2024-04-25T15:43:50.979 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T15:43:50.979 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235 --verbose --name pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi --owner scheduled_pdonnell@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} -- /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/orig.config.yaml 2024-04-25T15:43:51.034 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T15:43:51.127 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235 branch: wip-pdonnell-testing-20240425.015853-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} email: pdonnell@redhat.com first_in_suite: false job_id: '7673235' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240425.015853-debug ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported dry_run: true install: ceph: flavor: default sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240425.015853-debug sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1880 sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 sleep_before_teardown: 0 subset: 178/512 suite: fs suite_branch: wip-pdonnell-testing-20240425.015853-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChT6/LCQ1/LKPzuRgRC/VZuBjr67zixE3fSxe2aHFw5HtHrq5byzgAaFtV2zofLsx9KERq81U+b51VRTbVNU54= smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAS7EYXcVcYz06/nXE4TGDlAzeOlYvF8vs+pp68puOW0Y8X559RYoJgQyKDu+q/URZ1r/yhi6LX7ZDkStueOXI0= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDY64nFZK5zWYhFjmsS3fEjNZGNqtMaBaMNOdvQmFbrMEmUk+r7aKlDJGgUZpH6BAZN2zA6o1xU4Qn/50x0REoo= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/iogen.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_10:00:23 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T15:43:51.127 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa; will attempt to use it 2024-04-25T15:43:51.128 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_994e2f20102a20fa9a27966c14b625cc01884df8/qa/tasks 2024-04-25T15:43:51.128 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T15:43:51.130 INFO:teuthology.task.internal:Checking packages... 2024-04-25T15:43:51.152 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '994e2f20102a20fa9a27966c14b625cc01884df8' 2024-04-25T15:43:51.152 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T15:43:51.153 INFO:teuthology.packaging:ref: None 2024-04-25T15:43:51.153 INFO:teuthology.packaging:tag: None 2024-04-25T15:43:51.153 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240425.015853-debug 2024-04-25T15:43:51.153 INFO:teuthology.packaging:sha1: 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:43:51.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240425.015853-debug 2024-04-25T15:43:51.294 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3291.g994e2f20 2024-04-25T15:43:51.296 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T15:43:51.302 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T15:43:51.302 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T15:43:51.325 INFO:teuthology.task.internal:Saving configuration 2024-04-25T15:43:51.344 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T15:43:51.350 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T15:43:51.370 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:38:50.722697', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBChT6/LCQ1/LKPzuRgRC/VZuBjr67zixE3fSxe2aHFw5HtHrq5byzgAaFtV2zofLsx9KERq81U+b51VRTbVNU54='} 2024-04-25T15:43:51.390 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:38:50.724215', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAS7EYXcVcYz06/nXE4TGDlAzeOlYvF8vs+pp68puOW0Y8X559RYoJgQyKDu+q/URZ1r/yhi6LX7ZDkStueOXI0='} 2024-04-25T15:43:51.411 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 15:38:50.725398', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDY64nFZK5zWYhFjmsS3fEjNZGNqtMaBaMNOdvQmFbrMEmUk+r7aKlDJGgUZpH6BAZN2zA6o1xU4Qn/50x0REoo='} 2024-04-25T15:43:51.411 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T15:43:51.417 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-25T15:43:51.417 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-25T15:43:51.418 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-25T15:43:51.418 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T15:43:51.517 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f099fd8b0d0>, signals=[15]) 2024-04-25T15:43:51.517 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T15:43:51.523 INFO:teuthology.task.internal:Opening connections... 2024-04-25T15:43:51.524 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T15:43:51.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:43:51.604 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T15:43:51.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:43:51.677 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-04-25T15:43:51.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:43:51.751 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T15:43:51.757 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2024-04-25T15:43:51.778 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2024-04-25T15:43:51.778 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2024-04-25T15:43:51.836 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2024-04-25T15:43:51.836 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="9" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="9" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el9" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:LOGO="fedora-logo-icon" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T15:43:51.837 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T15:43:51.838 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2024-04-25T15:43:51.865 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-04-25T15:43:51.883 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-04-25T15:43:51.883 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-04-25T15:43:51.941 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-04-25T15:43:51.941 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-04-25T15:43:51.941 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T15:43:51.942 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T15:43:51.943 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-04-25T15:43:51.973 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-04-25T15:43:51.990 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-04-25T15:43:51.991 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-04-25T15:43:52.047 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-04-25T15:43:52.047 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-04-25T15:43:52.047 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-04-25T15:43:52.047 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-04-25T15:43:52.047 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-04-25T15:43:52.048 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-04-25T15:43:52.048 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-25T15:43:52.048 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-04-25T15:43:52.048 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-04-25T15:43:52.048 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-25T15:43:52.048 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-04-25T15:43:52.048 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-25T15:43:52.048 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-25T15:43:52.048 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-25T15:43:52.048 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-04-25T15:43:52.073 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T15:43:52.084 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T15:43:52.090 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T15:43:52.090 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2024-04-25T15:43:52.093 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-04-25T15:43:52.095 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-04-25T15:43:52.111 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T15:43:52.117 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T15:43:52.117 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2024-04-25T15:43:52.151 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-04-25T15:43:52.154 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-04-25T15:43:52.239 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T15:43:52.310 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T15:43:52.325 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-04-25T15:43:52.325 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-04-25T15:43:52.325 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-04-25T15:43:52.325 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T15:43:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:43:52.343 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2024-04-25T15:43:52.401 INFO:teuthology.orchestra.run.smithi031.stdout:5.14.0-437.el9.x86_64 2024-04-25T15:43:52.401 INFO:teuthology.task.kernel:Running kernel on smithi031: 5.14.0-437.el9.x86_64 2024-04-25T15:43:52.401 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2024-04-25T15:43:57.128 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 2.6 MB/s | 8.0 MB 00:03 2024-04-25T15:43:59.204 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-25T15:44:03.346 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-25T15:44:04.961 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-04-25T15:44:06.156 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-04-25T15:44:11.412 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T15:44:13.385 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-25T15:44:13.385 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-25T15:44:13.465 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-25T15:44:13.466 INFO:teuthology.orchestra.run.smithi031.stdout:Nothing to do. 2024-04-25T15:44:13.466 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-25T15:44:13.569 DEBUG:teuthology.orchestra.run.smithi031:> echo no | sudo yum reinstall kernel || true 2024-04-25T15:44:14.092 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 03:44:11 PM UTC. 2024-04-25T15:44:14.215 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-25T15:44:14.230 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T15:44:14.230 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-25T15:44:14.230 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T15:44:14.230 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalling: 2024-04-25T15:44:14.231 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:44:14.231 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:44:14.231 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-25T15:44:14.231 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T15:44:14.231 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:44:14.231 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 255 k 2024-04-25T15:44:14.231 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 0 2024-04-25T15:44:14.231 INFO:teuthology.orchestra.run.smithi031.stderr:Operation aborted. 2024-04-25T15:44:14.282 INFO:teuthology.orchestra.run.smithi031.stdout:Is this ok [y/N]: 2024-04-25T15:44:14.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum reinstall -y kernel || true 2024-04-25T15:44:14.799 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:00:03 ago on Thu 25 Apr 2024 03:44:11 PM UTC. 2024-04-25T15:44:14.919 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-25T15:44:14.920 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T15:44:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2024-04-25T15:44:14.920 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T15:44:14.920 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalling: 2024-04-25T15:44:14.920 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-25T15:44:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:44:14.921 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-25T15:44:14.921 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2024-04-25T15:44:14.921 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:44:14.921 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 255 k 2024-04-25T15:44:14.921 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 0 2024-04-25T15:44:14.921 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-25T15:44:15.164 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-437.el9.x86_64.rpm 2.0 MB/s | 255 kB 00:00 2024-04-25T15:44:15.164 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:44:15.164 INFO:teuthology.orchestra.run.smithi031.stdout:Total 1.0 MB/s | 255 kB 00:00 2024-04-25T15:44:15.185 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-25T15:44:15.395 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-25T15:44:15.395 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-25T15:44:15.400 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-25T15:44:15.401 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-25T15:44:15.531 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-25T15:44:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:44:15.607 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:44:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:44:16.257 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-25T15:44:16.923 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-25T15:44:16.923 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:44:16.923 INFO:teuthology.orchestra.run.smithi031.stdout:Reinstalled: 2024-04-25T15:44:16.923 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-25T15:44:16.923 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:44:16.923 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-25T15:44:17.102 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2024-04-25T15:44:17.150 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-25T15:44:17.151 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-25T15:44:17.151 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-25T15:44:17.151 DEBUG:teuthology.parallel:result is None 2024-04-25T15:44:17.152 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T15:44:17.158 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T15:44:17.159 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T15:44:17.161 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T15:44:17.163 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T15:44:17.182 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T15:44:17.188 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T15:44:17.196 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T15:44:17.196 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T15:44:17.216 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T15:44:17.222 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T15:44:17.342 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T15:44:17.349 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T15:44:17.349 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T15:44:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:44:17.367 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T15:44:17.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:44:17.384 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T15:44:17.386 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T15:44:17.410 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T15:44:17.434 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:44:17.443 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:44:17.447 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:44:17.460 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:44:17.472 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:44:17.482 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T15:44:17.483 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T15:44:17.491 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T15:44:17.491 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T15:44:17.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T15:44:17.516 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T15:44:17.550 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T15:44:17.558 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T15:44:17.559 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T15:44:17.561 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T15:44:17.584 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T15:44:17.607 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:44:17.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:44:17.708 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:44:17.788 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:44:17.828 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:44:17.828 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T15:44:17.892 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:44:17.935 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:44:17.989 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:44:18.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:44:18.115 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:44:18.115 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T15:44:18.181 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:44:18.223 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T15:44:18.277 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:44:18.359 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T15:44:18.402 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:44:18.402 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T15:44:18.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2024-04-25T15:44:18.470 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-04-25T15:44:18.472 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-04-25T15:44:18.505 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:44:18.508 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:44:18.546 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T15:44:18.993 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T15:44:19.001 INFO:teuthology.task.internal:Starting timer... 2024-04-25T15:44:19.001 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T15:44:19.036 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T15:44:19.044 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-25T15:44:19.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2024-04-25T15:44:19.100 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2024-04-25T15:44:19.101 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-04-25T15:44:19.162 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-04-25T15:44:19.164 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-04-25T15:44:19.231 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-04-25T15:44:19.233 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T15:44:19.233 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2024-04-25T15:44:19.253 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2024-04-25T15:44:19.253 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-04-25T15:44:19.278 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-04-25T15:44:19.278 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-04-25T15:44:19.301 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-04-25T15:44:19.301 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2024-04-25T15:44:19.301 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:44:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:44:19.328 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:44:19.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:44:19.356 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T15:44:19.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T15:44:19.382 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T15:44:19.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2024-04-25T15:44:19.407 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-04-25T15:44:19.433 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-04-25T15:44:19.458 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T15:44:19.467 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T15:44:19.482 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T15:44:19.484 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi063.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T15:48:10.148 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2024-04-25T15:48:10.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-25T15:48:10.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:48:10.230 DEBUG:teuthology.orchestra.run.smithi031:> true 2024-04-25T15:48:10.306 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2024-04-25T15:48:10.306 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-25T15:48:10.307 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:48:10.381 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-04-25T15:48:10.458 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-04-25T15:48:10.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-25T15:48:10.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T15:48:10.534 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-04-25T15:48:10.613 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-04-25T15:48:10.614 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T15:48:10.627 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T15:48:10.628 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T15:48:10.628 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:48:10.631 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T15:48:10.631 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:48:10.633 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T15:48:10.633 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T15:48:10.659 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T15:48:10.661 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T15:48:10.674 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T15:48:10.675 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T15:48:10.683 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-25T15:48:10.697 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-25T15:48:10.698 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2024-04-25T15:48:10.699 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-04-25T15:48:10.710 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2024-04-25T15:48:10.710 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-04-25T15:48:10.720 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-04-25T15:48:10.723 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T15:48:10.724 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T15:48:10.730 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-04-25T15:48:10.737 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T15:48:10.738 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T15:48:10.743 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-25T15:48:10.756 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-25T15:48:10.793 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:48:10.793 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:48:10.796 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:48:10.796 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-25T15:48:10.796 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.796 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.797 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.797 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.798 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:48:10.798 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-04-25T15:48:10.798 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.798 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.798 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.798 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.812 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-25T15:48:10.816 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T15:48:10.816 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-25T15:48:10.816 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.816 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.816 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.816 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T15:48:10.817 INFO:teuthology.run_tasks:Running task install... 2024-04-25T15:48:10.828 DEBUG:teuthology.task.install:project ceph 2024-04-25T15:48:10.829 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'}} 2024-04-25T15:48:10.829 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8'} 2024-04-25T15:48:10.829 INFO:teuthology.task.install:Using flavor: default 2024-04-25T15:48:10.834 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T15:48:10.835 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-25T15:48:10.835 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T15:48:10.835 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:48:10.837 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T15:48:10.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:48:10.838 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'tag': None, 'wait_for_package': False} 2024-04-25T15:48:10.839 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:48:10.985 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T15:48:10.985 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T15:48:10.989 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T15:48:10.989 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T15:48:11.058 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/ 2024-04-25T15:48:11.058 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3291.g994e2f20 2024-04-25T15:48:11.156 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T15:48:11.157 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:48:11.157 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T15:48:11.162 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T15:48:11.162 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:48:11.162 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T15:48:11.182 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T15:48:11.183 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T15:48:11.188 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T15:48:11.188 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T15:48:11.228 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-25T15:48:11.228 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:48:11.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-25T15:48:11.247 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T15:48:11.253 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-25T15:48:11.254 DEBUG:teuthology.orchestra.run.smithi031:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/994e2f20102a20fa9a27966c14b625cc01884df8/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-25T15:48:11.255 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T15:48:11.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-25T15:48:11.321 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T15:48:11.330 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T15:48:11.386 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-04-25T15:48:11.386 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-04-25T15:48:11.395 DEBUG:teuthology.orchestra.run.smithi031:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-25T15:48:11.395 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-04-25T15:48:11.396 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-04-25T15:48:11.459 INFO:teuthology.orchestra.run.smithi031.stdout:check_obsoletes = 1 2024-04-25T15:48:11.460 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum clean all 2024-04-25T15:48:11.612 INFO:teuthology.orchestra.run.smithi186.stdout:48 files removed 2024-04-25T15:48:11.630 INFO:teuthology.orchestra.run.smithi063.stdout:48 files removed 2024-04-25T15:48:11.638 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T15:48:11.658 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T15:48:11.693 INFO:teuthology.orchestra.run.smithi031.stdout:48 files removed 2024-04-25T15:48:11.720 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-25T15:48:12.304 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 207 kB/s | 84 kB 00:00 2024-04-25T15:48:12.418 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 172 kB/s | 84 kB 00:00 2024-04-25T15:48:12.663 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-04-25T15:48:12.793 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-04-25T15:48:13.038 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-04-25T15:48:13.219 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-04-25T15:48:13.741 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-25T15:48:13.755 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-04-25T15:48:14.650 INFO:teuthology.orchestra.run.smithi031.stdout:ceph packages for x86_64 31 kB/s | 84 kB 00:02 2024-04-25T15:48:15.392 INFO:teuthology.orchestra.run.smithi031.stdout:ceph noarch packages 17 kB/s | 12 kB 00:00 2024-04-25T15:48:15.534 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-25T15:48:15.840 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-04-25T15:48:16.044 INFO:teuthology.orchestra.run.smithi031.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-04-25T15:48:16.821 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-04-25T15:48:18.948 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-04-25T15:48:19.901 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 6.2 MB 00:00 2024-04-25T15:48:20.257 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 9.2 MB/s | 6.2 MB 00:00 2024-04-25T15:48:21.500 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-25T15:48:21.797 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-04-25T15:48:22.768 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-25T15:48:23.023 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-04-25T15:48:23.206 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-04-25T15:48:24.996 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-25T15:48:27.818 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-25T15:48:28.290 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-25T15:48:29.222 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 5.1 MB/s | 21 MB 00:04 2024-04-25T15:48:30.036 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:48:30.036 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:48:30.047 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T15:48:30.048 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T15:48:30.049 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T15:48:30.050 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T15:48:30.051 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T15:48:30.054 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T15:48:30.054 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T15:48:30.116 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-04-25T15:48:30.130 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-25T15:48:30.130 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-04-25T15:48:30.130 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-25T15:48:30.130 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-04-25T15:48:30.130 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T15:48:30.130 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T15:48:30.130 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T15:48:30.131 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T15:48:30.132 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T15:48:30.133 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T15:48:30.134 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T15:48:30.135 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T15:48:30.136 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T15:48:30.137 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T15:48:30.138 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T15:48:30.139 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T15:48:30.140 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T15:48:30.141 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T15:48:30.142 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T15:48:30.143 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout:Install 148 Packages 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 4 Packages 2024-04-25T15:48:30.144 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:48:30.145 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 214 M 2024-04-25T15:48:30.145 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-04-25T15:48:30.517 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:48:30.518 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:48:30.529 INFO:teuthology.orchestra.run.smithi063.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T15:48:30.530 INFO:teuthology.orchestra.run.smithi063.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T15:48:30.531 INFO:teuthology.orchestra.run.smithi063.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T15:48:30.532 INFO:teuthology.orchestra.run.smithi063.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T15:48:30.533 INFO:teuthology.orchestra.run.smithi063.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T15:48:30.536 INFO:teuthology.orchestra.run.smithi063.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T15:48:30.536 INFO:teuthology.orchestra.run.smithi063.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T15:48:30.600 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-04-25T15:48:30.619 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-04-25T15:48:30.619 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-04-25T15:48:30.619 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-04-25T15:48:30.619 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T15:48:30.620 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T15:48:30.621 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-04-25T15:48:30.622 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T15:48:30.623 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T15:48:30.624 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T15:48:30.625 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T15:48:30.626 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T15:48:30.627 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T15:48:30.628 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T15:48:30.629 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T15:48:30.630 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T15:48:30.630 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T15:48:30.630 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T15:48:30.630 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T15:48:30.630 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T15:48:30.631 INFO:teuthology.orchestra.run.smithi063.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T15:48:30.632 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-04-25T15:48:30.632 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T15:48:30.632 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:48:30.632 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-04-25T15:48:30.632 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-04-25T15:48:30.632 INFO:teuthology.orchestra.run.smithi063.stdout:Install 148 Packages 2024-04-25T15:48:30.632 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 4 Packages 2024-04-25T15:48:30.632 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:48:30.633 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 214 M 2024-04-25T15:48:30.633 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-04-25T15:48:31.077 INFO:teuthology.orchestra.run.smithi186.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-25T15:48:31.269 INFO:teuthology.orchestra.run.smithi186.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 4.4 MB/s | 863 kB 00:00 2024-04-25T15:48:31.319 INFO:teuthology.orchestra.run.smithi186.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 2.9 MB/s | 147 kB 00:00 2024-04-25T15:48:31.636 INFO:teuthology.orchestra.run.smithi186.stdout:(4/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 7.1 MB/s | 2.3 MB 00:00 2024-04-25T15:48:31.674 INFO:teuthology.orchestra.run.smithi063.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 26 kB/s | 6.6 kB 00:00 2024-04-25T15:48:31.711 INFO:teuthology.orchestra.run.smithi186.stdout:(5/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 6.7 MB/s | 5.4 MB 00:00 2024-04-25T15:48:31.928 INFO:teuthology.orchestra.run.smithi186.stdout:(6/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 5.2 MB/s | 1.5 MB 00:00 2024-04-25T15:48:32.370 INFO:teuthology.orchestra.run.smithi186.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 7.4 MB/s | 4.9 MB 00:00 2024-04-25T15:48:32.566 INFO:teuthology.orchestra.run.smithi063.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 968 kB/s | 863 kB 00:00 2024-04-25T15:48:32.769 INFO:teuthology.orchestra.run.smithi063.stdout:(3/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 4.0 MB/s | 5.4 MB 00:01 2024-04-25T15:48:32.794 INFO:teuthology.orchestra.run.smithi063.stdout:(4/152): ceph-immutable-object-cache-19.0.0-329 646 kB/s | 147 kB 00:00 2024-04-25T15:48:33.320 INFO:teuthology.orchestra.run.smithi063.stdout:(5/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 4.1 MB/s | 2.3 MB 00:00 2024-04-25T15:48:34.045 INFO:teuthology.orchestra.run.smithi186.stdout:(8/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 6.0 MB/s | 10 MB 00:01 2024-04-25T15:48:34.112 INFO:teuthology.orchestra.run.smithi186.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 379 kB/s | 25 kB 00:00 2024-04-25T15:48:34.144 INFO:teuthology.orchestra.run.smithi063.stdout:(6/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 5.9 MB/s | 4.9 MB 00:00 2024-04-25T15:48:34.469 INFO:teuthology.orchestra.run.smithi063.stdout:(7/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 919 kB/s | 1.5 MB 00:01 2024-04-25T15:48:34.496 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-04-25T15:48:35.087 INFO:teuthology.orchestra.run.smithi186.stdout:(10/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 5.6 MB/s | 18 MB 00:03 2024-04-25T15:48:35.179 INFO:teuthology.orchestra.run.smithi186.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.4 MB/s | 224 kB 00:00 2024-04-25T15:48:35.254 INFO:teuthology.orchestra.run.smithi186.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 361 kB/s | 27 kB 00:00 2024-04-25T15:48:35.338 INFO:teuthology.orchestra.run.smithi186.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 379 kB/s | 32 kB 00:00 2024-04-25T15:48:35.539 INFO:teuthology.orchestra.run.smithi186.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 3.7 MB/s | 756 kB 00:00 2024-04-25T15:48:35.622 INFO:teuthology.orchestra.run.smithi186.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 1.9 MB/s | 158 kB 00:00 2024-04-25T15:48:35.706 INFO:teuthology.orchestra.run.smithi186.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.5 MB/s | 127 kB 00:00 2024-04-25T15:48:35.914 INFO:teuthology.orchestra.run.smithi186.stdout:(17/152): ceph-common-19.0.0-3291.g994e2f20.el9 3.9 MB/s | 20 MB 00:05 2024-04-25T15:48:35.956 INFO:teuthology.orchestra.run.smithi186.stdout:(18/152): libradosstriper1-19.0.0-3291.g994e2f2 1.9 MB/s | 497 kB 00:00 2024-04-25T15:48:36.032 INFO:teuthology.orchestra.run.smithi186.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 601 kB/s | 45 kB 00:00 2024-04-25T15:48:36.090 INFO:teuthology.orchestra.run.smithi186.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 2.3 MB/s | 135 kB 00:00 2024-04-25T15:48:36.158 INFO:teuthology.orchestra.run.smithi186.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.4 MB/s | 161 kB 00:00 2024-04-25T15:48:36.178 INFO:teuthology.orchestra.run.smithi063.stdout:(8/152): ceph-common-19.0.0-3291.g994e2f20.el9. 4.1 MB/s | 20 MB 00:04 2024-04-25T15:48:36.233 INFO:teuthology.orchestra.run.smithi186.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 4.2 MB/s | 321 kB 00:00 2024-04-25T15:48:36.261 INFO:teuthology.orchestra.run.smithi063.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 303 kB/s | 25 kB 00:00 2024-04-25T15:48:36.300 INFO:teuthology.orchestra.run.smithi186.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 4.4 MB/s | 298 kB 00:00 2024-04-25T15:48:36.358 INFO:teuthology.orchestra.run.smithi186.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.7 MB/s | 100 kB 00:00 2024-04-25T15:48:36.425 INFO:teuthology.orchestra.run.smithi186.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 86 kB 00:00 2024-04-25T15:48:36.428 INFO:teuthology.orchestra.run.smithi063.stdout:(10/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x8 7.8 MB/s | 18 MB 00:02 2024-04-25T15:48:36.512 INFO:teuthology.orchestra.run.smithi063.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.6 MB/s | 224 kB 00:00 2024-04-25T15:48:36.587 INFO:teuthology.orchestra.run.smithi063.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 361 kB/s | 27 kB 00:00 2024-04-25T15:48:36.654 INFO:teuthology.orchestra.run.smithi063.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 476 kB/s | 32 kB 00:00 2024-04-25T15:48:36.715 INFO:teuthology.orchestra.run.smithi031.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:48:36.716 INFO:teuthology.orchestra.run.smithi031.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-25T15:48:36.727 INFO:teuthology.orchestra.run.smithi031.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-25T15:48:36.728 INFO:teuthology.orchestra.run.smithi031.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-25T15:48:36.729 INFO:teuthology.orchestra.run.smithi031.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-25T15:48:36.729 INFO:teuthology.orchestra.run.smithi031.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-25T15:48:36.734 INFO:teuthology.orchestra.run.smithi031.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-25T15:48:36.734 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-25T15:48:36.734 INFO:teuthology.orchestra.run.smithi031.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-25T15:48:36.798 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2024-04-25T15:48:36.804 INFO:teuthology.orchestra.run.smithi063.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 4.9 MB/s | 756 kB 00:00 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: ceph x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 6.6 k 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.4 M 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 863 k 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 147 k 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 1.5 M 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 142 k 2024-04-25T15:48:36.808 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 3.5 M 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 7.4 M 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 49 k 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 10 M 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 43 M 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 273 k 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 762 k 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 224 k 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 27 k 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 18 k 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-25T15:48:36.809 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 32 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 756 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 127 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 161 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 321 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 298 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 100 k 2024-04-25T15:48:36.810 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 86 k 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.0 M 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 169 k 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrading: 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: librados2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.3 M 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 3.1 M 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 20 M 2024-04-25T15:48:36.811 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 34 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 2.3 M 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 249 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 4.9 M 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 18 M 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3291.g994e2f20.el9 ceph-noarch 16 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 25 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-25T15:48:36.812 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 158 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 497 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2 x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 5.2 M 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-25T15:48:36.813 INFO:teuthology.orchestra.run.smithi031.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 45 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common x86_64 2:19.0.0-3291.g994e2f20.el9 ceph 135 k 2024-04-25T15:48:36.814 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-25T15:48:36.815 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-25T15:48:36.816 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-25T15:48:36.817 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-25T15:48:36.818 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout:======================================================================================= 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout:Install 148 Packages 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout:Upgrade 4 Packages 2024-04-25T15:48:36.819 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:48:36.820 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 214 M 2024-04-25T15:48:36.820 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2024-04-25T15:48:36.880 INFO:teuthology.orchestra.run.smithi063.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 2.1 MB/s | 158 kB 00:00 2024-04-25T15:48:36.934 INFO:teuthology.orchestra.run.smithi186.stdout:(26/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 6.0 MB/s | 3.0 MB 00:00 2024-04-25T15:48:36.938 INFO:teuthology.orchestra.run.smithi063.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 2.1 MB/s | 127 kB 00:00 2024-04-25T15:48:37.009 INFO:teuthology.orchestra.run.smithi186.stdout:(27/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 4.7 MB/s | 5.2 MB 00:01 2024-04-25T15:48:37.013 INFO:teuthology.orchestra.run.smithi063.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 6.5 MB/s | 497 kB 00:00 2024-04-25T15:48:37.043 INFO:teuthology.orchestra.run.smithi186.stdout:(28/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 1.5 MB/s | 169 kB 00:00 2024-04-25T15:48:37.077 INFO:teuthology.orchestra.run.smithi186.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3291.g 510 kB/s | 34 kB 00:00 2024-04-25T15:48:37.118 INFO:teuthology.orchestra.run.smithi186.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 1.8 MB/s | 142 kB 00:00 2024-04-25T15:48:37.747 INFO:teuthology.orchestra.run.smithi063.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 7.0 MB/s | 5.2 MB 00:00 2024-04-25T15:48:37.822 INFO:teuthology.orchestra.run.smithi063.stdout:(19/152): python3-ceph-argparse-19.0.0-3291.g99 603 kB/s | 45 kB 00:00 2024-04-25T15:48:37.898 INFO:teuthology.orchestra.run.smithi063.stdout:(20/152): python3-ceph-common-19.0.0-3291.g994e 1.8 MB/s | 135 kB 00:00 2024-04-25T15:48:37.973 INFO:teuthology.orchestra.run.smithi063.stdout:(21/152): python3-cephfs-19.0.0-3291.g994e2f20. 2.1 MB/s | 161 kB 00:00 2024-04-25T15:48:37.977 INFO:teuthology.orchestra.run.smithi186.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 3.9 MB/s | 3.5 MB 00:00 2024-04-25T15:48:37.998 INFO:teuthology.orchestra.run.smithi031.stdout:(1/152): ceph-19.0.0-3291.g994e2f20.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-04-25T15:48:38.040 INFO:teuthology.orchestra.run.smithi063.stdout:(22/152): python3-rados-19.0.0-3291.g994e2f20.e 4.7 MB/s | 321 kB 00:00 2024-04-25T15:48:38.077 INFO:teuthology.orchestra.run.smithi186.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.4 MB/s | 249 kB 00:00 2024-04-25T15:48:38.115 INFO:teuthology.orchestra.run.smithi063.stdout:(23/152): python3-rbd-19.0.0-3291.g994e2f20.el9 3.9 MB/s | 298 kB 00:00 2024-04-25T15:48:38.161 INFO:teuthology.orchestra.run.smithi186.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 592 kB/s | 49 kB 00:00 2024-04-25T15:48:38.190 INFO:teuthology.orchestra.run.smithi063.stdout:(24/152): python3-rgw-19.0.0-3291.g994e2f20.el9 1.3 MB/s | 100 kB 00:00 2024-04-25T15:48:38.236 INFO:teuthology.orchestra.run.smithi186.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 219 kB/s | 16 kB 00:00 2024-04-25T15:48:38.257 INFO:teuthology.orchestra.run.smithi063.stdout:(25/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 86 kB 00:00 2024-04-25T15:48:38.370 INFO:teuthology.orchestra.run.smithi186.stdout:(35/152): ceph-mgr-diskprediction-local-19.0.0- 5.9 MB/s | 7.4 MB 00:01 2024-04-25T15:48:38.395 INFO:teuthology.orchestra.run.smithi186.stdout:(36/152): ceph-volume-19.0.0-3291.g994e2f20.el9 1.7 MB/s | 273 kB 00:00 2024-04-25T15:48:38.487 INFO:teuthology.orchestra.run.smithi186.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 199 kB/s | 18 kB 00:00 2024-04-25T15:48:38.579 INFO:teuthology.orchestra.run.smithi186.stdout:(38/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 3.6 MB/s | 762 kB 00:00 2024-04-25T15:48:38.604 INFO:teuthology.orchestra.run.smithi186.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 687 kB/s | 80 kB 00:00 2024-04-25T15:48:38.656 INFO:teuthology.orchestra.run.smithi031.stdout:(2/152): ceph-fuse-19.0.0-3291.g994e2f20.el9.x8 1.3 MB/s | 863 kB 00:00 2024-04-25T15:48:38.657 INFO:teuthology.orchestra.run.smithi186.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 10 MB/s | 803 kB 00:00 2024-04-25T15:48:38.682 INFO:teuthology.orchestra.run.smithi186.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2024-04-25T15:48:38.707 INFO:teuthology.orchestra.run.smithi186.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-25T15:48:38.707 INFO:teuthology.orchestra.run.smithi063.stdout:(26/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 6.8 MB/s | 3.0 MB 00:00 2024-04-25T15:48:38.732 INFO:teuthology.orchestra.run.smithi031.stdout:(3/152): ceph-immutable-object-cache-19.0.0-329 1.9 MB/s | 147 kB 00:00 2024-04-25T15:48:38.741 INFO:teuthology.orchestra.run.smithi186.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 21 MB/s | 1.2 MB 00:00 2024-04-25T15:48:38.766 INFO:teuthology.orchestra.run.smithi063.stdout:(27/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 2.8 MB/s | 169 kB 00:00 2024-04-25T15:48:38.766 INFO:teuthology.orchestra.run.smithi186.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-25T15:48:38.791 INFO:teuthology.orchestra.run.smithi186.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T15:48:38.816 INFO:teuthology.orchestra.run.smithi063.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 679 kB/s | 34 kB 00:00 2024-04-25T15:48:38.825 INFO:teuthology.orchestra.run.smithi186.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-25T15:48:38.850 INFO:teuthology.orchestra.run.smithi186.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-04-25T15:48:38.875 INFO:teuthology.orchestra.run.smithi063.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 2.4 MB/s | 142 kB 00:00 2024-04-25T15:48:38.875 INFO:teuthology.orchestra.run.smithi186.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 13 MB/s | 689 kB 00:00 2024-04-25T15:48:38.917 INFO:teuthology.orchestra.run.smithi186.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-04-25T15:48:38.943 INFO:teuthology.orchestra.run.smithi186.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-25T15:48:38.968 INFO:teuthology.orchestra.run.smithi186.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-25T15:48:38.993 INFO:teuthology.orchestra.run.smithi186.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T15:48:39.019 INFO:teuthology.orchestra.run.smithi186.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 659 kB/s | 33 kB 00:00 2024-04-25T15:48:39.094 INFO:teuthology.orchestra.run.smithi186.stdout:(54/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-04-25T15:48:39.119 INFO:teuthology.orchestra.run.smithi186.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 186 kB/s | 18 kB 00:00 2024-04-25T15:48:39.144 INFO:teuthology.orchestra.run.smithi186.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-04-25T15:48:39.178 INFO:teuthology.orchestra.run.smithi186.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-25T15:48:39.195 INFO:teuthology.orchestra.run.smithi186.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 907 kB/s | 45 kB 00:00 2024-04-25T15:48:39.228 INFO:teuthology.orchestra.run.smithi186.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T15:48:39.262 INFO:teuthology.orchestra.run.smithi186.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-04-25T15:48:39.279 INFO:teuthology.orchestra.run.smithi186.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 515 kB/s | 25 kB 00:00 2024-04-25T15:48:39.305 INFO:teuthology.orchestra.run.smithi186.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-25T15:48:39.355 INFO:teuthology.orchestra.run.smithi186.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-25T15:48:39.358 INFO:teuthology.orchestra.run.smithi063.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 7.3 MB/s | 3.5 MB 00:00 2024-04-25T15:48:39.380 INFO:teuthology.orchestra.run.smithi186.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 188 kB 00:00 2024-04-25T15:48:39.405 INFO:teuthology.orchestra.run.smithi186.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 709 kB/s | 35 kB 00:00 2024-04-25T15:48:39.449 INFO:teuthology.orchestra.run.smithi031.stdout:(4/152): ceph-mds-19.0.0-3291.g994e2f20.el9.x86 3.2 MB/s | 2.3 MB 00:00 2024-04-25T15:48:39.766 INFO:teuthology.orchestra.run.smithi031.stdout:(5/152): ceph-mgr-19.0.0-3291.g994e2f20.el9.x86 4.8 MB/s | 1.5 MB 00:00 2024-04-25T15:48:39.856 INFO:teuthology.orchestra.run.smithi186.stdout:(66/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 7.5 MB/s | 43 MB 00:05 2024-04-25T15:48:39.923 INFO:teuthology.orchestra.run.smithi186.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 8.7 MB/s | 4.7 MB 00:00 2024-04-25T15:48:39.998 INFO:teuthology.orchestra.run.smithi186.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-04-25T15:48:40.040 INFO:teuthology.orchestra.run.smithi186.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 8.8 MB/s | 1.6 MB 00:00 2024-04-25T15:48:40.065 INFO:teuthology.orchestra.run.smithi186.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 6.4 MB/s | 911 kB 00:00 2024-04-25T15:48:40.091 INFO:teuthology.orchestra.run.smithi186.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.5 MB/s | 133 kB 00:00 2024-04-25T15:48:40.174 INFO:teuthology.orchestra.run.smithi186.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-04-25T15:48:40.234 INFO:teuthology.orchestra.run.smithi063.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 8.4 MB/s | 7.4 MB 00:00 2024-04-25T15:48:40.266 INFO:teuthology.orchestra.run.smithi186.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 29 MB/s | 5.8 MB 00:00 2024-04-25T15:48:40.291 INFO:teuthology.orchestra.run.smithi186.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 441 kB/s | 87 kB 00:00 2024-04-25T15:48:40.334 INFO:teuthology.orchestra.run.smithi063.stdout:(32/152): ceph-radosgw-19.0.0-3291.g994e2f20.el 1.7 MB/s | 10 MB 00:05 2024-04-25T15:48:40.340 INFO:teuthology.orchestra.run.smithi031.stdout:(6/152): ceph-base-19.0.0-3291.g994e2f20.el9.x8 2.1 MB/s | 5.4 MB 00:02 2024-04-25T15:48:40.368 INFO:teuthology.orchestra.run.smithi063.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3291.g99 1.8 MB/s | 249 kB 00:00 2024-04-25T15:48:40.383 INFO:teuthology.orchestra.run.smithi186.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 29 MB/s | 6.0 MB 00:00 2024-04-25T15:48:40.393 INFO:teuthology.orchestra.run.smithi063.stdout:(34/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 844 kB/s | 49 kB 00:00 2024-04-25T15:48:40.409 INFO:teuthology.orchestra.run.smithi186.stdout:(76/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.7 MB/s | 244 kB 00:00 2024-04-25T15:48:40.418 INFO:teuthology.orchestra.run.smithi063.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3291.g9 328 kB/s | 16 kB 00:00 2024-04-25T15:48:40.432 INFO:teuthology.orchestra.run.smithi031.stdout:(7/152): ceph-mon-19.0.0-3291.g994e2f20.el9.x86 7.3 MB/s | 4.9 MB 00:00 2024-04-25T15:48:40.434 INFO:teuthology.orchestra.run.smithi186.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-04-25T15:48:40.459 INFO:teuthology.orchestra.run.smithi186.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 345 kB/s | 26 kB 00:00 2024-04-25T15:48:40.484 INFO:teuthology.orchestra.run.smithi186.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-25T15:48:40.509 INFO:teuthology.orchestra.run.smithi186.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-25T15:48:40.519 INFO:teuthology.orchestra.run.smithi063.stdout:(36/152): ceph-volume-19.0.0-3291.g994e2f20.el9 2.1 MB/s | 273 kB 00:00 2024-04-25T15:48:40.535 INFO:teuthology.orchestra.run.smithi186.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 463 kB/s | 35 kB 00:00 2024-04-25T15:48:40.553 INFO:teuthology.orchestra.run.smithi063.stdout:(37/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 5.6 MB/s | 762 kB 00:00 2024-04-25T15:48:40.585 INFO:teuthology.orchestra.run.smithi186.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-04-25T15:48:40.611 INFO:teuthology.orchestra.run.smithi063.stdout:(38/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 199 kB/s | 18 kB 00:00 2024-04-25T15:48:40.652 INFO:teuthology.orchestra.run.smithi186.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 5.1 MB 00:00 2024-04-25T15:48:40.677 INFO:teuthology.orchestra.run.smithi186.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-04-25T15:48:40.702 INFO:teuthology.orchestra.run.smithi186.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 675 kB/s | 77 kB 00:00 2024-04-25T15:48:40.728 INFO:teuthology.orchestra.run.smithi186.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 849 kB/s | 42 kB 00:00 2024-04-25T15:48:40.744 INFO:teuthology.orchestra.run.smithi063.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 5.9 MB/s | 803 kB 00:00 2024-04-25T15:48:40.753 INFO:teuthology.orchestra.run.smithi186.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 269 kB 00:00 2024-04-25T15:48:40.786 INFO:teuthology.orchestra.run.smithi186.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-04-25T15:48:40.787 INFO:teuthology.orchestra.run.smithi063.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 344 kB/s | 80 kB 00:00 2024-04-25T15:48:40.811 INFO:teuthology.orchestra.run.smithi186.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.3 MB/s | 279 kB 00:00 2024-04-25T15:48:40.812 INFO:teuthology.orchestra.run.smithi063.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-25T15:48:40.837 INFO:teuthology.orchestra.run.smithi063.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-04-25T15:48:40.837 INFO:teuthology.orchestra.run.smithi186.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T15:48:40.862 INFO:teuthology.orchestra.run.smithi186.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 720 kB/s | 54 kB 00:00 2024-04-25T15:48:40.904 INFO:teuthology.orchestra.run.smithi063.stdout:(43/152): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.2 MB 00:00 2024-04-25T15:48:40.921 INFO:teuthology.orchestra.run.smithi063.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 106 kB 00:00 2024-04-25T15:48:40.954 INFO:teuthology.orchestra.run.smithi063.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-25T15:48:40.979 INFO:teuthology.orchestra.run.smithi063.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-25T15:48:41.005 INFO:teuthology.orchestra.run.smithi063.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-25T15:48:41.047 INFO:teuthology.orchestra.run.smithi063.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 10 MB/s | 689 kB 00:00 2024-04-25T15:48:41.080 INFO:teuthology.orchestra.run.smithi063.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-04-25T15:48:41.106 INFO:teuthology.orchestra.run.smithi063.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-04-25T15:48:41.131 INFO:teuthology.orchestra.run.smithi063.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-25T15:48:41.156 INFO:teuthology.orchestra.run.smithi063.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-04-25T15:48:41.179 INFO:teuthology.orchestra.run.smithi186.stdout:(92/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-04-25T15:48:41.182 INFO:teuthology.orchestra.run.smithi063.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 661 kB/s | 33 kB 00:00 2024-04-25T15:48:41.204 INFO:teuthology.orchestra.run.smithi186.stdout:(93/152): python3-toml-0.10.2-6.el9.noarch.rpm 123 kB/s | 42 kB 00:00 2024-04-25T15:48:41.224 INFO:teuthology.orchestra.run.smithi063.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 453 kB/s | 18 kB 00:00 2024-04-25T15:48:41.229 INFO:teuthology.orchestra.run.smithi186.stdout:(94/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 121 kB/s | 44 kB 00:00 2024-04-25T15:48:41.263 INFO:teuthology.orchestra.run.smithi186.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-04-25T15:48:41.274 INFO:teuthology.orchestra.run.smithi063.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-04-25T15:48:41.283 INFO:teuthology.orchestra.run.smithi031.stdout:(8/152): ceph-radosgw-19.0.0-3291.g994e2f20.el9 12 MB/s | 10 MB 00:00 2024-04-25T15:48:41.288 INFO:teuthology.orchestra.run.smithi186.stdout:(96/152): qatlib-service-24.02.0-1.el9.x86_64.r 618 kB/s | 36 kB 00:00 2024-04-25T15:48:41.299 INFO:teuthology.orchestra.run.smithi063.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 154 kB/s | 11 kB 00:00 2024-04-25T15:48:41.305 INFO:teuthology.orchestra.run.smithi186.stdout:(97/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 633 kB/s | 47 kB 00:00 2024-04-25T15:48:41.324 INFO:teuthology.orchestra.run.smithi063.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-25T15:48:41.330 INFO:teuthology.orchestra.run.smithi186.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-25T15:48:41.341 INFO:teuthology.orchestra.run.smithi063.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-04-25T15:48:41.349 INFO:teuthology.orchestra.run.smithi031.stdout:(9/152): ceph-selinux-19.0.0-3291.g994e2f20.el9 379 kB/s | 25 kB 00:00 2024-04-25T15:48:41.355 INFO:teuthology.orchestra.run.smithi186.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T15:48:41.375 INFO:teuthology.orchestra.run.smithi063.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T15:48:41.381 INFO:teuthology.orchestra.run.smithi186.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.2 MB/s | 94 kB 00:00 2024-04-25T15:48:41.406 INFO:teuthology.orchestra.run.smithi186.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 847 kB/s | 64 kB 00:00 2024-04-25T15:48:41.408 INFO:teuthology.orchestra.run.smithi063.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 582 kB 00:00 2024-04-25T15:48:41.425 INFO:teuthology.orchestra.run.smithi063.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 520 kB/s | 25 kB 00:00 2024-04-25T15:48:41.431 INFO:teuthology.orchestra.run.smithi186.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 890 kB/s | 45 kB 00:00 2024-04-25T15:48:41.443 INFO:teuthology.orchestra.run.smithi063.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-25T15:48:41.457 INFO:teuthology.orchestra.run.smithi186.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2024-04-25T15:48:41.476 INFO:teuthology.orchestra.run.smithi063.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-25T15:48:41.498 INFO:teuthology.orchestra.run.smithi186.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 9.2 MB/s | 863 kB 00:00 2024-04-25T15:48:41.501 INFO:teuthology.orchestra.run.smithi063.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-04-25T15:48:41.518 INFO:teuthology.orchestra.run.smithi063.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 852 kB/s | 35 kB 00:00 2024-04-25T15:48:41.635 INFO:teuthology.orchestra.run.smithi063.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 38 MB/s | 4.4 MB 00:00 2024-04-25T15:48:41.719 INFO:teuthology.orchestra.run.smithi186.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 130 kB/s | 19 kB 00:00 2024-04-25T15:48:41.736 INFO:teuthology.orchestra.run.smithi063.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-04-25T15:48:41.778 INFO:teuthology.orchestra.run.smithi063.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-25T15:48:41.819 INFO:teuthology.orchestra.run.smithi063.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-25T15:48:41.828 INFO:teuthology.orchestra.run.smithi186.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 837 kB/s | 309 kB 00:00 2024-04-25T15:48:41.845 INFO:teuthology.orchestra.run.smithi063.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-04-25T15:48:41.858 INFO:teuthology.orchestra.run.smithi186.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-04-25T15:48:41.891 INFO:teuthology.orchestra.run.smithi186.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 411 kB/s | 26 kB 00:00 2024-04-25T15:48:41.950 INFO:teuthology.orchestra.run.smithi186.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-04-25T15:48:41.953 INFO:teuthology.orchestra.run.smithi063.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 9.4 MB/s | 1.0 MB 00:00 2024-04-25T15:48:42.042 INFO:teuthology.orchestra.run.smithi186.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-25T15:48:42.070 INFO:teuthology.orchestra.run.smithi063.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-04-25T15:48:42.095 INFO:teuthology.orchestra.run.smithi063.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 629 kB/s | 87 kB 00:00 2024-04-25T15:48:42.137 INFO:teuthology.orchestra.run.smithi063.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 5.8 MB/s | 244 kB 00:00 2024-04-25T15:48:42.187 INFO:teuthology.orchestra.run.smithi063.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 5.1 MB/s | 248 kB 00:00 2024-04-25T15:48:42.296 INFO:teuthology.orchestra.run.smithi063.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 30 MB/s | 6.0 MB 00:00 2024-04-25T15:48:42.321 INFO:teuthology.orchestra.run.smithi063.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 195 kB/s | 26 kB 00:00 2024-04-25T15:48:42.346 INFO:teuthology.orchestra.run.smithi063.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 383 kB/s | 19 kB 00:00 2024-04-25T15:48:42.350 INFO:teuthology.orchestra.run.smithi186.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 147 kB/s | 67 kB 00:00 2024-04-25T15:48:42.372 INFO:teuthology.orchestra.run.smithi063.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-25T15:48:42.397 INFO:teuthology.orchestra.run.smithi063.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 696 kB/s | 35 kB 00:00 2024-04-25T15:48:42.409 INFO:teuthology.orchestra.run.smithi186.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-04-25T15:48:42.464 INFO:teuthology.orchestra.run.smithi063.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.7 MB/s | 250 kB 00:00 2024-04-25T15:48:42.484 INFO:teuthology.orchestra.run.smithi031.stdout:(10/152): ceph-common-19.0.0-3291.g994e2f20.el9 4.1 MB/s | 20 MB 00:04 2024-04-25T15:48:42.501 INFO:teuthology.orchestra.run.smithi186.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-04-25T15:48:42.548 INFO:teuthology.orchestra.run.smithi063.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-04-25T15:48:42.551 INFO:teuthology.orchestra.run.smithi186.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 646 kB/s | 32 kB 00:00 2024-04-25T15:48:42.558 INFO:teuthology.orchestra.run.smithi031.stdout:(11/152): cephfs-mirror-19.0.0-3291.g994e2f20.e 2.9 MB/s | 224 kB 00:00 2024-04-25T15:48:42.573 INFO:teuthology.orchestra.run.smithi063.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.1 MB/s | 222 kB 00:00 2024-04-25T15:48:42.598 INFO:teuthology.orchestra.run.smithi063.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T15:48:42.601 INFO:teuthology.orchestra.run.smithi186.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 301 kB/s | 15 kB 00:00 2024-04-25T15:48:42.624 INFO:teuthology.orchestra.run.smithi063.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-04-25T15:48:42.633 INFO:teuthology.orchestra.run.smithi031.stdout:(12/152): cephfs-shell-19.0.0-3291.g994e2f20.el 361 kB/s | 27 kB 00:00 2024-04-25T15:48:42.649 INFO:teuthology.orchestra.run.smithi063.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-25T15:48:42.674 INFO:teuthology.orchestra.run.smithi063.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T15:48:42.699 INFO:teuthology.orchestra.run.smithi063.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.5 MB/s | 279 kB 00:00 2024-04-25T15:48:42.710 INFO:teuthology.orchestra.run.smithi186.stdout:(116/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-25T15:48:42.725 INFO:teuthology.orchestra.run.smithi063.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-25T15:48:42.725 INFO:teuthology.orchestra.run.smithi031.stdout:(13/152): libcephfs-devel-19.0.0-3291.g994e2f20 347 kB/s | 32 kB 00:00 2024-04-25T15:48:42.744 INFO:teuthology.orchestra.run.smithi186.stdout:(117/152): python3-asyncssh-2.13.2-5.el9.noarch 1.4 MB/s | 548 kB 00:00 2024-04-25T15:48:42.750 INFO:teuthology.orchestra.run.smithi063.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.1 MB/s | 54 kB 00:00 2024-04-25T15:48:42.792 INFO:teuthology.orchestra.run.smithi063.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-25T15:48:42.834 INFO:teuthology.orchestra.run.smithi063.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.1 MB/s | 44 kB 00:00 2024-04-25T15:48:42.861 INFO:teuthology.orchestra.run.smithi186.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.3 MB/s | 358 kB 00:00 2024-04-25T15:48:42.878 INFO:teuthology.orchestra.run.smithi063.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 5.6 MB/s | 221 kB 00:00 2024-04-25T15:48:42.892 INFO:teuthology.orchestra.run.smithi031.stdout:(14/152): libcephfs2-19.0.0-3291.g994e2f20.el9. 4.4 MB/s | 756 kB 00:00 2024-04-25T15:48:42.911 INFO:teuthology.orchestra.run.smithi186.stdout:(119/152): python3-colorama-0.4.4-12.el9.noarch 707 kB/s | 35 kB 00:00 2024-04-25T15:48:42.917 INFO:teuthology.orchestra.run.smithi063.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 918 kB/s | 36 kB 00:00 2024-04-25T15:48:42.978 INFO:teuthology.orchestra.run.smithi186.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.3 MB/s | 303 kB 00:00 2024-04-25T15:48:42.985 INFO:teuthology.orchestra.run.smithi063.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 735 kB/s | 47 kB 00:00 2024-04-25T15:48:43.001 INFO:teuthology.orchestra.run.smithi031.stdout:(15/152): libcephsqlite-19.0.0-3291.g994e2f20.e 1.4 MB/s | 158 kB 00:00 2024-04-25T15:48:43.045 INFO:teuthology.orchestra.run.smithi186.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.5 MB/s | 206 kB 00:00 2024-04-25T15:48:43.101 INFO:teuthology.orchestra.run.smithi031.stdout:(16/152): librados-devel-19.0.0-3291.g994e2f20. 1.2 MB/s | 127 kB 00:00 2024-04-25T15:48:43.129 INFO:teuthology.orchestra.run.smithi186.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.7 MB/s | 144 kB 00:00 2024-04-25T15:48:43.143 INFO:teuthology.orchestra.run.smithi063.stdout:(96/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-04-25T15:48:43.169 INFO:teuthology.orchestra.run.smithi063.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-04-25T15:48:43.170 INFO:teuthology.orchestra.run.smithi186.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 257 kB/s | 11 kB 00:00 2024-04-25T15:48:43.194 INFO:teuthology.orchestra.run.smithi063.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-25T15:48:43.202 INFO:teuthology.orchestra.run.smithi031.stdout:(17/152): libradosstriper1-19.0.0-3291.g994e2f2 4.8 MB/s | 497 kB 00:00 2024-04-25T15:48:43.212 INFO:teuthology.orchestra.run.smithi186.stdout:(124/152): python3-jaraco-classes-3.2.1-5.el9.n 426 kB/s | 18 kB 00:00 2024-04-25T15:48:43.219 INFO:teuthology.orchestra.run.smithi063.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-25T15:48:43.236 INFO:teuthology.orchestra.run.smithi063.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-25T15:48:43.253 INFO:teuthology.orchestra.run.smithi063.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-25T15:48:43.254 INFO:teuthology.orchestra.run.smithi186.stdout:(125/152): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-04-25T15:48:43.270 INFO:teuthology.orchestra.run.smithi063.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 666 kB/s | 22 kB 00:00 2024-04-25T15:48:43.296 INFO:teuthology.orchestra.run.smithi186.stdout:(126/152): python3-jaraco-functools-3.5.0-2.el9 469 kB/s | 19 kB 00:00 2024-04-25T15:48:43.320 INFO:teuthology.orchestra.run.smithi063.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 13 MB/s | 863 kB 00:00 2024-04-25T15:48:43.338 INFO:teuthology.orchestra.run.smithi186.stdout:(127/152): python3-jaraco-text-3.2.0-6.el9.noar 473 kB/s | 20 kB 00:00 2024-04-25T15:48:43.531 INFO:teuthology.orchestra.run.smithi186.stdout:(128/152): python3-grpcio-1.46.7-10.el9.x86_64. 3.7 MB/s | 2.0 MB 00:00 2024-04-25T15:48:43.580 INFO:teuthology.orchestra.run.smithi186.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 931 kB/s | 46 kB 00:00 2024-04-25T15:48:43.631 INFO:teuthology.orchestra.run.smithi186.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-04-25T15:48:43.690 INFO:teuthology.orchestra.run.smithi186.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 2.9 MB/s | 1.0 MB 00:00 2024-04-25T15:48:43.715 INFO:teuthology.orchestra.run.smithi186.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 690 kB/s | 58 kB 00:00 2024-04-25T15:48:43.765 INFO:teuthology.orchestra.run.smithi186.stdout:(133/152): python3-portend-3.1.0-2.el9.noarch.r 329 kB/s | 16 kB 00:00 2024-04-25T15:48:43.770 INFO:teuthology.orchestra.run.smithi063.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 687 kB/s | 309 kB 00:00 2024-04-25T15:48:43.795 INFO:teuthology.orchestra.run.smithi063.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-25T15:48:43.824 INFO:teuthology.orchestra.run.smithi186.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-04-25T15:48:43.837 INFO:teuthology.orchestra.run.smithi063.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-04-25T15:48:43.849 INFO:teuthology.orchestra.run.smithi186.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-04-25T15:48:43.875 INFO:teuthology.orchestra.run.smithi186.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 563 kB/s | 28 kB 00:00 2024-04-25T15:48:43.900 INFO:teuthology.orchestra.run.smithi186.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-04-25T15:48:43.904 INFO:teuthology.orchestra.run.smithi063.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 389 kB/s | 26 kB 00:00 2024-04-25T15:48:43.919 INFO:teuthology.orchestra.run.smithi031.stdout:(18/152): librgw2-19.0.0-3291.g994e2f20.el9.x86 7.2 MB/s | 5.2 MB 00:00 2024-04-25T15:48:43.950 INFO:teuthology.orchestra.run.smithi186.stdout:(138/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-25T15:48:43.975 INFO:teuthology.orchestra.run.smithi186.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-25T15:48:43.988 INFO:teuthology.orchestra.run.smithi063.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 584 kB/s | 49 kB 00:00 2024-04-25T15:48:44.000 INFO:teuthology.orchestra.run.smithi186.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-04-25T15:48:44.059 INFO:teuthology.orchestra.run.smithi186.stdout:(141/152): libarrow-9.0.0-10.el9.x86_64.rpm 1.8 MB/s | 4.4 MB 00:02 2024-04-25T15:48:44.072 INFO:teuthology.orchestra.run.smithi063.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2024-04-25T15:48:44.085 INFO:teuthology.orchestra.run.smithi186.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 468 kB/s | 51 kB 00:00 2024-04-25T15:48:44.127 INFO:teuthology.orchestra.run.smithi186.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-04-25T15:48:44.152 INFO:teuthology.orchestra.run.smithi186.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 975 kB/s | 90 kB 00:00 2024-04-25T15:48:44.177 INFO:teuthology.orchestra.run.smithi186.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 400 kB/s | 20 kB 00:00 2024-04-25T15:48:44.245 INFO:teuthology.orchestra.run.smithi186.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.6 MB/s | 427 kB 00:00 2024-04-25T15:48:44.269 INFO:teuthology.orchestra.run.smithi186.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-25T15:48:44.352 INFO:teuthology.orchestra.run.smithi031.stdout:(19/152): ceph-test-19.0.0-3291.g994e2f20.el9.x 14 MB/s | 43 MB 00:03 2024-04-25T15:48:44.377 INFO:teuthology.orchestra.run.smithi031.stdout:(20/152): python3-ceph-argparse-19.0.0-3291.g99 98 kB/s | 45 kB 00:00 2024-04-25T15:48:44.419 INFO:teuthology.orchestra.run.smithi031.stdout:(21/152): python3-ceph-common-19.0.0-3291.g994e 2.0 MB/s | 135 kB 00:00 2024-04-25T15:48:44.439 INFO:teuthology.orchestra.run.smithi063.stdout:(110/152): ceph-test-19.0.0-3291.g994e2f20.el9. 5.3 MB/s | 43 MB 00:08 2024-04-25T15:48:44.464 INFO:teuthology.orchestra.run.smithi063.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 385 kB/s | 151 kB 00:00 2024-04-25T15:48:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:(22/152): python3-cephfs-19.0.0-3291.g994e2f20. 1.7 MB/s | 161 kB 00:00 2024-04-25T15:48:44.512 INFO:teuthology.orchestra.run.smithi031.stdout:(23/152): python3-rados-19.0.0-3291.g994e2f20.e 3.4 MB/s | 321 kB 00:00 2024-04-25T15:48:44.531 INFO:teuthology.orchestra.run.smithi063.stdout:(112/152): libarrow-9.0.0-10.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-04-25T15:48:44.578 INFO:teuthology.orchestra.run.smithi186.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-04-25T15:48:44.586 INFO:teuthology.orchestra.run.smithi031.stdout:(24/152): python3-rbd-19.0.0-3291.g994e2f20.el9 2.5 MB/s | 298 kB 00:00 2024-04-25T15:48:44.598 INFO:teuthology.orchestra.run.smithi063.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-04-25T15:48:44.612 INFO:teuthology.orchestra.run.smithi031.stdout:(25/152): python3-rgw-19.0.0-3291.g994e2f20.el9 996 kB/s | 100 kB 00:00 2024-04-25T15:48:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 695 kB/s | 43 kB 00:00 2024-04-25T15:48:44.636 INFO:teuthology.orchestra.run.smithi186.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-25T15:48:44.665 INFO:teuthology.orchestra.run.smithi063.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 482 kB/s | 32 kB 00:00 2024-04-25T15:48:44.690 INFO:teuthology.orchestra.run.smithi063.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 228 kB/s | 15 kB 00:00 2024-04-25T15:48:44.696 INFO:teuthology.orchestra.run.smithi031.stdout:(26/152): rbd-fuse-19.0.0-3291.g994e2f20.el9.x8 794 kB/s | 86 kB 00:00 2024-04-25T15:48:44.720 INFO:teuthology.orchestra.run.smithi186.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 29 MB/s | 2.4 MB 00:00 2024-04-25T15:48:44.749 INFO:teuthology.orchestra.run.smithi063.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-25T15:48:44.810 INFO:teuthology.orchestra.run.smithi063.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.9 MB/s | 358 kB 00:00 2024-04-25T15:48:44.841 INFO:teuthology.orchestra.run.smithi063.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-25T15:48:44.883 INFO:teuthology.orchestra.run.smithi063.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 488 kB/s | 35 kB 00:00 2024-04-25T15:48:44.924 INFO:teuthology.orchestra.run.smithi063.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-04-25T15:48:44.945 INFO:teuthology.orchestra.run.smithi186.stdout:(151/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 4.6 MB/s | 3.1 MB 00:00 2024-04-25T15:48:45.000 INFO:teuthology.orchestra.run.smithi063.stdout:(122/152): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-04-25T15:48:45.003 INFO:teuthology.orchestra.run.smithi186.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 4.4 MB/s | 3.3 MB 00:00 2024-04-25T15:48:45.008 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:48:45.008 INFO:teuthology.orchestra.run.smithi186.stdout:Total 14 MB/s | 214 MB 00:14 2024-04-25T15:48:45.025 INFO:teuthology.orchestra.run.smithi063.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-25T15:48:45.054 INFO:teuthology.orchestra.run.smithi031.stdout:(27/152): rbd-mirror-19.0.0-3291.g994e2f20.el9. 6.9 MB/s | 3.0 MB 00:00 2024-04-25T15:48:45.058 INFO:teuthology.orchestra.run.smithi063.stdout:(124/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-04-25T15:48:45.084 INFO:teuthology.orchestra.run.smithi063.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-04-25T15:48:45.096 INFO:teuthology.orchestra.run.smithi031.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3291.g 824 kB/s | 34 kB 00:00 2024-04-25T15:48:45.109 INFO:teuthology.orchestra.run.smithi063.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-04-25T15:48:45.135 INFO:teuthology.orchestra.run.smithi063.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 306 kB/s | 23 kB 00:00 2024-04-25T15:48:45.147 INFO:teuthology.orchestra.run.smithi031.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3291.g994e2f2 2.8 MB/s | 142 kB 00:00 2024-04-25T15:48:45.160 INFO:teuthology.orchestra.run.smithi063.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-25T15:48:45.185 INFO:teuthology.orchestra.run.smithi063.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 260 kB/s | 20 kB 00:00 2024-04-25T15:48:45.227 INFO:teuthology.orchestra.run.smithi063.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-04-25T15:48:45.252 INFO:teuthology.orchestra.run.smithi063.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-04-25T15:48:45.277 INFO:teuthology.orchestra.run.smithi063.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 858 kB/s | 79 kB 00:00 2024-04-25T15:48:45.303 INFO:teuthology.orchestra.run.smithi063.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 768 kB/s | 58 kB 00:00 2024-04-25T15:48:45.336 INFO:teuthology.orchestra.run.smithi063.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-25T15:48:45.362 INFO:teuthology.orchestra.run.smithi063.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-04-25T15:48:45.387 INFO:teuthology.orchestra.run.smithi063.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-04-25T15:48:45.412 INFO:teuthology.orchestra.run.smithi063.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-04-25T15:48:45.430 INFO:teuthology.orchestra.run.smithi031.stdout:(30/152): rbd-nbd-19.0.0-3291.g994e2f20.el9.x86 231 kB/s | 169 kB 00:00 2024-04-25T15:48:45.437 INFO:teuthology.orchestra.run.smithi063.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-04-25T15:48:45.463 INFO:teuthology.orchestra.run.smithi063.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-04-25T15:48:45.488 INFO:teuthology.orchestra.run.smithi063.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 783 kB/s | 59 kB 00:00 2024-04-25T15:48:45.513 INFO:teuthology.orchestra.run.smithi063.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-04-25T15:48:45.538 INFO:teuthology.orchestra.run.smithi063.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 678 kB/s | 51 kB 00:00 2024-04-25T15:48:45.539 INFO:teuthology.orchestra.run.smithi031.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3291.g994e2 9.0 MB/s | 3.5 MB 00:00 2024-04-25T15:48:45.614 INFO:teuthology.orchestra.run.smithi063.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-04-25T15:48:45.622 INFO:teuthology.orchestra.run.smithi031.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3291.g99 2.9 MB/s | 249 kB 00:00 2024-04-25T15:48:45.639 INFO:teuthology.orchestra.run.smithi063.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 714 kB/s | 90 kB 00:00 2024-04-25T15:48:45.664 INFO:teuthology.orchestra.run.smithi063.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-25T15:48:45.690 INFO:teuthology.orchestra.run.smithi063.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-25T15:48:45.697 INFO:teuthology.orchestra.run.smithi031.stdout:(33/152): ceph-mgr-rook-19.0.0-3291.g994e2f20.e 658 kB/s | 49 kB 00:00 2024-04-25T15:48:45.723 INFO:teuthology.orchestra.run.smithi063.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-25T15:48:45.765 INFO:teuthology.orchestra.run.smithi063.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-25T15:48:45.773 INFO:teuthology.orchestra.run.smithi031.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3291.g9 220 kB/s | 16 kB 00:00 2024-04-25T15:48:45.824 INFO:teuthology.orchestra.run.smithi063.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 23 MB/s | 1.3 MB 00:00 2024-04-25T15:48:45.856 INFO:teuthology.orchestra.run.smithi031.stdout:(35/152): ceph-volume-19.0.0-3291.g994e2f20.el9 3.2 MB/s | 273 kB 00:00 2024-04-25T15:48:45.916 INFO:teuthology.orchestra.run.smithi063.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-04-25T15:48:46.007 INFO:teuthology.orchestra.run.smithi031.stdout:(36/152): cephadm-19.0.0-3291.g994e2f20.el9.noa 5.0 MB/s | 762 kB 00:00 2024-04-25T15:48:46.090 INFO:teuthology.orchestra.run.smithi031.stdout:(37/152): cephfs-top-19.0.0-3291.g994e2f20.el9. 219 kB/s | 18 kB 00:00 2024-04-25T15:48:46.128 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-04-25T15:48:46.199 INFO:teuthology.orchestra.run.smithi031.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 743 kB/s | 80 kB 00:00 2024-04-25T15:48:46.273 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-04-25T15:48:46.273 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-04-25T15:48:46.274 INFO:teuthology.orchestra.run.smithi031.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 10 MB/s | 803 kB 00:00 2024-04-25T15:48:46.308 INFO:teuthology.orchestra.run.smithi031.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 5.7 MB/s | 195 kB 00:00 2024-04-25T15:48:46.349 INFO:teuthology.orchestra.run.smithi031.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 6.0 MB/s | 253 kB 00:00 2024-04-25T15:48:46.408 INFO:teuthology.orchestra.run.smithi031.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 21 MB/s | 1.2 MB 00:00 2024-04-25T15:48:46.432 INFO:teuthology.orchestra.run.smithi063.stdout:(151/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 4.4 MB/s | 3.1 MB 00:00 2024-04-25T15:48:46.442 INFO:teuthology.orchestra.run.smithi031.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 3.1 MB/s | 106 kB 00:00 2024-04-25T15:48:46.475 INFO:teuthology.orchestra.run.smithi031.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 4.0 MB/s | 135 kB 00:00 2024-04-25T15:48:46.509 INFO:teuthology.orchestra.run.smithi031.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 3.7 MB/s | 125 kB 00:00 2024-04-25T15:48:46.542 INFO:teuthology.orchestra.run.smithi031.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 6.3 MB/s | 215 kB 00:00 2024-04-25T15:48:46.584 INFO:teuthology.orchestra.run.smithi031.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 16 MB/s | 689 kB 00:00 2024-04-25T15:48:46.643 INFO:teuthology.orchestra.run.smithi031.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 16 MB/s | 943 kB 00:00 2024-04-25T15:48:46.677 INFO:teuthology.orchestra.run.smithi031.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 107 kB 00:00 2024-04-25T15:48:46.710 INFO:teuthology.orchestra.run.smithi031.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 645 kB/s | 21 kB 00:00 2024-04-25T15:48:46.752 INFO:teuthology.orchestra.run.smithi031.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 7.4 MB/s | 313 kB 00:00 2024-04-25T15:48:46.786 INFO:teuthology.orchestra.run.smithi031.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 985 kB/s | 33 kB 00:00 2024-04-25T15:48:46.869 INFO:teuthology.orchestra.run.smithi031.stdout:(53/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 36 MB/s | 3.0 MB 00:00 2024-04-25T15:48:46.903 INFO:teuthology.orchestra.run.smithi031.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 549 kB/s | 18 kB 00:00 2024-04-25T15:48:46.937 INFO:teuthology.orchestra.run.smithi031.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 334 kB/s | 11 kB 00:00 2024-04-25T15:48:47.020 INFO:teuthology.orchestra.run.smithi031.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-04-25T15:48:47.137 INFO:teuthology.orchestra.run.smithi031.stdout:(57/152): ceph-mgr-diskprediction-local-19.0.0- 4.3 MB/s | 7.4 MB 00:01 2024-04-25T15:48:47.154 INFO:teuthology.orchestra.run.smithi031.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2024-04-25T15:48:47.188 INFO:teuthology.orchestra.run.smithi031.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 662 kB 00:00 2024-04-25T15:48:47.222 INFO:teuthology.orchestra.run.smithi031.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 582 kB 00:00 2024-04-25T15:48:47.238 INFO:teuthology.orchestra.run.smithi031.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 517 kB/s | 25 kB 00:00 2024-04-25T15:48:47.256 INFO:teuthology.orchestra.run.smithi031.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-04-25T15:48:47.281 INFO:teuthology.orchestra.run.smithi031.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.9 MB/s | 292 kB 00:00 2024-04-25T15:48:47.306 INFO:teuthology.orchestra.run.smithi031.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-25T15:48:47.316 INFO:teuthology.orchestra.run.smithi063.stdout:(152/152): librados2-19.0.0-3291.g994e2f20.el9. 2.0 MB/s | 3.3 MB 00:01 2024-04-25T15:48:47.324 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:48:47.324 INFO:teuthology.orchestra.run.smithi063.stdout:Total 13 MB/s | 214 MB 00:16 2024-04-25T15:48:47.331 INFO:teuthology.orchestra.run.smithi031.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 710 kB/s | 35 kB 00:00 2024-04-25T15:48:47.424 INFO:teuthology.orchestra.run.smithi031.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-04-25T15:48:47.490 INFO:teuthology.orchestra.run.smithi031.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 29 MB/s | 4.4 MB 00:00 2024-04-25T15:48:47.532 INFO:teuthology.orchestra.run.smithi031.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-25T15:48:47.566 INFO:teuthology.orchestra.run.smithi031.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-25T15:48:47.591 INFO:teuthology.orchestra.run.smithi031.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.3 MB/s | 133 kB 00:00 2024-04-25T15:48:47.708 INFO:teuthology.orchestra.run.smithi031.stdout:(71/152): postgresql-server-13.14-1.el9.x86_64. 41 MB/s | 5.8 MB 00:00 2024-04-25T15:48:47.741 INFO:teuthology.orchestra.run.smithi031.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.9 MB/s | 1.0 MB 00:00 2024-04-25T15:48:47.766 INFO:teuthology.orchestra.run.smithi031.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-25T15:48:47.892 INFO:teuthology.orchestra.run.smithi031.stdout:(74/152): python3-babel-2.9.1-2.el9.noarch.rpm 40 MB/s | 6.0 MB 00:00 2024-04-25T15:48:47.917 INFO:teuthology.orchestra.run.smithi031.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.6 MB/s | 244 kB 00:00 2024-04-25T15:48:47.942 INFO:teuthology.orchestra.run.smithi031.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-25T15:48:47.968 INFO:teuthology.orchestra.run.smithi031.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 515 kB/s | 26 kB 00:00 2024-04-25T15:48:47.992 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-04-25T15:48:47.992 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-04-25T15:48:47.993 INFO:teuthology.orchestra.run.smithi031.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 381 kB/s | 19 kB 00:00 2024-04-25T15:48:48.018 INFO:teuthology.orchestra.run.smithi031.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-04-25T15:48:48.043 INFO:teuthology.orchestra.run.smithi031.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 701 kB/s | 35 kB 00:00 2024-04-25T15:48:48.144 INFO:teuthology.orchestra.run.smithi031.stdout:(81/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-04-25T15:48:48.169 INFO:teuthology.orchestra.run.smithi031.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 250 kB 00:00 2024-04-25T15:48:48.194 INFO:teuthology.orchestra.run.smithi031.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-04-25T15:48:48.219 INFO:teuthology.orchestra.run.smithi031.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-25T15:48:48.245 INFO:teuthology.orchestra.run.smithi031.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-04-25T15:48:48.270 INFO:teuthology.orchestra.run.smithi031.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-25T15:48:48.295 INFO:teuthology.orchestra.run.smithi031.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-25T15:48:48.437 INFO:teuthology.orchestra.run.smithi031.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.6 MB/s | 279 kB 00:00 2024-04-25T15:48:48.462 INFO:teuthology.orchestra.run.smithi031.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 306 kB/s | 51 kB 00:00 2024-04-25T15:48:48.473 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-04-25T15:48:48.539 INFO:teuthology.orchestra.run.smithi031.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 530 kB/s | 54 kB 00:00 2024-04-25T15:48:48.635 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-04-25T15:48:48.635 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-04-25T15:48:48.738 INFO:teuthology.orchestra.run.smithi031.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 212 kB/s | 42 kB 00:00 2024-04-25T15:48:48.889 INFO:teuthology.orchestra.run.smithi031.stdout:(92/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-04-25T15:48:48.914 INFO:teuthology.orchestra.run.smithi031.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 253 kB/s | 44 kB 00:00 2024-04-25T15:48:48.939 INFO:teuthology.orchestra.run.smithi031.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-25T15:48:48.965 INFO:teuthology.orchestra.run.smithi031.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 718 kB/s | 36 kB 00:00 2024-04-25T15:48:49.006 INFO:teuthology.orchestra.run.smithi031.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 710 kB/s | 47 kB 00:00 2024-04-25T15:48:49.031 INFO:teuthology.orchestra.run.smithi031.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 305 kB 00:00 2024-04-25T15:48:49.049 INFO:teuthology.orchestra.run.smithi031.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.4 MB/s | 58 kB 00:00 2024-04-25T15:48:49.065 INFO:teuthology.orchestra.run.smithi031.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.7 MB/s | 94 kB 00:00 2024-04-25T15:48:49.082 INFO:teuthology.orchestra.run.smithi031.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-04-25T15:48:49.099 INFO:teuthology.orchestra.run.smithi031.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-04-25T15:48:49.116 INFO:teuthology.orchestra.run.smithi031.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 670 kB/s | 22 kB 00:00 2024-04-25T15:48:49.150 INFO:teuthology.orchestra.run.smithi031.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 17 MB/s | 863 kB 00:00 2024-04-25T15:48:49.350 INFO:teuthology.orchestra.run.smithi031.stdout:(104/152): ceph-osd-19.0.0-3291.g994e2f20.el9.x 2.0 MB/s | 18 MB 00:09 2024-04-25T15:48:49.596 INFO:teuthology.orchestra.run.smithi031.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-25T15:48:49.617 INFO:teuthology.orchestra.run.smithi031.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 81 kB/s | 19 kB 00:00 2024-04-25T15:48:49.684 INFO:teuthology.orchestra.run.smithi031.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 608 kB/s | 309 kB 00:00 2024-04-25T15:48:49.809 INFO:teuthology.orchestra.run.smithi031.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 136 kB/s | 26 kB 00:00 2024-04-25T15:48:49.834 INFO:teuthology.orchestra.run.smithi031.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 325 kB/s | 49 kB 00:00 2024-04-25T15:48:49.901 INFO:teuthology.orchestra.run.smithi031.stdout:(110/152): libarrow-9.0.0-10.el9.x86_64.rpm 16 MB/s | 4.4 MB 00:00 2024-04-25T15:48:49.926 INFO:teuthology.orchestra.run.smithi031.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 756 kB/s | 67 kB 00:00 2024-04-25T15:48:49.952 INFO:teuthology.orchestra.run.smithi031.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-04-25T15:48:49.985 INFO:teuthology.orchestra.run.smithi031.stdout:(113/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.9 MB/s | 838 kB 00:00 2024-04-25T15:48:50.069 INFO:teuthology.orchestra.run.smithi031.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 4.7 MB/s | 548 kB 00:00 2024-04-25T15:48:50.094 INFO:teuthology.orchestra.run.smithi031.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 310 kB/s | 43 kB 00:00 2024-04-25T15:48:50.119 INFO:teuthology.orchestra.run.smithi031.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 241 kB/s | 32 kB 00:00 2024-04-25T15:48:50.144 INFO:teuthology.orchestra.run.smithi031.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-04-25T15:48:50.178 INFO:teuthology.orchestra.run.smithi031.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-04-25T15:48:50.203 INFO:teuthology.orchestra.run.smithi031.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-04-25T15:48:50.228 INFO:teuthology.orchestra.run.smithi031.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-04-25T15:48:50.254 INFO:teuthology.orchestra.run.smithi031.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 478 kB/s | 35 kB 00:00 2024-04-25T15:48:50.279 INFO:teuthology.orchestra.run.smithi031.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-04-25T15:48:50.429 INFO:teuthology.orchestra.run.smithi031.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 10 MB/s | 2.0 MB 00:00 2024-04-25T15:48:50.446 INFO:teuthology.orchestra.run.smithi031.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 754 kB/s | 144 kB 00:00 2024-04-25T15:48:50.479 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-04-25T15:48:50.480 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-04-25T15:48:50.480 INFO:teuthology.orchestra.run.smithi031.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 53 kB/s | 11 kB 00:00 2024-04-25T15:48:50.505 INFO:teuthology.orchestra.run.smithi031.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-04-25T15:48:50.530 INFO:teuthology.orchestra.run.smithi031.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 276 kB/s | 23 kB 00:00 2024-04-25T15:48:50.555 INFO:teuthology.orchestra.run.smithi031.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 259 kB/s | 19 kB 00:00 2024-04-25T15:48:50.581 INFO:teuthology.orchestra.run.smithi031.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 393 kB/s | 20 kB 00:00 2024-04-25T15:48:50.614 INFO:teuthology.orchestra.run.smithi031.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-04-25T15:48:50.640 INFO:teuthology.orchestra.run.smithi031.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-04-25T15:48:50.665 INFO:teuthology.orchestra.run.smithi031.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 967 kB/s | 79 kB 00:00 2024-04-25T15:48:50.690 INFO:teuthology.orchestra.run.smithi031.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 770 kB/s | 58 kB 00:00 2024-04-25T15:48:50.740 INFO:teuthology.orchestra.run.smithi031.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.6 MB/s | 272 kB 00:00 2024-04-25T15:48:50.765 INFO:teuthology.orchestra.run.smithi031.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-04-25T15:48:50.790 INFO:teuthology.orchestra.run.smithi031.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 897 kB/s | 90 kB 00:00 2024-04-25T15:48:50.816 INFO:teuthology.orchestra.run.smithi031.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 377 kB/s | 28 kB 00:00 2024-04-25T15:48:50.850 INFO:teuthology.orchestra.run.smithi031.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 526 kB/s | 31 kB 00:00 2024-04-25T15:48:50.875 INFO:teuthology.orchestra.run.smithi031.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-04-25T15:48:50.900 INFO:teuthology.orchestra.run.smithi031.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 707 kB/s | 59 kB 00:00 2024-04-25T15:48:50.925 INFO:teuthology.orchestra.run.smithi031.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 479 kB/s | 36 kB 00:00 2024-04-25T15:48:50.950 INFO:teuthology.orchestra.run.smithi031.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-04-25T15:48:51.009 INFO:teuthology.orchestra.run.smithi031.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-04-25T15:48:51.034 INFO:teuthology.orchestra.run.smithi031.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 826 kB/s | 90 kB 00:00 2024-04-25T15:48:51.068 INFO:teuthology.orchestra.run.smithi031.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-25T15:48:51.093 INFO:teuthology.orchestra.run.smithi031.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 240 kB/s | 20 kB 00:00 2024-04-25T15:48:51.118 INFO:teuthology.orchestra.run.smithi031.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-04-25T15:48:51.152 INFO:teuthology.orchestra.run.smithi031.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-04-25T15:48:51.219 INFO:teuthology.orchestra.run.smithi031.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 20 MB/s | 1.3 MB 00:00 2024-04-25T15:48:51.311 INFO:teuthology.orchestra.run.smithi031.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-04-25T15:48:51.744 INFO:teuthology.orchestra.run.smithi031.stdout:(151/152): librados2-19.0.0-3291.g994e2f20.el9. 5.1 MB/s | 3.3 MB 00:00 2024-04-25T15:48:52.744 INFO:teuthology.orchestra.run.smithi031.stdout:(152/152): librbd1-19.0.0-3291.g994e2f20.el9.x8 1.9 MB/s | 3.1 MB 00:01 2024-04-25T15:48:52.749 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:48:52.750 INFO:teuthology.orchestra.run.smithi031.stdout:Total 13 MB/s | 214 MB 00:15 2024-04-25T15:48:53.877 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2024-04-25T15:48:53.911 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-04-25T15:48:53.982 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T15:48:54.029 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T15:48:54.035 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2024-04-25T15:48:54.035 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2024-04-25T15:48:54.132 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T15:48:54.300 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T15:48:54.326 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T15:48:54.368 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T15:48:54.693 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T15:48:54.818 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:48:54.943 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:48:54.969 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:48:55.085 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:48:55.132 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:48:55.278 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T15:48:55.328 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T15:48:55.356 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:48:55.461 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:48:55.491 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:48:55.553 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:48:55.575 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:48:55.600 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:48:55.792 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:48:55.842 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:48:55.875 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2024-04-25T15:48:55.875 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2024-04-25T15:48:55.952 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T15:48:56.048 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T15:48:56.083 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T15:48:56.119 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T15:48:56.193 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T15:48:56.242 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T15:48:56.294 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T15:48:56.326 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T15:48:56.354 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T15:48:56.424 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T15:48:56.477 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T15:48:56.515 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T15:48:56.567 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T15:48:56.629 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-04-25T15:48:56.644 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T15:48:56.677 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T15:48:56.720 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T15:48:56.721 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T15:48:56.749 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T15:48:56.759 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T15:48:56.819 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T15:48:56.865 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T15:48:56.887 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T15:48:56.939 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T15:48:57.038 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T15:48:57.063 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T15:48:57.071 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T15:48:57.107 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T15:48:57.133 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T15:48:57.460 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T15:48:57.596 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:48:57.651 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T15:48:57.725 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:48:57.728 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T15:48:57.747 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:48:57.766 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T15:48:57.817 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T15:48:57.840 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:48:57.855 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T15:48:57.877 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:48:57.896 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T15:48:58.042 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T15:48:58.050 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T15:48:58.099 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T15:48:58.108 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T15:48:58.127 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:48:58.189 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T15:48:58.233 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:48:58.260 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T15:48:58.261 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:48:58.292 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T15:48:58.323 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:48:58.328 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T15:48:58.345 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:48:58.370 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:48:58.383 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T15:48:58.428 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T15:48:58.470 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T15:48:58.512 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T15:48:58.538 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:48:58.570 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:48:58.648 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T15:48:58.682 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T15:48:58.693 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T15:48:58.723 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T15:48:58.764 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T15:48:58.777 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T15:48:58.810 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T15:48:58.846 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T15:48:58.858 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T15:48:58.898 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T15:48:58.922 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T15:48:58.946 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T15:48:58.961 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T15:48:58.972 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T15:48:58.997 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T15:48:59.005 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T15:48:59.046 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T15:48:59.071 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T15:48:59.073 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T15:48:59.146 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T15:48:59.197 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T15:48:59.198 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T15:48:59.234 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T15:48:59.279 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T15:48:59.304 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T15:48:59.358 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T15:48:59.389 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T15:48:59.433 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T15:48:59.433 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T15:48:59.461 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T15:48:59.467 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T15:48:59.506 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T15:48:59.534 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T15:48:59.562 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T15:48:59.592 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T15:48:59.600 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T15:48:59.635 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T15:48:59.669 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T15:48:59.670 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T15:48:59.711 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T15:48:59.744 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T15:48:59.817 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T15:48:59.833 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T15:48:59.875 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T15:48:59.876 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T15:48:59.906 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T15:48:59.953 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T15:49:00.059 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T15:49:00.115 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T15:49:00.438 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T15:49:00.526 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T15:49:00.560 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T15:49:00.603 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T15:49:00.641 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T15:49:00.684 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T15:49:00.832 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T15:49:00.834 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T15:49:00.887 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T15:49:00.894 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T15:49:00.921 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T15:49:00.951 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T15:49:01.019 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T15:49:01.046 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T15:49:01.079 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T15:49:01.086 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T15:49:01.122 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T15:49:01.159 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T15:49:01.170 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T15:49:01.198 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T15:49:01.205 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T15:49:01.249 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T15:49:01.291 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T15:49:01.375 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T15:49:01.404 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T15:49:01.433 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T15:49:01.478 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T15:49:01.481 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T15:49:01.500 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T15:49:01.542 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T15:49:01.641 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T15:49:01.675 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T15:49:01.715 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T15:49:01.740 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T15:49:01.765 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T15:49:01.843 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T15:49:01.882 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2024-04-25T15:49:01.973 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T15:49:02.007 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-25T15:49:02.013 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T15:49:02.033 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T15:49:02.068 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T15:49:02.104 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-25T15:49:02.107 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T15:49:02.126 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T15:49:02.147 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T15:49:02.158 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-25T15:49:02.176 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T15:49:02.200 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-25T15:49:02.211 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T15:49:02.224 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-25T15:49:02.234 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T15:49:02.244 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T15:49:02.268 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-25T15:49:02.286 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T15:49:02.320 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T15:49:02.410 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T15:49:02.451 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T15:49:02.556 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T15:49:02.604 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T15:49:02.674 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-25T15:49:02.716 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T15:49:02.750 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:49:02.767 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T15:49:02.878 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:49:02.901 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:49:02.993 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:49:03.022 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:49:03.180 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-25T15:49:03.236 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-25T15:49:03.263 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:49:03.353 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:49:03.381 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:49:03.426 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 14/156 2024-04-25T15:49:03.448 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:49:03.454 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T15:49:03.474 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-25T15:49:03.541 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T15:49:03.542 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T15:49:03.569 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T15:49:03.588 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T15:49:03.602 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T15:49:03.621 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T15:49:03.632 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:49:03.664 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-25T15:49:03.728 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T15:49:03.775 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-25T15:49:03.780 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T15:49:03.871 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T15:49:03.872 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-25T15:49:03.905 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-25T15:49:03.906 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T15:49:03.929 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T15:49:03.956 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-25T15:49:03.982 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:49:04.032 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-25T15:49:04.072 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-25T15:49:04.090 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:49:04.095 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T15:49:04.116 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-25T15:49:04.120 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T15:49:04.137 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T15:49:04.148 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-25T15:49:04.155 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T15:49:04.176 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-25T15:49:04.219 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T15:49:04.248 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-25T15:49:04.299 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-25T15:49:04.328 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-25T15:49:04.372 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-25T15:49:04.451 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-25T15:49:04.482 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 31/156 2024-04-25T15:49:04.518 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 32/156 2024-04-25T15:49:04.546 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 33/156 2024-04-25T15:49:04.620 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-25T15:49:04.643 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T15:49:04.668 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:49:04.669 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-25T15:49:04.739 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:49:04.747 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-25T15:49:04.769 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T15:49:04.895 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-25T15:49:04.909 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T15:49:04.948 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-25T15:49:05.032 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T15:49:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-25T15:49:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-25T15:49:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-25T15:49:05.680 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-25T15:49:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 43/156 2024-04-25T15:49:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-25T15:49:05.908 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-25T15:49:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-25T15:49:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-25T15:49:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-25T15:49:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-25T15:49:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-25T15:49:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-25T15:49:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-25T15:49:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-25T15:49:06.341 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T15:49:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-25T15:49:06.424 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T15:49:06.452 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T15:49:06.485 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T15:49:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-25T15:49:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-25T15:49:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-25T15:49:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-25T15:49:06.726 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-25T15:49:06.768 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-25T15:49:06.809 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-25T15:49:06.826 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T15:49:06.842 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-25T15:49:06.864 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:49:06.866 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-25T15:49:06.946 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-25T15:49:06.947 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:49:06.977 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T15:49:07.013 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T15:49:07.037 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:49:07.074 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-25T15:49:07.083 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:49:07.107 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-25T15:49:07.135 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-25T15:49:07.169 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-25T15:49:07.209 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-25T15:49:07.249 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-25T15:49:07.278 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-25T15:49:07.312 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-25T15:49:07.346 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-25T15:49:07.388 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-25T15:49:07.422 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-25T15:49:07.532 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T15:49:07.566 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:49:07.638 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:49:07.640 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:49:07.668 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T15:49:07.677 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:49:07.718 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-25T15:49:07.762 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-25T15:49:07.767 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:49:07.768 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T15:49:07.768 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T15:49:07.768 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:07.804 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:49:07.808 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-25T15:49:07.857 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-25T15:49:07.967 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-25T15:49:08.018 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-25T15:49:08.789 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-25T15:49:08.839 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-25T15:49:08.873 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-25T15:49:08.978 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-25T15:49:09.057 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-25T15:49:09.148 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-25T15:49:09.183 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-25T15:49:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-25T15:49:09.397 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-25T15:49:09.477 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-25T15:49:09.925 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:49:09.956 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:49:10.168 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-25T15:49:10.283 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-25T15:49:10.552 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:49:10.592 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:49:10.683 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:49:10.683 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T15:49:10.683 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T15:49:10.683 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:10.710 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:49:11.601 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-25T15:49:11.692 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-25T15:49:11.720 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-25T15:49:11.753 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-25T15:49:12.090 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-25T15:49:12.132 INFO:teuthology.orchestra.run.smithi031.stdout: Upgrading : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:49:12.206 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 99/156 2024-04-25T15:49:12.236 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 100/156 2024-04-25T15:49:12.273 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-25T15:49:12.788 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-25T15:49:12.826 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:49:12.906 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 103/156 2024-04-25T15:49:12.936 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 104/156 2024-04-25T15:49:15.274 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:49:15.307 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:49:15.894 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 105/156 2024-04-25T15:49:15.935 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:49:16.036 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 106/156 2024-04-25T15:49:16.036 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-25T15:49:16.037 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-25T15:49:16.037 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:16.069 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:49:19.280 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:49:19.280 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-04-25T15:49:19.280 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-04-25T15:49:19.280 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-04-25T15:49:19.280 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-04-25T15:49:19.280 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-04-25T15:49:19.280 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-04-25T15:49:19.280 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-04-25T15:49:19.280 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:19.465 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:49:19.483 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:49:19.484 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:19.484 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T15:49:19.484 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:21.076 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:49:21.102 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:49:21.102 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:21.102 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T15:49:21.102 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:49:21.102 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:49:21.102 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:21.335 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:49:21.358 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:49:21.358 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:21.358 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T15:49:21.358 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:49:21.358 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:49:21.358 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:21.819 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:49:21.842 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:49:21.842 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:21.842 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T15:49:21.842 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:49:21.842 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:49:21.842 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:21.941 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T15:49:21.989 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T15:49:22.052 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T15:49:22.246 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T15:49:22.303 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T15:49:22.380 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T15:49:22.468 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:49:22.608 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:49:22.644 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T15:49:22.677 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T15:49:22.704 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T15:49:22.822 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:49:22.822 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-04-25T15:49:22.822 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-04-25T15:49:22.822 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-04-25T15:49:22.822 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-04-25T15:49:22.823 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-04-25T15:49:22.823 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-04-25T15:49:22.823 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-04-25T15:49:22.823 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:23.003 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:49:23.020 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:49:23.020 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:23.020 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T15:49:23.020 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:23.157 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:49:23.186 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:49:23.218 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T15:49:23.280 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T15:49:23.334 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:49:24.374 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:49:24.403 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:49:24.664 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:49:24.692 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:49:24.692 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:24.692 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T15:49:24.693 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:49:24.693 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:49:24.693 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:24.933 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:49:24.957 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:49:24.957 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:24.958 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T15:49:24.958 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:49:24.958 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:49:24.958 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:25.367 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:49:25.395 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:49:25.441 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:49:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:49:25.466 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:25.466 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T15:49:25.466 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:49:25.466 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:49:25.466 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:25.497 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:49:25.581 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T15:49:25.640 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T15:49:25.669 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T15:49:25.703 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T15:49:25.705 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:49:25.728 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:49:25.728 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:25.728 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T15:49:25.728 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:49:25.728 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:49:25.728 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:25.773 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:49:25.784 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:49:25.883 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T15:49:25.902 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T15:49:25.937 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T15:49:25.960 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T15:49:26.041 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T15:49:26.140 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:49:26.292 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:49:26.376 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T15:49:26.408 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T15:49:26.435 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T15:49:26.871 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T15:49:26.900 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:49:26.923 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:49:26.923 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:26.923 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T15:49:26.923 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:49:26.923 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:49:26.923 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:26.967 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:49:26.989 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:49:26.989 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:26.990 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T15:49:26.990 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:27.014 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:49:27.035 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:49:27.042 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:49:27.057 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:49:27.057 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:27.058 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T15:49:27.058 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:49:27.058 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:49:27.058 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:27.075 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T15:49:27.139 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T15:49:27.164 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:49:27.385 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:49:27.408 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:49:27.409 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:27.409 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T15:49:27.409 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:49:27.409 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:49:27.409 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:27.961 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 107/156 2024-04-25T15:49:27.961 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /sys 2024-04-25T15:49:27.961 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /proc 2024-04-25T15:49:27.961 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /mnt 2024-04-25T15:49:27.961 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /var/tmp 2024-04-25T15:49:27.961 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /home 2024-04-25T15:49:27.961 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /root 2024-04-25T15:49:27.961 INFO:teuthology.orchestra.run.smithi031.stdout:skipping the directory /tmp 2024-04-25T15:49:27.962 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:28.136 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:49:28.152 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 108/156 2024-04-25T15:49:28.152 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:28.152 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-25T15:49:28.153 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:28.273 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:49:28.316 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:49:29.343 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:49:29.375 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:49:29.485 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:49:29.658 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T15:49:29.685 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:49:29.710 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:49:29.710 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:29.710 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T15:49:29.710 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:49:29.710 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:49:29.710 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:29.761 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:49:29.772 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:49:29.780 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:49:29.807 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 109/156 2024-04-25T15:49:29.807 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:29.807 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-25T15:49:29.807 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:49:29.807 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-25T15:49:29.807 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:29.875 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T15:49:29.936 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T15:49:30.040 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:49:30.064 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 110/156 2024-04-25T15:49:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-25T15:49:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:49:30.064 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-25T15:49:30.064 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:30.550 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:49:30.574 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 111/156 2024-04-25T15:49:30.574 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:30.574 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-25T15:49:30.574 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:49:30.575 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-25T15:49:30.575 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:30.689 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-25T15:49:30.747 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-25T15:49:30.811 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-25T15:49:30.928 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T15:49:30.963 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:49:30.987 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:49:30.987 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:30.987 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T15:49:30.988 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:49:30.988 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:49:30.988 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:31.006 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-25T15:49:31.030 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:49:31.051 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-25T15:49:31.054 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:49:31.054 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:31.054 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T15:49:31.054 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:31.106 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:49:31.130 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:49:31.130 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:31.130 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T15:49:31.131 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:49:31.131 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:49:31.131 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:31.131 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-25T15:49:31.221 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:49:31.372 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:49:31.417 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-25T15:49:31.449 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-25T15:49:31.457 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:49:31.476 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-25T15:49:31.481 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:49:31.481 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:31.481 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T15:49:31.481 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:49:31.481 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:49:31.482 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:31.698 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T15:49:31.739 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T15:49:31.779 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T15:49:31.811 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T15:49:31.838 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T15:49:31.863 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:49:31.891 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 123/156 2024-04-25T15:49:31.924 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 124/156 2024-04-25T15:49:31.988 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 125/156 2024-04-25T15:49:32.023 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T15:49:32.023 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:49:32.068 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T15:49:32.106 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T15:49:32.654 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:49:32.711 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:49:32.731 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:49:33.122 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 126/156 2024-04-25T15:49:33.158 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:49:33.577 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T15:49:33.666 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T15:49:33.707 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T15:49:33.790 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T15:49:33.829 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T15:49:33.829 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:49:33.912 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:49:33.945 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:49:33.945 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:49:34.025 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:49:34.094 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:49:34.179 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 127/156 2024-04-25T15:49:34.208 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:49:34.315 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 128/156 2024-04-25T15:49:34.482 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 129/156 2024-04-25T15:49:34.510 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:49:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 130/156 2024-04-25T15:49:34.534 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:34.534 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-25T15:49:34.534 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:49:34.534 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-25T15:49:34.534 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:34.577 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:49:34.588 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 131/156 2024-04-25T15:49:34.691 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 132/156 2024-04-25T15:49:34.743 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 133/156 2024-04-25T15:49:35.731 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-25T15:49:35.762 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:49:35.787 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 135/156 2024-04-25T15:49:35.787 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:35.787 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-25T15:49:35.787 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:49:35.787 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-25T15:49:35.787 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:35.830 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:49:35.853 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3291.g994e2 136/156 2024-04-25T15:49:35.853 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:35.853 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-25T15:49:35.853 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:35.905 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:49:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 137/156 2024-04-25T15:49:35.929 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:35.929 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-25T15:49:35.929 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:49:35.929 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-25T15:49:35.929 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:36.027 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T15:49:36.069 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T15:49:36.110 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T15:49:36.141 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T15:49:36.169 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T15:49:36.256 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:49:36.280 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 138/156 2024-04-25T15:49:36.280 INFO:teuthology.orchestra.run.smithi031.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-25T15:49:36.280 INFO:teuthology.orchestra.run.smithi031.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-25T15:49:36.280 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:49:36.281 INFO:teuthology.orchestra.run.smithi031.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-25T15:49:36.281 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:36.414 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T15:49:36.456 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T15:49:36.486 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T15:49:37.095 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:49:37.132 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:49:37.154 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:49:38.076 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T15:49:38.287 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T15:49:38.330 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T15:49:38.425 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T15:49:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T15:49:38.465 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:49:38.558 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:49:38.591 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:49:38.591 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:49:38.679 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:49:38.751 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:49:40.228 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:49:40.228 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T15:49:40.228 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T15:49:40.228 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T15:49:40.228 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T15:49:40.228 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T15:49:40.228 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T15:49:40.229 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T15:49:40.230 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T15:49:40.231 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T15:49:40.232 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T15:49:40.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T15:49:40.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T15:49:40.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T15:49:40.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T15:49:40.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T15:49:40.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:49:40.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T15:49:40.793 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 139/156 2024-04-25T15:49:40.835 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 140/156 2024-04-25T15:49:40.884 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 141/156 2024-04-25T15:49:40.915 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-25T15:49:40.942 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-25T15:49:41.142 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 144/156 2024-04-25T15:49:41.188 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-25T15:49:41.218 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 146/156 2024-04-25T15:49:41.801 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:49:41.840 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:49:41.860 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-25T15:49:42.304 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:49:42.304 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.305 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.306 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T15:49:42.307 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.308 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T15:49:42.309 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T15:49:42.310 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T15:49:42.311 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T15:49:42.312 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T15:49:42.313 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T15:49:42.314 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T15:49:42.315 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T15:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T15:49:42.317 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T15:49:42.317 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:49:42.317 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-04-25T15:49:42.759 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 148/156 2024-04-25T15:49:42.764 DEBUG:teuthology.parallel:result is None 2024-04-25T15:49:42.844 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-25T15:49:42.886 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-25T15:49:42.973 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-25T15:49:43.008 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-25T15:49:43.008 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:49:43.093 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-25T15:49:43.125 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:49:43.126 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:49:43.213 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-25T15:49:43.276 INFO:teuthology.orchestra.run.smithi031.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:49:44.198 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T15:49:44.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T15:49:44.200 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T15:49:44.200 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T15:49:44.200 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T15:49:44.200 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T15:49:44.200 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T15:49:44.200 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T15:49:44.211 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T15:49:44.212 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T15:49:44.213 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T15:49:44.214 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T15:49:44.215 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T15:49:44.216 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T15:49:44.217 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T15:49:44.218 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T15:49:44.219 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:49:44.220 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T15:49:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:49:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:46.390 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-04-25T15:49:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T15:49:46.390 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.391 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.392 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T15:49:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T15:49:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T15:49:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T15:49:46.393 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T15:49:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T15:49:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T15:49:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T15:49:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T15:49:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T15:49:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T15:49:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.394 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T15:49:46.395 INFO:teuthology.orchestra.run.smithi063.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T15:49:46.396 INFO:teuthology.orchestra.run.smithi063.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T15:49:46.397 INFO:teuthology.orchestra.run.smithi063.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T15:49:46.398 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T15:49:46.399 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.400 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T15:49:46.401 INFO:teuthology.orchestra.run.smithi063.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T15:49:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T15:49:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T15:49:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T15:49:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T15:49:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T15:49:46.402 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T15:49:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T15:49:46.403 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-04-25T15:49:46.403 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-04-25T15:49:46.791 DEBUG:teuthology.parallel:result is None 2024-04-25T15:49:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:49:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 1/156 2024-04-25T15:49:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2/156 2024-04-25T15:49:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 3/156 2024-04-25T15:49:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 4/156 2024-04-25T15:49:49.058 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3291.g994e2 5/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 6/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 7/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 8/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 9/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 10/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 11/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 12/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 13/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 14/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_ 15/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 16/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 17/156 2024-04-25T15:49:49.059 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_6 18/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86 19/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 20/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el 21/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ceph-common-2:19.0.0-3291.g994e2f20.el9. 22/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_6 23/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 24/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 25/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 26/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 27/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 28/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 29/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20. 30/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noa 31/156 2024-04-25T15:49:49.060 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.n 32/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3291.g994 33/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el 34/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 35/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.e 36/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 37/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 38/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 39/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-25T15:49:49.061 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-25T15:49:49.062 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-25T15:49:49.063 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-25T15:49:49.064 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-25T15:49:49.065 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-25T15:49:49.066 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-25T15:49:49.067 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-25T15:49:49.068 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-25T15:49:49.068 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-25T15:49:49.068 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-25T15:49:49.068 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-25T15:49:49.068 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-25T15:49:49.068 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-25T15:49:49.068 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-25T15:49:49.068 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-25T15:49:49.068 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-25T15:49:49.069 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 149/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 151/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-25T15:49:49.070 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-25T15:49:51.165 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-25T15:49:51.165 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:51.165 INFO:teuthology.orchestra.run.smithi031.stdout:Upgraded: 2024-04-25T15:49:51.165 INFO:teuthology.orchestra.run.smithi031.stdout: librados2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.165 INFO:teuthology.orchestra.run.smithi031.stdout: librbd1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.165 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: automake-1.16.2-8.el9.noarch 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: bison-3.7.4-5.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-base-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-grafana-dashboards-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-immutable-object-cache-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mds-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.166 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-dashboard-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-modules-core-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mgr-rook-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-mon-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-osd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-prometheus-alerts-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-radosgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-selinux-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-test-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: ceph-volume-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: cephadm-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.167 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-shell-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: cephfs-top-2:19.0.0-3291.g994e2f20.el9.noarch 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: flex-2.6.4-9.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: libcephfs2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.168 INFO:teuthology.orchestra.run.smithi031.stdout: libcephsqlite-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: librados-devel-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: libradosstriper1-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: librgw2-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-25T15:49:51.169 INFO:teuthology.orchestra.run.smithi031.stdout: lua-5.4.4-4.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-13.14-1.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-25T15:49:51.170 INFO:teuthology.orchestra.run.smithi031.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-argparse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ceph-common-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cephfs-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-25T15:49:51.171 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-25T15:49:51.172 INFO:teuthology.orchestra.run.smithi031.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-ply-3.11-14.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-25T15:49:51.173 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rados-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rgw-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-25T15:49:51.174 INFO:teuthology.orchestra.run.smithi031.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-fuse-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-mirror-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: rbd-nbd-2:19.0.0-3291.g994e2f20.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: re2-1:20211101-3.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-25T15:49:51.175 INFO:teuthology.orchestra.run.smithi031.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-25T15:49:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-25T15:49:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-25T15:49:51.176 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:49:51.176 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2024-04-25T15:49:51.548 DEBUG:teuthology.parallel:result is None 2024-04-25T15:49:51.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:49:51.772 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T15:49:51.794 INFO:teuthology.orchestra.run.smithi031.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T15:49:51.795 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T15:49:51.796 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T15:49:51.797 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:49:52.032 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T15:49:52.057 INFO:teuthology.orchestra.run.smithi063.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T15:49:52.058 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T15:49:52.058 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T15:49:52.060 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:49:52.282 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-25T15:49:52.309 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-3291.g994e2f20.el9 2024-04-25T15:49:52.309 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3291.g994e2f20.el9 2024-04-25T15:49:52.309 INFO:teuthology.task.install:The correct ceph version 19.0.0-3291.g994e2f20 is installed. 2024-04-25T15:49:52.311 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T15:49:52.311 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:49:52.311 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T15:49:52.339 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:49:52.339 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T15:49:52.371 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:49:52.371 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T15:49:52.403 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T15:49:52.403 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:49:52.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T15:49:52.428 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T15:49:52.493 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:49:52.493 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T15:49:52.523 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T15:49:52.592 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:49:52.592 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T15:49:52.621 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T15:49:52.689 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T15:49:52.690 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:49:52.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T15:49:52.714 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T15:49:52.779 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:49:52.779 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T15:49:52.808 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T15:49:52.877 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:49:52.877 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T15:49:52.905 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T15:49:52.973 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T15:49:52.974 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:49:52.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T15:49:52.998 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T15:49:53.064 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:49:53.064 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T15:49:53.093 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T15:49:53.161 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:49:53.162 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T15:49:53.189 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T15:49:53.257 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-25T15:49:53.352 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T15:49:53.352 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:49:53.352 INFO:tasks.cephadm:Cluster fsid is 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:49:53.352 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-25T15:49:53.353 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.31', 'mon.b': '172.21.15.63', 'mon.c': '172.21.15.186'} 2024-04-25T15:49:53.353 INFO:tasks.cephadm:First mon is mon.a on smithi031 2024-04-25T15:49:53.353 INFO:tasks.cephadm:First mgr is x 2024-04-25T15:49:53.353 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-25T15:49:53.353 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2024-04-25T15:49:53.378 DEBUG:teuthology.orchestra.run.smithi063:> sudo hostname $(hostname -s) 2024-04-25T15:49:53.408 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-04-25T15:49:53.437 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-25T15:49:53.437 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:49:53.652 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '994e2f20102a20fa9a27966c14b625cc01884df8', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78872/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3291-g994e2f20', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3291.g994e2f20'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-25 04:02:18.990013', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240425.015853-debug', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-04-25T15:49:53.790 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-pdonnell-testing-20240425.015853-debug, sha1 994e2f20102a20fa9a27966c14b625cc01884df8 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:49:53.791 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm 2024-04-25T15:49:53.792 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T15:49:54.197 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 15:49 /home/ubuntu/cephtest/cephadm 2024-04-25T15:49:54.198 DEBUG:teuthology.orchestra.run.smithi063:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T15:49:54.570 INFO:teuthology.orchestra.run.smithi063.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 15:49 /home/ubuntu/cephtest/cephadm 2024-04-25T15:49:54.571 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240425.015853-debug/994e2f20102a20fa9a27966c14b625cc01884df8/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-25T15:49:54.959 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 781647 Apr 25 15:49 /home/ubuntu/cephtest/cephadm 2024-04-25T15:49:54.959 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T15:49:54.976 DEBUG:teuthology.orchestra.run.smithi063:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T15:49:54.995 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-25T15:49:55.029 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 on all hosts... 2024-04-25T15:49:55.029 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T15:49:55.032 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T15:49:55.038 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 pull 2024-04-25T15:49:56.577 INFO:teuthology.orchestra.run.smithi063.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T15:49:56.678 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T15:49:56.782 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T15:50:25.030 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T15:50:25.030 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:50:25.030 INFO:teuthology.orchestra.run.smithi063.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T15:50:25.030 INFO:teuthology.orchestra.run.smithi063.stdout: "repo_digests": [ 2024-04-25T15:50:25.030 INFO:teuthology.orchestra.run.smithi063.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T15:50:25.030 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-04-25T15:50:25.030 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T15:50:34.244 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T15:50:34.244 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:50:34.244 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T15:50:34.244 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2024-04-25T15:50:34.245 INFO:teuthology.orchestra.run.smithi031.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T15:50:34.245 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T15:50:34.245 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T15:50:34.328 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-25T15:50:34.328 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev)", 2024-04-25T15:50:34.328 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4", 2024-04-25T15:50:34.328 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-04-25T15:50:34.328 INFO:teuthology.orchestra.run.smithi186.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385" 2024-04-25T15:50:34.328 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-04-25T15:50:34.329 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-25T15:50:34.356 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2024-04-25T15:50:34.386 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph 2024-04-25T15:50:34.420 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-04-25T15:50:34.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2024-04-25T15:50:34.473 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /etc/ceph 2024-04-25T15:50:34.503 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-04-25T15:50:34.527 INFO:tasks.cephadm:Writing seed config... 2024-04-25T15:50:34.528 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-25T15:50:34.528 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-25T15:50:34.528 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-25T15:50:34.528 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-25T15:50:34.528 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-25T15:50:34.528 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-25T15:50:34.528 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-25T15:50:34.528 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-25T15:50:34.529 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-25T15:50:34.530 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-25T15:50:34.531 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-25T15:50:34.531 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-25T15:50:34.531 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-25T15:50:34.531 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-25T15:50:34.531 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-25T15:50:34.531 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-25T15:50:34.531 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-25T15:50:34.532 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:50:34.532 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-25T15:50:34.549 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 74fd3742-031b-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-25T15:50:34.549 DEBUG:teuthology.orchestra.run.smithi031:mon.a> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.a.service 2024-04-25T15:50:34.592 DEBUG:teuthology.orchestra.run.smithi031:mgr.x> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.x.service 2024-04-25T15:50:34.635 INFO:tasks.cephadm:Bootstrapping... 2024-04-25T15:50:34.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 -v bootstrap --fsid 74fd3742-031b-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.31 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-25T15:50:34.833 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2024-04-25T15:50:34.833 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8', '-v', 'bootstrap', '--fsid', '74fd3742-031b-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.31', '--skip-admin-label'] 2024-04-25T15:50:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-25T15:50:34.834 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying podman|docker is present... 2024-04-25T15:50:34.859 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T15:50:34.859 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying lvm2 is present... 2024-04-25T15:50:34.859 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying time synchronization is in place... 2024-04-25T15:50:34.867 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T15:50:34.867 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T15:50:34.873 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T15:50:34.873 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T15:50:34.880 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-04-25T15:50:34.886 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-04-25T15:50:34.886 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-04-25T15:50:34.886 INFO:teuthology.orchestra.run.smithi031.stdout:Repeating the final host check... 2024-04-25T15:50:34.911 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-25T15:50:34.912 INFO:teuthology.orchestra.run.smithi031.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-25T15:50:34.912 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl is present 2024-04-25T15:50:34.912 INFO:teuthology.orchestra.run.smithi031.stdout:lvcreate is present 2024-04-25T15:50:34.918 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-25T15:50:34.918 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-25T15:50:34.924 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-25T15:50:34.924 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T15:50:34.930 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2024-04-25T15:50:34.936 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2024-04-25T15:50:34.937 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2024-04-25T15:50:34.937 INFO:teuthology.orchestra.run.smithi031.stdout:Host looks OK 2024-04-25T15:50:34.937 INFO:teuthology.orchestra.run.smithi031.stdout:Cluster fsid: 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:34.937 INFO:teuthology.orchestra.run.smithi031.stdout:Acquiring lock 140174327000224 on /run/cephadm/74fd3742-031b-11ef-bc93-c7b262605968.lock 2024-04-25T15:50:34.937 INFO:teuthology.orchestra.run.smithi031.stdout:Lock 140174327000224 acquired on /run/cephadm/74fd3742-031b-11ef-bc93-c7b262605968.lock 2024-04-25T15:50:34.937 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 3300 ... 2024-04-25T15:50:34.938 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 6789 ... 2024-04-25T15:50:34.938 INFO:teuthology.orchestra.run.smithi031.stdout:Base mon IP(s) is [172.21.15.31:3300, 172.21.15.31:6789], mon addrv is [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-04-25T15:50:34.941 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.31 metric 100 2024-04-25T15:50:34.941 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.31 metric 100 2024-04-25T15:50:34.944 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-25T15:50:34.944 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-25T15:50:34.944 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-25T15:50:34.946 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-25T15:50:34.946 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-25T15:50:34.946 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T15:50:34.946 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-25T15:50:34.947 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-04-25T15:50:34.947 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-25T15:50:34.947 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-04-25T15:50:34.947 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2024-04-25T15:50:34.947 INFO:teuthology.orchestra.run.smithi031.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-25T15:50:34.948 INFO:teuthology.orchestra.run.smithi031.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-25T15:50:34.949 INFO:teuthology.orchestra.run.smithi031.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T15:50:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T15:50:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8... 2024-04-25T15:50:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Getting image source signatures 2024-04-25T15:50:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:665339d96aad80d083ba9988a62422bcef0a42210fb2b28340e33ab98589f7b8 2024-04-25T15:50:35.438 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-25T15:50:35.439 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying config sha256:d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 2024-04-25T15:50:35.439 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-25T15:50:36.400 INFO:teuthology.orchestra.run.smithi031.stdout:ceph: stdout ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T15:50:36.400 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph version: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev) 2024-04-25T15:50:36.400 INFO:teuthology.orchestra.run.smithi031.stdout:Extracting ceph user uid/gid from container image... 2024-04-25T15:50:37.273 INFO:teuthology.orchestra.run.smithi031.stdout:stat: stdout 167 167 2024-04-25T15:50:37.273 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial keys... 2024-04-25T15:50:38.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQDNeypmnh5zIxAA8RGG9tqBkIGXP3C3k4U7/A== 2024-04-25T15:50:39.037 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQDOeypmvupTHxAA3kbofcOE1u2uR9XasnHcbw== 2024-04-25T15:50:39.917 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQDPeypmjGF/FRAA87aAaDuRx7UUmcGKYKxHaA== 2024-04-25T15:50:39.918 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial monmap... 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool for a [v2:172.21.15.31:3300,v1:172.21.15.31:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = quincy 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: set fsid to 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:50:40.824 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mon... 2024-04-25T15:50:47.906 INFO:teuthology.orchestra.run.smithi031.stdout:create mon.a on 2024-04-25T15:50:48.497 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-25T15:50:48.849 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-25T15:50:49.355 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-74fd3742-031b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-74fd3742-031b-11ef-bc93-c7b262605968.target. 2024-04-25T15:50:49.355 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-74fd3742-031b-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-74fd3742-031b-11ef-bc93-c7b262605968.target. 2024-04-25T15:50:50.150 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.a 2024-04-25T15:50:50.151 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-25T15:50:50.380 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-74fd3742-031b-11ef-bc93-c7b262605968.target.wants/ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-74fd3742-031b-11ef-bc93-c7b262605968@.service. 2024-04-25T15:50:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:50 smithi031 podman[28749]: 2024-04-25 15:50:50.532307099 +0000 UTC m=+0.018327835 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:50:51.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:51 smithi031 podman[28749]: 2024-04-25 15:50:51.409495688 +0000 UTC m=+0.895516410 container create fd39c855a3c4ba5b430b151db60c423363d3385e4cefb8144738510ca02cc484 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T15:50:52.095 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T15:50:52.095 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-25T15:50:52.101 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T15:50:52.102 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T15:50:52.102 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-25T15:50:52.102 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T15:50:52.102 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon to start... 2024-04-25T15:50:52.102 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon... 2024-04-25T15:50:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:52 smithi031 podman[28749]: 2024-04-25 15:50:52.005613318 +0000 UTC m=+1.491634041 container init fd39c855a3c4ba5b430b151db60c423363d3385e4cefb8144738510ca02cc484 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T15:50:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:52 smithi031 podman[28749]: 2024-04-25 15:50:52.008794861 +0000 UTC m=+1.494815585 container start fd39c855a3c4ba5b430b151db60c423363d3385e4cefb8144738510ca02cc484 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5) 2024-04-25T15:50:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:52 smithi031 ceph-mon[28768]: mkfs 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:52 smithi031 ceph-mon[28768]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T15:50:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:52 smithi031 bash[28749]: fd39c855a3c4ba5b430b151db60c423363d3385e4cefb8144738510ca02cc484 2024-04-25T15:50:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:52 smithi031 systemd[1]: Started Ceph mon.a for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:50:53.186 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: monmap epoch 1 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: last_changed 2024-04-25T15:50:40.276173+0000 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: created 2024-04-25T15:50:40.276173+0000 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: min_mon_release 19 (squid) 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: election_strategy: 1 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: fsmap 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: mgrmap e1: no daemons active 2024-04-25T15:50:53.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:53 smithi031 ceph-mon[28768]: from='client.? 172.21.15.31:0/202263297' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout cluster: 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout id: 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout services: 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.577151s) 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout data: 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-25T15:50:53.195 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pgs: 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.561+0000 7f1782d10700 1 Processor -- start 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.561+0000 7f1782d10700 1 -- start start 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.561+0000 7f1782d10700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c10c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.561+0000 7f1782d10700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f177c10cb20 con 0x7f177c07d4f0 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.561+0000 7f1780aac700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c10c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.561+0000 7f1780aac700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c10c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48624/0 (socket says 172.21.15.31:48624) 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.561+0000 7f1780aac700 1 -- 172.21.15.31:0/3843805741 learned_addr learned my addr 172.21.15.31:0/3843805741 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.562+0000 7f1780aac700 1 -- 172.21.15.31:0/3843805741 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c10cca0 con 0x7f177c07d4f0 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.562+0000 7f1780aac700 1 --2- 172.21.15.31:0/3843805741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c10c510 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17640098a0 tx=0x7f1764009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf887cd63558ab29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.562+0000 7f177b7fe700 1 -- 172.21.15.31:0/3843805741 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1764004160 con 0x7f177c07d4f0 2024-04-25T15:50:53.196 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.562+0000 7f177b7fe700 1 -- 172.21.15.31:0/3843805741 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1764004300 con 0x7f177c07d4f0 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.562+0000 7f177b7fe700 1 -- 172.21.15.31:0/3843805741 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f17640036a0 con 0x7f177c07d4f0 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.563+0000 7f1782d10700 1 -- 172.21.15.31:0/3843805741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 msgr2=0x7f177c10c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.563+0000 7f1782d10700 1 --2- 172.21.15.31:0/3843805741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c10c510 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f17640098a0 tx=0x7f1764009bb0 comp rx=0 tx=0).stop 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.563+0000 7f1782d10700 1 -- 172.21.15.31:0/3843805741 shutdown_connections 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.563+0000 7f1782d10700 1 --2- 172.21.15.31:0/3843805741 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c10c510 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.563+0000 7f1782d10700 1 -- 172.21.15.31:0/3843805741 >> 172.21.15.31:0/3843805741 conn(0x7f177c103fa0 msgr2=0x7f177c1063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.563+0000 7f1782d10700 1 -- 172.21.15.31:0/3843805741 shutdown_connections 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.563+0000 7f1782d10700 1 -- 172.21.15.31:0/3843805741 wait complete. 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.563+0000 7f1782d10700 1 Processor -- start 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.564+0000 7f1782d10700 1 -- start start 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.564+0000 7f1782d10700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c19d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.564+0000 7f1782d10700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f177c10d1d0 con 0x7f177c07d4f0 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.564+0000 7f1780aac700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c19d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:53.197 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.564+0000 7f1780aac700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c19d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48636/0 (socket says 172.21.15.31:48636) 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.564+0000 7f1780aac700 1 -- 172.21.15.31:0/202263297 learned_addr learned my addr 172.21.15.31:0/202263297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.564+0000 7f1780aac700 1 -- 172.21.15.31:0/202263297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c19d940 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.564+0000 7f1780aac700 1 --2- 172.21.15.31:0/202263297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c19d400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1764000c00 tx=0x7f1764004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.565+0000 7f1779ffb700 1 -- 172.21.15.31:0/202263297 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1764003db0 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.565+0000 7f1779ffb700 1 -- 172.21.15.31:0/202263297 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1764011630 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.565+0000 7f1782d10700 1 -- 172.21.15.31:0/202263297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177c19dbd0 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.565+0000 7f1779ffb700 1 -- 172.21.15.31:0/202263297 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1764011950 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.565+0000 7f1782d10700 1 -- 172.21.15.31:0/202263297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f177c1a07e0 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.565+0000 7f1779ffb700 1 -- 172.21.15.31:0/202263297 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1764011af0 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.565+0000 7f1779ffb700 1 -- 172.21.15.31:0/202263297 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f176401c450 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.566+0000 7f1782d10700 1 -- 172.21.15.31:0/202263297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f177c064cf0 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.568+0000 7f1779ffb700 1 -- 172.21.15.31:0/202263297 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1764032020 con 0x7f177c07d4f0 2024-04-25T15:50:53.198 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.641+0000 7f1782d10700 1 -- 172.21.15.31:0/202263297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f177c02e810 con 0x7f177c07d4f0 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.641+0000 7f1779ffb700 1 -- 172.21.15.31:0/202263297 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f1764004260 con 0x7f177c07d4f0 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.642+0000 7f1782d10700 1 -- 172.21.15.31:0/202263297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 msgr2=0x7f177c19d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.642+0000 7f1782d10700 1 --2- 172.21.15.31:0/202263297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c19d400 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1764000c00 tx=0x7f1764004750 comp rx=0 tx=0).stop 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.642+0000 7f1782d10700 1 -- 172.21.15.31:0/202263297 shutdown_connections 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.642+0000 7f1782d10700 1 --2- 172.21.15.31:0/202263297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f177c07d4f0 0x7f177c19d400 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.642+0000 7f1782d10700 1 -- 172.21.15.31:0/202263297 >> 172.21.15.31:0/202263297 conn(0x7f177c103fa0 msgr2=0x7f177c105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.642+0000 7f1782d10700 1 -- 172.21.15.31:0/202263297 shutdown_connections 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:52.642+0000 7f1782d10700 1 -- 172.21.15.31:0/202263297 wait complete. 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:mon is available 2024-04-25T15:50:53.199 INFO:teuthology.orchestra.run.smithi031.stdout:Assimilating anything we can from ceph.conf... 2024-04-25T15:50:54.240 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:50:54.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T15:50:54.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:54.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T15:50:54.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T15:50:54.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.619+0000 7f16dd0f8700 1 Processor -- start 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.619+0000 7f16dd0f8700 1 -- start start 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.619+0000 7f16dd0f8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d8109db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.619+0000 7f16dd0f8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f16d810a3a0 con 0x7f16d81099a0 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.620+0000 7f16d6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d8109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.620+0000 7f16d6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d8109db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48652/0 (socket says 172.21.15.31:48652) 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.620+0000 7f16d6d9d700 1 -- 172.21.15.31:0/2953706009 learned_addr learned my addr 172.21.15.31:0/2953706009 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.620+0000 7f16d6d9d700 1 -- 172.21.15.31:0/2953706009 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d810aad0 con 0x7f16d81099a0 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.620+0000 7f16d6d9d700 1 --2- 172.21.15.31:0/2953706009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d8109db0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009e80 tx=0x7f16c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6951f929646e6ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:54.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.620+0000 7f16d5d9b700 1 -- 172.21.15.31:0/2953706009 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16c8017070 con 0x7f16d81099a0 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.620+0000 7f16d5d9b700 1 -- 172.21.15.31:0/2953706009 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c8003a20 con 0x7f16d81099a0 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.620+0000 7f16d5d9b700 1 -- 172.21.15.31:0/2953706009 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16c8003d20 con 0x7f16d81099a0 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.621+0000 7f16dd0f8700 1 -- 172.21.15.31:0/2953706009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 msgr2=0x7f16d8109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.621+0000 7f16dd0f8700 1 --2- 172.21.15.31:0/2953706009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d8109db0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009e80 tx=0x7f16c800b330 comp rx=0 tx=0).stop 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.621+0000 7f16dd0f8700 1 -- 172.21.15.31:0/2953706009 shutdown_connections 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.621+0000 7f16dd0f8700 1 --2- 172.21.15.31:0/2953706009 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d8109db0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.621+0000 7f16dd0f8700 1 -- 172.21.15.31:0/2953706009 >> 172.21.15.31:0/2953706009 conn(0x7f16d8105060 msgr2=0x7f16d81074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.621+0000 7f16dd0f8700 1 -- 172.21.15.31:0/2953706009 shutdown_connections 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.621+0000 7f16dd0f8700 1 -- 172.21.15.31:0/2953706009 wait complete. 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.622+0000 7f16dd0f8700 1 Processor -- start 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.622+0000 7f16dd0f8700 1 -- start start 2024-04-25T15:50:54.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.622+0000 7f16dd0f8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.622+0000 7f16dd0f8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f16d810b000 con 0x7f16d81099a0 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.622+0000 7f16d6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16d6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48660/0 (socket says 172.21.15.31:48660) 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16d6d9d700 1 -- 172.21.15.31:0/570738626 learned_addr learned my addr 172.21.15.31:0/570738626 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16d6d9d700 1 -- 172.21.15.31:0/570738626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d819fc90 con 0x7f16d81099a0 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16d6d9d700 1 --2- 172.21.15.31:0/570738626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d819f750 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f16c8004000 tx=0x7f16c8004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16cffff700 1 -- 172.21.15.31:0/570738626 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16c8022020 con 0x7f16d81099a0 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16cffff700 1 -- 172.21.15.31:0/570738626 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16c8018040 con 0x7f16d81099a0 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16dd0f8700 1 -- 172.21.15.31:0/570738626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16d819ff20 con 0x7f16d81099a0 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16cffff700 1 -- 172.21.15.31:0/570738626 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f16c8017070 con 0x7f16d81099a0 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16dd0f8700 1 -- 172.21.15.31:0/570738626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16d81a2b30 con 0x7f16d81099a0 2024-04-25T15:50:54.251 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.623+0000 7f16cffff700 1 -- 172.21.15.31:0/570738626 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f16c8025020 con 0x7f16d81099a0 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.624+0000 7f16cffff700 1 -- 172.21.15.31:0/570738626 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f16c800f050 con 0x7f16d81099a0 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.624+0000 7f16dd0f8700 1 -- 172.21.15.31:0/570738626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16d8064cf0 con 0x7f16d81099a0 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.627+0000 7f16cffff700 1 -- 172.21.15.31:0/570738626 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f16c802c030 con 0x7f16d81099a0 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.697+0000 7f16dd0f8700 1 -- 172.21.15.31:0/570738626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f16d81a2d00 con 0x7f16d81099a0 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.702+0000 7f16cffff700 1 -- 172.21.15.31:0/570738626 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f16c801ede0 con 0x7f16d81099a0 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.703+0000 7f16dd0f8700 1 -- 172.21.15.31:0/570738626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 msgr2=0x7f16d819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.703+0000 7f16cffff700 1 -- 172.21.15.31:0/570738626 <== mon.0 v2:172.21.15.31:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f16c8035070 con 0x7f16d81099a0 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.703+0000 7f16dd0f8700 1 --2- 172.21.15.31:0/570738626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d819f750 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f16c8004000 tx=0x7f16c8004ad0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.703+0000 7f16dd0f8700 1 -- 172.21.15.31:0/570738626 shutdown_connections 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.703+0000 7f16dd0f8700 1 --2- 172.21.15.31:0/570738626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16d81099a0 0x7f16d819f750 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.703+0000 7f16dd0f8700 1 -- 172.21.15.31:0/570738626 >> 172.21.15.31:0/570738626 conn(0x7f16d8105060 msgr2=0x7f16d807ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:54.252 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.703+0000 7f16dd0f8700 1 -- 172.21.15.31:0/570738626 shutdown_connections 2024-04-25T15:50:54.253 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:53.703+0000 7f16dd0f8700 1 -- 172.21.15.31:0/570738626 wait complete. 2024-04-25T15:50:54.253 INFO:teuthology.orchestra.run.smithi031.stdout:Generating new minimal ceph.conf... 2024-04-25T15:50:54.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:54 smithi031 ceph-mon[28768]: from='client.? 172.21.15.31:0/570738626' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T15:50:54.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:54 smithi031 ceph-mon[28768]: from='client.? 172.21.15.31:0/570738626' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-25T15:50:55.269 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.675+0000 7fddefc62700 1 Processor -- start 2024-04-25T15:50:55.269 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.675+0000 7fddefc62700 1 -- start start 2024-04-25T15:50:55.269 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.675+0000 7fddefc62700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde8108d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:55.269 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.675+0000 7fddefc62700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdde81092f0 con 0x7fdde81088f0 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.676+0000 7fdded9fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde8108d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.676+0000 7fdded9fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde8108d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44784/0 (socket says 172.21.15.31:44784) 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.676+0000 7fdded9fe700 1 -- 172.21.15.31:0/1934597549 learned_addr learned my addr 172.21.15.31:0/1934597549 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.676+0000 7fdded9fe700 1 -- 172.21.15.31:0/1934597549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde8109a20 con 0x7fdde81088f0 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.676+0000 7fdded9fe700 1 --2- 172.21.15.31:0/1934597549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde8108d00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdde40098a0 tx=0x7fdde4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a46416dabcba768 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.676+0000 7fddec9fc700 1 -- 172.21.15.31:0/1934597549 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdde4004160 con 0x7fdde81088f0 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.676+0000 7fddec9fc700 1 -- 172.21.15.31:0/1934597549 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdde4004300 con 0x7fdde81088f0 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.676+0000 7fddec9fc700 1 -- 172.21.15.31:0/1934597549 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdde4011af0 con 0x7fdde81088f0 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.677+0000 7fddefc62700 1 -- 172.21.15.31:0/1934597549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 msgr2=0x7fdde8108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.677+0000 7fddefc62700 1 --2- 172.21.15.31:0/1934597549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde8108d00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdde40098a0 tx=0x7fdde4009bb0 comp rx=0 tx=0).stop 2024-04-25T15:50:55.270 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.677+0000 7fddefc62700 1 -- 172.21.15.31:0/1934597549 shutdown_connections 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.677+0000 7fddefc62700 1 --2- 172.21.15.31:0/1934597549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde8108d00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.677+0000 7fddefc62700 1 -- 172.21.15.31:0/1934597549 >> 172.21.15.31:0/1934597549 conn(0x7fdde8103f70 msgr2=0x7fdde81063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.677+0000 7fddefc62700 1 -- 172.21.15.31:0/1934597549 shutdown_connections 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.677+0000 7fddefc62700 1 -- 172.21.15.31:0/1934597549 wait complete. 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.678+0000 7fddefc62700 1 Processor -- start 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.678+0000 7fddefc62700 1 -- start start 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.678+0000 7fddefc62700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde81060d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.678+0000 7fddefc62700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdde810cdd0 con 0x7fdde81088f0 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.678+0000 7fdded9fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde81060d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fdded9fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde81060d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44790/0 (socket says 172.21.15.31:44790) 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fdded9fe700 1 -- 172.21.15.31:0/2422579076 learned_addr learned my addr 172.21.15.31:0/2422579076 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:50:55.271 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fdded9fe700 1 -- 172.21.15.31:0/2422579076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdde8106610 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fdded9fe700 1 --2- 172.21.15.31:0/2422579076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde81060d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdde4011890 tx=0x7fdde4011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fdddeffd700 1 -- 172.21.15.31:0/2422579076 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdde40039e0 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fdddeffd700 1 -- 172.21.15.31:0/2422579076 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdde4003b80 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fddefc62700 1 -- 172.21.15.31:0/2422579076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdde8104860 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fdddeffd700 1 -- 172.21.15.31:0/2422579076 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdde40246e0 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fddefc62700 1 -- 172.21.15.31:0/2422579076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdde8104d40 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fdddeffd700 1 -- 172.21.15.31:0/2422579076 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdde4022020 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.679+0000 7fdddeffd700 1 -- 172.21.15.31:0/2422579076 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdde402dc80 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.680+0000 7fddefc62700 1 -- 172.21.15.31:0/2422579076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdde8064cf0 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.683+0000 7fdddeffd700 1 -- 172.21.15.31:0/2422579076 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdde4024e20 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.753+0000 7fddefc62700 1 -- 172.21.15.31:0/2422579076 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fdde8196ed0 con 0x7fdde81088f0 2024-04-25T15:50:55.272 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.754+0000 7fdddeffd700 1 -- 172.21.15.31:0/2422579076 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fdde401aa70 con 0x7fdde81088f0 2024-04-25T15:50:55.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.755+0000 7fddefc62700 1 -- 172.21.15.31:0/2422579076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 msgr2=0x7fdde81060d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:50:55.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.755+0000 7fddefc62700 1 --2- 172.21.15.31:0/2422579076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde81060d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdde4011890 tx=0x7fdde4011970 comp rx=0 tx=0).stop 2024-04-25T15:50:55.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.755+0000 7fddefc62700 1 -- 172.21.15.31:0/2422579076 shutdown_connections 2024-04-25T15:50:55.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.755+0000 7fddefc62700 1 --2- 172.21.15.31:0/2422579076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdde81088f0 0x7fdde81060d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:50:55.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.755+0000 7fddefc62700 1 -- 172.21.15.31:0/2422579076 >> 172.21.15.31:0/2422579076 conn(0x7fdde8103f70 msgr2=0x7fdde8194350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:50:55.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.755+0000 7fddefc62700 1 -- 172.21.15.31:0/2422579076 shutdown_connections 2024-04-25T15:50:55.273 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:54.755+0000 7fddefc62700 1 -- 172.21.15.31:0/2422579076 wait complete. 2024-04-25T15:50:55.273 INFO:teuthology.orchestra.run.smithi031.stdout:Restarting the monitor... 2024-04-25T15:50:55.285 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:55 smithi031 ceph-mon[28768]: from='client.? 172.21.15.31:0/2422579076' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:50:55.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:55 smithi031 systemd[1]: Stopping Ceph mon.a for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:50:56.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a[28764]: 2024-04-25T15:50:56.490+0000 7f98a8224700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T15:50:56.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a[28764]: 2024-04-25T15:50:56.490+0000 7f98a8224700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-25T15:50:56.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:56 smithi031 podman[29012]: 2024-04-25 15:50:56.531960817 +0000 UTC m=+1.096086175 container died fd39c855a3c4ba5b430b151db60c423363d3385e4cefb8144738510ca02cc484 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T15:50:57.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:56 smithi031 podman[29012]: 2024-04-25 15:50:56.848069705 +0000 UTC m=+1.412195053 container cleanup fd39c855a3c4ba5b430b151db60c423363d3385e4cefb8144738510ca02cc484 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T15:50:57.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:56 smithi031 bash[29012]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a 2024-04-25T15:50:57.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:57 smithi031 podman[29026]: 2024-04-25 15:50:57.661653086 +0000 UTC m=+1.132440661 container remove fd39c855a3c4ba5b430b151db60c423363d3385e4cefb8144738510ca02cc484 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T15:50:57.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:57 smithi031 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-25T15:50:57.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:57 smithi031 systemd[1]: Stopped Ceph mon.a for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:50:57.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:57 smithi031 systemd[1]: Starting Ceph mon.a for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:50:58.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:58 smithi031 podman[29121]: 2024-04-25 15:50:58.048804181 +0000 UTC m=+0.018904662 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:50:59.133 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:58 smithi031 podman[29121]: 2024-04-25 15:50:58.876045099 +0000 UTC m=+0.846145578 container create 3c77fdc22f43d6e43451cecbbab44433d99a3d1440a2a6dea41d3bf7aef738c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T15:50:59.133 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 podman[29121]: 2024-04-25 15:50:59.092605247 +0000 UTC m=+1.062705727 container init 3c77fdc22f43d6e43451cecbbab44433d99a3d1440a2a6dea41d3bf7aef738c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T15:50:59.133 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 podman[29121]: 2024-04-25 15:50:59.096409118 +0000 UTC m=+1.066509598 container start 3c77fdc22f43d6e43451cecbbab44433d99a3d1440a2a6dea41d3bf7aef738c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T15:50:59.172 INFO:teuthology.orchestra.run.smithi031.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-25T15:50:59.388 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T15:50:59.388 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T15:50:59.388 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: pidfile_write: ignore empty --pid-file 2024-04-25T15:50:59.388 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: load: jerasure load: lrc 2024-04-25T15:50:59.388 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: RocksDB version: 7.9.2 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Git sha 0 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: DB SUMMARY 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: DB Session ID: 5KA3MSN20IP4KANN44KH 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: CURRENT file: CURRENT 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104076 ; 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.error_if_exists: 0 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.create_if_missing: 0 2024-04-25T15:50:59.389 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.paranoid_checks: 1 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.env: 0x5629342f0500 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.info_log: 0x562934f23fc0 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.statistics: (nil) 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.use_fsync: 0 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_log_file_size: 0 2024-04-25T15:50:59.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.allow_fallocate: 1 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.use_direct_reads: 0 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.db_log_dir: 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.wal_dir: 2024-04-25T15:50:59.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.write_buffer_manager: 0x562934ef48c0 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T15:50:59.393 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.unordered_write: 0 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.row_cache: None 2024-04-25T15:50:59.394 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.wal_filter: None 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.two_write_queues: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.wal_compression: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.atomic_flush: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.log_readahead_size: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T15:50:59.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_background_jobs: 2 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_background_compactions: -1 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_subcompactions: 1 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T15:50:59.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_open_files: -1 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_background_flushes: -1 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Compression algorithms supported: 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: kZSTD supported: 0 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: kXpressCompression supported: 0 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: kZlibCompression supported: 1 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: kSnappyCompression supported: 1 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: kLZ4Compression supported: 1 2024-04-25T15:50:59.397 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: kBZip2Compression supported: 0 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.merge_operator: 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_filter: None 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56293502a1a0) 2024-04-25T15:50:59.398 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks: 1 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_top_level_index_and_filter: 1 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_type: 0 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_index_type: 0 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_shortening: 1 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: checksum: 4 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: no_block_cache: 0 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache: 0x562934f271f0 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_name: BinnedLRUCache 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_options: 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: capacity : 536870912 2024-04-25T15:50:59.399 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_shard_bits : 4 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: strict_capacity_limit : 0 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: high_pri_pool_ratio: 0.000 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_compressed: (nil) 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: persistent_cache: (nil) 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size: 4096 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size_deviation: 10 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_restart_interval: 16 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_block_restart_interval: 1 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: metadata_block_size: 4096 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: partition_filters: 0 2024-04-25T15:50:59.401 INFO:journalctl@ceph.mon.a.smithi031.stdout: use_delta_encoding: 1 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: filter_policy: bloomfilter 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: whole_key_filtering: 1 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: verify_compression: 0 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: read_amp_bytes_per_bit: 0 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: format_version: 5 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_index_compression: 1 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_align: 0 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: max_auto_readahead_size: 262144 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: prepopulate_block_cache: 0 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: initial_auto_readahead_size: 8192 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T15:50:59.402 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression: NoCompression 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.num_levels: 7 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T15:50:59.403 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T15:50:59.404 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T15:50:59.405 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T15:50:59.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.inplace_update_support: 0 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.bloom_locality: 0 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.max_successive_merges: 0 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T15:50:59.407 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.ttl: 2592000 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.enable_blob_files: false 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.min_blob_size: 0 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T15:50:59.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0e630218-5c62-4311-b034-dd3bcbf16bf8 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714060259139874, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714060259141163, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99304, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 322, "table_properties": {"data_size": 97250, "index_size": 310, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13917, "raw_average_key_size": 49, "raw_value_size": 89644, "raw_average_value_size": 319, "num_data_blocks": 12, "num_entries": 281, "num_filter_entries": 281, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714060259, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0e630218-5c62-4311-b034-dd3bcbf16bf8", "db_session_id": "5KA3MSN20IP4KANN44KH", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714060259141231, "job": 1, "event": "recovery_finished"} 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x562934f0bc00 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: DB pointer 0x562939cfe000 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** DB Stats ** 2024-04-25T15:50:59.409 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: L0 2/0 98.84 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: Sum 2/0 98.84 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 137.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T15:50:59.410 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 137.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative compaction: 0.00 GB write, 26.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T15:50:59.411 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval compaction: 0.00 GB write, 26.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache BinnedLRUCache@0x562934f271f0#2 capacity: 512.00 MB usage: 1.41 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.91 KB,0.000172853%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: starting mon.a rank 0 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mon.a@-1(???) e1 preinit fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mon.a@-1(???).mds e1 new map 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mon.a@-1(???).mds e1 print_map 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: e1 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: legacy client fscid: -1 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout: No filesystems configured 2024-04-25T15:50:59.412 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 bash[29121]: 3c77fdc22f43d6e43451cecbbab44433d99a3d1440a2a6dea41d3bf7aef738c5 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 systemd[1]: Started Ceph mon.a for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: monmap epoch 1 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: last_changed 2024-04-25T15:50:40.276173+0000 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: created 2024-04-25T15:50:40.276173+0000 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: min_mon_release 19 (squid) 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: election_strategy: 1 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: fsmap 2024-04-25T15:50:59.413 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: osdmap e1: 0 total, 0 up, 0 in 2024-04-25T15:50:59.414 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:50:59 smithi031 ceph-mon[29140]: mgrmap e1: no daemons active 2024-04-25T15:51:00.229 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.610+0000 7f21396f8700 1 Processor -- start 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.610+0000 7f21396f8700 1 -- start start 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.610+0000 7f21396f8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213407d680 0x7f213410c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.610+0000 7f21396f8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f213410cb20 con 0x7f213407d680 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.611+0000 7f2132ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213407d680 0x7f213410c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.611+0000 7f2132ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213407d680 0x7f213410c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44796/0 (socket says 172.21.15.31:44796) 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.611+0000 7f2132ffd700 1 -- 172.21.15.31:0/1384846795 learned_addr learned my addr 172.21.15.31:0/1384846795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.611+0000 7f2132ffd700 1 -- 172.21.15.31:0/1384846795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f213410cca0 con 0x7f213407d680 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.611+0000 7f2132ffd700 1 --2- 172.21.15.31:0/1384846795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213407d680 0x7f213410c510 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f21240098a0 tx=0x7f2124009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c34f925a2cf8a1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.611+0000 7f2131ffb700 1 -- 172.21.15.31:0/1384846795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2124004160 con 0x7f213407d680 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.611+0000 7f2131ffb700 1 -- 172.21.15.31:0/1384846795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2124004300 con 0x7f213407d680 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.611+0000 7f2131ffb700 1 -- 172.21.15.31:0/1384846795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2124011af0 con 0x7f213407d680 2024-04-25T15:51:00.230 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.612+0000 7f21396f8700 1 -- 172.21.15.31:0/1384846795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213407d680 msgr2=0x7f213410c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.612+0000 7f21396f8700 1 --2- 172.21.15.31:0/1384846795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213407d680 0x7f213410c510 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f21240098a0 tx=0x7f2124009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.612+0000 7f21396f8700 1 -- 172.21.15.31:0/1384846795 shutdown_connections 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.612+0000 7f21396f8700 1 --2- 172.21.15.31:0/1384846795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213407d680 0x7f213410c510 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.612+0000 7f21396f8700 1 -- 172.21.15.31:0/1384846795 >> 172.21.15.31:0/1384846795 conn(0x7f2134103f60 msgr2=0x7f2134106410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.612+0000 7f21396f8700 1 -- 172.21.15.31:0/1384846795 shutdown_connections 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.612+0000 7f21396f8700 1 -- 172.21.15.31:0/1384846795 wait complete. 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.613+0000 7f21396f8700 1 Processor -- start 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.613+0000 7f21396f8700 1 -- start start 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.613+0000 7f21396f8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213419d8a0 0x7f213419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.613+0000 7f21396f8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f213410dd20 con 0x7f213419d8a0 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f2132ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213419d8a0 0x7f213419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f2132ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213419d8a0 0x7f213419dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44806/0 (socket says 172.21.15.31:44806) 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f2132ffd700 1 -- 172.21.15.31:0/1175555008 learned_addr learned my addr 172.21.15.31:0/1175555008 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:00.231 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f2132ffd700 1 -- 172.21.15.31:0/1175555008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f213419e1f0 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f2132ffd700 1 --2- 172.21.15.31:0/1175555008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213419d8a0 0x7f213419dcb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2124009fb0 tx=0x7f2124011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f211bfff700 1 -- 172.21.15.31:0/1175555008 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f21240036a0 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f21396f8700 1 -- 172.21.15.31:0/1175555008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f213419e480 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f211bfff700 1 -- 172.21.15.31:0/1175555008 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f212401a430 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f211bfff700 1 -- 172.21.15.31:0/1175555008 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2124019550 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f21396f8700 1 -- 172.21.15.31:0/1175555008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21341a0f50 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.614+0000 7f211bfff700 1 -- 172.21.15.31:0/1175555008 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2124019be0 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.615+0000 7f211bfff700 1 -- 172.21.15.31:0/1175555008 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2124024730 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.615+0000 7f21396f8700 1 -- 172.21.15.31:0/1175555008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2134064cf0 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.618+0000 7f211bfff700 1 -- 172.21.15.31:0/1175555008 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f2124043e50 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.688+0000 7f21396f8700 1 -- 172.21.15.31:0/1175555008 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f21341a12a0 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.690+0000 7f211bfff700 1 -- 172.21.15.31:0/1175555008 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f2124032020 con 0x7f213419d8a0 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.691+0000 7f21396f8700 1 -- 172.21.15.31:0/1175555008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213419d8a0 msgr2=0x7f213419dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.691+0000 7f21396f8700 1 --2- 172.21.15.31:0/1175555008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213419d8a0 0x7f213419dcb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f2124009fb0 tx=0x7f2124011940 comp rx=0 tx=0).stop 2024-04-25T15:51:00.232 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.691+0000 7f21396f8700 1 -- 172.21.15.31:0/1175555008 shutdown_connections 2024-04-25T15:51:00.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.691+0000 7f21396f8700 1 --2- 172.21.15.31:0/1175555008 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f213419d8a0 0x7f213419dcb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:00.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.691+0000 7f21396f8700 1 -- 172.21.15.31:0/1175555008 >> 172.21.15.31:0/1175555008 conn(0x7f2134103f60 msgr2=0x7f2134105990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:00.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.692+0000 7f21396f8700 1 -- 172.21.15.31:0/1175555008 shutdown_connections 2024-04-25T15:51:00.233 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:50:59.692+0000 7f21396f8700 1 -- 172.21.15.31:0/1175555008 wait complete. 2024-04-25T15:51:00.233 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-25T15:51:00.255 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-25T15:51:00.256 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mgr... 2024-04-25T15:51:00.256 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-25T15:51:00.256 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-25T15:51:00.513 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.x 2024-04-25T15:51:00.513 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-25T15:51:00.663 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-74fd3742-031b-11ef-bc93-c7b262605968.target.wants/ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-74fd3742-031b-11ef-bc93-c7b262605968@.service. 2024-04-25T15:51:00.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:00 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1175555008' entity='client.admin' 2024-04-25T15:51:00.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:00 smithi031 podman[29360]: 2024-04-25 15:51:00.806255758 +0000 UTC m=+0.018615651 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T15:51:01.141 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T15:51:01.141 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-25T15:51:01.148 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T15:51:01.149 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T15:51:01.149 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-25T15:51:01.149 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2024-04-25T15:51:01.157 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T15:51:01.157 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-25T15:51:01.165 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T15:51:01.165 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T15:51:01.165 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-25T15:51:01.165 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-25T15:51:01.165 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr to start... 2024-04-25T15:51:01.165 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr... 2024-04-25T15:51:01.226 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:01 smithi031 podman[29360]: 2024-04-25 15:51:01.052292242 +0000 UTC m=+0.264652126 container init 8ac56c1f2c768fd7526ba22c4f7f3042b84356f5d215de729c304af1b4c2a315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-25T15:51:01.226 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:01 smithi031 podman[29360]: 2024-04-25 15:51:01.05484325 +0000 UTC m=+0.267203128 container start 8ac56c1f2c768fd7526ba22c4f7f3042b84356f5d215de729c304af1b4c2a315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-25T15:51:01.226 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:01 smithi031 bash[29360]: 8ac56c1f2c768fd7526ba22c4f7f3042b84356f5d215de729c304af1b4c2a315 2024-04-25T15:51:01.226 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:01 smithi031 systemd[1]: Started Ceph mgr.x for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:51:01.528 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:01 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:01.225+0000 7f9db5f28200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:51:01.528 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:01 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:01.420+0000 7f9db5f28200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:51:01.528 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:01 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:01.526+0000 7f9db5f28200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:51:02.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:01 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1341985318' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T15:51:02.200 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:01 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:01.864+0000 7f9db5f28200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:51:02.200 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:01 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:01.952+0000 7f9db5f28200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:51:02.962 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:02.973 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T15:51:02.974 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T15:51:02.975 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:02.976 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T15:50:52.066479+0000", 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.660+0000 7faba1329700 1 Processor -- start 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.660+0000 7faba1329700 1 -- start start 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.660+0000 7faba1329700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c10c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.660+0000 7faba1329700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c07cd70 con 0x7fab9c10a480 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.660+0000 7fab9affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c10c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.660+0000 7fab9affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c10c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44842/0 (socket says 172.21.15.31:44842) 2024-04-25T15:51:02.977 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.660+0000 7fab9affd700 1 -- 172.21.15.31:0/1999363518 learned_addr learned my addr 172.21.15.31:0/1999363518 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.660+0000 7fab9affd700 1 -- 172.21.15.31:0/1999363518 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c10cde0 con 0x7fab9c10a480 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.661+0000 7fab9affd700 1 --2- 172.21.15.31:0/1999363518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c10c8a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fab8c0098a0 tx=0x7fab8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44438b0a4bad9b82 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.661+0000 7fab99ffb700 1 -- 172.21.15.31:0/1999363518 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab8c004160 con 0x7fab9c10a480 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.661+0000 7fab99ffb700 1 -- 172.21.15.31:0/1999363518 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fab8c004300 con 0x7fab9c10a480 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.661+0000 7fab99ffb700 1 -- 172.21.15.31:0/1999363518 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab8c011af0 con 0x7fab9c10a480 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.662+0000 7faba1329700 1 -- 172.21.15.31:0/1999363518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 msgr2=0x7fab9c10c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.662+0000 7faba1329700 1 --2- 172.21.15.31:0/1999363518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c10c8a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fab8c0098a0 tx=0x7fab8c009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.662+0000 7faba1329700 1 -- 172.21.15.31:0/1999363518 shutdown_connections 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.662+0000 7faba1329700 1 --2- 172.21.15.31:0/1999363518 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c10c8a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.662+0000 7faba1329700 1 -- 172.21.15.31:0/1999363518 >> 172.21.15.31:0/1999363518 conn(0x7fab9c103f80 msgr2=0x7fab9c1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.662+0000 7faba1329700 1 -- 172.21.15.31:0/1999363518 shutdown_connections 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.662+0000 7faba1329700 1 -- 172.21.15.31:0/1999363518 wait complete. 2024-04-25T15:51:02.978 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.662+0000 7faba1329700 1 Processor -- start 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.663+0000 7faba1329700 1 -- start start 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.663+0000 7faba1329700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.663+0000 7faba1329700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c10dce0 con 0x7fab9c10a480 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.663+0000 7fab9affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.663+0000 7fab9affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c19dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44858/0 (socket says 172.21.15.31:44858) 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.663+0000 7fab9affd700 1 -- 172.21.15.31:0/1341985318 learned_addr learned my addr 172.21.15.31:0/1341985318 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.663+0000 7fab9affd700 1 -- 172.21.15.31:0/1341985318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c19e240 con 0x7fab9c10a480 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.663+0000 7fab9affd700 1 --2- 172.21.15.31:0/1341985318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c19dd00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009fb0 tx=0x7fab8c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.664+0000 7fab83fff700 1 -- 172.21.15.31:0/1341985318 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab8c0036a0 con 0x7fab9c10a480 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.664+0000 7fab83fff700 1 -- 172.21.15.31:0/1341985318 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fab8c01a430 con 0x7fab9c10a480 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.664+0000 7faba1329700 1 -- 172.21.15.31:0/1341985318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab9c19e4d0 con 0x7fab9c10a480 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.664+0000 7fab83fff700 1 -- 172.21.15.31:0/1341985318 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fab8c019550 con 0x7fab9c10a480 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.664+0000 7faba1329700 1 -- 172.21.15.31:0/1341985318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab9c1a10e0 con 0x7fab9c10a480 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.664+0000 7fab83fff700 1 -- 172.21.15.31:0/1341985318 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fab8c019b90 con 0x7fab9c10a480 2024-04-25T15:51:02.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.664+0000 7fab83fff700 1 -- 172.21.15.31:0/1341985318 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fab8c024730 con 0x7fab9c10a480 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.665+0000 7faba1329700 1 -- 172.21.15.31:0/1341985318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab9c064cf0 con 0x7fab9c10a480 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.667+0000 7fab83fff700 1 -- 172.21.15.31:0/1341985318 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fab8c043e50 con 0x7fab9c10a480 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.740+0000 7faba1329700 1 -- 172.21.15.31:0/1341985318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fab9c1a12b0 con 0x7fab9c10a480 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.741+0000 7fab83fff700 1 -- 172.21.15.31:0/1341985318 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fab8c032020 con 0x7fab9c10a480 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.742+0000 7faba1329700 1 -- 172.21.15.31:0/1341985318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 msgr2=0x7fab9c19dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.742+0000 7faba1329700 1 --2- 172.21.15.31:0/1341985318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c19dd00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009fb0 tx=0x7fab8c011940 comp rx=0 tx=0).stop 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.742+0000 7faba1329700 1 -- 172.21.15.31:0/1341985318 shutdown_connections 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.742+0000 7faba1329700 1 --2- 172.21.15.31:0/1341985318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab9c10a480 0x7fab9c19dd00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.742+0000 7faba1329700 1 -- 172.21.15.31:0/1341985318 >> 172.21.15.31:0/1341985318 conn(0x7fab9c103f80 msgr2=0x7fab9c105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.742+0000 7faba1329700 1 -- 172.21.15.31:0/1341985318 shutdown_connections 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:01.742+0000 7faba1329700 1 -- 172.21.15.31:0/1341985318 wait complete. 2024-04-25T15:51:02.980 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (1/15)... 2024-04-25T15:51:03.567 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:03 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:03.215+0000 7f9db5f28200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:51:03.568 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:03 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:03.391+0000 7f9db5f28200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:51:03.568 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:03 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:03.565+0000 7f9db5f28200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:51:04.170 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:03 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:03.865+0000 7f9db5f28200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:51:04.171 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:03 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:03.960+0000 7f9db5f28200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:51:04.171 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:04 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:04.169+0000 7f9db5f28200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:51:04.449 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:04 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:04.367+0000 7f9db5f28200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:51:05.311 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:05 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:05.003+0000 7f9db5f28200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T15:51:05.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:05 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1778414772' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T15:51:05.699 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:05 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:05.451+0000 7f9db5f28200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:51:05.699 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:05 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:05.554+0000 7f9db5f28200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:51:06.080 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:05 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:05.764+0000 7f9db5f28200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-25T15:51:06.094 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T15:51:06.095 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:06.096 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:06.097 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T15:50:52.066479+0000", 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.421+0000 7f4e35218700 1 Processor -- start 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.421+0000 7f4e35218700 1 -- start start 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.421+0000 7f4e35218700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3010c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.421+0000 7f4e35218700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3010cae0 con 0x7f4e3007cc40 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.421+0000 7f4e2ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3010c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.421+0000 7f4e2ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3010c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51684/0 (socket says 172.21.15.31:51684) 2024-04-25T15:51:06.098 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.421+0000 7f4e2ed9d700 1 -- 172.21.15.31:0/1103138040 learned_addr learned my addr 172.21.15.31:0/1103138040 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.421+0000 7f4e2ed9d700 1 -- 172.21.15.31:0/1103138040 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3010cc60 con 0x7f4e3007cc40 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.422+0000 7f4e2ed9d700 1 --2- 172.21.15.31:0/1103138040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3010c4f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4e200098a0 tx=0x7f4e20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89e24204a36e40d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.422+0000 7f4e2dd9b700 1 -- 172.21.15.31:0/1103138040 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e20004160 con 0x7f4e3007cc40 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.422+0000 7f4e2dd9b700 1 -- 172.21.15.31:0/1103138040 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4e20004300 con 0x7f4e3007cc40 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.422+0000 7f4e2dd9b700 1 -- 172.21.15.31:0/1103138040 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e20011af0 con 0x7f4e3007cc40 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.423+0000 7f4e35218700 1 -- 172.21.15.31:0/1103138040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 msgr2=0x7f4e3010c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.423+0000 7f4e35218700 1 --2- 172.21.15.31:0/1103138040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3010c4f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4e200098a0 tx=0x7f4e20009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.423+0000 7f4e35218700 1 -- 172.21.15.31:0/1103138040 shutdown_connections 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.423+0000 7f4e35218700 1 --2- 172.21.15.31:0/1103138040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3010c4f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.423+0000 7f4e35218700 1 -- 172.21.15.31:0/1103138040 >> 172.21.15.31:0/1103138040 conn(0x7f4e30103f80 msgr2=0x7f4e301063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.423+0000 7f4e35218700 1 -- 172.21.15.31:0/1103138040 shutdown_connections 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.423+0000 7f4e35218700 1 -- 172.21.15.31:0/1103138040 wait complete. 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.424+0000 7f4e35218700 1 Processor -- start 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.424+0000 7f4e35218700 1 -- start start 2024-04-25T15:51:06.099 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.424+0000 7f4e35218700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3019dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.424+0000 7f4e35218700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3010dce0 con 0x7f4e3007cc40 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.424+0000 7f4e2ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3019dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.424+0000 7f4e2ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3019dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51696/0 (socket says 172.21.15.31:51696) 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.424+0000 7f4e2ed9d700 1 -- 172.21.15.31:0/1778414772 learned_addr learned my addr 172.21.15.31:0/1778414772 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.425+0000 7f4e2ed9d700 1 -- 172.21.15.31:0/1778414772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3019e220 con 0x7f4e3007cc40 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.425+0000 7f4e2ed9d700 1 --2- 172.21.15.31:0/1778414772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3019dce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4e200117b0 tx=0x7f4e20011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.425+0000 7f4e17fff700 1 -- 172.21.15.31:0/1778414772 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e20003af0 con 0x7f4e3007cc40 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.425+0000 7f4e17fff700 1 -- 172.21.15.31:0/1778414772 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4e20020070 con 0x7f4e3007cc40 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.425+0000 7f4e17fff700 1 -- 172.21.15.31:0/1778414772 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e20019630 con 0x7f4e3007cc40 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.425+0000 7f4e35218700 1 -- 172.21.15.31:0/1778414772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3019e4b0 con 0x7f4e3007cc40 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.425+0000 7f4e35218700 1 -- 172.21.15.31:0/1778414772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e301a10c0 con 0x7f4e3007cc40 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.425+0000 7f4e17fff700 1 -- 172.21.15.31:0/1778414772 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4e20019c30 con 0x7f4e3007cc40 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.425+0000 7f4e17fff700 1 -- 172.21.15.31:0/1778414772 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4e2000b720 con 0x7f4e3007cc40 2024-04-25T15:51:06.100 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.426+0000 7f4e35218700 1 -- 172.21.15.31:0/1778414772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e30064cf0 con 0x7f4e3007cc40 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.429+0000 7f4e17fff700 1 -- 172.21.15.31:0/1778414772 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f4e200042a0 con 0x7f4e3007cc40 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.502+0000 7f4e35218700 1 -- 172.21.15.31:0/1778414772 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4e301a1290 con 0x7f4e3007cc40 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.503+0000 7f4e17fff700 1 -- 172.21.15.31:0/1778414772 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4e2001a9c0 con 0x7f4e3007cc40 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.504+0000 7f4e35218700 1 -- 172.21.15.31:0/1778414772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 msgr2=0x7f4e3019dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.504+0000 7f4e35218700 1 --2- 172.21.15.31:0/1778414772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3019dce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4e200117b0 tx=0x7f4e20011b70 comp rx=0 tx=0).stop 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.504+0000 7f4e35218700 1 -- 172.21.15.31:0/1778414772 shutdown_connections 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.504+0000 7f4e35218700 1 --2- 172.21.15.31:0/1778414772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4e3007cc40 0x7f4e3019dce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.504+0000 7f4e35218700 1 -- 172.21.15.31:0/1778414772 >> 172.21.15.31:0/1778414772 conn(0x7f4e30103f80 msgr2=0x7f4e30105970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.504+0000 7f4e35218700 1 -- 172.21.15.31:0/1778414772 shutdown_connections 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:05.504+0000 7f4e35218700 1 -- 172.21.15.31:0/1778414772 wait complete. 2024-04-25T15:51:06.101 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (2/15)... 2024-04-25T15:51:06.404 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:06 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:06.307+0000 7f9db5f28200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:51:06.404 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:06 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:06.402+0000 7f9db5f28200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:51:06.949 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:06 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:06.656+0000 7f9db5f28200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:51:06.949 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:06 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:06.794+0000 7f9db5f28200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:51:06.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:06 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:06.892+0000 7f9db5f28200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:51:07.340 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:07 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:07.012+0000 7f9db5f28200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:51:07.340 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:07 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:07.338+0000 7f9db5f28200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:51:08.247 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:07 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:07.964+0000 7f9db5f28200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:51:08.247 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:08.190+0000 7f9db5f28200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:51:08.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: Activating manager daemon x 2024-04-25T15:51:08.516 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: mgrmap e2: x(active, starting, since 0.00341515s) 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: Manager daemon x is now available 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' 2024-04-25T15:51:08.517 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:08 smithi031 ceph-mon[29140]: from='mgr.14100 172.21.15.31:0/1461547051' entity='mgr.x' 2024-04-25T15:51:09.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:51:09.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T15:51:09.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T15:51:09.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T15:51:09.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T15:51:09.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T15:51:09.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T15:51:09.176 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T15:51:09.177 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:09.177 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T15:51:09.177 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:09.177 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T15:51:09.177 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T15:51:09.177 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T15:51:09.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T15:51:09.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T15:51:09.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T15:51:09.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:09.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T15:51:09.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T15:51:09.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T15:51:09.185 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-25T15:51:09.186 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T15:50:52.066479+0000", 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T15:51:09.187 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.541+0000 7f334a29e700 1 Processor -- start 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.542+0000 7f334a29e700 1 -- start start 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.542+0000 7f334a29e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334407d680 0x7f334410c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.542+0000 7f334a29e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f334410cb20 con 0x7f334407d680 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.542+0000 7f3343fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334407d680 0x7f334410c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.542+0000 7f3343fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334407d680 0x7f334410c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51760/0 (socket says 172.21.15.31:51760) 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.542+0000 7f3343fff700 1 -- 172.21.15.31:0/1636576547 learned_addr learned my addr 172.21.15.31:0/1636576547 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.542+0000 7f3343fff700 1 -- 172.21.15.31:0/1636576547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334410cca0 con 0x7f334407d680 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.542+0000 7f3343fff700 1 --2- 172.21.15.31:0/1636576547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334407d680 0x7f334410c510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f332c0098a0 tx=0x7f332c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1650acb5bbb36727 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.543+0000 7f3342ffd700 1 -- 172.21.15.31:0/1636576547 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f332c004160 con 0x7f334407d680 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.543+0000 7f3342ffd700 1 -- 172.21.15.31:0/1636576547 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f332c004300 con 0x7f334407d680 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.543+0000 7f3342ffd700 1 -- 172.21.15.31:0/1636576547 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f332c011af0 con 0x7f334407d680 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.543+0000 7f334a29e700 1 -- 172.21.15.31:0/1636576547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334407d680 msgr2=0x7f334410c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.543+0000 7f334a29e700 1 --2- 172.21.15.31:0/1636576547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334407d680 0x7f334410c510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f332c0098a0 tx=0x7f332c009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.543+0000 7f334a29e700 1 -- 172.21.15.31:0/1636576547 shutdown_connections 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.543+0000 7f334a29e700 1 --2- 172.21.15.31:0/1636576547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334407d680 0x7f334410c510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.188 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.543+0000 7f334a29e700 1 -- 172.21.15.31:0/1636576547 >> 172.21.15.31:0/1636576547 conn(0x7f3344103f60 msgr2=0x7f3344106410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.543+0000 7f334a29e700 1 -- 172.21.15.31:0/1636576547 shutdown_connections 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.544+0000 7f334a29e700 1 -- 172.21.15.31:0/1636576547 wait complete. 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.544+0000 7f334a29e700 1 Processor -- start 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.544+0000 7f334a29e700 1 -- start start 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f334a29e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334419d8a0 0x7f334419dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f334a29e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f334410dd20 con 0x7f334419d8a0 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f3343fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334419d8a0 0x7f334419dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f3343fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334419d8a0 0x7f334419dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51772/0 (socket says 172.21.15.31:51772) 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f3343fff700 1 -- 172.21.15.31:0/1132062706 learned_addr learned my addr 172.21.15.31:0/1132062706 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f3343fff700 1 -- 172.21.15.31:0/1132062706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334419e1f0 con 0x7f334419d8a0 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f3343fff700 1 --2- 172.21.15.31:0/1132062706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334419d8a0 0x7f334419dcb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f332c0117b0 tx=0x7f332c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f33417fa700 1 -- 172.21.15.31:0/1132062706 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f332c003af0 con 0x7f334419d8a0 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f33417fa700 1 -- 172.21.15.31:0/1132062706 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f332c020070 con 0x7f334419d8a0 2024-04-25T15:51:09.189 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f334a29e700 1 -- 172.21.15.31:0/1132062706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f334419e480 con 0x7f334419d8a0 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f33417fa700 1 -- 172.21.15.31:0/1132062706 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f332c019630 con 0x7f334419d8a0 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.545+0000 7f334a29e700 1 -- 172.21.15.31:0/1132062706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33441a0f50 con 0x7f334419d8a0 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.547+0000 7f33417fa700 1 -- 172.21.15.31:0/1132062706 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 2) ==== 49272+0+0 (secure 0 0 0) 0x7f332c0198f0 con 0x7f334419d8a0 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.547+0000 7f334a29e700 1 -- 172.21.15.31:0/1132062706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3344064cf0 con 0x7f334419d8a0 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.547+0000 7f33417fa700 1 -- 172.21.15.31:0/1132062706 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f332c052500 con 0x7f334419d8a0 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.549+0000 7f33417fa700 1 -- 172.21.15.31:0/1132062706 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f332c0042a0 con 0x7f334419d8a0 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.623+0000 7f334a29e700 1 -- 172.21.15.31:0/1132062706 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f33441a1870 con 0x7f334419d8a0 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.623+0000 7f33417fa700 1 -- 172.21.15.31:0/1132062706 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f332c02bde0 con 0x7f334419d8a0 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.624+0000 7f334a29e700 1 -- 172.21.15.31:0/1132062706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334419d8a0 msgr2=0x7f334419dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.624+0000 7f334a29e700 1 --2- 172.21.15.31:0/1132062706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334419d8a0 0x7f334419dcb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f332c0117b0 tx=0x7f332c011b70 comp rx=0 tx=0).stop 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.624+0000 7f334a29e700 1 -- 172.21.15.31:0/1132062706 shutdown_connections 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.624+0000 7f334a29e700 1 --2- 172.21.15.31:0/1132062706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f334419d8a0 0x7f334419dcb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:09.190 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.624+0000 7f334a29e700 1 -- 172.21.15.31:0/1132062706 >> 172.21.15.31:0/1132062706 conn(0x7f3344103f60 msgr2=0x7f3344105990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:09.191 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.624+0000 7f334a29e700 1 -- 172.21.15.31:0/1132062706 shutdown_connections 2024-04-25T15:51:09.191 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:08.624+0000 7f334a29e700 1 -- 172.21.15.31:0/1132062706 wait complete. 2024-04-25T15:51:09.191 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (3/15)... 2024-04-25T15:51:09.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:09 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1132062706' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T15:51:09.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:09 smithi031 ceph-mon[29140]: mgrmap e3: x(active, since 1.00761s) 2024-04-25T15:51:11.539 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:11 smithi031 ceph-mon[29140]: mgrmap e4: x(active, since 2s) 2024-04-25T15:51:12.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:12 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/979318148' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-25T15:51:12.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:51:12.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T15:51:12.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T15:51:12.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-25T15:51:12.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-25T15:51:12.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-25T15:51:12.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2024-04-25T15:50:52.066479+0000", 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2024-04-25T15:51:12.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.648+0000 7ff882273700 1 Processor -- start 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.648+0000 7ff882273700 1 -- start start 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.648+0000 7ff882273700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c108cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.648+0000 7ff882273700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c1092a0 con 0x7ff87c1088a0 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.649+0000 7ff87bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c108cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.649+0000 7ff87bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c108cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51780/0 (socket says 172.21.15.31:51780) 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.649+0000 7ff87bfff700 1 -- 172.21.15.31:0/3306245409 learned_addr learned my addr 172.21.15.31:0/3306245409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.649+0000 7ff87bfff700 1 -- 172.21.15.31:0/3306245409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87c1099d0 con 0x7ff87c1088a0 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.649+0000 7ff87bfff700 1 --2- 172.21.15.31:0/3306245409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c108cb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff8700098a0 tx=0x7ff870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44474e4939ad48bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.650+0000 7ff87affd700 1 -- 172.21.15.31:0/3306245409 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff870004160 con 0x7ff87c1088a0 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.650+0000 7ff87affd700 1 -- 172.21.15.31:0/3306245409 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff870004300 con 0x7ff87c1088a0 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.650+0000 7ff87affd700 1 -- 172.21.15.31:0/3306245409 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff870011af0 con 0x7ff87c1088a0 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.650+0000 7ff882273700 1 -- 172.21.15.31:0/3306245409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 msgr2=0x7ff87c108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.650+0000 7ff882273700 1 --2- 172.21.15.31:0/3306245409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c108cb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ff8700098a0 tx=0x7ff870009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:12.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.650+0000 7ff882273700 1 -- 172.21.15.31:0/3306245409 shutdown_connections 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.650+0000 7ff882273700 1 --2- 172.21.15.31:0/3306245409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c108cb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.650+0000 7ff882273700 1 -- 172.21.15.31:0/3306245409 >> 172.21.15.31:0/3306245409 conn(0x7ff87c103f80 msgr2=0x7ff87c1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.651+0000 7ff882273700 1 -- 172.21.15.31:0/3306245409 shutdown_connections 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.651+0000 7ff882273700 1 -- 172.21.15.31:0/3306245409 wait complete. 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.651+0000 7ff882273700 1 Processor -- start 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.651+0000 7ff882273700 1 -- start start 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.652+0000 7ff882273700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c07c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.652+0000 7ff882273700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff87c10cd80 con 0x7ff87c1088a0 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.652+0000 7ff87bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c07c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.652+0000 7ff87bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c07c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51796/0 (socket says 172.21.15.31:51796) 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.652+0000 7ff87bfff700 1 -- 172.21.15.31:0/979318148 learned_addr learned my addr 172.21.15.31:0/979318148 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.652+0000 7ff87bfff700 1 -- 172.21.15.31:0/979318148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff87c07ce20 con 0x7ff87c1088a0 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.653+0000 7ff87bfff700 1 --2- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c07c8e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff870009fb0 tx=0x7ff870011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.653+0000 7ff8797fa700 1 -- 172.21.15.31:0/979318148 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8700036a0 con 0x7ff87c1088a0 2024-04-25T15:51:12.637 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.653+0000 7ff8797fa700 1 -- 172.21.15.31:0/979318148 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff870003cd0 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.653+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff87c1a0b10 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.653+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff87c1a0f90 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.653+0000 7ff8797fa700 1 -- 172.21.15.31:0/979318148 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff87001a430 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.654+0000 7ff8797fa700 1 -- 172.21.15.31:0/979318148 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7ff87001a6a0 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.654+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff87c064cf0 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.654+0000 7ff8797fa700 1 --2- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7ff86403d520 0x7ff86403fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.654+0000 7ff8797fa700 1 -- 172.21.15.31:0/979318148 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff870053430 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.654+0000 7ff87b7fe700 1 --2- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7ff86403d520 0x7ff86403fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.655+0000 7ff87b7fe700 1 --2- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7ff86403d520 0x7ff86403fa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff86c006fd0 tx=0x7ff86c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.660+0000 7ff8797fa700 1 -- 172.21.15.31:0/979318148 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff870017e90 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.971+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff87c1a1920 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.971+0000 7ff8797fa700 1 -- 172.21.15.31:0/979318148 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff87002c350 con 0x7ff87c1088a0 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7ff86403d520 msgr2=0x7ff86403fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:12.638 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 --2- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7ff86403d520 0x7ff86403fa30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff86c006fd0 tx=0x7ff86c006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:12.639 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 msgr2=0x7ff87c07c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:12.639 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 --2- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c07c8e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff870009fb0 tx=0x7ff870011890 comp rx=0 tx=0).stop 2024-04-25T15:51:12.639 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 shutdown_connections 2024-04-25T15:51:12.639 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 --2- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7ff86403d520 0x7ff86403fa30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:12.639 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 --2- 172.21.15.31:0/979318148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff87c1088a0 0x7ff87c07c8e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:12.639 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 >> 172.21.15.31:0/979318148 conn(0x7ff87c103f80 msgr2=0x7ff87c105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:12.639 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 shutdown_connections 2024-04-25T15:51:12.639 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:11.974+0000 7ff882273700 1 -- 172.21.15.31:0/979318148 wait complete. 2024-04-25T15:51:12.639 INFO:teuthology.orchestra.run.smithi031.stdout:mgr is available 2024-04-25T15:51:13.589 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:13 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2960665720' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-25T15:51:13.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:51:13.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2024-04-25T15:51:13.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:51:13.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-25T15:51:13.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-25T15:51:13.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-25T15:51:13.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.059+0000 7f49f2562700 1 Processor -- start 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.060+0000 7f49f2562700 1 -- start start 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.060+0000 7f49f2562700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec10c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:13.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.060+0000 7f49f2562700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec10cb30 con 0x7f49ec07d4f0 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.060+0000 7f49ebfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec10c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.060+0000 7f49ebfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec10c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51810/0 (socket says 172.21.15.31:51810) 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.060+0000 7f49ebfff700 1 -- 172.21.15.31:0/2168779950 learned_addr learned my addr 172.21.15.31:0/2168779950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.060+0000 7f49ebfff700 1 -- 172.21.15.31:0/2168779950 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec10ccb0 con 0x7f49ec07d4f0 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.060+0000 7f49ebfff700 1 --2- 172.21.15.31:0/2168779950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec10c520 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0098a0 tx=0x7f49dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e111773cff1739d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.061+0000 7f49eaffd700 1 -- 172.21.15.31:0/2168779950 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49dc004160 con 0x7f49ec07d4f0 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.061+0000 7f49eaffd700 1 -- 172.21.15.31:0/2168779950 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f49dc004300 con 0x7f49ec07d4f0 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.061+0000 7f49eaffd700 1 -- 172.21.15.31:0/2168779950 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49dc011af0 con 0x7f49ec07d4f0 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.061+0000 7f49f2562700 1 -- 172.21.15.31:0/2168779950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 msgr2=0x7f49ec10c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.061+0000 7f49f2562700 1 --2- 172.21.15.31:0/2168779950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec10c520 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f49dc0098a0 tx=0x7f49dc009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.061+0000 7f49f2562700 1 -- 172.21.15.31:0/2168779950 shutdown_connections 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.061+0000 7f49f2562700 1 --2- 172.21.15.31:0/2168779950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec10c520 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.061+0000 7f49f2562700 1 -- 172.21.15.31:0/2168779950 >> 172.21.15.31:0/2168779950 conn(0x7f49ec103f50 msgr2=0x7f49ec1063e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.062+0000 7f49f2562700 1 -- 172.21.15.31:0/2168779950 shutdown_connections 2024-04-25T15:51:13.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.062+0000 7f49f2562700 1 -- 172.21.15.31:0/2168779950 wait complete. 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.062+0000 7f49f2562700 1 Processor -- start 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.062+0000 7f49f2562700 1 -- start start 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49f2562700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec109c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49f2562700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f49ec10dd30 con 0x7f49ec07d4f0 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49ebfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec109c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49ebfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec109c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51824/0 (socket says 172.21.15.31:51824) 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49ebfff700 1 -- 172.21.15.31:0/2960665720 learned_addr learned my addr 172.21.15.31:0/2960665720 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49ebfff700 1 -- 172.21.15.31:0/2960665720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ec10a1d0 con 0x7f49ec07d4f0 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49ebfff700 1 --2- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec109c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009fb0 tx=0x7f49dc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49e97fa700 1 -- 172.21.15.31:0/2960665720 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49dc0036a0 con 0x7f49ec07d4f0 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49e97fa700 1 -- 172.21.15.31:0/2960665720 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f49dc01a430 con 0x7f49ec07d4f0 2024-04-25T15:51:13.894 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ec108420 con 0x7f49ec07d4f0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49e97fa700 1 -- 172.21.15.31:0/2960665720 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49dc019550 con 0x7f49ec07d4f0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.063+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ec108900 con 0x7f49ec07d4f0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.064+0000 7f49e97fa700 1 -- 172.21.15.31:0/2960665720 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f49dc019850 con 0x7f49ec07d4f0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.064+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ec064cf0 con 0x7f49ec07d4f0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.065+0000 7f49e97fa700 1 --2- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f49d403d4d0 0x7f49d403f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.065+0000 7f49e97fa700 1 -- 172.21.15.31:0/2960665720 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f49dc052690 con 0x7f49ec07d4f0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.065+0000 7f49eb7fe700 1 --2- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f49d403d4d0 0x7f49d403f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.065+0000 7f49eb7fe700 1 --2- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f49d403d4d0 0x7f49d403f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f49e0006fd0 tx=0x7f49e0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.070+0000 7f49e97fa700 1 -- 172.21.15.31:0/2960665720 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49dc017e90 con 0x7f49ec07d4f0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.277+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f49ec109040 con 0x7f49ec07d4f0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.278+0000 7f49e97fa700 1 -- 172.21.15.31:0/2960665720 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7f49dc023390 con 0x7f49ec07d4f0 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.280+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f49d403d4d0 msgr2=0x7f49d403f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.280+0000 7f49f2562700 1 --2- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f49d403d4d0 0x7f49d403f9e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f49e0006fd0 tx=0x7f49e0006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.280+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 msgr2=0x7f49ec109c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:13.895 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.280+0000 7f49f2562700 1 --2- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec109c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f49dc009fb0 tx=0x7f49dc011940 comp rx=0 tx=0).stop 2024-04-25T15:51:13.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.281+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 shutdown_connections 2024-04-25T15:51:13.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.281+0000 7f49f2562700 1 --2- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f49d403d4d0 0x7f49d403f9e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:13.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.281+0000 7f49f2562700 1 --2- 172.21.15.31:0/2960665720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f49ec07d4f0 0x7f49ec109c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:13.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.281+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 >> 172.21.15.31:0/2960665720 conn(0x7f49ec103f50 msgr2=0x7f49ec104880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:13.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.281+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 shutdown_connections 2024-04-25T15:51:13.896 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:13.281+0000 7f49f2562700 1 -- 172.21.15.31:0/2960665720 wait complete. 2024-04-25T15:51:13.896 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling cephadm module... 2024-04-25T15:51:14.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:14 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1548292532' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-25T15:51:14.836 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:14 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: ignoring --setuser ceph since I am not root 2024-04-25T15:51:14.836 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:14 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: ignoring --setgroup ceph since I am not root 2024-04-25T15:51:14.836 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:14 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:14.835+0000 7f323a061200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:51:15.138 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:15.030+0000 7f323a061200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:51:15.138 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:15.137+0000 7f323a061200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:51:15.246 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.304+0000 7f7cc7452700 1 Processor -- start 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.304+0000 7f7cc7452700 1 -- start start 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.305+0000 7f7cc7452700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc010c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.305+0000 7f7cc7452700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc010cae0 con 0x7f7cc007cc20 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.305+0000 7f7cc51ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc010c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.305+0000 7f7cc51ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc010c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34312/0 (socket says 172.21.15.31:34312) 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.305+0000 7f7cc51ee700 1 -- 172.21.15.31:0/2977848681 learned_addr learned my addr 172.21.15.31:0/2977848681 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.305+0000 7f7cc51ee700 1 -- 172.21.15.31:0/2977848681 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc010cc60 con 0x7f7cc007cc20 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.305+0000 7f7cc51ee700 1 --2- 172.21.15.31:0/2977848681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc010c510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009e80 tx=0x7f7cb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b1814078737fae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.306+0000 7f7cbffff700 1 -- 172.21.15.31:0/2977848681 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cb8017070 con 0x7f7cc007cc20 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.306+0000 7f7cbffff700 1 -- 172.21.15.31:0/2977848681 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7cb800b8c0 con 0x7f7cc007cc20 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.306+0000 7f7cbffff700 1 -- 172.21.15.31:0/2977848681 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cb8004490 con 0x7f7cc007cc20 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.306+0000 7f7cc7452700 1 -- 172.21.15.31:0/2977848681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 msgr2=0x7f7cc010c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.247 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.306+0000 7f7cc7452700 1 --2- 172.21.15.31:0/2977848681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc010c510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009e80 tx=0x7f7cb800b330 comp rx=0 tx=0).stop 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.306+0000 7f7cc7452700 1 -- 172.21.15.31:0/2977848681 shutdown_connections 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.306+0000 7f7cc7452700 1 --2- 172.21.15.31:0/2977848681 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc010c510 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.306+0000 7f7cc7452700 1 -- 172.21.15.31:0/2977848681 >> 172.21.15.31:0/2977848681 conn(0x7f7cc0103f80 msgr2=0x7f7cc01063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.307+0000 7f7cc7452700 1 -- 172.21.15.31:0/2977848681 shutdown_connections 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.307+0000 7f7cc7452700 1 -- 172.21.15.31:0/2977848681 wait complete. 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.307+0000 7f7cc7452700 1 Processor -- start 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.307+0000 7f7cc7452700 1 -- start start 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cc7452700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc019dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cc7452700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc010dce0 con 0x7f7cc007cc20 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cc51ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc019dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cc51ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc019dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34322/0 (socket says 172.21.15.31:34322) 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cc51ee700 1 -- 172.21.15.31:0/1548292532 learned_addr learned my addr 172.21.15.31:0/1548292532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cc51ee700 1 -- 172.21.15.31:0/1548292532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc019e1c0 con 0x7f7cc007cc20 2024-04-25T15:51:15.248 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cc51ee700 1 --2- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc019dc80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8004220 tx=0x7f7cb800bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cbe7fc700 1 -- 172.21.15.31:0/1548292532 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cb8020020 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cbe7fc700 1 -- 172.21.15.31:0/1548292532 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7cb8012430 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc019e450 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cbe7fc700 1 -- 172.21.15.31:0/1548292532 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cb8017040 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.308+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc01a1060 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.309+0000 7f7cbe7fc700 1 -- 172.21.15.31:0/1548292532 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f7cb8023020 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.309+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc0064cf0 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.310+0000 7f7cbe7fc700 1 --2- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f7ca803d520 0x7f7ca803fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.310+0000 7f7cbe7fc700 1 -- 172.21.15.31:0/1548292532 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7cb800f050 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.310+0000 7f7cc49ed700 1 --2- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f7ca803d520 0x7f7ca803fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.310+0000 7f7cc49ed700 1 --2- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f7ca803d520 0x7f7ca803fa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4006fd0 tx=0x7f7cb4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.315+0000 7f7cbe7fc700 1 -- 172.21.15.31:0/1548292532 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7cb802a020 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.583+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f7cc01a1280 con 0x7f7cc007cc20 2024-04-25T15:51:15.249 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.634+0000 7f7cbe7fc700 1 -- 172.21.15.31:0/1548292532 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f7cc01a1280 con 0x7f7cc007cc20 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.636+0000 7f7cbe7fc700 1 -- 172.21.15.31:0/1548292532 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f7cb802da40 con 0x7f7cc007cc20 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.637+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f7ca803d520 msgr2=0x7f7ca803fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.637+0000 7f7cc7452700 1 --2- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f7ca803d520 0x7f7ca803fa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7cb4006fd0 tx=0x7f7cb4006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.637+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 msgr2=0x7f7cc019dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.637+0000 7f7cc7452700 1 --2- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc019dc80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8004220 tx=0x7f7cb800bdc0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.637+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 shutdown_connections 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.637+0000 7f7cc7452700 1 --2- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f7ca803d520 0x7f7ca803fa30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.637+0000 7f7cc7452700 1 --2- 172.21.15.31:0/1548292532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7cc007cc20 0x7f7cc019dc80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.637+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 >> 172.21.15.31:0/1548292532 conn(0x7f7cc0103f80 msgr2=0x7f7cc0105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.638+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 shutdown_connections 2024-04-25T15:51:15.250 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:14.638+0000 7f7cc7452700 1 -- 172.21.15.31:0/1548292532 wait complete. 2024-04-25T15:51:15.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:15 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1548292532' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-25T15:51:15.902 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:15 smithi031 ceph-mon[29140]: mgrmap e5: x(active, since 6s) 2024-04-25T15:51:15.903 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:15.468+0000 7f323a061200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:51:15.903 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:15.559+0000 7f323a061200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:51:16.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T15:51:16.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-25T15:51:16.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T15:51:16.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T15:51:16.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T15:51:16.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.695+0000 7f747636f700 1 Processor -- start 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.695+0000 7f747636f700 1 -- start start 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.695+0000 7f747636f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f747010c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.695+0000 7f747636f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f747010cb10 con 0x7f747007d690 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.696+0000 7f746ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f747010c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.696+0000 7f746ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f747010c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34352/0 (socket says 172.21.15.31:34352) 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.696+0000 7f746ffff700 1 -- 172.21.15.31:0/3300973380 learned_addr learned my addr 172.21.15.31:0/3300973380 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.696+0000 7f746ffff700 1 -- 172.21.15.31:0/3300973380 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747010cc90 con 0x7f747007d690 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.696+0000 7f746ffff700 1 --2- 172.21.15.31:0/3300973380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f747010c500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f74600098a0 tx=0x7f7460009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3790c76ba5e6a584 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.696+0000 7f746effd700 1 -- 172.21.15.31:0/3300973380 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7460004160 con 0x7f747007d690 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.696+0000 7f746effd700 1 -- 172.21.15.31:0/3300973380 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7460004300 con 0x7f747007d690 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.696+0000 7f746effd700 1 -- 172.21.15.31:0/3300973380 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7460011af0 con 0x7f747007d690 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.697+0000 7f747636f700 1 -- 172.21.15.31:0/3300973380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 msgr2=0x7f747010c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.577 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.697+0000 7f747636f700 1 --2- 172.21.15.31:0/3300973380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f747010c500 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f74600098a0 tx=0x7f7460009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.697+0000 7f747636f700 1 -- 172.21.15.31:0/3300973380 shutdown_connections 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.697+0000 7f747636f700 1 --2- 172.21.15.31:0/3300973380 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f747010c500 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.697+0000 7f747636f700 1 -- 172.21.15.31:0/3300973380 >> 172.21.15.31:0/3300973380 conn(0x7f7470103f10 msgr2=0x7f74701063c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.697+0000 7f747636f700 1 -- 172.21.15.31:0/3300973380 shutdown_connections 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.697+0000 7f747636f700 1 -- 172.21.15.31:0/3300973380 wait complete. 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.698+0000 7f747636f700 1 Processor -- start 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.698+0000 7f747636f700 1 -- start start 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.698+0000 7f747636f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f7470109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.698+0000 7f747636f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f747010dd10 con 0x7f747007d690 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f746ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f7470109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f746ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f7470109c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34358/0 (socket says 172.21.15.31:34358) 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f746ffff700 1 -- 172.21.15.31:0/3953339601 learned_addr learned my addr 172.21.15.31:0/3953339601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f746ffff700 1 -- 172.21.15.31:0/3953339601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f747010a1b0 con 0x7f747007d690 2024-04-25T15:51:16.578 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f746ffff700 1 --2- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f7470109c70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f74600117b0 tx=0x7f7460011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f746d7fa700 1 -- 172.21.15.31:0/3953339601 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7460003af0 con 0x7f747007d690 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f746d7fa700 1 -- 172.21.15.31:0/3953339601 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7460020070 con 0x7f747007d690 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f746d7fa700 1 -- 172.21.15.31:0/3953339601 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7460019630 con 0x7f747007d690 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7470108400 con 0x7f747007d690 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.699+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74701088e0 con 0x7f747007d690 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.700+0000 7f746d7fa700 1 -- 172.21.15.31:0/3953339601 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f74600198a0 con 0x7f747007d690 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.700+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7470064cf0 con 0x7f747007d690 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.701+0000 7f746d7fa700 1 --2- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f745c03d5c0 0x7f745c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.701+0000 7f746d7fa700 1 -- 172.21.15.31:0/3953339601 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7460052870 con 0x7f747007d690 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.701+0000 7f746f7fe700 1 -- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f745c03d5c0 msgr2=0x7f745c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3549555101 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.701+0000 7f746f7fe700 1 --2- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f745c03d5c0 0x7f745c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.706+0000 7f746d7fa700 1 -- 172.21.15.31:0/3953339601 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74600042a0 con 0x7f747007d690 2024-04-25T15:51:16.579 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.902+0000 7f746f7fe700 1 -- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f745c03d5c0 msgr2=0x7f745c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3549555101 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:15.902+0000 7f746f7fe700 1 --2- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f745c03d5c0 0x7f745c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.010+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f7470109020 con 0x7f747007d690 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.012+0000 7f746d7fa700 1 -- 172.21.15.31:0/3953339601 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f746002bde0 con 0x7f747007d690 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.020+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f745c03d5c0 msgr2=0x7f745c03fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.020+0000 7f747636f700 1 --2- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f745c03d5c0 0x7f745c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.020+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 msgr2=0x7f7470109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.020+0000 7f747636f700 1 --2- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f7470109c70 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f74600117b0 tx=0x7f7460011b70 comp rx=0 tx=0).stop 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.025+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 shutdown_connections 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.025+0000 7f747636f700 1 --2- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f745c03d5c0 0x7f745c03fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.025+0000 7f747636f700 1 --2- 172.21.15.31:0/3953339601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747007d690 0x7f7470109c70 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.025+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 >> 172.21.15.31:0/3953339601 conn(0x7f7470103f10 msgr2=0x7f74701048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:16.580 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.027+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 shutdown_connections 2024-04-25T15:51:16.581 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:16.027+0000 7f747636f700 1 -- 172.21.15.31:0/3953339601 wait complete. 2024-04-25T15:51:16.581 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-04-25T15:51:16.581 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 5... 2024-04-25T15:51:16.945 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:16 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3953339601' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T15:51:16.945 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:16.851+0000 7f323a061200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:51:17.199 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:17.035+0000 7f323a061200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:51:17.519 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:17.214+0000 7f323a061200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:51:17.519 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:17.518+0000 7f323a061200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:51:17.826 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:17.615+0000 7f323a061200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:51:17.826 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:17.825+0000 7f323a061200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:51:18.199 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:18 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:18.028+0000 7f323a061200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:51:18.949 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:18 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:18.656+0000 7f323a061200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T15:51:19.415 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:19.102+0000 7f323a061200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:51:19.415 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:19.203+0000 7f323a061200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:51:19.415 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:19.414+0000 7f323a061200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T15:51:20.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:19.958+0000 7f323a061200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:51:20.310 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:20.054+0000 7f323a061200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:51:20.670 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:20.309+0000 7f323a061200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:51:20.670 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:20.449+0000 7f323a061200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:51:20.670 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:20.548+0000 7f323a061200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:51:20.670 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:20.669+0000 7f323a061200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:51:21.449 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:20.980+0000 7f323a061200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:51:21.909 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:21.627+0000 7f323a061200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:51:21.909 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:21.855+0000 7f323a061200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:51:22.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: Active manager daemon x restarted 2024-04-25T15:51:22.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: Activating manager daemon x 2024-04-25T15:51:22.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: osdmap e2: 0 total, 0 up, 0 in 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: mgrmap e6: x(active, starting, since 0.00414896s) 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: Manager daemon x is now available 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:51:22.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:21 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:51:23.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:22 smithi031 ceph-mon[29140]: Found migration_current of "None". Setting to last migration. 2024-04-25T15:51:23.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:22 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T15:51:23.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:22 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T15:51:23.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:22 smithi031 ceph-mon[29140]: mgrmap e7: x(active, since 1.00705s) 2024-04-25T15:51:23.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T15:51:23.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-25T15:51:23.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.058+0000 7f744fbf4700 1 Processor -- start 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.058+0000 7f744fbf4700 1 -- start start 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.058+0000 7f744fbf4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744810e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.058+0000 7f744fbf4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f744810ee80 con 0x7f744807cc20 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.058+0000 7f744d990700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744810e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.058+0000 7f744d990700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744810e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34368/0 (socket says 172.21.15.31:34368) 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.058+0000 7f744d990700 1 -- 172.21.15.31:0/1918150836 learned_addr learned my addr 172.21.15.31:0/1918150836 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.059+0000 7f744d990700 1 -- 172.21.15.31:0/1918150836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744810f000 con 0x7f744807cc20 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.059+0000 7f744d990700 1 --2- 172.21.15.31:0/1918150836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744810e940 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f74440098a0 tx=0x7f7444009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75d126021523f9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.059+0000 7f744c98e700 1 -- 172.21.15.31:0/1918150836 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7444004160 con 0x7f744807cc20 2024-04-25T15:51:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.059+0000 7f744c98e700 1 -- 172.21.15.31:0/1918150836 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7444004300 con 0x7f744807cc20 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.059+0000 7f744c98e700 1 -- 172.21.15.31:0/1918150836 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7444011af0 con 0x7f744807cc20 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.060+0000 7f744fbf4700 1 -- 172.21.15.31:0/1918150836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 msgr2=0x7f744810e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.060+0000 7f744fbf4700 1 --2- 172.21.15.31:0/1918150836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744810e940 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f74440098a0 tx=0x7f7444009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.060+0000 7f744fbf4700 1 -- 172.21.15.31:0/1918150836 shutdown_connections 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.060+0000 7f744fbf4700 1 --2- 172.21.15.31:0/1918150836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744810e940 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.060+0000 7f744fbf4700 1 -- 172.21.15.31:0/1918150836 >> 172.21.15.31:0/1918150836 conn(0x7f7448103fb0 msgr2=0x7f7448106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.060+0000 7f744fbf4700 1 -- 172.21.15.31:0/1918150836 shutdown_connections 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.060+0000 7f744fbf4700 1 -- 172.21.15.31:0/1918150836 wait complete. 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.061+0000 7f744fbf4700 1 Processor -- start 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.061+0000 7f744fbf4700 1 -- start start 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.061+0000 7f744fbf4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744819ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.061+0000 7f744fbf4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f744810ff10 con 0x7f744807cc20 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.061+0000 7f744d990700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744819ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.061+0000 7f744d990700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744819ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34372/0 (socket says 172.21.15.31:34372) 2024-04-25T15:51:23.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.061+0000 7f744d990700 1 -- 172.21.15.31:0/2550429697 learned_addr learned my addr 172.21.15.31:0/2550429697 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.062+0000 7f744d990700 1 -- 172.21.15.31:0/2550429697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74481a0450 con 0x7f744807cc20 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.062+0000 7f744d990700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744819ff10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7444011ac0 tx=0x7f7444011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.062+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7444003970 con 0x7f744807cc20 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.062+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7444003b10 con 0x7f744807cc20 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.062+0000 7f744fbf4700 1 -- 172.21.15.31:0/2550429697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74481a06e0 con 0x7f744807cc20 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.062+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7444024610 con 0x7f744807cc20 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.062+0000 7f744fbf4700 1 -- 172.21.15.31:0/2550429697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74481a32f0 con 0x7f744807cc20 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.063+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f7444024840 con 0x7f744807cc20 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.063+0000 7f743effd700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 0x7f743403fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.063+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 --> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f74340401e0 con 0x7f743403d5c0 2024-04-25T15:51:23.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.063+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f74440526a0 con 0x7f744807cc20 2024-04-25T15:51:23.471 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.063+0000 7f744d18f700 1 -- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 msgr2=0x7f743403fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3549555101 2024-04-25T15:51:23.471 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.063+0000 7f744d18f700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 0x7f743403fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:51:23.471 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.264+0000 7f744d18f700 1 -- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 msgr2=0x7f743403fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3549555101 2024-04-25T15:51:23.471 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.264+0000 7f744d18f700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 0x7f743403fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T15:51:23.471 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.665+0000 7f744d18f700 1 -- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 msgr2=0x7f743403fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3549555101 2024-04-25T15:51:23.471 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:17.665+0000 7f744d18f700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 0x7f743403fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T15:51:23.471 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:18.466+0000 7f744d18f700 1 -- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 msgr2=0x7f743403fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3549555101 2024-04-25T15:51:23.471 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:18.466+0000 7f744d18f700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 0x7f743403fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T15:51:23.472 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:20.067+0000 7f744d18f700 1 -- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 msgr2=0x7f743403fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3549555101 2024-04-25T15:51:23.472 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:20.067+0000 7f744d18f700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 0x7f743403fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T15:51:23.472 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:21.859+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 6) ==== 49283+0+0 (secure 0 0 0) 0x7f7444024b30 con 0x7f744807cc20 2024-04-25T15:51:23.472 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:21.859+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 msgr2=0x7f743403fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:51:23.472 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:21.859+0000 7f743effd700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 0x7f743403fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.472 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.860+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7f7444054490 con 0x7f744807cc20 2024-04-25T15:51:23.473 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.860+0000 7f743effd700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f7434040f00 0x7f7434043320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:23.473 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.861+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f74340401e0 con 0x7f7434040f00 2024-04-25T15:51:23.473 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.861+0000 7f744d18f700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f7434040f00 0x7f7434043320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:23.473 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.862+0000 7f744d18f700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f7434040f00 0x7f7434043320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7438003a10 tx=0x7f74380092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:23.473 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.863+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 <== mgr.14120 v2:172.21.15.31:6800/3023595010 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f74340401e0 con 0x7f7434040f00 2024-04-25T15:51:23.474 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.871+0000 7f744fbf4700 1 -- 172.21.15.31:0/2550429697 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f7448064cf0 con 0x7f7434040f00 2024-04-25T15:51:23.474 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.871+0000 7f743effd700 1 -- 172.21.15.31:0/2550429697 <== mgr.14120 v2:172.21.15.31:6800/3023595010 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f7448064cf0 con 0x7f7434040f00 2024-04-25T15:51:23.474 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 -- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f7434040f00 msgr2=0x7f7434043320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.474 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f7434040f00 0x7f7434043320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7438003a10 tx=0x7f74380092b0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.474 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 -- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 msgr2=0x7f744819ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:23.474 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744819ff10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7444011ac0 tx=0x7f7444011ba0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 -- 172.21.15.31:0/2550429697 shutdown_connections 2024-04-25T15:51:23.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f7434040f00 0x7f7434043320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f744807cc20 0x7f744819ff10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 --2- 172.21.15.31:0/2550429697 >> [v2:172.21.15.31:6800/3549555101,v1:172.21.15.31:6801/3549555101] conn(0x7f743403d5c0 0x7f743403fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:23.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 -- 172.21.15.31:0/2550429697 >> 172.21.15.31:0/2550429697 conn(0x7f7448103fb0 msgr2=0x7f7448105940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:23.475 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 -- 172.21.15.31:0/2550429697 shutdown_connections 2024-04-25T15:51:23.476 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:22.872+0000 7f744fbf4700 1 -- 172.21.15.31:0/2550429697 wait complete. 2024-04-25T15:51:23.476 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 5 is available 2024-04-25T15:51:23.476 INFO:teuthology.orchestra.run.smithi031.stdout:Setting orchestrator backend to cephadm... 2024-04-25T15:51:24.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:24 smithi031 ceph-mon[29140]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T15:51:24.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:24 smithi031 ceph-mon[29140]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T15:51:24.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:24 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:24.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:24 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:24.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:24 smithi031 ceph-mon[29140]: [25/Apr/2024:15:51:23] ENGINE Bus STARTING 2024-04-25T15:51:24.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:24 smithi031 ceph-mon[29140]: [25/Apr/2024:15:51:23] ENGINE Serving on http://172.21.15.31:8765 2024-04-25T15:51:24.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:24 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:51:24.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.939+0000 7f1dbe01e700 1 Processor -- start 2024-04-25T15:51:24.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.939+0000 7f1dbe01e700 1 -- start start 2024-04-25T15:51:24.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.940+0000 7f1dbe01e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db810a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:24.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.940+0000 7f1dbe01e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1db810a800 con 0x7f1db8109e00 2024-04-25T15:51:24.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.940+0000 7f1db77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db810a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:24.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.940+0000 7f1db77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db810a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49894/0 (socket says 172.21.15.31:49894) 2024-04-25T15:51:24.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.940+0000 7f1db77fe700 1 -- 172.21.15.31:0/3871264285 learned_addr learned my addr 172.21.15.31:0/3871264285 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:24.763 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.940+0000 7f1db77fe700 1 -- 172.21.15.31:0/3871264285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db8108b20 con 0x7f1db8109e00 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.940+0000 7f1db77fe700 1 --2- 172.21.15.31:0/3871264285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db810a210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1da80098a0 tx=0x7f1da8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81acc9dae73f7937 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1db67fc700 1 -- 172.21.15.31:0/3871264285 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1da8004160 con 0x7f1db8109e00 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1db67fc700 1 -- 172.21.15.31:0/3871264285 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1da8004300 con 0x7f1db8109e00 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1db67fc700 1 -- 172.21.15.31:0/3871264285 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1da8011af0 con 0x7f1db8109e00 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1dbe01e700 1 -- 172.21.15.31:0/3871264285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 msgr2=0x7f1db810a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1dbe01e700 1 --2- 172.21.15.31:0/3871264285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db810a210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1da80098a0 tx=0x7f1da8009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1dbe01e700 1 -- 172.21.15.31:0/3871264285 shutdown_connections 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1dbe01e700 1 --2- 172.21.15.31:0/3871264285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db810a210 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1dbe01e700 1 -- 172.21.15.31:0/3871264285 >> 172.21.15.31:0/3871264285 conn(0x7f1db8103720 msgr2=0x7f1db8105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1dbe01e700 1 -- 172.21.15.31:0/3871264285 shutdown_connections 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.941+0000 7f1dbe01e700 1 -- 172.21.15.31:0/3871264285 wait complete. 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.942+0000 7f1dbe01e700 1 Processor -- start 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.942+0000 7f1dbe01e700 1 -- start start 2024-04-25T15:51:24.764 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.942+0000 7f1dbe01e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db81083b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.942+0000 7f1dbe01e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1db810cd60 con 0x7f1db8109e00 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1db77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db81083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1db77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db81083b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49898/0 (socket says 172.21.15.31:49898) 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1db77fe700 1 -- 172.21.15.31:0/1184137345 learned_addr learned my addr 172.21.15.31:0/1184137345 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1db77fe700 1 -- 172.21.15.31:0/1184137345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db81088f0 con 0x7f1db8109e00 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1db77fe700 1 --2- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db81083b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1da8011890 tx=0x7f1da8011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1db4ff9700 1 -- 172.21.15.31:0/1184137345 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1da80039e0 con 0x7f1db8109e00 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1db4ff9700 1 -- 172.21.15.31:0/1184137345 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1da8003b80 con 0x7f1db8109e00 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1db81a0b90 con 0x7f1db8109e00 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1db4ff9700 1 -- 172.21.15.31:0/1184137345 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1da8024750 con 0x7f1db8109e00 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.943+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1db819c970 con 0x7f1db8109e00 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.944+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1db8064cf0 con 0x7f1db8109e00 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.944+0000 7f1db4ff9700 1 -- 172.21.15.31:0/1184137345 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7f1da8022020 con 0x7f1db8109e00 2024-04-25T15:51:24.765 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.945+0000 7f1db4ff9700 1 --2- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f1da003d4a0 0x7f1da003f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.945+0000 7f1db4ff9700 1 -- 172.21.15.31:0/1184137345 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f1da80524c0 con 0x7f1db8109e00 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.945+0000 7f1db6ffd700 1 --2- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f1da003d4a0 0x7f1da003f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.945+0000 7f1db6ffd700 1 --2- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f1da003d4a0 0x7f1da003f9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1dac006fd0 tx=0x7f1dac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:23.950+0000 7f1db4ff9700 1 -- 172.21.15.31:0/1184137345 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1da800fdd0 con 0x7f1db8109e00 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.163+0000 7f1db4ff9700 1 -- 172.21.15.31:0/1184137345 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f1da801ab50 con 0x7f1db8109e00 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.177+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f1db8105870 con 0x7f1da003d4a0 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.182+0000 7f1db4ff9700 1 -- 172.21.15.31:0/1184137345 <== mgr.14120 v2:172.21.15.31:6800/3023595010 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1db8105870 con 0x7f1da003d4a0 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.184+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f1da003d4a0 msgr2=0x7f1da003f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.184+0000 7f1dbe01e700 1 --2- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f1da003d4a0 0x7f1da003f9b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1dac006fd0 tx=0x7f1dac006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.184+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 msgr2=0x7f1db81083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.184+0000 7f1dbe01e700 1 --2- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db81083b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1da8011890 tx=0x7f1da8011970 comp rx=0 tx=0).stop 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.185+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 shutdown_connections 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.185+0000 7f1dbe01e700 1 --2- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f1da003d4a0 0x7f1da003f9b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:24.766 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.185+0000 7f1dbe01e700 1 --2- 172.21.15.31:0/1184137345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1db8109e00 0x7f1db81083b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:24.767 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.185+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 >> 172.21.15.31:0/1184137345 conn(0x7f1db8103720 msgr2=0x7f1db81050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:24.767 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.185+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 shutdown_connections 2024-04-25T15:51:24.767 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:24.185+0000 7f1dbe01e700 1 -- 172.21.15.31:0/1184137345 wait complete. 2024-04-25T15:51:25.408 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:25 smithi031 ceph-mon[29140]: [25/Apr/2024:15:51:23] ENGINE Serving on https://172.21.15.31:7150 2024-04-25T15:51:25.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:25 smithi031 ceph-mon[29140]: [25/Apr/2024:15:51:23] ENGINE Bus STARTED 2024-04-25T15:51:25.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:25 smithi031 ceph-mon[29140]: mgrmap e8: x(active, since 2s) 2024-04-25T15:51:25.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:25 smithi031 ceph-mon[29140]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:51:25.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:25 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:25.409 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:25 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.186+0000 7fda01987700 1 Processor -- start 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.186+0000 7fda01987700 1 -- start start 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.186+0000 7fda01987700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc10c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.186+0000 7fda01987700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc07cd70 con 0x7fd9fc10a480 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.187+0000 7fd9faffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc10c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.187+0000 7fd9faffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc10c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49912/0 (socket says 172.21.15.31:49912) 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.187+0000 7fd9faffd700 1 -- 172.21.15.31:0/3503170232 learned_addr learned my addr 172.21.15.31:0/3503170232 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.187+0000 7fd9faffd700 1 -- 172.21.15.31:0/3503170232 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9fc10cde0 con 0x7fd9fc10a480 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.187+0000 7fd9faffd700 1 --2- 172.21.15.31:0/3503170232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc10c8a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009e80 tx=0x7fd9ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43c719e579cf38f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:25.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.187+0000 7fd9f9ffb700 1 -- 172.21.15.31:0/3503170232 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9ec017070 con 0x7fd9fc10a480 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.187+0000 7fd9f9ffb700 1 -- 172.21.15.31:0/3503170232 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9ec00b8c0 con 0x7fd9fc10a480 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.187+0000 7fd9f9ffb700 1 -- 172.21.15.31:0/3503170232 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9ec004490 con 0x7fd9fc10a480 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.188+0000 7fda01987700 1 -- 172.21.15.31:0/3503170232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 msgr2=0x7fd9fc10c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.188+0000 7fda01987700 1 --2- 172.21.15.31:0/3503170232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc10c8a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec009e80 tx=0x7fd9ec00b330 comp rx=0 tx=0).stop 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.188+0000 7fda01987700 1 -- 172.21.15.31:0/3503170232 shutdown_connections 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.188+0000 7fda01987700 1 --2- 172.21.15.31:0/3503170232 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc10c8a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.188+0000 7fda01987700 1 -- 172.21.15.31:0/3503170232 >> 172.21.15.31:0/3503170232 conn(0x7fd9fc103f80 msgr2=0x7fd9fc1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.188+0000 7fda01987700 1 -- 172.21.15.31:0/3503170232 shutdown_connections 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.188+0000 7fda01987700 1 -- 172.21.15.31:0/3503170232 wait complete. 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.189+0000 7fda01987700 1 Processor -- start 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.189+0000 7fda01987700 1 -- start start 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.189+0000 7fda01987700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc109c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.189+0000 7fda01987700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd9fc10dce0 con 0x7fd9fc10a480 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.189+0000 7fd9faffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc109c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:25.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.189+0000 7fd9faffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc109c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49920/0 (socket says 172.21.15.31:49920) 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.189+0000 7fd9faffd700 1 -- 172.21.15.31:0/3313032022 learned_addr learned my addr 172.21.15.31:0/3313032022 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.190+0000 7fd9faffd700 1 -- 172.21.15.31:0/3313032022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9fc10a140 con 0x7fd9fc10a480 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.190+0000 7fd9faffd700 1 --2- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc109c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00b7b0 tx=0x7fd9ec00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.190+0000 7fda00985700 1 -- 172.21.15.31:0/3313032022 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9ec020020 con 0x7fd9fc10a480 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.190+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9fc108390 con 0x7fd9fc10a480 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.190+0000 7fda00985700 1 -- 172.21.15.31:0/3313032022 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fd9ec012430 con 0x7fd9fc10a480 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.190+0000 7fda00985700 1 -- 172.21.15.31:0/3313032022 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9ec017040 con 0x7fd9fc10a480 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.190+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9fc1088d0 con 0x7fd9fc10a480 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.191+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9fc064cf0 con 0x7fd9fc10a480 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.191+0000 7fda00985700 1 -- 172.21.15.31:0/3313032022 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fd9ec023020 con 0x7fd9fc10a480 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.191+0000 7fda00985700 1 --2- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fd9e403d570 0x7fd9e403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.192+0000 7fda00985700 1 -- 172.21.15.31:0/3313032022 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd9ec00f050 con 0x7fd9fc10a480 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.192+0000 7fd9fa7fc700 1 --2- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fd9e403d570 0x7fd9e403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:25.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.192+0000 7fd9fa7fc700 1 --2- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fd9e403d570 0x7fd9e403fa80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0006fd0 tx=0x7fd9f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.197+0000 7fda00985700 1 -- 172.21.15.31:0/3313032022 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9ec02a020 con 0x7fd9fc10a480 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.408+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fd9fc109030 con 0x7fd9e403d570 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.408+0000 7fda00985700 1 -- 172.21.15.31:0/3313032022 <== mgr.14120 v2:172.21.15.31:6800/3023595010 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fd9fc109030 con 0x7fd9e403d570 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fd9e403d570 msgr2=0x7fd9e403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 --2- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fd9e403d570 0x7fd9e403fa80 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd9f0006fd0 tx=0x7fd9f0006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 msgr2=0x7fd9fc109c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 --2- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc109c00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec00b7b0 tx=0x7fd9ec00bd50 comp rx=0 tx=0).stop 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 shutdown_connections 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 --2- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fd9e403d570 0x7fd9e403fa80 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 --2- 172.21.15.31:0/3313032022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9fc10a480 0x7fd9fc109c00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 >> 172.21.15.31:0/3313032022 conn(0x7fd9fc103f80 msgr2=0x7fd9fc1058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 shutdown_connections 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:25.411+0000 7fda01987700 1 -- 172.21.15.31:0/3313032022 wait complete. 2024-04-25T15:51:25.989 INFO:teuthology.orchestra.run.smithi031.stdout:Generating ssh key... 2024-04-25T15:51:26.304 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-mon[29140]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:51:26.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: Generating public/private rsa key pair. 2024-04-25T15:51:26.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: Your identification has been saved in /tmp/tmph1azaknd/key. 2024-04-25T15:51:26.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: Your public key has been saved in /tmp/tmph1azaknd/key.pub. 2024-04-25T15:51:26.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: The key fingerprint is: 2024-04-25T15:51:26.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: SHA256:qjjAMTG+a8ziup6QJd97bOxuTKrXGLYCWve+r7wrLxo ceph-74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:51:26.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: The key's randomart image is: 2024-04-25T15:51:26.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: +---[RSA 3072]----+ 2024-04-25T15:51:26.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: | | 2024-04-25T15:51:26.950 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: | o | 2024-04-25T15:51:26.951 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: |. o | 2024-04-25T15:51:26.951 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: | + | 2024-04-25T15:51:26.951 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: |o = S | 2024-04-25T15:51:26.951 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: |oB..+ .. | 2024-04-25T15:51:26.951 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: |*+Eo.%. | 2024-04-25T15:51:26.951 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: |+=+oB+X | 2024-04-25T15:51:26.951 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: |O=+*o@@*. | 2024-04-25T15:51:26.951 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: +----[SHA256]-----+ 2024-04-25T15:51:27.432 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:27 smithi031 ceph-mon[29140]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:51:27.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:27 smithi031 ceph-mon[29140]: Generating ssh key... 2024-04-25T15:51:27.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:27 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:27.433 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:27 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.423+0000 7fc102dca700 1 Processor -- start 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.423+0000 7fc102dca700 1 -- start start 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.423+0000 7fc102dca700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc108420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.423+0000 7fc102dca700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc108a10 con 0x7fc0fc108010 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.424+0000 7fc100b66700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc108420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.424+0000 7fc100b66700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc108420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49932/0 (socket says 172.21.15.31:49932) 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.424+0000 7fc100b66700 1 -- 172.21.15.31:0/321810541 learned_addr learned my addr 172.21.15.31:0/321810541 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.424+0000 7fc100b66700 1 -- 172.21.15.31:0/321810541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc109140 con 0x7fc0fc108010 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.424+0000 7fc100b66700 1 --2- 172.21.15.31:0/321810541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc108420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0098a0 tx=0x7fc0ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd2c64b1a6978c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.424+0000 7fc0fb7fe700 1 -- 172.21.15.31:0/321810541 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0ec004160 con 0x7fc0fc108010 2024-04-25T15:51:27.441 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.424+0000 7fc0fb7fe700 1 -- 172.21.15.31:0/321810541 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc0ec004300 con 0x7fc0fc108010 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.424+0000 7fc0fb7fe700 1 -- 172.21.15.31:0/321810541 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0ec011af0 con 0x7fc0fc108010 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.425+0000 7fc102dca700 1 -- 172.21.15.31:0/321810541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 msgr2=0x7fc0fc108420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.425+0000 7fc102dca700 1 --2- 172.21.15.31:0/321810541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc108420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec0098a0 tx=0x7fc0ec009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.425+0000 7fc102dca700 1 -- 172.21.15.31:0/321810541 shutdown_connections 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.425+0000 7fc102dca700 1 --2- 172.21.15.31:0/321810541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc108420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.425+0000 7fc102dca700 1 -- 172.21.15.31:0/321810541 >> 172.21.15.31:0/321810541 conn(0x7fc0fc1036f0 msgr2=0x7fc0fc105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.425+0000 7fc102dca700 1 -- 172.21.15.31:0/321810541 shutdown_connections 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.425+0000 7fc102dca700 1 -- 172.21.15.31:0/321810541 wait complete. 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.426+0000 7fc102dca700 1 Processor -- start 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.426+0000 7fc102dca700 1 -- start start 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.426+0000 7fc102dca700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc19dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.426+0000 7fc102dca700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc0fc10c4f0 con 0x7fc0fc108010 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.426+0000 7fc100b66700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc19dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:27.442 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.426+0000 7fc100b66700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc19dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49946/0 (socket says 172.21.15.31:49946) 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.426+0000 7fc100b66700 1 -- 172.21.15.31:0/4261533161 learned_addr learned my addr 172.21.15.31:0/4261533161 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.427+0000 7fc100b66700 1 -- 172.21.15.31:0/4261533161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0fc19e280 con 0x7fc0fc108010 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.427+0000 7fc100b66700 1 --2- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc19dd40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009fb0 tx=0x7fc0ec011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.427+0000 7fc0f9ffb700 1 -- 172.21.15.31:0/4261533161 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0ec0036a0 con 0x7fc0fc108010 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.427+0000 7fc0f9ffb700 1 -- 172.21.15.31:0/4261533161 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc0ec003cd0 con 0x7fc0fc108010 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.427+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0fc19e510 con 0x7fc0fc108010 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.427+0000 7fc0f9ffb700 1 -- 172.21.15.31:0/4261533161 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc0ec01a430 con 0x7fc0fc108010 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.427+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0fc1a1110 con 0x7fc0fc108010 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.428+0000 7fc0f9ffb700 1 -- 172.21.15.31:0/4261533161 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fc0ec01a6a0 con 0x7fc0fc108010 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.428+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0fc064cf0 con 0x7fc0fc108010 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.428+0000 7fc0f9ffb700 1 --2- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc0e403d570 0x7fc0e403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.429+0000 7fc0f9ffb700 1 -- 172.21.15.31:0/4261533161 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc0ec0525c0 con 0x7fc0fc108010 2024-04-25T15:51:27.443 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.429+0000 7fc0fbfff700 1 --2- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc0e403d570 0x7fc0e403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.429+0000 7fc0fbfff700 1 --2- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc0e403d570 0x7fc0e403fa80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0006fd0 tx=0x7fc0f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.434+0000 7fc0f9ffb700 1 -- 172.21.15.31:0/4261533161 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0ec02c350 con 0x7fc0fc108010 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.642+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fc0fc02e880 con 0x7fc0e403d570 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.819+0000 7fc0f9ffb700 1 -- 172.21.15.31:0/4261533161 <== mgr.14120 v2:172.21.15.31:6800/3023595010 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc0fc02e880 con 0x7fc0e403d570 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.821+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc0e403d570 msgr2=0x7fc0e403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.821+0000 7fc102dca700 1 --2- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc0e403d570 0x7fc0e403fa80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0006fd0 tx=0x7fc0f0006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.821+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 msgr2=0x7fc0fc19dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.821+0000 7fc102dca700 1 --2- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc19dd40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc0ec009fb0 tx=0x7fc0ec011890 comp rx=0 tx=0).stop 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.821+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 shutdown_connections 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.821+0000 7fc102dca700 1 --2- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc0e403d570 0x7fc0e403fa80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.821+0000 7fc102dca700 1 --2- 172.21.15.31:0/4261533161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc0fc108010 0x7fc0fc19dd40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:27.444 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.821+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 >> 172.21.15.31:0/4261533161 conn(0x7fc0fc1036f0 msgr2=0x7fc0fc103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:27.445 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.822+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 shutdown_connections 2024-04-25T15:51:27.445 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:26.822+0000 7fc102dca700 1 -- 172.21.15.31:0/4261533161 wait complete. 2024-04-25T15:51:28.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/CcJAaZhzxeOyxFMEYBPZ3NuOufE8PGKqsNPDzs00rnat7/9R7kUeuLtDKQwa5HGhFOJwhSCQtiUp1oec8R3Kblal9i9e7WNAl8VWYMkaRp/iZO/vQ+/K+FjkTn0OOSna3NtCpznkJN2uGegPDnHjj6EU/DxZjAjwgIb2tBaFtSRoHZb39lXHIrf6nrYJGIquUh7FjuKt18WnN2PPR1ugsoNxyo9USAaIHqtFy9NBMH1oEg8sE91UkoJs4PLR+GZrrd8XQKFZ21t9EYV8oXnhTxWl7ax/9uW17jh/1UIOrKUvsgE4baBNbVSqCuvn9p9je5Pa8oUaEi+dTNOaAYq6bKKS1379gNWClmvBIoM5uSRUyCoXx7ztOgUjTKdOqwC9QcSm7UCFkmaNMTDfAdDXGezmQqkLdPxDOFVMNLQLh1qIL1XpvmZttKLlbkyqmOL7AliFO5bI6lkXiq/gruXH0KuukhyTWzxOLnAp6cmHLToRm9z/9sy21Np9ZHU+Xm0= ceph-74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:51:28.723 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.911+0000 7f285854a700 1 Processor -- start 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.911+0000 7f285854a700 1 -- start start 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.912+0000 7f285854a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f2850109e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.912+0000 7f285854a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f285010a3f0 con 0x7f28501099f0 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.912+0000 7f28562e6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f2850109e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.912+0000 7f28562e6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f2850109e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49950/0 (socket says 172.21.15.31:49950) 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.912+0000 7f28562e6700 1 -- 172.21.15.31:0/4020511428 learned_addr learned my addr 172.21.15.31:0/4020511428 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.912+0000 7f28562e6700 1 -- 172.21.15.31:0/4020511428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285010aad0 con 0x7f28501099f0 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.913+0000 7f28562e6700 1 --2- 172.21.15.31:0/4020511428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f2850109e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f28440098a0 tx=0x7f2844009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f43f995d25f4fad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.913+0000 7f28552e4700 1 -- 172.21.15.31:0/4020511428 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2844004160 con 0x7f28501099f0 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.913+0000 7f28552e4700 1 -- 172.21.15.31:0/4020511428 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2844004300 con 0x7f28501099f0 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.913+0000 7f28552e4700 1 -- 172.21.15.31:0/4020511428 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2844011af0 con 0x7f28501099f0 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.914+0000 7f285854a700 1 -- 172.21.15.31:0/4020511428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 msgr2=0x7f2850109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.914+0000 7f285854a700 1 --2- 172.21.15.31:0/4020511428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f2850109e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f28440098a0 tx=0x7f2844009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.914+0000 7f285854a700 1 -- 172.21.15.31:0/4020511428 shutdown_connections 2024-04-25T15:51:28.724 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.914+0000 7f285854a700 1 --2- 172.21.15.31:0/4020511428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f2850109e00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.914+0000 7f285854a700 1 -- 172.21.15.31:0/4020511428 >> 172.21.15.31:0/4020511428 conn(0x7f2850105070 msgr2=0x7f28501074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.914+0000 7f285854a700 1 -- 172.21.15.31:0/4020511428 shutdown_connections 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.914+0000 7f285854a700 1 -- 172.21.15.31:0/4020511428 wait complete. 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.914+0000 7f285854a700 1 Processor -- start 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.914+0000 7f285854a700 1 -- start start 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.915+0000 7f285854a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f285019dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.915+0000 7f285854a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f285010de80 con 0x7f28501099f0 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.915+0000 7f28562e6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f285019dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.915+0000 7f28562e6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f285019dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49954/0 (socket says 172.21.15.31:49954) 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.915+0000 7f28562e6700 1 -- 172.21.15.31:0/3979433194 learned_addr learned my addr 172.21.15.31:0/3979433194 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.915+0000 7f28562e6700 1 -- 172.21.15.31:0/3979433194 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285019e2c0 con 0x7f28501099f0 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.915+0000 7f28562e6700 1 --2- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f285019dd80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2844011890 tx=0x7f2844011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.915+0000 7f28437fe700 1 -- 172.21.15.31:0/3979433194 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28440039e0 con 0x7f28501099f0 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.916+0000 7f28437fe700 1 -- 172.21.15.31:0/3979433194 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f2844003b80 con 0x7f28501099f0 2024-04-25T15:51:28.725 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.916+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f285019e550 con 0x7f28501099f0 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.916+0000 7f28437fe700 1 -- 172.21.15.31:0/3979433194 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f28440246e0 con 0x7f28501099f0 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.916+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28501a1160 con 0x7f28501099f0 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.916+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2850064cf0 con 0x7f28501099f0 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.916+0000 7f28437fe700 1 -- 172.21.15.31:0/3979433194 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f2844022020 con 0x7f28501099f0 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.917+0000 7f28437fe700 1 --2- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f283c03d570 0x7f283c03fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.917+0000 7f28437fe700 1 -- 172.21.15.31:0/3979433194 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f28440525d0 con 0x7f28501099f0 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.917+0000 7f2855ae5700 1 --2- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f283c03d570 0x7f283c03fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.917+0000 7f2855ae5700 1 --2- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f283c03d570 0x7f283c03fa80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f284c006fd0 tx=0x7f284c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:27.922+0000 7f28437fe700 1 -- 172.21.15.31:0/3979433194 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28440172b0 con 0x7f28501099f0 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.131+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f2850106020 con 0x7f283c03d570 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.132+0000 7f28437fe700 1 -- 172.21.15.31:0/3979433194 <== mgr.14120 v2:172.21.15.31:6800/3023595010 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f2850106020 con 0x7f283c03d570 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f283c03d570 msgr2=0x7f283c03fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 --2- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f283c03d570 0x7f283c03fa80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f284c006fd0 tx=0x7f284c006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 msgr2=0x7f285019dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:28.726 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 --2- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f285019dd80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2844011890 tx=0x7f2844011970 comp rx=0 tx=0).stop 2024-04-25T15:51:28.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 shutdown_connections 2024-04-25T15:51:28.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 --2- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f283c03d570 0x7f283c03fa80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 --2- 172.21.15.31:0/3979433194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28501099f0 0x7f285019dd80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:28.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 >> 172.21.15.31:0/3979433194 conn(0x7f2850105070 msgr2=0x7f2850105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:28.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 shutdown_connections 2024-04-25T15:51:28.727 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:28.134+0000 7f285854a700 1 -- 172.21.15.31:0/3979433194 wait complete. 2024-04-25T15:51:28.727 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-25T15:51:28.727 INFO:teuthology.orchestra.run.smithi031.stdout:Adding key to root@localhost authorized_keys... 2024-04-25T15:51:28.727 INFO:teuthology.orchestra.run.smithi031.stdout:Adding host smithi031... 2024-04-25T15:51:28.965 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:28 smithi031 ceph-mon[29140]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:51:29.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:29 smithi031 ceph-mon[29140]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:51:31.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:30 smithi031 ceph-mon[29140]: Deploying cephadm binary to smithi031 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Added host 'smithi031' with addr '172.21.15.31' 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.182+0000 7f95e768e700 1 Processor -- start 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.182+0000 7f95e768e700 1 -- start start 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.183+0000 7f95e768e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e0108cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.183+0000 7f95e768e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95e01092c0 con 0x7f95e01088c0 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.183+0000 7f95e542a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e0108cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.183+0000 7f95e542a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e0108cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49958/0 (socket says 172.21.15.31:49958) 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.183+0000 7f95e542a700 1 -- 172.21.15.31:0/2496378471 learned_addr learned my addr 172.21.15.31:0/2496378471 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.183+0000 7f95e542a700 1 -- 172.21.15.31:0/2496378471 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e01099f0 con 0x7f95e01088c0 2024-04-25T15:51:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.183+0000 7f95e542a700 1 --2- 172.21.15.31:0/2496378471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e0108cd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0098a0 tx=0x7f95dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b037a5d969fd689 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.183+0000 7f95d7fff700 1 -- 172.21.15.31:0/2496378471 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95dc004160 con 0x7f95e01088c0 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.183+0000 7f95d7fff700 1 -- 172.21.15.31:0/2496378471 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f95dc004300 con 0x7f95e01088c0 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.184+0000 7f95d7fff700 1 -- 172.21.15.31:0/2496378471 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95dc011af0 con 0x7f95e01088c0 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.184+0000 7f95e768e700 1 -- 172.21.15.31:0/2496378471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 msgr2=0x7f95e0108cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.184+0000 7f95e768e700 1 --2- 172.21.15.31:0/2496378471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e0108cd0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f95dc0098a0 tx=0x7f95dc009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.184+0000 7f95e768e700 1 -- 172.21.15.31:0/2496378471 shutdown_connections 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.184+0000 7f95e768e700 1 --2- 172.21.15.31:0/2496378471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e0108cd0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.184+0000 7f95e768e700 1 -- 172.21.15.31:0/2496378471 >> 172.21.15.31:0/2496378471 conn(0x7f95e0103f20 msgr2=0x7f95e01063b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.184+0000 7f95e768e700 1 -- 172.21.15.31:0/2496378471 shutdown_connections 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.184+0000 7f95e768e700 1 -- 172.21.15.31:0/2496378471 wait complete. 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.185+0000 7f95e768e700 1 Processor -- start 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.185+0000 7f95e768e700 1 -- start start 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.185+0000 7f95e768e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.185+0000 7f95e768e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95e010cda0 con 0x7f95e01088c0 2024-04-25T15:51:34.255 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95e542a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95e542a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e019dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:49972/0 (socket says 172.21.15.31:49972) 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95e542a700 1 -- 172.21.15.31:0/1264955644 learned_addr learned my addr 172.21.15.31:0/1264955644 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95e542a700 1 -- 172.21.15.31:0/1264955644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95e019e2e0 con 0x7f95e01088c0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95e542a700 1 --2- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e019dda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f95dc011890 tx=0x7f95dc011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95d67fc700 1 -- 172.21.15.31:0/1264955644 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95dc0039e0 con 0x7f95e01088c0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95d67fc700 1 -- 172.21.15.31:0/1264955644 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f95dc003b80 con 0x7f95e01088c0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95e019e570 con 0x7f95e01088c0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95d67fc700 1 -- 172.21.15.31:0/1264955644 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f95dc0246e0 con 0x7f95e01088c0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.186+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95e01a1180 con 0x7f95e01088c0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.187+0000 7f95d67fc700 1 -- 172.21.15.31:0/1264955644 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f95dc022020 con 0x7f95e01088c0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.187+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95e0064cf0 con 0x7f95e01088c0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.187+0000 7f95d67fc700 1 --2- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f95cc03d5c0 0x7f95cc03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.188+0000 7f95d67fc700 1 -- 172.21.15.31:0/1264955644 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f95dc052770 con 0x7f95e01088c0 2024-04-25T15:51:34.256 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.188+0000 7f95e4c29700 1 --2- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f95cc03d5c0 0x7f95cc03fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.188+0000 7f95e4c29700 1 --2- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f95cc03d5c0 0x7f95cc03fad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f95d0006fd0 tx=0x7f95d0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.193+0000 7f95d67fc700 1 -- 172.21.15.31:0/1264955644 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95dc0172b0 con 0x7f95e01088c0 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:29.406+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}) -- 0x7f95e002e7e0 con 0x7f95cc03d5c0 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.619+0000 7f95d67fc700 1 -- 172.21.15.31:0/1264955644 <== mgr.14120 v2:172.21.15.31:6800/3023595010 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f95e002e7e0 con 0x7f95cc03d5c0 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.621+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f95cc03d5c0 msgr2=0x7f95cc03fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.621+0000 7f95e768e700 1 --2- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f95cc03d5c0 0x7f95cc03fad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f95d0006fd0 tx=0x7f95d0006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.621+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 msgr2=0x7f95e019dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.621+0000 7f95e768e700 1 --2- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e019dda0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f95dc011890 tx=0x7f95dc011970 comp rx=0 tx=0).stop 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.621+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 shutdown_connections 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.621+0000 7f95e768e700 1 --2- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f95cc03d5c0 0x7f95cc03fad0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.621+0000 7f95e768e700 1 --2- 172.21.15.31:0/1264955644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95e01088c0 0x7f95e019dda0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.621+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 >> 172.21.15.31:0/1264955644 conn(0x7f95e0103f20 msgr2=0x7f95e01942f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:34.257 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.622+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 shutdown_connections 2024-04-25T15:51:34.258 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:33.622+0000 7f95e768e700 1 -- 172.21.15.31:0/1264955644 wait complete. 2024-04-25T15:51:34.258 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mon service... 2024-04-25T15:51:34.910 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:34 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:34.910 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:34 smithi031 ceph-mon[29140]: Added host smithi031 2024-04-25T15:51:34.910 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:34 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:51:35.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-25T15:51:35.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.674+0000 7f802f935700 1 Processor -- start 2024-04-25T15:51:35.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.674+0000 7f802f935700 1 -- start start 2024-04-25T15:51:35.571 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.675+0000 7f802f935700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281090f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.675+0000 7f802f935700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f80281096e0 con 0x7f8028108ce0 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.675+0000 7f802d6d1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281090f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.675+0000 7f802d6d1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281090f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44024/0 (socket says 172.21.15.31:44024) 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.675+0000 7f802d6d1700 1 -- 172.21.15.31:0/64499297 learned_addr learned my addr 172.21.15.31:0/64499297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.675+0000 7f802d6d1700 1 -- 172.21.15.31:0/64499297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8028107960 con 0x7f8028108ce0 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.675+0000 7f802d6d1700 1 --2- 172.21.15.31:0/64499297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281090f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8020009e80 tx=0x7f802000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265a997bf2bb7c7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.676+0000 7f801ffff700 1 -- 172.21.15.31:0/64499297 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8020017070 con 0x7f8028108ce0 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.676+0000 7f801ffff700 1 -- 172.21.15.31:0/64499297 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f802000b8c0 con 0x7f8028108ce0 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.676+0000 7f801ffff700 1 -- 172.21.15.31:0/64499297 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8020004490 con 0x7f8028108ce0 2024-04-25T15:51:35.572 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.676+0000 7f802f935700 1 -- 172.21.15.31:0/64499297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 msgr2=0x7f80281090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.676+0000 7f802f935700 1 --2- 172.21.15.31:0/64499297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281090f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8020009e80 tx=0x7f802000b330 comp rx=0 tx=0).stop 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.677+0000 7f802f935700 1 -- 172.21.15.31:0/64499297 shutdown_connections 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.677+0000 7f802f935700 1 --2- 172.21.15.31:0/64499297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281090f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.677+0000 7f802f935700 1 -- 172.21.15.31:0/64499297 >> 172.21.15.31:0/64499297 conn(0x7f8028103710 msgr2=0x7f8028105b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.677+0000 7f802f935700 1 -- 172.21.15.31:0/64499297 shutdown_connections 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.677+0000 7f802f935700 1 -- 172.21.15.31:0/64499297 wait complete. 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.677+0000 7f802f935700 1 Processor -- start 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.677+0000 7f802f935700 1 -- start start 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f802f935700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f802f935700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f802810bd40 con 0x7f8028108ce0 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f802d6d1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f802d6d1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281a42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44028/0 (socket says 172.21.15.31:44028) 2024-04-25T15:51:35.573 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f802d6d1700 1 -- 172.21.15.31:0/3747251117 learned_addr learned my addr 172.21.15.31:0/3747251117 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f802d6d1700 1 -- 172.21.15.31:0/3747251117 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80281a4830 con 0x7f8028108ce0 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f802d6d1700 1 --2- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281a42f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f802000bca0 tx=0x7f802000bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f801e7fc700 1 -- 172.21.15.31:0/3747251117 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8020020020 con 0x7f8028108ce0 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80281a4ac0 con 0x7f8028108ce0 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f801e7fc700 1 -- 172.21.15.31:0/3747251117 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8020012430 con 0x7f8028108ce0 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f801e7fc700 1 -- 172.21.15.31:0/3747251117 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8020017040 con 0x7f8028108ce0 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.678+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80281a76d0 con 0x7f8028108ce0 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.679+0000 7f801e7fc700 1 -- 172.21.15.31:0/3747251117 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f8020023020 con 0x7f8028108ce0 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.679+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8028064cf0 con 0x7f8028108ce0 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.680+0000 7f801e7fc700 1 --2- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f801403d570 0x7f801403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:35.574 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.680+0000 7f801e7fc700 1 -- 172.21.15.31:0/3747251117 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f802000f050 con 0x7f8028108ce0 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.680+0000 7f802ced0700 1 --2- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f801403d570 0x7f801403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.680+0000 7f802ced0700 1 --2- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f801403d570 0x7f801403fa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8018006fd0 tx=0x7f8018006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.685+0000 7f801e7fc700 1 -- 172.21.15.31:0/3747251117 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f802002a020 con 0x7f8028108ce0 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.909+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f802802e880 con 0x7f801403d570 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.916+0000 7f801e7fc700 1 -- 172.21.15.31:0/3747251117 <== mgr.14120 v2:172.21.15.31:6800/3023595010 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f802802e880 con 0x7f801403d570 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.918+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f801403d570 msgr2=0x7f801403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.918+0000 7f802f935700 1 --2- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f801403d570 0x7f801403fa80 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8018006fd0 tx=0x7f8018006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.918+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 msgr2=0x7f80281a42f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.918+0000 7f802f935700 1 --2- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281a42f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f802000bca0 tx=0x7f802000bd80 comp rx=0 tx=0).stop 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.919+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 shutdown_connections 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.919+0000 7f802f935700 1 --2- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f801403d570 0x7f801403fa80 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.575 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.919+0000 7f802f935700 1 --2- 172.21.15.31:0/3747251117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8028108ce0 0x7f80281a42f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:35.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.919+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 >> 172.21.15.31:0/3747251117 conn(0x7f8028103710 msgr2=0x7f802810dc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:35.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.919+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 shutdown_connections 2024-04-25T15:51:35.576 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:34.919+0000 7f802f935700 1 -- 172.21.15.31:0/3747251117 wait complete. 2024-04-25T15:51:35.576 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mgr service... 2024-04-25T15:51:36.090 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:35 smithi031 ceph-mon[29140]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:51:36.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:35 smithi031 ceph-mon[29140]: Saving service mon spec with placement count:5 2024-04-25T15:51:36.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:35 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:37.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:37 smithi031 ceph-mon[29140]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:51:37.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:37 smithi031 ceph-mon[29140]: Saving service mgr spec with placement count:2 2024-04-25T15:51:37.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:37 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:37.624 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:37 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:37.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-25T15:51:37.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.090+0000 7fec8743a700 1 Processor -- start 2024-04-25T15:51:37.631 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.090+0000 7fec8743a700 1 -- start start 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.090+0000 7fec8743a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec80108cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.090+0000 7fec8743a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec801092b0 con 0x7fec801088b0 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.090+0000 7fec851d6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec80108cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.090+0000 7fec851d6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec80108cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44042/0 (socket says 172.21.15.31:44042) 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.090+0000 7fec851d6700 1 -- 172.21.15.31:0/1840451875 learned_addr learned my addr 172.21.15.31:0/1840451875 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.091+0000 7fec851d6700 1 -- 172.21.15.31:0/1840451875 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec801099e0 con 0x7fec801088b0 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.091+0000 7fec851d6700 1 --2- 172.21.15.31:0/1840451875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec80108cc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00bb40 tx=0x7fec7c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca052e5f0c3d642c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.091+0000 7fec77fff700 1 -- 172.21.15.31:0/1840451875 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec7c00dbe0 con 0x7fec801088b0 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.091+0000 7fec77fff700 1 -- 172.21.15.31:0/1840451875 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fec7c0044c0 con 0x7fec801088b0 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.091+0000 7fec77fff700 1 -- 172.21.15.31:0/1840451875 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec7c010470 con 0x7fec801088b0 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.092+0000 7fec8743a700 1 -- 172.21.15.31:0/1840451875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 msgr2=0x7fec80108cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.092+0000 7fec8743a700 1 --2- 172.21.15.31:0/1840451875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec80108cc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00bb40 tx=0x7fec7c00be50 comp rx=0 tx=0).stop 2024-04-25T15:51:37.632 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.092+0000 7fec8743a700 1 -- 172.21.15.31:0/1840451875 shutdown_connections 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.092+0000 7fec8743a700 1 --2- 172.21.15.31:0/1840451875 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec80108cc0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.092+0000 7fec8743a700 1 -- 172.21.15.31:0/1840451875 >> 172.21.15.31:0/1840451875 conn(0x7fec80103f30 msgr2=0x7fec801063a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.092+0000 7fec8743a700 1 -- 172.21.15.31:0/1840451875 shutdown_connections 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.092+0000 7fec8743a700 1 -- 172.21.15.31:0/1840451875 wait complete. 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.093+0000 7fec8743a700 1 Processor -- start 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.093+0000 7fec8743a700 1 -- start start 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.093+0000 7fec8743a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec8019fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.093+0000 7fec8743a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec8007c970 con 0x7fec801088b0 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.093+0000 7fec851d6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec8019fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.093+0000 7fec851d6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec8019fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44052/0 (socket says 172.21.15.31:44052) 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.093+0000 7fec851d6700 1 -- 172.21.15.31:0/4226811575 learned_addr learned my addr 172.21.15.31:0/4226811575 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.094+0000 7fec851d6700 1 -- 172.21.15.31:0/4226811575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec801a03a0 con 0x7fec801088b0 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.094+0000 7fec851d6700 1 --2- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec8019fe60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fec7c003b90 tx=0x7fec7c003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.094+0000 7fec767fc700 1 -- 172.21.15.31:0/4226811575 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec7c010770 con 0x7fec801088b0 2024-04-25T15:51:37.633 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.094+0000 7fec767fc700 1 -- 172.21.15.31:0/4226811575 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fec7c01d070 con 0x7fec801088b0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.094+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec801a0630 con 0x7fec801088b0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.094+0000 7fec767fc700 1 -- 172.21.15.31:0/4226811575 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fec7c019b70 con 0x7fec801088b0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.094+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec801a3240 con 0x7fec801088b0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.095+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec80064cf0 con 0x7fec801088b0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.095+0000 7fec767fc700 1 -- 172.21.15.31:0/4226811575 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fec7c028060 con 0x7fec801088b0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.095+0000 7fec767fc700 1 --2- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fec6c03d570 0x7fec6c03fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.095+0000 7fec767fc700 1 -- 172.21.15.31:0/4226811575 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fec7c0527b0 con 0x7fec801088b0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.095+0000 7fec849d5700 1 --2- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fec6c03d570 0x7fec6c03fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.096+0000 7fec849d5700 1 --2- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fec6c03d570 0x7fec6c03fa80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fec70006fd0 tx=0x7fec70006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.100+0000 7fec767fc700 1 -- 172.21.15.31:0/4226811575 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec7c015070 con 0x7fec801088b0 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.325+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fec80063ad0 con 0x7fec6c03d570 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.333+0000 7fec767fc700 1 -- 172.21.15.31:0/4226811575 <== mgr.14120 v2:172.21.15.31:6800/3023595010 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fec80063ad0 con 0x7fec6c03d570 2024-04-25T15:51:37.634 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fec6c03d570 msgr2=0x7fec6c03fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:37.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 --2- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fec6c03d570 0x7fec6c03fa80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fec70006fd0 tx=0x7fec70006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:37.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 msgr2=0x7fec8019fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:37.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 --2- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec8019fe60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fec7c003b90 tx=0x7fec7c003dc0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 shutdown_connections 2024-04-25T15:51:37.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 --2- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fec6c03d570 0x7fec6c03fa80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 --2- 172.21.15.31:0/4226811575 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec801088b0 0x7fec8019fe60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:37.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 >> 172.21.15.31:0/4226811575 conn(0x7fec80103f30 msgr2=0x7fec80191d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:37.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 shutdown_connections 2024-04-25T15:51:37.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:36.335+0000 7fec8743a700 1 -- 172.21.15.31:0/4226811575 wait complete. 2024-04-25T15:51:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.076+0000 7f51ed65d700 1 Processor -- start 2024-04-25T15:51:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.077+0000 7f51ed65d700 1 -- start start 2024-04-25T15:51:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.077+0000 7f51ed65d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e8109e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.077+0000 7f51ed65d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f51e810a450 con 0x7f51e8109a50 2024-04-25T15:51:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.077+0000 7f51e6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e8109e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.077+0000 7f51e6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e8109e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44068/0 (socket says 172.21.15.31:44068) 2024-04-25T15:51:38.871 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.077+0000 7f51e6ffd700 1 -- 172.21.15.31:0/1164119660 learned_addr learned my addr 172.21.15.31:0/1164119660 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.077+0000 7f51e6ffd700 1 -- 172.21.15.31:0/1164119660 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e810ab30 con 0x7f51e8109a50 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.078+0000 7f51e6ffd700 1 --2- 172.21.15.31:0/1164119660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e8109e60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f51d80098a0 tx=0x7f51d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f22c0ce88b5a7f40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.078+0000 7f51e5ffb700 1 -- 172.21.15.31:0/1164119660 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51d8004160 con 0x7f51e8109a50 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.078+0000 7f51e5ffb700 1 -- 172.21.15.31:0/1164119660 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d8004300 con 0x7f51e8109a50 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.078+0000 7f51e5ffb700 1 -- 172.21.15.31:0/1164119660 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51d8011a20 con 0x7f51e8109a50 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.079+0000 7f51ed65d700 1 -- 172.21.15.31:0/1164119660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 msgr2=0x7f51e8109e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.079+0000 7f51ed65d700 1 --2- 172.21.15.31:0/1164119660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e8109e60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f51d80098a0 tx=0x7f51d8009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.079+0000 7f51ed65d700 1 -- 172.21.15.31:0/1164119660 shutdown_connections 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.079+0000 7f51ed65d700 1 --2- 172.21.15.31:0/1164119660 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e8109e60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.079+0000 7f51ed65d700 1 -- 172.21.15.31:0/1164119660 >> 172.21.15.31:0/1164119660 conn(0x7f51e81050b0 msgr2=0x7f51e8107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.079+0000 7f51ed65d700 1 -- 172.21.15.31:0/1164119660 shutdown_connections 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.079+0000 7f51ed65d700 1 -- 172.21.15.31:0/1164119660 wait complete. 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.080+0000 7f51ed65d700 1 Processor -- start 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.080+0000 7f51ed65d700 1 -- start start 2024-04-25T15:51:38.872 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.080+0000 7f51ed65d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e807ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.080+0000 7f51ed65d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f51e810dee0 con 0x7f51e8109a50 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.080+0000 7f51e6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e807ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.080+0000 7f51e6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e807ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44072/0 (socket says 172.21.15.31:44072) 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.080+0000 7f51e6ffd700 1 -- 172.21.15.31:0/2698660872 learned_addr learned my addr 172.21.15.31:0/2698660872 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.081+0000 7f51e6ffd700 1 -- 172.21.15.31:0/2698660872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51e807ef40 con 0x7f51e8109a50 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.081+0000 7f51e6ffd700 1 --2- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e807ea00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51d8003e70 tx=0x7f51d8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.081+0000 7f51cffff700 1 -- 172.21.15.31:0/2698660872 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51d8003af0 con 0x7f51e8109a50 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.081+0000 7f51ed65d700 1 -- 172.21.15.31:0/2698660872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51e807d190 con 0x7f51e8109a50 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.081+0000 7f51cffff700 1 -- 172.21.15.31:0/2698660872 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51d801a8e0 con 0x7f51e8109a50 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.081+0000 7f51cffff700 1 -- 172.21.15.31:0/2698660872 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51d8024870 con 0x7f51e8109a50 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.081+0000 7f51ed65d700 1 -- 172.21.15.31:0/2698660872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51e807d670 con 0x7f51e8109a50 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.082+0000 7f51cdffb700 1 -- 172.21.15.31:0/2698660872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51c8005330 con 0x7f51e8109a50 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.083+0000 7f51cffff700 1 -- 172.21.15.31:0/2698660872 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f51d8022020 con 0x7f51e8109a50 2024-04-25T15:51:38.873 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.083+0000 7f51cffff700 1 --2- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f51d003d520 0x7f51d003fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.083+0000 7f51cffff700 1 -- 172.21.15.31:0/2698660872 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f51d8052730 con 0x7f51e8109a50 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.083+0000 7f51e67fc700 1 --2- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f51d003d520 0x7f51d003fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.084+0000 7f51e67fc700 1 --2- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f51d003d520 0x7f51d003fa30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f51dc006fd0 tx=0x7f51dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.088+0000 7f51cffff700 1 -- 172.21.15.31:0/2698660872 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51d801ad90 con 0x7f51e8109a50 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.316+0000 7f51cdffb700 1 -- 172.21.15.31:0/2698660872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f51c8005620 con 0x7f51e8109a50 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.319+0000 7f51cffff700 1 -- 172.21.15.31:0/2698660872 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f51d8018cd0 con 0x7f51e8109a50 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 -- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f51d003d520 msgr2=0x7f51d003fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 --2- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f51d003d520 0x7f51d003fa30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f51dc006fd0 tx=0x7f51dc006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 -- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 msgr2=0x7f51e807ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 --2- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e807ea00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f51d8003e70 tx=0x7f51d8011aa0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 -- 172.21.15.31:0/2698660872 shutdown_connections 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 --2- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7f51d003d520 0x7f51d003fa30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 --2- 172.21.15.31:0/2698660872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51e8109a50 0x7f51e807ea00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:38.874 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 -- 172.21.15.31:0/2698660872 >> 172.21.15.31:0/2698660872 conn(0x7f51e81050b0 msgr2=0x7f51e81058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:38.875 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 -- 172.21.15.31:0/2698660872 shutdown_connections 2024-04-25T15:51:38.875 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:38.322+0000 7f51cdffb700 1 -- 172.21.15.31:0/2698660872 wait complete. 2024-04-25T15:51:39.100 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:38 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:39.100 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:38 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2698660872' entity='client.admin' 2024-04-25T15:51:40.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:40 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:40.881 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:40 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1754786082' entity='client.admin' 2024-04-25T15:51:40.889 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.027+0000 7fe3913fd700 1 Processor -- start 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.027+0000 7fe3913fd700 1 -- start start 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.027+0000 7fe3913fd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c108cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.027+0000 7fe3913fd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c1092a0 con 0x7fe38c1088a0 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.028+0000 7fe38affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c108cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.028+0000 7fe38affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c108cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44080/0 (socket says 172.21.15.31:44080) 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.028+0000 7fe38affd700 1 -- 172.21.15.31:0/841509636 learned_addr learned my addr 172.21.15.31:0/841509636 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.028+0000 7fe38affd700 1 -- 172.21.15.31:0/841509636 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38c1099d0 con 0x7fe38c1088a0 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.028+0000 7fe38affd700 1 --2- 172.21.15.31:0/841509636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c108cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00bb40 tx=0x7fe37c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df44345ef08735f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.028+0000 7fe389ffb700 1 -- 172.21.15.31:0/841509636 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe37c00dbe0 con 0x7fe38c1088a0 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.028+0000 7fe389ffb700 1 -- 172.21.15.31:0/841509636 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe37c0044e0 con 0x7fe38c1088a0 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.028+0000 7fe389ffb700 1 -- 172.21.15.31:0/841509636 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe37c010470 con 0x7fe38c1088a0 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.029+0000 7fe3913fd700 1 -- 172.21.15.31:0/841509636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 msgr2=0x7fe38c108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.890 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.029+0000 7fe3913fd700 1 --2- 172.21.15.31:0/841509636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c108cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00bb40 tx=0x7fe37c00be50 comp rx=0 tx=0).stop 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.029+0000 7fe3913fd700 1 -- 172.21.15.31:0/841509636 shutdown_connections 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.029+0000 7fe3913fd700 1 --2- 172.21.15.31:0/841509636 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c108cb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.029+0000 7fe3913fd700 1 -- 172.21.15.31:0/841509636 >> 172.21.15.31:0/841509636 conn(0x7fe38c103f80 msgr2=0x7fe38c1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.030+0000 7fe3913fd700 1 -- 172.21.15.31:0/841509636 shutdown_connections 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.030+0000 7fe3913fd700 1 -- 172.21.15.31:0/841509636 wait complete. 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.030+0000 7fe3913fd700 1 Processor -- start 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.030+0000 7fe3913fd700 1 -- start start 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.031+0000 7fe3913fd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.031+0000 7fe3913fd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe38c07c0e0 con 0x7fe38c1088a0 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.031+0000 7fe38affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.031+0000 7fe38affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44084/0 (socket says 172.21.15.31:44084) 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.031+0000 7fe38affd700 1 -- 172.21.15.31:0/1754786082 learned_addr learned my addr 172.21.15.31:0/1754786082 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.031+0000 7fe38affd700 1 -- 172.21.15.31:0/1754786082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe38c19c000 con 0x7fe38c1088a0 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.031+0000 7fe38affd700 1 --2- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c19bac0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00bb10 tx=0x7fe37c003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.891 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.032+0000 7fe373fff700 1 -- 172.21.15.31:0/1754786082 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe37c027070 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.032+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe38c19c290 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.032+0000 7fe373fff700 1 -- 172.21.15.31:0/1754786082 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe37c010c20 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.032+0000 7fe373fff700 1 -- 172.21.15.31:0/1754786082 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe37c022b70 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.032+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe38c19eea0 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.033+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe38c064cf0 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.033+0000 7fe373fff700 1 -- 172.21.15.31:0/1754786082 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fe37c022d10 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.033+0000 7fe373fff700 1 --2- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe37403d570 0x7fe37403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.033+0000 7fe373fff700 1 -- 172.21.15.31:0/1754786082 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe37c01f070 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.034+0000 7fe38a7fc700 1 --2- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe37403d570 0x7fe37403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.034+0000 7fe38a7fc700 1 --2- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe37403d570 0x7fe37403fa80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe380006fd0 tx=0x7fe380006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.039+0000 7fe373fff700 1 -- 172.21.15.31:0/1754786082 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe37c015070 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.269+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fe38c194cd0 con 0x7fe38c1088a0 2024-04-25T15:51:40.892 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.272+0000 7fe373fff700 1 -- 172.21.15.31:0/1754786082 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fe38c194cd0 con 0x7fe38c1088a0 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.275+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe37403d570 msgr2=0x7fe37403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.275+0000 7fe3913fd700 1 --2- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe37403d570 0x7fe37403fa80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fe380006fd0 tx=0x7fe380006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.275+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 msgr2=0x7fe38c19bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.275+0000 7fe3913fd700 1 --2- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c19bac0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe37c00bb10 tx=0x7fe37c003dc0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.275+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 shutdown_connections 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.275+0000 7fe3913fd700 1 --2- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe37403d570 0x7fe37403fa80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.275+0000 7fe3913fd700 1 --2- 172.21.15.31:0/1754786082 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe38c1088a0 0x7fe38c19bac0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.275+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 >> 172.21.15.31:0/1754786082 conn(0x7fe38c103f80 msgr2=0x7fe38c191d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.276+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 shutdown_connections 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:40.276+0000 7fe3913fd700 1 -- 172.21.15.31:0/1754786082 wait complete. 2024-04-25T15:51:40.893 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling the dashboard module... 2024-04-25T15:51:42.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:41 smithi031 ceph-mon[29140]: from='mgr.14120 172.21.15.31:0/943537648' entity='mgr.x' 2024-04-25T15:51:42.181 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:41 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2771297397' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-25T15:51:42.181 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: ignoring --setuser ceph since I am not root 2024-04-25T15:51:42.181 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: ignoring --setgroup ceph since I am not root 2024-04-25T15:51:42.181 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:42.180+0000 7fdb78fcb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:51:42.449 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:42.375+0000 7fdb78fcb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:51:42.677 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.336+0000 7fc825953700 1 Processor -- start 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.336+0000 7fc825953700 1 -- start start 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.336+0000 7fc825953700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc820109dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.336+0000 7fc825953700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc82010a3b0 con 0x7fc8201099b0 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.337+0000 7fc81effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc820109dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.337+0000 7fc81effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc820109dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44090/0 (socket says 172.21.15.31:44090) 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.337+0000 7fc81effd700 1 -- 172.21.15.31:0/2477297664 learned_addr learned my addr 172.21.15.31:0/2477297664 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.337+0000 7fc81effd700 1 -- 172.21.15.31:0/2477297664 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82010aae0 con 0x7fc8201099b0 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.337+0000 7fc81effd700 1 --2- 172.21.15.31:0/2477297664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc820109dc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc8100098a0 tx=0x7fc810009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eed5957e5543cd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.337+0000 7fc81dffb700 1 -- 172.21.15.31:0/2477297664 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc810004160 con 0x7fc8201099b0 2024-04-25T15:51:42.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.338+0000 7fc81dffb700 1 -- 172.21.15.31:0/2477297664 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc810004300 con 0x7fc8201099b0 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.338+0000 7fc81dffb700 1 -- 172.21.15.31:0/2477297664 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc810011a20 con 0x7fc8201099b0 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.338+0000 7fc825953700 1 -- 172.21.15.31:0/2477297664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 msgr2=0x7fc820109dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.338+0000 7fc825953700 1 --2- 172.21.15.31:0/2477297664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc820109dc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc8100098a0 tx=0x7fc810009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.338+0000 7fc825953700 1 -- 172.21.15.31:0/2477297664 shutdown_connections 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.338+0000 7fc825953700 1 --2- 172.21.15.31:0/2477297664 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc820109dc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.338+0000 7fc825953700 1 -- 172.21.15.31:0/2477297664 >> 172.21.15.31:0/2477297664 conn(0x7fc820105030 msgr2=0x7fc8201074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.339+0000 7fc825953700 1 -- 172.21.15.31:0/2477297664 shutdown_connections 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.339+0000 7fc825953700 1 -- 172.21.15.31:0/2477297664 wait complete. 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.339+0000 7fc825953700 1 Processor -- start 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.340+0000 7fc825953700 1 -- start start 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.340+0000 7fc825953700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc82019dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.340+0000 7fc825953700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc82010de90 con 0x7fc8201099b0 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.340+0000 7fc81effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc82019dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.340+0000 7fc81effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc82019dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44096/0 (socket says 172.21.15.31:44096) 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.340+0000 7fc81effd700 1 -- 172.21.15.31:0/2771297397 learned_addr learned my addr 172.21.15.31:0/2771297397 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.340+0000 7fc81effd700 1 -- 172.21.15.31:0/2771297397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc82019e2b0 con 0x7fc8201099b0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.340+0000 7fc81effd700 1 --2- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc82019dd70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc810011a20 tx=0x7fc810011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.341+0000 7fc824951700 1 -- 172.21.15.31:0/2771297397 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc8100039c0 con 0x7fc8201099b0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.341+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc82019e540 con 0x7fc8201099b0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.341+0000 7fc824951700 1 -- 172.21.15.31:0/2771297397 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc810003b60 con 0x7fc8201099b0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.341+0000 7fc824951700 1 -- 172.21.15.31:0/2771297397 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc810019700 con 0x7fc8201099b0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.342+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8201a1150 con 0x7fc8201099b0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.343+0000 7fc824951700 1 -- 172.21.15.31:0/2771297397 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fc81001ab00 con 0x7fc8201099b0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.343+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc820064cf0 con 0x7fc8201099b0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.343+0000 7fc824951700 1 --2- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc80803d520 0x7fc80803fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.343+0000 7fc824951700 1 -- 172.21.15.31:0/2771297397 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc8100527b0 con 0x7fc8201099b0 2024-04-25T15:51:42.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.344+0000 7fc81e7fc700 1 --2- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc80803d520 0x7fc80803fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.344+0000 7fc81e7fc700 1 --2- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc80803d520 0x7fc80803fa30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc814006fd0 tx=0x7fc814006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.349+0000 7fc824951700 1 -- 172.21.15.31:0/2771297397 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc810052510 con 0x7fc8201099b0 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.640+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fc8201a1320 con 0x7fc8201099b0 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.966+0000 7fc824951700 1 -- 172.21.15.31:0/2771297397 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fc81001a860 con 0x7fc8201099b0 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.968+0000 7fc824951700 1 -- 172.21.15.31:0/2771297397 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7fc81002d430 con 0x7fc8201099b0 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.969+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc80803d520 msgr2=0x7fc80803fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.969+0000 7fc825953700 1 --2- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc80803d520 0x7fc80803fa30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc814006fd0 tx=0x7fc814006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.970+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 msgr2=0x7fc82019dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.970+0000 7fc825953700 1 --2- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc82019dd70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc810011a20 tx=0x7fc810011b00 comp rx=0 tx=0).stop 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.970+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 shutdown_connections 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.970+0000 7fc825953700 1 --2- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fc80803d520 0x7fc80803fa30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:42.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.970+0000 7fc825953700 1 --2- 172.21.15.31:0/2771297397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8201099b0 0x7fc82019dd70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:42.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.970+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 >> 172.21.15.31:0/2771297397 conn(0x7fc820105030 msgr2=0x7fc820105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:42.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.970+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 shutdown_connections 2024-04-25T15:51:42.687 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:41.970+0000 7fc825953700 1 -- 172.21.15.31:0/2771297397 wait complete. 2024-04-25T15:51:42.803 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:42.484+0000 7fdb78fcb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:51:43.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:42 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2771297397' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-25T15:51:43.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:42 smithi031 ceph-mon[29140]: mgrmap e9: x(active, since 20s) 2024-04-25T15:51:43.200 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:42.812+0000 7fdb78fcb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:51:43.200 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:42.903+0000 7fdb78fcb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:51:44.180 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:43 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1677530508' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-25T15:51:44.449 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:44 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:44.179+0000 7fdb78fcb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:51:44.449 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:44 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:44.361+0000 7fdb78fcb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:51:44.465 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T15:51:44.465 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-25T15:51:44.465 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-25T15:51:44.465 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-25T15:51:44.465 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.593+0000 7fa72227a700 1 Processor -- start 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.593+0000 7fa72227a700 1 -- start start 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.593+0000 7fa72227a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c10c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.593+0000 7fa72227a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c10cae0 con 0x7fa71c07cc60 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.593+0000 7fa71bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c10c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.593+0000 7fa71bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c10c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44128/0 (socket says 172.21.15.31:44128) 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.593+0000 7fa71bfff700 1 -- 172.21.15.31:0/257883676 learned_addr learned my addr 172.21.15.31:0/257883676 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.594+0000 7fa71bfff700 1 -- 172.21.15.31:0/257883676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71c10cc60 con 0x7fa71c07cc60 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.594+0000 7fa71bfff700 1 --2- 172.21.15.31:0/257883676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c10c4d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0098a0 tx=0x7fa70c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861302bffbe96092 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.594+0000 7fa71affd700 1 -- 172.21.15.31:0/257883676 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa70c004160 con 0x7fa71c07cc60 2024-04-25T15:51:44.466 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.594+0000 7fa71affd700 1 -- 172.21.15.31:0/257883676 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa70c004300 con 0x7fa71c07cc60 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.594+0000 7fa71affd700 1 -- 172.21.15.31:0/257883676 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa70c011a20 con 0x7fa71c07cc60 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.595+0000 7fa72227a700 1 -- 172.21.15.31:0/257883676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 msgr2=0x7fa71c10c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.595+0000 7fa72227a700 1 --2- 172.21.15.31:0/257883676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c10c4d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0098a0 tx=0x7fa70c009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.595+0000 7fa72227a700 1 -- 172.21.15.31:0/257883676 shutdown_connections 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.595+0000 7fa72227a700 1 --2- 172.21.15.31:0/257883676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c10c4d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.595+0000 7fa72227a700 1 -- 172.21.15.31:0/257883676 >> 172.21.15.31:0/257883676 conn(0x7fa71c103f80 msgr2=0x7fa71c1063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.595+0000 7fa72227a700 1 -- 172.21.15.31:0/257883676 shutdown_connections 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.595+0000 7fa72227a700 1 -- 172.21.15.31:0/257883676 wait complete. 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.595+0000 7fa72227a700 1 Processor -- start 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.596+0000 7fa72227a700 1 -- start start 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.596+0000 7fa72227a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c109c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.596+0000 7fa72227a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa71c10dd20 con 0x7fa71c07cc60 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.596+0000 7fa71bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c109c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:44.467 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.596+0000 7fa71bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c109c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44132/0 (socket says 172.21.15.31:44132) 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.596+0000 7fa71bfff700 1 -- 172.21.15.31:0/1677530508 learned_addr learned my addr 172.21.15.31:0/1677530508 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.596+0000 7fa71bfff700 1 -- 172.21.15.31:0/1677530508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa71c10a180 con 0x7fa71c07cc60 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.597+0000 7fa71bfff700 1 --2- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c109c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0119f0 tx=0x7fa70c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.597+0000 7fa7197fa700 1 -- 172.21.15.31:0/1677530508 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa70c0039e0 con 0x7fa71c07cc60 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.597+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa71c1083d0 con 0x7fa71c07cc60 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.597+0000 7fa7197fa700 1 -- 172.21.15.31:0/1677530508 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa70c01b8f0 con 0x7fa71c07cc60 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.597+0000 7fa7197fa700 1 -- 172.21.15.31:0/1677530508 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa70c0237a0 con 0x7fa71c07cc60 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.597+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa71c1088b0 con 0x7fa71c07cc60 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.598+0000 7fa7197fa700 1 -- 172.21.15.31:0/1677530508 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7fa70c023aa0 con 0x7fa71c07cc60 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.598+0000 7fa7197fa700 1 --2- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fa70403d570 0x7fa70403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:44.468 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.598+0000 7fa7197fa700 1 -- 172.21.15.31:0/1677530508 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa70c052c80 con 0x7fa71c07cc60 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.598+0000 7fa71b7fe700 1 -- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fa70403d570 msgr2=0x7fa70403fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3023595010 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.598+0000 7fa71b7fe700 1 --2- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fa70403d570 0x7fa70403fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.598+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa71c064cf0 con 0x7fa71c07cc60 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.603+0000 7fa7197fa700 1 -- 172.21.15.31:0/1677530508 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa70c01f030 con 0x7fa71c07cc60 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.799+0000 7fa71b7fe700 1 -- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fa70403d570 msgr2=0x7fa70403fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3023595010 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.799+0000 7fa71b7fe700 1 --2- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fa70403d570 0x7fa70403fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.870+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa71c108fc0 con 0x7fa71c07cc60 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.870+0000 7fa7197fa700 1 -- 172.21.15.31:0/1677530508 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7fa70c017e90 con 0x7fa71c07cc60 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fa70403d570 msgr2=0x7fa70403fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 --2- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fa70403d570 0x7fa70403fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 msgr2=0x7fa71c109c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:44.469 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 --2- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c109c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa70c0119f0 tx=0x7fa70c011ad0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 shutdown_connections 2024-04-25T15:51:44.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 --2- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fa70403d570 0x7fa70403fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 --2- 172.21.15.31:0/1677530508 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa71c07cc60 0x7fa71c109c40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:44.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 >> 172.21.15.31:0/1677530508 conn(0x7fa71c103f80 msgr2=0x7fa71c105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:44.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 shutdown_connections 2024-04-25T15:51:44.470 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:43.873+0000 7fa72227a700 1 -- 172.21.15.31:0/1677530508 wait complete. 2024-04-25T15:51:44.470 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2024-04-25T15:51:44.470 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 9... 2024-04-25T15:51:44.760 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:44 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:44.543+0000 7fdb78fcb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:51:45.079 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:44 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:44.857+0000 7fdb78fcb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:51:45.080 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:44 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:44.957+0000 7fdb78fcb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:51:45.373 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:45 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:45.169+0000 7fdb78fcb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:51:45.699 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:45 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:45.373+0000 7fdb78fcb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:51:46.281 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:46.007+0000 7fdb78fcb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T15:51:46.557 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:46.455+0000 7fdb78fcb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:51:46.949 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:46.557+0000 7fdb78fcb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:51:46.949 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:46.770+0000 7fdb78fcb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T15:51:47.677 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:47.319+0000 7fdb78fcb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:51:47.677 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:47.418+0000 7fdb78fcb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:51:47.949 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:47.676+0000 7fdb78fcb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:51:47.949 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:47.817+0000 7fdb78fcb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:51:47.949 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:47.917+0000 7fdb78fcb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:51:48.355 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:48 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:48.039+0000 7fdb78fcb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:51:48.355 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:48 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:48.354+0000 7fdb78fcb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:51:49.274 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:48 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:48.988+0000 7fdb78fcb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:51:49.274 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:51:49.217+0000 7fdb78fcb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:51:49.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: Active manager daemon x restarted 2024-04-25T15:51:49.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: Activating manager daemon x 2024-04-25T15:51:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: osdmap e3: 0 total, 0 up, 0 in 2024-04-25T15:51:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: mgrmap e10: x(active, starting, since 0.00410399s) 2024-04-25T15:51:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:51:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-25T15:51:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-25T15:51:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-25T15:51:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-25T15:51:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: Manager daemon x is now available 2024-04-25T15:51:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:51:50.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-25T15:51:50.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-25T15:51:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:50 smithi031 ceph-mon[29140]: [25/Apr/2024:15:51:49] ENGINE Bus STARTING 2024-04-25T15:51:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:50 smithi031 ceph-mon[29140]: [25/Apr/2024:15:51:49] ENGINE Serving on http://172.21.15.31:8765 2024-04-25T15:51:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:50 smithi031 ceph-mon[29140]: [25/Apr/2024:15:51:50] ENGINE Serving on https://172.21.15.31:7150 2024-04-25T15:51:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:50 smithi031 ceph-mon[29140]: [25/Apr/2024:15:51:50] ENGINE Bus STARTED 2024-04-25T15:51:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:50 smithi031 ceph-mon[29140]: mgrmap e11: x(active, since 1.00683s) 2024-04-25T15:51:51.212 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2024-04-25T15:51:51.212 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.872+0000 7fe432d64700 1 Processor -- start 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.872+0000 7fe432d64700 1 -- start start 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.872+0000 7fe432d64700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c109e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.872+0000 7fe432d64700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c10a430 con 0x7fe42c109a30 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.873+0000 7fe430b00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c109e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.873+0000 7fe430b00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c109e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51686/0 (socket says 172.21.15.31:51686) 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.873+0000 7fe430b00700 1 -- 172.21.15.31:0/2192555951 learned_addr learned my addr 172.21.15.31:0/2192555951 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.873+0000 7fe430b00700 1 -- 172.21.15.31:0/2192555951 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42c10ab10 con 0x7fe42c109a30 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.873+0000 7fe430b00700 1 --2- 172.21.15.31:0/2192555951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c109e40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4200098a0 tx=0x7fe420009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=308bc5de8ea8c2e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.213 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.873+0000 7fe42b7fe700 1 -- 172.21.15.31:0/2192555951 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe420004160 con 0x7fe42c109a30 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.873+0000 7fe42b7fe700 1 -- 172.21.15.31:0/2192555951 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe420004300 con 0x7fe42c109a30 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.873+0000 7fe42b7fe700 1 -- 172.21.15.31:0/2192555951 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe420011a20 con 0x7fe42c109a30 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.874+0000 7fe432d64700 1 -- 172.21.15.31:0/2192555951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 msgr2=0x7fe42c109e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.874+0000 7fe432d64700 1 --2- 172.21.15.31:0/2192555951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c109e40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe4200098a0 tx=0x7fe420009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.874+0000 7fe432d64700 1 -- 172.21.15.31:0/2192555951 shutdown_connections 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.874+0000 7fe432d64700 1 --2- 172.21.15.31:0/2192555951 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c109e40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.874+0000 7fe432d64700 1 -- 172.21.15.31:0/2192555951 >> 172.21.15.31:0/2192555951 conn(0x7fe42c105070 msgr2=0x7fe42c107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.874+0000 7fe432d64700 1 -- 172.21.15.31:0/2192555951 shutdown_connections 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.874+0000 7fe432d64700 1 -- 172.21.15.31:0/2192555951 wait complete. 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.875+0000 7fe432d64700 1 Processor -- start 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.875+0000 7fe432d64700 1 -- start start 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.875+0000 7fe432d64700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c07ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.214 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.875+0000 7fe432d64700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe42c10dec0 con 0x7fe42c109a30 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.875+0000 7fe430b00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c07ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.875+0000 7fe430b00700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c07ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51702/0 (socket says 172.21.15.31:51702) 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.875+0000 7fe430b00700 1 -- 172.21.15.31:0/3166108042 learned_addr learned my addr 172.21.15.31:0/3166108042 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.876+0000 7fe430b00700 1 -- 172.21.15.31:0/3166108042 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42c07ef40 con 0x7fe42c109a30 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.876+0000 7fe430b00700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c07ea00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe420003e70 tx=0x7fe420011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.876+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe420003aa0 con 0x7fe42c109a30 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.876+0000 7fe432d64700 1 -- 172.21.15.31:0/3166108042 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42c07d190 con 0x7fe42c109a30 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.876+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe42001a8e0 con 0x7fe42c109a30 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.876+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe420024870 con 0x7fe42c109a30 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.876+0000 7fe432d64700 1 -- 172.21.15.31:0/3166108042 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe42c07d670 con 0x7fe42c109a30 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.878+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7fe420022020 con 0x7fe42c109a30 2024-04-25T15:51:51.215 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.878+0000 7fe429ffb700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 0x7fe41c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.878+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 --> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe41c0401e0 con 0x7fe41c03d5c0 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.878+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe4200526e0 con 0x7fe42c109a30 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.878+0000 7fe42bfff700 1 -- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 msgr2=0x7fe41c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3023595010 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:44.878+0000 7fe42bfff700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 0x7fe41c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:45.079+0000 7fe42bfff700 1 -- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 msgr2=0x7fe41c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3023595010 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:45.079+0000 7fe42bfff700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 0x7fe41c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:45.480+0000 7fe42bfff700 1 -- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 msgr2=0x7fe41c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3023595010 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:45.480+0000 7fe42bfff700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 0x7fe41c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:46.281+0000 7fe42bfff700 1 -- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 msgr2=0x7fe41c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3023595010 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:46.281+0000 7fe42bfff700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 0x7fe41c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:47.882+0000 7fe42bfff700 1 -- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 msgr2=0x7fe41c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.31:6800/3023595010 2024-04-25T15:51:51.216 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:47.882+0000 7fe42bfff700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 0x7fe41c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:49.221+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mgrmap(e 10) ==== 49296+0+0 (secure 0 0 0) 0x7fe420018a40 con 0x7fe42c109a30 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:49.221+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 msgr2=0x7fe41c03fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:49.221+0000 7fe429ffb700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 0x7fe41c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.223+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7fe4200546c0 con 0x7fe42c109a30 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.223+0000 7fe429ffb700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe420052270 0x7fe41c043000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.223+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe420050c60 con 0x7fe420052270 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.224+0000 7fe42bfff700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe420052270 0x7fe41c043000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.225+0000 7fe42bfff700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe420052270 0x7fe41c043000 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe418003a10 tx=0x7fe4180092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.226+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fe420050c60 con 0x7fe420052270 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 -- 172.21.15.31:0/3166108042 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe42c064cf0 con 0x7fe420052270 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe429ffb700 1 -- 172.21.15.31:0/3166108042 <== mgr.14152 v2:172.21.15.31:6800/3984865329 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fe42c064cf0 con 0x7fe420052270 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 -- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe420052270 msgr2=0x7fe41c043000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe420052270 0x7fe41c043000 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fe418003a10 tx=0x7fe4180092b0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.217 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 -- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 msgr2=0x7fe42c07ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c07ea00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe420003e70 tx=0x7fe420011aa0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 -- 172.21.15.31:0/3166108042 shutdown_connections 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe420052270 0x7fe41c043000 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe42c109a30 0x7fe42c07ea00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 --2- 172.21.15.31:0/3166108042 >> [v2:172.21.15.31:6800/3023595010,v1:172.21.15.31:6801/3023595010] conn(0x7fe41c03d5c0 0x7fe41c03fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.234+0000 7fe432d64700 1 -- 172.21.15.31:0/3166108042 >> 172.21.15.31:0/3166108042 conn(0x7fe42c105070 msgr2=0x7fe42c105920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.235+0000 7fe432d64700 1 -- 172.21.15.31:0/3166108042 shutdown_connections 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:50.235+0000 7fe432d64700 1 -- 172.21.15.31:0/3166108042 wait complete. 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 9 is available 2024-04-25T15:51:51.218 INFO:teuthology.orchestra.run.smithi031.stdout:Generating a dashboard self-signed certificate... 2024-04-25T15:51:52.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:51 smithi031 ceph-mon[29140]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-25T15:51:52.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:51 smithi031 ceph-mon[29140]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-25T15:51:52.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.669+0000 7f12bb21e700 1 Processor -- start 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.669+0000 7f12bb21e700 1 -- start start 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.669+0000 7f12bb21e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b4108cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.669+0000 7f12bb21e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f12b41092e0 con 0x7f12b41088e0 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.669+0000 7f12b8fba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b4108cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.669+0000 7f12b8fba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b4108cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51754/0 (socket says 172.21.15.31:51754) 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.669+0000 7f12b8fba700 1 -- 172.21.15.31:0/1022558529 learned_addr learned my addr 172.21.15.31:0/1022558529 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.669+0000 7f12b8fba700 1 -- 172.21.15.31:0/1022558529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b4109a10 con 0x7f12b41088e0 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.670+0000 7f12b8fba700 1 --2- 172.21.15.31:0/1022558529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b4108cf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f12a80098a0 tx=0x7f12a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e636f3daaab42bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:52.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.670+0000 7f12b37fe700 1 -- 172.21.15.31:0/1022558529 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12a8004160 con 0x7f12b41088e0 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.670+0000 7f12b37fe700 1 -- 172.21.15.31:0/1022558529 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12a8004300 con 0x7f12b41088e0 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.670+0000 7f12b37fe700 1 -- 172.21.15.31:0/1022558529 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12a8011a20 con 0x7f12b41088e0 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.671+0000 7f12bb21e700 1 -- 172.21.15.31:0/1022558529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 msgr2=0x7f12b4108cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.671+0000 7f12bb21e700 1 --2- 172.21.15.31:0/1022558529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b4108cf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f12a80098a0 tx=0x7f12a8009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.671+0000 7f12bb21e700 1 -- 172.21.15.31:0/1022558529 shutdown_connections 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.671+0000 7f12bb21e700 1 --2- 172.21.15.31:0/1022558529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b4108cf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.671+0000 7f12bb21e700 1 -- 172.21.15.31:0/1022558529 >> 172.21.15.31:0/1022558529 conn(0x7f12b4103f20 msgr2=0x7f12b41063d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.671+0000 7f12bb21e700 1 -- 172.21.15.31:0/1022558529 shutdown_connections 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.671+0000 7f12bb21e700 1 -- 172.21.15.31:0/1022558529 wait complete. 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.671+0000 7f12bb21e700 1 Processor -- start 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.671+0000 7f12bb21e700 1 -- start start 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12bb21e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b419dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12bb21e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f12b410cdc0 con 0x7f12b41088e0 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12b8fba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b419dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12b8fba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b419dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51768/0 (socket says 172.21.15.31:51768) 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12b8fba700 1 -- 172.21.15.31:0/2124462234 learned_addr learned my addr 172.21.15.31:0/2124462234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12b8fba700 1 -- 172.21.15.31:0/2124462234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b419e210 con 0x7f12b41088e0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12b8fba700 1 --2- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b419dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f12a8003e70 tx=0x7f12a8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12b1ffb700 1 -- 172.21.15.31:0/2124462234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12a8003aa0 con 0x7f12b41088e0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b419e4a0 con 0x7f12b41088e0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12b1ffb700 1 -- 172.21.15.31:0/2124462234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12a801a8e0 con 0x7f12b41088e0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.672+0000 7f12b1ffb700 1 -- 172.21.15.31:0/2124462234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12a8024870 con 0x7f12b41088e0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.673+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b41a10b0 con 0x7f12b41088e0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.673+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b4064cf0 con 0x7f12b41088e0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.673+0000 7f12b1ffb700 1 -- 172.21.15.31:0/2124462234 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7f12a8022020 con 0x7f12b41088e0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.674+0000 7f12b1ffb700 1 --2- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12a403d560 0x7f12a403fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.674+0000 7f12b1ffb700 1 -- 172.21.15.31:0/2124462234 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f12a8052640 con 0x7f12b41088e0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.674+0000 7f12b3fff700 1 --2- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12a403d560 0x7f12a403fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.674+0000 7f12b3fff700 1 --2- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12a403d560 0x7f12a403fa70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f12a0006fd0 tx=0x7f12a0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:52.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.679+0000 7f12b1ffb700 1 -- 172.21.15.31:0/2124462234 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f12a800e260 con 0x7f12b41088e0 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.835+0000 7f12b1ffb700 1 -- 172.21.15.31:0/2124462234 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f12a802dc90 con 0x7f12b41088e0 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.899+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f12b402e7f0 con 0x7f12a403d560 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.963+0000 7f12b1ffb700 1 -- 172.21.15.31:0/2124462234 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f12b402e7f0 con 0x7f12a403d560 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.965+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12a403d560 msgr2=0x7f12a403fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.965+0000 7f12bb21e700 1 --2- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12a403d560 0x7f12a403fa70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f12a0006fd0 tx=0x7f12a0006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.965+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 msgr2=0x7f12b419dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.965+0000 7f12bb21e700 1 --2- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b419dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f12a8003e70 tx=0x7f12a8011aa0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.965+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 shutdown_connections 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.965+0000 7f12bb21e700 1 --2- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12a403d560 0x7f12a403fa70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.965+0000 7f12bb21e700 1 --2- 172.21.15.31:0/2124462234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f12b41088e0 0x7f12b419dcd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.965+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 >> 172.21.15.31:0/2124462234 conn(0x7f12b4103f20 msgr2=0x7f12b4194280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.966+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 shutdown_connections 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:51.966+0000 7f12bb21e700 1 -- 172.21.15.31:0/2124462234 wait complete. 2024-04-25T15:51:52.560 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial admin user... 2024-04-25T15:51:52.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:52 smithi031 ceph-mon[29140]: mgrmap e12: x(active, since 2s) 2024-04-25T15:51:52.936 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:52 smithi031 ceph-mon[29140]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:51:52.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:51:52.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:51:52.937 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:51:54.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:54 smithi031 ceph-mon[29140]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:51:54.813 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:54 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:51:54.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$EW4olcJJ9/XZd8pE/nNG9eWOwpheqfqpsQ1Gm181enhaso/lxXA7K", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714060313, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-25T15:51:54.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.053+0000 7f3079cd3700 1 Processor -- start 2024-04-25T15:51:54.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.053+0000 7f3079cd3700 1 -- start start 2024-04-25T15:51:54.820 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.053+0000 7f3079cd3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307410c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.053+0000 7f3079cd3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f307410cb10 con 0x7f307407cc40 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.054+0000 7f30737fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307410c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.054+0000 7f30737fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307410c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51782/0 (socket says 172.21.15.31:51782) 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.054+0000 7f30737fe700 1 -- 172.21.15.31:0/2317475072 learned_addr learned my addr 172.21.15.31:0/2317475072 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.054+0000 7f30737fe700 1 -- 172.21.15.31:0/2317475072 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307410cc90 con 0x7f307407cc40 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.054+0000 7f30737fe700 1 --2- 172.21.15.31:0/2317475072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307410c520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f30680098a0 tx=0x7f3068009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aa5023f23db8f07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.054+0000 7f30727fc700 1 -- 172.21.15.31:0/2317475072 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3068004160 con 0x7f307407cc40 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.054+0000 7f30727fc700 1 -- 172.21.15.31:0/2317475072 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3068004300 con 0x7f307407cc40 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.054+0000 7f30727fc700 1 -- 172.21.15.31:0/2317475072 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3068011a20 con 0x7f307407cc40 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.055+0000 7f3079cd3700 1 -- 172.21.15.31:0/2317475072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 msgr2=0x7f307410c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.055+0000 7f3079cd3700 1 --2- 172.21.15.31:0/2317475072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307410c520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f30680098a0 tx=0x7f3068009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.055+0000 7f3079cd3700 1 -- 172.21.15.31:0/2317475072 shutdown_connections 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.055+0000 7f3079cd3700 1 --2- 172.21.15.31:0/2317475072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307410c520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.055+0000 7f3079cd3700 1 -- 172.21.15.31:0/2317475072 >> 172.21.15.31:0/2317475072 conn(0x7f3074103fb0 msgr2=0x7f3074106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:54.821 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.055+0000 7f3079cd3700 1 -- 172.21.15.31:0/2317475072 shutdown_connections 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.055+0000 7f3079cd3700 1 -- 172.21.15.31:0/2317475072 wait complete. 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.056+0000 7f3079cd3700 1 Processor -- start 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.056+0000 7f3079cd3700 1 -- start start 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f3079cd3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307419dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f3079cd3700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f307410dd50 con 0x7f307407cc40 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f30737fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307419dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f30737fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307419dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51784/0 (socket says 172.21.15.31:51784) 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f30737fe700 1 -- 172.21.15.31:0/4179662852 learned_addr learned my addr 172.21.15.31:0/4179662852 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f30737fe700 1 -- 172.21.15.31:0/4179662852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f307419e210 con 0x7f307407cc40 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f30737fe700 1 --2- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307419dcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3068000c00 tx=0x7f3068011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f3070ff9700 1 -- 172.21.15.31:0/4179662852 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3068003aa0 con 0x7f307407cc40 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f3070ff9700 1 -- 172.21.15.31:0/4179662852 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f306801b8f0 con 0x7f307407cc40 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f307419e4a0 con 0x7f307407cc40 2024-04-25T15:51:54.822 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.057+0000 7f3070ff9700 1 -- 172.21.15.31:0/4179662852 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3068023890 con 0x7f307407cc40 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.058+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30741a10b0 con 0x7f307407cc40 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.059+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3074064cf0 con 0x7f307407cc40 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.059+0000 7f3070ff9700 1 -- 172.21.15.31:0/4179662852 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f306802c3f0 con 0x7f307407cc40 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.059+0000 7f3070ff9700 1 --2- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f305c03d5c0 0x7f305c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.059+0000 7f3070ff9700 1 -- 172.21.15.31:0/4179662852 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3068052d90 con 0x7f307407cc40 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.059+0000 7f3072ffd700 1 --2- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f305c03d5c0 0x7f305c03fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.060+0000 7f3072ffd700 1 --2- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f305c03d5c0 0x7f305c03fad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3064006fd0 tx=0x7f3064006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.064+0000 7f3070ff9700 1 -- 172.21.15.31:0/4179662852 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f306801f030 con 0x7f307407cc40 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.296+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f30741a1340 con 0x7f305c03d5c0 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.543+0000 7f3070ff9700 1 -- 172.21.15.31:0/4179662852 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f30741a1340 con 0x7f305c03d5c0 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.545+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f305c03d5c0 msgr2=0x7f305c03fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.545+0000 7f3079cd3700 1 --2- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f305c03d5c0 0x7f305c03fad0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3064006fd0 tx=0x7f3064006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.545+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 msgr2=0x7f307419dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.545+0000 7f3079cd3700 1 --2- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307419dcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3068000c00 tx=0x7f3068011aa0 comp rx=0 tx=0).stop 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.546+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 shutdown_connections 2024-04-25T15:51:54.823 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.546+0000 7f3079cd3700 1 --2- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f305c03d5c0 0x7f305c03fad0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:54.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.546+0000 7f3079cd3700 1 --2- 172.21.15.31:0/4179662852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f307407cc40 0x7f307419dcd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:54.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.546+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 >> 172.21.15.31:0/4179662852 conn(0x7f3074103fb0 msgr2=0x7f3074105860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:54.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.546+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 shutdown_connections 2024-04-25T15:51:54.824 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:53.546+0000 7f3079cd3700 1 -- 172.21.15.31:0/4179662852 wait complete. 2024-04-25T15:51:54.824 INFO:teuthology.orchestra.run.smithi031.stdout:Fetching dashboard port number... 2024-04-25T15:51:55.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:55 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/428015596' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 8443 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.239+0000 7f7d1a260700 1 Processor -- start 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.239+0000 7f7d1a260700 1 -- start start 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.239+0000 7f7d1a260700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d14108ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.239+0000 7f7d1a260700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d141092d0 con 0x7f7d141088d0 2024-04-25T15:51:56.044 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.239+0000 7f7d137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d14108ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.239+0000 7f7d137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d14108ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56596/0 (socket says 172.21.15.31:56596) 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.239+0000 7f7d137fe700 1 -- 172.21.15.31:0/1539640482 learned_addr learned my addr 172.21.15.31:0/1539640482 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.239+0000 7f7d137fe700 1 -- 172.21.15.31:0/1539640482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d14109a00 con 0x7f7d141088d0 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.239+0000 7f7d137fe700 1 --2- 172.21.15.31:0/1539640482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d14108ce0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7d040098a0 tx=0x7f7d04009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10d9a2cf3d39916b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.240+0000 7f7d127fc700 1 -- 172.21.15.31:0/1539640482 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d04004160 con 0x7f7d141088d0 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.240+0000 7f7d127fc700 1 -- 172.21.15.31:0/1539640482 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d04004300 con 0x7f7d141088d0 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.240+0000 7f7d127fc700 1 -- 172.21.15.31:0/1539640482 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d04011a20 con 0x7f7d141088d0 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.240+0000 7f7d1a260700 1 -- 172.21.15.31:0/1539640482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 msgr2=0x7f7d14108ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.240+0000 7f7d1a260700 1 --2- 172.21.15.31:0/1539640482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d14108ce0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7d040098a0 tx=0x7f7d04009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.240+0000 7f7d1a260700 1 -- 172.21.15.31:0/1539640482 shutdown_connections 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.240+0000 7f7d1a260700 1 --2- 172.21.15.31:0/1539640482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d14108ce0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.240+0000 7f7d1a260700 1 -- 172.21.15.31:0/1539640482 >> 172.21.15.31:0/1539640482 conn(0x7f7d14103fb0 msgr2=0x7f7d14106400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:56.045 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.240+0000 7f7d1a260700 1 -- 172.21.15.31:0/1539640482 shutdown_connections 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.241+0000 7f7d1a260700 1 -- 172.21.15.31:0/1539640482 wait complete. 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.241+0000 7f7d1a260700 1 Processor -- start 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.241+0000 7f7d1a260700 1 -- start start 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d1a260700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d1407c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d1a260700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1410cdb0 con 0x7f7d141088d0 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d1407c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d137fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d1407c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56610/0 (socket says 172.21.15.31:56610) 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d137fe700 1 -- 172.21.15.31:0/428015596 learned_addr learned my addr 172.21.15.31:0/428015596 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d137fe700 1 -- 172.21.15.31:0/428015596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1407ce20 con 0x7f7d141088d0 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d137fe700 1 --2- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d1407c8e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d04003e70 tx=0x7f7d04011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d10ff9700 1 -- 172.21.15.31:0/428015596 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d04003af0 con 0x7f7d141088d0 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d10ff9700 1 -- 172.21.15.31:0/428015596 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d0401a8e0 con 0x7f7d141088d0 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d10ff9700 1 -- 172.21.15.31:0/428015596 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7d04024870 con 0x7f7d141088d0 2024-04-25T15:51:56.046 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d141a0ac0 con 0x7f7d141088d0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.242+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d141a0f40 con 0x7f7d141088d0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.243+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d14064cf0 con 0x7f7d141088d0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.243+0000 7f7d10ff9700 1 -- 172.21.15.31:0/428015596 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f7d04022020 con 0x7f7d141088d0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.244+0000 7f7d10ff9700 1 --2- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7cfc03d5d0 0x7f7cfc03fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.244+0000 7f7d10ff9700 1 -- 172.21.15.31:0/428015596 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7d04052870 con 0x7f7d141088d0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.244+0000 7f7d12ffd700 1 --2- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7cfc03d5d0 0x7f7cfc03fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.244+0000 7f7d12ffd700 1 --2- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7cfc03d5d0 0x7f7cfc03fae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7d08006fd0 tx=0x7f7d08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.249+0000 7f7d10ff9700 1 -- 172.21.15.31:0/428015596 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7d04029df0 con 0x7f7d141088d0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.456+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f7d141a1750 con 0x7f7d141088d0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.456+0000 7f7d10ff9700 1 -- 172.21.15.31:0/428015596 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f7d0401ad90 con 0x7f7d141088d0 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7cfc03d5d0 msgr2=0x7f7cfc03fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 --2- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7cfc03d5d0 0x7f7cfc03fae0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7d08006fd0 tx=0x7f7d08006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 msgr2=0x7f7d1407c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:56.047 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 --2- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d1407c8e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d04003e70 tx=0x7f7d04011aa0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 shutdown_connections 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 --2- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7cfc03d5d0 0x7f7cfc03fae0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 --2- 172.21.15.31:0/428015596 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d141088d0 0x7f7d1407c8e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 >> 172.21.15.31:0/428015596 conn(0x7f7d14103fb0 msgr2=0x7f7d14105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 shutdown_connections 2024-04-25T15:51:56.048 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:55.459+0000 7f7d1a260700 1 -- 172.21.15.31:0/428015596 wait complete. 2024-04-25T15:51:56.052 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-25T15:51:56.053 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2024-04-25T15:51:56.060 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2024-04-25T15:51:56.061 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph Dashboard is now available at: 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi031.stdout: URL: https://smithi031:8443/ 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi031.stdout: User: admin 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi031.stdout: Password: n5ux2f0nvp 2024-04-25T15:51:56.062 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi031.stdout:Saving cluster configuration to /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config directory 2024-04-25T15:51:56.063 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling autotune for osd_memory_target 2024-04-25T15:51:56.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:56 smithi031 ceph-mon[29140]: mgrmap e13: x(active, since 6s) 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.519+0000 7f127dc1e700 1 Processor -- start 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.519+0000 7f127dc1e700 1 -- start start 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.519+0000 7f127dc1e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f1278109e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.519+0000 7f127dc1e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f127810a430 con 0x7f1278109a30 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.520+0000 7f12777fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f1278109e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.520+0000 7f12777fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f1278109e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56626/0 (socket says 172.21.15.31:56626) 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.520+0000 7f12777fe700 1 -- 172.21.15.31:0/4204872303 learned_addr learned my addr 172.21.15.31:0/4204872303 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.520+0000 7f12777fe700 1 -- 172.21.15.31:0/4204872303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127810ab10 con 0x7f1278109a30 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.520+0000 7f12777fe700 1 --2- 172.21.15.31:0/4204872303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f1278109e40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1270009e80 tx=0x7f127000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d89f15df86311fcb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.520+0000 7f12767fc700 1 -- 172.21.15.31:0/4204872303 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1270017070 con 0x7f1278109a30 2024-04-25T15:51:57.359 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.520+0000 7f12767fc700 1 -- 172.21.15.31:0/4204872303 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f127000b8c0 con 0x7f1278109a30 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.520+0000 7f12767fc700 1 -- 172.21.15.31:0/4204872303 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1270004490 con 0x7f1278109a30 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.521+0000 7f127dc1e700 1 -- 172.21.15.31:0/4204872303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 msgr2=0x7f1278109e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.521+0000 7f127dc1e700 1 --2- 172.21.15.31:0/4204872303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f1278109e40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1270009e80 tx=0x7f127000b330 comp rx=0 tx=0).stop 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.521+0000 7f127dc1e700 1 -- 172.21.15.31:0/4204872303 shutdown_connections 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.521+0000 7f127dc1e700 1 --2- 172.21.15.31:0/4204872303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f1278109e40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.521+0000 7f127dc1e700 1 -- 172.21.15.31:0/4204872303 >> 172.21.15.31:0/4204872303 conn(0x7f1278105070 msgr2=0x7f1278107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.521+0000 7f127dc1e700 1 -- 172.21.15.31:0/4204872303 shutdown_connections 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.521+0000 7f127dc1e700 1 -- 172.21.15.31:0/4204872303 wait complete. 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.522+0000 7f127dc1e700 1 Processor -- start 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.522+0000 7f127dc1e700 1 -- start start 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.522+0000 7f127dc1e700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f127819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.522+0000 7f127dc1e700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f127810dec0 con 0x7f1278109a30 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.522+0000 7f12777fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f127819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:57.360 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.522+0000 7f12777fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f127819dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56630/0 (socket says 172.21.15.31:56630) 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.522+0000 7f12777fe700 1 -- 172.21.15.31:0/493969592 learned_addr learned my addr 172.21.15.31:0/493969592 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.523+0000 7f12777fe700 1 -- 172.21.15.31:0/493969592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f127819e2e0 con 0x7f1278109a30 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.523+0000 7f12777fe700 1 --2- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f127819dda0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1270009fd0 tx=0x7f127000bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.523+0000 7f1274ff9700 1 -- 172.21.15.31:0/493969592 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1270020020 con 0x7f1278109a30 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.523+0000 7f1274ff9700 1 -- 172.21.15.31:0/493969592 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1270003b50 con 0x7f1278109a30 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.523+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f127819e570 con 0x7f1278109a30 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.523+0000 7f1274ff9700 1 -- 172.21.15.31:0/493969592 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1270017040 con 0x7f1278109a30 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.523+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12781a1180 con 0x7f1278109a30 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.524+0000 7f1274ff9700 1 -- 172.21.15.31:0/493969592 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f1270023020 con 0x7f1278109a30 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.524+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1278064cf0 con 0x7f1278109a30 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.524+0000 7f1274ff9700 1 --2- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f126403d610 0x7f126403fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.524+0000 7f1274ff9700 1 -- 172.21.15.31:0/493969592 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f127000f050 con 0x7f1278109a30 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.524+0000 7f1276ffd700 1 --2- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f126403d610 0x7f126403fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.525+0000 7f1276ffd700 1 --2- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f126403d610 0x7f126403fb20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1260006fd0 tx=0x7f1260006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:57.361 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.530+0000 7f1274ff9700 1 -- 172.21.15.31:0/493969592 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f127002a020 con 0x7f1278109a30 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.739+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f12781a13a0 con 0x7f1278109a30 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.739+0000 7f1274ff9700 1 -- 172.21.15.31:0/493969592 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f127002dd60 con 0x7f1278109a30 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.741+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f126403d610 msgr2=0x7f126403fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.741+0000 7f127dc1e700 1 --2- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f126403d610 0x7f126403fb20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1260006fd0 tx=0x7f1260006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.741+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 msgr2=0x7f127819dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.741+0000 7f127dc1e700 1 --2- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f127819dda0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1270009fd0 tx=0x7f127000bd30 comp rx=0 tx=0).stop 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.741+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 shutdown_connections 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.741+0000 7f127dc1e700 1 --2- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f126403d610 0x7f126403fb20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.741+0000 7f127dc1e700 1 --2- 172.21.15.31:0/493969592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1278109a30 0x7f127819dda0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.741+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 >> 172.21.15.31:0/493969592 conn(0x7f1278105070 msgr2=0x7f1278105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.742+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 shutdown_connections 2024-04-25T15:51:57.362 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:56.742+0000 7f127dc1e700 1 -- 172.21.15.31:0/493969592 wait complete. 2024-04-25T15:51:58.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.803+0000 7f85788ca700 1 Processor -- start 2024-04-25T15:51:58.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.803+0000 7f85788ca700 1 -- start start 2024-04-25T15:51:58.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.803+0000 7f85788ca700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f857410a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.803+0000 7f85788ca700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f857410a830 con 0x7f8574109e50 2024-04-25T15:51:58.682 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.803+0000 7f857259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f857410a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.803+0000 7f857259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f857410a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56636/0 (socket says 172.21.15.31:56636) 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.803+0000 7f857259c700 1 -- 172.21.15.31:0/3183713631 learned_addr learned my addr 172.21.15.31:0/3183713631 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.803+0000 7f857259c700 1 -- 172.21.15.31:0/3183713631 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8574108ad0 con 0x7f8574109e50 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.804+0000 7f857259c700 1 --2- 172.21.15.31:0/3183713631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f857410a260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f85640098a0 tx=0x7f8564009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f66154bb069225d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.804+0000 7f8571d9b700 1 -- 172.21.15.31:0/3183713631 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8564004160 con 0x7f8574109e50 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.804+0000 7f8571d9b700 1 -- 172.21.15.31:0/3183713631 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8564004300 con 0x7f8574109e50 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.804+0000 7f8571d9b700 1 -- 172.21.15.31:0/3183713631 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8564011a20 con 0x7f8574109e50 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.804+0000 7f85788ca700 1 -- 172.21.15.31:0/3183713631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 msgr2=0x7f857410a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.805+0000 7f85788ca700 1 --2- 172.21.15.31:0/3183713631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f857410a260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f85640098a0 tx=0x7f8564009bb0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.805+0000 7f85788ca700 1 -- 172.21.15.31:0/3183713631 shutdown_connections 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.805+0000 7f85788ca700 1 --2- 172.21.15.31:0/3183713631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f857410a260 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.805+0000 7f85788ca700 1 -- 172.21.15.31:0/3183713631 >> 172.21.15.31:0/3183713631 conn(0x7f8574103770 msgr2=0x7f8574105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.805+0000 7f85788ca700 1 -- 172.21.15.31:0/3183713631 shutdown_connections 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.805+0000 7f85788ca700 1 -- 172.21.15.31:0/3183713631 wait complete. 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.805+0000 7f85788ca700 1 Processor -- start 2024-04-25T15:51:58.683 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.806+0000 7f85788ca700 1 -- start start 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.806+0000 7f85788ca700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f8574108340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.806+0000 7f85788ca700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f857410ce90 con 0x7f8574109e50 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.806+0000 7f857259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f8574108340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.806+0000 7f857259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f8574108340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56640/0 (socket says 172.21.15.31:56640) 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.806+0000 7f857259c700 1 -- 172.21.15.31:0/1372750732 learned_addr learned my addr 172.21.15.31:0/1372750732 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.806+0000 7f857259c700 1 -- 172.21.15.31:0/1372750732 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f857419e8b0 con 0x7f8574109e50 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.806+0000 7f857259c700 1 --2- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f8574108340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8564011a20 tx=0x7f8564011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.807+0000 7f856b7fe700 1 -- 172.21.15.31:0/1372750732 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8564003af0 con 0x7f8574109e50 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.807+0000 7f856b7fe700 1 -- 172.21.15.31:0/1372750732 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f856401b8f0 con 0x7f8574109e50 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.807+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8574108880 con 0x7f8574109e50 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.807+0000 7f856b7fe700 1 -- 172.21.15.31:0/1372750732 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8564023890 con 0x7f8574109e50 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.807+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85741a3170 con 0x7f8574109e50 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.808+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8574064cf0 con 0x7f8574109e50 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.808+0000 7f856b7fe700 1 -- 172.21.15.31:0/1372750732 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f856402c3f0 con 0x7f8574109e50 2024-04-25T15:51:58.684 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.808+0000 7f856b7fe700 1 --2- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f856003d6c0 0x7f856003fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.808+0000 7f856b7fe700 1 -- 172.21.15.31:0/1372750732 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8564053150 con 0x7f8574109e50 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.808+0000 7f856bfff700 1 --2- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f856003d6c0 0x7f856003fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.809+0000 7f856bfff700 1 --2- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f856003d6c0 0x7f856003fbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f855c006fd0 tx=0x7f855c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:57.813+0000 7f856b7fe700 1 -- 172.21.15.31:0/1372750732 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f856401f030 con 0x7f8574109e50 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.135+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f85741a3580 con 0x7f8574109e50 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.138+0000 7f856b7fe700 1 -- 172.21.15.31:0/1372750732 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f85741a3580 con 0x7f8574109e50 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f856003d6c0 msgr2=0x7f856003fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 --2- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f856003d6c0 0x7f856003fbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f855c006fd0 tx=0x7f855c006e40 comp rx=0 tx=0).stop 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 msgr2=0x7f8574108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 --2- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f8574108340 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8564011a20 tx=0x7f8564011b00 comp rx=0 tx=0).stop 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 shutdown_connections 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 --2- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f856003d6c0 0x7f856003fbd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 --2- 172.21.15.31:0/1372750732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8574109e50 0x7f8574108340 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 >> 172.21.15.31:0/1372750732 conn(0x7f8574103770 msgr2=0x7f8574105010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 shutdown_connections 2024-04-25T15:51:58.685 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr 2024-04-25T15:51:58.140+0000 7f85788ca700 1 -- 172.21.15.31:0/1372750732 wait complete. 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout:Or, if you are only running a single cluster on this host: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: ceph telemetry on 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout:For more information see: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-25T15:51:58.686 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:51:58.687 INFO:teuthology.orchestra.run.smithi031.stdout:Bootstrap complete. 2024-04-25T15:51:58.722 INFO:tasks.cephadm:Fetching config... 2024-04-25T15:51:58.723 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:51:58.723 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-25T15:51:58.739 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-25T15:51:58.739 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:51:58.739 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-25T15:51:58.795 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-25T15:51:58.795 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:51:58.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-25T15:51:58.860 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-25T15:51:58.860 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:51:58.861 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-25T15:51:58.916 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-25T15:51:58.916 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/CcJAaZhzxeOyxFMEYBPZ3NuOufE8PGKqsNPDzs00rnat7/9R7kUeuLtDKQwa5HGhFOJwhSCQtiUp1oec8R3Kblal9i9e7WNAl8VWYMkaRp/iZO/vQ+/K+FjkTn0OOSna3NtCpznkJN2uGegPDnHjj6EU/DxZjAjwgIb2tBaFtSRoHZb39lXHIrf6nrYJGIquUh7FjuKt18WnN2PPR1ugsoNxyo9USAaIHqtFy9NBMH1oEg8sE91UkoJs4PLR+GZrrd8XQKFZ21t9EYV8oXnhTxWl7ax/9uW17jh/1UIOrKUvsgE4baBNbVSqCuvn9p9je5Pa8oUaEi+dTNOaAYq6bKKS1379gNWClmvBIoM5uSRUyCoXx7ztOgUjTKdOqwC9QcSm7UCFkmaNMTDfAdDXGezmQqkLdPxDOFVMNLQLh1qIL1XpvmZttKLlbkyqmOL7AliFO5bI6lkXiq/gruXH0KuukhyTWzxOLnAp6cmHLToRm9z/9sy21Np9ZHU+Xm0= ceph-74fd3742-031b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T15:51:59.000 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/CcJAaZhzxeOyxFMEYBPZ3NuOufE8PGKqsNPDzs00rnat7/9R7kUeuLtDKQwa5HGhFOJwhSCQtiUp1oec8R3Kblal9i9e7WNAl8VWYMkaRp/iZO/vQ+/K+FjkTn0OOSna3NtCpznkJN2uGegPDnHjj6EU/DxZjAjwgIb2tBaFtSRoHZb39lXHIrf6nrYJGIquUh7FjuKt18WnN2PPR1ugsoNxyo9USAaIHqtFy9NBMH1oEg8sE91UkoJs4PLR+GZrrd8XQKFZ21t9EYV8oXnhTxWl7ax/9uW17jh/1UIOrKUvsgE4baBNbVSqCuvn9p9je5Pa8oUaEi+dTNOaAYq6bKKS1379gNWClmvBIoM5uSRUyCoXx7ztOgUjTKdOqwC9QcSm7UCFkmaNMTDfAdDXGezmQqkLdPxDOFVMNLQLh1qIL1XpvmZttKLlbkyqmOL7AliFO5bI6lkXiq/gruXH0KuukhyTWzxOLnAp6cmHLToRm9z/9sy21Np9ZHU+Xm0= ceph-74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:51:59.012 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/CcJAaZhzxeOyxFMEYBPZ3NuOufE8PGKqsNPDzs00rnat7/9R7kUeuLtDKQwa5HGhFOJwhSCQtiUp1oec8R3Kblal9i9e7WNAl8VWYMkaRp/iZO/vQ+/K+FjkTn0OOSna3NtCpznkJN2uGegPDnHjj6EU/DxZjAjwgIb2tBaFtSRoHZb39lXHIrf6nrYJGIquUh7FjuKt18WnN2PPR1ugsoNxyo9USAaIHqtFy9NBMH1oEg8sE91UkoJs4PLR+GZrrd8XQKFZ21t9EYV8oXnhTxWl7ax/9uW17jh/1UIOrKUvsgE4baBNbVSqCuvn9p9je5Pa8oUaEi+dTNOaAYq6bKKS1379gNWClmvBIoM5uSRUyCoXx7ztOgUjTKdOqwC9QcSm7UCFkmaNMTDfAdDXGezmQqkLdPxDOFVMNLQLh1qIL1XpvmZttKLlbkyqmOL7AliFO5bI6lkXiq/gruXH0KuukhyTWzxOLnAp6cmHLToRm9z/9sy21Np9ZHU+Xm0= ceph-74fd3742-031b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T15:51:59.052 INFO:teuthology.orchestra.run.smithi063.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/CcJAaZhzxeOyxFMEYBPZ3NuOufE8PGKqsNPDzs00rnat7/9R7kUeuLtDKQwa5HGhFOJwhSCQtiUp1oec8R3Kblal9i9e7WNAl8VWYMkaRp/iZO/vQ+/K+FjkTn0OOSna3NtCpznkJN2uGegPDnHjj6EU/DxZjAjwgIb2tBaFtSRoHZb39lXHIrf6nrYJGIquUh7FjuKt18WnN2PPR1ugsoNxyo9USAaIHqtFy9NBMH1oEg8sE91UkoJs4PLR+GZrrd8XQKFZ21t9EYV8oXnhTxWl7ax/9uW17jh/1UIOrKUvsgE4baBNbVSqCuvn9p9je5Pa8oUaEi+dTNOaAYq6bKKS1379gNWClmvBIoM5uSRUyCoXx7ztOgUjTKdOqwC9QcSm7UCFkmaNMTDfAdDXGezmQqkLdPxDOFVMNLQLh1qIL1XpvmZttKLlbkyqmOL7AliFO5bI6lkXiq/gruXH0KuukhyTWzxOLnAp6cmHLToRm9z/9sy21Np9ZHU+Xm0= ceph-74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:51:59.064 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/CcJAaZhzxeOyxFMEYBPZ3NuOufE8PGKqsNPDzs00rnat7/9R7kUeuLtDKQwa5HGhFOJwhSCQtiUp1oec8R3Kblal9i9e7WNAl8VWYMkaRp/iZO/vQ+/K+FjkTn0OOSna3NtCpznkJN2uGegPDnHjj6EU/DxZjAjwgIb2tBaFtSRoHZb39lXHIrf6nrYJGIquUh7FjuKt18WnN2PPR1ugsoNxyo9USAaIHqtFy9NBMH1oEg8sE91UkoJs4PLR+GZrrd8XQKFZ21t9EYV8oXnhTxWl7ax/9uW17jh/1UIOrKUvsgE4baBNbVSqCuvn9p9je5Pa8oUaEi+dTNOaAYq6bKKS1379gNWClmvBIoM5uSRUyCoXx7ztOgUjTKdOqwC9QcSm7UCFkmaNMTDfAdDXGezmQqkLdPxDOFVMNLQLh1qIL1XpvmZttKLlbkyqmOL7AliFO5bI6lkXiq/gruXH0KuukhyTWzxOLnAp6cmHLToRm9z/9sy21Np9ZHU+Xm0= ceph-74fd3742-031b-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-25T15:51:59.104 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/CcJAaZhzxeOyxFMEYBPZ3NuOufE8PGKqsNPDzs00rnat7/9R7kUeuLtDKQwa5HGhFOJwhSCQtiUp1oec8R3Kblal9i9e7WNAl8VWYMkaRp/iZO/vQ+/K+FjkTn0OOSna3NtCpznkJN2uGegPDnHjj6EU/DxZjAjwgIb2tBaFtSRoHZb39lXHIrf6nrYJGIquUh7FjuKt18WnN2PPR1ugsoNxyo9USAaIHqtFy9NBMH1oEg8sE91UkoJs4PLR+GZrrd8XQKFZ21t9EYV8oXnhTxWl7ax/9uW17jh/1UIOrKUvsgE4baBNbVSqCuvn9p9je5Pa8oUaEi+dTNOaAYq6bKKS1379gNWClmvBIoM5uSRUyCoXx7ztOgUjTKdOqwC9QcSm7UCFkmaNMTDfAdDXGezmQqkLdPxDOFVMNLQLh1qIL1XpvmZttKLlbkyqmOL7AliFO5bI6lkXiq/gruXH0KuukhyTWzxOLnAp6cmHLToRm9z/9sy21Np9ZHU+Xm0= ceph-74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:51:59.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-25T15:51:59.358 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:51:59.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:51:59 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1372750732' entity='client.admin' 2024-04-25T15:52:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.064+0000 7f7c0ff17700 1 -- 172.21.15.31:0/1576772588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 msgr2=0x7f7c08102df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.064+0000 7f7c0ff17700 1 --2- 172.21.15.31:0/1576772588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 0x7f7c08102df0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f7c04009910 tx=0x7f7c04009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.064+0000 7f7c0ff17700 1 -- 172.21.15.31:0/1576772588 shutdown_connections 2024-04-25T15:52:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.064+0000 7f7c0ff17700 1 --2- 172.21.15.31:0/1576772588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 0x7f7c08102df0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.064+0000 7f7c0ff17700 1 -- 172.21.15.31:0/1576772588 >> 172.21.15.31:0/1576772588 conn(0x7f7c080fe000 msgr2=0x7f7c081004d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.064+0000 7f7c0ff17700 1 -- 172.21.15.31:0/1576772588 shutdown_connections 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.064+0000 7f7c0ff17700 1 -- 172.21.15.31:0/1576772588 wait complete. 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.065+0000 7f7c0ff17700 1 Processor -- start 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.065+0000 7f7c0ff17700 1 -- start start 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.065+0000 7f7c0ff17700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 0x7f7c08199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.065+0000 7f7c0ff17700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7c081089a0 con 0x7f7c081029e0 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.065+0000 7f7c0dcb3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 0x7f7c08199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.065+0000 7f7c0dcb3700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 0x7f7c08199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56656/0 (socket says 172.21.15.31:56656) 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.065+0000 7f7c0dcb3700 1 -- 172.21.15.31:0/3746259292 learned_addr learned my addr 172.21.15.31:0/3746259292 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.065+0000 7f7c0dcb3700 1 -- 172.21.15.31:0/3746259292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c08199b70 con 0x7f7c081029e0 2024-04-25T15:52:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.066+0000 7f7c0dcb3700 1 --2- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 0x7f7c08199630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c04006010 tx=0x7f7c04004fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.066+0000 7f7bfeffd700 1 -- 172.21.15.31:0/3746259292 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c04026070 con 0x7f7c081029e0 2024-04-25T15:52:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.066+0000 7f7bfeffd700 1 -- 172.21.15.31:0/3746259292 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c0400bc00 con 0x7f7c081029e0 2024-04-25T15:52:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.066+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c08199e00 con 0x7f7c081029e0 2024-04-25T15:52:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.066+0000 7f7bfeffd700 1 -- 172.21.15.31:0/3746259292 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7c04003970 con 0x7f7c081029e0 2024-04-25T15:52:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.066+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0819ca10 con 0x7f7c081029e0 2024-04-25T15:52:02.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.067+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c08109120 con 0x7f7c081029e0 2024-04-25T15:52:02.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.067+0000 7f7bfeffd700 1 -- 172.21.15.31:0/3746259292 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f7c04029070 con 0x7f7c081029e0 2024-04-25T15:52:02.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.068+0000 7f7bfeffd700 1 --2- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7bf403d610 0x7f7bf403fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:02.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.068+0000 7f7bfeffd700 1 -- 172.21.15.31:0/3746259292 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7c0401e070 con 0x7f7c081029e0 2024-04-25T15:52:02.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.068+0000 7f7c0d4b2700 1 --2- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7bf403d610 0x7f7bf403fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:02.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.068+0000 7f7c0d4b2700 1 --2- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7bf403d610 0x7f7bf403fb20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8006fd0 tx=0x7f7bf8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.073+0000 7f7bfeffd700 1 -- 172.21.15.31:0/3746259292 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7c04013070 con 0x7f7c081029e0 2024-04-25T15:52:02.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.323+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f7c0802e620 con 0x7f7c081029e0 2024-04-25T15:52:02.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.326+0000 7f7bfeffd700 1 -- 172.21.15.31:0/3746259292 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f7c04015350 con 0x7f7c081029e0 2024-04-25T15:52:02.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.329+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7bf403d610 msgr2=0x7f7bf403fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:02.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.329+0000 7f7c0ff17700 1 --2- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7bf403d610 0x7f7bf403fb20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7bf8006fd0 tx=0x7f7bf8006e40 comp rx=0 tx=0).stop 2024-04-25T15:52:02.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.329+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 msgr2=0x7f7c08199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.329+0000 7f7c0ff17700 1 --2- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 0x7f7c08199630 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c04006010 tx=0x7f7c04004fa0 comp rx=0 tx=0).stop 2024-04-25T15:52:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.329+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 shutdown_connections 2024-04-25T15:52:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.329+0000 7f7c0ff17700 1 --2- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7bf403d610 0x7f7bf403fb20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.329+0000 7f7c0ff17700 1 --2- 172.21.15.31:0/3746259292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7c081029e0 0x7f7c08199630 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.329+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 >> 172.21.15.31:0/3746259292 conn(0x7f7c080fe000 msgr2=0x7f7c0806b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.330+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 shutdown_connections 2024-04-25T15:52:02.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:02.330+0000 7f7c0ff17700 1 -- 172.21.15.31:0/3746259292 wait complete. 2024-04-25T15:52:02.959 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-25T15:52:02.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-25T15:52:03.536 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:03 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3746259292' entity='client.admin' 2024-04-25T15:52:03.664 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:52:04.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:04.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:04.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:52:04.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:04.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:04.876 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.207+0000 7f6d49840700 1 -- 172.21.15.31:0/1654694993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 msgr2=0x7f6d44106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.207+0000 7f6d49840700 1 --2- 172.21.15.31:0/1654694993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 0x7f6d44106a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009910 tx=0x7f6d34009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.207+0000 7f6d49840700 1 -- 172.21.15.31:0/1654694993 shutdown_connections 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.207+0000 7f6d49840700 1 --2- 172.21.15.31:0/1654694993 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 0x7f6d44106a30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.207+0000 7f6d49840700 1 -- 172.21.15.31:0/1654694993 >> 172.21.15.31:0/1654694993 conn(0x7f6d440fe010 msgr2=0x7f6d441004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.207+0000 7f6d49840700 1 -- 172.21.15.31:0/1654694993 shutdown_connections 2024-04-25T15:52:05.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.207+0000 7f6d49840700 1 -- 172.21.15.31:0/1654694993 wait complete. 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.208+0000 7f6d49840700 1 Processor -- start 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.208+0000 7f6d49840700 1 -- start start 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.208+0000 7f6d49840700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 0x7f6d44103d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.208+0000 7f6d49840700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d44109a70 con 0x7f6d44104610 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.208+0000 7f6d42ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 0x7f6d44103d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.208+0000 7f6d42ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 0x7f6d44103d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33684/0 (socket says 172.21.15.31:33684) 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.208+0000 7f6d42ffd700 1 -- 172.21.15.31:0/2975238856 learned_addr learned my addr 172.21.15.31:0/2975238856 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.208+0000 7f6d42ffd700 1 -- 172.21.15.31:0/2975238856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d441042d0 con 0x7f6d44104610 2024-04-25T15:52:05.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.209+0000 7f6d42ffd700 1 --2- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 0x7f6d44103d90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6d340050d0 tx=0x7f6d34009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.209+0000 7f6d4883e700 1 -- 172.21.15.31:0/2975238856 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d34027070 con 0x7f6d44104610 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.209+0000 7f6d4883e700 1 -- 172.21.15.31:0/2975238856 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d3400be10 con 0x7f6d44104610 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.209+0000 7f6d4883e700 1 -- 172.21.15.31:0/2975238856 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6d34022d80 con 0x7f6d44104610 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.209+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d44102520 con 0x7f6d44104610 2024-04-25T15:52:05.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.209+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d44102a00 con 0x7f6d44104610 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.210+0000 7f6d4883e700 1 -- 172.21.15.31:0/2975238856 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f6d3402a070 con 0x7f6d44104610 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.210+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d4410a1f0 con 0x7f6d44104610 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.210+0000 7f6d4883e700 1 --2- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d2c03d5c0 0x7f6d2c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.210+0000 7f6d4883e700 1 -- 172.21.15.31:0/2975238856 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6d3401e030 con 0x7f6d44104610 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.210+0000 7f6d427fc700 1 --2- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d2c03d5c0 0x7f6d2c03fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:05.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.211+0000 7f6d427fc700 1 --2- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d2c03d5c0 0x7f6d2c03fad0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6d38006fd0 tx=0x7f6d38006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:05.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.215+0000 7f6d4883e700 1 -- 172.21.15.31:0/2975238856 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6d34013070 con 0x7f6d44104610 2024-04-25T15:52:05.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.429+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f6d44192410 con 0x7f6d2c03d5c0 2024-04-25T15:52:05.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.433+0000 7f6d4883e700 1 -- 172.21.15.31:0/2975238856 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6d44192410 con 0x7f6d2c03d5c0 2024-04-25T15:52:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.435+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d2c03d5c0 msgr2=0x7f6d2c03fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.435+0000 7f6d49840700 1 --2- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d2c03d5c0 0x7f6d2c03fad0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6d38006fd0 tx=0x7f6d38006e40 comp rx=0 tx=0).stop 2024-04-25T15:52:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.435+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 msgr2=0x7f6d44103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.435+0000 7f6d49840700 1 --2- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 0x7f6d44103d90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6d340050d0 tx=0x7f6d34009f80 comp rx=0 tx=0).stop 2024-04-25T15:52:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.435+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 shutdown_connections 2024-04-25T15:52:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.435+0000 7f6d49840700 1 --2- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d2c03d5c0 0x7f6d2c03fad0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.435+0000 7f6d49840700 1 --2- 172.21.15.31:0/2975238856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d44104610 0x7f6d44103d90 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:05.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.435+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 >> 172.21.15.31:0/2975238856 conn(0x7f6d440fe010 msgr2=0x7f6d440fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:05.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.436+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 shutdown_connections 2024-04-25T15:52:05.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:05.436+0000 7f6d49840700 1 -- 172.21.15.31:0/2975238856 wait complete. 2024-04-25T15:52:06.291 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi063 2024-04-25T15:52:06.291 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:52:06.291 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/etc/ceph/ceph.conf 2024-04-25T15:52:06.311 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:52:06.311 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:06.369 INFO:tasks.cephadm:Adding host smithi063 to orchestrator... 2024-04-25T15:52:06.369 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch host add smithi063 2024-04-25T15:52:06.612 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:52:06.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: Updating smithi031:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:07.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:07 smithi031 ceph-mon[29140]: Updating smithi031:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T15:52:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.153+0000 7f8b3ace5700 1 -- 172.21.15.31:0/464869778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 msgr2=0x7f8b34109e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.153+0000 7f8b3ace5700 1 --2- 172.21.15.31:0/464869778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 0x7f8b34109e50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8b28009910 tx=0x7f8b28009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.153+0000 7f8b3ace5700 1 -- 172.21.15.31:0/464869778 shutdown_connections 2024-04-25T15:52:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.153+0000 7f8b3ace5700 1 --2- 172.21.15.31:0/464869778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 0x7f8b34109e50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.153+0000 7f8b3ace5700 1 -- 172.21.15.31:0/464869778 >> 172.21.15.31:0/464869778 conn(0x7f8b340ff530 msgr2=0x7f8b34101980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.153+0000 7f8b3ace5700 1 -- 172.21.15.31:0/464869778 shutdown_connections 2024-04-25T15:52:08.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.153+0000 7f8b3ace5700 1 -- 172.21.15.31:0/464869778 wait complete. 2024-04-25T15:52:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.154+0000 7f8b3ace5700 1 Processor -- start 2024-04-25T15:52:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.154+0000 7f8b3ace5700 1 -- start start 2024-04-25T15:52:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.154+0000 7f8b3ace5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 0x7f8b3419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.154+0000 7f8b3ace5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8b3410df90 con 0x7f8b34109a40 2024-04-25T15:52:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.154+0000 7f8b38a81700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 0x7f8b3419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.154+0000 7f8b38a81700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 0x7f8b3419f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33708/0 (socket says 172.21.15.31:33708) 2024-04-25T15:52:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.154+0000 7f8b38a81700 1 -- 172.21.15.31:0/4022694134 learned_addr learned my addr 172.21.15.31:0/4022694134 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:52:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.154+0000 7f8b38a81700 1 -- 172.21.15.31:0/4022694134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3419fa30 con 0x7f8b34109a40 2024-04-25T15:52:08.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.155+0000 7f8b38a81700 1 --2- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 0x7f8b3419f4f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8b280056d0 tx=0x7f8b280057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.155+0000 7f8b31ffb700 1 -- 172.21.15.31:0/4022694134 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b28027070 con 0x7f8b34109a40 2024-04-25T15:52:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.155+0000 7f8b31ffb700 1 -- 172.21.15.31:0/4022694134 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b2800be10 con 0x7f8b34109a40 2024-04-25T15:52:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.155+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b3419fd20 con 0x7f8b34109a40 2024-04-25T15:52:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.155+0000 7f8b31ffb700 1 -- 172.21.15.31:0/4022694134 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8b28022d20 con 0x7f8b34109a40 2024-04-25T15:52:08.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.155+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b34199850 con 0x7f8b34109a40 2024-04-25T15:52:08.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.156+0000 7f8b31ffb700 1 -- 172.21.15.31:0/4022694134 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f8b2802a070 con 0x7f8b34109a40 2024-04-25T15:52:08.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.156+0000 7f8b31ffb700 1 --2- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8b2403d670 0x7f8b2403fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:08.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.156+0000 7f8b31ffb700 1 -- 172.21.15.31:0/4022694134 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8b2801e030 con 0x7f8b34109a40 2024-04-25T15:52:08.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.156+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b34105150 con 0x7f8b34109a40 2024-04-25T15:52:08.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.156+0000 7f8b33fff700 1 --2- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8b2403d670 0x7f8b2403fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:08.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.157+0000 7f8b33fff700 1 --2- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8b2403d670 0x7f8b2403fb80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b20006fd0 tx=0x7f8b20006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:08.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.162+0000 7f8b31ffb700 1 -- 172.21.15.31:0/4022694134 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b28013070 con 0x7f8b34109a40 2024-04-25T15:52:08.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:08.375+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi063", "target": ["mon-mgr", ""]}) -- 0x7f8b3419ff50 con 0x7f8b2403d670 2024-04-25T15:52:09.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:09 smithi031 ceph-mon[29140]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi063", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:52:09.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:09 smithi031 ceph-mon[29140]: Deploying cephadm binary to smithi063 2024-04-25T15:52:10.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:10 smithi031 ceph-mon[29140]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:12.615 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi063' with addr '172.21.15.63' 2024-04-25T15:52:12.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.614+0000 7f8b31ffb700 1 -- 172.21.15.31:0/4022694134 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f8b3419ff50 con 0x7f8b2403d670 2024-04-25T15:52:12.616 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:12 smithi031 ceph-mon[29140]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8b2403d670 msgr2=0x7f8b2403fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 --2- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8b2403d670 0x7f8b2403fb80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b20006fd0 tx=0x7f8b20006e40 comp rx=0 tx=0).stop 2024-04-25T15:52:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 msgr2=0x7f8b3419f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 --2- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 0x7f8b3419f4f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8b280056d0 tx=0x7f8b280057b0 comp rx=0 tx=0).stop 2024-04-25T15:52:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 shutdown_connections 2024-04-25T15:52:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 --2- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8b2403d670 0x7f8b2403fb80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 --2- 172.21.15.31:0/4022694134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8b34109a40 0x7f8b3419f4f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 >> 172.21.15.31:0/4022694134 conn(0x7f8b340ff530 msgr2=0x7f8b34100510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:12.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 shutdown_connections 2024-04-25T15:52:12.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:12.617+0000 7f8b3ace5700 1 -- 172.21.15.31:0/4022694134 wait complete. 2024-04-25T15:52:13.275 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T15:52:13.523 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:52:13.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:13.919 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:13 smithi031 ceph-mon[29140]: Added host smithi063 2024-04-25T15:52:13.919 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:13.919 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:14.878 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:14 smithi031 ceph-mon[29140]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:14.878 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:14 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.097+0000 7f04ba905700 1 -- 172.21.15.31:0/1677211753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 msgr2=0x7f04b4104ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.097+0000 7f04ba905700 1 --2- 172.21.15.31:0/1677211753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 0x7f04b4104ff0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f049c009910 tx=0x7f049c009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.097+0000 7f04ba905700 1 -- 172.21.15.31:0/1677211753 shutdown_connections 2024-04-25T15:52:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.097+0000 7f04ba905700 1 --2- 172.21.15.31:0/1677211753 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 0x7f04b4104ff0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.097+0000 7f04ba905700 1 -- 172.21.15.31:0/1677211753 >> 172.21.15.31:0/1677211753 conn(0x7f04b40fe490 msgr2=0x7f04b41008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.097+0000 7f04ba905700 1 -- 172.21.15.31:0/1677211753 shutdown_connections 2024-04-25T15:52:15.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.097+0000 7f04ba905700 1 -- 172.21.15.31:0/1677211753 wait complete. 2024-04-25T15:52:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.098+0000 7f04ba905700 1 Processor -- start 2024-04-25T15:52:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.098+0000 7f04ba905700 1 -- start start 2024-04-25T15:52:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.098+0000 7f04ba905700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 0x7f04b419f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.098+0000 7f04ba905700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f04b410dfe0 con 0x7f04b4104be0 2024-04-25T15:52:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.098+0000 7f04b3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 0x7f04b419f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.099+0000 7f04b3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 0x7f04b419f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50806/0 (socket says 172.21.15.31:50806) 2024-04-25T15:52:15.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.099+0000 7f04b3fff700 1 -- 172.21.15.31:0/2156992928 learned_addr learned my addr 172.21.15.31:0/2156992928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:52:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.099+0000 7f04b3fff700 1 -- 172.21.15.31:0/2156992928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04b419fa00 con 0x7f04b4104be0 2024-04-25T15:52:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.099+0000 7f04b3fff700 1 --2- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 0x7f04b419f4c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f049c006010 tx=0x7f049c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.099+0000 7f04b17fa700 1 -- 172.21.15.31:0/2156992928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f049c026070 con 0x7f04b4104be0 2024-04-25T15:52:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.099+0000 7f04b17fa700 1 -- 172.21.15.31:0/2156992928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f049c00bc00 con 0x7f04b4104be0 2024-04-25T15:52:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.099+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04b419fc90 con 0x7f04b4104be0 2024-04-25T15:52:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.099+0000 7f04b17fa700 1 -- 172.21.15.31:0/2156992928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f049c003910 con 0x7f04b4104be0 2024-04-25T15:52:15.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.099+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04b4199650 con 0x7f04b4104be0 2024-04-25T15:52:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.100+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04b406c270 con 0x7f04b4104be0 2024-04-25T15:52:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.100+0000 7f04b17fa700 1 -- 172.21.15.31:0/2156992928 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f049c029070 con 0x7f04b4104be0 2024-04-25T15:52:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.101+0000 7f04b17fa700 1 --2- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f04a003d670 0x7f04a003fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.101+0000 7f04b17fa700 1 -- 172.21.15.31:0/2156992928 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f049c01e070 con 0x7f04b4104be0 2024-04-25T15:52:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.101+0000 7f04b37fe700 1 --2- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f04a003d670 0x7f04a003fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:15.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.101+0000 7f04b37fe700 1 --2- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f04a003d670 0x7f04a003fb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04a4006fd0 tx=0x7f04a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:15.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.106+0000 7f04b17fa700 1 -- 172.21.15.31:0/2156992928 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f049c013070 con 0x7f04b4104be0 2024-04-25T15:52:15.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.323+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f04b419ab60 con 0x7f04a003d670 2024-04-25T15:52:15.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.324+0000 7f04b17fa700 1 -- 172.21.15.31:0/2156992928 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f04b419ab60 con 0x7f04a003d670 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:52:15.325 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}] 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f04a003d670 msgr2=0x7f04a003fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 --2- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f04a003d670 0x7f04a003fb80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f04a4006fd0 tx=0x7f04a4006e40 comp rx=0 tx=0).stop 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 msgr2=0x7f04b419f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 --2- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 0x7f04b419f4c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f049c006010 tx=0x7f049c005780 comp rx=0 tx=0).stop 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 shutdown_connections 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 --2- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f04a003d670 0x7f04a003fb80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 --2- 172.21.15.31:0/2156992928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f04b4104be0 0x7f04b419f4c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:15.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 >> 172.21.15.31:0/2156992928 conn(0x7f04b40fe490 msgr2=0x7f04b4109dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 shutdown_connections 2024-04-25T15:52:15.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:15.326+0000 7f04ba905700 1 -- 172.21.15.31:0/2156992928 wait complete. 2024-04-25T15:52:15.931 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-04-25T15:52:15.932 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:52:15.932 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-04-25T15:52:15.951 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:52:15.951 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:16.008 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-04-25T15:52:16.008 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch host add smithi186 2024-04-25T15:52:16.251 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:52:16.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:16 smithi031 ceph-mon[29140]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:16.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:16.573 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:16 smithi031 ceph-mon[29140]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:52:17.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.807+0000 7f79733b8700 1 -- 172.21.15.31:0/1936794411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 msgr2=0x7f796c108d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:17.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.807+0000 7f79733b8700 1 --2- 172.21.15.31:0/1936794411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 0x7f796c108d20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7968009910 tx=0x7f7968009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:17.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.807+0000 7f79733b8700 1 -- 172.21.15.31:0/1936794411 shutdown_connections 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.807+0000 7f79733b8700 1 --2- 172.21.15.31:0/1936794411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 0x7f796c108d20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.807+0000 7f79733b8700 1 -- 172.21.15.31:0/1936794411 >> 172.21.15.31:0/1936794411 conn(0x7f796c0fe440 msgr2=0x7f796c100890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.807+0000 7f79733b8700 1 -- 172.21.15.31:0/1936794411 shutdown_connections 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.807+0000 7f79733b8700 1 -- 172.21.15.31:0/1936794411 wait complete. 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.808+0000 7f79733b8700 1 Processor -- start 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.808+0000 7f79733b8700 1 -- start start 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.808+0000 7f79733b8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 0x7f796c19f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.808+0000 7f79733b8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f796c10ce60 con 0x7f796c108910 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.808+0000 7f7971154700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 0x7f796c19f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.808+0000 7f7971154700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 0x7f796c19f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50826/0 (socket says 172.21.15.31:50826) 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.808+0000 7f7971154700 1 -- 172.21.15.31:0/892414046 learned_addr learned my addr 172.21.15.31:0/892414046 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.808+0000 7f7971154700 1 -- 172.21.15.31:0/892414046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f796c19fa40 con 0x7f796c108910 2024-04-25T15:52:17.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.808+0000 7f7971154700 1 --2- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 0x7f796c19f500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f79680092e0 tx=0x7f7968005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.809+0000 7f79627fc700 1 -- 172.21.15.31:0/892414046 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7968026070 con 0x7f796c108910 2024-04-25T15:52:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.809+0000 7f79627fc700 1 -- 172.21.15.31:0/892414046 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f796800b6f0 con 0x7f796c108910 2024-04-25T15:52:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.809+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f796c19fd30 con 0x7f796c108910 2024-04-25T15:52:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.809+0000 7f79627fc700 1 -- 172.21.15.31:0/892414046 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79680037f0 con 0x7f796c108910 2024-04-25T15:52:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.809+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f796c199860 con 0x7f796c108910 2024-04-25T15:52:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.810+0000 7f79627fc700 1 -- 172.21.15.31:0/892414046 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f7968029070 con 0x7f796c108910 2024-04-25T15:52:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.810+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f796c104020 con 0x7f796c108910 2024-04-25T15:52:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.810+0000 7f79627fc700 1 --2- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795803d610 0x7f795803fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.810+0000 7f79627fc700 1 -- 172.21.15.31:0/892414046 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f796801e070 con 0x7f796c108910 2024-04-25T15:52:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.810+0000 7f7970953700 1 --2- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795803d610 0x7f795803fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.810+0000 7f7970953700 1 --2- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795803d610 0x7f795803fb20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f795c006fd0 tx=0x7f795c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:17.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:17.815+0000 7f79627fc700 1 -- 172.21.15.31:0/892414046 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7968013070 con 0x7f796c108910 2024-04-25T15:52:18.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:18.026+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}) -- 0x7f796c19ff60 con 0x7f795803d610 2024-04-25T15:52:18.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:18 smithi031 ceph-mon[29140]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:18.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:18 smithi031 ceph-mon[29140]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:52:19.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:19.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:19.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:19.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:19.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:52:19.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:19.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:19.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T15:52:19.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: Deploying cephadm binary to smithi186 2024-04-25T15:52:19.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: Updating smithi063:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:19.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:19 smithi031 ceph-mon[29140]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:20.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:20 smithi031 ceph-mon[29140]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.338+0000 7f79627fc700 1 -- 172.21.15.31:0/892414046 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f796c19ff60 con 0x7f795803d610 2024-04-25T15:52:22.340 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-04-25T15:52:22.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795803d610 msgr2=0x7f795803fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 --2- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795803d610 0x7f795803fb20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f795c006fd0 tx=0x7f795c006e40 comp rx=0 tx=0).stop 2024-04-25T15:52:22.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 msgr2=0x7f796c19f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:22.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 --2- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 0x7f796c19f500 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f79680092e0 tx=0x7f7968005780 comp rx=0 tx=0).stop 2024-04-25T15:52:22.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 shutdown_connections 2024-04-25T15:52:22.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 --2- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795803d610 0x7f795803fb20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 --2- 172.21.15.31:0/892414046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f796c108910 0x7f796c19f500 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:22.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 >> 172.21.15.31:0/892414046 conn(0x7f796c0fe440 msgr2=0x7f796c1063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:22.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 shutdown_connections 2024-04-25T15:52:22.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:22.341+0000 7f79733b8700 1 -- 172.21.15.31:0/892414046 wait complete. 2024-04-25T15:52:22.924 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:22 smithi031 ceph-mon[29140]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:22.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:22 smithi031 ceph-mon[29140]: Updating smithi063:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T15:52:22.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:22.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:22.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:22.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:22.925 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:22.957 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-25T15:52:23.203 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:52:24.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:23 smithi031 ceph-mon[29140]: Added host smithi186 2024-04-25T15:52:24.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:23 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:24.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:23 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:24.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.765+0000 7fe14bb5a700 1 -- 172.21.15.31:0/3784732799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 msgr2=0x7fe1441025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.765+0000 7fe14bb5a700 1 --2- 172.21.15.31:0/3784732799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 0x7fe1441025e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe140009910 tx=0x7fe140009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.765+0000 7fe14bb5a700 1 -- 172.21.15.31:0/3784732799 shutdown_connections 2024-04-25T15:52:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.765+0000 7fe14bb5a700 1 --2- 172.21.15.31:0/3784732799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 0x7fe1441025e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.765+0000 7fe14bb5a700 1 -- 172.21.15.31:0/3784732799 >> 172.21.15.31:0/3784732799 conn(0x7fe144078c30 msgr2=0x7fe144079070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.765+0000 7fe14bb5a700 1 -- 172.21.15.31:0/3784732799 shutdown_connections 2024-04-25T15:52:24.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.765+0000 7fe14bb5a700 1 -- 172.21.15.31:0/3784732799 wait complete. 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.766+0000 7fe14bb5a700 1 Processor -- start 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.766+0000 7fe14bb5a700 1 -- start start 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe14bb5a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 0x7fe144104cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe14bb5a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe14410dfb0 con 0x7fe1441021d0 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe1498f6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 0x7fe144104cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe1498f6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 0x7fe144104cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60662/0 (socket says 172.21.15.31:60662) 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe1498f6700 1 -- 172.21.15.31:0/2025594774 learned_addr learned my addr 172.21.15.31:0/2025594774 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe1498f6700 1 -- 172.21.15.31:0/2025594774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe144103120 con 0x7fe1441021d0 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe1498f6700 1 --2- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 0x7fe144104cc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe140006010 tx=0x7fe140005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe13affd700 1 -- 172.21.15.31:0/2025594774 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe140026070 con 0x7fe1441021d0 2024-04-25T15:52:24.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe13affd700 1 -- 172.21.15.31:0/2025594774 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe14000bba0 con 0x7fe1441021d0 2024-04-25T15:52:24.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1441033b0 con 0x7fe1441021d0 2024-04-25T15:52:24.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.767+0000 7fe13affd700 1 -- 172.21.15.31:0/2025594774 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe140003910 con 0x7fe1441021d0 2024-04-25T15:52:24.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.768+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe144103870 con 0x7fe1441021d0 2024-04-25T15:52:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.769+0000 7fe13affd700 1 -- 172.21.15.31:0/2025594774 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fe140029070 con 0x7fe1441021d0 2024-04-25T15:52:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.769+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe144064cf0 con 0x7fe1441021d0 2024-04-25T15:52:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.769+0000 7fe13affd700 1 --2- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe13003d670 0x7fe13003fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.769+0000 7fe13affd700 1 -- 172.21.15.31:0/2025594774 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe14001e070 con 0x7fe1441021d0 2024-04-25T15:52:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.769+0000 7fe1490f5700 1 --2- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe13003d670 0x7fe13003fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:24.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.770+0000 7fe1490f5700 1 --2- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe13003d670 0x7fe13003fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe134006fd0 tx=0x7fe134006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:24.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.774+0000 7fe13affd700 1 -- 172.21.15.31:0/2025594774 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe140013070 con 0x7fe1441021d0 2024-04-25T15:52:24.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.988+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe144104760 con 0x7fe13003d670 2024-04-25T15:52:24.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.989+0000 7fe13affd700 1 -- 172.21.15.31:0/2025594774 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fe144104760 con 0x7fe13003d670 2024-04-25T15:52:24.990 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:52:24.990 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-04-25T15:52:24.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe13003d670 msgr2=0x7fe13003fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 --2- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe13003d670 0x7fe13003fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe134006fd0 tx=0x7fe134006e40 comp rx=0 tx=0).stop 2024-04-25T15:52:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 msgr2=0x7fe144104cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 --2- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 0x7fe144104cc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe140006010 tx=0x7fe140005780 comp rx=0 tx=0).stop 2024-04-25T15:52:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 shutdown_connections 2024-04-25T15:52:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 --2- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe13003d670 0x7fe13003fb80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 --2- 172.21.15.31:0/2025594774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe1441021d0 0x7fe144104cc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 >> 172.21.15.31:0/2025594774 conn(0x7fe144078c30 msgr2=0x7fe144105660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 shutdown_connections 2024-04-25T15:52:24.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:24.992+0000 7fe14bb5a700 1 -- 172.21.15.31:0/2025594774 wait complete. 2024-04-25T15:52:25.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:24 smithi031 ceph-mon[29140]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:25.702 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-25T15:52:25.702 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-25T15:52:25.945 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:52:26.256 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:25 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:26.256 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:25 smithi031 ceph-mon[29140]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:52:27.231 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:26 smithi031 ceph-mon[29140]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:27.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.479+0000 7f75ad328700 1 -- 172.21.15.31:0/2789355804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 msgr2=0x7f75a8108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:27.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.479+0000 7f75ad328700 1 --2- 172.21.15.31:0/2789355804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 0x7f75a8108d50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7598009910 tx=0x7f7598009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:27.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.479+0000 7f75ad328700 1 -- 172.21.15.31:0/2789355804 shutdown_connections 2024-04-25T15:52:27.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.479+0000 7f75ad328700 1 --2- 172.21.15.31:0/2789355804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 0x7f75a8108d50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.479+0000 7f75ad328700 1 -- 172.21.15.31:0/2789355804 >> 172.21.15.31:0/2789355804 conn(0x7f75a80fe490 msgr2=0x7f75a81008e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:27.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.479+0000 7f75ad328700 1 -- 172.21.15.31:0/2789355804 shutdown_connections 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.479+0000 7f75ad328700 1 -- 172.21.15.31:0/2789355804 wait complete. 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.479+0000 7f75ad328700 1 Processor -- start 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.480+0000 7f75ad328700 1 -- start start 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.480+0000 7f75ad328700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 0x7f75a819a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.480+0000 7f75ad328700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f75a810ce90 con 0x7f75a8108940 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.480+0000 7f75a6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 0x7f75a819a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.480+0000 7f75a6ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 0x7f75a819a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60674/0 (socket says 172.21.15.31:60674) 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.480+0000 7f75a6ffd700 1 -- 172.21.15.31:0/1473919837 learned_addr learned my addr 172.21.15.31:0/1473919837 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.480+0000 7f75a6ffd700 1 -- 172.21.15.31:0/1473919837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75a806c940 con 0x7f75a8108940 2024-04-25T15:52:27.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.481+0000 7f75a6ffd700 1 --2- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 0x7f75a819a650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f75980056d0 tx=0x7f75980057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.481+0000 7f758ffff700 1 -- 172.21.15.31:0/1473919837 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7598027070 con 0x7f75a8108940 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.481+0000 7f758ffff700 1 -- 172.21.15.31:0/1473919837 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f759800be10 con 0x7f75a8108940 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.481+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75a819ac50 con 0x7f75a8108940 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.481+0000 7f758ffff700 1 -- 172.21.15.31:0/1473919837 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7598022d20 con 0x7f75a8108940 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.481+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75a819b110 con 0x7f75a8108940 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.482+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75a8104050 con 0x7f75a8108940 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.482+0000 7f758ffff700 1 -- 172.21.15.31:0/1473919837 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f759802a070 con 0x7f75a8108940 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.482+0000 7f758ffff700 1 --2- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f759003d670 0x7f759003fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.482+0000 7f758ffff700 1 -- 172.21.15.31:0/1473919837 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f759801e030 con 0x7f75a8108940 2024-04-25T15:52:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.482+0000 7f75a67fc700 1 --2- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f759003d670 0x7f759003fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.483+0000 7f75a67fc700 1 --2- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f759003d670 0x7f759003fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f759c006fd0 tx=0x7f759c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.488+0000 7f758ffff700 1 -- 172.21.15.31:0/1473919837 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7598013070 con 0x7f75a8108940 2024-04-25T15:52:27.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.702+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f75a819b3f0 con 0x7f75a8108940 2024-04-25T15:52:27.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.962+0000 7f758ffff700 1 -- 172.21.15.31:0/1473919837 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f759802b3b0 con 0x7f75a8108940 2024-04-25T15:52:27.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f759003d670 msgr2=0x7f759003fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:27.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 --2- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f759003d670 0x7f759003fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f759c006fd0 tx=0x7f759c006e40 comp rx=0 tx=0).stop 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 msgr2=0x7f75a819a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 --2- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 0x7f75a819a650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f75980056d0 tx=0x7f75980057b0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 shutdown_connections 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 --2- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f759003d670 0x7f759003fb80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 --2- 172.21.15.31:0/1473919837 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75a8108940 0x7f75a819a650 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 >> 172.21.15.31:0/1473919837 conn(0x7f75a80fe490 msgr2=0x7f75a80ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 shutdown_connections 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:27.966+0000 7f75ad328700 1 -- 172.21.15.31:0/1473919837 wait complete. 2024-04-25T15:52:27.967 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2024-04-25T15:52:28.275 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:27 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1473919837' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T15:52:28.630 INFO:tasks.cephadm:Adding mon.a on smithi031 2024-04-25T15:52:28.630 INFO:tasks.cephadm:Adding mon.b on smithi063 2024-04-25T15:52:28.630 INFO:tasks.cephadm:Adding mon.c on smithi186 2024-04-25T15:52:28.631 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi031:172.21.15.31=a;smithi063:172.21.15.63=b;smithi186:172.21.15.186=c' 2024-04-25T15:52:28.864 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:29.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1473919837' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T15:52:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T15:52:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:52:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:30.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:29 smithi031 ceph-mon[29140]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-25T15:52:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:29 smithi031 ceph-mon[29140]: Updating smithi186:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:29 smithi031 ceph-mon[29140]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:29 smithi031 ceph-mon[29140]: Updating smithi186:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T15:52:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.509+0000 7f9a185f5700 1 -- 172.21.15.186:0/3624208851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 msgr2=0x7f9a10108e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.509+0000 7f9a185f5700 1 --2- 172.21.15.186:0/3624208851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 0x7f9a10108e40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9a00009910 tx=0x7f9a00009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.510+0000 7f9a185f5700 1 -- 172.21.15.186:0/3624208851 shutdown_connections 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.510+0000 7f9a185f5700 1 --2- 172.21.15.186:0/3624208851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 0x7f9a10108e40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.510+0000 7f9a185f5700 1 -- 172.21.15.186:0/3624208851 >> 172.21.15.186:0/3624208851 conn(0x7f9a100fdd00 msgr2=0x7f9a10100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:30.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.510+0000 7f9a185f5700 1 -- 172.21.15.186:0/3624208851 shutdown_connections 2024-04-25T15:52:30.512 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.510+0000 7f9a185f5700 1 -- 172.21.15.186:0/3624208851 wait complete. 2024-04-25T15:52:30.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.510+0000 7f9a185f5700 1 Processor -- start 2024-04-25T15:52:30.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.510+0000 7f9a185f5700 1 -- start start 2024-04-25T15:52:30.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.511+0000 7f9a185f5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 0x7f9a10103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.511+0000 7f9a185f5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1010e0d0 con 0x7f9a10108a30 2024-04-25T15:52:30.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.511+0000 7f9a16391700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 0x7f9a10103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.511+0000 7f9a16391700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 0x7f9a10103110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.186:41942/0 (socket says 172.21.15.186:41942) 2024-04-25T15:52:30.513 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.511+0000 7f9a16391700 1 -- 172.21.15.186:0/2684972242 learned_addr learned my addr 172.21.15.186:0/2684972242 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:52:30.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.511+0000 7f9a16391700 1 -- 172.21.15.186:0/2684972242 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a10104f60 con 0x7f9a10108a30 2024-04-25T15:52:30.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.512+0000 7f9a16391700 1 --2- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 0x7f9a10103110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9a000056d0 tx=0x7f9a000057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.512+0000 7f9a077fe700 1 -- 172.21.15.186:0/2684972242 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a00027070 con 0x7f9a10108a30 2024-04-25T15:52:30.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.512+0000 7f9a077fe700 1 -- 172.21.15.186:0/2684972242 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a0000be10 con 0x7f9a10108a30 2024-04-25T15:52:30.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.512+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a10103650 con 0x7f9a10108a30 2024-04-25T15:52:30.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.512+0000 7f9a077fe700 1 -- 172.21.15.186:0/2684972242 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9a00022d20 con 0x7f9a10108a30 2024-04-25T15:52:30.514 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.512+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a10103a50 con 0x7f9a10108a30 2024-04-25T15:52:30.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.513+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a10064cf0 con 0x7f9a10108a30 2024-04-25T15:52:30.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.513+0000 7f9a077fe700 1 -- 172.21.15.186:0/2684972242 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f9a0002a070 con 0x7f9a10108a30 2024-04-25T15:52:30.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.513+0000 7f9a077fe700 1 --2- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99fc03d610 0x7f99fc03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:30.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.513+0000 7f9a077fe700 1 -- 172.21.15.186:0/2684972242 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f9a0001e030 con 0x7f9a10108a30 2024-04-25T15:52:30.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.514+0000 7f9a15b90700 1 --2- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99fc03d610 0x7f99fc03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:30.515 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.514+0000 7f9a15b90700 1 --2- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99fc03d610 0x7f99fc03fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c006fd0 tx=0x7f9a0c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:30.520 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.519+0000 7f9a077fe700 1 -- 172.21.15.186:0/2684972242 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a00013070 con 0x7f9a10108a30 2024-04-25T15:52:30.732 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.730+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi063:172.21.15.63=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}) -- 0x7f9a10103cd0 con 0x7f99fc03d610 2024-04-25T15:52:30.739 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.739+0000 7f9a077fe700 1 -- 172.21.15.186:0/2684972242 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9a10103cd0 con 0x7f99fc03d610 2024-04-25T15:52:30.739 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mon update... 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99fc03d610 msgr2=0x7f99fc03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 --2- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99fc03d610 0x7f99fc03fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9a0c006fd0 tx=0x7f9a0c006e40 comp rx=0 tx=0).stop 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 msgr2=0x7f9a10103110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 --2- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 0x7f9a10103110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9a000056d0 tx=0x7f9a000057b0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 shutdown_connections 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 --2- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99fc03d610 0x7f99fc03fb20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 --2- 172.21.15.186:0/2684972242 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a10108a30 0x7f9a10103110 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 >> 172.21.15.186:0/2684972242 conn(0x7f9a100fdd00 msgr2=0x7f9a10105650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 shutdown_connections 2024-04-25T15:52:30.742 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:30.740+0000 7f9a185f5700 1 -- 172.21.15.186:0/2684972242 wait complete. 2024-04-25T15:52:31.312 DEBUG:teuthology.orchestra.run.smithi063:mon.b> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.b.service 2024-04-25T15:52:31.315 DEBUG:teuthology.orchestra.run.smithi186:mon.c> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.c.service 2024-04-25T15:52:31.317 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T15:52:31.317 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T15:52:31.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:30 smithi031 ceph-mon[29140]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:31.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:31.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:31.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:31.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:31.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:31.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:31.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:32.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:31 smithi031 ceph-mon[29140]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi063:172.21.15.63=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:52:32.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:31 smithi031 ceph-mon[29140]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi063:172.21.15.63=b;smithi186:172.21.15.186=c;count:3 2024-04-25T15:52:32.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:31 smithi031 ceph-mon[29140]: Deploying daemon mon.c on smithi186 2024-04-25T15:52:32.528 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:33.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:32 smithi031 ceph-mon[29140]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:35.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.323+0000 7f79d0731700 1 -- 172.21.15.186:0/1701474315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 msgr2=0x7f79c8107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.323+0000 7f79d0731700 1 --2- 172.21.15.186:0/1701474315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 0x7f79c8107d70 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f79c4009910 tx=0x7f79c4009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:35.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.323+0000 7f79d0731700 1 -- 172.21.15.186:0/1701474315 shutdown_connections 2024-04-25T15:52:35.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.323+0000 7f79d0731700 1 --2- 172.21.15.186:0/1701474315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 0x7f79c8107d70 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.323+0000 7f79d0731700 1 -- 172.21.15.186:0/1701474315 >> 172.21.15.186:0/1701474315 conn(0x7f79c80fdd00 msgr2=0x7f79c8100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:35.325 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.323+0000 7f79d0731700 1 -- 172.21.15.186:0/1701474315 shutdown_connections 2024-04-25T15:52:35.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.323+0000 7f79d0731700 1 -- 172.21.15.186:0/1701474315 wait complete. 2024-04-25T15:52:35.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.324+0000 7f79d0731700 1 Processor -- start 2024-04-25T15:52:35.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.324+0000 7f79d0731700 1 -- start start 2024-04-25T15:52:35.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.324+0000 7f79d0731700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 0x7f79c8198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.326 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.324+0000 7f79d0731700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f79c810cef0 con 0x7f79c8107960 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.324+0000 7f79ce4cd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 0x7f79c8198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.324+0000 7f79ce4cd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 0x7f79c8198710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.186:41962/0 (socket says 172.21.15.186:41962) 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.324+0000 7f79ce4cd700 1 -- 172.21.15.186:0/2655456162 learned_addr learned my addr 172.21.15.186:0/2655456162 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.325+0000 7f79ce4cd700 1 -- 172.21.15.186:0/2655456162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c8198c50 con 0x7f79c8107960 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.325+0000 7f79ce4cd700 1 --2- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 0x7f79c8198710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f79c400b5c0 tx=0x7f79c4005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.325+0000 7f79bf7fe700 1 -- 172.21.15.186:0/2655456162 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c4026070 con 0x7f79c8107960 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.325+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c8198ee0 con 0x7f79c8107960 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.325+0000 7f79bf7fe700 1 -- 172.21.15.186:0/2655456162 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79c400b750 con 0x7f79c8107960 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.325+0000 7f79bf7fe700 1 -- 172.21.15.186:0/2655456162 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79c40037d0 con 0x7f79c8107960 2024-04-25T15:52:35.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.325+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c819cf40 con 0x7f79c8107960 2024-04-25T15:52:35.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.326+0000 7f79bf7fe700 1 -- 172.21.15.186:0/2655456162 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f79c4029070 con 0x7f79c8107960 2024-04-25T15:52:35.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.326+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c8104070 con 0x7f79c8107960 2024-04-25T15:52:35.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.326+0000 7f79bf7fe700 1 --2- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f79b403d610 0x7f79b403fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:35.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.326+0000 7f79bf7fe700 1 -- 172.21.15.186:0/2655456162 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f79c401e070 con 0x7f79c8107960 2024-04-25T15:52:35.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.327+0000 7f79cdccc700 1 --2- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f79b403d610 0x7f79b403fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:35.329 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.327+0000 7f79cdccc700 1 --2- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f79b403d610 0x7f79b403fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f79b8006fd0 tx=0x7f79b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:35.334 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.332+0000 7f79bf7fe700 1 -- 172.21.15.186:0/2655456162 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79c4013070 con 0x7f79c8107960 2024-04-25T15:52:35.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:34 smithi031 ceph-mon[29140]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:35.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.633+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f79c8199a70 con 0x7f79c8107960 2024-04-25T15:52:35.635 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.634+0000 7f79bf7fe700 1 -- 172.21.15.186:0/2655456162 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f79c400e3e0 con 0x7f79c8107960 2024-04-25T15:52:35.636 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:52:35.636 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":1,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","modified":"2024-04-25T15:50:40.276173Z","created":"2024-04-25T15:50:40.276173Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-25T15:52:35.637 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f79b403d610 msgr2=0x7f79b403fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 --2- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f79b403d610 0x7f79b403fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f79b8006fd0 tx=0x7f79b8006e40 comp rx=0 tx=0).stop 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 msgr2=0x7f79c8198710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 --2- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 0x7f79c8198710 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f79c400b5c0 tx=0x7f79c4005780 comp rx=0 tx=0).stop 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 shutdown_connections 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 --2- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f79b403d610 0x7f79b403fb20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 --2- 172.21.15.186:0/2655456162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79c8107960 0x7f79c8198710 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 >> 172.21.15.186:0/2655456162 conn(0x7f79c80fdd00 msgr2=0x7f79c8104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 shutdown_connections 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:35.636+0000 7f79d0731700 1 -- 172.21.15.186:0/2655456162 wait complete. 2024-04-25T15:52:35.638 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 1 2024-04-25T15:52:36.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:35 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/2655456162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T15:52:37.408 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T15:52:37.408 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T15:52:37.653 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:52:41.362 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:41 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:41.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:41.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:52:41.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:41.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: mon.a calling monitor election 2024-04-25T15:52:41.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: mon.c calling monitor election 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: monmap epoch 2 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: last_changed 2024-04-25T15:52:35.991291+0000 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: created 2024-04-25T15:50:40.276173+0000 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: min_mon_release 19 (squid) 2024-04-25T15:52:41.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: election_strategy: 1 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: fsmap 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: mgrmap e13: x(active, since 51s) 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: overall HEALTH_OK 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:41.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:41.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.993+0000 7fe89e63f700 1 -- 172.21.15.186:0/54903977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8981079d0 msgr2=0x7fe884005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:41.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.993+0000 7fe89e63f700 1 --2- 172.21.15.186:0/54903977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8981079d0 0x7fe884005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe888009bf0 tx=0x7fe888009f00 comp rx=0 tx=0).stop 2024-04-25T15:52:41.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.993+0000 7fe89e63f700 1 -- 172.21.15.186:0/54903977 shutdown_connections 2024-04-25T15:52:41.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.993+0000 7fe89e63f700 1 --2- 172.21.15.186:0/54903977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe8981079d0 0x7fe884005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.993+0000 7fe89e63f700 1 -- 172.21.15.186:0/54903977 >> 172.21.15.186:0/54903977 conn(0x7fe8980fddf0 msgr2=0x7fe898100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:41.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.993+0000 7fe89e63f700 1 -- 172.21.15.186:0/54903977 shutdown_connections 2024-04-25T15:52:41.994 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.993+0000 7fe89e63f700 1 -- 172.21.15.186:0/54903977 wait complete. 2024-04-25T15:52:41.996 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.994+0000 7fe89e63f700 1 Processor -- start 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.994+0000 7fe89e63f700 1 -- start start 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.994+0000 7fe89e63f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 0x7fe89819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.994+0000 7fe89e63f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe89819eb00 0x7fe898198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe89e63f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe89810aa70 con 0x7fe89819eb00 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe89e63f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe89810a8f0 con 0x7fe8981079d0 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe897fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 0x7fe89819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe897fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 0x7fe89819e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:39798/0 (socket says 172.21.15.186:39798) 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe897fff700 1 -- 172.21.15.186:0/3548500231 learned_addr learned my addr 172.21.15.186:0/3548500231 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe88f7fe700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe89819eb00 0x7fe898198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe897fff700 1 -- 172.21.15.186:0/3548500231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 msgr2=0x7fe89819e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe897fff700 1 -- 172.21.15.186:0/3548500231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 msgr2=0x7fe89819e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe897fff700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 0x7fe89819e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:52:41.997 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe897fff700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 0x7fe89819e5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:52:41.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe88f7fe700 1 -- 172.21.15.186:0/3548500231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 msgr2=0x7fe89819e5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:52:41.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe88f7fe700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 0x7fe89819e5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:41.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.995+0000 7fe88f7fe700 1 -- 172.21.15.186:0/3548500231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe89819f0a0 con 0x7fe89819eb00 2024-04-25T15:52:41.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.996+0000 7fe88f7fe700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe89819eb00 0x7fe898198750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe88000b4d0 tx=0x7fe88000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:41.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.996+0000 7fe895ffb700 1 -- 172.21.15.186:0/3548500231 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe880011630 con 0x7fe89819eb00 2024-04-25T15:52:41.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.996+0000 7fe895ffb700 1 -- 172.21.15.186:0/3548500231 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe880011c80 con 0x7fe89819eb00 2024-04-25T15:52:41.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.996+0000 7fe895ffb700 1 -- 172.21.15.186:0/3548500231 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fe880010da0 con 0x7fe89819eb00 2024-04-25T15:52:41.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.996+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe898198d50 con 0x7fe89819eb00 2024-04-25T15:52:41.998 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.996+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8981992e0 con 0x7fe89819eb00 2024-04-25T15:52:41.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.997+0000 7fe895ffb700 1 -- 172.21.15.186:0/3548500231 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fe880018020 con 0x7fe89819eb00 2024-04-25T15:52:41.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.997+0000 7fe895ffb700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe87803d3b0 0x7fe87803f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:41.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.997+0000 7fe895ffb700 1 -- 172.21.15.186:0/3548500231 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fe880050fc0 con 0x7fe89819eb00 2024-04-25T15:52:41.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.997+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8981995f0 con 0x7fe89819eb00 2024-04-25T15:52:41.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.998+0000 7fe897fff700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe87803d3b0 0x7fe87803f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:41.999 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:41.998+0000 7fe897fff700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe87803d3b0 0x7fe87803f8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe888007950 tx=0x7fe888009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:42.004 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.004+0000 7fe895ffb700 1 -- 172.21.15.186:0/3548500231 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe88001f020 con 0x7fe89819eb00 2024-04-25T15:52:42.289 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.287+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fe898064cf0 con 0x7fe89819eb00 2024-04-25T15:52:42.300 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:42 smithi186 ceph-mon[33258]: Deploying daemon mon.b on smithi063 2024-04-25T15:52:42.301 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:42 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:42.301 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.288+0000 7fe895ffb700 1 -- 172.21.15.186:0/3548500231 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fe8800187b0 con 0x7fe89819eb00 2024-04-25T15:52:42.301 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:52:42.302 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":2,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","modified":"2024-04-25T15:52:35.991291Z","created":"2024-04-25T15:50:40.276173Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.290+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe87803d3b0 msgr2=0x7fe87803f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.290+0000 7fe89e63f700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe87803d3b0 0x7fe87803f8c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe888007950 tx=0x7fe888009250 comp rx=0 tx=0).stop 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.290+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe89819eb00 msgr2=0x7fe898198750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.290+0000 7fe89e63f700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe89819eb00 0x7fe898198750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe88000b4d0 tx=0x7fe88000b890 comp rx=0 tx=0).stop 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.290+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 shutdown_connections 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.290+0000 7fe89e63f700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe87803d3b0 0x7fe87803f8c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.290+0000 7fe89e63f700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8981079d0 0x7fe89819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.290+0000 7fe89e63f700 1 --2- 172.21.15.186:0/3548500231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe89819eb00 0x7fe898198750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.291+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 >> 172.21.15.186:0/3548500231 conn(0x7fe8980fddf0 msgr2=0x7fe898106170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.291+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 shutdown_connections 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:42.291+0000 7fe89e63f700 1 -- 172.21.15.186:0/3548500231 wait complete. 2024-04-25T15:52:42.303 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 2 2024-04-25T15:52:42.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:42 smithi031 ceph-mon[29140]: Deploying daemon mon.b on smithi063 2024-04-25T15:52:42.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:42 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:43.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:43 smithi186 ceph-mon[33258]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:43 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/3548500231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T15:52:43.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:43 smithi031 ceph-mon[29140]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:43.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:43 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/3548500231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T15:52:44.012 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-25T15:52:44.013 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-25T15:52:44.284 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:52:44.385 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:44 smithi186 ceph-mon[33258]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:44.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:44 smithi031 ceph-mon[29140]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:44.634 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 podman[32395]: 2024-04-25 15:52:44.450609604 +0000 UTC m=+0.017580665 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:52:45.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 podman[32395]: 2024-04-25 15:52:44.706379511 +0000 UTC m=+0.273350565 container init c5ed416e9877ab5005402e48655d818e709866be333382ec51d4331ceeaefe59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-25T15:52:45.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 podman[32395]: 2024-04-25 15:52:44.709418045 +0000 UTC m=+0.276389099 container start c5ed416e9877ab5005402e48655d818e709866be333382ec51d4331ceeaefe59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T15:52:45.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: set uid:gid to 167:167 (ceph:ceph) 2024-04-25T15:52:45.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: ceph version 19.0.0-3291-g994e2f20 (994e2f20102a20fa9a27966c14b625cc01884df8) squid (dev), process ceph-mon, pid 2 2024-04-25T15:52:45.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pidfile_write: ignore empty --pid-file 2024-04-25T15:52:45.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: load: jerasure load: lrc 2024-04-25T15:52:45.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: RocksDB version: 7.9.2 2024-04-25T15:52:45.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Git sha 0 2024-04-25T15:52:45.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Compile date 2024-04-25 02:57:34 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: DB SUMMARY 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: DB Session ID: KDUN5CSKB39OITV6R1VX 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: CURRENT file: CURRENT 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: IDENTITY file: IDENTITY 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.error_if_exists: 0 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.create_if_missing: 0 2024-04-25T15:52:45.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.paranoid_checks: 1 2024-04-25T15:52:45.122 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-25T15:52:45.122 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-25T15:52:45.122 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-25T15:52:45.122 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.env: 0x56525a741500 2024-04-25T15:52:45.122 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.fs: PosixFileSystem 2024-04-25T15:52:45.122 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.info_log: 0x56525b0adfc0 2024-04-25T15:52:45.122 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_file_opening_threads: 16 2024-04-25T15:52:45.122 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.statistics: (nil) 2024-04-25T15:52:45.122 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.use_fsync: 0 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_log_file_size: 0 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.keep_log_file_num: 1000 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.recycle_log_file_num: 0 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.allow_fallocate: 1 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.allow_mmap_reads: 0 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.allow_mmap_writes: 0 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.use_direct_reads: 0 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.create_missing_column_families: 0 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.db_log_dir: 2024-04-25T15:52:45.123 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.wal_dir: 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.advise_random_on_open: 1 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.db_write_buffer_size: 0 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.write_buffer_manager: 0x56525b07e8c0 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-25T15:52:45.124 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.rate_limiter: (nil) 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.wal_recovery_mode: 2 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.enable_thread_tracking: 0 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.enable_pipelined_write: 0 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.unordered_write: 0 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-25T15:52:45.125 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-25T15:52:45.126 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.row_cache: None 2024-04-25T15:52:45.126 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.wal_filter: None 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.allow_ingest_behind: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.two_write_queues: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.manual_wal_flush: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.wal_compression: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.atomic_flush: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.log_readahead_size: 0 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-25T15:52:45.128 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.best_efforts_recovery: 0 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.allow_data_in_errors: 0 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.db_host_id: __hostname__ 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_background_jobs: 2 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_background_compactions: -1 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_subcompactions: 1 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-25T15:52:45.129 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_total_wal_size: 0 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_open_files: -1 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bytes_per_sync: 0 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_readahead_size: 0 2024-04-25T15:52:45.130 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_background_flushes: -1 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Compression algorithms supported: 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: kZSTD supported: 0 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: kXpressCompression supported: 0 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: kLZ4HCCompression supported: 1 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: kZlibCompression supported: 1 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: kSnappyCompression supported: 1 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: kLZ4Compression supported: 1 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: kBZip2Compression supported: 0 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-25T15:52:45.131 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.merge_operator: 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_filter: None 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_filter_factory: None 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.sst_partitioner_factory: None 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56525b1b41a0) 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout: cache_index_and_filter_blocks: 1 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-25T15:52:45.132 INFO:journalctl@ceph.mon.b.smithi063.stdout: pin_top_level_index_and_filter: 1 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: index_type: 0 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: data_block_index_type: 0 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: index_shortening: 1 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: checksum: 4 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: no_block_cache: 0 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: block_cache: 0x56525b0b11f0 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: block_cache_name: BinnedLRUCache 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: block_cache_options: 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: capacity : 536870912 2024-04-25T15:52:45.133 INFO:journalctl@ceph.mon.b.smithi063.stdout: num_shard_bits : 4 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: strict_capacity_limit : 0 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: high_pri_pool_ratio: 0.000 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: block_cache_compressed: (nil) 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: persistent_cache: (nil) 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: block_size: 4096 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: block_size_deviation: 10 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: block_restart_interval: 16 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: index_block_restart_interval: 1 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: metadata_block_size: 4096 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: partition_filters: 0 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: use_delta_encoding: 1 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: filter_policy: bloomfilter 2024-04-25T15:52:45.134 INFO:journalctl@ceph.mon.b.smithi063.stdout: whole_key_filtering: 1 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout: verify_compression: 0 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout: read_amp_bytes_per_bit: 0 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout: format_version: 5 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout: enable_index_compression: 1 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout: block_align: 0 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout: max_auto_readahead_size: 262144 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout: prepopulate_block_cache: 0 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout: initial_auto_readahead_size: 8192 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout: num_file_reads_for_auto_readahead: 2 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.write_buffer_size: 33554432 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_write_buffer_number: 2 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression: NoCompression 2024-04-25T15:52:45.135 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression: Disabled 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.prefix_extractor: nullptr 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.num_levels: 7 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-25T15:52:45.136 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression_opts.level: 32767 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression_opts.strategy: 0 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-25T15:52:45.137 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression_opts.enabled: false 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.target_file_size_base: 67108864 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-25T15:52:45.138 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.arena_block_size: 1048576 2024-04-25T15:52:45.139 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.disable_auto_compactions: 0 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-25T15:52:45.140 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.inplace_update_support: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.bloom_locality: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.max_successive_merges: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.paranoid_file_checks: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.force_consistency_checks: 1 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.report_bg_io_stats: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.ttl: 2592000 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-25T15:52:45.141 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.enable_blob_files: false 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.min_blob_size: 0 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.blob_file_size: 268435456 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.blob_file_starting_level: 0 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-25T15:52:45.142 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: b670bd00-e1ab-423e-a4e8-c7e14b9986a7 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714060364750271, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714060364750695, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714060364, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "b670bd00-e1ab-423e-a4e8-c7e14b9986a7", "db_session_id": "KDUN5CSKB39OITV6R1VX", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714060364750778, "job": 1, "event": "recovery_finished"} 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56525b095c00 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: DB pointer 0x56525fe88000 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-25T15:52:45.143 INFO:journalctl@ceph.mon.b.smithi063.stdout: ** DB Stats ** 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: ** Compaction Stats [default] ** 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-25T15:52:45.144 INFO:journalctl@ceph.mon.b.smithi063.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: ** Compaction Stats [default] ** 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-25T15:52:45.145 INFO:journalctl@ceph.mon.b.smithi063.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: Block cache BinnedLRUCache@0x56525b0b11f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-25T15:52:45.146 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: using public_addr v2:172.21.15.63:0/0 -> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: starting mon.b rank -1 at public addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] at bind addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(???) e0 preinit fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).mds e1 new map 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout: e1 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout: legacy client fscid: -1 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout: 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout: No filesystems configured 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-25T15:52:45.147 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:52:45.148 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Deploying cephadm binary to smithi186 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Updating smithi063:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.149 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Updating smithi063:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Added host smithi186 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.150 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1473919837' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1473919837' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-25T15:52:45.151 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Updating smithi186:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Updating smithi186:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:45.152 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi063:172.21.15.63=b;smithi186:172.21.15.186=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi063:172.21.15.63=b;smithi186:172.21.15.186=c;count:3 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Deploying daemon mon.c on smithi186 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/2655456162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.a calling monitor election 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:45.153 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.c calling monitor election 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: monmap epoch 2 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: last_changed 2024-04-25T15:52:35.991291+0000 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: created 2024-04-25T15:50:40.276173+0000 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: min_mon_release 19 (squid) 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: election_strategy: 1 2024-04-25T15:52:45.154 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: fsmap 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mgrmap e13: x(active, since 51s) 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: overall HEALTH_OK 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: Deploying daemon mon.b on smithi063 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:45.155 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.156 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/3548500231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T15:52:45.156 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:45.156 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 ceph-mon[32413]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-25T15:52:45.156 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 bash[32395]: c5ed416e9877ab5005402e48655d818e709866be333382ec51d4331ceeaefe59 2024-04-25T15:52:45.156 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:44 smithi063 systemd[1]: Started Ceph mon.b for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:52:50.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:52:50.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: mon.a calling monitor election 2024-04-25T15:52:50.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: mon.c calling monitor election 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: mon.b calling monitor election 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: monmap epoch 3 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: last_changed 2024-04-25T15:52:44.781122+0000 2024-04-25T15:52:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: created 2024-04-25T15:50:40.276173+0000 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: min_mon_release 19 (squid) 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: election_strategy: 1 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: 2: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.b 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: fsmap 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: mgrmap e13: x(active, since 60s) 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: overall HEALTH_OK 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:52:50.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: mon.a calling monitor election 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: mon.c calling monitor election 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: mon.b calling monitor election 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: monmap epoch 3 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:52:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: last_changed 2024-04-25T15:52:44.781122+0000 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: created 2024-04-25T15:50:40.276173+0000 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: min_mon_release 19 (squid) 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: election_strategy: 1 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: 2: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.b 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: fsmap 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: mgrmap e13: x(active, since 60s) 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: overall HEALTH_OK 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.158+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3571231853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5e40035c0 msgr2=0x7fd5e4005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.158+0000 7fd5ffc56700 1 --2- 172.21.15.186:0/3571231853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5e40035c0 0x7fd5e4005aa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd5f806b3a0 tx=0x7fd5f400fe60 comp rx=0 tx=0).stop 2024-04-25T15:52:50.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.159+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3571231853 shutdown_connections 2024-04-25T15:52:50.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.159+0000 7fd5ffc56700 1 --2- 172.21.15.186:0/3571231853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5e40035c0 0x7fd5e4005aa0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.159+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3571231853 >> 172.21.15.186:0/3571231853 conn(0x7fd5f8077ce0 msgr2=0x7fd5f8078120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:50.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.159+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3571231853 shutdown_connections 2024-04-25T15:52:50.159 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.159+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3571231853 wait complete. 2024-04-25T15:52:50.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.160+0000 7fd5ffc56700 1 Processor -- start 2024-04-25T15:52:50.160 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.160+0000 7fd5ffc56700 1 -- start start 2024-04-25T15:52:50.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.160+0000 7fd5ffc56700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 0x7fd5f819aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.160+0000 7fd5ffc56700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f819b0e0 0x7fd5f819b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.160+0000 7fd5ffc56700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5f81a3c10 0x7fd5f81a6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5ffc56700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f810ab80 con 0x7fd5f819b0e0 2024-04-25T15:52:50.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5ffc56700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f810aa00 con 0x7fd5f81a3c10 2024-04-25T15:52:50.162 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5ffc56700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd5f810ad00 con 0x7fd5e40035c0 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd9f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 0x7fd5f819aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd9f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 0x7fd5f819aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37666/0 (socket says 172.21.15.186:37666) 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd9f2700 1 -- 172.21.15.186:0/3548533277 learned_addr learned my addr 172.21.15.186:0/3548533277 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd9f2700 1 -- 172.21.15.186:0/3548533277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 msgr2=0x7fd5f819aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd9f2700 1 -- 172.21.15.186:0/3548533277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 msgr2=0x7fd5f819aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd9f2700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 0x7fd5f819aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd1f1700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f819b0e0 0x7fd5f819b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd9f2700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 0x7fd5f819aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fe1f3700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5f81a3c10 0x7fd5f81a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd1f1700 1 -- 172.21.15.186:0/3548533277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 msgr2=0x7fd5f819aba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd1f1700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 0x7fd5f819aba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd1f1700 1 -- 172.21.15.186:0/3548533277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5f81a3c10 msgr2=0x7fd5f81a6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd1f1700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5f81a3c10 0x7fd5f81a6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.163 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd1f1700 1 -- 172.21.15.186:0/3548533277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5f81a67c0 con 0x7fd5f819b0e0 2024-04-25T15:52:50.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.161+0000 7fd5fd1f1700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f819b0e0 0x7fd5f819b4f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800b550 tx=0x7fd5e800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.162+0000 7fd5eeffd700 1 -- 172.21.15.186:0/3548533277 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8004230 con 0x7fd5f819b0e0 2024-04-25T15:52:50.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.162+0000 7fd5eeffd700 1 -- 172.21.15.186:0/3548533277 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd5e80043d0 con 0x7fd5f819b0e0 2024-04-25T15:52:50.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.162+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5f81a6ab0 con 0x7fd5f819b0e0 2024-04-25T15:52:50.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.162+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5f81a6ff0 con 0x7fd5f819b0e0 2024-04-25T15:52:50.164 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.162+0000 7fd5eeffd700 1 -- 172.21.15.186:0/3548533277 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8011b90 con 0x7fd5f819b0e0 2024-04-25T15:52:50.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.163+0000 7fd5eeffd700 1 -- 172.21.15.186:0/3548533277 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fd5e801a400 con 0x7fd5f819b0e0 2024-04-25T15:52:50.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.163+0000 7fd5eeffd700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5dc03daa0 0x7fd5dc03ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:50.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.163+0000 7fd5eeffd700 1 -- 172.21.15.186:0/3548533277 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd5e80521d0 con 0x7fd5f819b0e0 2024-04-25T15:52:50.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.163+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f8064cf0 con 0x7fd5f819b0e0 2024-04-25T15:52:50.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.163+0000 7fd5fd9f2700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5dc03daa0 0x7fd5dc03ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:50.165 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.164+0000 7fd5fd9f2700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5dc03daa0 0x7fd5dc03ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400fa10 tx=0x7fd5f400f500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:50.169 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.169+0000 7fd5eeffd700 1 -- 172.21.15.186:0/3548533277 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd5e8020740 con 0x7fd5f819b0e0 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: mon.a calling monitor election 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: mon.c calling monitor election 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: mon.b calling monitor election 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: monmap epoch 3 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: last_changed 2024-04-25T15:52:44.781122+0000 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: created 2024-04-25T15:50:40.276173+0000 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: min_mon_release 19 (squid) 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: election_strategy: 1 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: 0: [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon.a 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: 1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.c 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: 2: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.b 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: fsmap 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: osdmap e4: 0 total, 0 up, 0 in 2024-04-25T15:52:50.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: mgrmap e13: x(active, since 60s) 2024-04-25T15:52:50.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: overall HEALTH_OK 2024-04-25T15:52:50.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.460+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fd5f806b990 con 0x7fd5f819b0e0 2024-04-25T15:52:50.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.460+0000 7fd5eeffd700 1 -- 172.21.15.186:0/3548533277 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1281 (secure 0 0 0) 0x7fd5e801a830 con 0x7fd5f819b0e0 2024-04-25T15:52:50.461 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-04-25T15:52:50.461 INFO:teuthology.orchestra.run.smithi186.stdout:{"epoch":3,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","modified":"2024-04-25T15:52:44.781122Z","created":"2024-04-25T15:50:40.276173Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-04-25T15:52:50.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5dc03daa0 msgr2=0x7fd5dc03ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5dc03daa0 0x7fd5dc03ffb0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd5f400fa10 tx=0x7fd5f400f500 comp rx=0 tx=0).stop 2024-04-25T15:52:50.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f819b0e0 msgr2=0x7fd5f819b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f819b0e0 0x7fd5f819b4f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd5e800b550 tx=0x7fd5e800b910 comp rx=0 tx=0).stop 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 shutdown_connections 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5dc03daa0 0x7fd5dc03ffb0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd5f81a3c10 0x7fd5f81a6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5e40035c0 0x7fd5f819aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 --2- 172.21.15.186:0/3548533277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd5f819b0e0 0x7fd5f819b4f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 >> 172.21.15.186:0/3548533277 conn(0x7fd5f8077ce0 msgr2=0x7fd5f81000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 shutdown_connections 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:50.463+0000 7fd5ffc56700 1 -- 172.21.15.186:0/3548533277 wait complete. 2024-04-25T15:52:50.464 INFO:teuthology.orchestra.run.smithi186.stderr:dumped monmap epoch 3 2024-04-25T15:52:51.064 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:50 smithi186 ceph-mon[33258]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:51.065 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:51.065 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:50 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/3548533277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T15:52:51.065 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:51.092 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-25T15:52:51.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-25T15:52:51.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:50 smithi063 ceph-mon[32413]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:51.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:51.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:50 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/3548533277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T15:52:51.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:51.120 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:50 smithi031 ceph-mon[29140]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:51.120 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:51.120 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:50 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/3548533277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-25T15:52:51.120 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:52:51.349 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:52:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T15:52:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T15:52:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:52 smithi063 ceph-mon[32413]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-25T15:52:52.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:52.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.629 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:52.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:52.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T15:52:52.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T15:52:52.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:52 smithi031 ceph-mon[29140]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-25T15:52:52.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:52.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:52.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: Updating smithi031:/etc/ceph/ceph.conf 2024-04-25T15:52:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: Updating smithi063:/etc/ceph/ceph.conf 2024-04-25T15:52:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:52 smithi186 ceph-mon[33258]: Updating smithi186:/etc/ceph/ceph.conf 2024-04-25T15:52:53.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.060+0000 7ff7b3a38700 1 -- 172.21.15.31:0/4213955677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac102310 msgr2=0x7ff7ac102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.060+0000 7ff7b3a38700 1 --2- 172.21.15.31:0/4213955677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac102310 0x7ff7ac102720 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8009910 tx=0x7ff7a8009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:53.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.060+0000 7ff7b3a38700 1 -- 172.21.15.31:0/4213955677 shutdown_connections 2024-04-25T15:52:53.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.060+0000 7ff7b3a38700 1 --2- 172.21.15.31:0/4213955677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac102310 0x7ff7ac102720 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.060+0000 7ff7b3a38700 1 -- 172.21.15.31:0/4213955677 >> 172.21.15.31:0/4213955677 conn(0x7ff7ac0fdd00 msgr2=0x7ff7ac100120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:53.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.060+0000 7ff7b3a38700 1 -- 172.21.15.31:0/4213955677 shutdown_connections 2024-04-25T15:52:53.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.060+0000 7ff7b3a38700 1 -- 172.21.15.31:0/4213955677 wait complete. 2024-04-25T15:52:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.061+0000 7ff7b3a38700 1 Processor -- start 2024-04-25T15:52:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.061+0000 7ff7b3a38700 1 -- start start 2024-04-25T15:52:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b3a38700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ac102310 0x7ff7ac19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b3a38700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7ac19eb80 0x7ff7ac198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b3a38700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac198da0 0x7ff7ac1991b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b3a38700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac10c600 con 0x7ff7ac198da0 2024-04-25T15:52:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b3a38700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac10c480 con 0x7ff7ac19eb80 2024-04-25T15:52:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b3a38700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7ac10c780 con 0x7ff7ac102310 2024-04-25T15:52:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b1fd5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac198da0 0x7ff7ac1991b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b1fd5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac198da0 0x7ff7ac1991b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50836/0 (socket says 172.21.15.31:50836) 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b1fd5700 1 -- 172.21.15.31:0/2126721338 learned_addr learned my addr 172.21.15.31:0/2126721338 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b17d4700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ac102310 0x7ff7ac19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b0fd3700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7ac19eb80 0x7ff7ac198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b1fd5700 1 -- 172.21.15.31:0/2126721338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ac102310 msgr2=0x7ff7ac19e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b1fd5700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ac102310 0x7ff7ac19e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b1fd5700 1 -- 172.21.15.31:0/2126721338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7ac19eb80 msgr2=0x7ff7ac198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b1fd5700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7ac19eb80 0x7ff7ac198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b1fd5700 1 -- 172.21.15.31:0/2126721338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7ac1a54d0 con 0x7ff7ac198da0 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b1fd5700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac198da0 0x7ff7ac1991b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400ba50 tx=0x7ff7a400bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7a27fc700 1 -- 172.21.15.31:0/2126721338 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4019070 con 0x7ff7ac198da0 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7a27fc700 1 -- 172.21.15.31:0/2126721338 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7a4014440 con 0x7ff7ac198da0 2024-04-25T15:52:53.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.062+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7ac1a5760 con 0x7ff7ac198da0 2024-04-25T15:52:53.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.063+0000 7ff7a27fc700 1 -- 172.21.15.31:0/2126721338 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7a400ce60 con 0x7ff7ac198da0 2024-04-25T15:52:53.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.063+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7ac1a5c70 con 0x7ff7ac198da0 2024-04-25T15:52:53.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.064+0000 7ff7a27fc700 1 -- 172.21.15.31:0/2126721338 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7ff7a401c070 con 0x7ff7ac198da0 2024-04-25T15:52:53.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.064+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7ac1037c0 con 0x7ff7ac198da0 2024-04-25T15:52:53.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.064+0000 7ff7a27fc700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff79803daa0 0x7ff79803ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:53.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.064+0000 7ff7a27fc700 1 -- 172.21.15.31:0/2126721338 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff7a4050cd0 con 0x7ff7ac198da0 2024-04-25T15:52:53.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.064+0000 7ff7b17d4700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff79803daa0 0x7ff79803ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:53.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.065+0000 7ff7b17d4700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff79803daa0 0x7ff79803ffb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8009750 tx=0x7ff7a801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:53.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.069+0000 7ff7a27fc700 1 -- 172.21.15.31:0/2126721338 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7a4056020 con 0x7ff7ac198da0 2024-04-25T15:52:53.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.286+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ff7ac1a5f30 con 0x7ff7ac198da0 2024-04-25T15:52:53.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.287+0000 7ff7a27fc700 1 -- 172.21.15.31:0/2126721338 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7ff7a4012c60 con 0x7ff7ac198da0 2024-04-25T15:52:53.287 INFO:teuthology.orchestra.run.smithi031.stdout:# minimal ceph.conf for 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:52:53.288 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2024-04-25T15:52:53.288 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T15:52:53.288 INFO:teuthology.orchestra.run.smithi031.stdout: mon_host = [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] 2024-04-25T15:52:53.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff79803daa0 msgr2=0x7ff79803ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff79803daa0 0x7ff79803ffb0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff7a8009750 tx=0x7ff7a801a040 comp rx=0 tx=0).stop 2024-04-25T15:52:53.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac198da0 msgr2=0x7ff7ac1991b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:53.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac198da0 0x7ff7ac1991b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7ff7a400ba50 tx=0x7ff7a400bd60 comp rx=0 tx=0).stop 2024-04-25T15:52:53.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 shutdown_connections 2024-04-25T15:52:53.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff79803daa0 0x7ff79803ffb0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7ac198da0 0x7ff7ac1991b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7ac102310 0x7ff7ac19e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 --2- 172.21.15.31:0/2126721338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7ac19eb80 0x7ff7ac198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:53.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 >> 172.21.15.31:0/2126721338 conn(0x7ff7ac0fdd00 msgr2=0x7ff7ac10a710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:53.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 shutdown_connections 2024-04-25T15:52:53.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:52:53.290+0000 7ff7b3a38700 1 -- 172.21.15.31:0/2126721338 wait complete. 2024-04-25T15:52:53.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: Updating smithi031:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: Updating smithi063:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: Updating smithi186:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T15:52:53.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:53.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T15:52:53.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:53.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: Reconfiguring daemon mon.a on smithi031 2024-04-25T15:52:53.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:53 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2126721338' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: Updating smithi031:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: Updating smithi063:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: Updating smithi186:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.674 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: Reconfiguring daemon mon.a on smithi031 2024-04-25T15:52:53.675 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:53 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2126721338' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:53.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: Updating smithi031:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: Updating smithi063:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: Updating smithi186:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-25T15:52:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: Reconfiguring mon.a (unknown last config time)... 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: Reconfiguring daemon mon.a on smithi031 2024-04-25T15:52:53.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:53 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2126721338' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:54.002 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-25T15:52:54.002 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:52:54.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T15:52:54.069 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:52:54.069 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:54.134 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:52:54.134 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T15:52:54.167 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:52:54.167 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:54.236 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:52:54.236 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-04-25T15:52:54.267 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:52:54.268 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-25T15:52:54.335 INFO:tasks.cephadm:Adding mgr.x on smithi031 2024-04-25T15:52:54.335 INFO:tasks.cephadm:Adding mgr.y on smithi063 2024-04-25T15:52:54.335 INFO:tasks.cephadm:Adding mgr.z on smithi186 2024-04-25T15:52:54.335 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi031=x;smithi063=y;smithi186=z' 2024-04-25T15:52:54.608 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:52:54.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:54 smithi031 ceph-mon[29140]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:54.696 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:54 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:54.846 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:54 smithi186 ceph-mon[33258]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:54.846 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:54 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:54.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:54 smithi063 ceph-mon[32413]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:54.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:54 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:56.106 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.101+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/2416503277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c14102ac0 msgr2=0x7f4c14102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.101+0000 7f4c1b4c4700 1 --2- 172.21.15.186:0/2416503277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c14102ac0 0x7f4c14102ed0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4c04009910 tx=0x7f4c04009c20 comp rx=0 tx=0).stop 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.101+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/2416503277 shutdown_connections 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.101+0000 7f4c1b4c4700 1 --2- 172.21.15.186:0/2416503277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c14102ac0 0x7f4c14102ed0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.101+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/2416503277 >> 172.21.15.186:0/2416503277 conn(0x7f4c140fe120 msgr2=0x7f4c141005b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.101+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/2416503277 shutdown_connections 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.101+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/2416503277 wait complete. 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.102+0000 7f4c1b4c4700 1 Processor -- start 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.102+0000 7f4c1b4c4700 1 -- start start 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.102+0000 7f4c1b4c4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c14102ac0 0x7f4c14198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.102+0000 7f4c1b4c4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c14198c60 0x7f4c1419dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.102+0000 7f4c1b4c4700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c1419e130 0x7f4c141a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:56.107 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.102+0000 7f4c1b4c4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4c14108a80 con 0x7f4c14102ac0 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.102+0000 7f4c1b4c4700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4c14108900 con 0x7f4c1419e130 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.102+0000 7f4c1b4c4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4c14108c00 con 0x7f4c14198c60 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.102+0000 7f4c18a5f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c14198c60 0x7f4c1419dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c18a5f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c14198c60 0x7f4c1419dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:37676/0 (socket says 172.21.15.186:37676) 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c18a5f700 1 -- 172.21.15.186:0/4054750855 learned_addr learned my addr 172.21.15.186:0/4054750855 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c19260700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c14102ac0 0x7f4c14198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c19a61700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c1419e130 0x7f4c141a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c19260700 1 -- 172.21.15.186:0/4054750855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c14198c60 msgr2=0x7f4c1419dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c19260700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c14198c60 0x7f4c1419dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c19260700 1 -- 172.21.15.186:0/4054750855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c1419e130 msgr2=0x7f4c141a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c19260700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c1419e130 0x7f4c141a0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c19260700 1 -- 172.21.15.186:0/4054750855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c141a0ac0 con 0x7f4c14102ac0 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c18a5f700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c14198c60 0x7f4c1419dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:52:56.108 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c19260700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c14102ac0 0x7f4c14198720 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c14103af0 tx=0x7f4c04004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c0a7fc700 1 -- 172.21.15.186:0/4054750855 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c04026070 con 0x7f4c14102ac0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c141a0d50 con 0x7f4c14102ac0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c0a7fc700 1 -- 172.21.15.186:0/4054750855 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c0400b750 con 0x7f4c14102ac0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c0a7fc700 1 -- 172.21.15.186:0/4054750855 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c040038e0 con 0x7f4c14102ac0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.103+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c141a1240 con 0x7f4c14102ac0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.104+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c141a1470 con 0x7f4c14102ac0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.105+0000 7f4c0a7fc700 1 -- 172.21.15.186:0/4054750855 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f4c04029070 con 0x7f4c14102ac0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.105+0000 7f4c0a7fc700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4c0003daa0 0x7f4c0003ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.105+0000 7f4c0a7fc700 1 -- 172.21.15.186:0/4054750855 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4c040520e0 con 0x7f4c14102ac0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.105+0000 7f4c18a5f700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4c0003daa0 0x7f4c0003ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:52:56.109 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.106+0000 7f4c18a5f700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4c0003daa0 0x7f4c0003ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c141997d0 tx=0x7f4c100092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:52:56.112 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.111+0000 7f4c0a7fc700 1 -- 172.21.15.186:0/4054750855 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c04013070 con 0x7f4c14102ac0 2024-04-25T15:52:56.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:56 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:56.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:56 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:56.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:56 smithi186 ceph-mon[33258]: Reconfiguring mon.b (monmap changed)... 2024-04-25T15:52:56.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:56 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:56.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:56 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T15:52:56.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:56 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:56.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:56 smithi186 ceph-mon[33258]: Reconfiguring daemon mon.b on smithi063 2024-04-25T15:52:56.320 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.319+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi063=y;smithi186=z", "target": ["mon-mgr", ""]}) -- 0x7f4c140fff70 con 0x7f4c0003daa0 2024-04-25T15:52:56.328 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.327+0000 7f4c0a7fc700 1 -- 172.21.15.186:0/4054750855 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4c140fff70 con 0x7f4c0003daa0 2024-04-25T15:52:56.328 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mgr update... 2024-04-25T15:52:56.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4c0003daa0 msgr2=0x7f4c0003ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:56.330 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4c0003daa0 0x7f4c0003ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4c141997d0 tx=0x7f4c100092c0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c14102ac0 msgr2=0x7f4c14198720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c14102ac0 0x7f4c14198720 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4c14103af0 tx=0x7f4c04004ed0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 shutdown_connections 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4c0003daa0 0x7f4c0003ffb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4c1419e130 0x7f4c141a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4c14102ac0 0x7f4c14198720 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 --2- 172.21.15.186:0/4054750855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c14198c60 0x7f4c1419dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 >> 172.21.15.186:0/4054750855 conn(0x7f4c140fe120 msgr2=0x7f4c140ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 shutdown_connections 2024-04-25T15:52:56.331 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:52:56.329+0000 7f4c1b4c4700 1 -- 172.21.15.186:0/4054750855 wait complete. 2024-04-25T15:52:56.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:56 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:56.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:56 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:56.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:56 smithi063 ceph-mon[32413]: Reconfiguring mon.b (monmap changed)... 2024-04-25T15:52:56.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:56 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:56.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:56 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T15:52:56.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:56 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:56.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:56 smithi063 ceph-mon[32413]: Reconfiguring daemon mon.b on smithi063 2024-04-25T15:52:56.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:56 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:56.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:56 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:56.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:56 smithi031 ceph-mon[29140]: Reconfiguring mon.b (monmap changed)... 2024-04-25T15:52:56.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:56 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:56.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:56 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T15:52:56.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:56 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:56.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:56 smithi031 ceph-mon[29140]: Reconfiguring daemon mon.b on smithi063 2024-04-25T15:52:56.950 DEBUG:teuthology.orchestra.run.smithi063:mgr.y> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.y.service 2024-04-25T15:52:56.953 DEBUG:teuthology.orchestra.run.smithi186:mgr.z> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.z.service 2024-04-25T15:52:56.955 INFO:tasks.cephadm:Deploying OSDs... 2024-04-25T15:52:56.955 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:52:56.955 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T15:52:56.975 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T15:52:56.975 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2024-04-25T15:52:57.034 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T15:52:57.035 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:57.035 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T15:52:57.035 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:57.035 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:57.035 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 15:52:02.006977390 +0000 2024-04-25T15:52:57.035 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 15:47:33.037450067 +0000 2024-04-25T15:52:57.035 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 15:47:33.037450067 +0000 2024-04-25T15:52:57.035 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-25 15:47:33.037450067 +0000 2024-04-25T15:52:57.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T15:52:57.104 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-25T15:52:57.104 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-25T15:52:57.104 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000234136 s, 2.2 MB/s 2024-04-25T15:52:57.105 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T15:52:57.165 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2024-04-25T15:52:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T15:52:57.224 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:57.224 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-25T15:52:57.224 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:57.224 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:57.224 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 15:52:02.053976601 +0000 2024-04-25T15:52:57.224 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 15:47:33.396443537 +0000 2024-04-25T15:52:57.224 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 15:47:33.396443537 +0000 2024-04-25T15:52:57.225 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-25 15:47:33.396443537 +0000 2024-04-25T15:52:57.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T15:52:57.289 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:57 smithi031 ceph-mon[29140]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:57.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:57.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:57.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:57.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:57.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T15:52:57.290 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-25T15:52:57.292 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-25T15:52:57.293 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000245505 s, 2.1 MB/s 2024-04-25T15:52:57.294 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T15:52:57.309 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:57 smithi186 ceph-mon[33258]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:57.309 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:57.309 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:57.309 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:57.309 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:57.309 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T15:52:57.309 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:57.354 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2024-04-25T15:52:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:57 smithi063 ceph-mon[32413]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-25T15:52:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-25T15:52:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:57.414 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T15:52:57.414 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T15:52:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 15:52:02.094975913 +0000 2024-04-25T15:52:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 15:47:33.747437152 +0000 2024-04-25T15:52:57.414 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 15:47:33.747437152 +0000 2024-04-25T15:52:57.414 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-25 15:47:33.747437152 +0000 2024-04-25T15:52:57.414 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T15:52:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-25T15:52:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-25T15:52:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000221439 s, 2.3 MB/s 2024-04-25T15:52:57.483 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T15:52:57.542 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2024-04-25T15:52:57.600 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T15:52:57.600 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:57.600 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-25T15:52:57.601 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:57.601 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:57.601 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 15:52:02.131975293 +0000 2024-04-25T15:52:57.601 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 15:47:34.095430822 +0000 2024-04-25T15:52:57.601 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 15:47:34.095430822 +0000 2024-04-25T15:52:57.601 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-25 15:47:34.095430822 +0000 2024-04-25T15:52:57.601 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T15:52:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2024-04-25T15:52:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2024-04-25T15:52:57.668 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000240238 s, 2.1 MB/s 2024-04-25T15:52:57.669 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T15:52:57.729 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-04-25T15:52:57.729 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T15:52:57.748 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T15:52:57.748 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-04-25T15:52:57.807 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T15:52:57.807 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:57.807 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-25T15:52:57.807 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:57.807 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:57.807 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-25 15:52:44.235911174 +0000 2024-04-25T15:52:57.807 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-25 15:47:50.523286294 +0000 2024-04-25T15:52:57.807 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-25 15:47:50.523286294 +0000 2024-04-25T15:52:57.807 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-04-25 15:47:50.523286294 +0000 2024-04-25T15:52:57.808 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T15:52:57.874 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-25T15:52:57.874 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-25T15:52:57.874 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.00020785 s, 2.5 MB/s 2024-04-25T15:52:57.876 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T15:52:57.935 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-04-25T15:52:57.992 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T15:52:57.992 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:57.992 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-25T15:52:57.993 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:57.993 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:57.993 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-25 15:52:44.235911174 +0000 2024-04-25T15:52:57.993 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-25 15:47:50.867282991 +0000 2024-04-25T15:52:57.993 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-25 15:47:50.867282991 +0000 2024-04-25T15:52:57.993 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-04-25 15:47:50.867282991 +0000 2024-04-25T15:52:57.993 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T15:52:58.058 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-25T15:52:58.058 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-25T15:52:58.058 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000296709 s, 1.7 MB/s 2024-04-25T15:52:58.060 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T15:52:58.118 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-04-25T15:52:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T15:52:58.175 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:58.175 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T15:52:58.176 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:58.176 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:58.176 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-25 15:52:44.236911164 +0000 2024-04-25T15:52:58.176 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-25 15:47:51.209279708 +0000 2024-04-25T15:52:58.176 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-25 15:47:51.209279708 +0000 2024-04-25T15:52:58.176 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-04-25 15:47:51.209279708 +0000 2024-04-25T15:52:58.176 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T15:52:58.238 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:58 smithi063 ceph-mon[32413]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi063=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:52:58.238 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:58 smithi063 ceph-mon[32413]: Saving service mgr spec with placement smithi031=x;smithi063=y;smithi186=z;count:3 2024-04-25T15:52:58.238 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:58 smithi063 ceph-mon[32413]: Reconfiguring mon.c (monmap changed)... 2024-04-25T15:52:58.238 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:58 smithi063 ceph-mon[32413]: Reconfiguring daemon mon.c on smithi186 2024-04-25T15:52:58.240 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-25T15:52:58.240 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-25T15:52:58.240 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000175484 s, 2.9 MB/s 2024-04-25T15:52:58.241 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T15:52:58.298 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-04-25T15:52:58.355 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T15:52:58.355 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:58.355 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-25T15:52:58.355 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:58.355 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:58.355 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-04-25 15:52:44.236911164 +0000 2024-04-25T15:52:58.355 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-04-25 15:47:51.540276531 +0000 2024-04-25T15:52:58.355 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-04-25 15:47:51.540276531 +0000 2024-04-25T15:52:58.355 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-04-25 15:47:51.540276531 +0000 2024-04-25T15:52:58.355 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T15:52:58.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:58 smithi186 ceph-mon[33258]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi063=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:52:58.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:58 smithi186 ceph-mon[33258]: Saving service mgr spec with placement smithi031=x;smithi063=y;smithi186=z;count:3 2024-04-25T15:52:58.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:58 smithi186 ceph-mon[33258]: Reconfiguring mon.c (monmap changed)... 2024-04-25T15:52:58.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:58 smithi186 ceph-mon[33258]: Reconfiguring daemon mon.c on smithi186 2024-04-25T15:52:58.420 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-04-25T15:52:58.420 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-04-25T15:52:58.420 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000195687 s, 2.6 MB/s 2024-04-25T15:52:58.421 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T15:52:58.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:58 smithi031 ceph-mon[29140]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi031=x;smithi063=y;smithi186=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:52:58.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:58 smithi031 ceph-mon[29140]: Saving service mgr spec with placement smithi031=x;smithi063=y;smithi186=z;count:3 2024-04-25T15:52:58.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:58 smithi031 ceph-mon[29140]: Reconfiguring mon.c (monmap changed)... 2024-04-25T15:52:58.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:58 smithi031 ceph-mon[29140]: Reconfiguring daemon mon.c on smithi186 2024-04-25T15:52:58.478 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-04-25T15:52:58.478 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-04-25T15:52:58.496 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-25T15:52:58.496 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-04-25T15:52:58.552 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-25T15:52:58.552 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:58.553 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-25T15:52:58.553 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:58.553 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:58.553 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-25 15:52:35.237307438 +0000 2024-04-25T15:52:58.553 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-25 15:47:44.739280575 +0000 2024-04-25T15:52:58.553 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-25 15:47:44.739280575 +0000 2024-04-25T15:52:58.553 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-25 15:47:44.739280575 +0000 2024-04-25T15:52:58.553 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-25T15:52:58.618 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-25T15:52:58.619 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.00026584 s, 1.9 MB/s 2024-04-25T15:52:58.620 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-25T15:52:58.678 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-04-25T15:52:58.734 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-25T15:52:58.735 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:58.735 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-25T15:52:58.735 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:58.735 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:58.735 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-25 15:52:35.237307438 +0000 2024-04-25T15:52:58.735 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-25 15:47:45.076274380 +0000 2024-04-25T15:52:58.735 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-25 15:47:45.076274380 +0000 2024-04-25T15:52:58.735 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-25 15:47:45.076274380 +0000 2024-04-25T15:52:58.735 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-25T15:52:58.799 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-25T15:52:58.799 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-25T15:52:58.799 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000233605 s, 2.2 MB/s 2024-04-25T15:52:58.801 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-25T15:52:58.857 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-04-25T15:52:58.911 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-25T15:52:58.912 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:58.912 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-25T15:52:58.912 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:58.912 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:58.912 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-25 15:52:35.238307421 +0000 2024-04-25T15:52:58.912 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-25 15:47:45.419268076 +0000 2024-04-25T15:52:58.912 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-25 15:47:45.419268076 +0000 2024-04-25T15:52:58.912 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-25 15:47:45.419268076 +0000 2024-04-25T15:52:58.912 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-25T15:52:58.974 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-25T15:52:58.974 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-25T15:52:58.975 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000182409 s, 2.8 MB/s 2024-04-25T15:52:58.976 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-25T15:52:59.031 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-04-25T15:52:59.085 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-25T15:52:59.086 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-25T15:52:59.086 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-25T15:52:59.086 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T15:52:59.086 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-04-25T15:52:59.086 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-04-25 15:52:35.238307421 +0000 2024-04-25T15:52:59.086 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-04-25 15:47:45.770261624 +0000 2024-04-25T15:52:59.086 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-04-25 15:47:45.770261624 +0000 2024-04-25T15:52:59.086 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-04-25 15:47:45.770261624 +0000 2024-04-25T15:52:59.086 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-25T15:52:59.148 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-04-25T15:52:59.148 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-04-25T15:52:59.148 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000228873 s, 2.2 MB/s 2024-04-25T15:52:59.149 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-25T15:52:59.205 INFO:tasks.cephadm:Deploying osd.0 on smithi031 with /dev/vg_nvme/lv_4... 2024-04-25T15:52:59.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T15:52:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:59.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:59.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T15:52:59.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T15:52:59.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T15:52:59.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:52:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:59.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:59.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T15:52:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T15:52:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T15:52:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:52:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:59.447 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:52:59.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:52:59.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:52:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:52:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:52:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:52:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T15:52:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T15:52:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T15:52:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:52:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:00.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:00 smithi031 ceph-mon[29140]: Deploying daemon mgr.z on smithi186 2024-04-25T15:53:00.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:00 smithi063 ceph-mon[32413]: Deploying daemon mgr.z on smithi186 2024-04-25T15:53:00.370 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:00 smithi186 ceph-mon[33258]: Deploying daemon mgr.z on smithi186 2024-04-25T15:53:00.898 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:00 smithi186 systemd[1]: Starting Ceph mgr.z for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:53:01.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:01 smithi031 ceph-mon[29140]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:01.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:01 smithi063 ceph-mon[32413]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:01.382 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:01 smithi186 ceph-mon[33258]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:01.382 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 podman[35102]: 2024-04-25 15:53:00.943209752 +0000 UTC m=+0.017843408 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:53:01.383 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 podman[35102]: 2024-04-25 15:53:01.155565054 +0000 UTC m=+0.230198740 container create 5ec702eb857bc1f8a4b494aaa1cc173195d3bcf3da5c3505a6509e5a97212faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-25T15:53:01.383 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 podman[35102]: 2024-04-25 15:53:01.382733748 +0000 UTC m=+0.457367391 container init 5ec702eb857bc1f8a4b494aaa1cc173195d3bcf3da5c3505a6509e5a97212faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T15:53:01.496 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T15:53:01.496 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T15:53:01.496 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-25T15:53:01.496 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-25T15:53:01.496 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114575 s, 915 MB/s 2024-04-25T15:53:01.496 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-25T15:53:01.648 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 podman[35102]: 2024-04-25 15:53:01.385440775 +0000 UTC m=+0.460074421 container start 5ec702eb857bc1f8a4b494aaa1cc173195d3bcf3da5c3505a6509e5a97212faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-25T15:53:01.648 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 ceph-mgr[35121]: -- 172.21.15.186:0/3163648701 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559cde5c85a0 con 0x559ce3352480 2024-04-25T15:53:01.649 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 bash[35102]: 5ec702eb857bc1f8a4b494aaa1cc173195d3bcf3da5c3505a6509e5a97212faa 2024-04-25T15:53:01.649 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 systemd[1]: Started Ceph mgr.z for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:53:01.649 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:01.554+0000 7f7b8d286200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:53:02.148 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:01.743+0000 7f7b8d286200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:53:02.148 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:01.845+0000 7f7b8d286200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:53:02.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2024-04-25T15:53:02.468 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:53:02.578 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:02.157+0000 7f7b8d286200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:53:02.578 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:02.241+0000 7f7b8d286200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:53:02.862 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:02 smithi031 ceph-mon[29140]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:02.862 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:02 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:02.862 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:02 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:02.862 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:02 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:02.862 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:02 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T15:53:02.862 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:02 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T15:53:02.862 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:02 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T15:53:02.862 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:02 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:02.862 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:02 smithi031 ceph-mon[29140]: Deploying daemon mgr.y on smithi063 2024-04-25T15:53:02.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:02 smithi063 ceph-mon[32413]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:02.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:02 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:02.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:02 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:02.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:02 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:02.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:02 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T15:53:02.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:02 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T15:53:02.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:02 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T15:53:02.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:02 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:02.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:02 smithi063 ceph-mon[32413]: Deploying daemon mgr.y on smithi063 2024-04-25T15:53:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-mon[33258]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T15:53:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-25T15:53:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T15:53:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:02.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:02 smithi186 ceph-mon[33258]: Deploying daemon mgr.y on smithi063 2024-04-25T15:53:03.781 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:03.439+0000 7f7b8d286200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:53:03.781 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:03.608+0000 7f7b8d286200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:53:03.869 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:03 smithi063 systemd[1]: Starting Ceph mgr.y for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:53:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.983+0000 7f1a3cc99700 1 -- 172.21.15.31:0/1137473836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 msgr2=0x7f1a381034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.983+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/1137473836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 0x7f1a381034f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c009990 tx=0x7f1a2c009ca0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.983+0000 7f1a3cc99700 1 -- 172.21.15.31:0/1137473836 shutdown_connections 2024-04-25T15:53:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.983+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/1137473836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a38104bb0 0x7f1a3810b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.983+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/1137473836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 0x7f1a381034f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.983+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/1137473836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38104210 0x7f1a38104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.983+0000 7f1a3cc99700 1 -- 172.21.15.31:0/1137473836 >> 172.21.15.31:0/1137473836 conn(0x7f1a380fe740 msgr2=0x7f1a38100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.983+0000 7f1a3cc99700 1 -- 172.21.15.31:0/1137473836 shutdown_connections 2024-04-25T15:53:03.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.983+0000 7f1a3cc99700 1 -- 172.21.15.31:0/1137473836 wait complete. 2024-04-25T15:53:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3cc99700 1 Processor -- start 2024-04-25T15:53:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3cc99700 1 -- start start 2024-04-25T15:53:03.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3cc99700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 0x7f1a3819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3cc99700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38104210 0x7f1a3819ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3cc99700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a38104bb0 0x7f1a381a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3cc99700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3810eb70 con 0x7f1a381030e0 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3cc99700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3810e9f0 con 0x7f1a38104210 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3cc99700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3810ecf0 con 0x7f1a38104bb0 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 0x7f1a3819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3659c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 0x7f1a3819a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50666/0 (socket says 172.21.15.31:50666) 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a3659c700 1 -- 172.21.15.31:0/3513371563 learned_addr learned my addr 172.21.15.31:0/3513371563 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a35d9b700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38104210 0x7f1a3819ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.984+0000 7f1a36d9d700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a38104bb0 0x7f1a381a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a3659c700 1 -- 172.21.15.31:0/3513371563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a38104bb0 msgr2=0x7f1a381a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a3659c700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a38104bb0 0x7f1a381a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a3659c700 1 -- 172.21.15.31:0/3513371563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38104210 msgr2=0x7f1a3819ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:03.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a3659c700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38104210 0x7f1a3819ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a3659c700 1 -- 172.21.15.31:0/3513371563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a381a2340 con 0x7f1a381030e0 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a36d9d700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a38104bb0 0x7f1a381a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a3659c700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 0x7f1a3819a7e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c0098e0 tx=0x7f1a2c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a277fe700 1 -- 172.21.15.31:0/3513371563 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c026070 con 0x7f1a381030e0 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a277fe700 1 -- 172.21.15.31:0/3513371563 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a2c004000 con 0x7f1a381030e0 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a381a25d0 con 0x7f1a381030e0 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a277fe700 1 -- 172.21.15.31:0/3513371563 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c00fd10 con 0x7f1a381030e0 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.985+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a381a2a80 con 0x7f1a381030e0 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.986+0000 7f1a277fe700 1 -- 172.21.15.31:0/3513371563 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f1a2c029070 con 0x7f1a381030e0 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.986+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a38051190 con 0x7f1a381030e0 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.986+0000 7f1a277fe700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a2003d850 0x7f1a2003fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:03.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.986+0000 7f1a277fe700 1 -- 172.21.15.31:0/3513371563 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f1a2c052330 con 0x7f1a381030e0 2024-04-25T15:53:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.987+0000 7f1a35d9b700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a2003d850 0x7f1a2003fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:03.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.987+0000 7f1a35d9b700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a2003d850 0x7f1a2003fd60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1a28005950 tx=0x7f1a2800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:03.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:03.992+0000 7f1a277fe700 1 -- 172.21.15.31:0/3513371563 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a2c013070 con 0x7f1a381030e0 2024-04-25T15:53:04.066 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:03.779+0000 7f7b8d286200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:53:04.066 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:04.064+0000 7f7b8d286200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:53:04.166 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:03 smithi063 podman[33785]: 2024-04-25 15:53:03.815791237 +0000 UTC m=+0.025599242 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:53:04.166 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:03 smithi063 podman[33785]: 2024-04-25 15:53:03.917032174 +0000 UTC m=+0.126840163 container create 1513da908ef814fa568b9849a2e30a87f4af28b59c2380bd402df6e13b855008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0) 2024-04-25T15:53:04.166 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:04 smithi063 podman[33785]: 2024-04-25 15:53:04.12761022 +0000 UTC m=+0.337418211 container init 1513da908ef814fa568b9849a2e30a87f4af28b59c2380bd402df6e13b855008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True) 2024-04-25T15:53:04.166 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:04 smithi063 podman[33785]: 2024-04-25 15:53:04.13015152 +0000 UTC m=+0.339959511 container start 1513da908ef814fa568b9849a2e30a87f4af28b59c2380bd402df6e13b855008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T15:53:04.166 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mgr[33804]: -- 172.21.15.63:0/114753187 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a81902e5a0 con 0x55a81ddb8900 2024-04-25T15:53:04.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:04.215+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1a38100790 con 0x7f1a2003d850 2024-04-25T15:53:04.362 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:04.155+0000 7f7b8d286200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:53:04.362 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:04.361+0000 7f7b8d286200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:53:04.490 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:04 smithi063 bash[33785]: 1513da908ef814fa568b9849a2e30a87f4af28b59c2380bd402df6e13b855008 2024-04-25T15:53:04.490 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:04 smithi063 systemd[1]: Started Ceph mgr.y for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:53:04.490 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:04.297+0000 7f9d179a7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-25T15:53:04.490 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:04.489+0000 7f9d179a7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:53:04.648 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:04.556+0000 7f7b8d286200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:53:04.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-mon[33258]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:53:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:53:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:04.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:04 smithi031 ceph-mon[29140]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:04.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:53:04.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:53:04.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:04.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.858 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.859 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:04.869 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:04.597+0000 7f9d179a7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-25T15:53:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mon[32413]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:53:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:53:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:05.307 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:04.929+0000 7f9d179a7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-25T15:53:05.307 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:05 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:05.024+0000 7f9d179a7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-25T15:53:05.585 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:05.205+0000 7f7b8d286200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T15:53:05.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:05 smithi063 ceph-mon[32413]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:53:05.888 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:05 smithi031 ceph-mon[29140]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:53:05.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:05 smithi186 ceph-mon[33258]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:53:05.898 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:05.639+0000 7f7b8d286200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:53:05.899 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:05.751+0000 7f7b8d286200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:53:06.398 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:05.954+0000 7f7b8d286200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T15:53:06.619 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:06.320+0000 7f9d179a7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-25T15:53:06.619 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:06.496+0000 7f9d179a7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-25T15:53:06.792 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:06.486+0000 7f7b8d286200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:53:06.792 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:06.577+0000 7f7b8d286200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:53:06.972 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:06.673+0000 7f9d179a7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-25T15:53:06.972 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:06.971+0000 7f9d179a7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-25T15:53:06.972 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T15:53:06.973 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:06.816+0000 7f7b8d286200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:06.948+0000 7f7b8d286200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:07.042+0000 7f7b8d286200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:07.044 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T15:53:07.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T15:53:07.045 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:07.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-25T15:53:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:07.276 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:07.066+0000 7f9d179a7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-25T15:53:07.277 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:07.276+0000 7f9d179a7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-25T15:53:07.398 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:07.158+0000 7f7b8d286200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:53:07.619 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:07.475+0000 7f9d179a7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-25T15:53:07.793 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:07.475+0000 7f7b8d286200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:53:08.070 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-mon[33258]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T15:53:08.071 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-mon[33258]: Reconfiguring daemon mgr.x on smithi031 2024-04-25T15:53:08.071 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2260992154' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd266854-c19f-4bb8-9461-5b329b2aa0b7"}]: dispatch 2024-04-25T15:53:08.071 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd266854-c19f-4bb8-9461-5b329b2aa0b7"}]: dispatch 2024-04-25T15:53:08.071 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd266854-c19f-4bb8-9461-5b329b2aa0b7"}]': finished 2024-04-25T15:53:08.071 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-mon[33258]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T15:53:08.071 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:08.071 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:07 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3010494059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:53:08.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-mon[32413]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T15:53:08.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-mon[32413]: Reconfiguring daemon mgr.x on smithi031 2024-04-25T15:53:08.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2260992154' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd266854-c19f-4bb8-9461-5b329b2aa0b7"}]: dispatch 2024-04-25T15:53:08.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd266854-c19f-4bb8-9461-5b329b2aa0b7"}]: dispatch 2024-04-25T15:53:08.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd266854-c19f-4bb8-9461-5b329b2aa0b7"}]': finished 2024-04-25T15:53:08.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-mon[32413]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T15:53:08.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:08.103 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:07 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3010494059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:53:08.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:07 smithi031 ceph-mon[29140]: Reconfiguring mgr.x (unknown last config time)... 2024-04-25T15:53:08.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:07 smithi031 ceph-mon[29140]: Reconfiguring daemon mgr.x on smithi031 2024-04-25T15:53:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:07 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2260992154' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd266854-c19f-4bb8-9461-5b329b2aa0b7"}]: dispatch 2024-04-25T15:53:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:07 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cd266854-c19f-4bb8-9461-5b329b2aa0b7"}]: dispatch 2024-04-25T15:53:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:07 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cd266854-c19f-4bb8-9461-5b329b2aa0b7"}]': finished 2024-04-25T15:53:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:07 smithi031 ceph-mon[29140]: osdmap e5: 1 total, 0 up, 1 in 2024-04-25T15:53:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:07 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:07 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3010494059' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:53:08.369 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:08.102+0000 7f9d179a7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-25T15:53:08.398 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:08.069+0000 7f7b8d286200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:53:08.398 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 15:53:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z[35117]: 2024-04-25T15:53:08.282+0000 7f7b8d286200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:53:08.796 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:08.539+0000 7f9d179a7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-25T15:53:08.796 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:08.638+0000 7f9d179a7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-25T15:53:09.119 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:08.846+0000 7f9d179a7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-25T15:53:09.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-mon[32413]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:09.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-mon[32413]: Standby manager daemon z started 2024-04-25T15:53:09.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-mon[32413]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T15:53:09.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-mon[32413]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T15:53:09.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-mon[32413]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T15:53:09.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:08 smithi063 ceph-mon[32413]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T15:53:09.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:09.135+0000 7f1a277fe700 1 -- 172.21.15.31:0/3513371563 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 14) ==== 98518+0+0 (secure 0 0 0) 0x7f1a2c02bce0 con 0x7f1a381030e0 2024-04-25T15:53:09.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:08 smithi031 ceph-mon[29140]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:09.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:08 smithi031 ceph-mon[29140]: Standby manager daemon z started 2024-04-25T15:53:09.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:08 smithi031 ceph-mon[29140]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T15:53:09.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:08 smithi031 ceph-mon[29140]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T15:53:09.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:08 smithi031 ceph-mon[29140]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T15:53:09.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:08 smithi031 ceph-mon[29140]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T15:53:09.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:08 smithi186 ceph-mon[33258]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:09.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:08 smithi186 ceph-mon[33258]: Standby manager daemon z started 2024-04-25T15:53:09.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:08 smithi186 ceph-mon[33258]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-25T15:53:09.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:08 smithi186 ceph-mon[33258]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T15:53:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:08 smithi186 ceph-mon[33258]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-25T15:53:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:08 smithi186 ceph-mon[33258]: from='mgr.? 172.21.15.186:0/3644115491' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T15:53:09.739 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:09 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:09.391+0000 7f9d179a7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-25T15:53:09.739 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:09 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:09.486+0000 7f9d179a7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-25T15:53:10.092 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:09 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:09.738+0000 7f9d179a7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-25T15:53:10.093 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:09 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:09.876+0000 7f9d179a7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-25T15:53:10.093 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:09 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:09.973+0000 7f9d179a7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-25T15:53:10.093 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:10.091+0000 7f9d179a7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-25T15:53:10.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:10 smithi063 ceph-mon[32413]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-25T15:53:10.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T15:53:10.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:10 smithi063 ceph-mon[32413]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:10.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:10.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:10 smithi186 ceph-mon[33258]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-25T15:53:10.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T15:53:10.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:10 smithi186 ceph-mon[33258]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:10.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:10.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:10 smithi031 ceph-mon[29140]: mgrmap e14: x(active, since 79s), standbys: z 2024-04-25T15:53:10.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-25T15:53:10.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:10 smithi031 ceph-mon[29140]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:10.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:10.869 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:10.411+0000 7f9d179a7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-25T15:53:11.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:11.316+0000 7f1a277fe700 1 -- 172.21.15.31:0/3513371563 <== mon.0 v2:172.21.15.31:3300/0 8 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f1a2c01f310 con 0x7f1a381030e0 2024-04-25T15:53:11.322 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:11 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:11.034+0000 7f9d179a7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-25T15:53:11.322 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 15:53:11 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y[33800]: 2024-04-25T15:53:11.256+0000 7f9d179a7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-25T15:53:11.533 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:11 smithi031 ceph-mon[29140]: Standby manager daemon y started 2024-04-25T15:53:11.533 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:11 smithi031 ceph-mon[29140]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T15:53:11.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:11 smithi031 ceph-mon[29140]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T15:53:11.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:11 smithi031 ceph-mon[29140]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T15:53:11.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:11 smithi031 ceph-mon[29140]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T15:53:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:11 smithi063 ceph-mon[32413]: Standby manager daemon y started 2024-04-25T15:53:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:11 smithi063 ceph-mon[32413]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T15:53:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:11 smithi063 ceph-mon[32413]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T15:53:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:11 smithi063 ceph-mon[32413]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T15:53:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:11 smithi063 ceph-mon[32413]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T15:53:11.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:11 smithi186 ceph-mon[33258]: Standby manager daemon y started 2024-04-25T15:53:11.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:11 smithi186 ceph-mon[33258]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-25T15:53:11.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:11 smithi186 ceph-mon[33258]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-25T15:53:11.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:11 smithi186 ceph-mon[33258]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-25T15:53:11.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:11 smithi186 ceph-mon[33258]: from='mgr.? 172.21.15.63:0/553121763' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-25T15:53:12.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:12 smithi031 ceph-mon[29140]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:12.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:12 smithi031 ceph-mon[29140]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-25T15:53:12.575 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T15:53:12.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:12 smithi063 ceph-mon[32413]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:12.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:12 smithi063 ceph-mon[32413]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-25T15:53:12.599 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T15:53:12.626 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:12 smithi186 ceph-mon[33258]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:12.626 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:12 smithi186 ceph-mon[33258]: mgrmap e15: x(active, since 82s), standbys: z, y 2024-04-25T15:53:12.626 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-25T15:53:14.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:14 smithi031 ceph-mon[29140]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:14.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:14 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:14.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:14 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:14.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:14 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:14.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:14 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:14.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:14 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:14.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:14 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:14 smithi063 ceph-mon[32413]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:14 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:14 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:14 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:14 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:14 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:14 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:15.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:14 smithi186 ceph-mon[33258]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:15.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:14 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:15.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:14 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:15.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:14 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:15.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:14 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:15.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:14 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:15.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:14 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:16.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:15 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T15:53:16.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:15 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:16.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:15 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T15:53:16.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:15 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:16.168 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:15 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-25T15:53:16.169 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:15 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:17.092 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:16 smithi031 ceph-mon[29140]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:17.093 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:16 smithi031 ceph-mon[29140]: Deploying daemon osd.0 on smithi031 2024-04-25T15:53:17.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:16 smithi186 ceph-mon[33258]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:17.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:16 smithi186 ceph-mon[33258]: Deploying daemon osd.0 on smithi031 2024-04-25T15:53:17.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:16 smithi063 ceph-mon[32413]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:17.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:16 smithi063 ceph-mon[32413]: Deploying daemon osd.0 on smithi031 2024-04-25T15:53:19.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:18 smithi031 ceph-mon[29140]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:19.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:18 smithi186 ceph-mon[33258]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:18 smithi063 ceph-mon[32413]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:21.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:20 smithi186 ceph-mon[33258]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:21.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:20 smithi031 ceph-mon[29140]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:21.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:20 smithi063 ceph-mon[32413]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:22.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:22.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:22.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:22.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:22 smithi186 ceph-mon[33258]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:22.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:22.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:22.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:22.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:22 smithi031 ceph-mon[29140]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:22.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:22.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:22.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:22.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:22 smithi063 ceph-mon[32413]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:24.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:24 smithi063 ceph-mon[32413]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:24.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:24 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:24.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:24 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:24.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:24 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:24.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:24 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:24.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:24 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:24.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:24 smithi031 ceph-mon[29140]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:24.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:24 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:24.620 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:24 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:24.621 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:24 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:24.621 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:24 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:24.621 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:24 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:24.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:24 smithi186 ceph-mon[33258]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:24.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:24 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:24.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:24 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:24.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:24 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:24.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:24 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:24.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:24 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:24.697 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 0 on host 'smithi031' 2024-04-25T15:53:24.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.696+0000 7f1a277fe700 1 -- 172.21.15.31:0/3513371563 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1a38100790 con 0x7f1a2003d850 2024-04-25T15:53:24.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.698+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a2003d850 msgr2=0x7f1a2003fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a2003d850 0x7f1a2003fd60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1a28005950 tx=0x7f1a2800a300 comp rx=0 tx=0).stop 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 msgr2=0x7f1a3819a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 0x7f1a3819a7e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c0098e0 tx=0x7f1a2c0047b0 comp rx=0 tx=0).stop 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 shutdown_connections 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a2003d850 0x7f1a2003fd60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a38104bb0 0x7f1a381a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a381030e0 0x7f1a3819a7e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 --2- 172.21.15.31:0/3513371563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a38104210 0x7f1a3819ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 >> 172.21.15.31:0/3513371563 conn(0x7f1a380fe740 msgr2=0x7f1a38100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 shutdown_connections 2024-04-25T15:53:24.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:53:24.699+0000 7f1a3cc99700 1 -- 172.21.15.31:0/3513371563 wait complete. 2024-04-25T15:53:25.351 DEBUG:teuthology.orchestra.run.smithi031:osd.0> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.0.service 2024-04-25T15:53:25.353 INFO:tasks.cephadm:Deploying osd.1 on smithi063 with /dev/vg_nvme/lv_4... 2024-04-25T15:53:25.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T15:53:25.622 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T15:53:25.940 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:25 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:25.941 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:25 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:25.941 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:25 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:25.941 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:25 smithi063 ceph-mon[32413]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T15:53:25.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:25 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:25.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:25 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:25.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:25 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:25.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:25 smithi031 ceph-mon[29140]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T15:53:26.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:25 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:26.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:25 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:26.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:25 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:26.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:25 smithi186 ceph-mon[33258]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-25T15:53:26.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:26 smithi031 ceph-mon[29140]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:26.783 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:26 smithi031 ceph-mon[29140]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T15:53:26.784 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:26 smithi031 ceph-mon[29140]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T15:53:26.784 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:26 smithi031 ceph-mon[29140]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:53:26.784 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:26.784 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:53:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0[44315]: 2024-04-25T15:53:26.673+0000 7f2b4efc1700 -1 osd.0 0 waiting for initial osdmap 2024-04-25T15:53:26.784 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:53:26 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0[44315]: 2024-04-25T15:53:26.676+0000 7f2b4ad76700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:53:26.822 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:26 smithi063 ceph-mon[32413]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:26.822 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:26 smithi063 ceph-mon[32413]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T15:53:26.822 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:26 smithi063 ceph-mon[32413]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T15:53:26.823 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:26 smithi063 ceph-mon[32413]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:53:26.823 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:27.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:26 smithi186 ceph-mon[33258]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:27.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:26 smithi186 ceph-mon[33258]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-25T15:53:27.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:26 smithi186 ceph-mon[33258]: osdmap e6: 1 total, 0 up, 1 in 2024-04-25T15:53:27.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:26 smithi186 ceph-mon[33258]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:53:27.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:27.434 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T15:53:27.434 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T15:53:27.434 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-04-25T15:53:27.434 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-04-25T15:53:27.434 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011319 s, 926 MB/s 2024-04-25T15:53:27.434 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-04-25T15:53:27.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:27 smithi031 ceph-mon[29140]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:53:27.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:27 smithi031 ceph-mon[29140]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T15:53:27.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:27.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:28.069 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:27 smithi063 ceph-mon[32413]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:53:28.069 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:27 smithi063 ceph-mon[32413]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T15:53:28.069 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:28.069 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:28.097 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_4 2024-04-25T15:53:28.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:27 smithi186 ceph-mon[33258]: from='osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:53:28.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:27 smithi186 ceph-mon[33258]: osdmap e7: 1 total, 0 up, 1 in 2024-04-25T15:53:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:28.346 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] boot 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:28.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:29.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:53:29.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:53:29.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:29.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] boot 2024-04-25T15:53:29.050 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T15:53:29.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:29.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:29.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:29.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:53:29.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:29.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:29.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:29.051 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:29.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:53:29.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:53:29.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-25T15:53:29.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: osd.0 [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] boot 2024-04-25T15:53:29.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: osdmap e8: 1 total, 1 up, 1 in 2024-04-25T15:53:29.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-25T15:53:29.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:29.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:29.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:53:29.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:29.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:29.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:29.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:29.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.898+0000 7f5f60283700 1 -- 172.21.15.63:0/1574284724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f58104bb0 msgr2=0x7f5f5810b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:29.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.898+0000 7f5f60283700 1 --2- 172.21.15.63:0/1574284724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f58104bb0 0x7f5f5810b460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5f5000b0a0 tx=0x7f5f5000b3b0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.898+0000 7f5f60283700 1 -- 172.21.15.63:0/1574284724 shutdown_connections 2024-04-25T15:53:29.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.898+0000 7f5f60283700 1 --2- 172.21.15.63:0/1574284724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f58104bb0 0x7f5f5810b460 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.898+0000 7f5f60283700 1 --2- 172.21.15.63:0/1574284724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f581030e0 0x7f5f581034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.898+0000 7f5f60283700 1 --2- 172.21.15.63:0/1574284724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f58104210 0x7f5f58104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.898+0000 7f5f60283700 1 -- 172.21.15.63:0/1574284724 >> 172.21.15.63:0/1574284724 conn(0x7f5f580fe740 msgr2=0x7f5f58100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:29.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.898+0000 7f5f60283700 1 -- 172.21.15.63:0/1574284724 shutdown_connections 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.898+0000 7f5f60283700 1 -- 172.21.15.63:0/1574284724 wait complete. 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f60283700 1 Processor -- start 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f60283700 1 -- start start 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f60283700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f581030e0 0x7f5f58199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f60283700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f58104210 0x7f5f58199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f60283700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f58104bb0 0x7f5f581a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f60283700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5810eb60 con 0x7f5f581030e0 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f60283700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5810e9e0 con 0x7f5f58104210 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f60283700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5810ece0 con 0x7f5f58104bb0 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f5d81e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f58104210 0x7f5f58199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f5d81e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f58104210 0x7f5f58199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42472/0 (socket says 172.21.15.63:42472) 2024-04-25T15:53:29.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f5d81e700 1 -- 172.21.15.63:0/3277627946 learned_addr learned my addr 172.21.15.63:0/3277627946 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f5e820700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f58104bb0 0x7f5f581a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f5e01f700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f581030e0 0x7f5f58199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f5e820700 1 -- 172.21.15.63:0/3277627946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f58104210 msgr2=0x7f5f58199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f5e820700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f58104210 0x7f5f58199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f5e820700 1 -- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f581030e0 msgr2=0x7f5f58199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.899+0000 7f5f5e820700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f581030e0 0x7f5f58199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.900+0000 7f5f5e820700 1 -- 172.21.15.63:0/3277627946 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f581a11a0 con 0x7f5f58104bb0 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.900+0000 7f5f5e820700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f58104bb0 0x7f5f581a0b20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5f50006130 tx=0x7f5f50004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.900+0000 7f5f4f7fe700 1 -- 172.21.15.63:0/3277627946 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50021070 con 0x7f5f58104bb0 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.900+0000 7f5f4f7fe700 1 -- 172.21.15.63:0/3277627946 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f50007c40 con 0x7f5f58104bb0 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.900+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f581a1430 con 0x7f5f58104bb0 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.900+0000 7f5f4f7fe700 1 -- 172.21.15.63:0/3277627946 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f50005060 con 0x7f5f58104bb0 2024-04-25T15:53:29.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.900+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f581a19c0 con 0x7f5f58104bb0 2024-04-25T15:53:29.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.901+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f58051190 con 0x7f5f58104bb0 2024-04-25T15:53:29.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.902+0000 7f5f4f7fe700 1 -- 172.21.15.63:0/3277627946 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f5f5001d450 con 0x7f5f58104bb0 2024-04-25T15:53:29.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.903+0000 7f5f4f7fe700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f440afb70 0x7f5f440b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:29.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.903+0000 7f5f4f7fe700 1 -- 172.21.15.63:0/3277627946 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f5f5000f020 con 0x7f5f58104bb0 2024-04-25T15:53:29.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.903+0000 7f5f5e01f700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f440afb70 0x7f5f440b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:29.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.904+0000 7f5f5e01f700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f440afb70 0x7f5f440b2080 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f54005950 tx=0x7f5f540058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:29.907 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:29.907+0000 7f5f4f7fe700 1 -- 172.21.15.63:0/3277627946 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f5002a020 con 0x7f5f58104bb0 2024-04-25T15:53:29.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:29 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi031 2024-04-25T15:53:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:29 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi031 to 17093M 2024-04-25T15:53:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:29 smithi031 ceph-mon[29140]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T15:53:30.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:29 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi031 2024-04-25T15:53:30.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:29 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi031 to 17093M 2024-04-25T15:53:30.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:29 smithi063 ceph-mon[32413]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T15:53:30.126 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:30.125+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f5f58063ad0 con 0x7f5f440afb70 2024-04-25T15:53:30.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:29 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi031 2024-04-25T15:53:30.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:29 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi031 to 17093M 2024-04-25T15:53:30.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:29 smithi186 ceph-mon[33258]: osdmap e9: 1 total, 1 up, 1 in 2024-04-25T15:53:30.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:30 smithi031 ceph-mon[29140]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:30.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:53:30.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:53:30.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:30.957 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:30 smithi063 ceph-mon[32413]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:30.958 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:53:30.958 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:53:30.958 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:31.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:30 smithi186 ceph-mon[33258]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:31.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:53:31.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:53:31.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:31.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:31 smithi031 ceph-mon[29140]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:53:31.996 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:31 smithi063 ceph-mon[32413]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:53:32.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:31 smithi186 ceph-mon[33258]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:53:32.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:32 smithi031 ceph-mon[29140]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:33.014 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:32 smithi063 ceph-mon[32413]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:33.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:32 smithi186 ceph-mon[33258]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:33.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:33 smithi031 ceph-mon[29140]: from='client.? 172.21.15.63:0/1684427873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7ee32d-a410-468a-ad35-a2113b0bc4d5"}]: dispatch 2024-04-25T15:53:33.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:33 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7ee32d-a410-468a-ad35-a2113b0bc4d5"}]: dispatch 2024-04-25T15:53:33.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:33 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b7ee32d-a410-468a-ad35-a2113b0bc4d5"}]': finished 2024-04-25T15:53:33.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:33 smithi031 ceph-mon[29140]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T15:53:33.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:33 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:33.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:33 smithi031 ceph-mon[29140]: from='client.? 172.21.15.63:0/4264375341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:53:34.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:33 smithi063 ceph-mon[32413]: from='client.? 172.21.15.63:0/1684427873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7ee32d-a410-468a-ad35-a2113b0bc4d5"}]: dispatch 2024-04-25T15:53:34.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:33 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7ee32d-a410-468a-ad35-a2113b0bc4d5"}]: dispatch 2024-04-25T15:53:34.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:33 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b7ee32d-a410-468a-ad35-a2113b0bc4d5"}]': finished 2024-04-25T15:53:34.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:33 smithi063 ceph-mon[32413]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T15:53:34.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:33 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:34.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:33 smithi063 ceph-mon[32413]: from='client.? 172.21.15.63:0/4264375341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:53:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:33 smithi186 ceph-mon[33258]: from='client.? 172.21.15.63:0/1684427873' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7ee32d-a410-468a-ad35-a2113b0bc4d5"}]: dispatch 2024-04-25T15:53:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:33 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4b7ee32d-a410-468a-ad35-a2113b0bc4d5"}]: dispatch 2024-04-25T15:53:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:33 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4b7ee32d-a410-468a-ad35-a2113b0bc4d5"}]': finished 2024-04-25T15:53:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:33 smithi186 ceph-mon[33258]: osdmap e10: 2 total, 1 up, 2 in 2024-04-25T15:53:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:33 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:33 smithi186 ceph-mon[33258]: from='client.? 172.21.15.63:0/4264375341' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:53:35.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:34 smithi186 ceph-mon[33258]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:35.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:34 smithi031 ceph-mon[29140]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:34 smithi063 ceph-mon[32413]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:37.144 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:36 smithi063 ceph-mon[32413]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:37.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:36 smithi031 ceph-mon[29140]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:37.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:36 smithi186 ceph-mon[33258]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:39.069 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:38 smithi063 ceph-mon[32413]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:39.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:38 smithi031 ceph-mon[29140]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:38 smithi186 ceph-mon[33258]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:41.033 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:40 smithi063 ceph-mon[32413]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:41.033 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T15:53:41.033 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:41.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:40 smithi031 ceph-mon[29140]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:41.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T15:53:41.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:41.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:40 smithi186 ceph-mon[33258]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:41.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-25T15:53:41.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:42.059 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:41 smithi063 ceph-mon[32413]: Deploying daemon osd.1 on smithi063 2024-04-25T15:53:42.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:41 smithi031 ceph-mon[29140]: Deploying daemon osd.1 on smithi063 2024-04-25T15:53:42.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:41 smithi186 ceph-mon[33258]: Deploying daemon osd.1 on smithi063 2024-04-25T15:53:43.081 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:42 smithi063 ceph-mon[32413]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:43.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:42 smithi031 ceph-mon[29140]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:43.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:42 smithi186 ceph-mon[33258]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:45.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:44 smithi031 ceph-mon[29140]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:45.249 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:44 smithi063 ceph-mon[32413]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:44 smithi186 ceph-mon[33258]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:46 smithi063 ceph-mon[32413]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:46.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:46 smithi186 ceph-mon[33258]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:46.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:46.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:46.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:46.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:46 smithi031 ceph-mon[29140]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:46.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:46.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:46.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:48.834 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:48 smithi063 ceph-mon[32413]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:48.835 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:48.835 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:48.835 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:48.835 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:48.835 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.895+0000 7f5f4f7fe700 1 -- 172.21.15.63:0/3277627946 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5f58063ad0 con 0x7f5f440afb70 2024-04-25T15:53:48.896 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 1 on host 'smithi063' 2024-04-25T15:53:48.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.897+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f440afb70 msgr2=0x7f5f440b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.897+0000 7f5f60283700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f440afb70 0x7f5f440b2080 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5f54005950 tx=0x7f5f540058e0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.897+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f58104bb0 msgr2=0x7f5f581a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.897+0000 7f5f60283700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f58104bb0 0x7f5f581a0b20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5f50006130 tx=0x7f5f50004d00 comp rx=0 tx=0).stop 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.898+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 shutdown_connections 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.898+0000 7f5f60283700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f440afb70 0x7f5f440b2080 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.898+0000 7f5f60283700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f58104bb0 0x7f5f581a0b20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.898+0000 7f5f60283700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f581030e0 0x7f5f58199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.898+0000 7f5f60283700 1 --2- 172.21.15.63:0/3277627946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f58104210 0x7f5f58199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.898+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 >> 172.21.15.63:0/3277627946 conn(0x7f5f580fe740 msgr2=0x7f5f58107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.898+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 shutdown_connections 2024-04-25T15:53:48.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:53:48.898+0000 7f5f60283700 1 -- 172.21.15.63:0/3277627946 wait complete. 2024-04-25T15:53:48.900 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:48 smithi186 ceph-mon[33258]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:48.900 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:48.900 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:48.900 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:48.900 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:48.900 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:48 smithi031 ceph-mon[29140]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:49.557 DEBUG:teuthology.orchestra.run.smithi063:osd.1> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.1.service 2024-04-25T15:53:49.559 INFO:tasks.cephadm:Deploying osd.2 on smithi186 with /dev/vg_nvme/lv_4... 2024-04-25T15:53:49.559 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-25T15:53:49.817 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:53:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:49 smithi063 ceph-mon[32413]: from='osd.1 [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T15:53:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:49 smithi063 ceph-mon[32413]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T15:53:50.128 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:50.128 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:50.128 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:50.128 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:49 smithi186 ceph-mon[33258]: from='osd.1 [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T15:53:50.128 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:49 smithi186 ceph-mon[33258]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T15:53:50.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:50.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:50.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:53:50.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:49 smithi031 ceph-mon[29140]: from='osd.1 [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T15:53:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:49 smithi031 ceph-mon[29140]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-25T15:53:50.944 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:50 smithi063 ceph-mon[32413]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:50.944 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:50 smithi063 ceph-mon[32413]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T15:53:50.944 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:50 smithi063 ceph-mon[32413]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T15:53:50.944 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:50.944 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:50 smithi063 ceph-mon[32413]: from='osd.1 [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:53:50.944 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:50 smithi063 ceph-mon[32413]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:53:50.944 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:53:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1[37016]: 2024-04-25T15:53:50.890+0000 7f1ea7ccd700 -1 osd.1 0 waiting for initial osdmap 2024-04-25T15:53:50.945 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:53:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1[37016]: 2024-04-25T15:53:50.894+0000 7f1ea3a82700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:53:50.999 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:50 smithi186 ceph-mon[33258]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:50.999 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:50 smithi186 ceph-mon[33258]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T15:53:50.999 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:50 smithi186 ceph-mon[33258]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T15:53:50.999 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:50.999 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:50 smithi186 ceph-mon[33258]: from='osd.1 [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:53:51.000 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:50 smithi186 ceph-mon[33258]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:53:51.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:50 smithi031 ceph-mon[29140]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:51.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:50 smithi031 ceph-mon[29140]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-25T15:53:51.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:50 smithi031 ceph-mon[29140]: osdmap e11: 2 total, 1 up, 2 in 2024-04-25T15:53:51.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:51.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:50 smithi031 ceph-mon[29140]: from='osd.1 [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:53:51.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:50 smithi031 ceph-mon[29140]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:53:51.584 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-25T15:53:51.584 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-25T15:53:51.584 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-25T15:53:51.584 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-25T15:53:51.584 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011362 s, 923 MB/s 2024-04-25T15:53:51.584 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-25T15:53:52.092 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:51 smithi063 ceph-mon[32413]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:53:52.092 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:51 smithi063 ceph-mon[32413]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T15:53:52.092 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:52.092 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:52.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:51 smithi186 ceph-mon[33258]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:53:52.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:51 smithi186 ceph-mon[33258]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T15:53:52.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:52.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:52.149 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-04-25T15:53:52.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:51 smithi031 ceph-mon[29140]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:53:52.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:51 smithi031 ceph-mon[29140]: osdmap e12: 2 total, 1 up, 2 in 2024-04-25T15:53:52.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:52.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:52.392 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:53:53.094 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:53:53.094 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:53:53.094 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: osd.1 [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] boot 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:53.095 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:53:53.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: osd.1 [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] boot 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:53.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:53:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:53:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: osd.1 [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] boot 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: osdmap e13: 2 total, 2 up, 2 in 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:53:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:53:53.839 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.836+0000 7f71f5557700 1 -- 172.21.15.186:0/3488606109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 msgr2=0x7f71f01070f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.836+0000 7f71f5557700 1 --2- 172.21.15.186:0/3488606109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 0x7f71f01070f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f71e0009990 tx=0x7f71e0009ca0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.836+0000 7f71f5557700 1 -- 172.21.15.186:0/3488606109 shutdown_connections 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.836+0000 7f71f5557700 1 --2- 172.21.15.186:0/3488606109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71f0109f90 0x7f71f010c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.836+0000 7f71f5557700 1 --2- 172.21.15.186:0/3488606109 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 0x7f71f01070f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.836+0000 7f71f5557700 1 --2- 172.21.15.186:0/3488606109 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f0107630 0x7f71f0109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.836+0000 7f71f5557700 1 -- 172.21.15.186:0/3488606109 >> 172.21.15.186:0/3488606109 conn(0x7f71f00fe710 msgr2=0x7f71f0100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.836+0000 7f71f5557700 1 -- 172.21.15.186:0/3488606109 shutdown_connections 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.836+0000 7f71f5557700 1 -- 172.21.15.186:0/3488606109 wait complete. 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71f5557700 1 Processor -- start 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71f5557700 1 -- start start 2024-04-25T15:53:53.840 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71f5557700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 0x7f71f019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71f5557700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f0107630 0x7f71f019acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71f5557700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71f0109f90 0x7f71f01a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71f5557700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f71f010fc10 con 0x7f71f0107630 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71f5557700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f71f010fa90 con 0x7f71f0109f90 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71f5557700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71f010fd90 con 0x7f71f0104cd0 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71eeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 0x7f71f019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71eeffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 0x7f71f019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:33610/0 (socket says 172.21.15.186:33610) 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.837+0000 7f71eeffd700 1 -- 172.21.15.186:0/2104716335 learned_addr learned my addr 172.21.15.186:0/2104716335 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71e67fc700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f0107630 0x7f71f019acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71ef7fe700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71f0109f90 0x7f71f01a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71eeffd700 1 -- 172.21.15.186:0/2104716335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71f0109f90 msgr2=0x7f71f01a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71eeffd700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71f0109f90 0x7f71f01a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.841 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71eeffd700 1 -- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f0107630 msgr2=0x7f71f019acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:53:53.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71eeffd700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f0107630 0x7f71f019acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:53:53.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71eeffd700 1 -- 172.21.15.186:0/2104716335 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f01a2260 con 0x7f71f0104cd0 2024-04-25T15:53:53.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71eeffd700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 0x7f71f019a790 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f71e0003800 tx=0x7f71e00047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71ecff9700 1 -- 172.21.15.186:0/2104716335 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0027070 con 0x7f71f0104cd0 2024-04-25T15:53:53.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71ecff9700 1 -- 172.21.15.186:0/2104716335 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71e0004000 con 0x7f71f0104cd0 2024-04-25T15:53:53.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f01a24f0 con 0x7f71f0104cd0 2024-04-25T15:53:53.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71ecff9700 1 -- 172.21.15.186:0/2104716335 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0022d30 con 0x7f71f0104cd0 2024-04-25T15:53:53.842 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.838+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f01a2940 con 0x7f71f0104cd0 2024-04-25T15:53:53.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.839+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71f0051190 con 0x7f71f0104cd0 2024-04-25T15:53:53.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.841+0000 7f71ecff9700 1 -- 172.21.15.186:0/2104716335 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f71e002a070 con 0x7f71f0104cd0 2024-04-25T15:53:53.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.841+0000 7f71ecff9700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f71dc0afaf0 0x7f71dc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:53:53.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.841+0000 7f71ecff9700 1 -- 172.21.15.186:0/2104716335 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f71e00de490 con 0x7f71f0104cd0 2024-04-25T15:53:53.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.841+0000 7f71e67fc700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f71dc0afaf0 0x7f71dc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:53:53.843 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.842+0000 7f71e67fc700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f71dc0afaf0 0x7f71dc0b2000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f71f019b840 tx=0x7f71d800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:53:53.846 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:53.845+0000 7f71ecff9700 1 -- 172.21.15.186:0/2104716335 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f71e005eff0 con 0x7f71f0104cd0 2024-04-25T15:53:54.056 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:53:54.055+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f71f0063ad0 con 0x7f71dc0afaf0 2024-04-25T15:53:54.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:53 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi063 2024-04-25T15:53:54.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:53 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi063 to 17088M 2024-04-25T15:53:54.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:53 smithi031 ceph-mon[29140]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T15:53:54.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:53 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi063 2024-04-25T15:53:54.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:53 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi063 to 17088M 2024-04-25T15:53:54.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:53 smithi063 ceph-mon[32413]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T15:53:54.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:53 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi063 2024-04-25T15:53:54.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:53 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi063 to 17088M 2024-04-25T15:53:54.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:53 smithi186 ceph-mon[33258]: osdmap e14: 2 total, 2 up, 2 in 2024-04-25T15:53:54.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:54 smithi186 ceph-mon[33258]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:53:54.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:54 smithi186 ceph-mon[33258]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:53:54.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:54 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:53:54.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:54 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:53:54.976 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:54 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:55.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:54 smithi031 ceph-mon[29140]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:53:55.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:54 smithi031 ceph-mon[29140]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:53:55.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:54 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:53:55.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:54 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:53:55.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:54 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:55.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:54 smithi063 ceph-mon[32413]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:53:55.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:54 smithi063 ceph-mon[32413]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:53:55.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:54 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:53:55.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:54 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:53:55.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:54 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:53:57.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:56 smithi031 ceph-mon[29140]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:53:57.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:56 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/1914075484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4"}]: dispatch 2024-04-25T15:53:57.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:56 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4"}]: dispatch 2024-04-25T15:53:57.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:56 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4"}]': finished 2024-04-25T15:53:57.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:56 smithi031 ceph-mon[29140]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T15:53:57.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:56 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:53:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:56 smithi063 ceph-mon[32413]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:53:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:56 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/1914075484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4"}]: dispatch 2024-04-25T15:53:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:56 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4"}]: dispatch 2024-04-25T15:53:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:56 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4"}]': finished 2024-04-25T15:53:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:56 smithi063 ceph-mon[32413]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T15:53:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:56 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:53:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:56 smithi186 ceph-mon[33258]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:53:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:56 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/1914075484' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4"}]: dispatch 2024-04-25T15:53:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:56 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4"}]: dispatch 2024-04-25T15:53:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:56 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4"}]': finished 2024-04-25T15:53:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:56 smithi186 ceph-mon[33258]: osdmap e15: 3 total, 2 up, 3 in 2024-04-25T15:53:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:56 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:53:58.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:57 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/318513682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:53:58.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:57 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/318513682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:53:58.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:57 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/318513682' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:53:59.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:53:58 smithi031 ceph-mon[29140]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:53:59.299 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:53:58 smithi186 ceph-mon[33258]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:53:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:53:58 smithi063 ceph-mon[32413]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:01.171 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:00 smithi063 ceph-mon[32413]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:01.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:00 smithi031 ceph-mon[29140]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:01.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:00 smithi186 ceph-mon[33258]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:03.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:02 smithi186 ceph-mon[33258]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:03.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:02 smithi031 ceph-mon[29140]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:03.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:02 smithi063 ceph-mon[32413]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:05.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:04 smithi186 ceph-mon[33258]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:05.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T15:54:05.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:05.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:04 smithi186 ceph-mon[33258]: Deploying daemon osd.2 on smithi186 2024-04-25T15:54:05.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:04 smithi031 ceph-mon[29140]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:05.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T15:54:05.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:05.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:04 smithi031 ceph-mon[29140]: Deploying daemon osd.2 on smithi186 2024-04-25T15:54:05.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:04 smithi063 ceph-mon[32413]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:05.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-25T15:54:05.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:05.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:04 smithi063 ceph-mon[32413]: Deploying daemon osd.2 on smithi186 2024-04-25T15:54:07.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:06 smithi031 ceph-mon[29140]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:07.317 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:06 smithi186 ceph-mon[33258]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:07.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:06 smithi063 ceph-mon[32413]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:09.049 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:08 smithi186 ceph-mon[33258]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:09.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:08 smithi031 ceph-mon[29140]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:09.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:08 smithi063 ceph-mon[32413]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:10.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:10 smithi186 ceph-mon[33258]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:10.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:10.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:10.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:10.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:10 smithi063 ceph-mon[32413]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:10.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:10.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:10.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:10.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:10 smithi031 ceph-mon[29140]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:10.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:10.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:12.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:12 smithi186 ceph-mon[33258]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:12.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:12.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:12.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:12.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:12.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:12 smithi063 ceph-mon[32413]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:12.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:12.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:12 smithi031 ceph-mon[29140]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:12.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:12.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:12.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:12.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:12.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:13.028 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.027+0000 7f71ecff9700 1 -- 172.21.15.186:0/2104716335 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f71f0063ad0 con 0x7f71dc0afaf0 2024-04-25T15:54:13.029 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 2 on host 'smithi186' 2024-04-25T15:54:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f71dc0afaf0 msgr2=0x7f71dc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f71dc0afaf0 0x7f71dc0b2000 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f71f019b840 tx=0x7f71d800a380 comp rx=0 tx=0).stop 2024-04-25T15:54:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 msgr2=0x7f71f019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 0x7f71f019a790 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f71e0003800 tx=0x7f71e00047b0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 shutdown_connections 2024-04-25T15:54:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f71dc0afaf0 0x7f71dc0b2000 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f71f0109f90 0x7f71f01a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71f0104cd0 0x7f71f019a790 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.034 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 --2- 172.21.15.186:0/2104716335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f71f0107630 0x7f71f019acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:13.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.029+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 >> 172.21.15.186:0/2104716335 conn(0x7f71f00fe710 msgr2=0x7f71f01057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:13.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.030+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 shutdown_connections 2024-04-25T15:54:13.035 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:54:13.030+0000 7f71f5557700 1 -- 172.21.15.186:0/2104716335 wait complete. 2024-04-25T15:54:13.659 DEBUG:teuthology.orchestra.run.smithi186:osd.2> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.2.service 2024-04-25T15:54:13.662 INFO:tasks.cephadm:Deploying osd.3 on smithi031 with /dev/vg_nvme/lv_3... 2024-04-25T15:54:13.662 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T15:54:13.920 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:54:14.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:14.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:14.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:14.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:13 smithi031 ceph-mon[29140]: from='osd.2 [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T15:54:14.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:13 smithi031 ceph-mon[29140]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T15:54:14.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:14.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:14.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:14.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:13 smithi063 ceph-mon[32413]: from='osd.2 [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T15:54:14.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:13 smithi063 ceph-mon[32413]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T15:54:14.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:14.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:14.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:14.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:13 smithi186 ceph-mon[33258]: from='osd.2 [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T15:54:14.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:13 smithi186 ceph-mon[33258]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-25T15:54:15.098 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:15 smithi186 ceph-mon[33258]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:15.099 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:15 smithi186 ceph-mon[33258]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T15:54:15.099 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:15 smithi186 ceph-mon[33258]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T15:54:15.099 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:15 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:15.099 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:15 smithi186 ceph-mon[33258]: from='osd.2 [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:54:15.099 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:15 smithi186 ceph-mon[33258]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:54:15.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:15 smithi031 ceph-mon[29140]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:15.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:15 smithi031 ceph-mon[29140]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T15:54:15.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:15 smithi031 ceph-mon[29140]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T15:54:15.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:15 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:15.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:15 smithi031 ceph-mon[29140]: from='osd.2 [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:54:15.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:15 smithi031 ceph-mon[29140]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:54:15.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:15 smithi063 ceph-mon[32413]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:15.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:15 smithi063 ceph-mon[32413]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-25T15:54:15.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:15 smithi063 ceph-mon[32413]: osdmap e16: 3 total, 2 up, 3 in 2024-04-25T15:54:15.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:15 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:15.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:15 smithi063 ceph-mon[32413]: from='osd.2 [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:54:15.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:15 smithi063 ceph-mon[32413]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:54:15.899 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:15 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2[38392]: 2024-04-25T15:54:15.493+0000 7f0fe1564700 -1 osd.2 0 waiting for initial osdmap 2024-04-25T15:54:15.899 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:15 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2[38392]: 2024-04-25T15:54:15.497+0000 7f0fd9311700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:54:15.990 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T15:54:15.991 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T15:54:15.991 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-25T15:54:15.991 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-25T15:54:15.991 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113067 s, 927 MB/s 2024-04-25T15:54:15.991 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-25T15:54:16.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:16 smithi031 ceph-mon[29140]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:54:16.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:16 smithi031 ceph-mon[29140]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T15:54:16.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:16.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:16.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:16 smithi031 ceph-mon[29140]: from='osd.2 ' entity='osd.2' 2024-04-25T15:54:16.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:16 smithi063 ceph-mon[32413]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:54:16.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:16 smithi063 ceph-mon[32413]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T15:54:16.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:16.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:16.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:16 smithi063 ceph-mon[32413]: from='osd.2 ' entity='osd.2' 2024-04-25T15:54:16.394 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:16 smithi186 ceph-mon[33258]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:54:16.394 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:16 smithi186 ceph-mon[33258]: osdmap e17: 3 total, 2 up, 3 in 2024-04-25T15:54:16.394 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:16.394 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:16.394 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:16 smithi186 ceph-mon[33258]: from='osd.2 ' entity='osd.2' 2024-04-25T15:54:16.703 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2024-04-25T15:54:16.961 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:54:17.301 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: osd.2 [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] boot 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.302 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:17.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:17.303 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:54:17.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:54:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: osd.2 [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] boot 2024-04-25T15:54:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T15:54:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:54:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:17.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:17.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: osd.2 [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] boot 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: osdmap e18: 3 total, 3 up, 3 in 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:17.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:18.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:18 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi186 2024-04-25T15:54:18.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:18 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-25T15:54:18.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:18 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T15:54:18.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:18 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi186 2024-04-25T15:54:18.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:18 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-25T15:54:18.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:18 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T15:54:18.401 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:18 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi186 2024-04-25T15:54:18.401 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:18 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi186 to 17088M 2024-04-25T15:54:18.401 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:18 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.657+0000 7f2dddcfe700 1 -- 172.21.15.31:0/1551335455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8105330 msgr2=0x7f2dd8105740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.657+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/1551335455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8105330 0x7f2dd8105740 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2dc0009990 tx=0x7f2dc0009ca0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.657+0000 7f2dddcfe700 1 -- 172.21.15.31:0/1551335455 shutdown_connections 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.657+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/1551335455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dd8104200 0x7f2dd8104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.657+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/1551335455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8105cd0 0x7f2dd810c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.657+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/1551335455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8105330 0x7f2dd8105740 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.657+0000 7f2dddcfe700 1 -- 172.21.15.31:0/1551335455 >> 172.21.15.31:0/1551335455 conn(0x7f2dd80ff840 msgr2=0x7f2dd8101cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.657+0000 7f2dddcfe700 1 -- 172.21.15.31:0/1551335455 shutdown_connections 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.657+0000 7f2dddcfe700 1 -- 172.21.15.31:0/1551335455 wait complete. 2024-04-25T15:54:18.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.658+0000 7f2dddcfe700 1 Processor -- start 2024-04-25T15:54:18.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.658+0000 7f2dddcfe700 1 -- start start 2024-04-25T15:54:18.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dddcfe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8104200 0x7f2dd819c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:18.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dddcfe700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dd8105330 0x7f2dd819cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:18.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dddcfe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8105cd0 0x7f2dd81a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:18.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dddcfe700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8111ec0 con 0x7f2dd8104200 2024-04-25T15:54:18.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dddcfe700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8111d40 con 0x7f2dd8105330 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8104200 0x7f2dd819c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dddcfe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2dd8112040 con 0x7f2dd8105cd0 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd77fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8104200 0x7f2dd819c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43710/0 (socket says 172.21.15.31:43710) 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd77fe700 1 -- 172.21.15.31:0/4238646620 learned_addr learned my addr 172.21.15.31:0/4238646620 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd6ffd700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dd8105330 0x7f2dd819cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd7fff700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8105cd0 0x7f2dd81a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd77fe700 1 -- 172.21.15.31:0/4238646620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8105cd0 msgr2=0x7f2dd81a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd77fe700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8105cd0 0x7f2dd81a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd77fe700 1 -- 172.21.15.31:0/4238646620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dd8105330 msgr2=0x7f2dd819cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd77fe700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dd8105330 0x7f2dd819cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd77fe700 1 -- 172.21.15.31:0/4238646620 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dd81a4530 con 0x7f2dd8104200 2024-04-25T15:54:18.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd7fff700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8105cd0 0x7f2dd81a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:54:18.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd77fe700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8104200 0x7f2dd819c9d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000ef70 tx=0x7f2dd000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:18.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.659+0000 7f2dd4ff9700 1 -- 172.21.15.31:0/4238646620 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd00092e0 con 0x7f2dd8104200 2024-04-25T15:54:18.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.660+0000 7f2dd4ff9700 1 -- 172.21.15.31:0/4238646620 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2dd0003a00 con 0x7f2dd8104200 2024-04-25T15:54:18.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.660+0000 7f2dd4ff9700 1 -- 172.21.15.31:0/4238646620 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0004580 con 0x7f2dd8104200 2024-04-25T15:54:18.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.660+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dd81a47c0 con 0x7f2dd8104200 2024-04-25T15:54:18.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.660+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dd81a4cb0 con 0x7f2dd8104200 2024-04-25T15:54:18.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.661+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dd8051190 con 0x7f2dd8104200 2024-04-25T15:54:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.662+0000 7f2dd4ff9700 1 -- 172.21.15.31:0/4238646620 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f2dd001b030 con 0x7f2dd8104200 2024-04-25T15:54:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.663+0000 7f2dd4ff9700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2dc40afb70 0x7f2dc40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.663+0000 7f2dd6ffd700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2dc40afb70 0x7f2dc40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.663+0000 7f2dd4ff9700 1 -- 172.21.15.31:0/4238646620 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 3226+0+0 (secure 0 0 0) 0x7f2dd00dd9e0 con 0x7f2dd8104200 2024-04-25T15:54:18.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.664+0000 7f2dd6ffd700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2dc40afb70 0x7f2dc40b2080 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000b2e0 tx=0x7f2dc0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:18.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.667+0000 7f2dd4ff9700 1 -- 172.21.15.31:0/4238646620 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2dd0010040 con 0x7f2dd8104200 2024-04-25T15:54:18.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:18.883+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f2dd8063ad0 con 0x7f2dc40afb70 2024-04-25T15:54:19.294 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 ceph-mon[29140]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:19.294 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T15:54:19.294 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 ceph-mon[29140]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T15:54:19.294 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T15:54:19.294 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:54:19.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:54:19.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:19.295 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48635]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T15:54:19.295 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48635]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.295 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48635]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.295 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48635]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.295 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48638]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T15:54:19.295 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48638]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.295 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48638]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.295 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48638]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.332 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 ceph-mon[33258]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:19.332 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T15:54:19.332 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 ceph-mon[33258]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T15:54:19.332 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T15:54:19.332 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:54:19.332 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:54:19.332 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:19.333 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42290]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T15:54:19.333 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42290]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.333 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42290]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.333 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42290]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.333 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42293]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T15:54:19.333 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42293]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.333 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42293]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.333 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42293]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 ceph-mon[32413]: pgmap v82: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-25T15:54:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 ceph-mon[32413]: osdmap e19: 3 total, 3 up, 3 in 2024-04-25T15:54:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T15:54:19.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:54:19.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:54:19.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:19.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41142]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T15:54:19.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41142]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41142]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41142]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41145]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T15:54:19.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41145]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41145]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41145]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42296]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T15:54:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42296]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42296]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42296]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42299]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T15:54:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42299]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42299]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:19 smithi186 sudo[42299]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.677 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48641]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T15:54:19.677 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48641]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.677 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48641]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.677 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48641]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.677 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48644]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T15:54:19.677 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48644]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.677 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48644]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.677 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:19 smithi031 sudo[48644]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41148]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-25T15:54:19.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41148]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41148]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41148]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:19.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41151]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-25T15:54:19.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41151]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-25T15:54:19.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41151]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-25T15:54:19.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:19 smithi063 sudo[41151]: pam_unix(sudo:session): session closed for user root 2024-04-25T15:54:20.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:20.227+0000 7f2dd4ff9700 1 -- 172.21.15.31:0/4238646620 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2dd0012050 con 0x7f2dd8104200 2024-04-25T15:54:20.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:54:20.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:54:20.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:54:20.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T15:54:20.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T15:54:20.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:54:20.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:54:20.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:54:20.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:20 smithi063 ceph-mon[32413]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T15:54:20.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:54:20.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T15:54:20.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T15:54:20.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:20.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:20.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:20.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T15:54:20.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:54:20.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:20 smithi186 ceph-mon[33258]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T15:54:20.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: osdmap e20: 3 total, 3 up, 3 in 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T15:54:20.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:54:20.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:54:20.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:54:20.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T15:54:20.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-25T15:54:20.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-25T15:54:20.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-25T15:54:20.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-25T15:54:20.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:20 smithi031 ceph-mon[29140]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-25T15:54:21.075 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:21 smithi031 ceph-mon[29140]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:21.076 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:21 smithi031 ceph-mon[29140]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T15:54:21.076 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:21 smithi031 ceph-mon[29140]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T15:54:21.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:21 smithi063 ceph-mon[32413]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:21.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:21 smithi063 ceph-mon[32413]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T15:54:21.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:21 smithi063 ceph-mon[32413]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T15:54:21.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:21 smithi186 ceph-mon[33258]: pgmap v85: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:21.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:21 smithi186 ceph-mon[33258]: osdmap e21: 3 total, 3 up, 3 in 2024-04-25T15:54:21.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:21 smithi186 ceph-mon[33258]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-25T15:54:22.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:22 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1352781529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c56990d-8026-4d79-bb6f-f1ee793e6a66"}]: dispatch 2024-04-25T15:54:22.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:22 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1352781529' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c56990d-8026-4d79-bb6f-f1ee793e6a66"}]': finished 2024-04-25T15:54:22.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:22 smithi063 ceph-mon[32413]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T15:54:22.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:22.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:22 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1352781529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c56990d-8026-4d79-bb6f-f1ee793e6a66"}]: dispatch 2024-04-25T15:54:22.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:22 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1352781529' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c56990d-8026-4d79-bb6f-f1ee793e6a66"}]': finished 2024-04-25T15:54:22.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:22 smithi186 ceph-mon[33258]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T15:54:22.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:22.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:22 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1352781529' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5c56990d-8026-4d79-bb6f-f1ee793e6a66"}]: dispatch 2024-04-25T15:54:22.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:22 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1352781529' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5c56990d-8026-4d79-bb6f-f1ee793e6a66"}]': finished 2024-04-25T15:54:22.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:22 smithi031 ceph-mon[29140]: osdmap e22: 4 total, 3 up, 4 in 2024-04-25T15:54:22.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:23.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:23 smithi063 ceph-mon[32413]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:23.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:23 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3866577874' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:54:23.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:23 smithi186 ceph-mon[33258]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:23.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:23 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3866577874' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:54:23.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:23 smithi031 ceph-mon[29140]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:23.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:23 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3866577874' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:54:25.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:25 smithi063 ceph-mon[32413]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:25.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:25 smithi186 ceph-mon[33258]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:25.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:25 smithi031 ceph-mon[29140]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:27.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:27 smithi031 ceph-mon[29140]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:27.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:27 smithi063 ceph-mon[32413]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:27.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:27 smithi186 ceph-mon[33258]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:29.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:29 smithi031 ceph-mon[29140]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:29.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T15:54:29.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:29.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:29 smithi063 ceph-mon[32413]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:29.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:29 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T15:54:29.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:29 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:29.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:29 smithi186 ceph-mon[33258]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:29.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:29 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-25T15:54:29.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:29 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:30.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:30 smithi031 ceph-mon[29140]: Deploying daemon osd.3 on smithi031 2024-04-25T15:54:30.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:30 smithi063 ceph-mon[32413]: Deploying daemon osd.3 on smithi031 2024-04-25T15:54:30.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:30 smithi186 ceph-mon[33258]: Deploying daemon osd.3 on smithi031 2024-04-25T15:54:31.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:31 smithi031 ceph-mon[29140]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:31 smithi063 ceph-mon[32413]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:31.390 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:31 smithi186 ceph-mon[33258]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:33.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:33 smithi063 ceph-mon[32413]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:33.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:33 smithi186 ceph-mon[33258]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:33.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:33 smithi031 ceph-mon[29140]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:35 smithi063 ceph-mon[32413]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:35 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:35 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:35 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:35.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:35 smithi186 ceph-mon[33258]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:35.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:35 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:35.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:35 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:35 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:35.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:35 smithi031 ceph-mon[29140]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:35.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:35 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:35.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:35 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:35.438 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:35 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:37.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:37 smithi063 ceph-mon[32413]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:37.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:37 smithi031 ceph-mon[29140]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:37.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:37 smithi186 ceph-mon[33258]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:38.746 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 3 on host 'smithi031' 2024-04-25T15:54:38.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.745+0000 7f2dd4ff9700 1 -- 172.21.15.31:0/4238646620 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2dd8063ad0 con 0x7f2dc40afb70 2024-04-25T15:54:38.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2dc40afb70 msgr2=0x7f2dc40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2dc40afb70 0x7f2dc40b2080 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2dc000b2e0 tx=0x7f2dc0005fb0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8104200 msgr2=0x7f2dd819c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8104200 0x7f2dd819c9d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000ef70 tx=0x7f2dd000c3a0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 shutdown_connections 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2dc40afb70 0x7f2dc40b2080 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2dd8105cd0 0x7f2dd81a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2dd8104200 0x7f2dd819c9d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 --2- 172.21.15.31:0/4238646620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2dd8105330 0x7f2dd819cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 >> 172.21.15.31:0/4238646620 conn(0x7f2dd80ff840 msgr2=0x7f2dd8108560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 shutdown_connections 2024-04-25T15:54:38.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:54:38.748+0000 7f2dddcfe700 1 -- 172.21.15.31:0/4238646620 wait complete. 2024-04-25T15:54:38.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T15:54:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:39.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:39.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:39.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:39.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T15:54:39.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:39.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-25T15:54:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:54:39.430 DEBUG:teuthology.orchestra.run.smithi031:osd.3> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.3.service 2024-04-25T15:54:39.434 INFO:tasks.cephadm:Deploying osd.4 on smithi063 with /dev/vg_nvme/lv_3... 2024-04-25T15:54:39.435 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T15:54:39.689 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T15:54:40.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:40 smithi063 ceph-mon[32413]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T15:54:40.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:40 smithi063 ceph-mon[32413]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T15:54:40.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:40 smithi063 ceph-mon[32413]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:54:40.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:40.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:40 smithi186 ceph-mon[33258]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T15:54:40.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:40 smithi186 ceph-mon[33258]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T15:54:40.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:40 smithi186 ceph-mon[33258]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:54:40.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:40.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:40 smithi031 ceph-mon[29140]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-25T15:54:40.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:40 smithi031 ceph-mon[29140]: osdmap e23: 4 total, 3 up, 4 in 2024-04-25T15:54:40.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:40 smithi031 ceph-mon[29140]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:54:40.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:40.450 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 15:54:40 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3[50849]: 2024-04-25T15:54:40.082+0000 7f02bbd07700 -1 osd.3 0 waiting for initial osdmap 2024-04-25T15:54:40.450 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 15:54:40 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3[50849]: 2024-04-25T15:54:40.088+0000 7f02b7abc700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:54:41.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:41 smithi186 ceph-mon[33258]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:41.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:41 smithi186 ceph-mon[33258]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:54:41.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:41 smithi186 ceph-mon[33258]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T15:54:41.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:41 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:41.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:41 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:41.173 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:41 smithi031 ceph-mon[29140]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:41.173 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:41 smithi031 ceph-mon[29140]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:54:41.173 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:41 smithi031 ceph-mon[29140]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T15:54:41.173 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:41.173 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:41.330 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:41 smithi063 ceph-mon[32413]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-25T15:54:41.331 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:41 smithi063 ceph-mon[32413]: from='osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:54:41.331 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:41 smithi063 ceph-mon[32413]: osdmap e24: 4 total, 3 up, 4 in 2024-04-25T15:54:41.331 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:41 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:41.331 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:41 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:41.788 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T15:54:41.788 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T15:54:41.788 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-04-25T15:54:41.788 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-04-25T15:54:41.788 INFO:teuthology.orchestra.run.smithi063.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115088 s, 911 MB/s 2024-04-25T15:54:41.788 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-04-25T15:54:42.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:42 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:54:42.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:42 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:54:42.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:42 smithi063 ceph-mon[32413]: osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] boot 2024-04-25T15:54:42.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:42 smithi063 ceph-mon[32413]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T15:54:42.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:42 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:42.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:42 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:54:42.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:42 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:54:42.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:42 smithi186 ceph-mon[33258]: osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] boot 2024-04-25T15:54:42.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:42 smithi186 ceph-mon[33258]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T15:54:42.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:42 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:42.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:42 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:54:42.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:42 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:54:42.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:42 smithi031 ceph-mon[29140]: osd.3 [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] boot 2024-04-25T15:54:42.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:42 smithi031 ceph-mon[29140]: osdmap e25: 4 total, 4 up, 4 in 2024-04-25T15:54:42.435 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:42 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-25T15:54:42.467 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_3 2024-04-25T15:54:42.714 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T15:54:43.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:43.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi031 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi031 to 8546M 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:43 smithi063 ceph-mon[32413]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T15:54:43.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi031 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi031 to 8546M 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:43 smithi186 ceph-mon[33258]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T15:54:43.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:43.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: osdmap e26: 4 total, 4 up, 4 in 2024-04-25T15:54:43.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi031 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi031 to 8546M 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:54:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:43 smithi031 ceph-mon[29140]: osdmap e27: 4 total, 4 up, 4 in 2024-04-25T15:54:44.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.314+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/3363373623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104210 msgr2=0x7fa9c0104620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.314+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/3363373623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104210 0x7fa9c0104620 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009990 tx=0x7fa9a8009ca0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.314+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/3363373623 shutdown_connections 2024-04-25T15:54:44.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.314+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/3363373623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9c0104bb0 0x7fa9c010b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.314+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/3363373623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9c01030e0 0x7fa9c01034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.314+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/3363373623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104210 0x7fa9c0104620 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.314+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/3363373623 >> 172.21.15.63:0/3363373623 conn(0x7fa9c00fe740 msgr2=0x7fa9c0100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:54:44.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.314+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/3363373623 shutdown_connections 2024-04-25T15:54:44.317 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.314+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/3363373623 wait complete. 2024-04-25T15:54:44.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.315+0000 7fa9c4b2b700 1 Processor -- start 2024-04-25T15:54:44.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9c4b2b700 1 -- start start 2024-04-25T15:54:44.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9c4b2b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9c01030e0 0x7fa9c01996f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9c4b2b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9c0104210 0x7fa9c0199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9c4b2b700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104bb0 0x7fa9c01a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.319 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9c4b2b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c010ebd0 con 0x7fa9c0104210 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9c4b2b700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c010ea50 con 0x7fa9c0104bb0 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9c4b2b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa9c010ed50 con 0x7fa9c01030e0 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bed9d700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104bb0 0x7fa9c01a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bed9d700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104bb0 0x7fa9c01a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:60732/0 (socket says 172.21.15.63:60732) 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bdd9b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9c0104210 0x7fa9c0199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bed9d700 1 -- 172.21.15.63:0/1723087746 learned_addr learned my addr 172.21.15.63:0/1723087746 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9be59c700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9c01030e0 0x7fa9c01996f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bdd9b700 1 -- 172.21.15.63:0/1723087746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9c01030e0 msgr2=0x7fa9c01996f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bdd9b700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9c01030e0 0x7fa9c01996f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bdd9b700 1 -- 172.21.15.63:0/1723087746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104bb0 msgr2=0x7fa9c01a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bdd9b700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104bb0 0x7fa9c01a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:54:44.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bdd9b700 1 -- 172.21.15.63:0/1723087746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9c01a1250 con 0x7fa9c0104210 2024-04-25T15:54:44.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.316+0000 7fa9bed9d700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104bb0 0x7fa9c01a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:54:44.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.317+0000 7fa9bdd9b700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9c0104210 0x7fa9c0199c30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800b6c0 tx=0x7fa9a80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:44.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.317+0000 7fa9b77fe700 1 -- 172.21.15.63:0/1723087746 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8026070 con 0x7fa9c0104210 2024-04-25T15:54:44.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.317+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9c006b4e0 con 0x7fa9c0104210 2024-04-25T15:54:44.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.317+0000 7fa9b77fe700 1 -- 172.21.15.63:0/1723087746 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9a8021410 con 0x7fa9c0104210 2024-04-25T15:54:44.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.317+0000 7fa9b77fe700 1 -- 172.21.15.63:0/1723087746 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a8017650 con 0x7fa9c0104210 2024-04-25T15:54:44.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.317+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9c006ba60 con 0x7fa9c0104210 2024-04-25T15:54:44.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.318+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9c0051190 con 0x7fa9c0104210 2024-04-25T15:54:44.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.319+0000 7fa9b77fe700 1 -- 172.21.15.63:0/1723087746 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa9a8029070 con 0x7fa9c0104210 2024-04-25T15:54:44.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.320+0000 7fa9b77fe700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9ac0afb80 0x7fa9ac0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:54:44.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.320+0000 7fa9b77fe700 1 -- 172.21.15.63:0/1723087746 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3669+0+0 (secure 0 0 0) 0x7fa9a80def00 con 0x7fa9c0104210 2024-04-25T15:54:44.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.320+0000 7fa9be59c700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9ac0afb80 0x7fa9ac0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:54:44.322 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.321+0000 7fa9be59c700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9ac0afb80 0x7fa9ac0b2090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0104110 tx=0x7fa9b800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:54:44.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.324+0000 7fa9b77fe700 1 -- 172.21.15.63:0/1723087746 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9a8030020 con 0x7fa9c0104210 2024-04-25T15:54:44.325 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:44 smithi063 ceph-mon[32413]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:44.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:44 smithi186 ceph-mon[33258]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:44.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:44 smithi031 ceph-mon[29140]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:44.545 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:54:44.544+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa9c006bcc0 con 0x7fa9ac0afb80 2024-04-25T15:54:45.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:45 smithi063 ceph-mon[32413]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:54:45.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:45 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:54:45.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:45 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:54:45.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:45 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:45 smithi186 ceph-mon[33258]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:54:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:45 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:54:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:45 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:54:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:45 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:45.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:45 smithi031 ceph-mon[29140]: from='client.14316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:54:45.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:45 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:54:45.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:45 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:54:45.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:45 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:46.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:46 smithi063 ceph-mon[32413]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:46.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:46 smithi186 ceph-mon[33258]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:46.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:46 smithi031 ceph-mon[29140]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:48.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:48 smithi063 ceph-mon[32413]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:48.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:48 smithi063 ceph-mon[32413]: from='client.? 172.21.15.63:0/1876593738' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e948c581-682c-4d78-9e92-d70304b5076e"}]: dispatch 2024-04-25T15:54:48.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:48 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e948c581-682c-4d78-9e92-d70304b5076e"}]: dispatch 2024-04-25T15:54:48.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:48 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e948c581-682c-4d78-9e92-d70304b5076e"}]': finished 2024-04-25T15:54:48.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:48 smithi063 ceph-mon[32413]: osdmap e28: 5 total, 4 up, 5 in 2024-04-25T15:54:48.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:54:48.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:48 smithi063 ceph-mon[32413]: from='client.? 172.21.15.63:0/4127879080' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:54:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:48 smithi186 ceph-mon[33258]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:48 smithi186 ceph-mon[33258]: from='client.? 172.21.15.63:0/1876593738' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e948c581-682c-4d78-9e92-d70304b5076e"}]: dispatch 2024-04-25T15:54:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:48 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e948c581-682c-4d78-9e92-d70304b5076e"}]: dispatch 2024-04-25T15:54:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:48 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e948c581-682c-4d78-9e92-d70304b5076e"}]': finished 2024-04-25T15:54:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:48 smithi186 ceph-mon[33258]: osdmap e28: 5 total, 4 up, 5 in 2024-04-25T15:54:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:54:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:48 smithi186 ceph-mon[33258]: from='client.? 172.21.15.63:0/4127879080' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:54:48.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:48 smithi031 ceph-mon[29140]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:54:48.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:48 smithi031 ceph-mon[29140]: from='client.? 172.21.15.63:0/1876593738' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e948c581-682c-4d78-9e92-d70304b5076e"}]: dispatch 2024-04-25T15:54:48.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:48 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e948c581-682c-4d78-9e92-d70304b5076e"}]: dispatch 2024-04-25T15:54:48.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:48 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e948c581-682c-4d78-9e92-d70304b5076e"}]': finished 2024-04-25T15:54:48.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:48 smithi031 ceph-mon[29140]: osdmap e28: 5 total, 4 up, 5 in 2024-04-25T15:54:48.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:54:48.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:48 smithi031 ceph-mon[29140]: from='client.? 172.21.15.63:0/4127879080' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:54:50.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:50 smithi063 ceph-mon[32413]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T15:54:50.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:50 smithi186 ceph-mon[33258]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T15:54:50.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:50 smithi031 ceph-mon[29140]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-04-25T15:54:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:52 smithi063 ceph-mon[32413]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-25T15:54:52.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:52 smithi186 ceph-mon[33258]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-25T15:54:52.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:52 smithi031 ceph-mon[29140]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-25T15:54:54.574 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:54 smithi063 ceph-mon[32413]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:54:54.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:54 smithi186 ceph-mon[33258]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:54:54.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:54 smithi031 ceph-mon[29140]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:54:55.593 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T15:54:55.593 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:55.593 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:55 smithi063 ceph-mon[32413]: Deploying daemon osd.4 on smithi063 2024-04-25T15:54:55.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T15:54:55.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:55.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:55 smithi186 ceph-mon[33258]: Deploying daemon osd.4 on smithi063 2024-04-25T15:54:55.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-25T15:54:55.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:54:55.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:55 smithi031 ceph-mon[29140]: Deploying daemon osd.4 on smithi063 2024-04-25T15:54:56.540 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:56 smithi063 ceph-mon[32413]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:54:56.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:56 smithi186 ceph-mon[33258]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:54:56.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:56 smithi031 ceph-mon[29140]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:54:58.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:54:58 smithi063 ceph-mon[32413]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:54:58.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:54:58 smithi186 ceph-mon[33258]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:54:58.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:54:58 smithi031 ceph-mon[29140]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:55:00.519 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:00 smithi063 ceph-mon[32413]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-25T15:55:00.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:00 smithi186 ceph-mon[33258]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-25T15:55:00.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:00 smithi031 ceph-mon[29140]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-25T15:55:02.050 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:02.050 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:02.050 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:02.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:02.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:02.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:02.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:02.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:02.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:03.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:02 smithi063 ceph-mon[32413]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:03.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:02 smithi186 ceph-mon[33258]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:03.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:02 smithi031 ceph-mon[29140]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:04.818 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 4 on host 'smithi063' 2024-04-25T15:55:04.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.817+0000 7fa9b77fe700 1 -- 172.21.15.63:0/1723087746 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa9c006bcc0 con 0x7fa9ac0afb80 2024-04-25T15:55:04.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.819+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9ac0afb80 msgr2=0x7fa9ac0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:04.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.819+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9ac0afb80 0x7fa9ac0b2090 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0104110 tx=0x7fa9b800b040 comp rx=0 tx=0).stop 2024-04-25T15:55:04.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.819+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9c0104210 msgr2=0x7fa9c0199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:04.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.819+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9c0104210 0x7fa9c0199c30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9a800b6c0 tx=0x7fa9a80047b0 comp rx=0 tx=0).stop 2024-04-25T15:55:04.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.820+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 shutdown_connections 2024-04-25T15:55:04.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.820+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9ac0afb80 0x7fa9ac0b2090 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:04.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.820+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9c0104bb0 0x7fa9c01a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:04.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.820+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9c01030e0 0x7fa9c01996f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:04.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.820+0000 7fa9c4b2b700 1 --2- 172.21.15.63:0/1723087746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa9c0104210 0x7fa9c0199c30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:04.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.820+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 >> 172.21.15.63:0/1723087746 conn(0x7fa9c00fe740 msgr2=0x7fa9c0107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:04.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.820+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 shutdown_connections 2024-04-25T15:55:04.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:55:04.820+0000 7fa9c4b2b700 1 -- 172.21.15.63:0/1723087746 wait complete. 2024-04-25T15:55:05.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:04 smithi063 ceph-mon[32413]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:05.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:05.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:05.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:05.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:05.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:04 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:05.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:04 smithi063 ceph-mon[32413]: from='osd.4 [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T15:55:05.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:04 smithi063 ceph-mon[32413]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T15:55:05.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:04 smithi186 ceph-mon[33258]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:05.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:05.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:05.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:05.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:05.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:04 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:05.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:04 smithi186 ceph-mon[33258]: from='osd.4 [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T15:55:05.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:04 smithi186 ceph-mon[33258]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T15:55:05.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:04 smithi031 ceph-mon[29140]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:05.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:05.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:05.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:05.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:05.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:04 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:05.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:04 smithi031 ceph-mon[29140]: from='osd.4 [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T15:55:05.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:04 smithi031 ceph-mon[29140]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-25T15:55:05.464 DEBUG:teuthology.orchestra.run.smithi063:osd.4> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.4.service 2024-04-25T15:55:05.467 INFO:tasks.cephadm:Deploying osd.5 on smithi186 with /dev/vg_nvme/lv_3... 2024-04-25T15:55:05.467 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-25T15:55:05.714 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:55:06.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:05 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:06.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:05 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:06.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:05 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:06.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:05 smithi186 ceph-mon[33258]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T15:55:06.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:05 smithi186 ceph-mon[33258]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T15:55:06.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:05 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:06.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:05 smithi186 ceph-mon[33258]: from='osd.4 [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:55:06.021 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:05 smithi186 ceph-mon[33258]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:55:06.035 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:06.035 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:06.035 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:06.035 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-mon[32413]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T15:55:06.035 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-mon[32413]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T15:55:06.035 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:06.035 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-mon[32413]: from='osd.4 [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:55:06.035 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-mon[32413]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:55:06.035 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4[43959]: 2024-04-25T15:55:05.810+0000 7f9adb871700 -1 osd.4 0 waiting for initial osdmap 2024-04-25T15:55:06.035 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 15:55:05 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4[43959]: 2024-04-25T15:55:05.817+0000 7f9ad7626700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:55:06.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:05 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:06.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:05 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:06.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:05 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:06.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:05 smithi031 ceph-mon[29140]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-25T15:55:06.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:05 smithi031 ceph-mon[29140]: osdmap e29: 5 total, 4 up, 5 in 2024-04-25T15:55:06.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:05 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:06.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:05 smithi031 ceph-mon[29140]: from='osd.4 [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:55:06.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:05 smithi031 ceph-mon[29140]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:55:07.018 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:06 smithi063 ceph-mon[32413]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:07.018 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:06 smithi063 ceph-mon[32413]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:55:07.018 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:06 smithi063 ceph-mon[32413]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T15:55:07.018 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:07.018 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:07.073 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:06 smithi186 ceph-mon[33258]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:07.074 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:06 smithi186 ceph-mon[33258]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:55:07.074 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:06 smithi186 ceph-mon[33258]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T15:55:07.074 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:07.074 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:06 smithi031 ceph-mon[29140]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:06 smithi031 ceph-mon[29140]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:55:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:06 smithi031 ceph-mon[29140]: osdmap e30: 5 total, 4 up, 5 in 2024-04-25T15:55:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:07.628 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-25T15:55:07.628 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-25T15:55:07.628 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-25T15:55:07.628 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-25T15:55:07.629 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116902 s, 897 MB/s 2024-04-25T15:55:07.629 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-25T15:55:07.965 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:07 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:55:07.965 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:07 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:55:07.965 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:07 smithi186 ceph-mon[33258]: osd.4 [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] boot 2024-04-25T15:55:07.965 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:07 smithi186 ceph-mon[33258]: osdmap e31: 5 total, 5 up, 5 in 2024-04-25T15:55:07.965 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:07 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:08.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:07 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:55:08.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:07 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:55:08.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:07 smithi063 ceph-mon[32413]: osd.4 [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] boot 2024-04-25T15:55:08.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:07 smithi063 ceph-mon[32413]: osdmap e31: 5 total, 5 up, 5 in 2024-04-25T15:55:08.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:07 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:08.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:07 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:55:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:07 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:55:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:07 smithi031 ceph-mon[29140]: osd.4 [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] boot 2024-04-25T15:55:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:07 smithi031 ceph-mon[29140]: osdmap e31: 5 total, 5 up, 5 in 2024-04-25T15:55:08.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:07 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-25T15:55:08.295 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-04-25T15:55:08.537 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:55:08.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:08.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T15:55:08.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:09.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T15:55:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:09.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-25T15:55:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: osdmap e32: 5 total, 5 up, 5 in 2024-04-25T15:55:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:09.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:09 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi063 2024-04-25T15:55:09.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:09 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi063 to 8544M 2024-04-25T15:55:10.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:09 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi063 2024-04-25T15:55:10.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:09 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi063 to 8544M 2024-04-25T15:55:10.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.183+0000 7f848eeba700 1 -- 172.21.15.186:0/257907856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488103120 msgr2=0x7f8488103530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.183+0000 7f848eeba700 1 --2- 172.21.15.186:0/257907856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488103120 0x7f8488103530 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f847c009990 tx=0x7f847c009ca0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.183+0000 7f848eeba700 1 -- 172.21.15.186:0/257907856 shutdown_connections 2024-04-25T15:55:10.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.183+0000 7f848eeba700 1 --2- 172.21.15.186:0/257907856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8488104bf0 0x7f848810b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.183+0000 7f848eeba700 1 --2- 172.21.15.186:0/257907856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488103120 0x7f8488103530 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.183+0000 7f848eeba700 1 --2- 172.21.15.186:0/257907856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8488104250 0x7f8488104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.186 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.183+0000 7f848eeba700 1 -- 172.21.15.186:0/257907856 >> 172.21.15.186:0/257907856 conn(0x7f84880fe760 msgr2=0x7f8488100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.183+0000 7f848eeba700 1 -- 172.21.15.186:0/257907856 shutdown_connections 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.183+0000 7f848eeba700 1 -- 172.21.15.186:0/257907856 wait complete. 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848eeba700 1 Processor -- start 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848eeba700 1 -- start start 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848eeba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8488103120 0x7f8488199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848eeba700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8488104250 0x7f8488199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848eeba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488104bf0 0x7f84881a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848eeba700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f848810ebe0 con 0x7f8488103120 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848eeba700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f848810ea60 con 0x7f8488104250 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848eeba700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f848810ed60 con 0x7f8488104bf0 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848d457700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488104bf0 0x7f84881a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848d457700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488104bf0 0x7f84881a0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:47378/0 (socket says 172.21.15.186:47378) 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848d457700 1 -- 172.21.15.186:0/1780397815 learned_addr learned my addr 172.21.15.186:0/1780397815 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:55:10.187 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848cc56700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8488103120 0x7f8488199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f8487fff700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8488104250 0x7f8488199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848d457700 1 -- 172.21.15.186:0/1780397815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8488104250 msgr2=0x7f8488199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848d457700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8488104250 0x7f8488199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848d457700 1 -- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8488103120 msgr2=0x7f8488199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848d457700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8488103120 0x7f8488199750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.184+0000 7f848d457700 1 -- 172.21.15.186:0/1780397815 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84881a1220 con 0x7f8488104bf0 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.185+0000 7f848d457700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488104bf0 0x7f84881a0c30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f848000b970 tx=0x7f848000bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.185+0000 7f8485ffb700 1 -- 172.21.15.186:0/1780397815 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f848000d860 con 0x7f8488104bf0 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.185+0000 7f8485ffb700 1 -- 172.21.15.186:0/1780397815 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f848000e430 con 0x7f8488104bf0 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.185+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84881a1510 con 0x7f8488104bf0 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.185+0000 7f8485ffb700 1 -- 172.21.15.186:0/1780397815 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8480017660 con 0x7f8488104bf0 2024-04-25T15:55:10.188 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.185+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84881a1ad0 con 0x7f8488104bf0 2024-04-25T15:55:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.186+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8488051190 con 0x7f8488104bf0 2024-04-25T15:55:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.187+0000 7f8485ffb700 1 -- 172.21.15.186:0/1780397815 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f848000da00 con 0x7f8488104bf0 2024-04-25T15:55:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.188+0000 7f8485ffb700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f84780afc00 0x7f84780b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:10.189 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.188+0000 7f8485ffb700 1 -- 172.21.15.186:0/1780397815 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4101+0+0 (secure 0 0 0) 0x7f84800dd8e0 con 0x7f8488104bf0 2024-04-25T15:55:10.190 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.188+0000 7f848cc56700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f84780afc00 0x7f84780b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:10.190 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.189+0000 7f848cc56700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f84780afc00 0x7f84780b2110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f847c0038a0 tx=0x7f847c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:10.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.192+0000 7f8485ffb700 1 -- 172.21.15.186:0/1780397815 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8480052ff0 con 0x7f8488104bf0 2024-04-25T15:55:10.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:09 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi063 2024-04-25T15:55:10.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:09 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi063 to 8544M 2024-04-25T15:55:10.400 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:10.399+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8488063ad0 con 0x7f84780afc00 2024-04-25T15:55:11.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:10 smithi063 ceph-mon[32413]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:11.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:55:11.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:55:11.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:11.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:10 smithi186 ceph-mon[33258]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:11.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:55:11.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:55:11.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:11.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:10 smithi031 ceph-mon[29140]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:11.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:55:11.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:55:11.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:12.101 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:11 smithi031 ceph-mon[29140]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:55:12.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:11 smithi063 ceph-mon[32413]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:55:12.134 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:11 smithi186 ceph-mon[33258]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:55:13.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:12 smithi063 ceph-mon[32413]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:13.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:12 smithi186 ceph-mon[33258]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:13.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:12 smithi031 ceph-mon[29140]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:14.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:13 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/961770869' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9c971-2a3e-4117-843f-efd5270a8de3"}]: dispatch 2024-04-25T15:55:14.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:13 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9c971-2a3e-4117-843f-efd5270a8de3"}]: dispatch 2024-04-25T15:55:14.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:13 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79c9c971-2a3e-4117-843f-efd5270a8de3"}]': finished 2024-04-25T15:55:14.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:13 smithi063 ceph-mon[32413]: osdmap e33: 6 total, 5 up, 6 in 2024-04-25T15:55:14.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:14.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:13 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/468485051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:55:14.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:13 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/961770869' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9c971-2a3e-4117-843f-efd5270a8de3"}]: dispatch 2024-04-25T15:55:14.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:13 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9c971-2a3e-4117-843f-efd5270a8de3"}]: dispatch 2024-04-25T15:55:14.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:13 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79c9c971-2a3e-4117-843f-efd5270a8de3"}]': finished 2024-04-25T15:55:14.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:13 smithi186 ceph-mon[33258]: osdmap e33: 6 total, 5 up, 6 in 2024-04-25T15:55:14.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:14.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:13 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/468485051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:55:14.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:13 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/961770869' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9c971-2a3e-4117-843f-efd5270a8de3"}]: dispatch 2024-04-25T15:55:14.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:13 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79c9c971-2a3e-4117-843f-efd5270a8de3"}]: dispatch 2024-04-25T15:55:14.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:13 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79c9c971-2a3e-4117-843f-efd5270a8de3"}]': finished 2024-04-25T15:55:14.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:13 smithi031 ceph-mon[29140]: osdmap e33: 6 total, 5 up, 6 in 2024-04-25T15:55:14.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:14.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:13 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/468485051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:55:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:14 smithi063 ceph-mon[32413]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:15.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:14 smithi186 ceph-mon[33258]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:15.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:14 smithi031 ceph-mon[29140]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:17.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:16 smithi063 ceph-mon[32413]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:17.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:16 smithi186 ceph-mon[33258]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:17.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:16 smithi031 ceph-mon[29140]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:19.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:18 smithi063 ceph-mon[32413]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:19.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:18 smithi186 ceph-mon[33258]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:19.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:18 smithi031 ceph-mon[29140]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:21.109 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:20 smithi186 ceph-mon[33258]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:21.109 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T15:55:21.109 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:21.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:20 smithi063 ceph-mon[32413]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:21.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T15:55:21.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:21.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:20 smithi031 ceph-mon[29140]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:21.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-25T15:55:21.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:22.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:21 smithi063 ceph-mon[32413]: Deploying daemon osd.5 on smithi186 2024-04-25T15:55:22.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:21 smithi186 ceph-mon[33258]: Deploying daemon osd.5 on smithi186 2024-04-25T15:55:22.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:21 smithi031 ceph-mon[29140]: Deploying daemon osd.5 on smithi186 2024-04-25T15:55:23.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:22 smithi063 ceph-mon[32413]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:22 smithi186 ceph-mon[33258]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:23.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:22 smithi031 ceph-mon[29140]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:24 smithi063 ceph-mon[32413]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:25.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:24 smithi186 ceph-mon[33258]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:25.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:24 smithi031 ceph-mon[29140]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:27.080 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:26 smithi186 ceph-mon[33258]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:27.080 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:27.080 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:27.080 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:27.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:26 smithi063 ceph-mon[32413]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:27.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:27.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:27.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:27.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:26 smithi031 ceph-mon[29140]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:27.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:27.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:29.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:28 smithi063 ceph-mon[32413]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:29.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:28 smithi186 ceph-mon[33258]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:29.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:28 smithi031 ceph-mon[29140]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:29.975 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 5 on host 'smithi186' 2024-04-25T15:55:29.975 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.973+0000 7f8485ffb700 1 -- 172.21.15.186:0/1780397815 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8488063ad0 con 0x7f84780afc00 2024-04-25T15:55:29.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f84780afc00 msgr2=0x7f84780b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:29.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f84780afc00 0x7f84780b2110 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f847c0038a0 tx=0x7f847c01a040 comp rx=0 tx=0).stop 2024-04-25T15:55:29.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488104bf0 msgr2=0x7f84881a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:29.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488104bf0 0x7f84881a0c30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f848000b970 tx=0x7f848000bc80 comp rx=0 tx=0).stop 2024-04-25T15:55:29.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 shutdown_connections 2024-04-25T15:55:29.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f84780afc00 0x7f84780b2110 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:29.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8488104bf0 0x7f84881a0c30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:29.978 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8488103120 0x7f8488199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:29.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 --2- 172.21.15.186:0/1780397815 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8488104250 0x7f8488199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:29.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 >> 172.21.15.186:0/1780397815 conn(0x7f84880fe760 msgr2=0x7f8488107480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:29.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 shutdown_connections 2024-04-25T15:55:29.979 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:55:29.977+0000 7f848eeba700 1 -- 172.21.15.186:0/1780397815 wait complete. 2024-04-25T15:55:30.609 DEBUG:teuthology.orchestra.run.smithi186:osd.5> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.5.service 2024-04-25T15:55:30.612 INFO:tasks.cephadm:Deploying osd.6 on smithi031 with /dev/vg_nvme/lv_2... 2024-04-25T15:55:30.612 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T15:55:30.870 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:55:31.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:30 smithi031 ceph-mon[29140]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:31.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:30 smithi031 ceph-mon[29140]: from='osd.5 [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T15:55:31.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:30 smithi031 ceph-mon[29140]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T15:55:31.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.337 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:30 smithi186 ceph-mon[33258]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:31.338 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.338 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.338 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:30 smithi186 ceph-mon[33258]: from='osd.5 [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T15:55:31.338 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:30 smithi186 ceph-mon[33258]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T15:55:31.338 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.338 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:30 smithi063 ceph-mon[32413]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:30 smithi063 ceph-mon[32413]: from='osd.5 [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T15:55:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:30 smithi063 ceph-mon[32413]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-25T15:55:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:31.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:32.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:31 smithi031 ceph-mon[29140]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T15:55:32.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:31 smithi031 ceph-mon[29140]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T15:55:32.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:31 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:32.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:31 smithi031 ceph-mon[29140]: from='osd.5 [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:55:32.293 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:31 smithi031 ceph-mon[29140]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:55:32.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:31 smithi063 ceph-mon[32413]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T15:55:32.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:31 smithi063 ceph-mon[32413]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T15:55:32.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:31 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:32.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:31 smithi063 ceph-mon[32413]: from='osd.5 [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:55:32.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:31 smithi063 ceph-mon[32413]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:55:32.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:31 smithi186 ceph-mon[33258]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-25T15:55:32.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:31 smithi186 ceph-mon[33258]: osdmap e34: 6 total, 5 up, 6 in 2024-04-25T15:55:32.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:31 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:32.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:31 smithi186 ceph-mon[33258]: from='osd.5 [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:55:32.375 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:31 smithi186 ceph-mon[33258]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:55:32.648 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 15:55:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T15:55:32.434+0000 7f85f46d9700 -1 osd.5 0 waiting for initial osdmap 2024-04-25T15:55:32.649 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 15:55:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T15:55:32.442+0000 7f85ec486700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:55:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T15:55:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T15:55:32.872 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-25T15:55:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-25T15:55:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112593 s, 931 MB/s 2024-04-25T15:55:32.872 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-25T15:55:33.259 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:32 smithi031 ceph-mon[29140]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:33.260 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:32 smithi031 ceph-mon[29140]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:55:33.260 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:32 smithi031 ceph-mon[29140]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T15:55:33.260 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:32 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:33.260 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:32 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:33.260 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:32 smithi031 ceph-mon[29140]: from='osd.5 ' entity='osd.5' 2024-04-25T15:55:33.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:32 smithi063 ceph-mon[32413]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:33.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:32 smithi063 ceph-mon[32413]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:55:33.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:32 smithi063 ceph-mon[32413]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T15:55:33.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:32 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:33.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:32 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:33.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:32 smithi063 ceph-mon[32413]: from='osd.5 ' entity='osd.5' 2024-04-25T15:55:33.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:32 smithi186 ceph-mon[33258]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-25T15:55:33.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:32 smithi186 ceph-mon[33258]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:55:33.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:32 smithi186 ceph-mon[33258]: osdmap e35: 6 total, 5 up, 6 in 2024-04-25T15:55:33.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:32 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:33.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:32 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:33.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:32 smithi186 ceph-mon[33258]: from='osd.5 ' entity='osd.5' 2024-04-25T15:55:33.688 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_2 2024-04-25T15:55:33.949 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:55:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:33 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:55:34.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:33 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:55:34.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:33 smithi186 ceph-mon[33258]: osd.5 [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] boot 2024-04-25T15:55:34.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:33 smithi186 ceph-mon[33258]: osdmap e36: 6 total, 6 up, 6 in 2024-04-25T15:55:34.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:33 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:34.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:33 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:55:34.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:33 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:55:34.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:33 smithi031 ceph-mon[29140]: osd.5 [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] boot 2024-04-25T15:55:34.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:33 smithi031 ceph-mon[29140]: osdmap e36: 6 total, 6 up, 6 in 2024-04-25T15:55:34.288 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:33 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:34.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:33 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:55:34.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:33 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:55:34.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:33 smithi063 ceph-mon[32413]: osd.5 [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] boot 2024-04-25T15:55:34.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:33 smithi063 ceph-mon[32413]: osdmap e36: 6 total, 6 up, 6 in 2024-04-25T15:55:34.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:33 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-25T15:55:35.247 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:35.248 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:35.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:35.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: osdmap e37: 6 total, 6 up, 6 in 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:55:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:35.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 -- 172.21.15.31:0/3632512040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4103120 msgr2=0x7f09d4103530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 --2- 172.21.15.31:0/3632512040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4103120 0x7f09d4103530 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009990 tx=0x7f09c8009ca0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 -- 172.21.15.31:0/3632512040 shutdown_connections 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 --2- 172.21.15.31:0/3632512040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09d4104bf0 0x7f09d410b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 --2- 172.21.15.31:0/3632512040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4103120 0x7f09d4103530 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 --2- 172.21.15.31:0/3632512040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09d4104250 0x7f09d4104660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 -- 172.21.15.31:0/3632512040 >> 172.21.15.31:0/3632512040 conn(0x7f09d40fe760 msgr2=0x7f09d4100c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 -- 172.21.15.31:0/3632512040 shutdown_connections 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 -- 172.21.15.31:0/3632512040 wait complete. 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.604+0000 7f09dad89700 1 Processor -- start 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09dad89700 1 -- start start 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09dad89700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09d4103120 0x7f09d419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09dad89700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09d4104250 0x7f09d419ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09dad89700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4104bf0 0x7f09d41a1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09dad89700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f09d410eb60 con 0x7f09d4104bf0 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09dad89700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f09d410e9e0 con 0x7f09d4103120 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09dad89700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f09d410ece0 con 0x7f09d4104250 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d9326700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4104bf0 0x7f09d41a1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d9326700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4104bf0 0x7f09d41a1d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44076/0 (socket says 172.21.15.31:44076) 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d9326700 1 -- 172.21.15.31:0/3343210127 learned_addr learned my addr 172.21.15.31:0/3343210127 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d8b25700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09d4103120 0x7f09d419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d3fff700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09d4104250 0x7f09d419ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d9326700 1 -- 172.21.15.31:0/3343210127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09d4104250 msgr2=0x7f09d419ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d9326700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09d4104250 0x7f09d419ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d9326700 1 -- 172.21.15.31:0/3343210127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09d4103120 msgr2=0x7f09d419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d9326700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09d4103120 0x7f09d419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:35.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d9326700 1 -- 172.21.15.31:0/3343210127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09d4105470 con 0x7f09d4104bf0 2024-04-25T15:55:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d8b25700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09d4103120 0x7f09d419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:55:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.605+0000 7f09d9326700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4104bf0 0x7f09d41a1d20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00ef30 tx=0x7f09cc00c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.606+0000 7f09d1ffb700 1 -- 172.21.15.31:0/3343210127 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc0092e0 con 0x7f09d4104bf0 2024-04-25T15:55:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.606+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09d4105700 con 0x7f09d4104bf0 2024-04-25T15:55:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.606+0000 7f09d1ffb700 1 -- 172.21.15.31:0/3343210127 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f09cc004bb0 con 0x7f09d4104bf0 2024-04-25T15:55:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.606+0000 7f09d1ffb700 1 -- 172.21.15.31:0/3343210127 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09cc004280 con 0x7f09d4104bf0 2024-04-25T15:55:35.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.606+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09d4105c40 con 0x7f09d4104bf0 2024-04-25T15:55:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.607+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09d4051190 con 0x7f09d4104bf0 2024-04-25T15:55:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.608+0000 7f09d1ffb700 1 -- 172.21.15.31:0/3343210127 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f09cc01b030 con 0x7f09d4104bf0 2024-04-25T15:55:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.609+0000 7f09d1ffb700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f09c40afb80 0x7f09c40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:55:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.609+0000 7f09d8b25700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f09c40afb80 0x7f09c40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:55:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.609+0000 7f09d1ffb700 1 -- 172.21.15.31:0/3343210127 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4533+0+0 (secure 0 0 0) 0x7f09cc0dd9f0 con 0x7f09d4104bf0 2024-04-25T15:55:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.609+0000 7f09d8b25700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f09c40afb80 0x7f09c40b2090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f09c800b760 tx=0x7f09c8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:55:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.614+0000 7f09d1ffb700 1 -- 172.21.15.31:0/3343210127 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f09cc022020 con 0x7f09d4104bf0 2024-04-25T15:55:35.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:35.832+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f09d41007d0 con 0x7f09c40afb80 2024-04-25T15:55:36.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:35 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi186 2024-04-25T15:55:36.391 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:35 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-25T15:55:36.391 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:35 smithi063 ceph-mon[32413]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T15:55:36.391 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:35 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:55:36.391 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:35 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:55:36.391 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:35 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:36.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:35 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi186 2024-04-25T15:55:36.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:35 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-25T15:55:36.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:35 smithi186 ceph-mon[33258]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T15:55:36.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:35 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:55:36.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:35 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:55:36.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:35 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:36.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:35 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi186 2024-04-25T15:55:36.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:35 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi186 to 8544M 2024-04-25T15:55:36.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:35 smithi031 ceph-mon[29140]: osdmap e38: 6 total, 6 up, 6 in 2024-04-25T15:55:36.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:35 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:55:36.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:35 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:55:36.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:35 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:37.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:36 smithi031 ceph-mon[29140]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:37.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:36 smithi031 ceph-mon[29140]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:55:37.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:36 smithi063 ceph-mon[32413]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:37.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:36 smithi063 ceph-mon[32413]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:55:37.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:36 smithi186 ceph-mon[33258]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:37.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:36 smithi186 ceph-mon[33258]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:55:39.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:38 smithi031 ceph-mon[29140]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:39.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:38 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2347848893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8955799-23fe-4161-890c-c77eb55760b2"}]: dispatch 2024-04-25T15:55:39.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:38 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2347848893' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8955799-23fe-4161-890c-c77eb55760b2"}]': finished 2024-04-25T15:55:39.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:38 smithi031 ceph-mon[29140]: osdmap e39: 7 total, 6 up, 7 in 2024-04-25T15:55:39.238 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:38 smithi063 ceph-mon[32413]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:38 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2347848893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8955799-23fe-4161-890c-c77eb55760b2"}]: dispatch 2024-04-25T15:55:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:38 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2347848893' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8955799-23fe-4161-890c-c77eb55760b2"}]': finished 2024-04-25T15:55:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:38 smithi063 ceph-mon[32413]: osdmap e39: 7 total, 6 up, 7 in 2024-04-25T15:55:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:38 smithi186 ceph-mon[33258]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 192 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:38 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2347848893' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8955799-23fe-4161-890c-c77eb55760b2"}]: dispatch 2024-04-25T15:55:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:38 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2347848893' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8955799-23fe-4161-890c-c77eb55760b2"}]': finished 2024-04-25T15:55:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:38 smithi186 ceph-mon[33258]: osdmap e39: 7 total, 6 up, 7 in 2024-04-25T15:55:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:40.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:39 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4206368740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:55:40.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:39 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4206368740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:55:40.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:39 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4206368740' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:55:41.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:40 smithi063 ceph-mon[32413]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T15:55:41.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:40 smithi186 ceph-mon[33258]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T15:55:41.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:40 smithi031 ceph-mon[29140]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-25T15:55:43.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:42 smithi186 ceph-mon[33258]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-25T15:55:43.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:42 smithi063 ceph-mon[32413]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-25T15:55:43.400 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:42 smithi031 ceph-mon[29140]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-25T15:55:45.295 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:44 smithi031 ceph-mon[29140]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T15:55:45.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:44 smithi063 ceph-mon[32413]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T15:55:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:44 smithi186 ceph-mon[33258]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T15:55:47.258 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:46 smithi031 ceph-mon[29140]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:55:47.258 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T15:55:47.258 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:47.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:46 smithi063 ceph-mon[32413]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:55:47.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T15:55:47.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:47.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:46 smithi186 ceph-mon[33258]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:55:47.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-25T15:55:47.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:55:48.249 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:47 smithi031 ceph-mon[29140]: Deploying daemon osd.6 on smithi031 2024-04-25T15:55:48.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:47 smithi063 ceph-mon[32413]: Deploying daemon osd.6 on smithi031 2024-04-25T15:55:48.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:47 smithi186 ceph-mon[33258]: Deploying daemon osd.6 on smithi031 2024-04-25T15:55:49.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:48 smithi031 ceph-mon[29140]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:55:49.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:48 smithi063 ceph-mon[32413]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:55:49.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:48 smithi186 ceph-mon[33258]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:55:51.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:51 smithi186 ceph-mon[33258]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:51.298 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:51 smithi031 ceph-mon[29140]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:51.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:51 smithi063 ceph-mon[32413]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:53.251 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:53 smithi031 ceph-mon[29140]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:53.251 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:53.251 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:53.252 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:53 smithi063 ceph-mon[32413]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:53.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:53 smithi186 ceph-mon[33258]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:53.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:55:55.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:55 smithi031 ceph-mon[29140]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:55.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:55 smithi063 ceph-mon[32413]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:55.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:55 smithi186 ceph-mon[33258]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:56.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.497+0000 7f09d1ffb700 1 -- 172.21.15.31:0/3343210127 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f09d41007d0 con 0x7f09c40afb80 2024-04-25T15:55:56.498 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 6 on host 'smithi031' 2024-04-25T15:55:56.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f09c40afb80 msgr2=0x7f09c40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:56.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f09c40afb80 0x7f09c40b2090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f09c800b760 tx=0x7f09c8005fb0 comp rx=0 tx=0).stop 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4104bf0 msgr2=0x7f09d41a1d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4104bf0 0x7f09d41a1d20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f09cc00ef30 tx=0x7f09cc00c450 comp rx=0 tx=0).stop 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 shutdown_connections 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f09c40afb80 0x7f09c40b2090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f09d4103120 0x7f09d419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f09d4104bf0 0x7f09d41a1d20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 --2- 172.21.15.31:0/3343210127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09d4104250 0x7f09d419ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 >> 172.21.15.31:0/3343210127 conn(0x7f09d40fe760 msgr2=0x7f09d41000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 shutdown_connections 2024-04-25T15:55:56.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:55:56.499+0000 7f09dad89700 1 -- 172.21.15.31:0/3343210127 wait complete. 2024-04-25T15:55:57.158 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:57 smithi031 ceph-mon[29140]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:57.158 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:57 smithi031 ceph-mon[29140]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T15:55:57.158 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.158 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.158 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.158 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.185 DEBUG:teuthology.orchestra.run.smithi031:osd.6> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.6.service 2024-04-25T15:55:57.188 INFO:tasks.cephadm:Deploying osd.7 on smithi063 with /dev/vg_nvme/lv_2... 2024-04-25T15:55:57.189 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T15:55:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:57 smithi063 ceph-mon[32413]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:57 smithi063 ceph-mon[32413]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T15:55:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:57 smithi186 ceph-mon[33258]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:57 smithi186 ceph-mon[33258]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-25T15:55:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:55:57.446 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T15:55:58.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:58 smithi063 ceph-mon[32413]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T15:55:58.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:58 smithi063 ceph-mon[32413]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T15:55:58.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:58 smithi063 ceph-mon[32413]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:55:58.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:58 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:58.391 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:58 smithi031 ceph-mon[29140]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T15:55:58.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:58 smithi031 ceph-mon[29140]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T15:55:58.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:58 smithi031 ceph-mon[29140]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:55:58.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:58 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:58.392 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 15:55:58 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T15:55:58.032+0000 7f3ad4df1700 -1 osd.6 0 waiting for initial osdmap 2024-04-25T15:55:58.392 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 15:55:58 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T15:55:58.041+0000 7f3ad0ba6700 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:55:58.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:58 smithi186 ceph-mon[33258]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-25T15:55:58.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:58 smithi186 ceph-mon[33258]: osdmap e40: 7 total, 6 up, 7 in 2024-04-25T15:55:58.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:58 smithi186 ceph-mon[33258]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:55:58.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:58 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:59.118 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:59 smithi063 ceph-mon[32413]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:59.118 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:59 smithi063 ceph-mon[32413]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:55:59.118 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:59 smithi063 ceph-mon[32413]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T15:55:59.118 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:59.118 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:55:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:59.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:59 smithi186 ceph-mon[33258]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:59 smithi186 ceph-mon[33258]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:55:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:59 smithi186 ceph-mon[33258]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T15:55:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:55:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:59 smithi031 ceph-mon[29140]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-25T15:55:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:59 smithi031 ceph-mon[29140]: from='osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:55:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:59 smithi031 ceph-mon[29140]: osdmap e41: 7 total, 6 up, 7 in 2024-04-25T15:55:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:55:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:55:59.708 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T15:55:59.708 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T15:55:59.708 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-04-25T15:55:59.709 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-04-25T15:55:59.709 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011609 s, 903 MB/s 2024-04-25T15:55:59.709 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-04-25T15:56:00.300 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:00 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:56:00.300 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:00 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:56:00.300 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:00 smithi186 ceph-mon[33258]: osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] boot 2024-04-25T15:56:00.300 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:00 smithi186 ceph-mon[33258]: osdmap e42: 7 total, 7 up, 7 in 2024-04-25T15:56:00.300 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:00 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:56:00.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:00 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:56:00.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:00 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:56:00.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:00 smithi063 ceph-mon[32413]: osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] boot 2024-04-25T15:56:00.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:00 smithi063 ceph-mon[32413]: osdmap e42: 7 total, 7 up, 7 in 2024-04-25T15:56:00.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:00 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:56:00.411 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_2 2024-04-25T15:56:00.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:00 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:56:00.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:00 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:56:00.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:00 smithi031 ceph-mon[29140]: osd.6 [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] boot 2024-04-25T15:56:00.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:00 smithi031 ceph-mon[29140]: osdmap e42: 7 total, 7 up, 7 in 2024-04-25T15:56:00.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:00 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-25T15:56:00.666 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T15:56:01.056 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.057 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:01.058 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:01.058 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:01.058 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:01.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T15:56:01.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:01.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 195 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: osdmap e43: 7 total, 7 up, 7 in 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:01.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:01.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:01.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:01.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:02.224 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:02 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi031 2024-04-25T15:56:02.224 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:02 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi031 to 5697M 2024-04-25T15:56:02.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:02 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi031 2024-04-25T15:56:02.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:02 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi031 to 5697M 2024-04-25T15:56:02.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:02 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi031 2024-04-25T15:56:02.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:02 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi031 to 5697M 2024-04-25T15:56:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.502+0000 7f2637384700 1 -- 172.21.15.63:0/4041794001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630102870 msgr2=0x7f2630102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.502+0000 7f2637384700 1 --2- 172.21.15.63:0/4041794001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630102870 0x7f2630102c80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2624009990 tx=0x7f2624009ca0 comp rx=0 tx=0).stop 2024-04-25T15:56:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.503+0000 7f2637384700 1 -- 172.21.15.63:0/4041794001 shutdown_connections 2024-04-25T15:56:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.503+0000 7f2637384700 1 --2- 172.21.15.63:0/4041794001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630102870 0x7f2630102c80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.503+0000 7f2637384700 1 --2- 172.21.15.63:0/4041794001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2630104340 0x7f263010abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.503+0000 7f2637384700 1 --2- 172.21.15.63:0/4041794001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26301039a0 0x7f2630103db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.503+0000 7f2637384700 1 -- 172.21.15.63:0/4041794001 >> 172.21.15.63:0/4041794001 conn(0x7f26300fded0 msgr2=0x7f2630100360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:56:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.503+0000 7f2637384700 1 -- 172.21.15.63:0/4041794001 shutdown_connections 2024-04-25T15:56:02.504 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.503+0000 7f2637384700 1 -- 172.21.15.63:0/4041794001 wait complete. 2024-04-25T15:56:02.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.503+0000 7f2637384700 1 Processor -- start 2024-04-25T15:56:02.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.503+0000 7f2637384700 1 -- start start 2024-04-25T15:56:02.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2637384700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2630102870 0x7f263019a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:02.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2637384700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26301039a0 0x7f263019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:02.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2637384700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630104340 0x7f26301a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:02.506 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2637384700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f263010da90 con 0x7f2630102870 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2637384700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f263010d910 con 0x7f2630104340 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2637384700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f263010dc10 con 0x7f26301039a0 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635921700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630104340 0x7f26301a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635921700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630104340 0x7f26301a1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55944/0 (socket says 172.21.15.63:55944) 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635921700 1 -- 172.21.15.63:0/224930627 learned_addr learned my addr 172.21.15.63:0/224930627 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635120700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2630102870 0x7f263019a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f263491f700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26301039a0 0x7f263019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635921700 1 -- 172.21.15.63:0/224930627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26301039a0 msgr2=0x7f263019adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635921700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26301039a0 0x7f263019adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635921700 1 -- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2630102870 msgr2=0x7f263019a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635921700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2630102870 0x7f263019a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635921700 1 -- 172.21.15.63:0/224930627 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26301a23d0 con 0x7f2630104340 2024-04-25T15:56:02.507 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f2635921700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630104340 0x7f26301a1d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f262800d370 tx=0x7f262800d680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:56:02.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.504+0000 7f26227fc700 1 -- 172.21.15.63:0/224930627 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f262800ddf0 con 0x7f2630104340 2024-04-25T15:56:02.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.505+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26301a2660 con 0x7f2630104340 2024-04-25T15:56:02.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.505+0000 7f26227fc700 1 -- 172.21.15.63:0/224930627 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2628003a00 con 0x7f2630104340 2024-04-25T15:56:02.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.505+0000 7f26227fc700 1 -- 172.21.15.63:0/224930627 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2628008d80 con 0x7f2630104340 2024-04-25T15:56:02.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.505+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26301a2ba0 con 0x7f2630104340 2024-04-25T15:56:02.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.506+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2630051190 con 0x7f2630104340 2024-04-25T15:56:02.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.507+0000 7f26227fc700 1 -- 172.21.15.63:0/224930627 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f262801a020 con 0x7f2630104340 2024-04-25T15:56:02.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.508+0000 7f26227fc700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f261c0afb80 0x7f261c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:02.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.508+0000 7f26227fc700 1 -- 172.21.15.63:0/224930627 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4965+0+0 (secure 0 0 0) 0x7f26280de0c0 con 0x7f2630104340 2024-04-25T15:56:02.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.508+0000 7f2635120700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f261c0afb80 0x7f261c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:02.509 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.508+0000 7f2635120700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f261c0afb80 0x7f261c0b2090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f26240038a0 tx=0x7f262401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:56:02.512 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.511+0000 7f26227fc700 1 -- 172.21.15.63:0/224930627 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f26280a79b0 con 0x7f2630104340 2024-04-25T15:56:02.729 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:02.729+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f26301a2ec0 con 0x7f261c0afb80 2024-04-25T15:56:03.357 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:03 smithi063 ceph-mon[32413]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:03.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:56:03.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:56:03.358 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:03.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:03 smithi186 ceph-mon[33258]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:03.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:56:03.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:56:03.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:03.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:03 smithi031 ceph-mon[29140]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:03.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:56:03.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:56:03.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:04.323 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:04 smithi063 ceph-mon[32413]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:56:04.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:04 smithi186 ceph-mon[33258]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:56:04.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:04 smithi031 ceph-mon[29140]: from='client.24287 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:56:05.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:05 smithi063 ceph-mon[32413]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:05.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:05 smithi186 ceph-mon[33258]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:05.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:05 smithi031 ceph-mon[29140]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:06 smithi063 ceph-mon[32413]: from='client.? 172.21.15.63:0/3946739730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33003c67-ad68-41a1-98eb-aea9138f1432"}]: dispatch 2024-04-25T15:56:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:06 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33003c67-ad68-41a1-98eb-aea9138f1432"}]: dispatch 2024-04-25T15:56:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:06 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33003c67-ad68-41a1-98eb-aea9138f1432"}]': finished 2024-04-25T15:56:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:06 smithi063 ceph-mon[32413]: osdmap e44: 8 total, 7 up, 8 in 2024-04-25T15:56:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:06 smithi186 ceph-mon[33258]: from='client.? 172.21.15.63:0/3946739730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33003c67-ad68-41a1-98eb-aea9138f1432"}]: dispatch 2024-04-25T15:56:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:06 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33003c67-ad68-41a1-98eb-aea9138f1432"}]: dispatch 2024-04-25T15:56:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:06 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33003c67-ad68-41a1-98eb-aea9138f1432"}]': finished 2024-04-25T15:56:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:06 smithi186 ceph-mon[33258]: osdmap e44: 8 total, 7 up, 8 in 2024-04-25T15:56:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:06.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:06 smithi031 ceph-mon[29140]: from='client.? 172.21.15.63:0/3946739730' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33003c67-ad68-41a1-98eb-aea9138f1432"}]: dispatch 2024-04-25T15:56:06.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:06 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33003c67-ad68-41a1-98eb-aea9138f1432"}]: dispatch 2024-04-25T15:56:06.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:06 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33003c67-ad68-41a1-98eb-aea9138f1432"}]': finished 2024-04-25T15:56:06.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:06 smithi031 ceph-mon[29140]: osdmap e44: 8 total, 7 up, 8 in 2024-04-25T15:56:06.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:07.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:07 smithi063 ceph-mon[32413]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:07.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:07 smithi063 ceph-mon[32413]: from='client.? 172.21.15.63:0/2759865601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:56:07.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:07 smithi186 ceph-mon[33258]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:07.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:07 smithi186 ceph-mon[33258]: from='client.? 172.21.15.63:0/2759865601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:56:07.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:07 smithi031 ceph-mon[29140]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:07.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:07 smithi031 ceph-mon[29140]: from='client.? 172.21.15.63:0/2759865601' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:56:09.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:09 smithi063 ceph-mon[32413]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:09.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:09 smithi186 ceph-mon[33258]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:09.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:09 smithi031 ceph-mon[29140]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:11.227 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:11 smithi063 ceph-mon[32413]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:11.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:11 smithi186 ceph-mon[33258]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:11.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:11 smithi031 ceph-mon[29140]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:13.281 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:13 smithi063 ceph-mon[32413]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:13.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:13 smithi186 ceph-mon[33258]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:13.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:13 smithi031 ceph-mon[29140]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:14.234 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:14 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T15:56:14.234 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:14 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:14.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:14 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T15:56:14.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:14 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:14.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:14 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-25T15:56:14.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:14 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:15.317 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:15 smithi063 ceph-mon[32413]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:15.318 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:15 smithi063 ceph-mon[32413]: Deploying daemon osd.7 on smithi063 2024-04-25T15:56:15.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:15 smithi186 ceph-mon[33258]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:15.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:15 smithi186 ceph-mon[33258]: Deploying daemon osd.7 on smithi063 2024-04-25T15:56:15.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:15 smithi031 ceph-mon[29140]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:15.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:15 smithi031 ceph-mon[29140]: Deploying daemon osd.7 on smithi063 2024-04-25T15:56:17.334 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:17 smithi063 ceph-mon[32413]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:17.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:17 smithi186 ceph-mon[33258]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:17.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:17 smithi031 ceph-mon[29140]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:19 smithi063 ceph-mon[32413]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:19 smithi186 ceph-mon[33258]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:19.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:19 smithi031 ceph-mon[29140]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:20.591 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:20.591 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:20.591 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:20 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:20.591 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:20 smithi063 ceph-mon[32413]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:20.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:20.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:20.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:20 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:20.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:20 smithi186 ceph-mon[33258]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:20.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:20.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:20.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:20 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:20.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:20 smithi031 ceph-mon[29140]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:22.536 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:22 smithi063 ceph-mon[32413]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:22.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:22 smithi186 ceph-mon[33258]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:22.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:22 smithi031 ceph-mon[29140]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:22.902 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 7 on host 'smithi063' 2024-04-25T15:56:22.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.899+0000 7f26227fc700 1 -- 172.21.15.63:0/224930627 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f26301a2ec0 con 0x7f261c0afb80 2024-04-25T15:56:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f261c0afb80 msgr2=0x7f261c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:22.903 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f261c0afb80 0x7f261c0b2090 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f26240038a0 tx=0x7f262401a040 comp rx=0 tx=0).stop 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630104340 msgr2=0x7f26301a1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630104340 0x7f26301a1d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f262800d370 tx=0x7f262800d680 comp rx=0 tx=0).stop 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 shutdown_connections 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f261c0afb80 0x7f261c0b2090 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2630104340 0x7f26301a1d50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2630102870 0x7f263019a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 --2- 172.21.15.63:0/224930627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26301039a0 0x7f263019adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 >> 172.21.15.63:0/224930627 conn(0x7f26300fded0 msgr2=0x7f2630108de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 shutdown_connections 2024-04-25T15:56:22.904 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:56:22.901+0000 7f2637384700 1 -- 172.21.15.63:0/224930627 wait complete. 2024-04-25T15:56:23.563 DEBUG:teuthology.orchestra.run.smithi063:osd.7> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.7.service 2024-04-25T15:56:23.566 INFO:tasks.cephadm:Deploying osd.8 on smithi186 with /dev/vg_nvme/lv_2... 2024-04-25T15:56:23.566 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-25T15:56:23.820 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:56:24.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:23 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:23 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:23 smithi063 ceph-mon[32413]: from='osd.7 [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T15:56:24.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:23 smithi063 ceph-mon[32413]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T15:56:24.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:23 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:23 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:23 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:23 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:23 smithi186 ceph-mon[33258]: from='osd.7 [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T15:56:24.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:23 smithi186 ceph-mon[33258]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T15:56:24.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:23 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:23 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:23 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:23 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:23 smithi031 ceph-mon[29140]: from='osd.7 [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T15:56:24.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:23 smithi031 ceph-mon[29140]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-25T15:56:24.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:23 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:23 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:24.948 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:24 smithi186 ceph-mon[33258]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:24.948 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:24 smithi186 ceph-mon[33258]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T15:56:24.949 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:24 smithi186 ceph-mon[33258]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T15:56:24.949 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:24 smithi186 ceph-mon[33258]: from='osd.7 [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:56:24.949 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:24 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:24.949 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:24 smithi186 ceph-mon[33258]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:56:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:24 smithi063 ceph-mon[32413]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:24 smithi063 ceph-mon[32413]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T15:56:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:24 smithi063 ceph-mon[32413]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T15:56:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:24 smithi063 ceph-mon[32413]: from='osd.7 [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:56:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:24 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:24 smithi063 ceph-mon[32413]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:56:25.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:24 smithi031 ceph-mon[29140]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:25.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:24 smithi031 ceph-mon[29140]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-25T15:56:25.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:24 smithi031 ceph-mon[29140]: osdmap e45: 8 total, 7 up, 8 in 2024-04-25T15:56:25.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:24 smithi031 ceph-mon[29140]: from='osd.7 [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:56:25.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:24 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:25.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:24 smithi031 ceph-mon[29140]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:56:25.619 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 15:56:25 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T15:56:25.291+0000 7f02f711e700 -1 osd.7 0 waiting for initial osdmap 2024-04-25T15:56:25.619 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 15:56:25 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T15:56:25.303+0000 7f02eeecb700 -1 osd.7 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:56:25.832 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-25T15:56:25.832 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-25T15:56:25.832 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-25T15:56:25.832 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-25T15:56:25.832 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113667 s, 923 MB/s 2024-04-25T15:56:25.832 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-25T15:56:26.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:25 smithi063 ceph-mon[32413]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:56:26.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:25 smithi063 ceph-mon[32413]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T15:56:26.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:25 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:26.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:25 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:26.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:25 smithi063 ceph-mon[32413]: from='osd.7 ' entity='osd.7' 2024-04-25T15:56:26.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:25 smithi186 ceph-mon[33258]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:56:26.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:25 smithi186 ceph-mon[33258]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T15:56:26.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:25 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:26.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:25 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:26.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:25 smithi186 ceph-mon[33258]: from='osd.7 ' entity='osd.7' 2024-04-25T15:56:26.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:25 smithi031 ceph-mon[29140]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:56:26.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:25 smithi031 ceph-mon[29140]: osdmap e46: 8 total, 7 up, 8 in 2024-04-25T15:56:26.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:25 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:26.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:25 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:26.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:25 smithi031 ceph-mon[29140]: from='osd.7 ' entity='osd.7' 2024-04-25T15:56:26.475 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-04-25T15:56:26.717 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:56:27.010 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:26 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:56:27.011 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:26 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:56:27.011 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:26 smithi186 ceph-mon[33258]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:27.011 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:27.011 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:26 smithi186 ceph-mon[33258]: osd.7 [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] boot 2024-04-25T15:56:27.011 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:26 smithi186 ceph-mon[33258]: osdmap e47: 8 total, 8 up, 8 in 2024-04-25T15:56:27.011 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:26 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:56:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:26 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:56:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:26 smithi031 ceph-mon[29140]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:26 smithi031 ceph-mon[29140]: osd.7 [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] boot 2024-04-25T15:56:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:26 smithi031 ceph-mon[29140]: osdmap e47: 8 total, 8 up, 8 in 2024-04-25T15:56:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:27.217 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:26 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:56:27.217 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:26 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:56:27.217 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:26 smithi063 ceph-mon[32413]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-25T15:56:27.217 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:27.217 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:26 smithi063 ceph-mon[32413]: osd.7 [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] boot 2024-04-25T15:56:27.217 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:26 smithi063 ceph-mon[32413]: osdmap e47: 8 total, 8 up, 8 in 2024-04-25T15:56:27.217 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-25T15:56:28.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:27 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:28.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T15:56:28.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:28.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:28.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:28.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:28.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:28.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:27 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:28.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: osdmap e48: 8 total, 8 up, 8 in 2024-04-25T15:56:28.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:28.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:28.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:28.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:28.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:28.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:28.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:27 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.456+0000 7fceb0b3c700 1 -- 172.21.15.186:0/1569128403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac1030d0 msgr2=0x7fceac1034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.456+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/1569128403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac1030d0 0x7fceac1034e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009990 tx=0x7fce9c009ca0 comp rx=0 tx=0).stop 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.456+0000 7fceb0b3c700 1 -- 172.21.15.186:0/1569128403 shutdown_connections 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.456+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/1569128403 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fceac104ba0 0x7fceac10b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.456+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/1569128403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac1030d0 0x7fceac1034e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.456+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/1569128403 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fceac104200 0x7fceac104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.456+0000 7fceb0b3c700 1 -- 172.21.15.186:0/1569128403 >> 172.21.15.186:0/1569128403 conn(0x7fceac0fe730 msgr2=0x7fceac100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.456+0000 7fceb0b3c700 1 -- 172.21.15.186:0/1569128403 shutdown_connections 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.456+0000 7fceb0b3c700 1 -- 172.21.15.186:0/1569128403 wait complete. 2024-04-25T15:56:28.459 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceb0b3c700 1 Processor -- start 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceb0b3c700 1 -- start start 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceb0b3c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac1030d0 0x7fceac19a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceb0b3c700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fceac104200 0x7fceac19ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceb0b3c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fceac104ba0 0x7fceac1a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceb0b3c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fceac10fcb0 con 0x7fceac104ba0 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceb0b3c700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fceac10fb30 con 0x7fceac104200 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceb0b3c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fceac10fe30 con 0x7fceac1030d0 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceaa59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac1030d0 0x7fceac19a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceaad9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fceac104ba0 0x7fceac1a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceaa59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac1030d0 0x7fceac19a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:51642/0 (socket says 172.21.15.186:51642) 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fcea1d9b700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fceac104200 0x7fceac19ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:28.460 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.457+0000 7fceaa59c700 1 -- 172.21.15.186:0/483088674 learned_addr learned my addr 172.21.15.186:0/483088674 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:56:28.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fceaad9d700 1 -- 172.21.15.186:0/483088674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac1030d0 msgr2=0x7fceac19a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:28.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fceaad9d700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac1030d0 0x7fceac19a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:28.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fceaad9d700 1 -- 172.21.15.186:0/483088674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fceac104200 msgr2=0x7fceac19ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:28.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fceaad9d700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fceac104200 0x7fceac19ad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:28.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fceaad9d700 1 -- 172.21.15.186:0/483088674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceac1a2350 con 0x7fceac104ba0 2024-04-25T15:56:28.461 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fceaad9d700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fceac104ba0 0x7fceac1a1cd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcea400efc0 tx=0x7fcea400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fcea3fff700 1 -- 172.21.15.186:0/483088674 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea40092e0 con 0x7fceac104ba0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceac10bfa0 con 0x7fceac104ba0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fcea3fff700 1 -- 172.21.15.186:0/483088674 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcea4004bb0 con 0x7fceac104ba0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fcea3fff700 1 -- 172.21.15.186:0/483088674 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea4004020 con 0x7fceac104ba0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.458+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceac10c490 con 0x7fceac104ba0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.459+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceac051190 con 0x7fceac104ba0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.461+0000 7fcea3fff700 1 -- 172.21.15.186:0/483088674 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fcea401b030 con 0x7fceac104ba0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.461+0000 7fcea3fff700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce980afb80 0x7fce980b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.461+0000 7fcea3fff700 1 -- 172.21.15.186:0/483088674 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5397+0+0 (secure 0 0 0) 0x7fcea40de560 con 0x7fceac104ba0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.461+0000 7fceaa59c700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce980afb80 0x7fce980b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:28.463 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.462+0000 7fceaa59c700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce980afb80 0x7fce980b2090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00b2e0 tx=0x7fce9c009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:56:28.466 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.465+0000 7fcea3fff700 1 -- 172.21.15.186:0/483088674 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcea4022020 con 0x7fceac104ba0 2024-04-25T15:56:28.672 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:28.670+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fceac063ad0 con 0x7fce980afb80 2024-04-25T15:56:29.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:29.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi063 2024-04-25T15:56:29.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi063 to 5696M 2024-04-25T15:56:29.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T15:56:29.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:29.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:29.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:29.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:56:29.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:56:29.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:29.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:29.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi063 2024-04-25T15:56:29.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi063 to 5696M 2024-04-25T15:56:29.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T15:56:29.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:29.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:29.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:29.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:56:29.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:56:29.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:29.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:29.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi063 2024-04-25T15:56:29.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi063 to 5696M 2024-04-25T15:56:29.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: osdmap e49: 8 total, 8 up, 8 in 2024-04-25T15:56:29.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:29.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:29.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:29.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:56:29.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:56:29.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:30.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:29 smithi186 ceph-mon[33258]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:56:30.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:29 smithi031 ceph-mon[29140]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:56:30.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:29 smithi063 ceph-mon[32413]: from='client.14388 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:56:31.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:30 smithi031 ceph-mon[29140]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:31.271 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:30 smithi063 ceph-mon[32413]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:31.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:30 smithi186 ceph-mon[33258]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:32.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:31 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/2315766867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d"}]: dispatch 2024-04-25T15:56:32.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:31 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d"}]: dispatch 2024-04-25T15:56:32.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:31 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d"}]': finished 2024-04-25T15:56:32.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:31 smithi031 ceph-mon[29140]: osdmap e50: 9 total, 8 up, 9 in 2024-04-25T15:56:32.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:31 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:32.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:31 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/2315766867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d"}]: dispatch 2024-04-25T15:56:32.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:31 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d"}]: dispatch 2024-04-25T15:56:32.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:31 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d"}]': finished 2024-04-25T15:56:32.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:31 smithi063 ceph-mon[32413]: osdmap e50: 9 total, 8 up, 9 in 2024-04-25T15:56:32.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:31 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:32.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:31 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/2315766867' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d"}]: dispatch 2024-04-25T15:56:32.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:31 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d"}]: dispatch 2024-04-25T15:56:32.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:31 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d"}]': finished 2024-04-25T15:56:32.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:31 smithi186 ceph-mon[33258]: osdmap e50: 9 total, 8 up, 9 in 2024-04-25T15:56:32.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:31 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:33.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:32 smithi031 ceph-mon[29140]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:33.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:32 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/1197307388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:56:33.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:32 smithi063 ceph-mon[32413]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:33.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:32 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/1197307388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:56:33.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:32 smithi186 ceph-mon[33258]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:33.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:32 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/1197307388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:56:35.198 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:34 smithi186 ceph-mon[33258]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-25T15:56:35.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:34 smithi031 ceph-mon[29140]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-25T15:56:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:34 smithi063 ceph-mon[32413]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-25T15:56:37.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:36 smithi186 ceph-mon[33258]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T15:56:37.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:36 smithi031 ceph-mon[29140]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T15:56:37.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:36 smithi063 ceph-mon[32413]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T15:56:39.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:38 smithi186 ceph-mon[33258]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T15:56:39.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T15:56:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:39.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:38 smithi031 ceph-mon[29140]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T15:56:39.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T15:56:39.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:38 smithi063 ceph-mon[32413]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-25T15:56:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-25T15:56:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:40.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:39 smithi186 ceph-mon[33258]: Deploying daemon osd.8 on smithi186 2024-04-25T15:56:40.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:39 smithi031 ceph-mon[29140]: Deploying daemon osd.8 on smithi186 2024-04-25T15:56:40.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:39 smithi063 ceph-mon[32413]: Deploying daemon osd.8 on smithi186 2024-04-25T15:56:41.127 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:40 smithi186 ceph-mon[33258]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:56:41.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:40 smithi031 ceph-mon[29140]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:56:41.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:40 smithi063 ceph-mon[32413]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:56:43.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:42 smithi186 ceph-mon[33258]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:56:43.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:42 smithi031 ceph-mon[29140]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:56:43.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:42 smithi063 ceph-mon[32413]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:56:45.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:44 smithi031 ceph-mon[29140]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-25T15:56:45.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:44 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:45.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:44 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:45.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:44 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:45.209 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:44 smithi186 ceph-mon[33258]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-25T15:56:45.209 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:44 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:45.209 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:44 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:45.209 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:44 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:45.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:44 smithi063 ceph-mon[32413]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-25T15:56:45.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:45.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:45.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:47.187 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:46 smithi186 ceph-mon[33258]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:47.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:46 smithi031 ceph-mon[29140]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:47.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:46 smithi063 ceph-mon[32413]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:48.539 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 8 on host 'smithi186' 2024-04-25T15:56:48.539 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.538+0000 7fcea3fff700 1 -- 172.21.15.186:0/483088674 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fceac063ad0 con 0x7fce980afb80 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce980afb80 msgr2=0x7fce980b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce980afb80 0x7fce980b2090 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fce9c00b2e0 tx=0x7fce9c009590 comp rx=0 tx=0).stop 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fceac104ba0 msgr2=0x7fceac1a1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fceac104ba0 0x7fceac1a1cd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcea400efc0 tx=0x7fcea400c3a0 comp rx=0 tx=0).stop 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 shutdown_connections 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce980afb80 0x7fce980b2090 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fceac104ba0 0x7fceac1a1cd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac1030d0 0x7fceac19a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 --2- 172.21.15.186:0/483088674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fceac104200 0x7fceac19ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:48.541 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 >> 172.21.15.186:0/483088674 conn(0x7fceac0fe730 msgr2=0x7fceac107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:56:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 shutdown_connections 2024-04-25T15:56:48.542 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:56:48.541+0000 7fceb0b3c700 1 -- 172.21.15.186:0/483088674 wait complete. 2024-04-25T15:56:48.837 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:48.837 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.837 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.837 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:48.837 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:48.838 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.838 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='osd.8 [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T15:56:48.838 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T15:56:48.838 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.838 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.838 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:48.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='osd.8 [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:48.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='osd.8 [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:48.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:56:49.166 DEBUG:teuthology.orchestra.run.smithi186:osd.8> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.8.service 2024-04-25T15:56:49.168 INFO:tasks.cephadm:Deploying osd.9 on smithi031 with /dev/vg_nvme/lv_1... 2024-04-25T15:56:49.168 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T15:56:49.433 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:56:50.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:49 smithi031 ceph-mon[29140]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T15:56:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:49 smithi031 ceph-mon[29140]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T15:56:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:49 smithi031 ceph-mon[29140]: from='osd.8 [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:56:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:50.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:49 smithi031 ceph-mon[29140]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:56:50.295 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:49 smithi186 ceph-mon[33258]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T15:56:50.295 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:49 smithi186 ceph-mon[33258]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T15:56:50.295 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:49 smithi186 ceph-mon[33258]: from='osd.8 [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:56:50.296 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:50.296 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:49 smithi186 ceph-mon[33258]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:56:50.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:49 smithi063 ceph-mon[32413]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-25T15:56:50.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:49 smithi063 ceph-mon[32413]: osdmap e51: 9 total, 8 up, 9 in 2024-04-25T15:56:50.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:49 smithi063 ceph-mon[32413]: from='osd.8 [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:56:50.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:50.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:49 smithi063 ceph-mon[32413]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:56:50.606 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 15:56:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T15:56:50.407+0000 7f7697d18700 -1 osd.8 0 waiting for initial osdmap 2024-04-25T15:56:50.606 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 15:56:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T15:56:50.420+0000 7f768fac5700 -1 osd.8 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:56:51.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:50 smithi186 ceph-mon[33258]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:51.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:50 smithi186 ceph-mon[33258]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:56:51.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:50 smithi186 ceph-mon[33258]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T15:56:51.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:51.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:51.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:50 smithi186 ceph-mon[33258]: from='osd.8 ' entity='osd.8' 2024-04-25T15:56:51.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:50 smithi031 ceph-mon[29140]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:51.187 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:50 smithi031 ceph-mon[29140]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:56:51.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:50 smithi031 ceph-mon[29140]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T15:56:51.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:51.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:51.188 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:50 smithi031 ceph-mon[29140]: from='osd.8 ' entity='osd.8' 2024-04-25T15:56:51.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:50 smithi063 ceph-mon[32413]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:51.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:50 smithi063 ceph-mon[32413]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:56:51.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:50 smithi063 ceph-mon[32413]: osdmap e52: 9 total, 8 up, 9 in 2024-04-25T15:56:51.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:51.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:51.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:50 smithi063 ceph-mon[32413]: from='osd.8 ' entity='osd.8' 2024-04-25T15:56:51.663 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T15:56:51.663 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T15:56:51.664 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2024-04-25T15:56:51.664 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2024-04-25T15:56:51.664 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0133163 s, 787 MB/s 2024-04-25T15:56:51.664 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2024-04-25T15:56:52.121 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:51 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:56:52.122 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:51 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:56:52.122 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:52.122 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:51 smithi186 ceph-mon[33258]: osd.8 [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] boot 2024-04-25T15:56:52.122 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:51 smithi186 ceph-mon[33258]: osdmap e53: 9 total, 9 up, 9 in 2024-04-25T15:56:52.122 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:52.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:51 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:56:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:51 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:56:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:51 smithi031 ceph-mon[29140]: osd.8 [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] boot 2024-04-25T15:56:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:51 smithi031 ceph-mon[29140]: osdmap e53: 9 total, 9 up, 9 in 2024-04-25T15:56:52.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:52.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:51 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:56:52.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:51 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:56:52.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:52.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:51 smithi063 ceph-mon[32413]: osd.8 [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] boot 2024-04-25T15:56:52.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:51 smithi063 ceph-mon[32413]: osdmap e53: 9 total, 9 up, 9 in 2024-04-25T15:56:52.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-25T15:56:52.437 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi031:vg_nvme/lv_1 2024-04-25T15:56:52.690 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:56:53.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:53.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T15:56:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:53.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-25T15:56:53.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: osdmap e54: 9 total, 9 up, 9 in 2024-04-25T15:56:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:56:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:56:53.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:56:54.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:53 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi186 2024-04-25T15:56:54.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:53 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-25T15:56:54.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:53 smithi063 ceph-mon[32413]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T15:56:54.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:53 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi186 2024-04-25T15:56:54.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:53 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-25T15:56:54.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:53 smithi031 ceph-mon[29140]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T15:56:54.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:53 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi186 2024-04-25T15:56:54.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:53 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi186 to 5696M 2024-04-25T15:56:54.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:53 smithi186 ceph-mon[33258]: osdmap e55: 9 total, 9 up, 9 in 2024-04-25T15:56:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.446+0000 7f2d91921700 1 -- 172.21.15.31:0/693150978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c1057b0 msgr2=0x7f2d8c105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.446+0000 7f2d91921700 1 --2- 172.21.15.31:0/693150978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c1057b0 0x7f2d8c105bc0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009990 tx=0x7f2d7c009ca0 comp rx=0 tx=0).stop 2024-04-25T15:56:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.446+0000 7f2d91921700 1 -- 172.21.15.31:0/693150978 shutdown_connections 2024-04-25T15:56:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.446+0000 7f2d91921700 1 --2- 172.21.15.31:0/693150978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d8c102b50 0x7f2d8c10c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.446+0000 7f2d91921700 1 --2- 172.21.15.31:0/693150978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c1057b0 0x7f2d8c105bc0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.446+0000 7f2d91921700 1 --2- 172.21.15.31:0/693150978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8c102200 0x7f2d8c102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.446+0000 7f2d91921700 1 -- 172.21.15.31:0/693150978 >> 172.21.15.31:0/693150978 conn(0x7f2d8c0fded0 msgr2=0x7f2d8c100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:56:54.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.446+0000 7f2d91921700 1 -- 172.21.15.31:0/693150978 shutdown_connections 2024-04-25T15:56:54.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.446+0000 7f2d91921700 1 -- 172.21.15.31:0/693150978 wait complete. 2024-04-25T15:56:54.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.447+0000 7f2d91921700 1 Processor -- start 2024-04-25T15:56:54.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.447+0000 7f2d91921700 1 -- start start 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.447+0000 7f2d91921700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c102200 0x7f2d8c104d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.447+0000 7f2d91921700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d8c102b50 0x7f2d8c103430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.447+0000 7f2d91921700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8c1057b0 0x7f2d8c103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d91921700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c10fc60 con 0x7f2d8c102200 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d91921700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c10fae0 con 0x7f2d8c102b50 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d91921700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8c10fde0 con 0x7f2d8c1057b0 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c102200 0x7f2d8c104d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c102200 0x7f2d8c104d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40360/0 (socket says 172.21.15.31:40360) 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8affd700 1 -- 172.21.15.31:0/1085422358 learned_addr learned my addr 172.21.15.31:0/1085422358 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8a7fc700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d8c102b50 0x7f2d8c103430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8b7fe700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8c1057b0 0x7f2d8c103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8affd700 1 -- 172.21.15.31:0/1085422358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8c1057b0 msgr2=0x7f2d8c103970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:54.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8affd700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8c1057b0 0x7f2d8c103970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8affd700 1 -- 172.21.15.31:0/1085422358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d8c102b50 msgr2=0x7f2d8c103430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8affd700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d8c102b50 0x7f2d8c103430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8affd700 1 -- 172.21.15.31:0/1085422358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8c1040d0 con 0x7f2d8c102200 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d8affd700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c102200 0x7f2d8c104d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c000c00 tx=0x7f2d7c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d9091f700 1 -- 172.21.15.31:0/1085422358 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c004830 con 0x7f2d8c102200 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8c1a48e0 con 0x7f2d8c102200 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d9091f700 1 -- 172.21.15.31:0/1085422358 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d7c005020 con 0x7f2d8c102200 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.448+0000 7f2d9091f700 1 -- 172.21.15.31:0/1085422358 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c003d90 con 0x7f2d8c102200 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.449+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d8c1a4c80 con 0x7f2d8c102200 2024-04-25T15:56:54.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.450+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d8c051190 con 0x7f2d8c102200 2024-04-25T15:56:54.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.451+0000 7f2d9091f700 1 -- 172.21.15.31:0/1085422358 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2d7c0051d0 con 0x7f2d8c102200 2024-04-25T15:56:54.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.452+0000 7f2d9091f700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d740afb80 0x7f2d740b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:56:54.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.452+0000 7f2d8a7fc700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d740afb80 0x7f2d740b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:56:54.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.452+0000 7f2d9091f700 1 -- 172.21.15.31:0/1085422358 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(55..55 src has 1..55) ==== 5829+0+0 (secure 0 0 0) 0x7f2d7c0df210 con 0x7f2d8c102200 2024-04-25T15:56:54.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.453+0000 7f2d8a7fc700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d740afb80 0x7f2d740b2090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2d80005950 tx=0x7f2d8000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:56:54.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.457+0000 7f2d9091f700 1 -- 172.21.15.31:0/1085422358 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d7c013070 con 0x7f2d8c102200 2024-04-25T15:56:54.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:56:54.673+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f2d8c063ad0 con 0x7f2d740afb80 2024-04-25T15:56:55.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:54 smithi031 ceph-mon[29140]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:56:55.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:54 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:56:55.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:54 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:56:55.312 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:54 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:55.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:54 smithi063 ceph-mon[32413]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:56:55.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:54 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:56:55.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:54 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:56:55.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:54 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:55.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:54 smithi186 ceph-mon[33258]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 258 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:56:55.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:54 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:56:55.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:54 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:56:55.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:54 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:56:56.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:55 smithi031 ceph-mon[29140]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:56:56.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:55 smithi063 ceph-mon[32413]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:56:56.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:55 smithi186 ceph-mon[33258]: from='client.14400 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:56:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:56 smithi063 ceph-mon[32413]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:56:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:56 smithi186 ceph-mon[33258]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:56:57.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:56 smithi031 ceph-mon[29140]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:56:58.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:57 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2559230829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37886022-d7c3-400c-aa91-02eedf2a12ee"}]: dispatch 2024-04-25T15:56:58.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:57 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2559230829' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37886022-d7c3-400c-aa91-02eedf2a12ee"}]': finished 2024-04-25T15:56:58.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:57 smithi031 ceph-mon[29140]: osdmap e56: 10 total, 9 up, 10 in 2024-04-25T15:56:58.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:56:58.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:57 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2559230829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37886022-d7c3-400c-aa91-02eedf2a12ee"}]: dispatch 2024-04-25T15:56:58.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:57 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2559230829' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37886022-d7c3-400c-aa91-02eedf2a12ee"}]': finished 2024-04-25T15:56:58.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:57 smithi063 ceph-mon[32413]: osdmap e56: 10 total, 9 up, 10 in 2024-04-25T15:56:58.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:56:58.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:57 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2559230829' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "37886022-d7c3-400c-aa91-02eedf2a12ee"}]: dispatch 2024-04-25T15:56:58.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:57 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2559230829' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "37886022-d7c3-400c-aa91-02eedf2a12ee"}]': finished 2024-04-25T15:56:58.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:57 smithi186 ceph-mon[33258]: osdmap e56: 10 total, 9 up, 10 in 2024-04-25T15:56:58.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:56:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:58 smithi063 ceph-mon[32413]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-25T15:56:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:56:58 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1232087665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:56:59.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:58 smithi186 ceph-mon[33258]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-25T15:56:59.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:56:58 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1232087665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:56:59.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:58 smithi031 ceph-mon[29140]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-25T15:56:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:56:58 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1232087665' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:57:01.281 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:00 smithi031 ceph-mon[29140]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-25T15:57:01.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:00 smithi063 ceph-mon[32413]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-25T15:57:01.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:00 smithi186 ceph-mon[33258]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-04-25T15:57:03.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:02 smithi063 ceph-mon[32413]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T15:57:03.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:02 smithi186 ceph-mon[33258]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T15:57:03.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:02 smithi031 ceph-mon[29140]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-25T15:57:05.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:04 smithi031 ceph-mon[29140]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:57:05.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:04 smithi063 ceph-mon[32413]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:57:05.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:04 smithi186 ceph-mon[33258]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:57:06.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T15:57:06.354 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T15:57:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-25T15:57:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:07.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:07 smithi031 ceph-mon[29140]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:57:07.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:07 smithi031 ceph-mon[29140]: Deploying daemon osd.9 on smithi031 2024-04-25T15:57:07.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:07 smithi063 ceph-mon[32413]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:57:07.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:07 smithi063 ceph-mon[32413]: Deploying daemon osd.9 on smithi031 2024-04-25T15:57:07.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:07 smithi186 ceph-mon[33258]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:57:07.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:07 smithi186 ceph-mon[33258]: Deploying daemon osd.9 on smithi031 2024-04-25T15:57:09.297 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:09 smithi031 ceph-mon[29140]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:09.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:09 smithi063 ceph-mon[32413]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:09.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:09 smithi186 ceph-mon[33258]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:11.091 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:11 smithi031 ceph-mon[29140]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:11.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:11 smithi063 ceph-mon[32413]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:11.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:11 smithi186 ceph-mon[33258]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:13.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:12 smithi063 ceph-mon[32413]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:13.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:13.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:12 smithi186 ceph-mon[33258]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:13.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:13.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:13.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:13.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:12 smithi031 ceph-mon[29140]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:13.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:13.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:13.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:14 smithi063 ceph-mon[32413]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:15.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:14 smithi186 ceph-mon[33258]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:15.173 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:14 smithi031 ceph-mon[29140]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:16.089 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 9 on host 'smithi031' 2024-04-25T15:57:16.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.087+0000 7f2d9091f700 1 -- 172.21.15.31:0/1085422358 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2d8c063ad0 con 0x7f2d740afb80 2024-04-25T15:57:16.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d740afb80 msgr2=0x7f2d740b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:16.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d740afb80 0x7f2d740b2090 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2d80005950 tx=0x7f2d8000a300 comp rx=0 tx=0).stop 2024-04-25T15:57:16.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c102200 msgr2=0x7f2d8c104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c102200 0x7f2d8c104d80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c000c00 tx=0x7f2d7c0047b0 comp rx=0 tx=0).stop 2024-04-25T15:57:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 shutdown_connections 2024-04-25T15:57:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d740afb80 0x7f2d740b2090 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8c1057b0 0x7f2d8c103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d8c102200 0x7f2d8c104d80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 --2- 172.21.15.31:0/1085422358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d8c102b50 0x7f2d8c103430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.089+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 >> 172.21.15.31:0/1085422358 conn(0x7f2d8c0fded0 msgr2=0x7f2d8c108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:57:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.090+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 shutdown_connections 2024-04-25T15:57:16.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:57:16.090+0000 7f2d91921700 1 -- 172.21.15.31:0/1085422358 wait complete. 2024-04-25T15:57:16.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:16.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:16.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:16 smithi063 ceph-mon[32413]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T15:57:16.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:16.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:16 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:16.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:16.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:16 smithi186 ceph-mon[33258]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T15:57:16.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:16.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:16 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:16.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:16.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:16 smithi031 ceph-mon[29140]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-25T15:57:16.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:16.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:16 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:16.898 DEBUG:teuthology.orchestra.run.smithi031:osd.9> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.9.service 2024-04-25T15:57:16.901 INFO:tasks.cephadm:Deploying osd.10 on smithi063 with /dev/vg_nvme/lv_1... 2024-04-25T15:57:16.901 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T15:57:17.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:17 smithi031 ceph-mon[29140]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:17.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:17 smithi031 ceph-mon[29140]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T15:57:17.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:17 smithi031 ceph-mon[29140]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T15:57:17.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:17 smithi031 ceph-mon[29140]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:57:17.134 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:17 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:17.134 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 15:57:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T15:57:17.104+0000 7f452f3c9700 -1 osd.9 0 waiting for initial osdmap 2024-04-25T15:57:17.134 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 15:57:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T15:57:17.119+0000 7f452b17e700 -1 osd.9 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:57:17.158 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T15:57:17.281 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:17 smithi063 ceph-mon[32413]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:17.281 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:17 smithi063 ceph-mon[32413]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T15:57:17.281 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:17 smithi063 ceph-mon[32413]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T15:57:17.282 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:17 smithi063 ceph-mon[32413]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:57:17.282 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:17 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:17.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:17 smithi186 ceph-mon[33258]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:17.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:17 smithi186 ceph-mon[33258]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-25T15:57:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:17 smithi186 ceph-mon[33258]: osdmap e57: 10 total, 9 up, 10 in 2024-04-25T15:57:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:17 smithi186 ceph-mon[33258]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2024-04-25T15:57:17.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:17 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:18.362 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:18 smithi031 ceph-mon[29140]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:57:18.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:18 smithi031 ceph-mon[29140]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T15:57:18.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:18 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:18.363 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:18 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:18.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:18 smithi063 ceph-mon[32413]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:57:18.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:18 smithi063 ceph-mon[32413]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T15:57:18.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:18 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:18.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:18 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:18.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:18 smithi186 ceph-mon[33258]: from='osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2024-04-25T15:57:18.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:18 smithi186 ceph-mon[33258]: osdmap e58: 10 total, 9 up, 10 in 2024-04-25T15:57:18.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:18 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:18.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:18 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:19.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:19 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:57:19.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:19 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:57:19.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:19 smithi063 ceph-mon[32413]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:19.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:19 smithi063 ceph-mon[32413]: osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] boot 2024-04-25T15:57:19.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:19 smithi063 ceph-mon[32413]: osdmap e59: 10 total, 10 up, 10 in 2024-04-25T15:57:19.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:19 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:19.384 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:19 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:57:19.385 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:19 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:57:19.385 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:19 smithi031 ceph-mon[29140]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:19.385 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:19 smithi031 ceph-mon[29140]: osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] boot 2024-04-25T15:57:19.385 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:19 smithi031 ceph-mon[29140]: osdmap e59: 10 total, 10 up, 10 in 2024-04-25T15:57:19.385 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:19 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:19 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:57:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:19 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:57:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:19 smithi186 ceph-mon[33258]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-25T15:57:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:19 smithi186 ceph-mon[33258]: osd.9 [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] boot 2024-04-25T15:57:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:19 smithi186 ceph-mon[33258]: osdmap e59: 10 total, 10 up, 10 in 2024-04-25T15:57:19.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:19 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-25T15:57:19.423 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T15:57:19.423 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T15:57:19.423 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-04-25T15:57:19.423 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-04-25T15:57:19.423 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114097 s, 919 MB/s 2024-04-25T15:57:19.423 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-04-25T15:57:20.112 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_1 2024-04-25T15:57:20.368 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T15:57:20.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:20 smithi063 ceph-mon[32413]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T15:57:20.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:20 smithi063 ceph-mon[32413]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:20.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:20 smithi186 ceph-mon[33258]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T15:57:20.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:20 smithi186 ceph-mon[33258]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:20.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:20 smithi031 ceph-mon[29140]: osdmap e60: 10 total, 10 up, 10 in 2024-04-25T15:57:20.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:20 smithi031 ceph-mon[29140]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:21.608 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi031 2024-04-25T15:57:21.608 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.608 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.608 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.608 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.608 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.608 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.609 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi031 to 4273M 2024-04-25T15:57:21.609 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.609 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:21.609 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:21.609 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:21 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi031 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi031 to 4273M 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:21 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi031 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi031 to 4273M 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:21.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:21 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:22.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.173+0000 7f0621a27700 1 -- 172.21.15.63:0/3403056862 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 msgr2=0x7f061c1034e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:22.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.173+0000 7f0621a27700 1 --2- 172.21.15.63:0/3403056862 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 0x7f061c1034e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f060c009990 tx=0x7f060c009ca0 comp rx=0 tx=0).stop 2024-04-25T15:57:22.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.173+0000 7f0621a27700 1 -- 172.21.15.63:0/3403056862 shutdown_connections 2024-04-25T15:57:22.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.173+0000 7f0621a27700 1 --2- 172.21.15.63:0/3403056862 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 0x7f061c1034e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:22.176 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.173+0000 7f0621a27700 1 --2- 172.21.15.63:0/3403056862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f061c104ba0 0x7f061c10b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.173+0000 7f0621a27700 1 --2- 172.21.15.63:0/3403056862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f061c104200 0x7f061c104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.173+0000 7f0621a27700 1 -- 172.21.15.63:0/3403056862 >> 172.21.15.63:0/3403056862 conn(0x7f061c0fe710 msgr2=0x7f061c100bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.173+0000 7f0621a27700 1 -- 172.21.15.63:0/3403056862 shutdown_connections 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.173+0000 7f0621a27700 1 -- 172.21.15.63:0/3403056862 wait complete. 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f0621a27700 1 Processor -- start 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f0621a27700 1 -- start start 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f0621a27700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 0x7f061c199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f0621a27700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f061c104200 0x7f061c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f0621a27700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f061c104ba0 0x7f061c1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f0621a27700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f061c10eb10 con 0x7f061c104ba0 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f0621a27700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f061c10e990 con 0x7f061c1030d0 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f0621a27700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f061c10ec90 con 0x7f061c104200 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f061affd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 0x7f061c199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:57:22.177 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f061affd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 0x7f061c199680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46532/0 (socket says 172.21.15.63:46532) 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.174+0000 7f061affd700 1 -- 172.21.15.63:0/2417844957 learned_addr learned my addr 172.21.15.63:0/2417844957 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f061a7fc700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f061c104200 0x7f061c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f061b7fe700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f061c104ba0 0x7f061c1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f061affd700 1 -- 172.21.15.63:0/2417844957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f061c104200 msgr2=0x7f061c199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f061affd700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f061c104200 0x7f061c199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f061affd700 1 -- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f061c104ba0 msgr2=0x7f061c1a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f061affd700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f061c104ba0 0x7f061c1a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f061affd700 1 -- 172.21.15.63:0/2417844957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c1a1150 con 0x7f061c1030d0 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f061affd700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 0x7f061c199680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f060c000c00 tx=0x7f060c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f0620a25700 1 -- 172.21.15.63:0/2417844957 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c027070 con 0x7f061c1030d0 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061c1a13e0 con 0x7f061c1030d0 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f0620a25700 1 -- 172.21.15.63:0/2417844957 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f060c004000 con 0x7f061c1030d0 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f0620a25700 1 -- 172.21.15.63:0/2417844957 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c022d80 con 0x7f061c1030d0 2024-04-25T15:57:22.178 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.175+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061c1a1890 con 0x7f061c1030d0 2024-04-25T15:57:22.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.176+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f061c051190 con 0x7f061c1030d0 2024-04-25T15:57:22.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.178+0000 7f0620a25700 1 -- 172.21.15.63:0/2417844957 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f060c02a070 con 0x7f061c1030d0 2024-04-25T15:57:22.179 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.179+0000 7f0620a25700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f06040afb80 0x7f06040b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:57:22.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.179+0000 7f0620a25700 1 -- 172.21.15.63:0/2417844957 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(60..60 src has 1..60) ==== 6261+0+0 (secure 0 0 0) 0x7f060c0df670 con 0x7f061c1030d0 2024-04-25T15:57:22.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.179+0000 7f061a7fc700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f06040afb80 0x7f06040b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:57:22.180 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.180+0000 7f061a7fc700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f06040afb80 0x7f06040b2090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0610005ea0 tx=0x7f0610005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:57:22.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.183+0000 7f0620a25700 1 -- 172.21.15.63:0/2417844957 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f060c031080 con 0x7f061c1030d0 2024-04-25T15:57:22.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:22.400+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f061c063ad0 con 0x7f06040afb80 2024-04-25T15:57:22.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:22 smithi063 ceph-mon[32413]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:22.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:22 smithi186 ceph-mon[33258]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:22.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:22 smithi031 ceph-mon[29140]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:23.299 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:23 smithi063 ceph-mon[32413]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:57:23.299 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:23 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:57:23.299 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:23 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:57:23.300 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:23 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:23.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:23 smithi186 ceph-mon[33258]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:57:23.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:23 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:57:23.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:23 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:57:23.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:23 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:23.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:23 smithi031 ceph-mon[29140]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:57:23.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:23 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:57:23.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:23 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:57:23.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:23 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:24.355 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:24 smithi063 ceph-mon[32413]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:24.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:24 smithi186 ceph-mon[33258]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:24.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:24 smithi031 ceph-mon[29140]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:25.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:25 smithi063 ceph-mon[32413]: from='client.? 172.21.15.63:0/2072826346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97d9d39e-2bd0-47a2-b5a4-cca5bde83613"}]: dispatch 2024-04-25T15:57:25.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:25 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97d9d39e-2bd0-47a2-b5a4-cca5bde83613"}]: dispatch 2024-04-25T15:57:25.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:25 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97d9d39e-2bd0-47a2-b5a4-cca5bde83613"}]': finished 2024-04-25T15:57:25.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:25 smithi063 ceph-mon[32413]: osdmap e61: 11 total, 10 up, 11 in 2024-04-25T15:57:25.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:25 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:25.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:25 smithi031 ceph-mon[29140]: from='client.? 172.21.15.63:0/2072826346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97d9d39e-2bd0-47a2-b5a4-cca5bde83613"}]: dispatch 2024-04-25T15:57:25.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:25 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97d9d39e-2bd0-47a2-b5a4-cca5bde83613"}]: dispatch 2024-04-25T15:57:25.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:25 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97d9d39e-2bd0-47a2-b5a4-cca5bde83613"}]': finished 2024-04-25T15:57:25.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:25 smithi031 ceph-mon[29140]: osdmap e61: 11 total, 10 up, 11 in 2024-04-25T15:57:25.631 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:25 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:25.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:25 smithi186 ceph-mon[33258]: from='client.? 172.21.15.63:0/2072826346' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97d9d39e-2bd0-47a2-b5a4-cca5bde83613"}]: dispatch 2024-04-25T15:57:25.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:25 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97d9d39e-2bd0-47a2-b5a4-cca5bde83613"}]: dispatch 2024-04-25T15:57:25.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:25 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97d9d39e-2bd0-47a2-b5a4-cca5bde83613"}]': finished 2024-04-25T15:57:25.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:25 smithi186 ceph-mon[33258]: osdmap e61: 11 total, 10 up, 11 in 2024-04-25T15:57:25.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:25 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:26.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:26 smithi063 ceph-mon[32413]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:26.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:26 smithi063 ceph-mon[32413]: from='client.? 172.21.15.63:0/415092214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:57:26.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:26 smithi186 ceph-mon[33258]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:26.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:26 smithi186 ceph-mon[33258]: from='client.? 172.21.15.63:0/415092214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:57:26.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:26 smithi031 ceph-mon[29140]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:26.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:26 smithi031 ceph-mon[29140]: from='client.? 172.21.15.63:0/415092214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:57:28.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:28 smithi063 ceph-mon[32413]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:28.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:28 smithi186 ceph-mon[33258]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:28.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:28 smithi031 ceph-mon[29140]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:30.529 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:30 smithi063 ceph-mon[32413]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:30.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:30 smithi186 ceph-mon[33258]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:30.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:30 smithi031 ceph-mon[29140]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:32.474 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:32 smithi063 ceph-mon[32413]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:32.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:32 smithi186 ceph-mon[33258]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:32.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:32 smithi031 ceph-mon[29140]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:33.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:33 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T15:57:33.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:33 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:33.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:33 smithi063 ceph-mon[32413]: Deploying daemon osd.10 on smithi063 2024-04-25T15:57:33.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:33 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T15:57:33.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:33 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:33.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:33 smithi186 ceph-mon[33258]: Deploying daemon osd.10 on smithi063 2024-04-25T15:57:33.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:33 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-25T15:57:33.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:33 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:33.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:33 smithi031 ceph-mon[29140]: Deploying daemon osd.10 on smithi063 2024-04-25T15:57:34.626 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:34 smithi063 ceph-mon[32413]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:34.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:34 smithi186 ceph-mon[33258]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:34.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:34 smithi031 ceph-mon[29140]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:36.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:36 smithi186 ceph-mon[33258]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:36.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:36 smithi063 ceph-mon[32413]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:36.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:36 smithi031 ceph-mon[29140]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:38.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:38 smithi186 ceph-mon[33258]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:38.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:38 smithi031 ceph-mon[29140]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:38.732 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:38 smithi063 ceph-mon[32413]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:40.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:40.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:40.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:40.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:40 smithi031 ceph-mon[29140]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:40.602 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:40.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:40.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:40.603 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:40 smithi063 ceph-mon[32413]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:40.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:40.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:40.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:40.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:40 smithi186 ceph-mon[33258]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:42.464 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:42 smithi063 ceph-mon[32413]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:42.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:42 smithi186 ceph-mon[33258]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:42.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:42 smithi031 ceph-mon[29140]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:42.928 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 10 on host 'smithi063' 2024-04-25T15:57:42.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.926+0000 7f0620a25700 1 -- 172.21.15.63:0/2417844957 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f061c063ad0 con 0x7f06040afb80 2024-04-25T15:57:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f06040afb80 msgr2=0x7f06040b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f06040afb80 0x7f06040b2090 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0610005ea0 tx=0x7f0610005e30 comp rx=0 tx=0).stop 2024-04-25T15:57:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 msgr2=0x7f061c199680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 0x7f061c199680 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f060c000c00 tx=0x7f060c0047b0 comp rx=0 tx=0).stop 2024-04-25T15:57:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 shutdown_connections 2024-04-25T15:57:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f06040afb80 0x7f06040b2090 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f061c1030d0 0x7f061c199680 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:42.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f061c104ba0 0x7f061c1a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 --2- 172.21.15.63:0/2417844957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f061c104200 0x7f061c199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 >> 172.21.15.63:0/2417844957 conn(0x7f061c0fe710 msgr2=0x7f061c107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:57:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 shutdown_connections 2024-04-25T15:57:42.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-04-25T15:57:42.929+0000 7f0621a27700 1 -- 172.21.15.63:0/2417844957 wait complete. 2024-04-25T15:57:43.621 DEBUG:teuthology.orchestra.run.smithi063:osd.10> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.10.service 2024-04-25T15:57:43.624 INFO:tasks.cephadm:Deploying osd.11 on smithi186 with /dev/vg_nvme/lv_1... 2024-04-25T15:57:43.624 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-25T15:57:43.873 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:57:44.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:43 smithi031 ceph-mon[29140]: from='osd.10 [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T15:57:44.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:43 smithi031 ceph-mon[29140]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T15:57:44.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.221 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:43 smithi186 ceph-mon[33258]: from='osd.10 [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T15:57:44.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:43 smithi186 ceph-mon[33258]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T15:57:44.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.222 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:43 smithi063 ceph-mon[32413]: from='osd.10 [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T15:57:44.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:43 smithi063 ceph-mon[32413]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-25T15:57:44.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:44.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:45.085 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:44 smithi186 ceph-mon[33258]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:45.086 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:44 smithi186 ceph-mon[33258]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T15:57:45.086 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:44 smithi186 ceph-mon[33258]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T15:57:45.086 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:44 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:45.086 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:44 smithi186 ceph-mon[33258]: from='osd.10 [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:57:45.087 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:44 smithi186 ceph-mon[33258]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:57:45.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:44 smithi031 ceph-mon[29140]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:45.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:44 smithi031 ceph-mon[29140]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T15:57:45.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:44 smithi031 ceph-mon[29140]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T15:57:45.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:44 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:45.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:44 smithi031 ceph-mon[29140]: from='osd.10 [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:57:45.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:44 smithi031 ceph-mon[29140]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:57:45.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:44 smithi063 ceph-mon[32413]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:45.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:44 smithi063 ceph-mon[32413]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-25T15:57:45.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:44 smithi063 ceph-mon[32413]: osdmap e62: 11 total, 10 up, 11 in 2024-04-25T15:57:45.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:45.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:44 smithi063 ceph-mon[32413]: from='osd.10 [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:57:45.367 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:44 smithi063 ceph-mon[32413]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-04-25T15:57:45.619 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 15:57:45 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T15:57:45.365+0000 7f91670f7700 -1 osd.10 0 waiting for initial osdmap 2024-04-25T15:57:45.619 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 15:57:45 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T15:57:45.377+0000 7f915eea4700 -1 osd.10 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:57:45.965 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-25T15:57:45.965 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-25T15:57:45.965 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-04-25T15:57:45.965 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-04-25T15:57:45.966 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120331 s, 871 MB/s 2024-04-25T15:57:45.966 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-04-25T15:57:46.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:45 smithi031 ceph-mon[29140]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:57:46.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:45 smithi031 ceph-mon[29140]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T15:57:46.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:45 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:46.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:45 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:46.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:45 smithi031 ceph-mon[29140]: from='osd.10 ' entity='osd.10' 2024-04-25T15:57:46.343 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:45 smithi186 ceph-mon[33258]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:57:46.344 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:45 smithi186 ceph-mon[33258]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T15:57:46.344 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:45 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:46.344 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:45 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:46.344 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:45 smithi186 ceph-mon[33258]: from='osd.10 ' entity='osd.10' 2024-04-25T15:57:46.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:45 smithi063 ceph-mon[32413]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-04-25T15:57:46.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:45 smithi063 ceph-mon[32413]: osdmap e63: 11 total, 10 up, 11 in 2024-04-25T15:57:46.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:45 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:46.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:45 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:46.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:45 smithi063 ceph-mon[32413]: from='osd.10 ' entity='osd.10' 2024-04-25T15:57:46.654 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_1 2024-04-25T15:57:46.898 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:57:47.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:46 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:57:47.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:46 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:57:47.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:46 smithi031 ceph-mon[29140]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:47.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:47.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:46 smithi031 ceph-mon[29140]: osd.10 [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] boot 2024-04-25T15:57:47.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:46 smithi031 ceph-mon[29140]: osdmap e64: 11 total, 11 up, 11 in 2024-04-25T15:57:47.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:47.232 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:46 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:57:47.232 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:46 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:57:47.232 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:46 smithi063 ceph-mon[32413]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:47.232 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:47.232 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:46 smithi063 ceph-mon[32413]: osd.10 [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] boot 2024-04-25T15:57:47.232 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:46 smithi063 ceph-mon[32413]: osdmap e64: 11 total, 11 up, 11 in 2024-04-25T15:57:47.232 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:47.265 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:46 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:57:47.265 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:46 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:57:47.265 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:46 smithi186 ceph-mon[33258]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-04-25T15:57:47.265 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:47.265 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:46 smithi186 ceph-mon[33258]: osd.10 [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] boot 2024-04-25T15:57:47.265 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:46 smithi186 ceph-mon[33258]: osdmap e64: 11 total, 11 up, 11 in 2024-04-25T15:57:47.265 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-25T15:57:48.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: osdmap e65: 11 total, 11 up, 11 in 2024-04-25T15:57:48.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.028 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:48.029 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: osdmap e65: 11 total, 11 up, 11 in 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:48.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:48.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:48.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:48.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:48.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: osdmap e65: 11 total, 11 up, 11 in 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:48.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:57:48.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:57:48.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.502+0000 7f299e7ab700 1 -- 172.21.15.186:0/1815009648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 msgr2=0x7f299810b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:48.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.502+0000 7f299e7ab700 1 --2- 172.21.15.186:0/1815009648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 0x7f299810b460 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f299000b0a0 tx=0x7f299000b3b0 comp rx=0 tx=0).stop 2024-04-25T15:57:48.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.502+0000 7f299e7ab700 1 -- 172.21.15.186:0/1815009648 shutdown_connections 2024-04-25T15:57:48.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.502+0000 7f299e7ab700 1 --2- 172.21.15.186:0/1815009648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29981030e0 0x7f29981034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:48.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.502+0000 7f299e7ab700 1 --2- 172.21.15.186:0/1815009648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 0x7f299810b460 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:48.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.502+0000 7f299e7ab700 1 --2- 172.21.15.186:0/1815009648 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2998104210 0x7f2998104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:48.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.502+0000 7f299e7ab700 1 -- 172.21.15.186:0/1815009648 >> 172.21.15.186:0/1815009648 conn(0x7f29980fe740 msgr2=0x7f2998100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:57:48.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.502+0000 7f299e7ab700 1 -- 172.21.15.186:0/1815009648 shutdown_connections 2024-04-25T15:57:48.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.502+0000 7f299e7ab700 1 -- 172.21.15.186:0/1815009648 wait complete. 2024-04-25T15:57:48.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.503+0000 7f299e7ab700 1 Processor -- start 2024-04-25T15:57:48.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.503+0000 7f299e7ab700 1 -- start start 2024-04-25T15:57:48.505 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299e7ab700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29981030e0 0x7f2998199780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299e7ab700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2998104210 0x7f2998199cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299e7ab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 0x7f29981a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299e7ab700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f299810ec40 con 0x7f2998104210 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299e7ab700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f299810eac0 con 0x7f29981030e0 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299e7ab700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f299810edc0 con 0x7f2998104bb0 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299cd48700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 0x7f29981a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299cd48700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 0x7f29981a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:58180/0 (socket says 172.21.15.186:58180) 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299cd48700 1 -- 172.21.15.186:0/2809081754 learned_addr learned my addr 172.21.15.186:0/2809081754 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f2997fff700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29981030e0 0x7f2998199780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f29977fe700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2998104210 0x7f2998199cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299cd48700 1 -- 172.21.15.186:0/2809081754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29981030e0 msgr2=0x7f2998199780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:48.506 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299cd48700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29981030e0 0x7f2998199780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299cd48700 1 -- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2998104210 msgr2=0x7f2998199cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299cd48700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2998104210 0x7f2998199cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299cd48700 1 -- 172.21.15.186:0/2809081754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29981a12e0 con 0x7f2998104bb0 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f29977fe700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2998104210 0x7f2998199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.504+0000 7f299cd48700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 0x7f29981a0c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2990006130 tx=0x7f2990004f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.505+0000 7f29957fa700 1 -- 172.21.15.186:0/2809081754 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2990020070 con 0x7f2998104bb0 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.505+0000 7f29957fa700 1 -- 172.21.15.186:0/2809081754 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2990004020 con 0x7f2998104bb0 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.505+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29980758c0 con 0x7f2998104bb0 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.505+0000 7f29957fa700 1 -- 172.21.15.186:0/2809081754 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f299001a720 con 0x7f2998104bb0 2024-04-25T15:57:48.507 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.505+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2998075e80 con 0x7f2998104bb0 2024-04-25T15:57:48.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.506+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2998051190 con 0x7f2998104bb0 2024-04-25T15:57:48.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.507+0000 7f29957fa700 1 -- 172.21.15.186:0/2809081754 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f29900044d0 con 0x7f2998104bb0 2024-04-25T15:57:48.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.508+0000 7f29957fa700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f29800afb80 0x7f29800b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:57:48.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.508+0000 7f29957fa700 1 -- 172.21.15.186:0/2809081754 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(66..66 src has 1..66) ==== 6694+0+0 (secure 0 0 0) 0x7f299000f030 con 0x7f2998104bb0 2024-04-25T15:57:48.508 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.508+0000 7f2997fff700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f29800afb80 0x7f29800b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:57:48.509 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.509+0000 7f2997fff700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f29800afb80 0x7f29800b2090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f298c006fd0 tx=0x7f298c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:57:48.511 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.511+0000 7f29957fa700 1 -- 172.21.15.186:0/2809081754 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29900aa960 con 0x7f2998104bb0 2024-04-25T15:57:48.719 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:57:48.719+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f29980760e0 con 0x7f29800afb80 2024-04-25T15:57:49.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:48 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi063 2024-04-25T15:57:49.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:48 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi063 to 4272M 2024-04-25T15:57:49.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:48 smithi186 ceph-mon[33258]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:57:49.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:48 smithi186 ceph-mon[33258]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T15:57:49.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:57:49.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:57:49.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:48 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:49.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:48 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi063 2024-04-25T15:57:49.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:48 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi063 to 4272M 2024-04-25T15:57:49.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:48 smithi063 ceph-mon[32413]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:57:49.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:48 smithi063 ceph-mon[32413]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T15:57:49.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:57:49.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:57:49.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:48 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:49.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:48 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi063 2024-04-25T15:57:49.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:48 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi063 to 4272M 2024-04-25T15:57:49.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:48 smithi031 ceph-mon[29140]: pgmap v234: 1 pgs: 1 remapped+peering; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:57:49.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:48 smithi031 ceph-mon[29140]: osdmap e66: 11 total, 11 up, 11 in 2024-04-25T15:57:49.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-25T15:57:49.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-25T15:57:49.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:48 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:57:50.346 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:49 smithi186 ceph-mon[33258]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:57:50.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:49 smithi063 ceph-mon[32413]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:57:50.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:49 smithi031 ceph-mon[29140]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:57:51.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:50 smithi063 ceph-mon[32413]: pgmap v236: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:57:51.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:50 smithi186 ceph-mon[33258]: pgmap v236: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:57:51.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:50 smithi031 ceph-mon[29140]: pgmap v236: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:57:52.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:51 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/2824448739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba832dc5-0d0e-420c-9430-12fdc737a86a"}]: dispatch 2024-04-25T15:57:52.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:51 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba832dc5-0d0e-420c-9430-12fdc737a86a"}]: dispatch 2024-04-25T15:57:52.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:51 smithi063 ceph-mon[32413]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba832dc5-0d0e-420c-9430-12fdc737a86a"}]': finished 2024-04-25T15:57:52.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:51 smithi063 ceph-mon[32413]: osdmap e67: 12 total, 11 up, 12 in 2024-04-25T15:57:52.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:57:52.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:51 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/2824448739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba832dc5-0d0e-420c-9430-12fdc737a86a"}]: dispatch 2024-04-25T15:57:52.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:51 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba832dc5-0d0e-420c-9430-12fdc737a86a"}]: dispatch 2024-04-25T15:57:52.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:51 smithi186 ceph-mon[33258]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba832dc5-0d0e-420c-9430-12fdc737a86a"}]': finished 2024-04-25T15:57:52.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:51 smithi186 ceph-mon[33258]: osdmap e67: 12 total, 11 up, 12 in 2024-04-25T15:57:52.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:57:52.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:51 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/2824448739' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba832dc5-0d0e-420c-9430-12fdc737a86a"}]: dispatch 2024-04-25T15:57:52.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:51 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba832dc5-0d0e-420c-9430-12fdc737a86a"}]: dispatch 2024-04-25T15:57:52.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:51 smithi031 ceph-mon[29140]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba832dc5-0d0e-420c-9430-12fdc737a86a"}]': finished 2024-04-25T15:57:52.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:51 smithi031 ceph-mon[29140]: osdmap e67: 12 total, 11 up, 12 in 2024-04-25T15:57:52.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:57:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:52 smithi063 ceph-mon[32413]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:57:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:52 smithi063 ceph-mon[32413]: from='client.? 172.21.15.186:0/269435234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:57:53.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:52 smithi186 ceph-mon[33258]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:57:53.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:52 smithi186 ceph-mon[33258]: from='client.? 172.21.15.186:0/269435234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:57:53.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:52 smithi031 ceph-mon[29140]: pgmap v237: 1 pgs: 1 remapped+peering; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:57:53.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:52 smithi031 ceph-mon[29140]: from='client.? 172.21.15.186:0/269435234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-25T15:57:55.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:54 smithi031 ceph-mon[29140]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-25T15:57:55.343 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:54 smithi186 ceph-mon[33258]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-25T15:57:55.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:54 smithi063 ceph-mon[32413]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-25T15:57:57.139 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:56 smithi186 ceph-mon[33258]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T15:57:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:56 smithi063 ceph-mon[32413]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T15:57:57.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:56 smithi031 ceph-mon[29140]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-25T15:57:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:58 smithi063 ceph-mon[32413]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-25T15:57:59.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:58 smithi186 ceph-mon[33258]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-25T15:57:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:58 smithi031 ceph-mon[29140]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-04-25T15:58:00.234 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T15:58:00.234 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:57:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:00.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T15:58:00.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:57:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:00.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-25T15:58:00.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:57:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:01.263 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:00 smithi186 ceph-mon[33258]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:58:01.264 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:00 smithi186 ceph-mon[33258]: Deploying daemon osd.11 on smithi186 2024-04-25T15:58:01.270 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:00 smithi063 ceph-mon[32413]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:58:01.270 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:00 smithi063 ceph-mon[32413]: Deploying daemon osd.11 on smithi186 2024-04-25T15:58:01.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:00 smithi031 ceph-mon[29140]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:58:01.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:00 smithi031 ceph-mon[29140]: Deploying daemon osd.11 on smithi186 2024-04-25T15:58:03.264 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:02 smithi186 ceph-mon[33258]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:58:03.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:02 smithi063 ceph-mon[32413]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:58:03.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:02 smithi031 ceph-mon[29140]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-25T15:58:05.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:04 smithi063 ceph-mon[32413]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-25T15:58:05.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:04 smithi186 ceph-mon[33258]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-25T15:58:05.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:04 smithi031 ceph-mon[29140]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-04-25T15:58:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:06.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:06 smithi186 ceph-mon[33258]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:06.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:06.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:06.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:06.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:06 smithi063 ceph-mon[32413]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:06.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:06 smithi031 ceph-mon[29140]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:08.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:08 smithi063 ceph-mon[32413]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:08.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:08.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:08.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:08.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:58:08.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:08 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:08.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:08 smithi186 ceph-mon[33258]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:08.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:08.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:08.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:08.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:58:08.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:08 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:08.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:08 smithi031 ceph-mon[29140]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:08.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:08.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:08.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:08.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:58:08.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:08 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:09.190 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 11 on host 'smithi186' 2024-04-25T15:58:09.191 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.189+0000 7f29957fa700 1 -- 172.21.15.186:0/2809081754 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f29980760e0 con 0x7f29800afb80 2024-04-25T15:58:09.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f29800afb80 msgr2=0x7f29800b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:09.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f29800afb80 0x7f29800b2090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f298c006fd0 tx=0x7f298c008040 comp rx=0 tx=0).stop 2024-04-25T15:58:09.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 msgr2=0x7f29981a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:09.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 0x7f29981a0c60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2990006130 tx=0x7f2990004f10 comp rx=0 tx=0).stop 2024-04-25T15:58:09.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 shutdown_connections 2024-04-25T15:58:09.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f29800afb80 0x7f29800b2090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:09.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f29981030e0 0x7f2998199780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:09.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2998104bb0 0x7f29981a0c60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:09.193 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 --2- 172.21.15.186:0/2809081754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2998104210 0x7f2998199cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:09.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.191+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 >> 172.21.15.186:0/2809081754 conn(0x7f29980fe740 msgr2=0x7f2998107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:09.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.192+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 shutdown_connections 2024-04-25T15:58:09.194 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:09.192+0000 7f299e7ab700 1 -- 172.21.15.186:0/2809081754 wait complete. 2024-04-25T15:58:09.844 DEBUG:teuthology.orchestra.run.smithi186:osd.11> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.11.service 2024-04-25T15:58:09.847 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-25T15:58:09.847 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-25T15:58:10.108 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:58:10.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:10.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:10.355 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:10.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:10 smithi186 ceph-mon[33258]: from='osd.11 [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T15:58:10.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:10 smithi186 ceph-mon[33258]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T15:58:10.356 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:10 smithi186 ceph-mon[33258]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:10.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:10.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:10.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:10.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:10 smithi031 ceph-mon[29140]: from='osd.11 [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T15:58:10.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:10 smithi031 ceph-mon[29140]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T15:58:10.406 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:10 smithi031 ceph-mon[29140]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:10.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:10.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:10.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:10.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:10 smithi063 ceph-mon[32413]: from='osd.11 [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T15:58:10.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:10 smithi063 ceph-mon[32413]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-25T15:58:10.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:10 smithi063 ceph-mon[32413]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:11.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:11 smithi031 ceph-mon[29140]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T15:58:11.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:11 smithi031 ceph-mon[29140]: osdmap e68: 12 total, 11 up, 12 in 2024-04-25T15:58:11.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:11 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:11.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:11 smithi031 ceph-mon[29140]: from='osd.11 [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:58:11.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:11 smithi031 ceph-mon[29140]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:58:11.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:11 smithi186 ceph-mon[33258]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T15:58:11.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:11 smithi186 ceph-mon[33258]: osdmap e68: 12 total, 11 up, 12 in 2024-04-25T15:58:11.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:11 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:11.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:11 smithi186 ceph-mon[33258]: from='osd.11 [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:58:11.473 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:11 smithi186 ceph-mon[33258]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:58:11.474 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 15:58:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T15:58:11.163+0000 7f84daae9700 -1 osd.11 0 waiting for initial osdmap 2024-04-25T15:58:11.474 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 15:58:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T15:58:11.176+0000 7f84d689e700 -1 osd.11 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-25T15:58:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:11 smithi063 ceph-mon[32413]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-25T15:58:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:11 smithi063 ceph-mon[32413]: osdmap e68: 12 total, 11 up, 12 in 2024-04-25T15:58:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:11 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:11 smithi063 ceph-mon[32413]: from='osd.11 [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:58:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:11 smithi063 ceph-mon[32413]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-04-25T15:58:12.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.117+0000 7f8e10915700 1 -- 172.21.15.31:0/4005650826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c105ce0 msgr2=0x7f8e0c10c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:12.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.117+0000 7f8e10915700 1 --2- 172.21.15.31:0/4005650826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c105ce0 0x7f8e0c10c590 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400b0a0 tx=0x7f8e0400b3b0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.117+0000 7f8e10915700 1 -- 172.21.15.31:0/4005650826 shutdown_connections 2024-04-25T15:58:12.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.117+0000 7f8e10915700 1 --2- 172.21.15.31:0/4005650826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e0c104210 0x7f8e0c104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.117+0000 7f8e10915700 1 --2- 172.21.15.31:0/4005650826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c105ce0 0x7f8e0c10c590 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.117+0000 7f8e10915700 1 --2- 172.21.15.31:0/4005650826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e0c105340 0x7f8e0c105750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.117+0000 7f8e10915700 1 -- 172.21.15.31:0/4005650826 >> 172.21.15.31:0/4005650826 conn(0x7f8e0c0ff870 msgr2=0x7f8e0c101d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:12.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.117+0000 7f8e10915700 1 -- 172.21.15.31:0/4005650826 shutdown_connections 2024-04-25T15:58:12.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.117+0000 7f8e10915700 1 -- 172.21.15.31:0/4005650826 wait complete. 2024-04-25T15:58:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.118+0000 7f8e10915700 1 Processor -- start 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.118+0000 7f8e10915700 1 -- start start 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.118+0000 7f8e10915700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c104210 0x7f8e0c19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.118+0000 7f8e10915700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e0c105340 0x7f8e0c19ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.118+0000 7f8e10915700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e0c105ce0 0x7f8e0c1a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.118+0000 7f8e10915700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c10fc70 con 0x7f8e0c104210 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.118+0000 7f8e10915700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c10faf0 con 0x7f8e0c105ce0 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.118+0000 7f8e10915700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c10fdf0 con 0x7f8e0c105340 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.118+0000 7f8e0a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c104210 0x7f8e0c19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e09d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e0c105340 0x7f8e0c19ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c104210 0x7f8e0c19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59310/0 (socket says 172.21.15.31:59310) 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0a59c700 1 -- 172.21.15.31:0/3632025627 learned_addr learned my addr 172.21.15.31:0/3632025627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0ad9d700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e0c105ce0 0x7f8e0c1a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0a59c700 1 -- 172.21.15.31:0/3632025627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e0c105340 msgr2=0x7f8e0c19ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0a59c700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e0c105340 0x7f8e0c19ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0a59c700 1 -- 172.21.15.31:0/3632025627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e0c105ce0 msgr2=0x7f8e0c1a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0a59c700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e0c105ce0 0x7f8e0c1a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0a59c700 1 -- 172.21.15.31:0/3632025627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0c1a2260 con 0x7f8e0c104210 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e09d9b700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e0c105340 0x7f8e0c19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0ad9d700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e0c105ce0 0x7f8e0c1a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e0a59c700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c104210 0x7f8e0c19a700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00e940 tx=0x7f8dfc00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8dfb7fe700 1 -- 172.21.15.31:0/3632025627 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc00cbd0 con 0x7f8e0c104210 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8dfb7fe700 1 -- 172.21.15.31:0/3632025627 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8dfc00ee90 con 0x7f8e0c104210 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e0c1a2550 con 0x7f8e0c104210 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8dfb7fe700 1 -- 172.21.15.31:0/3632025627 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dfc003dc0 con 0x7f8e0c104210 2024-04-25T15:58:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.119+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e0c1a2ae0 con 0x7f8e0c104210 2024-04-25T15:58:12.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.121+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e0c051190 con 0x7f8e0c104210 2024-04-25T15:58:12.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.122+0000 7f8dfb7fe700 1 -- 172.21.15.31:0/3632025627 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8dfc003f60 con 0x7f8e0c104210 2024-04-25T15:58:12.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.123+0000 7f8dfb7fe700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8df40afc00 0x7f8df40b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:12.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.123+0000 7f8e09d9b700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8df40afc00 0x7f8df40b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:12.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.123+0000 7f8dfb7fe700 1 -- 172.21.15.31:0/3632025627 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6847+0+0 (secure 0 0 0) 0x7f8dfc0de830 con 0x7f8e0c104210 2024-04-25T15:58:12.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.123+0000 7f8e09d9b700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8df40afc00 0x7f8df40b2110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c19b7b0 tx=0x7f8e0000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:12.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.128+0000 7f8dfb7fe700 1 -- 172.21.15.31:0/3632025627 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8dfc0a79d0 con 0x7f8e0c104210 2024-04-25T15:58:12.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.337+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f8e0c02e510 con 0x7f8e0c104210 2024-04-25T15:58:12.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.338+0000 7f8dfb7fe700 1 -- 172.21.15.31:0/3632025627 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7f8dfc0a72d0 con 0x7f8e0c104210 2024-04-25T15:58:12.340 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:12.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8df40afc00 msgr2=0x7f8df40b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:12.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8df40afc00 0x7f8df40b2110 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c19b7b0 tx=0x7f8e0000b410 comp rx=0 tx=0).stop 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c104210 msgr2=0x7f8e0c19a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c104210 0x7f8e0c19a700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc00e940 tx=0x7f8dfc00ec50 comp rx=0 tx=0).stop 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 shutdown_connections 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8df40afc00 0x7f8df40b2110 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e0c105ce0 0x7f8e0c1a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8e0c104210 0x7f8e0c19a700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 --2- 172.21.15.31:0/3632025627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e0c105340 0x7f8e0c19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 >> 172.21.15.31:0/3632025627 conn(0x7f8e0c0ff870 msgr2=0x7f8e0c108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.340+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 shutdown_connections 2024-04-25T15:58:12.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:12.341+0000 7f8e10915700 1 -- 172.21.15.31:0/3632025627 wait complete. 2024-04-25T15:58:12.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:12 smithi031 ceph-mon[29140]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:58:12.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:12 smithi031 ceph-mon[29140]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T15:58:12.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:12.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:12.378 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:12 smithi031 ceph-mon[29140]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:12.617 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:12 smithi186 ceph-mon[33258]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:58:12.617 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:12 smithi186 ceph-mon[33258]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T15:58:12.617 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:12.618 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:12.618 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:12 smithi186 ceph-mon[33258]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:12.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:12 smithi063 ceph-mon[32413]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-04-25T15:58:12.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:12 smithi063 ceph-mon[32413]: osdmap e69: 12 total, 11 up, 12 in 2024-04-25T15:58:12.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:12.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:12.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:12 smithi063 ceph-mon[32413]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-04-25T15:58:13.263 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":70,"num_osds":12,"num_up_osds":12,"osd_up_since":1714060692,"num_in_osds":12,"osd_in_since":1714060671,"num_remapped_pgs":0} 2024-04-25T15:58:13.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T15:58:13.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: purged_snaps scrub starts 2024-04-25T15:58:13.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: purged_snaps scrub ok 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: osd.11 [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] boot 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: osdmap e70: 12 total, 12 up, 12 in 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3632025627' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: Detected new or changed devices on smithi186 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:13.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:58:13.371 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:13 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: purged_snaps scrub starts 2024-04-25T15:58:13.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: purged_snaps scrub ok 2024-04-25T15:58:13.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: osd.11 [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] boot 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: osdmap e70: 12 total, 12 up, 12 in 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3632025627' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: Detected new or changed devices on smithi186 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:58:13.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:13 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: purged_snaps scrub starts 2024-04-25T15:58:13.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: purged_snaps scrub ok 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: osd.11 [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] boot 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: osdmap e70: 12 total, 12 up, 12 in 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3632025627' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: Detected new or changed devices on smithi186 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi186 to 4272M 2024-04-25T15:58:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:13.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:58:13.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:13 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:13.521 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:58:15.121 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:15 smithi031 ceph-mon[29140]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:15.121 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:15 smithi031 ceph-mon[29140]: osdmap e71: 12 total, 12 up, 12 in 2024-04-25T15:58:15.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.359+0000 7f8bcc535700 1 -- 172.21.15.31:0/2269346329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc41044a0 msgr2=0x7f8bc41048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:15.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.359+0000 7f8bcc535700 1 --2- 172.21.15.31:0/2269346329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc41044a0 0x7f8bc41048b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8bc00099c0 tx=0x7f8bc0009cd0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.359+0000 7f8bcc535700 1 -- 172.21.15.31:0/2269346329 shutdown_connections 2024-04-25T15:58:15.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.359+0000 7f8bcc535700 1 --2- 172.21.15.31:0/2269346329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8bc4109720 0x7f8bc4111080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.359+0000 7f8bcc535700 1 --2- 172.21.15.31:0/2269346329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc41044a0 0x7f8bc41048b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.359+0000 7f8bcc535700 1 --2- 172.21.15.31:0/2269346329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc4108dd0 0x7f8bc41091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.359+0000 7f8bcc535700 1 -- 172.21.15.31:0/2269346329 >> 172.21.15.31:0/2269346329 conn(0x7f8bc40ffd70 msgr2=0x7f8bc41021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.359+0000 7f8bcc535700 1 -- 172.21.15.31:0/2269346329 shutdown_connections 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.359+0000 7f8bcc535700 1 -- 172.21.15.31:0/2269346329 wait complete. 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcc535700 1 Processor -- start 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcc535700 1 -- start start 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcc535700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8bc41044a0 0x7f8bc41a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcc535700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc4108dd0 0x7f8bc41a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcc535700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc4109720 0x7f8bc419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcc535700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc4114350 con 0x7f8bc4109720 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcc535700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41141d0 con 0x7f8bc41044a0 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcc535700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8bc41144d0 con 0x7f8bc4108dd0 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcaad2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc4109720 0x7f8bc419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcaad2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc4109720 0x7f8bc419aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54226/0 (socket says 172.21.15.31:54226) 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.360+0000 7f8bcaad2700 1 -- 172.21.15.31:0/1781004646 learned_addr learned my addr 172.21.15.31:0/1781004646 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:15.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.361+0000 7f8bc9ad0700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc4108dd0 0x7f8bc41a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.361+0000 7f8bca2d1700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8bc41044a0 0x7f8bc41a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.361+0000 7f8bcaad2700 1 -- 172.21.15.31:0/1781004646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc4108dd0 msgr2=0x7f8bc41a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.361+0000 7f8bcaad2700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc4108dd0 0x7f8bc41a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.361+0000 7f8bcaad2700 1 -- 172.21.15.31:0/1781004646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8bc41044a0 msgr2=0x7f8bc41a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.361+0000 7f8bcaad2700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8bc41044a0 0x7f8bc41a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.361+0000 7f8bcaad2700 1 -- 172.21.15.31:0/1781004646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bc419b140 con 0x7f8bc4109720 2024-04-25T15:58:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.361+0000 7f8bca2d1700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8bc41044a0 0x7f8bc41a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:58:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.361+0000 7f8bcaad2700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc4109720 0x7f8bc419aa90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc007ee0 tx=0x7f8bbc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:15.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.362+0000 7f8bbb7fe700 1 -- 172.21.15.31:0/1781004646 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00dbd0 con 0x7f8bc4109720 2024-04-25T15:58:15.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.362+0000 7f8bbb7fe700 1 -- 172.21.15.31:0/1781004646 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8bbc00dd70 con 0x7f8bc4109720 2024-04-25T15:58:15.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.362+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bc419b430 con 0x7f8bc4109720 2024-04-25T15:58:15.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.362+0000 7f8bbb7fe700 1 -- 172.21.15.31:0/1781004646 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc005990 con 0x7f8bc4109720 2024-04-25T15:58:15.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.362+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bc41a7700 con 0x7f8bc4109720 2024-04-25T15:58:15.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.363+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bc4051190 con 0x7f8bc4109720 2024-04-25T15:58:15.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.365+0000 7f8bbb7fe700 1 -- 172.21.15.31:0/1781004646 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8bbc01a020 con 0x7f8bc4109720 2024-04-25T15:58:15.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.367+0000 7f8bbb7fe700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8bb00afcd0 0x7f8bb00b21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:15.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.367+0000 7f8bca2d1700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8bb00afcd0 0x7f8bb00b21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:15.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.367+0000 7f8bbb7fe700 1 -- 172.21.15.31:0/1781004646 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f8bbc0de550 con 0x7f8bc4109720 2024-04-25T15:58:15.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.367+0000 7f8bca2d1700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8bb00afcd0 0x7f8bb00b21e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8bc000b580 tx=0x7f8bc0009610 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:15.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:15 smithi063 ceph-mon[32413]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:15.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:15 smithi063 ceph-mon[32413]: osdmap e71: 12 total, 12 up, 12 in 2024-04-25T15:58:15.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.370+0000 7f8bbb7fe700 1 -- 172.21.15.31:0/1781004646 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8bbc010070 con 0x7f8bc4109720 2024-04-25T15:58:15.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:15 smithi186 ceph-mon[33258]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:15.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:15 smithi186 ceph-mon[33258]: osdmap e71: 12 total, 12 up, 12 in 2024-04-25T15:58:15.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.579+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8bc402e510 con 0x7f8bc4109720 2024-04-25T15:58:15.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.581+0000 7f8bbb7fe700 1 -- 172.21.15.31:0/1781004646 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+18420 (secure 0 0 0) 0x7f8bbc0a75b0 con 0x7f8bc4109720 2024-04-25T15:58:15.583 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:15.583 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:58:14.039554+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8bb00afcd0 msgr2=0x7f8bb00b21e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8bb00afcd0 0x7f8bb00b21e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8bc000b580 tx=0x7f8bc0009610 comp rx=0 tx=0).stop 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc4109720 msgr2=0x7f8bc419aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc4109720 0x7f8bc419aa90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc007ee0 tx=0x7f8bbc00d3b0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 shutdown_connections 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8bb00afcd0 0x7f8bb00b21e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8bc41044a0 0x7f8bc41a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8bc4109720 0x7f8bc419aa90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 --2- 172.21.15.31:0/1781004646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8bc4108dd0 0x7f8bc41a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:15.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 >> 172.21.15.31:0/1781004646 conn(0x7f8bc40ffd70 msgr2=0x7f8bc4108040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 shutdown_connections 2024-04-25T15:58:15.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:15.583+0000 7f8bcc535700 1 -- 172.21.15.31:0/1781004646 wait complete. 2024-04-25T15:58:16.367 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:16 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1781004646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:58:16.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:16 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1781004646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:58:16.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:16 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1781004646' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:58:16.403 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:58:16.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-25T15:58:16.657 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:58:17.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:17 smithi063 ceph-mon[32413]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:17.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:17 smithi186 ceph-mon[33258]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:17.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:17 smithi031 ceph-mon[29140]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.604+0000 7f962038d700 1 -- 172.21.15.31:0/3376427117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618107c80 msgr2=0x7f9618108090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.604+0000 7f962038d700 1 --2- 172.21.15.31:0/3376427117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618107c80 0x7f9618108090 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f96080099c0 tx=0x7f9608009cd0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.604+0000 7f962038d700 1 -- 172.21.15.31:0/3376427117 shutdown_connections 2024-04-25T15:58:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.604+0000 7f962038d700 1 --2- 172.21.15.31:0/3376427117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96181085d0 0x7f961810ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.604+0000 7f962038d700 1 --2- 172.21.15.31:0/3376427117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9618103350 0x7f9618103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.604+0000 7f962038d700 1 --2- 172.21.15.31:0/3376427117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618107c80 0x7f9618108090 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.604+0000 7f962038d700 1 -- 172.21.15.31:0/3376427117 >> 172.21.15.31:0/3376427117 conn(0x7f96180fec50 msgr2=0x7f9618101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.604+0000 7f962038d700 1 -- 172.21.15.31:0/3376427117 shutdown_connections 2024-04-25T15:58:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.604+0000 7f962038d700 1 -- 172.21.15.31:0/3376427117 wait complete. 2024-04-25T15:58:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f962038d700 1 Processor -- start 2024-04-25T15:58:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f962038d700 1 -- start start 2024-04-25T15:58:18.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f962038d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618103350 0x7f9618075b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f962038d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9618107c80 0x7f96180760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f962038d700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96181085d0 0x7f961819f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f962038d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96181131e0 con 0x7f9618103350 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f962038d700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9618113060 con 0x7f96181085d0 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f962038d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9618113360 con 0x7f9618107c80 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f961e129700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618103350 0x7f9618075b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f961e129700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618103350 0x7f9618075b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54234/0 (socket says 172.21.15.31:54234) 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f961e129700 1 -- 172.21.15.31:0/686324821 learned_addr learned my addr 172.21.15.31:0/686324821 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f961e92a700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96181085d0 0x7f961819f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.605+0000 7f961d928700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9618107c80 0x7f96180760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f961e129700 1 -- 172.21.15.31:0/686324821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9618107c80 msgr2=0x7f96180760c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f961e129700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9618107c80 0x7f96180760c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f961e129700 1 -- 172.21.15.31:0/686324821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96181085d0 msgr2=0x7f961819f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f961e129700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96181085d0 0x7f961819f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f961e129700 1 -- 172.21.15.31:0/686324821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f961819fc40 con 0x7f9618103350 2024-04-25T15:58:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f961e129700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618103350 0x7f9618075b60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f961400d980 tx=0x7f961400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f960f7fe700 1 -- 172.21.15.31:0/686324821 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9614016070 con 0x7f9618103350 2024-04-25T15:58:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f961819ff30 con 0x7f9618103350 2024-04-25T15:58:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f960f7fe700 1 -- 172.21.15.31:0/686324821 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9614004830 con 0x7f9618103350 2024-04-25T15:58:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f960f7fe700 1 -- 172.21.15.31:0/686324821 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9614005540 con 0x7f9618103350 2024-04-25T15:58:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.606+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96181a7850 con 0x7f9618103350 2024-04-25T15:58:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.607+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9618051190 con 0x7f9618103350 2024-04-25T15:58:18.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.609+0000 7f960f7fe700 1 -- 172.21.15.31:0/686324821 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9614004220 con 0x7f9618103350 2024-04-25T15:58:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.609+0000 7f960f7fe700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96040afb80 0x7f96040b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.610+0000 7f961d928700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96040afb80 0x7f96040b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.610+0000 7f960f7fe700 1 -- 172.21.15.31:0/686324821 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f96140deab0 con 0x7f9618103350 2024-04-25T15:58:18.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.610+0000 7f961d928700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96040afb80 0x7f96040b2090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9608006010 tx=0x7f9608000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.613+0000 7f960f7fe700 1 -- 172.21.15.31:0/686324821 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f961401f020 con 0x7f9618103350 2024-04-25T15:58:18.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.820+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f961802e510 con 0x7f9618103350 2024-04-25T15:58:18.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.821+0000 7f960f7fe700 1 -- 172.21.15.31:0/686324821 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7f96140a7b30 con 0x7f9618103350 2024-04-25T15:58:18.822 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T15:58:18.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96040afb80 msgr2=0x7f96040b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:18.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96040afb80 0x7f96040b2090 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9608006010 tx=0x7f9608000bc0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618103350 msgr2=0x7f9618075b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:18.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618103350 0x7f9618075b60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f961400d980 tx=0x7f961400dc90 comp rx=0 tx=0).stop 2024-04-25T15:58:18.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 shutdown_connections 2024-04-25T15:58:18.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96040afb80 0x7f96040b2090 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96181085d0 0x7f961819f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9618103350 0x7f9618075b60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 --2- 172.21.15.31:0/686324821 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9618107c80 0x7f96180760c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:18.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 >> 172.21.15.31:0/686324821 conn(0x7f96180fec50 msgr2=0x7f9618109530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:18.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 shutdown_connections 2024-04-25T15:58:18.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:18.824+0000 7f962038d700 1 -- 172.21.15.31:0/686324821 wait complete. 2024-04-25T15:58:19.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:19 smithi031 ceph-mon[29140]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:19.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:19 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/686324821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T15:58:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:19 smithi063 ceph-mon[32413]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:19 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/686324821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T15:58:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:19 smithi186 ceph-mon[33258]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:19 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/686324821' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T15:58:19.627 INFO:tasks.cephadm:Adding mds.a on smithi031 2024-04-25T15:58:19.627 INFO:tasks.cephadm:Adding mds.d on smithi031 2024-04-25T15:58:19.627 INFO:tasks.cephadm:Adding mds.g on smithi031 2024-04-25T15:58:19.627 INFO:tasks.cephadm:Adding mds.j on smithi031 2024-04-25T15:58:19.627 INFO:tasks.cephadm:Adding mds.b on smithi063 2024-04-25T15:58:19.627 INFO:tasks.cephadm:Adding mds.e on smithi063 2024-04-25T15:58:19.628 INFO:tasks.cephadm:Adding mds.h on smithi063 2024-04-25T15:58:19.628 INFO:tasks.cephadm:Adding mds.k on smithi063 2024-04-25T15:58:19.628 INFO:tasks.cephadm:Adding mds.c on smithi186 2024-04-25T15:58:19.628 INFO:tasks.cephadm:Adding mds.f on smithi186 2024-04-25T15:58:19.628 INFO:tasks.cephadm:Adding mds.i on smithi186 2024-04-25T15:58:19.628 INFO:tasks.cephadm:Adding mds.l on smithi186 2024-04-25T15:58:19.629 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l' 2024-04-25T15:58:19.879 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T15:58:21.321 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:21 smithi186 ceph-mon[33258]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:21.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:21 smithi063 ceph-mon[32413]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:21.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:21 smithi031 ceph-mon[29140]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:21.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 -- 172.21.15.186:0/3629225401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104080 msgr2=0x7f8a74104550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:21.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 --2- 172.21.15.186:0/3629225401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104080 0x7f8a74104550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8a68009990 tx=0x7f8a68009ca0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 -- 172.21.15.186:0/3629225401 shutdown_connections 2024-04-25T15:58:21.562 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 --2- 172.21.15.186:0/3629225401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a741036a0 0x7f8a74103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 --2- 172.21.15.186:0/3629225401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a74104be0 0x7f8a7410b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 --2- 172.21.15.186:0/3629225401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104080 0x7f8a74104550 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 -- 172.21.15.186:0/3629225401 >> 172.21.15.186:0/3629225401 conn(0x7f8a740fdef0 msgr2=0x7f8a741003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:21.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 -- 172.21.15.186:0/3629225401 shutdown_connections 2024-04-25T15:58:21.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 -- 172.21.15.186:0/3629225401 wait complete. 2024-04-25T15:58:21.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 Processor -- start 2024-04-25T15:58:21.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.561+0000 7f8a7a325700 1 -- start start 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a7a325700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a741036a0 0x7f8a741996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a7a325700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a74104080 0x7f8a74199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a7a325700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104be0 0x7f8a741a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a7a325700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7410eba0 con 0x7f8a741036a0 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a7a325700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7410ea20 con 0x7f8a74104080 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a7a325700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a7410ed20 con 0x7f8a74104be0 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a788c2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104be0 0x7f8a741a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a788c2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104be0 0x7f8a741a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:54930/0 (socket says 172.21.15.186:54930) 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a788c2700 1 -- 172.21.15.186:0/277766539 learned_addr learned my addr 172.21.15.186:0/277766539 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a737fe700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a74104080 0x7f8a74199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a73fff700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a741036a0 0x7f8a741996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a788c2700 1 -- 172.21.15.186:0/277766539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a74104080 msgr2=0x7f8a74199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a788c2700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a74104080 0x7f8a74199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.565 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a788c2700 1 -- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a741036a0 msgr2=0x7f8a741996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a788c2700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a741036a0 0x7f8a741996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a788c2700 1 -- 172.21.15.186:0/277766539 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a741a1210 con 0x7f8a74104be0 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a73fff700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a741036a0 0x7f8a741996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a788c2700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104be0 0x7f8a741a0b90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c007ee0 tx=0x7f8a6c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.562+0000 7f8a717fa700 1 -- 172.21.15.186:0/277766539 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c00dbd0 con 0x7f8a74104be0 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.563+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a741a1500 con 0x7f8a74104be0 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.563+0000 7f8a717fa700 1 -- 172.21.15.186:0/277766539 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a6c00dd70 con 0x7f8a74104be0 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.563+0000 7f8a717fa700 1 -- 172.21.15.186:0/277766539 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c005990 con 0x7f8a74104be0 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.563+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a741a1a40 con 0x7f8a74104be0 2024-04-25T15:58:21.566 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.564+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a74051190 con 0x7f8a74104be0 2024-04-25T15:58:21.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.565+0000 7f8a717fa700 1 -- 172.21.15.186:0/277766539 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8a6c01a020 con 0x7f8a74104be0 2024-04-25T15:58:21.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.566+0000 7f8a717fa700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a5c0afb80 0x7f8a5c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:21.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.566+0000 7f8a717fa700 1 -- 172.21.15.186:0/277766539 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f8a6c0de580 con 0x7f8a74104be0 2024-04-25T15:58:21.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.566+0000 7f8a73fff700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a5c0afb80 0x7f8a5c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:21.567 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.567+0000 7f8a73fff700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a5c0afb80 0x7f8a5c0b2090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8a64005950 tx=0x7f8a640058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:21.570 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.569+0000 7f8a717fa700 1 -- 172.21.15.186:0/277766539 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a6c010070 con 0x7f8a74104be0 2024-04-25T15:58:21.779 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.777+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}) -- 0x7f8a74063bd0 con 0x7f8a5c0afb80 2024-04-25T15:58:21.788 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.787+0000 7f8a717fa700 1 -- 172.21.15.186:0/277766539 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f8a74063bd0 con 0x7f8a5c0afb80 2024-04-25T15:58:21.788 INFO:teuthology.orchestra.run.smithi186.stdout:Scheduled mds.all update... 2024-04-25T15:58:21.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a5c0afb80 msgr2=0x7f8a5c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:21.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a5c0afb80 0x7f8a5c0b2090 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8a64005950 tx=0x7f8a640058e0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.790 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104be0 msgr2=0x7f8a741a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:21.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104be0 0x7f8a741a0b90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c007ee0 tx=0x7f8a6c00d3b0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 shutdown_connections 2024-04-25T15:58:21.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a5c0afb80 0x7f8a5c0b2090 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a74104be0 0x7f8a741a0b90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a741036a0 0x7f8a741996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 --2- 172.21.15.186:0/277766539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a74104080 0x7f8a74199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:21.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 >> 172.21.15.186:0/277766539 conn(0x7f8a740fdef0 msgr2=0x7f8a74107650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:21.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 shutdown_connections 2024-04-25T15:58:21.791 INFO:teuthology.orchestra.run.smithi186.stderr:2024-04-25T15:58:21.789+0000 7f8a7a325700 1 -- 172.21.15.186:0/277766539 wait complete. 2024-04-25T15:58:22.420 DEBUG:teuthology.orchestra.run.smithi031:mds.a> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.a.service 2024-04-25T15:58:22.423 DEBUG:teuthology.orchestra.run.smithi031:mds.d> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.d.service 2024-04-25T15:58:22.467 DEBUG:teuthology.orchestra.run.smithi031:mds.g> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.g.service 2024-04-25T15:58:22.509 DEBUG:teuthology.orchestra.run.smithi031:mds.j> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.j.service 2024-04-25T15:58:22.553 DEBUG:teuthology.orchestra.run.smithi063:mds.b> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.b.service 2024-04-25T15:58:22.555 DEBUG:teuthology.orchestra.run.smithi063:mds.e> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.e.service 2024-04-25T15:58:22.598 DEBUG:teuthology.orchestra.run.smithi063:mds.h> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.h.service 2024-04-25T15:58:22.640 DEBUG:teuthology.orchestra.run.smithi063:mds.k> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.k.service 2024-04-25T15:58:22.683 DEBUG:teuthology.orchestra.run.smithi186:mds.c> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.c.service 2024-04-25T15:58:22.686 DEBUG:teuthology.orchestra.run.smithi186:mds.f> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.f.service 2024-04-25T15:58:22.729 DEBUG:teuthology.orchestra.run.smithi186:mds.i> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.i.service 2024-04-25T15:58:22.773 DEBUG:teuthology.orchestra.run.smithi186:mds.l> sudo journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.l.service 2024-04-25T15:58:22.816 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-25T15:58:22.816 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:58:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:58:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-25T15:58:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:22.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:58:22.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:22.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:22.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:22.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:22 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.942+0000 7f50cd642640 1 -- 172.21.15.31:0/3828172952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 msgr2=0x7f50c81090e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.942+0000 7f50cd642640 1 --2- 172.21.15.31:0/3828172952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 0x7f50c81090e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f50bc009900 tx=0x7f50bc02a410 comp rx=0 tx=0).stop 2024-04-25T15:58:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.942+0000 7f50cd642640 1 -- 172.21.15.31:0/3828172952 shutdown_connections 2024-04-25T15:58:22.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.942+0000 7f50cd642640 1 --2- 172.21.15.31:0/3828172952 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50c8109770 0x7f50c810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.942+0000 7f50cd642640 1 --2- 172.21.15.31:0/3828172952 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 0x7f50c81090e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.942+0000 7f50cd642640 1 --2- 172.21.15.31:0/3828172952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c8104210 0x7f50c8106630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.942+0000 7f50cd642640 1 -- 172.21.15.31:0/3828172952 >> 172.21.15.31:0/3828172952 conn(0x7f50c80fdbd0 msgr2=0x7f50c8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.942+0000 7f50cd642640 1 -- 172.21.15.31:0/3828172952 shutdown_connections 2024-04-25T15:58:22.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.942+0000 7f50cd642640 1 -- 172.21.15.31:0/3828172952 wait complete. 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50cd642640 1 Processor -- start 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50cd642640 1 -- start start 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50cd642640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50c8104210 0x7f50c8103a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50cd642640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 0x7f50c8102120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50cd642640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c8109770 0x7f50c8102660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50cd642640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f50c810f5b0 con 0x7f50c8106c00 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50cd642640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f50c810f430 con 0x7f50c8104210 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50cd642640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f50c810f730 con 0x7f50c8109770 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50c67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 0x7f50c8102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50c67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 0x7f50c8102120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54260/0 (socket says 172.21.15.31:54260) 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.943+0000 7f50c67fc640 1 -- 172.21.15.31:0/1127502402 learned_addr learned my addr 172.21.15.31:0/1127502402 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50c77fe640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c8109770 0x7f50c8102660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50c6ffd640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50c8104210 0x7f50c8103a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:22.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50c67fc640 1 -- 172.21.15.31:0/1127502402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c8109770 msgr2=0x7f50c8102660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50c67fc640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c8109770 0x7f50c8102660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50c67fc640 1 -- 172.21.15.31:0/1127502402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50c8104210 msgr2=0x7f50c8103a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50c67fc640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50c8104210 0x7f50c8103a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50c67fc640 1 -- 172.21.15.31:0/1127502402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c8102d90 con 0x7f50c8106c00 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50c67fc640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 0x7f50c8102120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f50bc006010 tx=0x7f50bc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50affff640 1 -- 172.21.15.31:0/1127502402 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc03f030 con 0x7f50c8106c00 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50affff640 1 -- 172.21.15.31:0/1127502402 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50bc003aa0 con 0x7f50c8106c00 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50c81a4280 con 0x7f50c8106c00 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50affff640 1 -- 172.21.15.31:0/1127502402 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50bc035070 con 0x7f50c8106c00 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.944+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50c81a4680 con 0x7f50c8106c00 2024-04-25T15:58:22.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.945+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50c80ff6f0 con 0x7f50c8106c00 2024-04-25T15:58:22.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.947+0000 7f50affff640 1 -- 172.21.15.31:0/1127502402 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f50bc042030 con 0x7f50c8106c00 2024-04-25T15:58:22.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.948+0000 7f50affff640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50b00afaf0 0x7f50b00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:22.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.948+0000 7f50c6ffd640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50b00afaf0 0x7f50b00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:22.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.948+0000 7f50affff640 1 -- 172.21.15.31:0/1127502402 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f50bc0ff490 con 0x7f50c8106c00 2024-04-25T15:58:22.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.949+0000 7f50c6ffd640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50b00afaf0 0x7f50b00b2000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f50b8009b60 tx=0x7f50b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:22.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:22.952+0000 7f50affff640 1 -- 172.21.15.31:0/1127502402 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f50bc0c8470 con 0x7f50c8106c00 2024-04-25T15:58:23.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.140+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f50c80637a0 con 0x7f50c8106c00 2024-04-25T15:58:23.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.141+0000 7f50affff640 1 -- 172.21.15.31:0/1127502402 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+18420 (secure 0 0 0) 0x7f50bc0c7d70 con 0x7f50c8106c00 2024-04-25T15:58:23.142 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:23.142 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:58:14.039554+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:58:23.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50b00afaf0 msgr2=0x7f50b00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50b00afaf0 0x7f50b00b2000 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f50b8009b60 tx=0x7f50b8005f90 comp rx=0 tx=0).stop 2024-04-25T15:58:23.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 msgr2=0x7f50c8102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 0x7f50c8102120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f50bc006010 tx=0x7f50bc031a50 comp rx=0 tx=0).stop 2024-04-25T15:58:23.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 shutdown_connections 2024-04-25T15:58:23.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50b00afaf0 0x7f50b00b2000 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50c8109770 0x7f50c8102660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50c8106c00 0x7f50c8102120 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 --2- 172.21.15.31:0/1127502402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50c8104210 0x7f50c8103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 >> 172.21.15.31:0/1127502402 conn(0x7f50c80fdbd0 msgr2=0x7f50c8106210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:23.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 shutdown_connections 2024-04-25T15:58:23.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.143+0000 7f50cd642640 1 -- 172.21.15.31:0/1127502402 wait complete. 2024-04-25T15:58:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:58:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-25T15:58:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:23.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:58:23.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:23.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:23.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:23.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:22 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:23.154 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: Saving service mds.all spec with placement smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:23.155 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:22 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:23.157 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:58:23.157 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:58:23.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 -- 172.21.15.31:0/1167610284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 msgr2=0x7f0e7c10ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 --2- 172.21.15.31:0/1167610284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 0x7f0e7c10ad50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009ca0 tx=0x7f0e7402a8c0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 -- 172.21.15.31:0/1167610284 shutdown_connections 2024-04-25T15:58:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 --2- 172.21.15.31:0/1167610284 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 0x7f0e7c10ad50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 --2- 172.21.15.31:0/1167610284 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e7c103c30 0x7f0e7c104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 --2- 172.21.15.31:0/1167610284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e7c1029f0 0x7f0e7c1036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 -- 172.21.15.31:0/1167610284 >> 172.21.15.31:0/1167610284 conn(0x7f0e7c0fdc00 msgr2=0x7f0e7c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 -- 172.21.15.31:0/1167610284 shutdown_connections 2024-04-25T15:58:23.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 -- 172.21.15.31:0/1167610284 wait complete. 2024-04-25T15:58:23.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.271+0000 7f0e80a92640 1 Processor -- start 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e80a92640 1 -- start start 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e80a92640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e7c1029f0 0x7f0e7c06bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e80a92640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e7c103c30 0x7f0e7c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e80a92640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 0x7f0e7c1a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e80a92640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c10e520 con 0x7f0e7c108930 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e80a92640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c10e3a0 con 0x7f0e7c1029f0 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e80a92640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e7c10e6a0 con 0x7f0e7c103c30 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 0x7f0e7c1a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 0x7f0e7c1a14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54270/0 (socket says 172.21.15.31:54270) 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7ad9d640 1 -- 172.21.15.31:0/658036928 learned_addr learned my addr 172.21.15.31:0/658036928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e73fff640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e7c103c30 0x7f0e7c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7a59c640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e7c1029f0 0x7f0e7c06bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7ad9d640 1 -- 172.21.15.31:0/658036928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e7c103c30 msgr2=0x7f0e7c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7ad9d640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e7c103c30 0x7f0e7c19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7ad9d640 1 -- 172.21.15.31:0/658036928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e7c1029f0 msgr2=0x7f0e7c06bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7ad9d640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e7c1029f0 0x7f0e7c06bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7ad9d640 1 -- 172.21.15.31:0/658036928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7c1a1ce0 con 0x7f0e7c108930 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e73fff640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e7c103c30 0x7f0e7c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.272+0000 7f0e7ad9d640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 0x7f0e7c1a14c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0e74004af0 tx=0x7f0e74031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.273+0000 7f0e737fe640 1 -- 172.21.15.31:0/658036928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e74003b70 con 0x7f0e7c108930 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.273+0000 7f0e737fe640 1 -- 172.21.15.31:0/658036928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e74003d10 con 0x7f0e7c108930 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.273+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e7c1a1f70 con 0x7f0e7c108930 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.273+0000 7f0e737fe640 1 -- 172.21.15.31:0/658036928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7402fb20 con 0x7f0e7c108930 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.273+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e7c1a23d0 con 0x7f0e7c108930 2024-04-25T15:58:23.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.274+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e7c0692e0 con 0x7f0e7c108930 2024-04-25T15:58:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.275+0000 7f0e737fe640 1 -- 172.21.15.31:0/658036928 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f0e74004580 con 0x7f0e7c108930 2024-04-25T15:58:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.276+0000 7f0e737fe640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e680afaf0 0x7f0e680b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.276+0000 7f0e7a59c640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e680afaf0 0x7f0e680b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.276+0000 7f0e737fe640 1 -- 172.21.15.31:0/658036928 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f0e740fff30 con 0x7f0e7c108930 2024-04-25T15:58:23.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.276+0000 7f0e7a59c640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e680afaf0 0x7f0e680b2000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c1035f0 tx=0x7f0e6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:23.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.279+0000 7f0e737fe640 1 -- 172.21.15.31:0/658036928 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0e7412eff0 con 0x7f0e7c108930 2024-04-25T15:58:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.467+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0e7c02dbd0 con 0x7f0e7c108930 2024-04-25T15:58:23.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.467+0000 7f0e737fe640 1 -- 172.21.15.31:0/658036928 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7f0e740c8f10 con 0x7f0e7c108930 2024-04-25T15:58:23.468 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e680afaf0 msgr2=0x7f0e680b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e680afaf0 0x7f0e680b2000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c1035f0 tx=0x7f0e6c005f90 comp rx=0 tx=0).stop 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 msgr2=0x7f0e7c1a14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 0x7f0e7c1a14c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f0e74004af0 tx=0x7f0e74031700 comp rx=0 tx=0).stop 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 shutdown_connections 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e680afaf0 0x7f0e680b2000 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e7c108930 0x7f0e7c1a14c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e7c103c30 0x7f0e7c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 --2- 172.21.15.31:0/658036928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e7c1029f0 0x7f0e7c06bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 >> 172.21.15.31:0/658036928 conn(0x7f0e7c0fdc00 msgr2=0x7f0e7c104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 shutdown_connections 2024-04-25T15:58:23.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.469+0000 7f0e80a92640 1 -- 172.21.15.31:0/658036928 wait complete. 2024-04-25T15:58:23.481 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:23.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.593+0000 7fa0f3c77640 1 -- 172.21.15.31:0/856786797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec1098b0 msgr2=0x7fa0ec10bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.593+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/856786797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec1098b0 0x7fa0ec10bcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009ca0 tx=0x7fa0e402a8c0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.593+0000 7fa0f3c77640 1 -- 172.21.15.31:0/856786797 shutdown_connections 2024-04-25T15:58:23.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.593+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/856786797 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec1098b0 0x7fa0ec10bcd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.593+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/856786797 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0ec106f50 0x7fa0ec109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.593+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/856786797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0ec1045f0 0x7fa0ec106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.593+0000 7fa0f3c77640 1 -- 172.21.15.31:0/856786797 >> 172.21.15.31:0/856786797 conn(0x7fa0ec0fdbf0 msgr2=0x7fa0ec1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.593+0000 7fa0f3c77640 1 -- 172.21.15.31:0/856786797 shutdown_connections 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.593+0000 7fa0f3c77640 1 -- 172.21.15.31:0/856786797 wait complete. 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f3c77640 1 Processor -- start 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f3c77640 1 -- start start 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f3c77640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0ec1045f0 0x7fa0ec19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f3c77640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec106f50 0x7fa0ec19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f3c77640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0ec1098b0 0x7fa0ec1a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f3c77640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ec10f630 con 0x7fa0ec106f50 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f3c77640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ec10f4b0 con 0x7fa0ec1098b0 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f3c77640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa0ec10f7b0 con 0x7fa0ec1045f0 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1212640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec106f50 0x7fa0ec19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1212640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec106f50 0x7fa0ec19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:54280/0 (socket says 172.21.15.31:54280) 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1212640 1 -- 172.21.15.31:0/2036704851 learned_addr learned my addr 172.21.15.31:0/2036704851 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:23.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1a13640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0ec1045f0 0x7fa0ec19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f2214640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0ec1098b0 0x7fa0ec1a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1212640 1 -- 172.21.15.31:0/2036704851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0ec1045f0 msgr2=0x7fa0ec19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1212640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0ec1045f0 0x7fa0ec19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1212640 1 -- 172.21.15.31:0/2036704851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0ec1098b0 msgr2=0x7fa0ec1a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1212640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0ec1098b0 0x7fa0ec1a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1212640 1 -- 172.21.15.31:0/2036704851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0ec1a1c30 con 0x7fa0ec106f50 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f2214640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0ec1098b0 0x7fa0ec1a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.594+0000 7fa0f1212640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec106f50 0x7fa0ec19a6e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00c920 tx=0x7fa0dc00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.595+0000 7fa0e2ffd640 1 -- 172.21.15.31:0/2036704851 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc007c70 con 0x7fa0ec106f50 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.595+0000 7fa0e2ffd640 1 -- 172.21.15.31:0/2036704851 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0dc007e10 con 0x7fa0ec106f50 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.595+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0ec1a1f20 con 0x7fa0ec106f50 2024-04-25T15:58:23.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.595+0000 7fa0e2ffd640 1 -- 172.21.15.31:0/2036704851 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc018820 con 0x7fa0ec106f50 2024-04-25T15:58:23.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.595+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0ec1a2390 con 0x7fa0ec106f50 2024-04-25T15:58:23.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.596+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0ec0ff710 con 0x7fa0ec106f50 2024-04-25T15:58:23.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.597+0000 7fa0e2ffd640 1 -- 172.21.15.31:0/2036704851 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa0dc00f450 con 0x7fa0ec106f50 2024-04-25T15:58:23.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.598+0000 7fa0e2ffd640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0d80afaf0 0x7fa0d80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:23.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.598+0000 7fa0f1a13640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0d80afaf0 0x7fa0d80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:23.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.598+0000 7fa0e2ffd640 1 -- 172.21.15.31:0/2036704851 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fa0dc0dec00 con 0x7fa0ec106f50 2024-04-25T15:58:23.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.598+0000 7fa0f1a13640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0d80afaf0 0x7fa0d80b2000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0757f0 tx=0x7fa0e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:23.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.601+0000 7fa0e2ffd640 1 -- 172.21.15.31:0/2036704851 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa0dc014030 con 0x7fa0ec106f50 2024-04-25T15:58:23.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.829+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0ec02dda0 con 0x7fa0ec106f50 2024-04-25T15:58:23.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.830+0000 7fa0e2ffd640 1 -- 172.21.15.31:0/2036704851 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fa0dc0a7c70 con 0x7fa0ec106f50 2024-04-25T15:58:23.831 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:23.831 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-25T15:58:23.831 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 1 2024-04-25T15:58:23.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.831+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0d80afaf0 msgr2=0x7fa0d80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.831+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0d80afaf0 0x7fa0d80b2000 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0757f0 tx=0x7fa0e8005f90 comp rx=0 tx=0).stop 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.831+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec106f50 msgr2=0x7fa0ec19a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.831+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec106f50 0x7fa0ec19a6e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00c920 tx=0x7fa0dc00cde0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.832+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 shutdown_connections 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.832+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0d80afaf0 0x7fa0d80b2000 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.832+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa0ec1098b0 0x7fa0ec1a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.832+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa0ec106f50 0x7fa0ec19a6e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.832+0000 7fa0f3c77640 1 --2- 172.21.15.31:0/2036704851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0ec1045f0 0x7fa0ec19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.832+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 >> 172.21.15.31:0/2036704851 conn(0x7fa0ec0fdbf0 msgr2=0x7fa0ec1054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.832+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 shutdown_connections 2024-04-25T15:58:23.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:23.832+0000 7fa0f3c77640 1 -- 172.21.15.31:0/2036704851 wait complete. 2024-04-25T15:58:24.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:23 smithi063 ceph-mon[32413]: Deploying daemon mds.l on smithi186 2024-04-25T15:58:24.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:23 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1127502402' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:58:24.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:23 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/658036928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T15:58:24.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:23 smithi186 ceph-mon[33258]: Deploying daemon mds.l on smithi186 2024-04-25T15:58:24.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:23 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1127502402' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:58:24.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:23 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/658036928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T15:58:24.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:23 smithi031 ceph-mon[29140]: Deploying daemon mds.l on smithi186 2024-04-25T15:58:24.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:23 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1127502402' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:58:24.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:23 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/658036928' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T15:58:24.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:24 smithi186 ceph-mon[33258]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:24.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:24 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2036704851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:24 smithi063 ceph-mon[32413]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:24 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2036704851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:25.190 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:24 smithi031 ceph-mon[29140]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:25.191 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:24 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2036704851' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:25.648 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 15:58:25 smithi186 systemd[1]: Started Ceph mds.l for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:25.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:25.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.972+0000 7f3843d9d640 1 -- 172.21.15.31:0/2636692802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c104d40 msgr2=0x7f383c105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:25.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.972+0000 7f3843d9d640 1 --2- 172.21.15.31:0/2636692802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c104d40 0x7f383c105150 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f382c009900 tx=0x7f382c02a410 comp rx=0 tx=0).stop 2024-04-25T15:58:25.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.972+0000 7f3843d9d640 1 -- 172.21.15.31:0/2636692802 shutdown_connections 2024-04-25T15:58:25.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.972+0000 7f3843d9d640 1 --2- 172.21.15.31:0/2636692802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f383c109a40 0x7f383c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.972+0000 7f3843d9d640 1 --2- 172.21.15.31:0/2636692802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c104d40 0x7f383c105150 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.972+0000 7f3843d9d640 1 --2- 172.21.15.31:0/2636692802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f383c103b00 0x7f383c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.972+0000 7f3843d9d640 1 -- 172.21.15.31:0/2636692802 >> 172.21.15.31:0/2636692802 conn(0x7f383c0fed10 msgr2=0x7f383c1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.972+0000 7f3843d9d640 1 -- 172.21.15.31:0/2636692802 shutdown_connections 2024-04-25T15:58:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.972+0000 7f3843d9d640 1 -- 172.21.15.31:0/2636692802 wait complete. 2024-04-25T15:58:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3843d9d640 1 Processor -- start 2024-04-25T15:58:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3843d9d640 1 -- start start 2024-04-25T15:58:25.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3843d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c103b00 0x7f383c19a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:25.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3843d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f383c104d40 0x7f383c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:25.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3843d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f383c109a40 0x7f383c1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:25.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3843d9d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f383c10f630 con 0x7f383c103b00 2024-04-25T15:58:25.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3843d9d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f383c10f4b0 con 0x7f383c109a40 2024-04-25T15:58:25.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3843d9d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f383c10f7b0 con 0x7f383c104d40 2024-04-25T15:58:25.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3841b39640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c103b00 0x7f383c19a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:25.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3841b39640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c103b00 0x7f383c19a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33594/0 (socket says 172.21.15.31:33594) 2024-04-25T15:58:25.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3841b39640 1 -- 172.21.15.31:0/3497137883 learned_addr learned my addr 172.21.15.31:0/3497137883 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3841338640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f383c104d40 0x7f383c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f384233a640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f383c109a40 0x7f383c1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3841b39640 1 -- 172.21.15.31:0/3497137883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f383c104d40 msgr2=0x7f383c19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3841b39640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f383c104d40 0x7f383c19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3841b39640 1 -- 172.21.15.31:0/3497137883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f383c109a40 msgr2=0x7f383c1a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3841b39640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f383c109a40 0x7f383c1a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.973+0000 7f3841b39640 1 -- 172.21.15.31:0/3497137883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c1a1d20 con 0x7f383c103b00 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.974+0000 7f3841b39640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c103b00 0x7f383c19a1e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f383800e9f0 tx=0x7f383800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.974+0000 7f3832ffd640 1 -- 172.21.15.31:0/3497137883 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383800ccc0 con 0x7f383c103b00 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.974+0000 7f3832ffd640 1 -- 172.21.15.31:0/3497137883 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38380035f0 con 0x7f383c103b00 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.974+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c1a1fb0 con 0x7f383c103b00 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.974+0000 7f3832ffd640 1 -- 172.21.15.31:0/3497137883 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f383800ce60 con 0x7f383c103b00 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.974+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c1a2470 con 0x7f383c103b00 2024-04-25T15:58:25.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.975+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c108850 con 0x7f383c103b00 2024-04-25T15:58:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.977+0000 7f3832ffd640 1 -- 172.21.15.31:0/3497137883 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3838003aa0 con 0x7f383c103b00 2024-04-25T15:58:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.978+0000 7f3832ffd640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38280afaf0 0x7f38280b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.978+0000 7f3841338640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38280afaf0 0x7f38280b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.978+0000 7f3832ffd640 1 -- 172.21.15.31:0/3497137883 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f38380dee30 con 0x7f383c103b00 2024-04-25T15:58:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.979+0000 7f3841338640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38280afaf0 0x7f38280b2000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f382c000c00 tx=0x7f382c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:25.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:25.982+0000 7f3832ffd640 1 -- 172.21.15.31:0/3497137883 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3838053ff0 con 0x7f383c103b00 2024-04-25T15:58:26.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.213+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f383c02db50 con 0x7f383c103b00 2024-04-25T15:58:26.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.214+0000 7f3832ffd640 1 -- 172.21.15.31:0/3497137883 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f38380a7e10 con 0x7f383c103b00 2024-04-25T15:58:26.215 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:26.215 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T15:58:26.215 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 2 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38280afaf0 msgr2=0x7f38280b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38280afaf0 0x7f38280b2000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f382c000c00 tx=0x7f382c031400 comp rx=0 tx=0).stop 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c103b00 msgr2=0x7f383c19a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c103b00 0x7f383c19a1e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f383800e9f0 tx=0x7f383800eeb0 comp rx=0 tx=0).stop 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 shutdown_connections 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38280afaf0 0x7f38280b2000 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f383c109a40 0x7f383c1a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f383c104d40 0x7f383c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 --2- 172.21.15.31:0/3497137883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f383c103b00 0x7f383c19a1e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 >> 172.21.15.31:0/3497137883 conn(0x7f383c0fed10 msgr2=0x7f383c105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 shutdown_connections 2024-04-25T15:58:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:26.216+0000 7f3843d9d640 1 -- 172.21.15.31:0/3497137883 wait complete. 2024-04-25T15:58:26.417 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:26.417 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6832/1701597397,v1:172.21.15.186:6833/1701597397] up:boot 2024-04-25T15:58:26.417 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: fsmap 1 up:standby 2024-04-25T15:58:26.417 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T15:58:26.417 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:26.417 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:26.417 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:26.418 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:26.418 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:26.418 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:26.418 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: Deploying daemon mds.k on smithi063 2024-04-25T15:58:26.418 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:26 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3497137883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:26.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:26.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6832/1701597397,v1:172.21.15.186:6833/1701597397] up:boot 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: fsmap 1 up:standby 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: Deploying daemon mds.k on smithi063 2024-04-25T15:58:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:26 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3497137883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:26.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:26.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6832/1701597397,v1:172.21.15.186:6833/1701597397] up:boot 2024-04-25T15:58:26.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: fsmap 1 up:standby 2024-04-25T15:58:26.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T15:58:26.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:26.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:26.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:26.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:26.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:26.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:26.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: Deploying daemon mds.k on smithi063 2024-04-25T15:58:26.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:26 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3497137883' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:28.119 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 15:58:27 smithi063 systemd[1]: Starting Ceph mds.k for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:58:28.229 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.355+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2855324689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 msgr2=0x7f61e8105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.355+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2855324689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 0x7f61e8105150 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f61dc009900 tx=0x7f61dc02a410 comp rx=0 tx=0).stop 2024-04-25T15:58:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.355+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2855324689 shutdown_connections 2024-04-25T15:58:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.355+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2855324689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e8109a40 0x7f61e810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.355+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2855324689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 0x7f61e8105150 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.355+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2855324689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61e8103b00 0x7f61e8104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:28.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.355+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2855324689 >> 172.21.15.31:0/2855324689 conn(0x7f61e80fed10 msgr2=0x7f61e81011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:28.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.355+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2855324689 shutdown_connections 2024-04-25T15:58:28.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.355+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2855324689 wait complete. 2024-04-25T15:58:28.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61ed1ce640 1 Processor -- start 2024-04-25T15:58:28.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61ed1ce640 1 -- start start 2024-04-25T15:58:28.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61ed1ce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e8103b00 0x7f61e819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:28.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61ed1ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 0x7f61e819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:28.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61ed1ce640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61e8109a40 0x7f61e81a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:28.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61ed1ce640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f61e810f680 con 0x7f61e8104d40 2024-04-25T15:58:28.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61ed1ce640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61e810f500 con 0x7f61e8109a40 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61ed1ce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61e810f800 con 0x7f61e8103b00 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61e659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 0x7f61e819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61e659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 0x7f61e819a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33618/0 (socket says 172.21.15.31:33618) 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.356+0000 7f61e659c640 1 -- 172.21.15.31:0/2821881121 learned_addr learned my addr 172.21.15.31:0/2821881121 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61e759e640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61e8109a40 0x7f61e81a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61e6d9d640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e8103b00 0x7f61e819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61e659c640 1 -- 172.21.15.31:0/2821881121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e8103b00 msgr2=0x7f61e819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61e659c640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e8103b00 0x7f61e819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61e659c640 1 -- 172.21.15.31:0/2821881121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61e8109a40 msgr2=0x7f61e81a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61e659c640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61e8109a40 0x7f61e81a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61e659c640 1 -- 172.21.15.31:0/2821881121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e81a1d90 con 0x7f61e8104d40 2024-04-25T15:58:28.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61e759e640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61e8109a40 0x7f61e81a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:58:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61e659c640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 0x7f61e819a790 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f61dc006010 tx=0x7f61dc031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61cffff640 1 -- 172.21.15.31:0/2821881121 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc03f030 con 0x7f61e8104d40 2024-04-25T15:58:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61e81a2020 con 0x7f61e8104d40 2024-04-25T15:58:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61cffff640 1 -- 172.21.15.31:0/2821881121 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61dc003aa0 con 0x7f61e8104d40 2024-04-25T15:58:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61cffff640 1 -- 172.21.15.31:0/2821881121 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61dc035070 con 0x7f61e8104d40 2024-04-25T15:58:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.357+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e81a25e0 con 0x7f61e8104d40 2024-04-25T15:58:28.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.358+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61e8108850 con 0x7f61e8104d40 2024-04-25T15:58:28.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.360+0000 7f61cffff640 1 -- 172.21.15.31:0/2821881121 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f61dc042030 con 0x7f61e8104d40 2024-04-25T15:58:28.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.361+0000 7f61cffff640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61d00afaf0 0x7f61d00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:28.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.361+0000 7f61e6d9d640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61d00afaf0 0x7f61d00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:28.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.361+0000 7f61cffff640 1 -- 172.21.15.31:0/2821881121 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f61dc0ff340 con 0x7f61e8104d40 2024-04-25T15:58:28.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.361+0000 7f61e6d9d640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61d00afaf0 0x7f61d00b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f61e8104700 tx=0x7f61d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:28.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.364+0000 7f61cffff640 1 -- 172.21.15.31:0/2821881121 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61dc0c7d70 con 0x7f61e8104d40 2024-04-25T15:58:28.411 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:28 smithi063 ceph-mon[32413]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:28.412 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 15:58:28 smithi063 podman[62009]: 2024-04-25 15:58:28.04872374 +0000 UTC m=+0.018237399 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:28.412 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 15:58:28 smithi063 podman[62009]: 2024-04-25 15:58:28.158695574 +0000 UTC m=+0.128209214 container create 184ad245f379892aa90c82db4d8f6ca103857c91743809d920d3f33f685790b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T15:58:28.412 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 15:58:28 smithi063 podman[62009]: 2024-04-25 15:58:28.411628158 +0000 UTC m=+0.381141798 container init 184ad245f379892aa90c82db4d8f6ca103857c91743809d920d3f33f685790b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD) 2024-04-25T15:58:28.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.589+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f61e802db50 con 0x7f61e8104d40 2024-04-25T15:58:28.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.590+0000 7f61cffff640 1 -- 172.21.15.31:0/2821881121 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f61dc0cc070 con 0x7f61e8104d40 2024-04-25T15:58:28.591 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:28.591 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T15:58:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 2 2024-04-25T15:58:28.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61d00afaf0 msgr2=0x7f61d00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61d00afaf0 0x7f61d00b2000 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f61e8104700 tx=0x7f61d8005f90 comp rx=0 tx=0).stop 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 msgr2=0x7f61e819a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 0x7f61e819a790 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f61dc006010 tx=0x7f61dc031a50 comp rx=0 tx=0).stop 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 shutdown_connections 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61d00afaf0 0x7f61d00b2000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61e8109a40 0x7f61e81a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61e8104d40 0x7f61e819a790 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 --2- 172.21.15.31:0/2821881121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e8103b00 0x7f61e819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 >> 172.21.15.31:0/2821881121 conn(0x7f61e80fed10 msgr2=0x7f61e8105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 shutdown_connections 2024-04-25T15:58:28.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:28.592+0000 7f61ed1ce640 1 -- 172.21.15.31:0/2821881121 wait complete. 2024-04-25T15:58:28.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:28 smithi031 ceph-mon[29140]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:28.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:28 smithi186 ceph-mon[33258]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:28.869 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 15:58:28 smithi063 podman[62009]: 2024-04-25 15:58:28.414725961 +0000 UTC m=+0.384239608 container start 184ad245f379892aa90c82db4d8f6ca103857c91743809d920d3f33f685790b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T15:58:28.869 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 15:58:28 smithi063 ceph-mds[62028]: -- 172.21.15.63:0/1350419780 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8be31ba40 con 0x55f8be363a80 2024-04-25T15:58:28.870 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 15:58:28 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k[62024]: starting mds.k at 2024-04-25T15:58:28.870 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 15:58:28 smithi063 bash[62009]: 184ad245f379892aa90c82db4d8f6ca103857c91743809d920d3f33f685790b7 2024-04-25T15:58:28.870 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 15:58:28 smithi063 systemd[1]: Started Ceph mds.k for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:29.846 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:29.846 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:29.846 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2821881121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:29.846 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:29.846 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:29.846 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:29.846 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:29.847 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: Deploying daemon mds.j on smithi031 2024-04-25T15:58:29.847 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6832/927313408,v1:172.21.15.63:6833/927313408] up:boot 2024-04-25T15:58:29.847 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: fsmap 2 up:standby 2024-04-25T15:58:29.847 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:29 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T15:58:29.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:29.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:29.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2821881121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:29.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:29.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:29.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:29.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:29.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: Deploying daemon mds.j on smithi031 2024-04-25T15:58:29.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6832/927313408,v1:172.21.15.63:6833/927313408] up:boot 2024-04-25T15:58:29.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: fsmap 2 up:standby 2024-04-25T15:58:29.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:29 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T15:58:29.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2821881121' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: Deploying daemon mds.j on smithi031 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6832/927313408,v1:172.21.15.63:6833/927313408] up:boot 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: fsmap 2 up:standby 2024-04-25T15:58:29.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:29 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T15:58:30.605 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:30.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.791+0000 7f687a957640 1 -- 172.21.15.31:0/2505353799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68741029c0 msgr2=0x7f68741036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:30.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.791+0000 7f687a957640 1 --2- 172.21.15.31:0/2505353799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68741029c0 0x7f68741036c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6864009900 tx=0x7f686402a410 comp rx=0 tx=0).stop 2024-04-25T15:58:30.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.791+0000 7f687a957640 1 -- 172.21.15.31:0/2505353799 shutdown_connections 2024-04-25T15:58:30.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.791+0000 7f687a957640 1 --2- 172.21.15.31:0/2505353799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6874108900 0x7f687410ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:30.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.791+0000 7f687a957640 1 --2- 172.21.15.31:0/2505353799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6874103c00 0x7f6874104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:30.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.791+0000 7f687a957640 1 --2- 172.21.15.31:0/2505353799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f68741029c0 0x7f68741036c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:30.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.791+0000 7f687a957640 1 -- 172.21.15.31:0/2505353799 >> 172.21.15.31:0/2505353799 conn(0x7f68740fdbd0 msgr2=0x7f6874100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:30.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.791+0000 7f687a957640 1 -- 172.21.15.31:0/2505353799 shutdown_connections 2024-04-25T15:58:30.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.791+0000 7f687a957640 1 -- 172.21.15.31:0/2505353799 wait complete. 2024-04-25T15:58:30.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f687a957640 1 Processor -- start 2024-04-25T15:58:30.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f687a957640 1 -- start start 2024-04-25T15:58:30.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f687a957640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68741029c0 0x7f687406a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:30.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f687a957640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874103c00 0x7f68741a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:30.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f687a957640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6874108900 0x7f68741a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:30.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f687a957640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f687410e4f0 con 0x7f6874103c00 2024-04-25T15:58:30.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f687a957640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f687410e370 con 0x7f6874108900 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f687a957640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f687410e670 con 0x7f68741029c0 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f68737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874103c00 0x7f68741a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f68737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874103c00 0x7f68741a13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33642/0 (socket says 172.21.15.31:33642) 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f68737fe640 1 -- 172.21.15.31:0/4283660529 learned_addr learned my addr 172.21.15.31:0/4283660529 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.792+0000 7f6878ef4640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6874108900 0x7f68741a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f6873fff640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68741029c0 0x7f687406a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f68737fe640 1 -- 172.21.15.31:0/4283660529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68741029c0 msgr2=0x7f687406a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f68737fe640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68741029c0 0x7f687406a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f68737fe640 1 -- 172.21.15.31:0/4283660529 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6874108900 msgr2=0x7f68741a3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f68737fe640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6874108900 0x7f68741a3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f68737fe640 1 -- 172.21.15.31:0/4283660529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68741a3f50 con 0x7f6874103c00 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f6873fff640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68741029c0 0x7f687406a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f68737fe640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874103c00 0x7f68741a13c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f686800d950 tx=0x7f686800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f68717fa640 1 -- 172.21.15.31:0/4283660529 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6868017030 con 0x7f6874103c00 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68741a4240 con 0x7f6874103c00 2024-04-25T15:58:30.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f68717fa640 1 -- 172.21.15.31:0/4283660529 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6868004020 con 0x7f6874103c00 2024-04-25T15:58:30.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f68717fa640 1 -- 172.21.15.31:0/4283660529 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68680047a0 con 0x7f6874103c00 2024-04-25T15:58:30.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.793+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68741a47e0 con 0x7f6874103c00 2024-04-25T15:58:30.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.794+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6874107710 con 0x7f6874103c00 2024-04-25T15:58:30.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.796+0000 7f68717fa640 1 -- 172.21.15.31:0/4283660529 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6868009a30 con 0x7f6874103c00 2024-04-25T15:58:30.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.797+0000 7f68717fa640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f685c0afaf0 0x7f685c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:30.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.797+0000 7f6873fff640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f685c0afaf0 0x7f685c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:30.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.797+0000 7f68717fa640 1 -- 172.21.15.31:0/4283660529 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f68680de950 con 0x7f6874103c00 2024-04-25T15:58:30.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.797+0000 7f6873fff640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f685c0afaf0 0x7f685c0b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f686402aa00 tx=0x7f6864031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:30.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:30.805+0000 7f68717fa640 1 -- 172.21.15.31:0/4283660529 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f68680a7930 con 0x7f6874103c00 2024-04-25T15:58:30.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:30 smithi063 ceph-mon[32413]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:30.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:30 smithi186 ceph-mon[33258]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:30.915 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 15:58:30 smithi031 systemd[1]: Starting Ceph mds.j for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:58:30.915 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:30 smithi031 ceph-mon[29140]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.043+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f687402db50 con 0x7f6874103c00 2024-04-25T15:58:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.044+0000 7f68717fa640 1 -- 172.21.15.31:0/4283660529 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f68680a7230 con 0x7f6874103c00 2024-04-25T15:58:31.045 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:31.045 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T15:58:31.045 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 3 2024-04-25T15:58:31.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.046+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f685c0afaf0 msgr2=0x7f685c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f685c0afaf0 0x7f685c0b2000 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f686402aa00 tx=0x7f6864031400 comp rx=0 tx=0).stop 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874103c00 msgr2=0x7f68741a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874103c00 0x7f68741a13c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f686800d950 tx=0x7f686800de10 comp rx=0 tx=0).stop 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 shutdown_connections 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f685c0afaf0 0x7f685c0b2000 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6874108900 0x7f68741a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6874103c00 0x7f68741a13c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 --2- 172.21.15.31:0/4283660529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68741029c0 0x7f687406a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 >> 172.21.15.31:0/4283660529 conn(0x7f68740fdbd0 msgr2=0x7f68741048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 shutdown_connections 2024-04-25T15:58:31.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:31.047+0000 7f687a957640 1 -- 172.21.15.31:0/4283660529 wait complete. 2024-04-25T15:58:31.182 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 15:58:31 smithi031 podman[69780]: 2024-04-25 15:58:31.044126237 +0000 UTC m=+0.018743780 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:31.182 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 15:58:31 smithi031 podman[69780]: 2024-04-25 15:58:31.182231797 +0000 UTC m=+0.156849326 container create 43107973a76c1b3991bc7af971f9d9fd2c5120391a98a0f7d04ca07c493eb629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0) 2024-04-25T15:58:31.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:31 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4283660529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:31.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:31 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:31.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:31 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:31.700 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 15:58:31 smithi031 podman[69780]: 2024-04-25 15:58:31.443359806 +0000 UTC m=+0.417977336 container init 43107973a76c1b3991bc7af971f9d9fd2c5120391a98a0f7d04ca07c493eb629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424) 2024-04-25T15:58:31.700 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 15:58:31 smithi031 podman[69780]: 2024-04-25 15:58:31.446281645 +0000 UTC m=+0.420899175 container start 43107973a76c1b3991bc7af971f9d9fd2c5120391a98a0f7d04ca07c493eb629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T15:58:31.700 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 15:58:31 smithi031 ceph-mds[69801]: -- 172.21.15.31:0/1967122130 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557871421a40 con 0x557871469600 2024-04-25T15:58:31.700 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 15:58:31 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j[69797]: starting mds.j at 2024-04-25T15:58:31.700 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 15:58:31 smithi031 bash[69780]: 43107973a76c1b3991bc7af971f9d9fd2c5120391a98a0f7d04ca07c493eb629 2024-04-25T15:58:31.700 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 15:58:31 smithi031 systemd[1]: Started Ceph mds.j for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:31.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:31 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4283660529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:31.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:31 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:31.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:31 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:32.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:31 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4283660529' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:32.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:31 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:32.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:31 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:32.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:32 smithi186 ceph-mon[33258]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:32.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:32 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.31:6834/3224554516,v1:172.21.15.31:6835/3224554516] up:boot 2024-04-25T15:58:32.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:32 smithi186 ceph-mon[33258]: fsmap 3 up:standby 2024-04-25T15:58:32.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:32 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T15:58:32.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:32 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:32.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:32 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:32.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:32 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:32.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:32 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:32.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:32 smithi186 ceph-mon[33258]: Deploying daemon mds.d on smithi031 2024-04-25T15:58:32.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:32 smithi031 ceph-mon[29140]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:32.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:32 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.31:6834/3224554516,v1:172.21.15.31:6835/3224554516] up:boot 2024-04-25T15:58:32.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:32 smithi031 ceph-mon[29140]: fsmap 3 up:standby 2024-04-25T15:58:32.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:32 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T15:58:32.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:32 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:32.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:32 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:32.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:32 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:32.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:32 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:32.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:32 smithi031 ceph-mon[29140]: Deploying daemon mds.d on smithi031 2024-04-25T15:58:33.063 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:33.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:32 smithi063 ceph-mon[32413]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:33.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:32 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.31:6834/3224554516,v1:172.21.15.31:6835/3224554516] up:boot 2024-04-25T15:58:33.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:32 smithi063 ceph-mon[32413]: fsmap 3 up:standby 2024-04-25T15:58:33.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:32 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-25T15:58:33.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:32 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:33.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:32 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:33.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:32 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:33.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:32 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:33.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:32 smithi063 ceph-mon[32413]: Deploying daemon mds.d on smithi031 2024-04-25T15:58:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 -- 172.21.15.31:0/896050562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 msgr2=0x7f28d4106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 --2- 172.21.15.31:0/896050562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 0x7f28d4106610 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f28c4009900 tx=0x7f28c402a410 comp rx=0 tx=0).stop 2024-04-25T15:58:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 -- 172.21.15.31:0/896050562 shutdown_connections 2024-04-25T15:58:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 --2- 172.21.15.31:0/896050562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d4109750 0x7f28d410bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 --2- 172.21.15.31:0/896050562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4106be0 0x7f28d41090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 --2- 172.21.15.31:0/896050562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 0x7f28d4106610 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 -- 172.21.15.31:0/896050562 >> 172.21.15.31:0/896050562 conn(0x7f28d40fdbd0 msgr2=0x7f28d4100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:33.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 -- 172.21.15.31:0/896050562 shutdown_connections 2024-04-25T15:58:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 -- 172.21.15.31:0/896050562 wait complete. 2024-04-25T15:58:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 Processor -- start 2024-04-25T15:58:33.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.188+0000 7f28da36c640 1 -- start start 2024-04-25T15:58:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28da36c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 0x7f28d4103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28da36c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d4106be0 0x7f28d4102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28da36c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4109750 0x7f28d4102640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28da36c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28d410f5b0 con 0x7f28d41041f0 2024-04-25T15:58:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28da36c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f28d410f430 con 0x7f28d4109750 2024-04-25T15:58:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28da36c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28d410f730 con 0x7f28d4106be0 2024-04-25T15:58:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 0x7f28d4103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 0x7f28d4103a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33670/0 (socket says 172.21.15.31:33670) 2024-04-25T15:58:33.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d3fff640 1 -- 172.21.15.31:0/100926107 learned_addr learned my addr 172.21.15.31:0/100926107 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d8909640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4109750 0x7f28d4102640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d37fe640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d4106be0 0x7f28d4102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d3fff640 1 -- 172.21.15.31:0/100926107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d4106be0 msgr2=0x7f28d4102100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d3fff640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d4106be0 0x7f28d4102100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d3fff640 1 -- 172.21.15.31:0/100926107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4109750 msgr2=0x7f28d4102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d3fff640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4109750 0x7f28d4102640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d3fff640 1 -- 172.21.15.31:0/100926107 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d4102d70 con 0x7f28d41041f0 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d8909640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4109750 0x7f28d4102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.189+0000 7f28d37fe640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d4106be0 0x7f28d4102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.190+0000 7f28d3fff640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 0x7f28d4103a60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f28c4006010 tx=0x7f28c4031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.190+0000 7f28d17fa640 1 -- 172.21.15.31:0/100926107 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c403f030 con 0x7f28d41041f0 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.190+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d41a42b0 con 0x7f28d41041f0 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.190+0000 7f28d17fa640 1 -- 172.21.15.31:0/100926107 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28c4003aa0 con 0x7f28d41041f0 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.190+0000 7f28d17fa640 1 -- 172.21.15.31:0/100926107 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4035070 con 0x7f28d41041f0 2024-04-25T15:58:33.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.190+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d41a46b0 con 0x7f28d41041f0 2024-04-25T15:58:33.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.191+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28d406af80 con 0x7f28d41041f0 2024-04-25T15:58:33.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.192+0000 7f28d17fa640 1 -- 172.21.15.31:0/100926107 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f28c4042030 con 0x7f28d41041f0 2024-04-25T15:58:33.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.193+0000 7f28d17fa640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28bc0afaf0 0x7f28bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:33.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.193+0000 7f28d37fe640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28bc0afaf0 0x7f28bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:33.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.193+0000 7f28d17fa640 1 -- 172.21.15.31:0/100926107 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f28c40fee90 con 0x7f28d41041f0 2024-04-25T15:58:33.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.194+0000 7f28d37fe640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28bc0afaf0 0x7f28bc0b2000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28d4103400 tx=0x7f28c8005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:33.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.198+0000 7f28d17fa640 1 -- 172.21.15.31:0/100926107 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28c40c7e70 con 0x7f28d41041f0 2024-04-25T15:58:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.426+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f28d40ff6f0 con 0x7f28d41041f0 2024-04-25T15:58:33.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.426+0000 7f28d17fa640 1 -- 172.21.15.31:0/100926107 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f28c40fec00 con 0x7f28d41041f0 2024-04-25T15:58:33.427 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:33.428 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T15:58:33.428 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 4 2024-04-25T15:58:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28bc0afaf0 msgr2=0x7f28bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28bc0afaf0 0x7f28bc0b2000 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f28d4103400 tx=0x7f28c8005e70 comp rx=0 tx=0).stop 2024-04-25T15:58:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 msgr2=0x7f28d4103a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:33.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 0x7f28d4103a60 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f28c4006010 tx=0x7f28c4031a50 comp rx=0 tx=0).stop 2024-04-25T15:58:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 shutdown_connections 2024-04-25T15:58:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28bc0afaf0 0x7f28bc0b2000 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28d4109750 0x7f28d4102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28d4106be0 0x7f28d4102100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 --2- 172.21.15.31:0/100926107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28d41041f0 0x7f28d4103a60 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.428+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 >> 172.21.15.31:0/100926107 conn(0x7f28d40fdbd0 msgr2=0x7f28d4104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.429+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 shutdown_connections 2024-04-25T15:58:33.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:33.429+0000 7f28da36c640 1 -- 172.21.15.31:0/100926107 wait complete. 2024-04-25T15:58:33.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:33 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/100926107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:33.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:33 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/100926107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:33.950 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 15:58:33 smithi031 systemd[1]: Starting Ceph mds.d for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:58:34.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:33 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/100926107' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:34.322 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 15:58:34 smithi031 podman[70130]: 2024-04-25 15:58:33.951821879 +0000 UTC m=+0.018328461 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:34.323 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 15:58:34 smithi031 podman[70130]: 2024-04-25 15:58:34.077799699 +0000 UTC m=+0.144306324 container create 625af22b8d0cedd6dbeaee5d46a6a77056058a46f7f0a3a68ce637a9196e76bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-d, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T15:58:34.323 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 15:58:34 smithi031 podman[70130]: 2024-04-25 15:58:34.322153433 +0000 UTC m=+0.388659996 container init 625af22b8d0cedd6dbeaee5d46a6a77056058a46f7f0a3a68ce637a9196e76bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-d, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-25T15:58:34.633 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 15:58:34 smithi031 podman[70130]: 2024-04-25 15:58:34.325093602 +0000 UTC m=+0.391600178 container start 625af22b8d0cedd6dbeaee5d46a6a77056058a46f7f0a3a68ce637a9196e76bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_CLEAN=True) 2024-04-25T15:58:34.633 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 15:58:34 smithi031 ceph-mds[70150]: -- 172.21.15.31:0/3097571768 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562755083a40 con 0x5627550cb180 2024-04-25T15:58:34.633 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 15:58:34 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-d[70146]: starting mds.d at 2024-04-25T15:58:34.633 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 15:58:34 smithi031 bash[70130]: 625af22b8d0cedd6dbeaee5d46a6a77056058a46f7f0a3a68ce637a9196e76bb 2024-04-25T15:58:34.633 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 15:58:34 smithi031 systemd[1]: Started Ceph mds.d for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:34.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:34 smithi186 ceph-mon[33258]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:34.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:34.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:34.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:34.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:34.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:34.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:34 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:34.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:34 smithi031 ceph-mon[29140]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:34.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:34.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:34.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:34.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:34.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:34.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:34 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:35.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:34 smithi063 ceph-mon[32413]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:35.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:35.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:35.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:35.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:35.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:35.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:34 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:35.441 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:35.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.570+0000 7f5018f6b640 1 -- 172.21.15.31:0/1517784289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014109750 msgr2=0x7f501410bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.570+0000 7f5018f6b640 1 --2- 172.21.15.31:0/1517784289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014109750 0x7f501410bc60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f500c009900 tx=0x7f500c02a410 comp rx=0 tx=0).stop 2024-04-25T15:58:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.571+0000 7f5018f6b640 1 -- 172.21.15.31:0/1517784289 shutdown_connections 2024-04-25T15:58:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.571+0000 7f5018f6b640 1 --2- 172.21.15.31:0/1517784289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014109750 0x7f501410bc60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.571+0000 7f5018f6b640 1 --2- 172.21.15.31:0/1517784289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5014106be0 0x7f50141090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.571+0000 7f5018f6b640 1 --2- 172.21.15.31:0/1517784289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50141041f0 0x7f5014106610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.571+0000 7f5018f6b640 1 -- 172.21.15.31:0/1517784289 >> 172.21.15.31:0/1517784289 conn(0x7f50140fdbd0 msgr2=0x7f5014100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.571+0000 7f5018f6b640 1 -- 172.21.15.31:0/1517784289 shutdown_connections 2024-04-25T15:58:35.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.571+0000 7f5018f6b640 1 -- 172.21.15.31:0/1517784289 wait complete. 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.571+0000 7f5018f6b640 1 Processor -- start 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5018f6b640 1 -- start start 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5018f6b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50141041f0 0x7f5014103a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5018f6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014106be0 0x7f5014102100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5018f6b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5014109750 0x7f5014102640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5018f6b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f501410f5a0 con 0x7f5014106be0 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5018f6b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f501410f420 con 0x7f5014109750 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5018f6b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f501410f720 con 0x7f50141041f0 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5011d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014106be0 0x7f5014102100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5011d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014106be0 0x7f5014102100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56814/0 (socket says 172.21.15.31:56814) 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5011d9b640 1 -- 172.21.15.31:0/2483245303 learned_addr learned my addr 172.21.15.31:0/2483245303 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5012d9d640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5014109750 0x7f5014102640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f501259c640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50141041f0 0x7f5014103a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:35.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5011d9b640 1 -- 172.21.15.31:0/2483245303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50141041f0 msgr2=0x7f5014103a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5011d9b640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50141041f0 0x7f5014103a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5011d9b640 1 -- 172.21.15.31:0/2483245303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5014109750 msgr2=0x7f5014102640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5011d9b640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5014109750 0x7f5014102640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5011d9b640 1 -- 172.21.15.31:0/2483245303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5014102d70 con 0x7f5014106be0 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5012d9d640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5014109750 0x7f5014102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.572+0000 7f5011d9b640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014106be0 0x7f5014102100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f500800e9f0 tx=0x7f500800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.573+0000 7f50037fe640 1 -- 172.21.15.31:0/2483245303 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500800cd80 con 0x7f5014106be0 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.573+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50141a4210 con 0x7f5014106be0 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.573+0000 7f50037fe640 1 -- 172.21.15.31:0/2483245303 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50080108e0 con 0x7f5014106be0 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.573+0000 7f50037fe640 1 -- 172.21.15.31:0/2483245303 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5008018980 con 0x7f5014106be0 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.573+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50141a4700 con 0x7f5014106be0 2024-04-25T15:58:35.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.574+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50140ff6f0 con 0x7f5014106be0 2024-04-25T15:58:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.575+0000 7f50037fe640 1 -- 172.21.15.31:0/2483245303 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5008010430 con 0x7f5014106be0 2024-04-25T15:58:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.576+0000 7f50037fe640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4ffc0afaf0 0x7f4ffc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.576+0000 7f50037fe640 1 -- 172.21.15.31:0/2483245303 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f50080defa0 con 0x7f5014106be0 2024-04-25T15:58:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.576+0000 7f501259c640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4ffc0afaf0 0x7f4ffc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.576+0000 7f501259c640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4ffc0afaf0 0x7f4ffc0b2000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f501406a850 tx=0x7f50040098f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:35.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.580+0000 7f50037fe640 1 -- 172.21.15.31:0/2483245303 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f50080a8020 con 0x7f5014106be0 2024-04-25T15:58:35.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.807+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f501402db50 con 0x7f5014106be0 2024-04-25T15:58:35.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.808+0000 7f50037fe640 1 -- 172.21.15.31:0/2483245303 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3865 (secure 0 0 0) 0x7f50080b00a0 con 0x7f5014106be0 2024-04-25T15:58:35.808 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:35.808 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-25T15:58:35.809 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 5 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4ffc0afaf0 msgr2=0x7f4ffc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4ffc0afaf0 0x7f4ffc0b2000 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f501406a850 tx=0x7f50040098f0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014106be0 msgr2=0x7f5014102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014106be0 0x7f5014102100 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f500800e9f0 tx=0x7f500800eeb0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 shutdown_connections 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4ffc0afaf0 0x7f4ffc0b2000 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5014109750 0x7f5014102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5014106be0 0x7f5014102100 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 --2- 172.21.15.31:0/2483245303 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50141041f0 0x7f5014103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:35.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 >> 172.21.15.31:0/2483245303 conn(0x7f50140fdbd0 msgr2=0x7f5014104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:35.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 shutdown_connections 2024-04-25T15:58:35.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:35.810+0000 7f5018f6b640 1 -- 172.21.15.31:0/2483245303 wait complete. 2024-04-25T15:58:35.821 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:35 smithi031 ceph-mon[29140]: Deploying daemon mds.e on smithi063 2024-04-25T15:58:35.821 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:35 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.31:6836/533105912,v1:172.21.15.31:6837/533105912] up:boot 2024-04-25T15:58:35.821 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:35 smithi031 ceph-mon[29140]: fsmap 4 up:standby 2024-04-25T15:58:35.822 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:35 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T15:58:35.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:35 smithi063 ceph-mon[32413]: Deploying daemon mds.e on smithi063 2024-04-25T15:58:35.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:35 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.31:6836/533105912,v1:172.21.15.31:6837/533105912] up:boot 2024-04-25T15:58:35.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:35 smithi063 ceph-mon[32413]: fsmap 4 up:standby 2024-04-25T15:58:35.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:35 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T15:58:35.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:35 smithi186 ceph-mon[33258]: Deploying daemon mds.e on smithi063 2024-04-25T15:58:35.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:35 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.31:6836/533105912,v1:172.21.15.31:6837/533105912] up:boot 2024-04-25T15:58:35.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:35 smithi186 ceph-mon[33258]: fsmap 4 up:standby 2024-04-25T15:58:35.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:35 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-25T15:58:36.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:36 smithi186 ceph-mon[33258]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:36.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:36 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2483245303' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:36.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:36 smithi031 ceph-mon[29140]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:36.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:36 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2483245303' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:37.081 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:36 smithi063 ceph-mon[32413]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:37.081 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:36 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2483245303' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:37.081 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 15:58:36 smithi063 systemd[1]: Starting Ceph mds.e for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:58:37.357 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 15:58:37 smithi063 podman[62328]: 2024-04-25 15:58:36.984272315 +0000 UTC m=+0.018790968 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:37.357 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 15:58:37 smithi063 podman[62328]: 2024-04-25 15:58:37.113345017 +0000 UTC m=+0.147863653 container create 5d6e749ec588691d3c8cc5e75885914b8fea85c3b3bbc7cdb27d65db30a96479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T15:58:37.619 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 15:58:37 smithi063 podman[62328]: 2024-04-25 15:58:37.357515057 +0000 UTC m=+0.392033696 container init 5d6e749ec588691d3c8cc5e75885914b8fea85c3b3bbc7cdb27d65db30a96479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-25T15:58:37.619 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 15:58:37 smithi063 podman[62328]: 2024-04-25 15:58:37.360304371 +0000 UTC m=+0.394823008 container start 5d6e749ec588691d3c8cc5e75885914b8fea85c3b3bbc7cdb27d65db30a96479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240424, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-25T15:58:37.620 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 15:58:37 smithi063 ceph-mds[62347]: -- 172.21.15.63:0/3612028857 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563f528f9a40 con 0x563f52941600 2024-04-25T15:58:37.620 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 15:58:37 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e[62343]: starting mds.e at 2024-04-25T15:58:37.620 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 15:58:37 smithi063 bash[62328]: 5d6e749ec588691d3c8cc5e75885914b8fea85c3b3bbc7cdb27d65db30a96479 2024-04-25T15:58:37.620 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 15:58:37 smithi063 systemd[1]: Started Ceph mds.e for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:37.824 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:37.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.950+0000 7fdca3c61640 1 -- 172.21.15.31:0/109395075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c103c40 msgr2=0x7fdc9c104050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:37.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.950+0000 7fdca3c61640 1 --2- 172.21.15.31:0/109395075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c103c40 0x7fdc9c104050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c009900 tx=0x7fdc8c02a410 comp rx=0 tx=0).stop 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.951+0000 7fdca3c61640 1 -- 172.21.15.31:0/109395075 shutdown_connections 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.951+0000 7fdca3c61640 1 --2- 172.21.15.31:0/109395075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc9c108940 0x7fdc9c10ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.951+0000 7fdca3c61640 1 --2- 172.21.15.31:0/109395075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c103c40 0x7fdc9c104050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.951+0000 7fdca3c61640 1 --2- 172.21.15.31:0/109395075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc9c102a00 0x7fdc9c103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.951+0000 7fdca3c61640 1 -- 172.21.15.31:0/109395075 >> 172.21.15.31:0/109395075 conn(0x7fdc9c0fdbf0 msgr2=0x7fdc9c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.951+0000 7fdca3c61640 1 -- 172.21.15.31:0/109395075 shutdown_connections 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.951+0000 7fdca3c61640 1 -- 172.21.15.31:0/109395075 wait complete. 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.951+0000 7fdca3c61640 1 Processor -- start 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.951+0000 7fdca3c61640 1 -- start start 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca3c61640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c102a00 0x7fdc9c0ffa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca3c61640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc9c103c40 0x7fdc9c0fe120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:37.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca3c61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc9c108940 0x7fdc9c0fe660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:37.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca3c61640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c10e530 con 0x7fdc9c102a00 2024-04-25T15:58:37.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca3c61640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c10e3b0 con 0x7fdc9c103c40 2024-04-25T15:58:37.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca3c61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9c10e6b0 con 0x7fdc9c108940 2024-04-25T15:58:37.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca19fd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c102a00 0x7fdc9c0ffa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:37.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca19fd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c102a00 0x7fdc9c0ffa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56832/0 (socket says 172.21.15.31:56832) 2024-04-25T15:58:37.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca19fd640 1 -- 172.21.15.31:0/673129437 learned_addr learned my addr 172.21.15.31:0/673129437 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:37.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca21fe640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc9c108940 0x7fdc9c0fe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:37.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca11fc640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc9c103c40 0x7fdc9c0fe120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:37.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca19fd640 1 -- 172.21.15.31:0/673129437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc9c108940 msgr2=0x7fdc9c0fe660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:37.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca19fd640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc9c108940 0x7fdc9c0fe660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:37.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca19fd640 1 -- 172.21.15.31:0/673129437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc9c103c40 msgr2=0x7fdc9c0fe120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:37.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca19fd640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc9c103c40 0x7fdc9c0fe120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:37.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca19fd640 1 -- 172.21.15.31:0/673129437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc9c0fed90 con 0x7fdc9c102a00 2024-04-25T15:58:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca11fc640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc9c103c40 0x7fdc9c0fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca19fd640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c102a00 0x7fdc9c0ffa80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc98005fd0 tx=0x7fdc9800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdc92ffd640 1 -- 172.21.15.31:0/673129437 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc98017070 con 0x7fdc9c102a00 2024-04-25T15:58:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc9c1a4290 con 0x7fdc9c102a00 2024-04-25T15:58:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.952+0000 7fdc92ffd640 1 -- 172.21.15.31:0/673129437 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc98009c30 con 0x7fdc9c102a00 2024-04-25T15:58:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.953+0000 7fdc92ffd640 1 -- 172.21.15.31:0/673129437 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc98003d10 con 0x7fdc9c102a00 2024-04-25T15:58:37.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.953+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc9c1a4780 con 0x7fdc9c102a00 2024-04-25T15:58:37.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.954+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc9c1a49e0 con 0x7fdc9c102a00 2024-04-25T15:58:37.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.955+0000 7fdc92ffd640 1 -- 172.21.15.31:0/673129437 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fdc98003eb0 con 0x7fdc9c102a00 2024-04-25T15:58:37.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.956+0000 7fdc92ffd640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc880afaf0 0x7fdc880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:37.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.956+0000 7fdca11fc640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc880afaf0 0x7fdc880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:37.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.957+0000 7fdc92ffd640 1 -- 172.21.15.31:0/673129437 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fdc980dec10 con 0x7fdc9c102a00 2024-04-25T15:58:37.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.957+0000 7fdca11fc640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc880afaf0 0x7fdc880b2000 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c000c00 tx=0x7fdc8c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:37.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:37.960+0000 7fdc92ffd640 1 -- 172.21.15.31:0/673129437 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc98020030 con 0x7fdc9c102a00 2024-04-25T15:58:38.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.186+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc9c1a4bb0 con 0x7fdc9c102a00 2024-04-25T15:58:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.187+0000 7fdc92ffd640 1 -- 172.21.15.31:0/673129437 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7fdc98009de0 con 0x7fdc9c102a00 2024-04-25T15:58:38.188 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:38.188 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-25T15:58:38.188 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 6 2024-04-25T15:58:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc880afaf0 msgr2=0x7fdc880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc880afaf0 0x7fdc880b2000 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fdc8c000c00 tx=0x7fdc8c005d50 comp rx=0 tx=0).stop 2024-04-25T15:58:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c102a00 msgr2=0x7fdc9c0ffa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c102a00 0x7fdc9c0ffa80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdc98005fd0 tx=0x7fdc9800dde0 comp rx=0 tx=0).stop 2024-04-25T15:58:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 shutdown_connections 2024-04-25T15:58:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc880afaf0 0x7fdc880b2000 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc9c108940 0x7fdc9c0fe660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:38.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc9c103c40 0x7fdc9c0fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 --2- 172.21.15.31:0/673129437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc9c102a00 0x7fdc9c0ffa80 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 >> 172.21.15.31:0/673129437 conn(0x7fdc9c0fdbf0 msgr2=0x7fdc9c109db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 shutdown_connections 2024-04-25T15:58:38.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:38.189+0000 7fdca3c61640 1 -- 172.21.15.31:0/673129437 wait complete. 2024-04-25T15:58:38.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: Deploying daemon mds.f on smithi186 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6834/1584006856,v1:172.21.15.63:6835/1584006856] up:boot 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: fsmap 5 up:standby 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T15:58:38.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:38 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/673129437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:38.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:38.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:38.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:38.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:38.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:38.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:38.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:38.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: Deploying daemon mds.f on smithi186 2024-04-25T15:58:38.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6834/1584006856,v1:172.21.15.63:6835/1584006856] up:boot 2024-04-25T15:58:38.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: fsmap 5 up:standby 2024-04-25T15:58:38.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T15:58:38.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:38 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/673129437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:38.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: Deploying daemon mds.f on smithi186 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6834/1584006856,v1:172.21.15.63:6835/1584006856] up:boot 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: fsmap 5 up:standby 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T15:58:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:38 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/673129437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:39.882 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 15:58:39 smithi186 systemd[1]: Starting Ceph mds.f for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:58:40.137 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 15:58:39 smithi186 podman[63886]: 2024-04-25 15:58:39.784410739 +0000 UTC m=+0.017890973 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:40.137 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 15:58:39 smithi186 podman[63886]: 2024-04-25 15:58:39.892425121 +0000 UTC m=+0.125905345 container create 2d7978c8c71f54cf6c87c51bbf100fcdd151df00c0a8f57cc70ad9fbeb8a774a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-25T15:58:40.137 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 15:58:40 smithi186 podman[63886]: 2024-04-25 15:58:40.13696493 +0000 UTC m=+0.370445154 container init 2d7978c8c71f54cf6c87c51bbf100fcdd151df00c0a8f57cc70ad9fbeb8a774a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, ceph=True) 2024-04-25T15:58:40.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:40.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.330+0000 7f91b6e84640 1 -- 172.21.15.31:0/168723010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0109a20 msgr2=0x7f91b010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:40.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.330+0000 7f91b6e84640 1 --2- 172.21.15.31:0/168723010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0109a20 0x7f91b010be40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f91a8009ca0 tx=0x7f91a802a8c0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.330+0000 7f91b6e84640 1 -- 172.21.15.31:0/168723010 shutdown_connections 2024-04-25T15:58:40.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.330+0000 7f91b6e84640 1 --2- 172.21.15.31:0/168723010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0109a20 0x7f91b010be40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.330+0000 7f91b6e84640 1 --2- 172.21.15.31:0/168723010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b0104d20 0x7f91b0105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.330+0000 7f91b6e84640 1 --2- 172.21.15.31:0/168723010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91b0103ae0 0x7f91b01047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.330+0000 7f91b6e84640 1 -- 172.21.15.31:0/168723010 >> 172.21.15.31:0/168723010 conn(0x7f91b00fecd0 msgr2=0x7f91b0101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.330+0000 7f91b6e84640 1 -- 172.21.15.31:0/168723010 shutdown_connections 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.330+0000 7f91b6e84640 1 -- 172.21.15.31:0/168723010 wait complete. 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b6e84640 1 Processor -- start 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b6e84640 1 -- start start 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b6e84640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b0103ae0 0x7f91b0076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b6e84640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0104d20 0x7f91b0074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b6e84640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91b0109a20 0x7f91b0075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b6e84640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f91b010f610 con 0x7f91b0104d20 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b6e84640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f91b010f490 con 0x7f91b0109a20 2024-04-25T15:58:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b6e84640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f91b010f790 con 0x7f91b0103ae0 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0104d20 0x7f91b0074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0104d20 0x7f91b0074ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56856/0 (socket says 172.21.15.31:56856) 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91affff640 1 -- 172.21.15.31:0/3767311899 learned_addr learned my addr 172.21.15.31:0/3767311899 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b5421640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91b0109a20 0x7f91b0075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b4c20640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b0103ae0 0x7f91b0076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91affff640 1 -- 172.21.15.31:0/3767311899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b0103ae0 msgr2=0x7f91b0076850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91affff640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b0103ae0 0x7f91b0076850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91affff640 1 -- 172.21.15.31:0/3767311899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91b0109a20 msgr2=0x7f91b0075430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91affff640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91b0109a20 0x7f91b0075430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91affff640 1 -- 172.21.15.31:0/3767311899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91b0075b60 con 0x7f91b0104d20 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.331+0000 7f91b4c20640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b0103ae0 0x7f91b0076850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.332+0000 7f91affff640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0104d20 0x7f91b0074ef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f91a400e9f0 tx=0x7f91a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.332+0000 7f91adffb640 1 -- 172.21.15.31:0/3767311899 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a400cd80 con 0x7f91b0104d20 2024-04-25T15:58:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.332+0000 7f91adffb640 1 -- 172.21.15.31:0/3767311899 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91a40054d0 con 0x7f91b0104d20 2024-04-25T15:58:40.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.332+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91b01a4210 con 0x7f91b0104d20 2024-04-25T15:58:40.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.332+0000 7f91adffb640 1 -- 172.21.15.31:0/3767311899 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a4010690 con 0x7f91b0104d20 2024-04-25T15:58:40.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.332+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91b01a4680 con 0x7f91b0104d20 2024-04-25T15:58:40.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.333+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91b0108830 con 0x7f91b0104d20 2024-04-25T15:58:40.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.334+0000 7f91adffb640 1 -- 172.21.15.31:0/3767311899 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f91a4004e90 con 0x7f91b0104d20 2024-04-25T15:58:40.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.335+0000 7f91adffb640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f91980afaf0 0x7f91980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:40.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.335+0000 7f91b4c20640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f91980afaf0 0x7f91980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:40.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.335+0000 7f91adffb640 1 -- 172.21.15.31:0/3767311899 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f91a40dec40 con 0x7f91b0104d20 2024-04-25T15:58:40.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.336+0000 7f91b4c20640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f91980afaf0 0x7f91980b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f91b01046e0 tx=0x7f91a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:40.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.340+0000 7f91adffb640 1 -- 172.21.15.31:0/3767311899 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91a40a7cc0 con 0x7f91b0104d20 2024-04-25T15:58:40.398 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 15:58:40 smithi186 podman[63886]: 2024-04-25 15:58:40.139479012 +0000 UTC m=+0.372959236 container start 2d7978c8c71f54cf6c87c51bbf100fcdd151df00c0a8f57cc70ad9fbeb8a774a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-25T15:58:40.398 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 15:58:40 smithi186 ceph-mds[63905]: -- 172.21.15.186:0/4112255664 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d0c5da5a40 con 0x55d0c5deda80 2024-04-25T15:58:40.398 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 15:58:40 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f[63901]: starting mds.f at 2024-04-25T15:58:40.398 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 15:58:40 smithi186 bash[63886]: 2d7978c8c71f54cf6c87c51bbf100fcdd151df00c0a8f57cc70ad9fbeb8a774a 2024-04-25T15:58:40.399 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 15:58:40 smithi186 systemd[1]: Started Ceph mds.f for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:40.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.567+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f91b002db50 con 0x7f91b0104d20 2024-04-25T15:58:40.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.567+0000 7f91adffb640 1 -- 172.21.15.31:0/3767311899 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7f91a40a75c0 con 0x7f91b0104d20 2024-04-25T15:58:40.568 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:40.569 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T15:58:40.569 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 7 2024-04-25T15:58:40.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.569+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f91980afaf0 msgr2=0x7f91980b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:40.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.569+0000 7f91b6e84640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f91980afaf0 0x7f91980b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f91b01046e0 tx=0x7f91a0005f90 comp rx=0 tx=0).stop 2024-04-25T15:58:40.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.569+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0104d20 msgr2=0x7f91b0074ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:40.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.569+0000 7f91b6e84640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0104d20 0x7f91b0074ef0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f91a400e9f0 tx=0x7f91a400edf0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.570+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 shutdown_connections 2024-04-25T15:58:40.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.570+0000 7f91b6e84640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f91980afaf0 0x7f91980b2000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.570+0000 7f91b6e84640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f91b0109a20 0x7f91b0075430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.570+0000 7f91b6e84640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f91b0104d20 0x7f91b0074ef0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.570+0000 7f91b6e84640 1 --2- 172.21.15.31:0/3767311899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91b0103ae0 0x7f91b0076850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:40.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.570+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 >> 172.21.15.31:0/3767311899 conn(0x7f91b00fecd0 msgr2=0x7f91b0105a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:40.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.570+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 shutdown_connections 2024-04-25T15:58:40.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:40.570+0000 7f91b6e84640 1 -- 172.21.15.31:0/3767311899 wait complete. 2024-04-25T15:58:40.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:40 smithi063 ceph-mon[32413]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:40.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:40.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:40.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:40.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:40.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:40.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:40.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:40 smithi186 ceph-mon[33258]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:40.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:40.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:40.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:40.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:40.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:40.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:40.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:40 smithi031 ceph-mon[29140]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:40.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:40.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:40.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:40.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:40.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:40.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:41.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:41 smithi063 ceph-mon[32413]: Deploying daemon mds.c on smithi186 2024-04-25T15:58:41.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:41 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6834/1885074975,v1:172.21.15.186:6835/1885074975] up:boot 2024-04-25T15:58:41.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:41 smithi063 ceph-mon[32413]: fsmap 6 up:standby 2024-04-25T15:58:41.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:41 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T15:58:41.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:41 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3767311899' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:41.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:41 smithi186 ceph-mon[33258]: Deploying daemon mds.c on smithi186 2024-04-25T15:58:41.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:41 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6834/1885074975,v1:172.21.15.186:6835/1885074975] up:boot 2024-04-25T15:58:41.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:41 smithi186 ceph-mon[33258]: fsmap 6 up:standby 2024-04-25T15:58:41.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:41 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T15:58:41.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:41 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3767311899' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:41.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:41 smithi031 ceph-mon[29140]: Deploying daemon mds.c on smithi186 2024-04-25T15:58:41.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:41 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6834/1885074975,v1:172.21.15.186:6835/1885074975] up:boot 2024-04-25T15:58:41.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:41 smithi031 ceph-mon[29140]: fsmap 6 up:standby 2024-04-25T15:58:41.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:41 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T15:58:41.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:41 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3767311899' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:42.583 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:42.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.709+0000 7f26886bf640 1 -- 172.21.15.31:0/1490364671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680108900 msgr2=0x7f268010ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:42.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.709+0000 7f26886bf640 1 --2- 172.21.15.31:0/1490364671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680108900 0x7f268010ad20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2678009ca0 tx=0x7f267802a8c0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.709+0000 7f26886bf640 1 -- 172.21.15.31:0/1490364671 shutdown_connections 2024-04-25T15:58:42.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.709+0000 7f26886bf640 1 --2- 172.21.15.31:0/1490364671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2680108900 0x7f268010ad20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.709+0000 7f26886bf640 1 --2- 172.21.15.31:0/1490364671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2680103c00 0x7f2680104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.709+0000 7f26886bf640 1 --2- 172.21.15.31:0/1490364671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26801029c0 0x7f26801036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.709+0000 7f26886bf640 1 -- 172.21.15.31:0/1490364671 >> 172.21.15.31:0/1490364671 conn(0x7f26800fdbd0 msgr2=0x7f2680100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.709+0000 7f26886bf640 1 -- 172.21.15.31:0/1490364671 shutdown_connections 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.709+0000 7f26886bf640 1 -- 172.21.15.31:0/1490364671 wait complete. 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f26886bf640 1 Processor -- start 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f26886bf640 1 -- start start 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f26886bf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26801029c0 0x7f268006a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f26886bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2680103c00 0x7f268019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f26886bf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2680108900 0x7f26801a14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f26886bf640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f268010e4d0 con 0x7f26801029c0 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f26886bf640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f268010e350 con 0x7f2680108900 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f26886bf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f268010e650 con 0x7f2680103c00 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f268645b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26801029c0 0x7f268006a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:42.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f268645b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26801029c0 0x7f268006a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56876/0 (socket says 172.21.15.31:56876) 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f268645b640 1 -- 172.21.15.31:0/4097053162 learned_addr learned my addr 172.21.15.31:0/4097053162 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.710+0000 7f2685c5a640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2680103c00 0x7f268019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f2686c5c640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2680108900 0x7f26801a14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f268645b640 1 -- 172.21.15.31:0/4097053162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2680103c00 msgr2=0x7f268019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f268645b640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2680103c00 0x7f268019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f268645b640 1 -- 172.21.15.31:0/4097053162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2680108900 msgr2=0x7f26801a14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f268645b640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2680108900 0x7f26801a14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f268645b640 1 -- 172.21.15.31:0/4097053162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26801a1cd0 con 0x7f26801029c0 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f2686c5c640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2680108900 0x7f26801a14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f268645b640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26801029c0 0x7f268006a960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e9f0 tx=0x7f267c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f26777fe640 1 -- 172.21.15.31:0/4097053162 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c00ccc0 con 0x7f26801029c0 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f26777fe640 1 -- 172.21.15.31:0/4097053162 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f267c0035f0 con 0x7f26801029c0 2024-04-25T15:58:42.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26801a1fc0 con 0x7f26801029c0 2024-04-25T15:58:42.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f26777fe640 1 -- 172.21.15.31:0/4097053162 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c00ce60 con 0x7f26801029c0 2024-04-25T15:58:42.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.711+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26801a2480 con 0x7f26801029c0 2024-04-25T15:58:42.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.712+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2680107710 con 0x7f26801029c0 2024-04-25T15:58:42.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.714+0000 7f26777fe640 1 -- 172.21.15.31:0/4097053162 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f267c003aa0 con 0x7f26801029c0 2024-04-25T15:58:42.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.714+0000 7f26777fe640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f266c0afb70 0x7f266c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:42.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.714+0000 7f2685c5a640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f266c0afb70 0x7f266c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:42.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.714+0000 7f26777fe640 1 -- 172.21.15.31:0/4097053162 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f267c0deb50 con 0x7f26801029c0 2024-04-25T15:58:42.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.715+0000 7f2685c5a640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f266c0afb70 0x7f266c0b2080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f268006a300 tx=0x7f2670005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:42.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.718+0000 7f26777fe640 1 -- 172.21.15.31:0/4097053162 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f267c0a7bd0 con 0x7f26801029c0 2024-04-25T15:58:42.842 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:42 smithi186 ceph-mon[33258]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:42.843 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 15:58:42 smithi186 systemd[1]: Starting Ceph mds.c for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:58:42.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:42 smithi063 ceph-mon[32413]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:42.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.945+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f268002dde0 con 0x7f26801029c0 2024-04-25T15:58:42.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.945+0000 7f26777fe640 1 -- 172.21.15.31:0/4097053162 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7f267c0a74d0 con 0x7f26801029c0 2024-04-25T15:58:42.946 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:42.947 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-25T15:58:42.947 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 7 2024-04-25T15:58:42.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.947+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f266c0afb70 msgr2=0x7f266c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:42.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.947+0000 7f26886bf640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f266c0afb70 0x7f266c0b2080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f268006a300 tx=0x7f2670005f90 comp rx=0 tx=0).stop 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.947+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26801029c0 msgr2=0x7f268006a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.947+0000 7f26886bf640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26801029c0 0x7f268006a960 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f267c00e9f0 tx=0x7f267c00eeb0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.947+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 shutdown_connections 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.947+0000 7f26886bf640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f266c0afb70 0x7f266c0b2080 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.947+0000 7f26886bf640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2680108900 0x7f26801a14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.948+0000 7f26886bf640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2680103c00 0x7f268019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.948+0000 7f26886bf640 1 --2- 172.21.15.31:0/4097053162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26801029c0 0x7f268006a960 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.948+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 >> 172.21.15.31:0/4097053162 conn(0x7f26800fdbd0 msgr2=0x7f26801048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.948+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 shutdown_connections 2024-04-25T15:58:42.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:42.948+0000 7f26886bf640 1 -- 172.21.15.31:0/4097053162 wait complete. 2024-04-25T15:58:42.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:42 smithi031 ceph-mon[29140]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:43.143 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 15:58:42 smithi186 podman[64196]: 2024-04-25 15:58:42.774745754 +0000 UTC m=+0.018236222 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:43.143 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 15:58:42 smithi186 podman[64196]: 2024-04-25 15:58:42.888370266 +0000 UTC m=+0.131860727 container create 0d441945cd3360d707634244da39c99453b4833afbce40849d48abfe41ef7844 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c, org.label-schema.build-date=20240424, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T15:58:43.143 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 15:58:43 smithi186 podman[64196]: 2024-04-25 15:58:43.142791869 +0000 UTC m=+0.386282330 container init 0d441945cd3360d707634244da39c99453b4833afbce40849d48abfe41ef7844 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T15:58:43.398 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 15:58:43 smithi186 podman[64196]: 2024-04-25 15:58:43.145575737 +0000 UTC m=+0.389066198 container start 0d441945cd3360d707634244da39c99453b4833afbce40849d48abfe41ef7844 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T15:58:43.398 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 15:58:43 smithi186 ceph-mds[64216]: -- 172.21.15.186:0/582110929 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c4157e5a40 con 0x55c41582da80 2024-04-25T15:58:43.399 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 15:58:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c[64212]: starting mds.c at 2024-04-25T15:58:43.399 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 15:58:43 smithi186 bash[64196]: 0d441945cd3360d707634244da39c99453b4833afbce40849d48abfe41ef7844 2024-04-25T15:58:43.399 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 15:58:43 smithi186 systemd[1]: Started Ceph mds.c for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:43.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:43 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4097053162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:43.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:43.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:43.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:43.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:43.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:43.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:43.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:43 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4097053162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:43.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:43.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:43.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:43.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:43.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:43.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:43.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:43 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4097053162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:43.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:43.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:43.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:43.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:43.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:43.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:44.851 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:44 smithi031 ceph-mon[29140]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:44.852 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:44 smithi031 ceph-mon[29140]: Deploying daemon mds.h on smithi063 2024-04-25T15:58:44.852 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:44 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] up:boot 2024-04-25T15:58:44.852 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:44 smithi031 ceph-mon[29140]: fsmap 7 up:standby 2024-04-25T15:58:44.852 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:44 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T15:58:44.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:44 smithi063 ceph-mon[32413]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:44.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:44 smithi063 ceph-mon[32413]: Deploying daemon mds.h on smithi063 2024-04-25T15:58:44.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:44 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] up:boot 2024-04-25T15:58:44.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:44 smithi063 ceph-mon[32413]: fsmap 7 up:standby 2024-04-25T15:58:44.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:44 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T15:58:44.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:44 smithi186 ceph-mon[33258]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:44.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:44 smithi186 ceph-mon[33258]: Deploying daemon mds.h on smithi063 2024-04-25T15:58:44.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:44 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] up:boot 2024-04-25T15:58:44.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:44 smithi186 ceph-mon[33258]: fsmap 7 up:standby 2024-04-25T15:58:44.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:44 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T15:58:44.961 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 -- 172.21.15.31:0/3096095794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef301029c0 msgr2=0x7fef301036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 --2- 172.21.15.31:0/3096095794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef301029c0 0x7fef301036c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fef18009900 tx=0x7fef1802a410 comp rx=0 tx=0).stop 2024-04-25T15:58:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 -- 172.21.15.31:0/3096095794 shutdown_connections 2024-04-25T15:58:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 --2- 172.21.15.31:0/3096095794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef30108900 0x7fef3010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 --2- 172.21.15.31:0/3096095794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef30103c00 0x7fef30104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 --2- 172.21.15.31:0/3096095794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef301029c0 0x7fef301036c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:45.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 -- 172.21.15.31:0/3096095794 >> 172.21.15.31:0/3096095794 conn(0x7fef300fdbd0 msgr2=0x7fef30100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 -- 172.21.15.31:0/3096095794 shutdown_connections 2024-04-25T15:58:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 -- 172.21.15.31:0/3096095794 wait complete. 2024-04-25T15:58:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.076+0000 7fef36642640 1 Processor -- start 2024-04-25T15:58:45.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef36642640 1 -- start start 2024-04-25T15:58:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef36642640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef301029c0 0x7fef3019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef36642640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef30103c00 0x7fef3019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef36642640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef30108900 0x7fef301a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef36642640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef3010e4c0 con 0x7fef30103c00 2024-04-25T15:58:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef36642640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef3010e340 con 0x7fef30108900 2024-04-25T15:58:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef36642640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef3010e640 con 0x7fef301029c0 2024-04-25T15:58:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef2f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef30103c00 0x7fef3019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:45.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef2f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef30103c00 0x7fef3019a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48370/0 (socket says 172.21.15.31:48370) 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef2f7fe640 1 -- 172.21.15.31:0/1514494074 learned_addr learned my addr 172.21.15.31:0/1514494074 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef34bdf640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef30108900 0x7fef301a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef2ffff640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef301029c0 0x7fef3019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef2f7fe640 1 -- 172.21.15.31:0/1514494074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef301029c0 msgr2=0x7fef3019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef2f7fe640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef301029c0 0x7fef3019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef2f7fe640 1 -- 172.21.15.31:0/1514494074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef30108900 msgr2=0x7fef301a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef2f7fe640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef30108900 0x7fef301a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.077+0000 7fef2f7fe640 1 -- 172.21.15.31:0/1514494074 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef301a1d80 con 0x7fef30103c00 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.078+0000 7fef2f7fe640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef30103c00 0x7fef3019a780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c970 tx=0x7fef2000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.078+0000 7fef2d7fa640 1 -- 172.21.15.31:0/1514494074 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef20007bb0 con 0x7fef30103c00 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.078+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef301a2070 con 0x7fef30103c00 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.078+0000 7fef2d7fa640 1 -- 172.21.15.31:0/1514494074 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef20007d50 con 0x7fef30103c00 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.078+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef301a25c0 con 0x7fef30103c00 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.078+0000 7fef2d7fa640 1 -- 172.21.15.31:0/1514494074 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef2000f450 con 0x7fef30103c00 2024-04-25T15:58:45.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.079+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef301a28e0 con 0x7fef30103c00 2024-04-25T15:58:45.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.080+0000 7fef2d7fa640 1 -- 172.21.15.31:0/1514494074 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fef20004f60 con 0x7fef30103c00 2024-04-25T15:58:45.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.081+0000 7fef2d7fa640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef1c0afaf0 0x7fef1c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:45.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.081+0000 7fef2ffff640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef1c0afaf0 0x7fef1c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:45.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.082+0000 7fef2d7fa640 1 -- 172.21.15.31:0/1514494074 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fef200de500 con 0x7fef30103c00 2024-04-25T15:58:45.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.082+0000 7fef2ffff640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef1c0afaf0 0x7fef1c0b2000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fef18005ab0 tx=0x7fef18031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:45.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.086+0000 7fef2d7fa640 1 -- 172.21.15.31:0/1514494074 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef20014030 con 0x7fef30103c00 2024-04-25T15:58:45.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.337+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef301a2ab0 con 0x7fef30103c00 2024-04-25T15:58:45.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.338+0000 7fef2d7fa640 1 -- 172.21.15.31:0/1514494074 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6343 (secure 0 0 0) 0x7fef200a6e90 con 0x7fef30103c00 2024-04-25T15:58:45.339 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:45.339 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T15:58:45.339 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 8 2024-04-25T15:58:45.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef1c0afaf0 msgr2=0x7fef1c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:45.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef1c0afaf0 0x7fef1c0b2000 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fef18005ab0 tx=0x7fef18031400 comp rx=0 tx=0).stop 2024-04-25T15:58:45.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef30103c00 msgr2=0x7fef3019a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:45.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef30103c00 0x7fef3019a780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fef2000c970 tx=0x7fef2000ce30 comp rx=0 tx=0).stop 2024-04-25T15:58:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 shutdown_connections 2024-04-25T15:58:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef1c0afaf0 0x7fef1c0b2000 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef30108900 0x7fef301a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef30103c00 0x7fef3019a780 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:45.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 --2- 172.21.15.31:0/1514494074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef301029c0 0x7fef3019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:45.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 >> 172.21.15.31:0/1514494074 conn(0x7fef300fdbd0 msgr2=0x7fef30069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:45.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 shutdown_connections 2024-04-25T15:58:45.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:45.340+0000 7fef36642640 1 -- 172.21.15.31:0/1514494074 wait complete. 2024-04-25T15:58:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:45 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1514494074' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:45.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:45 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1514494074' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:45.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:45 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1514494074' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:46.119 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 15:58:46 smithi063 podman[62636]: 2024-04-25 15:58:46.001755 +0000 UTC m=+0.134705305 container create 09bf47fc2733c8196aec808b0035b4001e13e8b2a721b9582c5251ea46dc5ec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T15:58:46.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 15:58:46 smithi063 podman[62636]: 2024-04-25 15:58:46.287989839 +0000 UTC m=+0.420940102 container init 09bf47fc2733c8196aec808b0035b4001e13e8b2a721b9582c5251ea46dc5ec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T15:58:46.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 15:58:46 smithi063 podman[62636]: 2024-04-25 15:58:46.290809755 +0000 UTC m=+0.423760017 container start 09bf47fc2733c8196aec808b0035b4001e13e8b2a721b9582c5251ea46dc5ec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T15:58:46.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 15:58:46 smithi063 ceph-mds[62655]: -- 172.21.15.63:0/1035723025 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559748eafa40 con 0x559748ef7a80 2024-04-25T15:58:46.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 15:58:46 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h[62651]: starting mds.h at 2024-04-25T15:58:46.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 15:58:46 smithi063 bash[62636]: 09bf47fc2733c8196aec808b0035b4001e13e8b2a721b9582c5251ea46dc5ec7 2024-04-25T15:58:46.549 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 15:58:46 smithi063 systemd[1]: Started Ceph mds.h for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:46 smithi063 ceph-mon[32413]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:46.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:46.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:46.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:46 smithi186 ceph-mon[33258]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:46.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:46.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:46.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:46.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:46.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:46.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:46.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:46 smithi031 ceph-mon[29140]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:46.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:46.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:46.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:46.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:46.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:46.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:47.354 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.482+0000 7f1b10a1a640 1 -- 172.21.15.31:0/1097445668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c109890 msgr2=0x7f1b0c10bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.482+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/1097445668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c109890 0x7f1b0c10bcb0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1b04009ca0 tx=0x7f1b0402a8c0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.482+0000 7f1b10a1a640 1 -- 172.21.15.31:0/1097445668 shutdown_connections 2024-04-25T15:58:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.482+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/1097445668 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c109890 0x7f1b0c10bcb0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.482+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/1097445668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c106f30 0x7f1b0c109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.482+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/1097445668 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b0c1045d0 0x7f1b0c1069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.482+0000 7f1b10a1a640 1 -- 172.21.15.31:0/1097445668 >> 172.21.15.31:0/1097445668 conn(0x7f1b0c0fdbd0 msgr2=0x7f1b0c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:47.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.482+0000 7f1b10a1a640 1 -- 172.21.15.31:0/1097445668 shutdown_connections 2024-04-25T15:58:47.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.482+0000 7f1b10a1a640 1 -- 172.21.15.31:0/1097445668 wait complete. 2024-04-25T15:58:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b10a1a640 1 Processor -- start 2024-04-25T15:58:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b10a1a640 1 -- start start 2024-04-25T15:58:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b10a1a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c1045d0 0x7f1b0c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b10a1a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c106f30 0x7f1b0c19a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b10a1a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b0c109890 0x7f1b0c1a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b10a1a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c10f660 con 0x7f1b0c1045d0 2024-04-25T15:58:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b10a1a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c10f4e0 con 0x7f1b0c109890 2024-04-25T15:58:47.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b10a1a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0c10f7e0 con 0x7f1b0c106f30 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b0a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c1045d0 0x7f1b0c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.483+0000 7f1b0a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c1045d0 0x7f1b0c19a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48376/0 (socket says 172.21.15.31:48376) 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b0a59c640 1 -- 172.21.15.31:0/2109277676 learned_addr learned my addr 172.21.15.31:0/2109277676 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b09d9b640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c106f30 0x7f1b0c19a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b0ad9d640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b0c109890 0x7f1b0c1a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b0a59c640 1 -- 172.21.15.31:0/2109277676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c106f30 msgr2=0x7f1b0c19a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b0a59c640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c106f30 0x7f1b0c19a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b0a59c640 1 -- 172.21.15.31:0/2109277676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b0c109890 msgr2=0x7f1b0c1a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b0a59c640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b0c109890 0x7f1b0c1a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b0a59c640 1 -- 172.21.15.31:0/2109277676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0c1a1d50 con 0x7f1b0c1045d0 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b0ad9d640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b0c109890 0x7f1b0c1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b0a59c640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c1045d0 0x7f1b0c19a210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00d770 tx=0x7f1afc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1afb7fe640 1 -- 172.21.15.31:0/2109277676 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc017070 con 0x7f1b0c1045d0 2024-04-25T15:58:47.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0c1a1fe0 con 0x7f1b0c1045d0 2024-04-25T15:58:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1afb7fe640 1 -- 172.21.15.31:0/2109277676 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1afc009c30 con 0x7f1b0c1045d0 2024-04-25T15:58:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1afb7fe640 1 -- 172.21.15.31:0/2109277676 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc00f620 con 0x7f1b0c1045d0 2024-04-25T15:58:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.484+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b0c1a24f0 con 0x7f1b0c1045d0 2024-04-25T15:58:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.486+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b0c0ff6f0 con 0x7f1b0c1045d0 2024-04-25T15:58:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.487+0000 7f1afb7fe640 1 -- 172.21.15.31:0/2109277676 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1afc01a070 con 0x7f1b0c1045d0 2024-04-25T15:58:47.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.488+0000 7f1afb7fe640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1af40afaf0 0x7f1af40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.488+0000 7f1b09d9b640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1af40afaf0 0x7f1af40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.488+0000 7f1afb7fe640 1 -- 172.21.15.31:0/2109277676 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f1afc0de320 con 0x7f1b0c1045d0 2024-04-25T15:58:47.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.488+0000 7f1b09d9b640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1af40afaf0 0x7f1af40b2000 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c19b2b0 tx=0x7f1b0000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:47.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.493+0000 7f1afb7fe640 1 -- 172.21.15.31:0/2109277676 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1afc009de0 con 0x7f1b0c1045d0 2024-04-25T15:58:47.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:47 smithi186 ceph-mon[33258]: Deploying daemon mds.i on smithi186 2024-04-25T15:58:47.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:47 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] up:boot 2024-04-25T15:58:47.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:47 smithi186 ceph-mon[33258]: fsmap 8 up:standby 2024-04-25T15:58:47.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T15:58:47.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:47 smithi031 ceph-mon[29140]: Deploying daemon mds.i on smithi186 2024-04-25T15:58:47.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:47 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] up:boot 2024-04-25T15:58:47.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:47 smithi031 ceph-mon[29140]: fsmap 8 up:standby 2024-04-25T15:58:47.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T15:58:47.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.724+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1b0c0637a0 con 0x7f1b0c1045d0 2024-04-25T15:58:47.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.724+0000 7f1afb7fe640 1 -- 172.21.15.31:0/2109277676 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7165 (secure 0 0 0) 0x7f1afc0a7300 con 0x7f1b0c1045d0 2024-04-25T15:58:47.725 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:47.725 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-25T15:58:47.726 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 9 2024-04-25T15:58:47.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.726+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1af40afaf0 msgr2=0x7f1af40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.726+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1af40afaf0 0x7f1af40b2000 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c19b2b0 tx=0x7f1b0000b390 comp rx=0 tx=0).stop 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.726+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c1045d0 msgr2=0x7f1b0c19a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.726+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c1045d0 0x7f1b0c19a210 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00d770 tx=0x7f1afc00dc30 comp rx=0 tx=0).stop 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.727+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 shutdown_connections 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.727+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1af40afaf0 0x7f1af40b2000 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.727+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b0c109890 0x7f1b0c1a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.727+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b0c106f30 0x7f1b0c19a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.727+0000 7f1b10a1a640 1 --2- 172.21.15.31:0/2109277676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b0c1045d0 0x7f1b0c19a210 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.727+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 >> 172.21.15.31:0/2109277676 conn(0x7f1b0c0fdbd0 msgr2=0x7f1b0c1051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.727+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 shutdown_connections 2024-04-25T15:58:47.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:47.727+0000 7f1b10a1a640 1 -- 172.21.15.31:0/2109277676 wait complete. 2024-04-25T15:58:47.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:47 smithi063 ceph-mon[32413]: Deploying daemon mds.i on smithi186 2024-04-25T15:58:47.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:47 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] up:boot 2024-04-25T15:58:47.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:47 smithi063 ceph-mon[32413]: fsmap 8 up:standby 2024-04-25T15:58:47.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T15:58:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:48 smithi186 ceph-mon[33258]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:48 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2109277676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:48.649 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 15:58:48 smithi186 systemd[1]: Starting Ceph mds.i for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:58:48.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:48 smithi063 ceph-mon[32413]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:48.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:48 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2109277676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:48.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:48 smithi031 ceph-mon[29140]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:48.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:48 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2109277676' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:49.060 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 15:58:48 smithi186 podman[64504]: 2024-04-25 15:58:48.703372645 +0000 UTC m=+0.017793284 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:49.061 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 15:58:48 smithi186 podman[64504]: 2024-04-25 15:58:48.816342246 +0000 UTC m=+0.130762872 container create 55b78effe522dd14a8bdce313f11226064acb33cd85d53dc9a85798c61aeebea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0) 2024-04-25T15:58:49.061 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 15:58:49 smithi186 podman[64504]: 2024-04-25 15:58:49.060837471 +0000 UTC m=+0.375258096 container init 55b78effe522dd14a8bdce313f11226064acb33cd85d53dc9a85798c61aeebea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T15:58:49.399 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 15:58:49 smithi186 podman[64504]: 2024-04-25 15:58:49.063556562 +0000 UTC m=+0.377977200 container start 55b78effe522dd14a8bdce313f11226064acb33cd85d53dc9a85798c61aeebea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS) 2024-04-25T15:58:49.399 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 15:58:49 smithi186 ceph-mds[64523]: -- 172.21.15.186:0/465463770 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564f1483da40 con 0x564f14885a80 2024-04-25T15:58:49.399 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 15:58:49 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i[64519]: starting mds.i at 2024-04-25T15:58:49.399 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 15:58:49 smithi186 bash[64504]: 55b78effe522dd14a8bdce313f11226064acb33cd85d53dc9a85798c61aeebea 2024-04-25T15:58:49.399 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 15:58:49 smithi186 systemd[1]: Started Ceph mds.i for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:49.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:49.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.859+0000 7f93f7af0640 1 -- 172.21.15.31:0/4133128853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0109a50 msgr2=0x7f93f010be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.859+0000 7f93f7af0640 1 --2- 172.21.15.31:0/4133128853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0109a50 0x7f93f010be70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f93e8009ca0 tx=0x7f93e802a8c0 comp rx=0 tx=0).stop 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.859+0000 7f93f7af0640 1 -- 172.21.15.31:0/4133128853 shutdown_connections 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.859+0000 7f93f7af0640 1 --2- 172.21.15.31:0/4133128853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0109a50 0x7f93f010be70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.859+0000 7f93f7af0640 1 --2- 172.21.15.31:0/4133128853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93f0104d50 0x7f93f0105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.859+0000 7f93f7af0640 1 --2- 172.21.15.31:0/4133128853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f0103b10 0x7f93f0104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.859+0000 7f93f7af0640 1 -- 172.21.15.31:0/4133128853 >> 172.21.15.31:0/4133128853 conn(0x7f93f00fed40 msgr2=0x7f93f01011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.859+0000 7f93f7af0640 1 -- 172.21.15.31:0/4133128853 shutdown_connections 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.859+0000 7f93f7af0640 1 -- 172.21.15.31:0/4133128853 wait complete. 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f7af0640 1 Processor -- start 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f7af0640 1 -- start start 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f7af0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f0103b10 0x7f93f019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:49.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f7af0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0104d50 0x7f93f019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f7af0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93f0109a50 0x7f93f01a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f7af0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f93f010f620 con 0x7f93f0104d50 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f7af0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f93f010f4a0 con 0x7f93f0103b10 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f7af0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f93f010f7a0 con 0x7f93f0109a50 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f508b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0104d50 0x7f93f019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f508b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0104d50 0x7f93f019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48390/0 (socket says 172.21.15.31:48390) 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.860+0000 7f93f508b640 1 -- 172.21.15.31:0/811399129 learned_addr learned my addr 172.21.15.31:0/811399129 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f608d640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93f0109a50 0x7f93f01a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f588c640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f0103b10 0x7f93f019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f508b640 1 -- 172.21.15.31:0/811399129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93f0109a50 msgr2=0x7f93f01a1660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f508b640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93f0109a50 0x7f93f01a1660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f508b640 1 -- 172.21.15.31:0/811399129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f0103b10 msgr2=0x7f93f019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f508b640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f0103b10 0x7f93f019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f508b640 1 -- 172.21.15.31:0/811399129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93f01a1c20 con 0x7f93f0104d50 2024-04-25T15:58:49.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f588c640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f0103b10 0x7f93f019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:58:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f508b640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0104d50 0x7f93f019a790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c920 tx=0x7f93e000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93e6ffd640 1 -- 172.21.15.31:0/811399129 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e0007c70 con 0x7f93f0104d50 2024-04-25T15:58:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93e6ffd640 1 -- 172.21.15.31:0/811399129 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f93e0007e10 con 0x7f93f0104d50 2024-04-25T15:58:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93f01a1f10 con 0x7f93f0104d50 2024-04-25T15:58:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93e6ffd640 1 -- 172.21.15.31:0/811399129 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93e0018820 con 0x7f93f0104d50 2024-04-25T15:58:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.861+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93f01a23d0 con 0x7f93f0104d50 2024-04-25T15:58:49.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.862+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93f0108860 con 0x7f93f0104d50 2024-04-25T15:58:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.863+0000 7f93e6ffd640 1 -- 172.21.15.31:0/811399129 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f93e000f450 con 0x7f93f0104d50 2024-04-25T15:58:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.864+0000 7f93e6ffd640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f93dc0afb70 0x7f93dc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.865+0000 7f93f588c640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f93dc0afb70 0x7f93dc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.865+0000 7f93e6ffd640 1 -- 172.21.15.31:0/811399129 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f93e00deb90 con 0x7f93f0104d50 2024-04-25T15:58:49.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.865+0000 7f93f588c640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f93dc0afb70 0x7f93dc0b2080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f93f0104710 tx=0x7f93ec005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:49.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:49.868+0000 7f93e6ffd640 1 -- 172.21.15.31:0/811399129 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f93e0014030 con 0x7f93f0104d50 2024-04-25T15:58:50.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.100+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f93f002dda0 con 0x7f93f0104d50 2024-04-25T15:58:50.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.101+0000 7f93e6ffd640 1 -- 172.21.15.31:0/811399129 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7995 (secure 0 0 0) 0x7f93e00a7d00 con 0x7f93f0104d50 2024-04-25T15:58:50.101 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:50.102 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T15:58:50.102 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 10 2024-04-25T15:58:50.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f93dc0afb70 msgr2=0x7f93dc0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:50.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f93dc0afb70 0x7f93dc0b2080 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f93f0104710 tx=0x7f93ec005e70 comp rx=0 tx=0).stop 2024-04-25T15:58:50.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0104d50 msgr2=0x7f93f019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:50.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0104d50 0x7f93f019a790 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c920 tx=0x7f93e000cde0 comp rx=0 tx=0).stop 2024-04-25T15:58:50.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 shutdown_connections 2024-04-25T15:58:50.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f93dc0afb70 0x7f93dc0b2080 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:50.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93f0109a50 0x7f93f01a1660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:50.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f93f0104d50 0x7f93f019a790 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:50.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 --2- 172.21.15.31:0/811399129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f0103b10 0x7f93f019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:50.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 >> 172.21.15.31:0/811399129 conn(0x7f93f00fed40 msgr2=0x7f93f0105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:50.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 shutdown_connections 2024-04-25T15:58:50.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:50.103+0000 7f93f7af0640 1 -- 172.21.15.31:0/811399129 wait complete. 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: Deploying daemon mds.g on smithi031 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] up:boot 2024-04-25T15:58:50.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: fsmap 9 up:standby 2024-04-25T15:58:50.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T15:58:50.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:50 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/811399129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:50.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: Deploying daemon mds.g on smithi031 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] up:boot 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: fsmap 9 up:standby 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T15:58:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:50 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/811399129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:50.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: Deploying daemon mds.g on smithi031 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] up:boot 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: fsmap 9 up:standby 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T15:58:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:50 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/811399129' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:51.700 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 15:58:51 smithi031 systemd[1]: Starting Ceph mds.g for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:58:52.012 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 15:58:51 smithi031 podman[70741]: 2024-04-25 15:58:51.609344249 +0000 UTC m=+0.018764525 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:52.012 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 15:58:51 smithi031 podman[70741]: 2024-04-25 15:58:51.766277454 +0000 UTC m=+0.175697718 container create ad720772e5c1ecbe794667350dbd088298d8d89ccac9d7f108750223e0792bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T15:58:52.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 -- 172.21.15.31:0/4083839795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c103520 msgr2=0x7f684c103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 --2- 172.21.15.31:0/4083839795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c103520 0x7f684c103a20 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6844009ca0 tx=0x7f684402a8c0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 -- 172.21.15.31:0/4083839795 shutdown_connections 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 --2- 172.21.15.31:0/4083839795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c103520 0x7f684c103a20 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 --2- 172.21.15.31:0/4083839795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f684c1029c0 0x7f684c102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 --2- 172.21.15.31:0/4083839795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f684c101fe0 0x7f684c1023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 -- 172.21.15.31:0/4083839795 >> 172.21.15.31:0/4083839795 conn(0x7f684c0fdbd0 msgr2=0x7f684c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 -- 172.21.15.31:0/4083839795 shutdown_connections 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 -- 172.21.15.31:0/4083839795 wait complete. 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.240+0000 7f6851f27640 1 Processor -- start 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f6851f27640 1 -- start start 2024-04-25T15:58:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f6851f27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f684c101fe0 0x7f684c19c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f6851f27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c1029c0 0x7f684c19c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f6851f27640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f684c103520 0x7f684c1a3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f6851f27640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f684c1117c0 con 0x7f684c1029c0 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f6851f27640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f684c111640 con 0x7f684c103520 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f6851f27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f684c111940 con 0x7f684c101fe0 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c1029c0 0x7f684c19c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c1029c0 0x7f684c19c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48432/0 (socket says 172.21.15.31:48432) 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684affd640 1 -- 172.21.15.31:0/1631644362 learned_addr learned my addr 172.21.15.31:0/1631644362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684bfff640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f684c103520 0x7f684c1a3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684b7fe640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f684c101fe0 0x7f684c19c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684affd640 1 -- 172.21.15.31:0/1631644362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f684c101fe0 msgr2=0x7f684c19c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684affd640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f684c101fe0 0x7f684c19c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684affd640 1 -- 172.21.15.31:0/1631644362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f684c103520 msgr2=0x7f684c1a3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:52.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684affd640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f684c103520 0x7f684c1a3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684affd640 1 -- 172.21.15.31:0/1631644362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f684c1a3ee0 con 0x7f684c1029c0 2024-04-25T15:58:52.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.241+0000 7f684bfff640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f684c103520 0x7f684c1a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:52.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.242+0000 7f684affd640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c1029c0 0x7f684c19c8e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f683400d950 tx=0x7f683400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:52.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.242+0000 7f6848ff9640 1 -- 172.21.15.31:0/1631644362 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6834017070 con 0x7f684c1029c0 2024-04-25T15:58:52.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.242+0000 7f6848ff9640 1 -- 172.21.15.31:0/1631644362 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6834009c30 con 0x7f684c1029c0 2024-04-25T15:58:52.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.242+0000 7f6848ff9640 1 -- 172.21.15.31:0/1631644362 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f683400f620 con 0x7f684c1029c0 2024-04-25T15:58:52.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.242+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f684c1a41d0 con 0x7f684c1029c0 2024-04-25T15:58:52.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.242+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f684c1a46d0 con 0x7f684c1029c0 2024-04-25T15:58:52.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.243+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f684c10d440 con 0x7f684c1029c0 2024-04-25T15:58:52.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.244+0000 7f6848ff9640 1 -- 172.21.15.31:0/1631644362 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6834010460 con 0x7f684c1029c0 2024-04-25T15:58:52.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.245+0000 7f6848ff9640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f68380afb70 0x7f68380b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:52.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.245+0000 7f684b7fe640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f68380afb70 0x7f68380b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:52.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.246+0000 7f6848ff9640 1 -- 172.21.15.31:0/1631644362 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f68340de430 con 0x7f684c1029c0 2024-04-25T15:58:52.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.246+0000 7f684b7fe640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f68380afb70 0x7f68380b2080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f683c005fd0 tx=0x7f683c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:52.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.249+0000 7f6848ff9640 1 -- 172.21.15.31:0/1631644362 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f68340a7490 con 0x7f684c1029c0 2024-04-25T15:58:52.363 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 15:58:52 smithi031 podman[70741]: 2024-04-25 15:58:52.011678837 +0000 UTC m=+0.421099104 container init ad720772e5c1ecbe794667350dbd088298d8d89ccac9d7f108750223e0792bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T15:58:52.363 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 15:58:52 smithi031 podman[70741]: 2024-04-25 15:58:52.015268272 +0000 UTC m=+0.424688543 container start ad720772e5c1ecbe794667350dbd088298d8d89ccac9d7f108750223e0792bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-25T15:58:52.363 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 15:58:52 smithi031 ceph-mds[70760]: -- 172.21.15.31:0/3515243597 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562127c8fa40 con 0x562127cd7180 2024-04-25T15:58:52.363 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 15:58:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g[70756]: starting mds.g at 2024-04-25T15:58:52.363 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 15:58:52 smithi031 bash[70741]: ad720772e5c1ecbe794667350dbd088298d8d89ccac9d7f108750223e0792bb5 2024-04-25T15:58:52.363 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 15:58:52 smithi031 systemd[1]: Started Ceph mds.g for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:52.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.489+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f684c02db50 con 0x7f684c1029c0 2024-04-25T15:58:52.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.490+0000 7f6848ff9640 1 -- 172.21.15.31:0/1631644362 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8821 (secure 0 0 0) 0x7f6834009de0 con 0x7f684c1029c0 2024-04-25T15:58:52.491 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:52.491 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T15:58:52.491 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 11 2024-04-25T15:58:52.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.492+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f68380afb70 msgr2=0x7f68380b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:52.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.492+0000 7f6851f27640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f68380afb70 0x7f68380b2080 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f683c005fd0 tx=0x7f683c005850 comp rx=0 tx=0).stop 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.492+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c1029c0 msgr2=0x7f684c19c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.492+0000 7f6851f27640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c1029c0 0x7f684c19c8e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f683400d950 tx=0x7f683400de10 comp rx=0 tx=0).stop 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.493+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 shutdown_connections 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.493+0000 7f6851f27640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f68380afb70 0x7f68380b2080 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.493+0000 7f6851f27640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f684c103520 0x7f684c1a3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.493+0000 7f6851f27640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f684c1029c0 0x7f684c19c8e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.493+0000 7f6851f27640 1 --2- 172.21.15.31:0/1631644362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f684c101fe0 0x7f684c19c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.493+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 >> 172.21.15.31:0/1631644362 conn(0x7f684c0fdbd0 msgr2=0x7f684c10a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.493+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 shutdown_connections 2024-04-25T15:58:52.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:52.493+0000 7f6851f27640 1 -- 172.21.15.31:0/1631644362 wait complete. 2024-04-25T15:58:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:52 smithi063 ceph-mon[32413]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:52.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:52 smithi186 ceph-mon[33258]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:52.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:52 smithi031 ceph-mon[29140]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:52.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:52.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:52.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:52.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:52.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:52.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:53.602 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:53 smithi031 ceph-mon[29140]: Deploying daemon mds.a on smithi031 2024-04-25T15:58:53.602 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:53 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.31:6838/3019802236,v1:172.21.15.31:6839/3019802236] up:boot 2024-04-25T15:58:53.602 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:53 smithi031 ceph-mon[29140]: fsmap 10 up:standby 2024-04-25T15:58:53.602 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:53 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T15:58:53.603 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:53 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1631644362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:53.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:53 smithi186 ceph-mon[33258]: Deploying daemon mds.a on smithi031 2024-04-25T15:58:53.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:53 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.31:6838/3019802236,v1:172.21.15.31:6839/3019802236] up:boot 2024-04-25T15:58:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:53 smithi186 ceph-mon[33258]: fsmap 10 up:standby 2024-04-25T15:58:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:53 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T15:58:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:53 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1631644362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:53.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:53 smithi063 ceph-mon[32413]: Deploying daemon mds.a on smithi031 2024-04-25T15:58:53.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:53 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.31:6838/3019802236,v1:172.21.15.31:6839/3019802236] up:boot 2024-04-25T15:58:53.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:53 smithi063 ceph-mon[32413]: fsmap 10 up:standby 2024-04-25T15:58:53.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:53 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T15:58:53.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:53 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1631644362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:54.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:54 smithi031 ceph-mon[29140]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:54.450 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 15:58:54 smithi031 systemd[1]: Starting Ceph mds.a for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T15:58:54.506 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.622+0000 7f967360c640 1 -- 172.21.15.31:0/2803186245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 msgr2=0x7f966c103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.622+0000 7f967360c640 1 --2- 172.21.15.31:0/2803186245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 0x7f966c103700 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f965c009900 tx=0x7f965c02a410 comp rx=0 tx=0).stop 2024-04-25T15:58:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.622+0000 7f967360c640 1 -- 172.21.15.31:0/2803186245 shutdown_connections 2024-04-25T15:58:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.622+0000 7f967360c640 1 --2- 172.21.15.31:0/2803186245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c108940 0x7f966c10ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.622+0000 7f967360c640 1 --2- 172.21.15.31:0/2803186245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f966c103c40 0x7f966c104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.622+0000 7f967360c640 1 --2- 172.21.15.31:0/2803186245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 0x7f966c103700 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.622+0000 7f967360c640 1 -- 172.21.15.31:0/2803186245 >> 172.21.15.31:0/2803186245 conn(0x7f966c0fdbf0 msgr2=0x7f966c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.622+0000 7f967360c640 1 -- 172.21.15.31:0/2803186245 shutdown_connections 2024-04-25T15:58:54.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.622+0000 7f967360c640 1 -- 172.21.15.31:0/2803186245 wait complete. 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f967360c640 1 Processor -- start 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f967360c640 1 -- start start 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f967360c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 0x7f966c19a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f967360c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c103c40 0x7f966c19a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f967360c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f966c108940 0x7f966c1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f967360c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f966c10e4e0 con 0x7f966c102a00 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f967360c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f966c10e360 con 0x7f966c103c40 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f967360c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f966c10e660 con 0x7f966c108940 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f96713a8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 0x7f966c19a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f96713a8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 0x7f966c19a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55106/0 (socket says 172.21.15.31:55106) 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.623+0000 7f96713a8640 1 -- 172.21.15.31:0/1200646051 learned_addr learned my addr 172.21.15.31:0/1200646051 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f9670ba7640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c103c40 0x7f966c19a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f9671ba9640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f966c108940 0x7f966c1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:54.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f96713a8640 1 -- 172.21.15.31:0/1200646051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f966c108940 msgr2=0x7f966c1a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f96713a8640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f966c108940 0x7f966c1a16d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f96713a8640 1 -- 172.21.15.31:0/1200646051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c103c40 msgr2=0x7f966c19a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f96713a8640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c103c40 0x7f966c19a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f96713a8640 1 -- 172.21.15.31:0/1200646051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f966c1a1d40 con 0x7f966c102a00 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f96713a8640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 0x7f966c19a200 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f965c006010 tx=0x7f965c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f96627fc640 1 -- 172.21.15.31:0/1200646051 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965c03f030 con 0x7f966c102a00 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f96627fc640 1 -- 172.21.15.31:0/1200646051 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f965c003aa0 con 0x7f966c102a00 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f966c1a1fd0 con 0x7f966c102a00 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f96627fc640 1 -- 172.21.15.31:0/1200646051 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965c035070 con 0x7f966c102a00 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.624+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f966c1a2430 con 0x7f966c102a00 2024-04-25T15:58:54.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.625+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f966c1a2660 con 0x7f966c102a00 2024-04-25T15:58:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.627+0000 7f96627fc640 1 -- 172.21.15.31:0/1200646051 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f965c042030 con 0x7f966c102a00 2024-04-25T15:58:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.628+0000 7f96627fc640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96580afb70 0x7f96580b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.628+0000 7f9670ba7640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96580afb70 0x7f96580b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.628+0000 7f96627fc640 1 -- 172.21.15.31:0/1200646051 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f965c0ff220 con 0x7f966c102a00 2024-04-25T15:58:54.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.628+0000 7f9670ba7640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96580afb70 0x7f96580b2080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9668005d00 tx=0x7f9668005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:54.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.631+0000 7f96627fc640 1 -- 172.21.15.31:0/1200646051 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f965c0c82a0 con 0x7f966c102a00 2024-04-25T15:58:54.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:54 smithi186 ceph-mon[33258]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:54.788 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 15:58:54 smithi031 podman[71089]: 2024-04-25 15:58:54.508733754 +0000 UTC m=+0.018724300 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:54.789 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 15:58:54 smithi031 podman[71089]: 2024-04-25 15:58:54.638311195 +0000 UTC m=+0.148301727 container create 98c0e7e3c52e760fafe76452976403b380cd01a5c6c367425227d925ad804b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-04-25T15:58:54.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:54 smithi063 ceph-mon[32413]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:54.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.871+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f966c1a2b50 con 0x7f966c102a00 2024-04-25T15:58:54.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.872+0000 7f96627fc640 1 -- 172.21.15.31:0/1200646051 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8821 (secure 0 0 0) 0x7f965c0c7ba0 con 0x7f966c102a00 2024-04-25T15:58:54.873 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:54.873 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T15:58:54.873 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 11 2024-04-25T15:58:54.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96580afb70 msgr2=0x7f96580b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:54.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96580afb70 0x7f96580b2080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9668005d00 tx=0x7f9668005c70 comp rx=0 tx=0).stop 2024-04-25T15:58:54.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 msgr2=0x7f966c19a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:54.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 0x7f966c19a200 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f965c006010 tx=0x7f965c004270 comp rx=0 tx=0).stop 2024-04-25T15:58:54.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 shutdown_connections 2024-04-25T15:58:54.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96580afb70 0x7f96580b2080 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f9668005d00 tx=0x7f9668005c70 comp rx=0 tx=0).stop 2024-04-25T15:58:54.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f966c108940 0x7f966c1a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:54.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f966c103c40 0x7f966c19a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:54.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 --2- 172.21.15.31:0/1200646051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f966c102a00 0x7f966c19a200 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:54.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 >> 172.21.15.31:0/1200646051 conn(0x7f966c0fdbf0 msgr2=0x7f966c109ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:54.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 shutdown_connections 2024-04-25T15:58:54.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:54.874+0000 7f967360c640 1 -- 172.21.15.31:0/1200646051 wait complete. 2024-04-25T15:58:55.199 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 15:58:54 smithi031 podman[71089]: 2024-04-25 15:58:54.907684734 +0000 UTC m=+0.417675267 container init 98c0e7e3c52e760fafe76452976403b380cd01a5c6c367425227d925ad804b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T15:58:55.200 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 15:58:54 smithi031 podman[71089]: 2024-04-25 15:58:54.9105489 +0000 UTC m=+0.420539433 container start 98c0e7e3c52e760fafe76452976403b380cd01a5c6c367425227d925ad804b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T15:58:55.200 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 15:58:54 smithi031 ceph-mds[71149]: -- 172.21.15.31:0/795196717 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557514427a40 con 0x55751446f180 2024-04-25T15:58:55.200 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 15:58:54 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a[71143]: starting mds.a at 2024-04-25T15:58:55.200 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 15:58:55 smithi031 bash[71089]: 98c0e7e3c52e760fafe76452976403b380cd01a5c6c367425227d925ad804b78 2024-04-25T15:58:55.200 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 15:58:55 smithi031 systemd[1]: Started Ceph mds.a for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:55 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1200646051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:55 smithi186 ceph-mon[33258]: Deploying daemon mds.b on smithi063 2024-04-25T15:58:55.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:55 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1200646051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:55.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:55.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:55.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:55.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:55.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:55.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:55.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:55 smithi031 ceph-mon[29140]: Deploying daemon mds.b on smithi063 2024-04-25T15:58:55.773 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:55 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1200646051' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:55.773 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:55.773 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:55.773 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:55.773 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-25T15:58:55.773 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-25T15:58:55.774 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:58:55.774 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:55 smithi063 ceph-mon[32413]: Deploying daemon mds.b on smithi063 2024-04-25T15:58:56.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:56 smithi186 ceph-mon[33258]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:56.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:56 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] up:boot 2024-04-25T15:58:56.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:56 smithi186 ceph-mon[33258]: fsmap 11 up:standby 2024-04-25T15:58:56.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:56 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T15:58:56.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:56 smithi031 ceph-mon[29140]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:56.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:56 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] up:boot 2024-04-25T15:58:56.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:56 smithi031 ceph-mon[29140]: fsmap 11 up:standby 2024-04-25T15:58:56.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:56 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T15:58:56.800 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:56 smithi063 ceph-mon[32413]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:56.800 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:56 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] up:boot 2024-04-25T15:58:56.801 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:56 smithi063 ceph-mon[32413]: fsmap 11 up:standby 2024-04-25T15:58:56.801 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:56 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T15:58:56.887 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:57.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.014+0000 7faadbc92640 1 -- 172.21.15.31:0/2535232084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad41050a0 msgr2=0x7faad41054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:57.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.014+0000 7faadbc92640 1 --2- 172.21.15.31:0/2535232084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad41050a0 0x7faad41054b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7faad0009900 tx=0x7faad002a410 comp rx=0 tx=0).stop 2024-04-25T15:58:57.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.015+0000 7faadbc92640 1 -- 172.21.15.31:0/2535232084 shutdown_connections 2024-04-25T15:58:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.015+0000 7faadbc92640 1 --2- 172.21.15.31:0/2535232084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faad4102250 0x7faad410bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.015+0000 7faadbc92640 1 --2- 172.21.15.31:0/2535232084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad4101750 0x7faad4101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.015+0000 7faadbc92640 1 --2- 172.21.15.31:0/2535232084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad41050a0 0x7faad41054b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.015+0000 7faadbc92640 1 -- 172.21.15.31:0/2535232084 >> 172.21.15.31:0/2535232084 conn(0x7faad40fd3b0 msgr2=0x7faad40ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.015+0000 7faadbc92640 1 -- 172.21.15.31:0/2535232084 shutdown_connections 2024-04-25T15:58:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.015+0000 7faadbc92640 1 -- 172.21.15.31:0/2535232084 wait complete. 2024-04-25T15:58:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.015+0000 7faadbc92640 1 Processor -- start 2024-04-25T15:58:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.015+0000 7faadbc92640 1 -- start start 2024-04-25T15:58:57.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faadbc92640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faad4101750 0x7faad419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:57.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faadbc92640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad4102250 0x7faad419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:57.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faadbc92640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad41050a0 0x7faad41a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:57.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faadbc92640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faad410f5f0 con 0x7faad4102250 2024-04-25T15:58:57.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faadbc92640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faad410f470 con 0x7faad4101750 2024-04-25T15:58:57.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faadbc92640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faad410f770 con 0x7faad41050a0 2024-04-25T15:58:57.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad922d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad4102250 0x7faad419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:57.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad922d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad4102250 0x7faad419a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55150/0 (socket says 172.21.15.31:55150) 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad922d640 1 -- 172.21.15.31:0/292860106 learned_addr learned my addr 172.21.15.31:0/292860106 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad9a2e640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faad4101750 0x7faad419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faada22f640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad41050a0 0x7faad41a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad922d640 1 -- 172.21.15.31:0/292860106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad41050a0 msgr2=0x7faad41a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad922d640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad41050a0 0x7faad41a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad922d640 1 -- 172.21.15.31:0/292860106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faad4101750 msgr2=0x7faad419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad922d640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faad4101750 0x7faad419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad922d640 1 -- 172.21.15.31:0/292860106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faad41a1d20 con 0x7faad4102250 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faad922d640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad4102250 0x7faad419a770 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faac400d950 tx=0x7faac400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faacaffd640 1 -- 172.21.15.31:0/292860106 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac4017070 con 0x7faad4102250 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faacaffd640 1 -- 172.21.15.31:0/292860106 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faac4009c30 con 0x7faad4102250 2024-04-25T15:58:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faad41a1fb0 con 0x7faad4102250 2024-04-25T15:58:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.016+0000 7faacaffd640 1 -- 172.21.15.31:0/292860106 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac400f620 con 0x7faad4102250 2024-04-25T15:58:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.017+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faad41a24c0 con 0x7faad4102250 2024-04-25T15:58:57.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.018+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faad4108a30 con 0x7faad4102250 2024-04-25T15:58:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.019+0000 7faacaffd640 1 -- 172.21.15.31:0/292860106 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7faac4005030 con 0x7faad4102250 2024-04-25T15:58:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.020+0000 7faacaffd640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faac00afb70 0x7faac00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.020+0000 7faad9a2e640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faac00afb70 0x7faac00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.020+0000 7faacaffd640 1 -- 172.21.15.31:0/292860106 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7faac40de320 con 0x7faad4102250 2024-04-25T15:58:57.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.021+0000 7faad9a2e640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faac00afb70 0x7faac00b2080 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faad0005ab0 tx=0x7faad0031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:57.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.023+0000 7faacaffd640 1 -- 172.21.15.31:0/292860106 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faac40a7420 con 0x7faad4102250 2024-04-25T15:58:57.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.250+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faad402db50 con 0x7faad4102250 2024-04-25T15:58:57.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.250+0000 7faacaffd640 1 -- 172.21.15.31:0/292860106 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9647 (secure 0 0 0) 0x7faac4009de0 con 0x7faad4102250 2024-04-25T15:58:57.251 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:57.251 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-25T15:58:57.252 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 12 2024-04-25T15:58:57.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faac00afb70 msgr2=0x7faac00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:57.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faac00afb70 0x7faac00b2080 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7faad0005ab0 tx=0x7faad0031400 comp rx=0 tx=0).stop 2024-04-25T15:58:57.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad4102250 msgr2=0x7faad419a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:57.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad4102250 0x7faad419a770 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faac400d950 tx=0x7faac400de10 comp rx=0 tx=0).stop 2024-04-25T15:58:57.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 shutdown_connections 2024-04-25T15:58:57.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faac00afb70 0x7faac00b2080 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:57.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad41050a0 0x7faad41a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:57.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faad4102250 0x7faad419a770 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:57.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 --2- 172.21.15.31:0/292860106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faad4101750 0x7faad419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:57.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 >> 172.21.15.31:0/292860106 conn(0x7faad40fd3b0 msgr2=0x7faad4105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:57.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 shutdown_connections 2024-04-25T15:58:57.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:57.252+0000 7faadbc92640 1 -- 172.21.15.31:0/292860106 wait complete. 2024-04-25T15:58:57.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:57 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/292860106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:57.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:57 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/292860106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:57.730 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:57 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/292860106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:58:57.982 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:58:57 smithi063 podman[62944]: 2024-04-25 15:58:57.853060973 +0000 UTC m=+0.018358789 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T15:58:57.983 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:58:57 smithi063 podman[62944]: 2024-04-25 15:58:57.982204839 +0000 UTC m=+0.147507531 container create 497d4c2b60ca14b9ddc7b51803b7ee3e90fb717b6aeeb3d6d1ea7f7bbe8c9894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T15:58:58.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:58 smithi063 ceph-mon[32413]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:58.620 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:58:58 smithi063 podman[62944]: 2024-04-25 15:58:58.243340364 +0000 UTC m=+0.408638165 container init 497d4c2b60ca14b9ddc7b51803b7ee3e90fb717b6aeeb3d6d1ea7f7bbe8c9894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T15:58:58.620 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:58:58 smithi063 podman[62944]: 2024-04-25 15:58:58.246459355 +0000 UTC m=+0.411757162 container start 497d4c2b60ca14b9ddc7b51803b7ee3e90fb717b6aeeb3d6d1ea7f7bbe8c9894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-25T15:58:58.620 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:58:58 smithi063 ceph-mds[62963]: -- 172.21.15.63:0/12092174 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563abeef7a40 con 0x563abef3f180 2024-04-25T15:58:58.620 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:58:58 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: starting mds.b at 2024-04-25T15:58:58.620 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:58:58 smithi063 bash[62944]: 497d4c2b60ca14b9ddc7b51803b7ee3e90fb717b6aeeb3d6d1ea7f7bbe8c9894 2024-04-25T15:58:58.620 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:58:58 smithi063 systemd[1]: Started Ceph mds.b for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T15:58:58.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:58 smithi031 ceph-mon[29140]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:58.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:58 smithi186 ceph-mon[33258]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:58:59.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:58:59.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.378+0000 7f9e282f2640 1 -- 172.21.15.31:0/3181102305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 msgr2=0x7f9e2010be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.378+0000 7f9e282f2640 1 --2- 172.21.15.31:0/3181102305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 0x7f9e2010be70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009ca0 tx=0x7f9e1802a8c0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 -- 172.21.15.31:0/3181102305 shutdown_connections 2024-04-25T15:58:59.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 --2- 172.21.15.31:0/3181102305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 0x7f9e2010be70 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 --2- 172.21.15.31:0/3181102305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e20104d50 0x7f9e20105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 --2- 172.21.15.31:0/3181102305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e20103b10 0x7f9e20104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 -- 172.21.15.31:0/3181102305 >> 172.21.15.31:0/3181102305 conn(0x7f9e200fed40 msgr2=0x7f9e201011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 -- 172.21.15.31:0/3181102305 shutdown_connections 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 -- 172.21.15.31:0/3181102305 wait complete. 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 Processor -- start 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 -- start start 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e20103b10 0x7f9e2019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e20104d50 0x7f9e2019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 0x7f9e201a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2010f640 con 0x7f9e20109a50 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2010f4c0 con 0x7f9e20103b10 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.379+0000 7f9e282f2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9e2010f7c0 con 0x7f9e20104d50 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2688f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 0x7f9e201a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:59.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2688f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 0x7f9e201a16e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55162/0 (socket says 172.21.15.31:55162) 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2688f640 1 -- 172.21.15.31:0/1178613418 learned_addr learned my addr 172.21.15.31:0/1178613418 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2588d640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e20104d50 0x7f9e2019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2608e640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e20103b10 0x7f9e2019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2688f640 1 -- 172.21.15.31:0/1178613418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e20104d50 msgr2=0x7f9e2019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2688f640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e20104d50 0x7f9e2019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2688f640 1 -- 172.21.15.31:0/1178613418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e20103b10 msgr2=0x7f9e2019a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2688f640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e20103b10 0x7f9e2019a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2688f640 1 -- 172.21.15.31:0/1178613418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e201a1d50 con 0x7f9e20109a50 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2588d640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e20104d50 0x7f9e2019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2608e640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e20103b10 0x7f9e2019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e2688f640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 0x7f9e201a16e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9e18004af0 tx=0x7f9e18031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e177fe640 1 -- 172.21.15.31:0/1178613418 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e18003b70 con 0x7f9e20109a50 2024-04-25T15:58:59.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e177fe640 1 -- 172.21.15.31:0/1178613418 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e18003d10 con 0x7f9e20109a50 2024-04-25T15:58:59.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e201a1fe0 con 0x7f9e20109a50 2024-04-25T15:58:59.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e177fe640 1 -- 172.21.15.31:0/1178613418 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1802fb20 con 0x7f9e20109a50 2024-04-25T15:58:59.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.380+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e201a24a0 con 0x7f9e20109a50 2024-04-25T15:58:59.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.381+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e20108860 con 0x7f9e20109a50 2024-04-25T15:58:59.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.383+0000 7f9e177fe640 1 -- 172.21.15.31:0/1178613418 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9e18004580 con 0x7f9e20109a50 2024-04-25T15:58:59.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.383+0000 7f9e177fe640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e0c0afb70 0x7f9e0c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:59.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.383+0000 7f9e2608e640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e0c0afb70 0x7f9e0c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:59.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.383+0000 7f9e177fe640 1 -- 172.21.15.31:0/1178613418 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f9e180fff60 con 0x7f9e20109a50 2024-04-25T15:58:59.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.384+0000 7f9e2608e640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e0c0afb70 0x7f9e0c0b2080 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9e20104710 tx=0x7f9e1c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:59.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.387+0000 7f9e177fe640 1 -- 172.21.15.31:0/1178613418 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9e180c8fe0 con 0x7f9e20109a50 2024-04-25T15:58:59.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.617+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9e2002db50 con 0x7f9e20109a50 2024-04-25T15:58:59.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.617+0000 7f9e177fe640 1 -- 172.21.15.31:0/1178613418 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10470 (secure 0 0 0) 0x7f9e180d1060 con 0x7f9e20109a50 2024-04-25T15:58:59.618 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:59.618 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14613,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14631,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24455,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24460,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24521,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24553,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24563,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-25T15:58:59.619 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 13 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e0c0afb70 msgr2=0x7f9e0c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e0c0afb70 0x7f9e0c0b2080 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9e20104710 tx=0x7f9e1c005f90 comp rx=0 tx=0).stop 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 msgr2=0x7f9e201a16e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 0x7f9e201a16e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9e18004af0 tx=0x7f9e18031700 comp rx=0 tx=0).stop 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 shutdown_connections 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e0c0afb70 0x7f9e0c0b2080 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e20109a50 0x7f9e201a16e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e20104d50 0x7f9e2019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 --2- 172.21.15.31:0/1178613418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e20103b10 0x7f9e2019a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 >> 172.21.15.31:0/1178613418 conn(0x7f9e200fed40 msgr2=0x7f9e20105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:59.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 shutdown_connections 2024-04-25T15:58:59.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.620+0000 7f9e282f2640 1 -- 172.21.15.31:0/1178613418 wait complete. 2024-04-25T15:58:59.633 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] up:boot 2024-04-25T15:58:59.633 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: fsmap 12 up:standby 2024-04-25T15:58:59.633 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T15:58:59.633 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.633 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.633 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.633 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.633 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.633 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:59.634 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:58:59 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.635 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] up:boot 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: fsmap 12 up:standby 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:59.665 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:58:59 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.667 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] up:boot 2024-04-25T15:58:59.667 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: fsmap 12 up:standby 2024-04-25T15:58:59.667 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T15:58:59.668 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.668 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.668 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.668 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.668 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.668 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:58:59.668 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:58:59 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:58:59.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 -- 172.21.15.31:0/1800810270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4103b00 msgr2=0x7fd4b4104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 --2- 172.21.15.31:0/1800810270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4103b00 0x7fd4b4104800 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4009900 tx=0x7fd4a402a410 comp rx=0 tx=0).stop 2024-04-25T15:58:59.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 -- 172.21.15.31:0/1800810270 shutdown_connections 2024-04-25T15:58:59.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 --2- 172.21.15.31:0/1800810270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b4109a40 0x7fd4b410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 --2- 172.21.15.31:0/1800810270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4b4104d40 0x7fd4b4105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 --2- 172.21.15.31:0/1800810270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4103b00 0x7fd4b4104800 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 -- 172.21.15.31:0/1800810270 >> 172.21.15.31:0/1800810270 conn(0x7fd4b40fed30 msgr2=0x7fd4b41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:59.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 -- 172.21.15.31:0/1800810270 shutdown_connections 2024-04-25T15:58:59.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 -- 172.21.15.31:0/1800810270 wait complete. 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 Processor -- start 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.748+0000 7fd4ba451640 1 -- start start 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4ba451640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4b4103b00 0x7fd4b419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4ba451640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b4104d40 0x7fd4b419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4ba451640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4109a40 0x7fd4b41a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4ba451640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b410f600 con 0x7fd4b4109a40 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4ba451640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b410f480 con 0x7fd4b4103b00 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4ba451640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b410f780 con 0x7fd4b4104d40 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b89ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4109a40 0x7fd4b41a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b89ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4109a40 0x7fd4b41a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55182/0 (socket says 172.21.15.31:55182) 2024-04-25T15:58:59.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b89ee640 1 -- 172.21.15.31:0/2021630262 learned_addr learned my addr 172.21.15.31:0/2021630262 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b3fff640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4b4103b00 0x7fd4b419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b37fe640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b4104d40 0x7fd4b419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b89ee640 1 -- 172.21.15.31:0/2021630262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b4104d40 msgr2=0x7fd4b419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b89ee640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b4104d40 0x7fd4b419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b89ee640 1 -- 172.21.15.31:0/2021630262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4b4103b00 msgr2=0x7fd4b419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b89ee640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4b4103b00 0x7fd4b419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b89ee640 1 -- 172.21.15.31:0/2021630262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b41a1d20 con 0x7fd4b4109a40 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.749+0000 7fd4b89ee640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4109a40 0x7fd4b41a16b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00eb10 tx=0x7fd4ac00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.750+0000 7fd4b17fa640 1 -- 172.21.15.31:0/2021630262 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac004b20 con 0x7fd4b4109a40 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.750+0000 7fd4b17fa640 1 -- 172.21.15.31:0/2021630262 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4ac004cc0 con 0x7fd4b4109a40 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.750+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4b41a1fb0 con 0x7fd4b4109a40 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.750+0000 7fd4b17fa640 1 -- 172.21.15.31:0/2021630262 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac010670 con 0x7fd4b4109a40 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.750+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4b41a24c0 con 0x7fd4b4109a40 2024-04-25T15:58:59.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.751+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4b4108850 con 0x7fd4b4109a40 2024-04-25T15:58:59.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.752+0000 7fd4b17fa640 1 -- 172.21.15.31:0/2021630262 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd4ac005000 con 0x7fd4b4109a40 2024-04-25T15:58:59.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.753+0000 7fd4b17fa640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd49c0afb70 0x7fd49c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:58:59.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.753+0000 7fd4b17fa640 1 -- 172.21.15.31:0/2021630262 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fd4ac0ded50 con 0x7fd4b4109a40 2024-04-25T15:58:59.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.753+0000 7fd4b3fff640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd49c0afb70 0x7fd49c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:58:59.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.753+0000 7fd4b3fff640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd49c0afb70 0x7fd49c0b2080 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4005ab0 tx=0x7fd4a4031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:58:59.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.756+0000 7fd4b17fa640 1 -- 172.21.15.31:0/2021630262 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd4ac053ff0 con 0x7fd4b4109a40 2024-04-25T15:58:59.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.944+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd4b402dd20 con 0x7fd4b4109a40 2024-04-25T15:58:59.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.945+0000 7fd4b17fa640 1 -- 172.21.15.31:0/2021630262 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) ==== 74+0+18420 (secure 0 0 0) 0x7fd4ac0a7dd0 con 0x7fd4b4109a40 2024-04-25T15:58:59.946 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:58:59.946 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":71,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:58:14.039554+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:58:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd49c0afb70 msgr2=0x7fd49c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd49c0afb70 0x7fd49c0b2080 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd4a4005ab0 tx=0x7fd4a4031400 comp rx=0 tx=0).stop 2024-04-25T15:58:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4109a40 msgr2=0x7fd4b41a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:58:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4109a40 0x7fd4b41a16b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac00eb10 tx=0x7fd4ac00c2d0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 shutdown_connections 2024-04-25T15:58:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd49c0afb70 0x7fd49c0b2080 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4b4109a40 0x7fd4b41a16b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b4104d40 0x7fd4b419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 --2- 172.21.15.31:0/2021630262 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4b4103b00 0x7fd4b419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:58:59.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 >> 172.21.15.31:0/2021630262 conn(0x7fd4b40fed30 msgr2=0x7fd4b4105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:58:59.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 shutdown_connections 2024-04-25T15:58:59.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:58:59.947+0000 7fd4ba451640 1 -- 172.21.15.31:0/2021630262 wait complete. 2024-04-25T15:58:59.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T15:58:59.960 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T15:59:00.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.081+0000 7fd445d1f640 1 -- 172.21.15.31:0/3377930215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4401029f0 msgr2=0x7fd4401036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 --2- 172.21.15.31:0/3377930215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4401029f0 0x7fd4401036f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd430009900 tx=0x7fd43002a410 comp rx=0 tx=0).stop 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 -- 172.21.15.31:0/3377930215 shutdown_connections 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 --2- 172.21.15.31:0/3377930215 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd440108930 0x7fd44010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 --2- 172.21.15.31:0/3377930215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd440103c30 0x7fd440104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 --2- 172.21.15.31:0/3377930215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd4401029f0 0x7fd4401036f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 -- 172.21.15.31:0/3377930215 >> 172.21.15.31:0/3377930215 conn(0x7fd4400fdbe0 msgr2=0x7fd440100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 -- 172.21.15.31:0/3377930215 shutdown_connections 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 -- 172.21.15.31:0/3377930215 wait complete. 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 Processor -- start 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.082+0000 7fd445d1f640 1 -- start start 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd445d1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4401029f0 0x7fd44006bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd445d1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd440103c30 0x7fd44019f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd445d1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd440108930 0x7fd4401a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd445d1f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd44010e4d0 con 0x7fd440108930 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd445d1f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd44010e350 con 0x7fd440103c30 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd445d1f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd44010e650 con 0x7fd4401029f0 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd440108930 0x7fd4401a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd440108930 0x7fd4401a15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55198/0 (socket says 172.21.15.31:55198) 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4401029f0 0x7fd44006bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43ffff640 1 -- 172.21.15.31:0/2556787375 learned_addr learned my addr 172.21.15.31:0/2556787375 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43effd640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd440103c30 0x7fd44019f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43ffff640 1 -- 172.21.15.31:0/2556787375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4401029f0 msgr2=0x7fd44006bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43ffff640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4401029f0 0x7fd44006bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43ffff640 1 -- 172.21.15.31:0/2556787375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd440103c30 msgr2=0x7fd44019f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43ffff640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd440103c30 0x7fd44019f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43ffff640 1 -- 172.21.15.31:0/2556787375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4401a1de0 con 0x7fd440108930 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.083+0000 7fd43effd640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd440103c30 0x7fd44019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:59:00.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.084+0000 7fd43ffff640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd440108930 0x7fd4401a15c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd43800eaa0 tx=0x7fd43800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:00.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.084+0000 7fd43cff9640 1 -- 172.21.15.31:0/2556787375 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd43800cdb0 con 0x7fd440108930 2024-04-25T15:59:00.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.084+0000 7fd43cff9640 1 -- 172.21.15.31:0/2556787375 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd438005020 con 0x7fd440108930 2024-04-25T15:59:00.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.084+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4401a20d0 con 0x7fd440108930 2024-04-25T15:59:00.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.084+0000 7fd43cff9640 1 -- 172.21.15.31:0/2556787375 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4380116d0 con 0x7fd440108930 2024-04-25T15:59:00.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.084+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4401a2620 con 0x7fd440108930 2024-04-25T15:59:00.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.085+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd440107740 con 0x7fd440108930 2024-04-25T15:59:00.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.086+0000 7fd43cff9640 1 -- 172.21.15.31:0/2556787375 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd4380051d0 con 0x7fd440108930 2024-04-25T15:59:00.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.087+0000 7fd43cff9640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd4280afb70 0x7fd4280b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.088+0000 7fd43f7fe640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd4280afb70 0x7fd4280b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.088+0000 7fd43cff9640 1 -- 172.21.15.31:0/2556787375 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7fd4380debc0 con 0x7fd440108930 2024-04-25T15:59:00.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.088+0000 7fd43f7fe640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd4280afb70 0x7fd4280b2080 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd430005ab0 tx=0x7fd430031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:00.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.091+0000 7fd43cff9640 1 -- 172.21.15.31:0/2556787375 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd438016030 con 0x7fd440108930 2024-04-25T15:59:00.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.294+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd44002db50 con 0x7fd440108930 2024-04-25T15:59:00.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.294+0000 7fd43cff9640 1 -- 172.21.15.31:0/2556787375 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v71) ==== 93+0+10 (secure 0 0 0) 0x7fd4380a7670 con 0x7fd440108930 2024-04-25T15:59:00.295 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T15:59:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd4280afb70 msgr2=0x7fd4280b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd4280afb70 0x7fd4280b2080 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd430005ab0 tx=0x7fd430031400 comp rx=0 tx=0).stop 2024-04-25T15:59:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd440108930 msgr2=0x7fd4401a15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd440108930 0x7fd4401a15c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd43800eaa0 tx=0x7fd43800ef60 comp rx=0 tx=0).stop 2024-04-25T15:59:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 shutdown_connections 2024-04-25T15:59:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd4280afb70 0x7fd4280b2080 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd440108930 0x7fd4401a15c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd440103c30 0x7fd44019f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 --2- 172.21.15.31:0/2556787375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4401029f0 0x7fd44006bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 >> 172.21.15.31:0/2556787375 conn(0x7fd4400fdbe0 msgr2=0x7fd440104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:00.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 shutdown_connections 2024-04-25T15:59:00.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.297+0000 7fd445d1f640 1 -- 172.21.15.31:0/2556787375 wait complete. 2024-04-25T15:59:00.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-25T15:59:00.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.433+0000 7f45fa649640 1 -- 172.21.15.31:0/945630805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 msgr2=0x7f45f410be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.433+0000 7f45fa649640 1 --2- 172.21.15.31:0/945630805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 0x7f45f410be60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009ca0 tx=0x7f45ec02ecd0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.433+0000 7f45fa649640 1 -- 172.21.15.31:0/945630805 shutdown_connections 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.433+0000 7f45fa649640 1 --2- 172.21.15.31:0/945630805 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 0x7f45f410be60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.433+0000 7f45fa649640 1 --2- 172.21.15.31:0/945630805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45f4104d40 0x7f45f4105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.433+0000 7f45fa649640 1 --2- 172.21.15.31:0/945630805 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45f4103b00 0x7f45f4104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.433+0000 7f45fa649640 1 -- 172.21.15.31:0/945630805 >> 172.21.15.31:0/945630805 conn(0x7f45f40fed30 msgr2=0x7f45f41011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.433+0000 7f45fa649640 1 -- 172.21.15.31:0/945630805 shutdown_connections 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.433+0000 7f45fa649640 1 -- 172.21.15.31:0/945630805 wait complete. 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45fa649640 1 Processor -- start 2024-04-25T15:59:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45fa649640 1 -- start start 2024-04-25T15:59:00.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45fa649640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45f4103b00 0x7f45f4197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45fa649640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45f4104d40 0x7f45f41984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45fa649640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 0x7f45f419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45fa649640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f45f410d400 con 0x7f45f4109a40 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45fa649640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f45f410d280 con 0x7f45f4104d40 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45fa649640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45f410d580 con 0x7f45f4103b00 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45f8be6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 0x7f45f419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45f37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45f4104d40 0x7f45f41984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45f8be6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 0x7f45f419f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55226/0 (socket says 172.21.15.31:55226) 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45f8be6640 1 -- 172.21.15.31:0/2864675154 learned_addr learned my addr 172.21.15.31:0/2864675154 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.434+0000 7f45f3fff640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45f4103b00 0x7f45f4197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.435+0000 7f45f8be6640 1 -- 172.21.15.31:0/2864675154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45f4103b00 msgr2=0x7f45f4197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.435+0000 7f45f8be6640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45f4103b00 0x7f45f4197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.435+0000 7f45f8be6640 1 -- 172.21.15.31:0/2864675154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45f4104d40 msgr2=0x7f45f41984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.435+0000 7f45f8be6640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45f4104d40 0x7f45f41984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.435+0000 7f45f8be6640 1 -- 172.21.15.31:0/2864675154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45f419faa0 con 0x7f45f4109a40 2024-04-25T15:59:00.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.435+0000 7f45f8be6640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 0x7f45f419f430 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f45ec037ec0 tx=0x7f45ec003ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:00.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.438+0000 7f45f17fa640 1 -- 172.21.15.31:0/2864675154 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec03c040 con 0x7f45f4109a40 2024-04-25T15:59:00.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.438+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45f419fd30 con 0x7f45f4109a40 2024-04-25T15:59:00.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.438+0000 7f45f17fa640 1 -- 172.21.15.31:0/2864675154 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45ec047070 con 0x7f45f4109a40 2024-04-25T15:59:00.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.438+0000 7f45f17fa640 1 -- 172.21.15.31:0/2864675154 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ec037850 con 0x7f45f4109a40 2024-04-25T15:59:00.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.438+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45f41a01c0 con 0x7f45f4109a40 2024-04-25T15:59:00.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.439+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45f4108850 con 0x7f45f4109a40 2024-04-25T15:59:00.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.442+0000 7f45f17fa640 1 -- 172.21.15.31:0/2864675154 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f45ec02f940 con 0x7f45f4109a40 2024-04-25T15:59:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.443+0000 7f45f17fa640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45dc0afaf0 0x7f45dc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.443+0000 7f45f3fff640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45dc0afaf0 0x7f45dc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.443+0000 7f45f3fff640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45dc0afaf0 0x7f45dc0b2000 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f45e40095f0 tx=0x7f45e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:00.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.445+0000 7f45f17fa640 1 -- 172.21.15.31:0/2864675154 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f45ec0c8f80 con 0x7f45f4109a40 2024-04-25T15:59:00.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.450+0000 7f45f17fa640 1 -- 172.21.15.31:0/2864675154 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45ec0cd090 con 0x7f45f4109a40 2024-04-25T15:59:00.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.644+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f45f402dd20 con 0x7f45f4109a40 2024-04-25T15:59:00.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.644+0000 7f45f17fa640 1 -- 172.21.15.31:0/2864675154 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v71) ==== 84+0+65 (secure 0 0 0) 0x7f45ec0cc990 con 0x7f45f4109a40 2024-04-25T15:59:00.645 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:00.645 INFO:teuthology.orchestra.run.smithi031.stdout:[ 2024-04-25T15:59:00.645 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T15:59:00.645 INFO:teuthology.orchestra.run.smithi031.stdout: "poolnum": 1, 2024-04-25T15:59:00.645 INFO:teuthology.orchestra.run.smithi031.stdout: "poolname": ".mgr" 2024-04-25T15:59:00.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T15:59:00.645 INFO:teuthology.orchestra.run.smithi031.stdout:] 2024-04-25T15:59:00.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45dc0afaf0 msgr2=0x7f45dc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45dc0afaf0 0x7f45dc0b2000 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f45e40095f0 tx=0x7f45e4005f90 comp rx=0 tx=0).stop 2024-04-25T15:59:00.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 msgr2=0x7f45f419f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 0x7f45f419f430 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f45ec037ec0 tx=0x7f45ec003ee0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 shutdown_connections 2024-04-25T15:59:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45dc0afaf0 0x7f45dc0b2000 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45f4109a40 0x7f45f419f430 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45f4104d40 0x7f45f41984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 --2- 172.21.15.31:0/2864675154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45f4103b00 0x7f45f4197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 >> 172.21.15.31:0/2864675154 conn(0x7f45f40fed30 msgr2=0x7f45f4105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 shutdown_connections 2024-04-25T15:59:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.646+0000 7f45fa649640 1 -- 172.21.15.31:0/2864675154 wait complete. 2024-04-25T15:59:00.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:00 smithi031 ceph-mon[29140]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:59:00.657 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:00 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1178613418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:59:00.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:00 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2021630262' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:00.658 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:00 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2556787375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T15:59:00.659 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-25T15:59:00.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-25T15:59:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.775+0000 7ff6b5985640 1 -- 172.21.15.31:0/3340842599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b01029e0 msgr2=0x7ff6b01036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.775+0000 7ff6b5985640 1 --2- 172.21.15.31:0/3340842599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b01029e0 0x7ff6b01036e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff6a400d910 tx=0x7ff6a402c380 comp rx=0 tx=0).stop 2024-04-25T15:59:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.775+0000 7ff6b5985640 1 -- 172.21.15.31:0/3340842599 shutdown_connections 2024-04-25T15:59:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.775+0000 7ff6b5985640 1 --2- 172.21.15.31:0/3340842599 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b0108920 0x7ff6b010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.775+0000 7ff6b5985640 1 --2- 172.21.15.31:0/3340842599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b0103c20 0x7ff6b0104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.775+0000 7ff6b5985640 1 --2- 172.21.15.31:0/3340842599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b01029e0 0x7ff6b01036e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.775+0000 7ff6b5985640 1 -- 172.21.15.31:0/3340842599 >> 172.21.15.31:0/3340842599 conn(0x7ff6b00fdbd0 msgr2=0x7ff6b0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.775+0000 7ff6b5985640 1 -- 172.21.15.31:0/3340842599 shutdown_connections 2024-04-25T15:59:00.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.775+0000 7ff6b5985640 1 -- 172.21.15.31:0/3340842599 wait complete. 2024-04-25T15:59:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6b5985640 1 Processor -- start 2024-04-25T15:59:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6b5985640 1 -- start start 2024-04-25T15:59:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6b5985640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b01029e0 0x7ff6b0198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6b5985640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b0103c20 0x7ff6b0198880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6b5985640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b0108920 0x7ff6b019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6b5985640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b010d3c0 con 0x7ff6b0108920 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6b5985640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b010d240 con 0x7ff6b01029e0 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6b5985640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff6b010d540 con 0x7ff6b0103c20 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b0108920 0x7ff6b019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b0108920 0x7ff6b019f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55250/0 (socket says 172.21.15.31:55250) 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6af7fe640 1 -- 172.21.15.31:0/233278246 learned_addr learned my addr 172.21.15.31:0/233278246 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6aeffd640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b01029e0 0x7ff6b0198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6ae7fc640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b0103c20 0x7ff6b0198880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6af7fe640 1 -- 172.21.15.31:0/233278246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b0103c20 msgr2=0x7ff6b0198880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6af7fe640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b0103c20 0x7ff6b0198880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6af7fe640 1 -- 172.21.15.31:0/233278246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b01029e0 msgr2=0x7ff6b0198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6af7fe640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b01029e0 0x7ff6b0198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.776+0000 7ff6af7fe640 1 -- 172.21.15.31:0/233278246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6b019fe80 con 0x7ff6b0108920 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.777+0000 7ff6af7fe640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b0108920 0x7ff6b019f810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800eb10 tx=0x7ff6a800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.777+0000 7ff6b4983640 1 -- 172.21.15.31:0/233278246 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8004b20 con 0x7ff6b0108920 2024-04-25T15:59:00.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.777+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6b01a0170 con 0x7ff6b0108920 2024-04-25T15:59:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.777+0000 7ff6b4983640 1 -- 172.21.15.31:0/233278246 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6a8004cc0 con 0x7ff6b0108920 2024-04-25T15:59:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.777+0000 7ff6b4983640 1 -- 172.21.15.31:0/233278246 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a8010670 con 0x7ff6b0108920 2024-04-25T15:59:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.777+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6b01a0670 con 0x7ff6b0108920 2024-04-25T15:59:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.778+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6b0107730 con 0x7ff6b0108920 2024-04-25T15:59:00.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.779+0000 7ff6b4983640 1 -- 172.21.15.31:0/233278246 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff6a8005020 con 0x7ff6b0108920 2024-04-25T15:59:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.780+0000 7ff6b4983640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff6980afaf0 0x7ff6980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.780+0000 7ff6b4983640 1 -- 172.21.15.31:0/233278246 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7ff6a80ded50 con 0x7ff6b0108920 2024-04-25T15:59:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.780+0000 7ff6aeffd640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff6980afaf0 0x7ff6980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:00.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.781+0000 7ff6aeffd640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff6980afaf0 0x7ff6980b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4000c00 tx=0x7ff6a4004810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:00.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.785+0000 7ff6b4983640 1 -- 172.21.15.31:0/233278246 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6a80e2030 con 0x7ff6b0108920 2024-04-25T15:59:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:00 smithi063 ceph-mon[32413]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:59:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:00 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1178613418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:59:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:00 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2021630262' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:00 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2556787375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T15:59:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:00 smithi186 ceph-mon[33258]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:59:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:00 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1178613418' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:59:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:00 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2021630262' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:00 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2556787375' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T15:59:00.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:00.973+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff6b002dca0 con 0x7ff6b0108920 2024-04-25T15:59:01.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.483+0000 7ff6b4983640 1 -- 172.21.15.31:0/233278246 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v72) ==== 138+0+0 (secure 0 0 0) 0x7ff6a80a7dd0 con 0x7ff6b0108920 2024-04-25T15:59:01.484 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' created 2024-04-25T15:59:01.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.487+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff6980afaf0 msgr2=0x7ff6980b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:01.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.487+0000 7ff6b5985640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff6980afaf0 0x7ff6980b2000 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff6a4000c00 tx=0x7ff6a4004810 comp rx=0 tx=0).stop 2024-04-25T15:59:01.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.487+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b0108920 msgr2=0x7ff6b019f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:01.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.487+0000 7ff6b5985640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b0108920 0x7ff6b019f810 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff6a800eb10 tx=0x7ff6a800c2d0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.487+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 shutdown_connections 2024-04-25T15:59:01.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.488+0000 7ff6b5985640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff6980afaf0 0x7ff6980b2000 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.488+0000 7ff6b5985640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff6b0108920 0x7ff6b019f810 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.488+0000 7ff6b5985640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6b0103c20 0x7ff6b0198880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.488+0000 7ff6b5985640 1 --2- 172.21.15.31:0/233278246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6b01029e0 0x7ff6b0198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.488+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 >> 172.21.15.31:0/233278246 conn(0x7ff6b00fdbd0 msgr2=0x7ff6b0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:01.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.489+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 shutdown_connections 2024-04-25T15:59:01.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.489+0000 7ff6b5985640 1 -- 172.21.15.31:0/233278246 wait complete. 2024-04-25T15:59:01.503 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-25T15:59:01.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.622+0000 7f821f784640 1 -- 172.21.15.31:0/1821242318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218103c20 msgr2=0x7f8218104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:01.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.622+0000 7f821f784640 1 --2- 172.21.15.31:0/1821242318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218103c20 0x7f8218104030 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f8208009900 tx=0x7f820802a410 comp rx=0 tx=0).stop 2024-04-25T15:59:01.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.622+0000 7f821f784640 1 -- 172.21.15.31:0/1821242318 shutdown_connections 2024-04-25T15:59:01.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.622+0000 7f821f784640 1 --2- 172.21.15.31:0/1821242318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8218108920 0x7f821810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.622+0000 7f821f784640 1 --2- 172.21.15.31:0/1821242318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218103c20 0x7f8218104030 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.622+0000 7f821f784640 1 --2- 172.21.15.31:0/1821242318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82181029e0 0x7f82181036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.622+0000 7f821f784640 1 -- 172.21.15.31:0/1821242318 >> 172.21.15.31:0/1821242318 conn(0x7f82180fdbd0 msgr2=0x7f8218100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.622+0000 7f821f784640 1 -- 172.21.15.31:0/1821242318 shutdown_connections 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.622+0000 7f821f784640 1 -- 172.21.15.31:0/1821242318 wait complete. 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821f784640 1 Processor -- start 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821f784640 1 -- start start 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821f784640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82181029e0 0x7f8218075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821f784640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8218103c20 0x7f82181a1400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821f784640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218108920 0x7f82181a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821f784640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f821810e4e0 con 0x7f8218108920 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821f784640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f821810e360 con 0x7f82181029e0 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821f784640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f821810e660 con 0x7f8218103c20 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821dd21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218108920 0x7f82181a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821dd21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218108920 0x7f82181a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55278/0 (socket says 172.21.15.31:55278) 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821dd21640 1 -- 172.21.15.31:0/2621654914 learned_addr learned my addr 172.21.15.31:0/2621654914 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:01.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.623+0000 7f821d520640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82181029e0 0x7f8218075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f821cd1f640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8218103c20 0x7f82181a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f821dd21640 1 -- 172.21.15.31:0/2621654914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8218103c20 msgr2=0x7f82181a1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f821dd21640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8218103c20 0x7f82181a1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f821dd21640 1 -- 172.21.15.31:0/2621654914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82181029e0 msgr2=0x7f8218075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f821dd21640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82181029e0 0x7f8218075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f821dd21640 1 -- 172.21.15.31:0/2621654914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82181a3f90 con 0x7f8218108920 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f821dd21640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218108920 0x7f82181a3950 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f821400d950 tx=0x7f821400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f820e7fc640 1 -- 172.21.15.31:0/2621654914 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8214017030 con 0x7f8218108920 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82181a4280 con 0x7f8218108920 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f820e7fc640 1 -- 172.21.15.31:0/2621654914 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8214004020 con 0x7f8218108920 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f820e7fc640 1 -- 172.21.15.31:0/2621654914 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8214003b50 con 0x7f8218108920 2024-04-25T15:59:01.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.624+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82181a4780 con 0x7f8218108920 2024-04-25T15:59:01.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.625+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8218107730 con 0x7f8218108920 2024-04-25T15:59:01.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.627+0000 7f820e7fc640 1 -- 172.21.15.31:0/2621654914 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f821401a070 con 0x7f8218108920 2024-04-25T15:59:01.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.628+0000 7f820e7fc640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82040afaf0 0x7f82040b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:01.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.628+0000 7f821d520640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82040afaf0 0x7f82040b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:01.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.628+0000 7f820e7fc640 1 -- 172.21.15.31:0/2621654914 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7497+0+0 (secure 0 0 0) 0x7f82140de310 con 0x7f8218108920 2024-04-25T15:59:01.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.628+0000 7f821d520640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82040afaf0 0x7f82040b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f82181035e0 tx=0x7f8210009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:01.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.632+0000 7f820e7fc640 1 -- 172.21.15.31:0/2621654914 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8214021070 con 0x7f8218108920 2024-04-25T15:59:01.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2864675154' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/233278246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:59:01.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:01.824+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f821802db50 con 0x7f8218108920 2024-04-25T15:59:01.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2864675154' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/233278246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:59:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2864675154' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-25T15:59:01.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/233278246' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-25T15:59:01.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:59:01.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:59:01.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:01.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T15:59:02.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.493+0000 7f820e7fc640 1 -- 172.21.15.31:0/2621654914 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v73) ==== 170+0+0 (secure 0 0 0) 0x7f82140a7220 con 0x7f8218108920 2024-04-25T15:59:02.494 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' created 2024-04-25T15:59:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82040afaf0 msgr2=0x7f82040b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82040afaf0 0x7f82040b2000 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f82181035e0 tx=0x7f8210009500 comp rx=0 tx=0).stop 2024-04-25T15:59:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218108920 msgr2=0x7f82181a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218108920 0x7f82181a3950 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f821400d950 tx=0x7f821400de10 comp rx=0 tx=0).stop 2024-04-25T15:59:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 shutdown_connections 2024-04-25T15:59:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82040afaf0 0x7f82040b2000 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8218108920 0x7f82181a3950 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8218103c20 0x7f82181a1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 --2- 172.21.15.31:0/2621654914 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f82181029e0 0x7f8218075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 >> 172.21.15.31:0/2621654914 conn(0x7f82180fdbd0 msgr2=0x7f8218104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 shutdown_connections 2024-04-25T15:59:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.496+0000 7f821f784640 1 -- 172.21.15.31:0/2621654914 wait complete. 2024-04-25T15:59:02.521 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-25T15:59:02.535 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:02 smithi063 ceph-mon[32413]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:59:02.535 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:02 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/233278246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T15:59:02.535 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:02 smithi063 ceph-mon[32413]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T15:59:02.535 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:02 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2621654914' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T15:59:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.637+0000 7f4fe3c15640 1 -- 172.21.15.31:0/654664946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc102170 msgr2=0x7f4fdc102e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.637+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/654664946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc102170 0x7f4fdc102e70 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009900 tx=0x7f4fd802a410 comp rx=0 tx=0).stop 2024-04-25T15:59:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.637+0000 7f4fe3c15640 1 -- 172.21.15.31:0/654664946 shutdown_connections 2024-04-25T15:59:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.637+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/654664946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fdc1080b0 0x7f4fdc10a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:02.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.637+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/654664946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fdc1033b0 0x7f4fdc1037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.637+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/654664946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc102170 0x7f4fdc102e70 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.637+0000 7f4fe3c15640 1 -- 172.21.15.31:0/654664946 >> 172.21.15.31:0/654664946 conn(0x7f4fdc0fd340 msgr2=0x7f4fdc0ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.637+0000 7f4fe3c15640 1 -- 172.21.15.31:0/654664946 shutdown_connections 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.638+0000 7f4fe3c15640 1 -- 172.21.15.31:0/654664946 wait complete. 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.638+0000 7f4fe3c15640 1 Processor -- start 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.638+0000 7f4fe3c15640 1 -- start start 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.638+0000 7f4fe3c15640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fdc102170 0x7f4fdc19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.638+0000 7f4fe3c15640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc1033b0 0x7f4fdc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.638+0000 7f4fe3c15640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fdc1080b0 0x7f4fdc1a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.638+0000 7f4fe3c15640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc10d450 con 0x7f4fdc1033b0 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.638+0000 7f4fe3c15640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc10d2d0 con 0x7f4fdc102170 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.638+0000 7f4fe3c15640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4fdc10d5d0 con 0x7f4fdc1080b0 2024-04-25T15:59:02.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe11b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc1033b0 0x7f4fdc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe11b0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc1033b0 0x7f4fdc19a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55306/0 (socket says 172.21.15.31:55306) 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe11b0640 1 -- 172.21.15.31:0/1972446771 learned_addr learned my addr 172.21.15.31:0/1972446771 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe21b2640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fdc1080b0 0x7f4fdc1a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe19b1640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fdc102170 0x7f4fdc19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe11b0640 1 -- 172.21.15.31:0/1972446771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fdc1080b0 msgr2=0x7f4fdc1a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe11b0640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fdc1080b0 0x7f4fdc1a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe11b0640 1 -- 172.21.15.31:0/1972446771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fdc102170 msgr2=0x7f4fdc19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe11b0640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fdc102170 0x7f4fdc19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe11b0640 1 -- 172.21.15.31:0/1972446771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fdc1a1dc0 con 0x7f4fdc1033b0 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe21b2640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fdc1080b0 0x7f4fdc1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe11b0640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc1033b0 0x7f4fdc19a7c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00e9f0 tx=0x7f4fcc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fd2ffd640 1 -- 172.21.15.31:0/1972446771 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc00cd80 con 0x7f4fdc1033b0 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fd2ffd640 1 -- 172.21.15.31:0/1972446771 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fcc0054d0 con 0x7f4fdc1033b0 2024-04-25T15:59:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fdc1a20b0 con 0x7f4fdc1033b0 2024-04-25T15:59:02.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fd2ffd640 1 -- 172.21.15.31:0/1972446771 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fcc010690 con 0x7f4fdc1033b0 2024-04-25T15:59:02.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.639+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fdc1a25b0 con 0x7f4fdc1033b0 2024-04-25T15:59:02.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.641+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4fdc1a28d0 con 0x7f4fdc1033b0 2024-04-25T15:59:02.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.643+0000 7f4fd2ffd640 1 -- 172.21.15.31:0/1972446771 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4fcc004e50 con 0x7f4fdc1033b0 2024-04-25T15:59:02.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.643+0000 7f4fd2ffd640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4fc80afaf0 0x7f4fc80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:02.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.643+0000 7f4fe19b1640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4fc80afaf0 0x7f4fc80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:02.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.643+0000 7f4fd2ffd640 1 -- 172.21.15.31:0/1972446771 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7879+0+0 (secure 0 0 0) 0x7f4fcc0df090 con 0x7f4fdc1033b0 2024-04-25T15:59:02.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.644+0000 7f4fe19b1640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4fc80afaf0 0x7f4fc80b2000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4fd802aa00 tx=0x7f4fd80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:02.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.647+0000 7f4fd2ffd640 1 -- 172.21.15.31:0/1972446771 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4fcc053ff0 con 0x7f4fdc1033b0 2024-04-25T15:59:02.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:02 smithi031 ceph-mon[29140]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:59:02.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:02 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/233278246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T15:59:02.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:02 smithi031 ceph-mon[29140]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T15:59:02.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:02 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2621654914' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T15:59:02.703 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:02 smithi186 ceph-mon[33258]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:59:02.703 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:02 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/233278246' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-25T15:59:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:02 smithi186 ceph-mon[33258]: osdmap e72: 12 total, 12 up, 12 in 2024-04-25T15:59:02.704 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:02 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2621654914' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-25T15:59:02.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:02.904+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f4fdc1a2aa0 con 0x7f4fdc1033b0 2024-04-25T15:59:03.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.494+0000 7f4fd2ffd640 1 -- 172.21.15.31:0/1972446771 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f4fcc0a7e20 con 0x7f4fdc1033b0 2024-04-25T15:59:03.495 INFO:teuthology.orchestra.run.smithi031.stderr:new fs with metadata pool 2 and data pool 3 2024-04-25T15:59:03.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.496+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4fc80afaf0 msgr2=0x7f4fc80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4fc80afaf0 0x7f4fc80b2000 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f4fd802aa00 tx=0x7f4fd80314a0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc1033b0 msgr2=0x7f4fdc19a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc1033b0 0x7f4fdc19a7c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4fcc00e9f0 tx=0x7f4fcc00edf0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 shutdown_connections 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4fc80afaf0 0x7f4fc80b2000 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fdc1080b0 0x7f4fdc1a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4fdc1033b0 0x7f4fdc19a7c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 --2- 172.21.15.31:0/1972446771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fdc102170 0x7f4fdc19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 >> 172.21.15.31:0/1972446771 conn(0x7f4fdc0fd340 msgr2=0x7f4fdc109c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 shutdown_connections 2024-04-25T15:59:03.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.497+0000 7f4fe3c15640 1 -- 172.21.15.31:0/1972446771 wait complete. 2024-04-25T15:59:03.512 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:59:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.623+0000 7f2542867640 1 -- 172.21.15.31:0/3910879023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c103c20 msgr2=0x7f253c104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.623+0000 7f2542867640 1 --2- 172.21.15.31:0/3910879023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c103c20 0x7f253c104030 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2530009900 tx=0x7f253002a410 comp rx=0 tx=0).stop 2024-04-25T15:59:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.623+0000 7f2542867640 1 -- 172.21.15.31:0/3910879023 shutdown_connections 2024-04-25T15:59:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.623+0000 7f2542867640 1 --2- 172.21.15.31:0/3910879023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c108920 0x7f253c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.623+0000 7f2542867640 1 --2- 172.21.15.31:0/3910879023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c103c20 0x7f253c104030 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.623+0000 7f2542867640 1 --2- 172.21.15.31:0/3910879023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f253c1029e0 0x7f253c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.623+0000 7f2542867640 1 -- 172.21.15.31:0/3910879023 >> 172.21.15.31:0/3910879023 conn(0x7f253c0fdbd0 msgr2=0x7f253c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.623+0000 7f2542867640 1 -- 172.21.15.31:0/3910879023 shutdown_connections 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.623+0000 7f2542867640 1 -- 172.21.15.31:0/3910879023 wait complete. 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.624+0000 7f2542867640 1 Processor -- start 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.624+0000 7f2542867640 1 -- start start 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.624+0000 7f2542867640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c1029e0 0x7f253c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.624+0000 7f2542867640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f253c103c20 0x7f253c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.624+0000 7f2542867640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c108920 0x7f253c1a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.624+0000 7f2542867640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f253c10e500 con 0x7f253c108920 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.624+0000 7f2542867640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f253c10e380 con 0x7f253c103c20 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.624+0000 7f2542867640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f253c10e680 con 0x7f253c1029e0 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2540e04640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c108920 0x7f253c1a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2540e04640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c108920 0x7f253c1a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55324/0 (socket says 172.21.15.31:55324) 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f253bfff640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c1029e0 0x7f253c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2540e04640 1 -- 172.21.15.31:0/2517812687 learned_addr learned my addr 172.21.15.31:0/2517812687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f253b7fe640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f253c103c20 0x7f253c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2540e04640 1 -- 172.21.15.31:0/2517812687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c1029e0 msgr2=0x7f253c19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2540e04640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c1029e0 0x7f253c19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2540e04640 1 -- 172.21.15.31:0/2517812687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f253c103c20 msgr2=0x7f253c19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2540e04640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f253c103c20 0x7f253c19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2540e04640 1 -- 172.21.15.31:0/2517812687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f253c1a1dd0 con 0x7f253c108920 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f253b7fe640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f253c103c20 0x7f253c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2540e04640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c108920 0x7f253c1a1760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f253400eb10 tx=0x7f253400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f25397fa640 1 -- 172.21.15.31:0/2517812687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534004b20 con 0x7f253c108920 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f25397fa640 1 -- 172.21.15.31:0/2517812687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2534004cc0 con 0x7f253c108920 2024-04-25T15:59:03.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253c1a20c0 con 0x7f253c108920 2024-04-25T15:59:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f25397fa640 1 -- 172.21.15.31:0/2517812687 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2534010670 con 0x7f253c108920 2024-04-25T15:59:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.625+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f253c1a25c0 con 0x7f253c108920 2024-04-25T15:59:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.627+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f253c1a2880 con 0x7f253c108920 2024-04-25T15:59:03.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.628+0000 7f25397fa640 1 -- 172.21.15.31:0/2517812687 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2534005020 con 0x7f253c108920 2024-04-25T15:59:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.629+0000 7f25397fa640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25240afb70 0x7f25240b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.629+0000 7f253bfff640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25240afb70 0x7f25240b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.629+0000 7f25397fa640 1 -- 172.21.15.31:0/2517812687 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7979+0+0 (secure 0 0 0) 0x7f25340df1e0 con 0x7f253c108920 2024-04-25T15:59:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.629+0000 7f253bfff640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25240afb70 0x7f25240b2080 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f253c1035e0 tx=0x7f252c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:03.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.633+0000 7f25397fa640 1 -- 172.21.15.31:0/2517812687 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f25340a7e70 con 0x7f253c108920 2024-04-25T15:59:03.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.822+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f253c1a2a50 con 0x7f253c108920 2024-04-25T15:59:03.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.824+0000 7f25397fa640 1 -- 172.21.15.31:0/2517812687 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v74) ==== 74+0+22196 (secure 0 0 0) 0x7f25340a7770 con 0x7f253c108920 2024-04-25T15:59:03.826 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:03.826 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":74,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:02.905464+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25240afb70 msgr2=0x7f25240b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25240afb70 0x7f25240b2080 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f253c1035e0 tx=0x7f252c005f90 comp rx=0 tx=0).stop 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c108920 msgr2=0x7f253c1a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c108920 0x7f253c1a1760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f253400eb10 tx=0x7f253400c2d0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 shutdown_connections 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25240afb70 0x7f25240b2080 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f253c108920 0x7f253c1a1760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f253c103c20 0x7f253c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 --2- 172.21.15.31:0/2517812687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f253c1029e0 0x7f253c19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 >> 172.21.15.31:0/2517812687 conn(0x7f253c0fdbd0 msgr2=0x7f253c109f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 shutdown_connections 2024-04-25T15:59:03.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.826+0000 7f2542867640 1 -- 172.21.15.31:0/2517812687 wait complete. 2024-04-25T15:59:03.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2621654914' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T15:59:03.838 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1972446771' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:59:03.839 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.841 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:59:03.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2621654914' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T15:59:03.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T15:59:03.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1972446771' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T15:59:03.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T15:59:03.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T15:59:03.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:59:03.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:59:03.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2621654914' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-25T15:59:03.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: osdmap e73: 12 total, 12 up, 12 in 2024-04-25T15:59:03.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1972446771' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: osdmap e74: 12 total, 12 up, 12 in 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T15:59:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T15:59:03.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.951+0000 7fc902ae7640 1 -- 172.21.15.31:0/302587632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 msgr2=0x7fc8fc10be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.951+0000 7fc902ae7640 1 --2- 172.21.15.31:0/302587632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 0x7fc8fc10be60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009ca0 tx=0x7fc8f402a8c0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.951+0000 7fc902ae7640 1 -- 172.21.15.31:0/302587632 shutdown_connections 2024-04-25T15:59:03.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.951+0000 7fc902ae7640 1 --2- 172.21.15.31:0/302587632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 0x7fc8fc10be60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.951+0000 7fc902ae7640 1 --2- 172.21.15.31:0/302587632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8fc104d40 0x7fc8fc105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.951+0000 7fc902ae7640 1 --2- 172.21.15.31:0/302587632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8fc103b00 0x7fc8fc104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.951+0000 7fc902ae7640 1 -- 172.21.15.31:0/302587632 >> 172.21.15.31:0/302587632 conn(0x7fc8fc0fed10 msgr2=0x7fc8fc1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.951+0000 7fc902ae7640 1 -- 172.21.15.31:0/302587632 shutdown_connections 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.951+0000 7fc902ae7640 1 -- 172.21.15.31:0/302587632 wait complete. 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc902ae7640 1 Processor -- start 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc902ae7640 1 -- start start 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc902ae7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8fc103b00 0x7fc8fc19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc902ae7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8fc104d40 0x7fc8fc19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc902ae7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 0x7fc8fc1a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc902ae7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc8fc10f680 con 0x7fc8fc109a40 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc902ae7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc8fc10f500 con 0x7fc8fc103b00 2024-04-25T15:59:03.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc902ae7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc8fc10f800 con 0x7fc8fc104d40 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc901084640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 0x7fc8fc1a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc901084640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 0x7fc8fc1a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39438/0 (socket says 172.21.15.31:39438) 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc901084640 1 -- 172.21.15.31:0/3513197808 learned_addr learned my addr 172.21.15.31:0/3513197808 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc8fbfff640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8fc104d40 0x7fc8fc19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.952+0000 7fc900883640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8fc103b00 0x7fc8fc19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc901084640 1 -- 172.21.15.31:0/3513197808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8fc104d40 msgr2=0x7fc8fc19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc901084640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8fc104d40 0x7fc8fc19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc901084640 1 -- 172.21.15.31:0/3513197808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8fc103b00 msgr2=0x7fc8fc19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc901084640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8fc103b00 0x7fc8fc19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc901084640 1 -- 172.21.15.31:0/3513197808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8fc1a1d70 con 0x7fc8fc109a40 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc901084640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 0x7fc8fc1a1700 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009430 tx=0x7fc8f40316f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc8f9ffb640 1 -- 172.21.15.31:0/3513197808 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f4003b70 con 0x7fc8fc109a40 2024-04-25T15:59:03.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc8f9ffb640 1 -- 172.21.15.31:0/3513197808 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8f4003d10 con 0x7fc8fc109a40 2024-04-25T15:59:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8fc1a2000 con 0x7fc8fc109a40 2024-04-25T15:59:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc8f9ffb640 1 -- 172.21.15.31:0/3513197808 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8f402fb40 con 0x7fc8fc109a40 2024-04-25T15:59:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.953+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8fc1a25a0 con 0x7fc8fc109a40 2024-04-25T15:59:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.954+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8fc108140 con 0x7fc8fc109a40 2024-04-25T15:59:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.956+0000 7fc8f9ffb640 1 -- 172.21.15.31:0/3513197808 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc8f4004610 con 0x7fc8fc109a40 2024-04-25T15:59:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.956+0000 7fc8f9ffb640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc8ec0afb70 0x7fc8ec0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.956+0000 7fc900883640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc8ec0afb70 0x7fc8ec0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:03.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.956+0000 7fc8f9ffb640 1 -- 172.21.15.31:0/3513197808 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7979+0+0 (secure 0 0 0) 0x7fc8f41000c0 con 0x7fc8fc109a40 2024-04-25T15:59:03.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.957+0000 7fc900883640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc8ec0afb70 0x7fc8ec0b2080 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc104700 tx=0x7fc8f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:03.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:03.960+0000 7fc8f9ffb640 1 -- 172.21.15.31:0/3513197808 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc8f40c8d50 con 0x7fc8fc109a40 2024-04-25T15:59:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.152+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc8fc0637a0 con 0x7fc8fc109a40 2024-04-25T15:59:04.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.154+0000 7fc8f9ffb640 1 -- 172.21.15.31:0/3513197808 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+22183 (secure 0 0 0) 0x7fc8f40c8650 con 0x7fc8fc109a40 2024-04-25T15:59:04.156 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:04.156 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":75,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:03.977416+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:59:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc8ec0afb70 msgr2=0x7fc8ec0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc8ec0afb70 0x7fc8ec0b2080 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc104700 tx=0x7fc8f0005f90 comp rx=0 tx=0).stop 2024-04-25T15:59:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 msgr2=0x7fc8fc1a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 0x7fc8fc1a1700 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc8f4009430 tx=0x7fc8f40316f0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 shutdown_connections 2024-04-25T15:59:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc8ec0afb70 0x7fc8ec0b2080 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8fc109a40 0x7fc8fc1a1700 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8fc104d40 0x7fc8fc19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 --2- 172.21.15.31:0/3513197808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8fc103b00 0x7fc8fc19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 >> 172.21.15.31:0/3513197808 conn(0x7fc8fc0fed10 msgr2=0x7fc8fc102100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:04.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 shutdown_connections 2024-04-25T15:59:04.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.156+0000 7fc902ae7640 1 -- 172.21.15.31:0/3513197808 wait complete. 2024-04-25T15:59:04.170 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-25T15:59:04.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.286+0000 7f0019be3640 1 -- 172.21.15.31:0/3655382927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 msgr2=0x7f0014105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:04.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.286+0000 7f0019be3640 1 --2- 172.21.15.31:0/3655382927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 0x7f0014105140 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0008009900 tx=0x7f000802a410 comp rx=0 tx=0).stop 2024-04-25T15:59:04.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.286+0000 7f0019be3640 1 -- 172.21.15.31:0/3655382927 shutdown_connections 2024-04-25T15:59:04.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.286+0000 7f0019be3640 1 --2- 172.21.15.31:0/3655382927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014109a30 0x7f001410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.286+0000 7f0019be3640 1 --2- 172.21.15.31:0/3655382927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 0x7f0014105140 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.286+0000 7f0019be3640 1 --2- 172.21.15.31:0/3655382927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0014103af0 0x7f00141047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.286+0000 7f0019be3640 1 -- 172.21.15.31:0/3655382927 >> 172.21.15.31:0/3655382927 conn(0x7f00140fece0 msgr2=0x7f0014101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.286+0000 7f0019be3640 1 -- 172.21.15.31:0/3655382927 shutdown_connections 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.286+0000 7f0019be3640 1 -- 172.21.15.31:0/3655382927 wait complete. 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.287+0000 7f0019be3640 1 Processor -- start 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.287+0000 7f0019be3640 1 -- start start 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.287+0000 7f0019be3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103af0 0x7f001419a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.287+0000 7f0019be3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 0x7f001419a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.287+0000 7f0019be3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0014109a30 0x7f00141a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.287+0000 7f0019be3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f001410f640 con 0x7f0014104d30 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.287+0000 7f0019be3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f001410f4c0 con 0x7f0014103af0 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.287+0000 7f0019be3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f001410f7c0 con 0x7f0014109a30 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.287+0000 7f0012ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 0x7f001419a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0012ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 0x7f001419a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39454/0 (socket says 172.21.15.31:39454) 2024-04-25T15:59:04.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0012ffd640 1 -- 172.21.15.31:0/4226202285 learned_addr learned my addr 172.21.15.31:0/4226202285 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0013fff640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0014109a30 0x7f00141a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f00137fe640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103af0 0x7f001419a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0012ffd640 1 -- 172.21.15.31:0/4226202285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0014109a30 msgr2=0x7f00141a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0012ffd640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0014109a30 0x7f00141a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0012ffd640 1 -- 172.21.15.31:0/4226202285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103af0 msgr2=0x7f001419a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0012ffd640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103af0 0x7f001419a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0012ffd640 1 -- 172.21.15.31:0/4226202285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00141a1d60 con 0x7f0014104d30 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0012ffd640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 0x7f001419a760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0008031440 tx=0x7f0008003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0010ff9640 1 -- 172.21.15.31:0/4226202285 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f000803f070 con 0x7f0014104d30 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0010ff9640 1 -- 172.21.15.31:0/4226202285 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0008031c50 con 0x7f0014104d30 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00141a1ff0 con 0x7f0014104d30 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0010ff9640 1 -- 172.21.15.31:0/4226202285 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0008035030 con 0x7f0014104d30 2024-04-25T15:59:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.288+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00141a2400 con 0x7f0014104d30 2024-04-25T15:59:04.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.290+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0014108840 con 0x7f0014104d30 2024-04-25T15:59:04.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.291+0000 7f0010ff9640 1 -- 172.21.15.31:0/4226202285 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f0008042070 con 0x7f0014104d30 2024-04-25T15:59:04.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.292+0000 7f0010ff9640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efffc0afaf0 0x7efffc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:04.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.292+0000 7f00137fe640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efffc0afaf0 0x7efffc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:04.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.292+0000 7f0010ff9640 1 -- 172.21.15.31:0/4226202285 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f00080ffa10 con 0x7f0014104d30 2024-04-25T15:59:04.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.293+0000 7f00137fe640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efffc0afaf0 0x7efffc0b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f00141046f0 tx=0x7f000400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:04.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.296+0000 7f0010ff9640 1 -- 172.21.15.31:0/4226202285 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f00080c80f0 con 0x7f0014104d30 2024-04-25T15:59:04.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:04.526+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f001402db50 con 0x7f0014104d30 2024-04-25T15:59:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:04 smithi063 ceph-mon[32413]: pgmap v281: 97 pgs: 28 creating+peering, 59 unknown, 10 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:59:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:04 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1972446771' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T15:59:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:04 smithi063 ceph-mon[32413]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T15:59:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:04 smithi063 ceph-mon[32413]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T15:59:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:04 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2517812687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:04 smithi063 ceph-mon[32413]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T15:59:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:04 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3513197808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:04 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] up:active 2024-04-25T15:59:04.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:04 smithi063 ceph-mon[32413]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T15:59:04.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:04 smithi186 ceph-mon[33258]: pgmap v281: 97 pgs: 28 creating+peering, 59 unknown, 10 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:59:04.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:04 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1972446771' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T15:59:04.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:04 smithi186 ceph-mon[33258]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T15:59:04.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:04 smithi186 ceph-mon[33258]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T15:59:04.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:04 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2517812687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:04.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:04 smithi186 ceph-mon[33258]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T15:59:04.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:04 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3513197808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:04.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:04 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] up:active 2024-04-25T15:59:04.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:04 smithi186 ceph-mon[33258]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T15:59:04.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:04 smithi031 ceph-mon[29140]: pgmap v281: 97 pgs: 28 creating+peering, 59 unknown, 10 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-25T15:59:04.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:04 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1972446771' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-25T15:59:04.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:04 smithi031 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-25T15:59:04.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:04 smithi031 ceph-mon[29140]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-25T15:59:04.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:04 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2517812687' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:04.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:04 smithi031 ceph-mon[29140]: osdmap e75: 12 total, 12 up, 12 in 2024-04-25T15:59:04.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:04 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3513197808' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:04.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:04 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] up:active 2024-04-25T15:59:04.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:04 smithi031 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T15:59:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.512+0000 7f0010ff9640 1 -- 172.21.15.31:0/4226202285 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v16) ==== 118+0+0 (secure 0 0 0) 0x7f00080cc3f0 con 0x7f0014104d30 2024-04-25T15:59:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efffc0afaf0 msgr2=0x7efffc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efffc0afaf0 0x7efffc0b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f00141046f0 tx=0x7f000400b390 comp rx=0 tx=0).stop 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 msgr2=0x7f001419a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 0x7f001419a760 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0008031440 tx=0x7f0008003c10 comp rx=0 tx=0).stop 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 shutdown_connections 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efffc0afaf0 0x7efffc0b2000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0014109a30 0x7f00141a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0014104d30 0x7f001419a760 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 --2- 172.21.15.31:0/4226202285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0014103af0 0x7f001419a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.515+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 >> 172.21.15.31:0/4226202285 conn(0x7f00140fece0 msgr2=0x7f0014105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.516+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 shutdown_connections 2024-04-25T15:59:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.516+0000 7f0019be3640 1 -- 172.21.15.31:0/4226202285 wait complete. 2024-04-25T15:59:05.531 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-25T15:59:05.532 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-25T15:59:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.652+0000 7fcb3bbee640 1 -- 172.21.15.31:0/2651278316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 msgr2=0x7fcb3410be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.652+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/2651278316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 0x7fcb3410be60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fcb30009ca0 tx=0x7fcb3002a8c0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.652+0000 7fcb3bbee640 1 -- 172.21.15.31:0/2651278316 shutdown_connections 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.652+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/2651278316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 0x7fcb3410be60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.652+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/2651278316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb34104d40 0x7fcb34105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.652+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/2651278316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb34103b00 0x7fcb34104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.652+0000 7fcb3bbee640 1 -- 172.21.15.31:0/2651278316 >> 172.21.15.31:0/2651278316 conn(0x7fcb340fecf0 msgr2=0x7fcb341011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.652+0000 7fcb3bbee640 1 -- 172.21.15.31:0/2651278316 shutdown_connections 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.652+0000 7fcb3bbee640 1 -- 172.21.15.31:0/2651278316 wait complete. 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3bbee640 1 Processor -- start 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3bbee640 1 -- start start 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3bbee640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb34103b00 0x7fcb3419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3bbee640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb34104d40 0x7fcb3419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3bbee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 0x7fcb341a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3bbee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3410f630 con 0x7fcb34109a40 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3bbee640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3410f4b0 con 0x7fcb34104d40 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3bbee640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb3410f7b0 con 0x7fcb34103b00 2024-04-25T15:59:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3a18b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 0x7fcb341a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3a18b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 0x7fcb341a16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39472/0 (socket says 172.21.15.31:39472) 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3a18b640 1 -- 172.21.15.31:0/3985311506 learned_addr learned my addr 172.21.15.31:0/3985311506 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3998a640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb34103b00 0x7fcb3419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb39189640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb34104d40 0x7fcb3419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3a18b640 1 -- 172.21.15.31:0/3985311506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb34103b00 msgr2=0x7fcb3419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3a18b640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb34103b00 0x7fcb3419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3a18b640 1 -- 172.21.15.31:0/3985311506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb34104d40 msgr2=0x7fcb3419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3a18b640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb34104d40 0x7fcb3419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb3a18b640 1 -- 172.21.15.31:0/3985311506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb341a1d10 con 0x7fcb34109a40 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.653+0000 7fcb39189640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb34104d40 0x7fcb3419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.654+0000 7fcb3a18b640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 0x7fcb341a16a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcb300097b0 tx=0x7fcb30003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.654+0000 7fcb2affd640 1 -- 172.21.15.31:0/3985311506 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb30004240 con 0x7fcb34109a40 2024-04-25T15:59:05.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.654+0000 7fcb2affd640 1 -- 172.21.15.31:0/3985311506 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb3002f380 con 0x7fcb34109a40 2024-04-25T15:59:05.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.654+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb341a1fa0 con 0x7fcb34109a40 2024-04-25T15:59:05.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.654+0000 7fcb2affd640 1 -- 172.21.15.31:0/3985311506 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb3002fc30 con 0x7fcb34109a40 2024-04-25T15:59:05.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.654+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb341a2400 con 0x7fcb34109a40 2024-04-25T15:59:05.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.655+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb3406afa0 con 0x7fcb34109a40 2024-04-25T15:59:05.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.656+0000 7fcb2affd640 1 -- 172.21.15.31:0/3985311506 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fcb300043e0 con 0x7fcb34109a40 2024-04-25T15:59:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.657+0000 7fcb2affd640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb200afaf0 0x7fcb200b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.657+0000 7fcb3998a640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb200afaf0 0x7fcb200b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.657+0000 7fcb2affd640 1 -- 172.21.15.31:0/3985311506 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fcb30100420 con 0x7fcb34109a40 2024-04-25T15:59:05.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.657+0000 7fcb3998a640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb200afaf0 0x7fcb200b2000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c005fd0 tx=0x7fcb2c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.660+0000 7fcb2affd640 1 -- 172.21.15.31:0/3985311506 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcb300c90b0 con 0x7fcb34109a40 2024-04-25T15:59:05.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:05 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4226202285' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T15:59:05.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:05.886+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7fcb3402dbd0 con 0x7fcb34109a40 2024-04-25T15:59:05.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:05 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4226202285' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T15:59:05.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:05 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4226202285' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-25T15:59:06.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.514+0000 7fcb2affd640 1 -- 172.21.15.31:0/3985311506 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v17) ==== 105+0+0 (secure 0 0 0) 0x7fcb300c89b0 con 0x7fcb34109a40 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb200afaf0 msgr2=0x7fcb200b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb200afaf0 0x7fcb200b2000 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c005fd0 tx=0x7fcb2c00b390 comp rx=0 tx=0).stop 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 msgr2=0x7fcb341a16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 0x7fcb341a16a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcb300097b0 tx=0x7fcb30003c20 comp rx=0 tx=0).stop 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 shutdown_connections 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb200afaf0 0x7fcb200b2000 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb34109a40 0x7fcb341a16a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb34104d40 0x7fcb3419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 --2- 172.21.15.31:0/3985311506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb34103b00 0x7fcb3419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:06.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 >> 172.21.15.31:0/3985311506 conn(0x7fcb340fecf0 msgr2=0x7fcb34105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 shutdown_connections 2024-04-25T15:59:06.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.517+0000 7fcb3bbee640 1 -- 172.21.15.31:0/3985311506 wait complete. 2024-04-25T15:59:06.535 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-25T15:59:06.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.654+0000 7f012dd1f640 1 -- 172.21.15.31:0/660706536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 msgr2=0x7f0128105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:06.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.654+0000 7f012dd1f640 1 --2- 172.21.15.31:0/660706536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 0x7f0128105140 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f011c009900 tx=0x7f011c02a410 comp rx=0 tx=0).stop 2024-04-25T15:59:06.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.654+0000 7f012dd1f640 1 -- 172.21.15.31:0/660706536 shutdown_connections 2024-04-25T15:59:06.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.654+0000 7f012dd1f640 1 --2- 172.21.15.31:0/660706536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128109a30 0x7f012810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:06.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.654+0000 7f012dd1f640 1 --2- 172.21.15.31:0/660706536 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 0x7f0128105140 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:06.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.654+0000 7f012dd1f640 1 --2- 172.21.15.31:0/660706536 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0128103af0 0x7f01281047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:06.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.654+0000 7f012dd1f640 1 -- 172.21.15.31:0/660706536 >> 172.21.15.31:0/660706536 conn(0x7f01280fece0 msgr2=0x7f0128101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:06.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.655+0000 7f012dd1f640 1 -- 172.21.15.31:0/660706536 shutdown_connections 2024-04-25T15:59:06.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.655+0000 7f012dd1f640 1 -- 172.21.15.31:0/660706536 wait complete. 2024-04-25T15:59:06.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.655+0000 7f012dd1f640 1 Processor -- start 2024-04-25T15:59:06.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.655+0000 7f012dd1f640 1 -- start start 2024-04-25T15:59:06.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f012dd1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128103af0 0x7f0128077960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:06.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f012dd1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 0x7f0128076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:06.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f012dd1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0128109a30 0x7f0128076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:06.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f012dd1f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f012810f610 con 0x7f0128104d30 2024-04-25T15:59:06.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f012dd1f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f012810f490 con 0x7f0128103af0 2024-04-25T15:59:06.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f012dd1f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f012810f790 con 0x7f0128109a30 2024-04-25T15:59:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0126ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 0x7f0128076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0126ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 0x7f0128076000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39484/0 (socket says 172.21.15.31:39484) 2024-04-25T15:59:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0126ffd640 1 -- 172.21.15.31:0/2550727677 learned_addr learned my addr 172.21.15.31:0/2550727677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f01277fe640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128103af0 0x7f0128077960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0127fff640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0128109a30 0x7f0128076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0126ffd640 1 -- 172.21.15.31:0/2550727677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0128109a30 msgr2=0x7f0128076540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0126ffd640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0128109a30 0x7f0128076540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:06.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0126ffd640 1 -- 172.21.15.31:0/2550727677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128103af0 msgr2=0x7f0128077960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:06.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0126ffd640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128103af0 0x7f0128077960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:06.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0126ffd640 1 -- 172.21.15.31:0/2550727677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0128076be0 con 0x7f0128104d30 2024-04-25T15:59:06.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0126ffd640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 0x7f0128076000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f011c006010 tx=0x7f011c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:06.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0124ff9640 1 -- 172.21.15.31:0/2550727677 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011c03f030 con 0x7f0128104d30 2024-04-25T15:59:06.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01281a4300 con 0x7f0128104d30 2024-04-25T15:59:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0124ff9640 1 -- 172.21.15.31:0/2550727677 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f011c003cb0 con 0x7f0128104d30 2024-04-25T15:59:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.656+0000 7f0124ff9640 1 -- 172.21.15.31:0/2550727677 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f011c035070 con 0x7f0128104d30 2024-04-25T15:59:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.657+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01281a46b0 con 0x7f0128104d30 2024-04-25T15:59:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.657+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0128108840 con 0x7f0128104d30 2024-04-25T15:59:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.659+0000 7f0124ff9640 1 -- 172.21.15.31:0/2550727677 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f011c042030 con 0x7f0128104d30 2024-04-25T15:59:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.659+0000 7f0124ff9640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f01100afaf0 0x7f01100b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.660+0000 7f0124ff9640 1 -- 172.21.15.31:0/2550727677 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f011c0ff950 con 0x7f0128104d30 2024-04-25T15:59:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.660+0000 7f01277fe640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f01100afaf0 0x7f01100b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.660+0000 7f01277fe640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f01100afaf0 0x7f01100b2000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f01180058c0 tx=0x7f011800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:06.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.663+0000 7f0124ff9640 1 -- 172.21.15.31:0/2550727677 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f011c0c85e0 con 0x7f0128104d30 2024-04-25T15:59:06.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:06 smithi063 ceph-mon[32413]: pgmap v283: 97 pgs: 28 creating+peering, 37 unknown, 32 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-04-25T15:59:06.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:06 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4226202285' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T15:59:06.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:06 smithi063 ceph-mon[32413]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T15:59:06.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:06 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3985311506' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T15:59:06.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:06.890+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f012802db50 con 0x7f0128104d30 2024-04-25T15:59:06.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:06 smithi186 ceph-mon[33258]: pgmap v283: 97 pgs: 28 creating+peering, 37 unknown, 32 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-04-25T15:59:06.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:06 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4226202285' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T15:59:06.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:06 smithi186 ceph-mon[33258]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T15:59:06.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:06 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3985311506' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T15:59:06.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:06 smithi031 ceph-mon[29140]: pgmap v283: 97 pgs: 28 creating+peering, 37 unknown, 32 active+clean; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 1 op/s 2024-04-25T15:59:06.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:06 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4226202285' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-25T15:59:06.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:06 smithi031 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T15:59:06.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:06 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3985311506' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-25T15:59:07.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.530+0000 7f0124ff9640 1 -- 172.21.15.31:0/2550727677 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v19) ==== 121+0+0 (secure 0 0 0) 0x7f011c0c7ee0 con 0x7f0128104d30 2024-04-25T15:59:07.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.533+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f01100afaf0 msgr2=0x7f01100b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.533+0000 7f012dd1f640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f01100afaf0 0x7f01100b2000 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f01180058c0 tx=0x7f011800a2d0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.533+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 msgr2=0x7f0128076000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.533+0000 7f012dd1f640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 0x7f0128076000 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f011c006010 tx=0x7f011c0043b0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.533+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 shutdown_connections 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.533+0000 7f012dd1f640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f01100afaf0 0x7f01100b2000 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.533+0000 7f012dd1f640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0128109a30 0x7f0128076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.533+0000 7f012dd1f640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0128104d30 0x7f0128076000 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.533+0000 7f012dd1f640 1 --2- 172.21.15.31:0/2550727677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0128103af0 0x7f0128077960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.534+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 >> 172.21.15.31:0/2550727677 conn(0x7f01280fece0 msgr2=0x7f0128105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.534+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 shutdown_connections 2024-04-25T15:59:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.534+0000 7f012dd1f640 1 -- 172.21.15.31:0/2550727677 wait complete. 2024-04-25T15:59:07.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-25T15:59:07.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 -- 172.21.15.31:0/1056500761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0104d40 msgr2=0x7f6ba0105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:07.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 --2- 172.21.15.31:0/1056500761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0104d40 0x7f6ba0105150 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009900 tx=0x7f6b9002a410 comp rx=0 tx=0).stop 2024-04-25T15:59:07.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 -- 172.21.15.31:0/1056500761 shutdown_connections 2024-04-25T15:59:07.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 --2- 172.21.15.31:0/1056500761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba0109a40 0x7f6ba010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 --2- 172.21.15.31:0/1056500761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0104d40 0x7f6ba0105150 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 --2- 172.21.15.31:0/1056500761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ba0103b00 0x7f6ba0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 -- 172.21.15.31:0/1056500761 >> 172.21.15.31:0/1056500761 conn(0x7f6ba00fed10 msgr2=0x7f6ba01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 -- 172.21.15.31:0/1056500761 shutdown_connections 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 -- 172.21.15.31:0/1056500761 wait complete. 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.672+0000 7f6ba8300640 1 Processor -- start 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba8300640 1 -- start start 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba8300640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0103b00 0x7f6ba019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba8300640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba0104d40 0x7f6ba019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba8300640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ba0109a40 0x7f6ba01a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba8300640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba010f620 con 0x7f6ba0103b00 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba8300640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba010f4a0 con 0x7f6ba0104d40 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba8300640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6ba010f7a0 con 0x7f6ba0109a40 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba609c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0103b00 0x7f6ba019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba609c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0103b00 0x7f6ba019a2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39508/0 (socket says 172.21.15.31:39508) 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba609c640 1 -- 172.21.15.31:0/866492808 learned_addr learned my addr 172.21.15.31:0/866492808 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba589b640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba0104d40 0x7f6ba019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba689d640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ba0109a40 0x7f6ba01a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba609c640 1 -- 172.21.15.31:0/866492808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ba0109a40 msgr2=0x7f6ba01a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba609c640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ba0109a40 0x7f6ba01a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba609c640 1 -- 172.21.15.31:0/866492808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba0104d40 msgr2=0x7f6ba019a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba609c640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba0104d40 0x7f6ba019a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba609c640 1 -- 172.21.15.31:0/866492808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ba01a1dc0 con 0x7f6ba0103b00 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.673+0000 7f6ba609c640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0103b00 0x7f6ba019a2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00e9f0 tx=0x7f6b9c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.674+0000 7f6b977fe640 1 -- 172.21.15.31:0/866492808 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c00ccc0 con 0x7f6ba0103b00 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.674+0000 7f6b977fe640 1 -- 172.21.15.31:0/866492808 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b9c0035f0 con 0x7f6ba0103b00 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.674+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ba01a20b0 con 0x7f6ba0103b00 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.674+0000 7f6b977fe640 1 -- 172.21.15.31:0/866492808 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c00ce60 con 0x7f6ba0103b00 2024-04-25T15:59:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.674+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ba01a25b0 con 0x7f6ba0103b00 2024-04-25T15:59:07.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.674+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ba0108850 con 0x7f6ba0103b00 2024-04-25T15:59:07.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.677+0000 7f6b977fe640 1 -- 172.21.15.31:0/866492808 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6b9c003aa0 con 0x7f6ba0103b00 2024-04-25T15:59:07.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.678+0000 7f6b977fe640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b8c0afaf0 0x7f6b8c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:07.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.678+0000 7f6ba589b640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b8c0afaf0 0x7f6b8c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:07.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.678+0000 7f6b977fe640 1 -- 172.21.15.31:0/866492808 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f6b9c0def20 con 0x7f6ba0103b00 2024-04-25T15:59:07.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.678+0000 7f6ba589b640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b8c0afaf0 0x7f6b8c0b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009230 tx=0x7f6b90031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:07.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.681+0000 7f6b977fe640 1 -- 172.21.15.31:0/866492808 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b9c0a7c50 con 0x7f6ba0103b00 2024-04-25T15:59:07.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:07 smithi063 ceph-mon[32413]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T15:59:07.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:07 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3985311506' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T15:59:07.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:07 smithi063 ceph-mon[32413]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T15:59:07.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:07 smithi063 ceph-mon[32413]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T15:59:07.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:07 smithi063 ceph-mon[32413]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T15:59:07.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:07 smithi063 ceph-mon[32413]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T15:59:07.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:07 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2550727677' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T15:59:07.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:07 smithi186 ceph-mon[33258]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T15:59:07.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:07 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3985311506' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T15:59:07.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:07 smithi186 ceph-mon[33258]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T15:59:07.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:07 smithi186 ceph-mon[33258]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T15:59:07.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:07 smithi186 ceph-mon[33258]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T15:59:07.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:07 smithi186 ceph-mon[33258]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T15:59:07.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:07 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2550727677' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T15:59:07.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:07.906+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f6ba002db50 con 0x7f6ba0103b00 2024-04-25T15:59:07.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:07 smithi031 ceph-mon[29140]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-25T15:59:07.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:07 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3985311506' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-25T15:59:07.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:07 smithi031 ceph-mon[29140]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-25T15:59:07.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:07 smithi031 ceph-mon[29140]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-25T15:59:07.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:07 smithi031 ceph-mon[29140]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-04-25T15:59:07.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:07 smithi031 ceph-mon[29140]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-04-25T15:59:07.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:07 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2550727677' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-25T15:59:08.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.541+0000 7f6b977fe640 1 -- 172.21.15.31:0/866492808 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v21) ==== 115+0+0 (secure 0 0 0) 0x7f6b9c0a7550 con 0x7f6ba0103b00 2024-04-25T15:59:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b8c0afaf0 msgr2=0x7f6b8c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b8c0afaf0 0x7f6b8c0b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6b90009230 tx=0x7f6b90031400 comp rx=0 tx=0).stop 2024-04-25T15:59:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0103b00 msgr2=0x7f6ba019a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:08.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0103b00 0x7f6ba019a2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c00e9f0 tx=0x7f6b9c00eeb0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 shutdown_connections 2024-04-25T15:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b8c0afaf0 0x7f6b8c0b2000 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ba0109a40 0x7f6ba01a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6ba0104d40 0x7f6ba019a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 --2- 172.21.15.31:0/866492808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6ba0103b00 0x7f6ba019a2f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 >> 172.21.15.31:0/866492808 conn(0x7f6ba00fed10 msgr2=0x7f6ba0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 shutdown_connections 2024-04-25T15:59:08.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.543+0000 7f6ba8300640 1 -- 172.21.15.31:0/866492808 wait complete. 2024-04-25T15:59:08.561 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-25T15:59:08.562 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-25T15:59:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.678+0000 7f1af48d8640 1 -- 172.21.15.31:0/3816112702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0109790 msgr2=0x7f1af010bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.678+0000 7f1af48d8640 1 --2- 172.21.15.31:0/3816112702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0109790 0x7f1af010bc70 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009ca0 tx=0x7f1ae002a8c0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.678+0000 7f1af48d8640 1 -- 172.21.15.31:0/3816112702 shutdown_connections 2024-04-25T15:59:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.678+0000 7f1af48d8640 1 --2- 172.21.15.31:0/3816112702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0109790 0x7f1af010bc70 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.678+0000 7f1af48d8640 1 --2- 172.21.15.31:0/3816112702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1af0106c20 0x7f1af0109100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.678+0000 7f1af48d8640 1 --2- 172.21.15.31:0/3816112702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1af0104230 0x7f1af0106650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.678+0000 7f1af48d8640 1 -- 172.21.15.31:0/3816112702 >> 172.21.15.31:0/3816112702 conn(0x7f1af00fdbd0 msgr2=0x7f1af0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:08.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.678+0000 7f1af48d8640 1 -- 172.21.15.31:0/3816112702 shutdown_connections 2024-04-25T15:59:08.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.678+0000 7f1af48d8640 1 -- 172.21.15.31:0/3816112702 wait complete. 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1af48d8640 1 Processor -- start 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1af48d8640 1 -- start start 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1af48d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1af0104230 0x7f1af019a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1af48d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0106c20 0x7f1af019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1af48d8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1af0109790 0x7f1af01a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1af48d8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1af010f5d0 con 0x7f1af0106c20 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1af48d8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1af010f450 con 0x7f1af0104230 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1af48d8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1af010f750 con 0x7f1af0109790 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aedd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0106c20 0x7f1af019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:08.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aedd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0106c20 0x7f1af019aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39526/0 (socket says 172.21.15.31:39526) 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aedd9b640 1 -- 172.21.15.31:0/940665623 learned_addr learned my addr 172.21.15.31:0/940665623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aee59c640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1af0104230 0x7f1af019a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aeed9d640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1af0109790 0x7f1af01a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aedd9b640 1 -- 172.21.15.31:0/940665623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1af0109790 msgr2=0x7f1af01a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aedd9b640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1af0109790 0x7f1af01a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aedd9b640 1 -- 172.21.15.31:0/940665623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1af0104230 msgr2=0x7f1af019a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aedd9b640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1af0104230 0x7f1af019a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.679+0000 7f1aedd9b640 1 -- 172.21.15.31:0/940665623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1af01a2010 con 0x7f1af0106c20 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.680+0000 7f1aedd9b640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0106c20 0x7f1af019aa10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ae400d770 tx=0x7f1ae400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.680+0000 7f1adf7fe640 1 -- 172.21.15.31:0/940665623 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4017070 con 0x7f1af0106c20 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.680+0000 7f1adf7fe640 1 -- 172.21.15.31:0/940665623 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ae40054d0 con 0x7f1af0106c20 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.680+0000 7f1adf7fe640 1 -- 172.21.15.31:0/940665623 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4004c30 con 0x7f1af0106c20 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.680+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1af01a2300 con 0x7f1af0106c20 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.680+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1af01a2800 con 0x7f1af0106c20 2024-04-25T15:59:08.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.681+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1af006c2a0 con 0x7f1af0106c20 2024-04-25T15:59:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.683+0000 7f1adf7fe640 1 -- 172.21.15.31:0/940665623 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1ae4004dd0 con 0x7f1af0106c20 2024-04-25T15:59:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.683+0000 7f1adf7fe640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ad80afaf0 0x7f1ad80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.683+0000 7f1aee59c640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ad80afaf0 0x7f1ad80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.683+0000 7f1adf7fe640 1 -- 172.21.15.31:0/940665623 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f1ae40de8f0 con 0x7f1af0106c20 2024-04-25T15:59:08.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.684+0000 7f1aee59c640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ad80afaf0 0x7f1ad80b2000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009bd0 tx=0x7f1ae8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:08.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.687+0000 7f1adf7fe640 1 -- 172.21.15.31:0/940665623 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ae40a6fd0 con 0x7f1af0106c20 2024-04-25T15:59:08.869 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: 2024-04-25T15:59:08.397+0000 7ff2c1c23700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-25T15:59:08.869 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: 2024-04-25T15:59:08.397+0000 7ff2c1c23700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-04-25T15:59:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-mon[32413]: pgmap v284: 97 pgs: 13 creating+peering, 84 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 9 op/s 2024-04-25T15:59:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2550727677' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T15:59:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] up:active 2024-04-25T15:59:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-mon[32413]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T15:59:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-mon[32413]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T15:59:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-mon[32413]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T15:59:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-mon[32413]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T15:59:08.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:08 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/866492808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T15:59:08.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:08 smithi186 ceph-mon[33258]: pgmap v284: 97 pgs: 13 creating+peering, 84 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 9 op/s 2024-04-25T15:59:08.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:08 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2550727677' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T15:59:08.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:08 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] up:active 2024-04-25T15:59:08.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:08 smithi186 ceph-mon[33258]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T15:59:08.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:08 smithi186 ceph-mon[33258]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T15:59:08.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:08 smithi186 ceph-mon[33258]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T15:59:08.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:08 smithi186 ceph-mon[33258]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T15:59:08.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:08 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/866492808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T15:59:08.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:08.898+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1af01a2a60 con 0x7f1ad80afaf0 2024-04-25T15:59:08.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-mon[29140]: pgmap v284: 97 pgs: 13 creating+peering, 84 active+clean; 579 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s wr, 9 op/s 2024-04-25T15:59:08.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2550727677' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-25T15:59:08.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] up:active 2024-04-25T15:59:08.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-mon[29140]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-25T15:59:08.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-mon[29140]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-04-25T15:59:08.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-mon[29140]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-04-25T15:59:08.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-mon[29140]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-04-25T15:59:08.907 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/866492808' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-25T15:59:09.199 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:59:08.903+0000 7fdadaa3a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:59:09.200 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:59:08.903+0000 7fdadaa3a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:59:09.200 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:59:08.903+0000 7fdadaa3a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:59:09.200 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:59:08.903+0000 7fdadaa3a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:59:09.200 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 15:59:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T15:59:08.903+0000 7fdadaa3a700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-25T15:59:09.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.261+0000 7f1adf7fe640 1 -- 172.21.15.31:0/940665623 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1af01a2a60 con 0x7f1ad80afaf0 2024-04-25T15:59:09.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ad80afaf0 msgr2=0x7f1ad80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ad80afaf0 0x7f1ad80b2000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009bd0 tx=0x7f1ae8005f90 comp rx=0 tx=0).stop 2024-04-25T15:59:09.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0106c20 msgr2=0x7f1af019aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0106c20 0x7f1af019aa10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1ae400d770 tx=0x7f1ae400dc30 comp rx=0 tx=0).stop 2024-04-25T15:59:09.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 shutdown_connections 2024-04-25T15:59:09.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ad80afaf0 0x7f1ad80b2000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1af0109790 0x7f1af01a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1af0106c20 0x7f1af019aa10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 --2- 172.21.15.31:0/940665623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1af0104230 0x7f1af019a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 >> 172.21.15.31:0/940665623 conn(0x7f1af00fdbd0 msgr2=0x7f1af0105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:09.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 shutdown_connections 2024-04-25T15:59:09.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.263+0000 7f1af48d8640 1 -- 172.21.15.31:0/940665623 wait complete. 2024-04-25T15:59:09.277 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-25T15:59:09.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.392+0000 7fd858b1f640 1 -- 172.21.15.31:0/3690288172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8541029e0 msgr2=0x7fd8541036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.392+0000 7fd858b1f640 1 --2- 172.21.15.31:0/3690288172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8541029e0 0x7fd8541036e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd844009900 tx=0x7fd84402a410 comp rx=0 tx=0).stop 2024-04-25T15:59:09.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.392+0000 7fd858b1f640 1 -- 172.21.15.31:0/3690288172 shutdown_connections 2024-04-25T15:59:09.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.392+0000 7fd858b1f640 1 --2- 172.21.15.31:0/3690288172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd854108920 0x7fd85410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.392+0000 7fd858b1f640 1 --2- 172.21.15.31:0/3690288172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd854103c20 0x7fd854104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.392+0000 7fd858b1f640 1 --2- 172.21.15.31:0/3690288172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8541029e0 0x7fd8541036e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.392+0000 7fd858b1f640 1 -- 172.21.15.31:0/3690288172 >> 172.21.15.31:0/3690288172 conn(0x7fd8540fdbd0 msgr2=0x7fd854100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.392+0000 7fd858b1f640 1 -- 172.21.15.31:0/3690288172 shutdown_connections 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.392+0000 7fd858b1f640 1 -- 172.21.15.31:0/3690288172 wait complete. 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd858b1f640 1 Processor -- start 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd858b1f640 1 -- start start 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd858b1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8541029e0 0x7fd85406a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd858b1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd854103c20 0x7fd85419f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd858b1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd854108920 0x7fd8541a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd858b1f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd85410e540 con 0x7fd854103c20 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd858b1f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd85410e3c0 con 0x7fd8541029e0 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd858b1f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd85410e6c0 con 0x7fd854108920 2024-04-25T15:59:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd851d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd854103c20 0x7fd85419f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd851d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd854103c20 0x7fd85419f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39558/0 (socket says 172.21.15.31:39558) 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd851d9b640 1 -- 172.21.15.31:0/2701614137 learned_addr learned my addr 172.21.15.31:0/2701614137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd85259c640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8541029e0 0x7fd85406a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd852d9d640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd854108920 0x7fd8541a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd851d9b640 1 -- 172.21.15.31:0/2701614137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd854108920 msgr2=0x7fd8541a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd851d9b640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd854108920 0x7fd8541a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd851d9b640 1 -- 172.21.15.31:0/2701614137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8541029e0 msgr2=0x7fd85406a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd851d9b640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8541029e0 0x7fd85406a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd851d9b640 1 -- 172.21.15.31:0/2701614137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8541a20d0 con 0x7fd854103c20 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd852d9d640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd854108920 0x7fd8541a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:59:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.393+0000 7fd851d9b640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd854103c20 0x7fd85419f340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd84800e9f0 tx=0x7fd84800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.394+0000 7fd8437fe640 1 -- 172.21.15.31:0/2701614137 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84800cd80 con 0x7fd854103c20 2024-04-25T15:59:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.394+0000 7fd8437fe640 1 -- 172.21.15.31:0/2701614137 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8480108e0 con 0x7fd854103c20 2024-04-25T15:59:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.394+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8541a23c0 con 0x7fd854103c20 2024-04-25T15:59:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.394+0000 7fd8437fe640 1 -- 172.21.15.31:0/2701614137 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd848018980 con 0x7fd854103c20 2024-04-25T15:59:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.394+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8541a2990 con 0x7fd854103c20 2024-04-25T15:59:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.395+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd854107730 con 0x7fd854103c20 2024-04-25T15:59:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.396+0000 7fd8437fe640 1 -- 172.21.15.31:0/2701614137 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd848010430 con 0x7fd854103c20 2024-04-25T15:59:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.397+0000 7fd8437fe640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd83c0afaf0 0x7fd83c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.397+0000 7fd85259c640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd83c0afaf0 0x7fd83c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.397+0000 7fd8437fe640 1 -- 172.21.15.31:0/2701614137 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fd8480df330 con 0x7fd854103c20 2024-04-25T15:59:09.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.398+0000 7fd85259c640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd83c0afaf0 0x7fd83c0b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd844005ab0 tx=0x7fd844031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:09.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.401+0000 7fd8437fe640 1 -- 172.21.15.31:0/2701614137 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8480a8060 con 0x7fd854103c20 2024-04-25T15:59:09.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.617+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fd85402dd40 con 0x7fd83c0afaf0 2024-04-25T15:59:09.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.642+0000 7fd8437fe640 1 -- 172.21.15.31:0/2701614137 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd85402dd40 con 0x7fd83c0afaf0 2024-04-25T15:59:09.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd83c0afaf0 msgr2=0x7fd83c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd83c0afaf0 0x7fd83c0b2000 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd844005ab0 tx=0x7fd844031400 comp rx=0 tx=0).stop 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd854103c20 msgr2=0x7fd85419f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd854103c20 0x7fd85419f340 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd84800e9f0 tx=0x7fd84800eeb0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 shutdown_connections 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd83c0afaf0 0x7fd83c0b2000 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd854108920 0x7fd8541a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd854103c20 0x7fd85419f340 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 --2- 172.21.15.31:0/2701614137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8541029e0 0x7fd85406a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.644+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 >> 172.21.15.31:0/2701614137 conn(0x7fd8540fdbd0 msgr2=0x7fd854104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.645+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 shutdown_connections 2024-04-25T15:59:09.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.645+0000 7fd858b1f640 1 -- 172.21.15.31:0/2701614137 wait complete. 2024-04-25T15:59:09.658 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-25T15:59:09.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.772+0000 7f5f275ab640 1 -- 172.21.15.31:0/212807100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f201029f0 msgr2=0x7f5f201036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.772+0000 7f5f275ab640 1 --2- 172.21.15.31:0/212807100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f201029f0 0x7f5f201036f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c009900 tx=0x7f5f1c02a410 comp rx=0 tx=0).stop 2024-04-25T15:59:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.772+0000 7f5f275ab640 1 -- 172.21.15.31:0/212807100 shutdown_connections 2024-04-25T15:59:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 --2- 172.21.15.31:0/212807100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f20108930 0x7f5f2010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 --2- 172.21.15.31:0/212807100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f20103c30 0x7f5f20104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 --2- 172.21.15.31:0/212807100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f201029f0 0x7f5f201036f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 -- 172.21.15.31:0/212807100 >> 172.21.15.31:0/212807100 conn(0x7f5f200fdbc0 msgr2=0x7f5f20100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 -- 172.21.15.31:0/212807100 shutdown_connections 2024-04-25T15:59:09.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 -- 172.21.15.31:0/212807100 wait complete. 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 Processor -- start 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 -- start start 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f201029f0 0x7f5f2019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.773+0000 7f5f275ab640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f20103c30 0x7f5f2019a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f275ab640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f20108930 0x7f5f201a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f275ab640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2010e4f0 con 0x7f5f20103c30 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f275ab640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2010e370 con 0x7f5f201029f0 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f275ab640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f2010e670 con 0x7f5f20108930 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f24b46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f20103c30 0x7f5f2019a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f24b46640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f20103c30 0x7f5f2019a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39578/0 (socket says 172.21.15.31:39578) 2024-04-25T15:59:09.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f24b46640 1 -- 172.21.15.31:0/3447324017 learned_addr learned my addr 172.21.15.31:0/3447324017 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f25347640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f201029f0 0x7f5f2019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f25b48640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f20108930 0x7f5f201a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f24b46640 1 -- 172.21.15.31:0/3447324017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f20108930 msgr2=0x7f5f201a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f24b46640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f20108930 0x7f5f201a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f24b46640 1 -- 172.21.15.31:0/3447324017 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f201029f0 msgr2=0x7f5f2019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f24b46640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f201029f0 0x7f5f2019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f24b46640 1 -- 172.21.15.31:0/3447324017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f201a1d90 con 0x7f5f20103c30 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f24b46640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f20103c30 0x7f5f2019a790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000d770 tx=0x7f5f1000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f167fc640 1 -- 172.21.15.31:0/3447324017 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f10017070 con 0x7f5f20103c30 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f201a2080 con 0x7f5f20103c30 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f167fc640 1 -- 172.21.15.31:0/3447324017 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f100054d0 con 0x7f5f20103c30 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f167fc640 1 -- 172.21.15.31:0/3447324017 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f10004c30 con 0x7f5f20103c30 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.774+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f201a2580 con 0x7f5f20103c30 2024-04-25T15:59:09.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.775+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f201a27e0 con 0x7f5f20103c30 2024-04-25T15:59:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.777+0000 7f5f167fc640 1 -- 172.21.15.31:0/3447324017 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5f10004e10 con 0x7f5f20103c30 2024-04-25T15:59:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.777+0000 7f5f167fc640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f0c0afaf0 0x7f5f0c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.778+0000 7f5f25347640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f0c0afaf0 0x7f5f0c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.778+0000 7f5f167fc640 1 -- 172.21.15.31:0/3447324017 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f5f100de4e0 con 0x7f5f20103c30 2024-04-25T15:59:09.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.778+0000 7f5f25347640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f0c0afaf0 0x7f5f0c0b2000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c005ab0 tx=0x7f5f1c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:09.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.782+0000 7f5f167fc640 1 -- 172.21.15.31:0/3447324017 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5f100a6bc0 con 0x7f5f20103c30 2024-04-25T15:59:09.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/866492808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T15:59:09.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] up:active 2024-04-25T15:59:09.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T15:59:09.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-25T15:59:09.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-04-25T15:59:09.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-25T15:59:09.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:09.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] up:active 2024-04-25T15:59:09.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T15:59:09.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T15:59:09.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: Cluster is now healthy 2024-04-25T15:59:09.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-04-25T15:59:09.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-04-25T15:59:09.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:09 smithi063 ceph-mon[32413]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-25T15:59:09.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/866492808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T15:59:09.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] up:active 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] up:active 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: Cluster is now healthy 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-04-25T15:59:09.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:09 smithi186 ceph-mon[33258]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/866492808' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] up:active 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: from='client.14727 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] up:active 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: Cluster is now healthy 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-04-25T15:59:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-04-25T15:59:09.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:09 smithi031 ceph-mon[29140]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-04-25T15:59:10.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:09.999+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f5f2010a610 con 0x7f5f0c0afaf0 2024-04-25T15:59:10.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.004+0000 7f5f167fc640 1 -- 172.21.15.31:0/3447324017 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f5f2010a610 con 0x7f5f0c0afaf0 2024-04-25T15:59:10.006 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/qa/sv_0/9a1f324c-caed-44b8-a551-dcce57bab72d 2024-04-25T15:59:10.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f0c0afaf0 msgr2=0x7f5f0c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f0c0afaf0 0x7f5f0c0b2000 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5f1c005ab0 tx=0x7f5f1c031400 comp rx=0 tx=0).stop 2024-04-25T15:59:10.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f20103c30 msgr2=0x7f5f2019a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f20103c30 0x7f5f2019a790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000d770 tx=0x7f5f1000dc30 comp rx=0 tx=0).stop 2024-04-25T15:59:10.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 shutdown_connections 2024-04-25T15:59:10.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f0c0afaf0 0x7f5f0c0b2000 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f20108930 0x7f5f201a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f20103c30 0x7f5f2019a790 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 --2- 172.21.15.31:0/3447324017 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f201029f0 0x7f5f2019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.006+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 >> 172.21.15.31:0/3447324017 conn(0x7f5f200fdbc0 msgr2=0x7f5f20109f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:10.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.007+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 shutdown_connections 2024-04-25T15:59:10.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.007+0000 7f5f275ab640 1 -- 172.21.15.31:0/3447324017 wait complete. 2024-04-25T15:59:10.020 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-25T15:59:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.135+0000 7fc39f261640 1 -- 172.21.15.31:0/2492764589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 msgr2=0x7fc398101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.135+0000 7fc39f261640 1 --2- 172.21.15.31:0/2492764589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 0x7fc398101c40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc38c009900 tx=0x7fc38c02a410 comp rx=0 tx=0).stop 2024-04-25T15:59:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.135+0000 7fc39f261640 1 -- 172.21.15.31:0/2492764589 shutdown_connections 2024-04-25T15:59:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.135+0000 7fc39f261640 1 --2- 172.21.15.31:0/2492764589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3981097a0 0x7fc39810bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.135+0000 7fc39f261640 1 --2- 172.21.15.31:0/2492764589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc398106df0 0x7fc398109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.135+0000 7fc39f261640 1 --2- 172.21.15.31:0/2492764589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 0x7fc398101c40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.135+0000 7fc39f261640 1 -- 172.21.15.31:0/2492764589 >> 172.21.15.31:0/2492764589 conn(0x7fc3980fcf20 msgr2=0x7fc3980ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.135+0000 7fc39f261640 1 -- 172.21.15.31:0/2492764589 shutdown_connections 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.135+0000 7fc39f261640 1 -- 172.21.15.31:0/2492764589 wait complete. 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39f261640 1 Processor -- start 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39f261640 1 -- start start 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39f261640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 0x7fc39819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39f261640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc398106df0 0x7fc39819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39f261640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3981097a0 0x7fc3981a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39f261640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc39810f5c0 con 0x7fc398101830 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39f261640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc39810f440 con 0x7fc3981097a0 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39f261640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc39810f740 con 0x7fc398106df0 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39cffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 0x7fc39819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39cffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 0x7fc39819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39592/0 (socket says 172.21.15.31:39592) 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39cffd640 1 -- 172.21.15.31:0/304234252 learned_addr learned my addr 172.21.15.31:0/304234252 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39d7fe640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3981097a0 0x7fc3981a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc397fff640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc398106df0 0x7fc39819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39cffd640 1 -- 172.21.15.31:0/304234252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc398106df0 msgr2=0x7fc39819a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39cffd640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc398106df0 0x7fc39819a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39cffd640 1 -- 172.21.15.31:0/304234252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3981097a0 msgr2=0x7fc3981a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39cffd640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3981097a0 0x7fc3981a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.136+0000 7fc39cffd640 1 -- 172.21.15.31:0/304234252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3981a1ce0 con 0x7fc398101830 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.137+0000 7fc39cffd640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 0x7fc39819a250 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc38c006010 tx=0x7fc38c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.137+0000 7fc395ffb640 1 -- 172.21.15.31:0/304234252 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c03f030 con 0x7fc398101830 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.137+0000 7fc395ffb640 1 -- 172.21.15.31:0/304234252 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc38c003aa0 con 0x7fc398101830 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.137+0000 7fc395ffb640 1 -- 172.21.15.31:0/304234252 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c035070 con 0x7fc398101830 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.137+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3981a1f70 con 0x7fc398101830 2024-04-25T15:59:10.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.137+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3981a2380 con 0x7fc398101830 2024-04-25T15:59:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.138+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3980fea40 con 0x7fc398101830 2024-04-25T15:59:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.139+0000 7fc395ffb640 1 -- 172.21.15.31:0/304234252 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc38c042030 con 0x7fc398101830 2024-04-25T15:59:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.140+0000 7fc395ffb640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3880afaf0 0x7fc3880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.140+0000 7fc395ffb640 1 -- 172.21.15.31:0/304234252 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fc38c0ff600 con 0x7fc398101830 2024-04-25T15:59:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.140+0000 7fc397fff640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3880afaf0 0x7fc3880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.141+0000 7fc397fff640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3880afaf0 0x7fc3880b2000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc39819b2f0 tx=0x7fc384005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:10.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.144+0000 7fc395ffb640 1 -- 172.21.15.31:0/304234252 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc38c0c8290 con 0x7fc398101830 2024-04-25T15:59:10.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.262+0000 7fc395ffb640 1 -- 172.21.15.31:0/304234252 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fc38c003c50 con 0x7fc398101830 2024-04-25T15:59:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.368+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fc398063070 con 0x7fc3880afaf0 2024-04-25T15:59:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.396+0000 7fc395ffb640 1 -- 172.21.15.31:0/304234252 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc398063070 con 0x7fc3880afaf0 2024-04-25T15:59:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3880afaf0 msgr2=0x7fc3880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3880afaf0 0x7fc3880b2000 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc39819b2f0 tx=0x7fc384005f90 comp rx=0 tx=0).stop 2024-04-25T15:59:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 msgr2=0x7fc39819a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 0x7fc39819a250 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc38c006010 tx=0x7fc38c004270 comp rx=0 tx=0).stop 2024-04-25T15:59:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 shutdown_connections 2024-04-25T15:59:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3880afaf0 0x7fc3880b2000 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3981097a0 0x7fc3981a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc398106df0 0x7fc39819a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 --2- 172.21.15.31:0/304234252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc398101830 0x7fc39819a250 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 >> 172.21.15.31:0/304234252 conn(0x7fc3980fcf20 msgr2=0x7fc398107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 shutdown_connections 2024-04-25T15:59:10.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.398+0000 7fc39f261640 1 -- 172.21.15.31:0/304234252 wait complete. 2024-04-25T15:59:10.411 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-25T15:59:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 -- 172.21.15.31:0/3220668406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc1029e0 msgr2=0x7fe2bc1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 --2- 172.21.15.31:0/3220668406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc1029e0 0x7fe2bc1036e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac009900 tx=0x7fe2ac02a410 comp rx=0 tx=0).stop 2024-04-25T15:59:10.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 -- 172.21.15.31:0/3220668406 shutdown_connections 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 --2- 172.21.15.31:0/3220668406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2bc108920 0x7fe2bc10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 --2- 172.21.15.31:0/3220668406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2bc103c20 0x7fe2bc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 --2- 172.21.15.31:0/3220668406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc1029e0 0x7fe2bc1036e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 -- 172.21.15.31:0/3220668406 >> 172.21.15.31:0/3220668406 conn(0x7fe2bc0fdbd0 msgr2=0x7fe2bc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 -- 172.21.15.31:0/3220668406 shutdown_connections 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 -- 172.21.15.31:0/3220668406 wait complete. 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.528+0000 7fe2c439d640 1 Processor -- start 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c439d640 1 -- start start 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c439d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2bc1029e0 0x7fe2bc19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c439d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2bc103c20 0x7fe2bc19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c439d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc108920 0x7fe2bc1a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c439d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe2bc10f610 con 0x7fe2bc108920 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c439d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe2bc10f490 con 0x7fe2bc1029e0 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c439d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe2bc10f790 con 0x7fe2bc103c20 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c293a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc108920 0x7fe2bc1a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c2139640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2bc1029e0 0x7fe2bc19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c293a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc108920 0x7fe2bc1a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39608/0 (socket says 172.21.15.31:39608) 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c293a640 1 -- 172.21.15.31:0/1093732613 learned_addr learned my addr 172.21.15.31:0/1093732613 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c1938640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2bc103c20 0x7fe2bc19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c293a640 1 -- 172.21.15.31:0/1093732613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2bc103c20 msgr2=0x7fe2bc19aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c293a640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2bc103c20 0x7fe2bc19aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c293a640 1 -- 172.21.15.31:0/1093732613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2bc1029e0 msgr2=0x7fe2bc19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c293a640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2bc1029e0 0x7fe2bc19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c293a640 1 -- 172.21.15.31:0/1093732613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2bc1a20b0 con 0x7fe2bc108920 2024-04-25T15:59:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c2139640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2bc1029e0 0x7fe2bc19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:59:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2c293a640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc108920 0x7fe2bc1a1a40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400eb10 tx=0x7fe2b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.529+0000 7fe2b37fe640 1 -- 172.21.15.31:0/1093732613 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4004b20 con 0x7fe2bc108920 2024-04-25T15:59:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.530+0000 7fe2b37fe640 1 -- 172.21.15.31:0/1093732613 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2b4004cc0 con 0x7fe2bc108920 2024-04-25T15:59:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.530+0000 7fe2b37fe640 1 -- 172.21.15.31:0/1093732613 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2b4010670 con 0x7fe2bc108920 2024-04-25T15:59:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.530+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2bc1a23a0 con 0x7fe2bc108920 2024-04-25T15:59:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.530+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2bc1a28a0 con 0x7fe2bc108920 2024-04-25T15:59:10.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.531+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2bc107730 con 0x7fe2bc108920 2024-04-25T15:59:10.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.532+0000 7fe2b37fe640 1 -- 172.21.15.31:0/1093732613 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fe2b4005020 con 0x7fe2bc108920 2024-04-25T15:59:10.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.533+0000 7fe2b37fe640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2a80afb80 0x7fe2a80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.533+0000 7fe2c2139640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2a80afb80 0x7fe2a80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.533+0000 7fe2b37fe640 1 -- 172.21.15.31:0/1093732613 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fe2b40df3a0 con 0x7fe2bc108920 2024-04-25T15:59:10.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.533+0000 7fe2c2139640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2a80afb80 0x7fe2a80b2090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac005ab0 tx=0x7fe2ac031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:10.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.537+0000 7fe2b37fe640 1 -- 172.21.15.31:0/1093732613 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe2b40a7ed0 con 0x7fe2bc108920 2024-04-25T15:59:10.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:10 smithi031 ceph-mon[29140]: pgmap v285: 97 pgs: 97 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s wr, 15 op/s 2024-04-25T15:59:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:10 smithi031 ceph-mon[29140]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:10 smithi031 ceph-mon[29140]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:10 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] up:active 2024-04-25T15:59:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:10 smithi031 ceph-mon[29140]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T15:59:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:10 smithi031 ceph-mon[29140]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T15:59:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:10 smithi031 ceph-mon[29140]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T15:59:10.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.756+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fe2bc02dd80 con 0x7fe2a80afb80 2024-04-25T15:59:10.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.761+0000 7fe2b37fe640 1 -- 172.21.15.31:0/1093732613 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fe2bc02dd80 con 0x7fe2a80afb80 2024-04-25T15:59:10.763 INFO:teuthology.orchestra.run.smithi031.stdout:/volumes/qa/sv_1/80cd7e96-3109-49c9-8578-0278f2cf668e 2024-04-25T15:59:10.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2a80afb80 msgr2=0x7fe2a80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2a80afb80 0x7fe2a80b2090 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac005ab0 tx=0x7fe2ac031400 comp rx=0 tx=0).stop 2024-04-25T15:59:10.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc108920 msgr2=0x7fe2bc1a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc108920 0x7fe2bc1a1a40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe2b400eb10 tx=0x7fe2b400c2d0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 shutdown_connections 2024-04-25T15:59:10.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2a80afb80 0x7fe2a80b2090 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe2bc108920 0x7fe2bc1a1a40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2bc103c20 0x7fe2bc19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 --2- 172.21.15.31:0/1093732613 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe2bc1029e0 0x7fe2bc19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 >> 172.21.15.31:0/1093732613 conn(0x7fe2bc0fdbd0 msgr2=0x7fe2bc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:10.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 shutdown_connections 2024-04-25T15:59:10.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.763+0000 7fe2c439d640 1 -- 172.21.15.31:0/1093732613 wait complete. 2024-04-25T15:59:10.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T15:59:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:10 smithi063 ceph-mon[32413]: pgmap v285: 97 pgs: 97 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s wr, 15 op/s 2024-04-25T15:59:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:10 smithi063 ceph-mon[32413]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:10 smithi063 ceph-mon[32413]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:10 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] up:active 2024-04-25T15:59:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:10 smithi063 ceph-mon[32413]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T15:59:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:10 smithi063 ceph-mon[32413]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T15:59:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:10 smithi063 ceph-mon[32413]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 -- 172.21.15.31:0/3162122181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 msgr2=0x7f890010ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 --2- 172.21.15.31:0/3162122181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 0x7f890010ad20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f88f8009ca0 tx=0x7f88f802a8c0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 -- 172.21.15.31:0/3162122181 shutdown_connections 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 --2- 172.21.15.31:0/3162122181 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 0x7f890010ad20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 --2- 172.21.15.31:0/3162122181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8900103c00 0x7f8900104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 --2- 172.21.15.31:0/3162122181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89001029c0 0x7f89001036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 -- 172.21.15.31:0/3162122181 >> 172.21.15.31:0/3162122181 conn(0x7f89000fdbd0 msgr2=0x7f8900100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 -- 172.21.15.31:0/3162122181 shutdown_connections 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 -- 172.21.15.31:0/3162122181 wait complete. 2024-04-25T15:59:10.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.887+0000 7f8905b1c640 1 Processor -- start 2024-04-25T15:59:10.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f8905b1c640 1 -- start start 2024-04-25T15:59:10.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f8905b1c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89001029c0 0x7f890006a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f8905b1c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8900103c00 0x7f890019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f8905b1c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 0x7f89001a14c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f8905b1c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f890010e4f0 con 0x7f8900108900 2024-04-25T15:59:10.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f8905b1c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f890010e370 con 0x7f8900103c00 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f8905b1c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f890010e670 con 0x7f89001029c0 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88fffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 0x7f89001a14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88fffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 0x7f89001a14c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39626/0 (socket says 172.21.15.31:39626) 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88fffff640 1 -- 172.21.15.31:0/274341499 learned_addr learned my addr 172.21.15.31:0/274341499 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88feffd640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8900103c00 0x7f890019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88ff7fe640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89001029c0 0x7f890006a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88fffff640 1 -- 172.21.15.31:0/274341499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89001029c0 msgr2=0x7f890006a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88fffff640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89001029c0 0x7f890006a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88fffff640 1 -- 172.21.15.31:0/274341499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8900103c00 msgr2=0x7f890019ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88fffff640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8900103c00 0x7f890019ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88fffff640 1 -- 172.21.15.31:0/274341499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89001a1ce0 con 0x7f8900108900 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.888+0000 7f88fffff640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 0x7f89001a14c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f88f8004af0 tx=0x7f88f8031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.889+0000 7f88fcff9640 1 -- 172.21.15.31:0/274341499 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8003b70 con 0x7f8900108900 2024-04-25T15:59:10.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.889+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89001a1f70 con 0x7f8900108900 2024-04-25T15:59:10.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.889+0000 7f88fcff9640 1 -- 172.21.15.31:0/274341499 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f88f8003d10 con 0x7f8900108900 2024-04-25T15:59:10.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.889+0000 7f88fcff9640 1 -- 172.21.15.31:0/274341499 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f802fb20 con 0x7f8900108900 2024-04-25T15:59:10.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.889+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89001a2430 con 0x7f8900108900 2024-04-25T15:59:10.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.890+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8900107710 con 0x7f8900108900 2024-04-25T15:59:10.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.892+0000 7f88fcff9640 1 -- 172.21.15.31:0/274341499 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f88f8004580 con 0x7f8900108900 2024-04-25T15:59:10.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.892+0000 7f88fcff9640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f88e80afb80 0x7f88e80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:10.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.892+0000 7f88ff7fe640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f88e80afb80 0x7f88e80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:10.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.892+0000 7f88fcff9640 1 -- 172.21.15.31:0/274341499 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f88f8100350 con 0x7f8900108900 2024-04-25T15:59:10.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.893+0000 7f88ff7fe640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f88e80afb80 0x7f88e80b2090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f89001035c0 tx=0x7f88f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:10.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:10.896+0000 7f88fcff9640 1 -- 172.21.15.31:0/274341499 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88f80c9090 con 0x7f8900108900 2024-04-25T15:59:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:10 smithi186 ceph-mon[33258]: pgmap v285: 97 pgs: 97 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s wr, 15 op/s 2024-04-25T15:59:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:10 smithi186 ceph-mon[33258]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:10 smithi186 ceph-mon[33258]: from='client.14745 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:10 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] up:active 2024-04-25T15:59:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:10 smithi186 ceph-mon[33258]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-04-25T15:59:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:10 smithi186 ceph-mon[33258]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-04-25T15:59:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:10 smithi186 ceph-mon[33258]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T15:59:11.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.127+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f890002db50 con 0x7f8900108900 2024-04-25T15:59:11.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.128+0000 7f88fcff9640 1 -- 172.21.15.31:0/274341499 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11917 (secure 0 0 0) 0x7f88f80c8990 con 0x7f8900108900 2024-04-25T15:59:11.129 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:11.130 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T15:59:10.264996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[14631,24553,24563,24521,24535]},"id":1}]} 2024-04-25T15:59:11.130 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f88e80afb80 msgr2=0x7f88e80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f88e80afb80 0x7f88e80b2090 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f89001035c0 tx=0x7f88f0005f90 comp rx=0 tx=0).stop 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 msgr2=0x7f89001a14c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 0x7f89001a14c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f88f8004af0 tx=0x7f88f8031700 comp rx=0 tx=0).stop 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 shutdown_connections 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f88e80afb80 0x7f88e80b2090 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8900108900 0x7f89001a14c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8900103c00 0x7f890019ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 --2- 172.21.15.31:0/274341499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89001029c0 0x7f890006a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 >> 172.21.15.31:0/274341499 conn(0x7f89000fdbd0 msgr2=0x7f89001048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:11.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 shutdown_connections 2024-04-25T15:59:11.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.130+0000 7f8905b1c640 1 -- 172.21.15.31:0/274341499 wait complete. 2024-04-25T15:59:11.144 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T15:59:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 -- 172.21.15.31:0/3321443360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0108900 msgr2=0x7fa1c010ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 --2- 172.21.15.31:0/3321443360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0108900 0x7fa1c010ad20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa1b8009ca0 tx=0x7fa1b802a8c0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 -- 172.21.15.31:0/3321443360 shutdown_connections 2024-04-25T15:59:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 --2- 172.21.15.31:0/3321443360 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c0108900 0x7fa1c010ad20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 --2- 172.21.15.31:0/3321443360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1c0103c00 0x7fa1c0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 --2- 172.21.15.31:0/3321443360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1c01029c0 0x7fa1c01036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 -- 172.21.15.31:0/3321443360 >> 172.21.15.31:0/3321443360 conn(0x7fa1c00fdbd0 msgr2=0x7fa1c0100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 -- 172.21.15.31:0/3321443360 shutdown_connections 2024-04-25T15:59:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 -- 172.21.15.31:0/3321443360 wait complete. 2024-04-25T15:59:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.254+0000 7fa1c63be640 1 Processor -- start 2024-04-25T15:59:11.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1c63be640 1 -- start start 2024-04-25T15:59:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1c63be640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c01029c0 0x7fa1c019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1c63be640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1c0103c00 0x7fa1c019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1c63be640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1c0108900 0x7fa1c01a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1c63be640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c010f5e0 con 0x7fa1c01029c0 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1c63be640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c010f460 con 0x7fa1c0108900 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1c63be640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c010f760 con 0x7fa1c0103c00 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1bffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c01029c0 0x7fa1c019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1bffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c01029c0 0x7fa1c019a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39638/0 (socket says 172.21.15.31:39638) 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1bffff640 1 -- 172.21.15.31:0/349894771 learned_addr learned my addr 172.21.15.31:0/349894771 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1bf7fe640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1c0103c00 0x7fa1c019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1c495b640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1c0108900 0x7fa1c01a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1bffff640 1 -- 172.21.15.31:0/349894771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1c0103c00 msgr2=0x7fa1c019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1bffff640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1c0103c00 0x7fa1c019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1bffff640 1 -- 172.21.15.31:0/349894771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1c0108900 msgr2=0x7fa1c01a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1bffff640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1c0108900 0x7fa1c01a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1bffff640 1 -- 172.21.15.31:0/349894771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c01a1cf0 con 0x7fa1c01029c0 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.255+0000 7fa1c495b640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1c0108900 0x7fa1c01a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T15:59:11.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.256+0000 7fa1bffff640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c01029c0 0x7fa1c019a1b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa1b000c970 tx=0x7fa1b000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.256+0000 7fa1bd7fa640 1 -- 172.21.15.31:0/349894771 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0004220 con 0x7fa1c01029c0 2024-04-25T15:59:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.256+0000 7fa1bd7fa640 1 -- 172.21.15.31:0/349894771 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1b00043c0 con 0x7fa1c01029c0 2024-04-25T15:59:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.256+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c01a1fe0 con 0x7fa1c01029c0 2024-04-25T15:59:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.256+0000 7fa1bd7fa640 1 -- 172.21.15.31:0/349894771 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0004a00 con 0x7fa1c01029c0 2024-04-25T15:59:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.256+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c01a2440 con 0x7fa1c01029c0 2024-04-25T15:59:11.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.257+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c0107710 con 0x7fa1c01029c0 2024-04-25T15:59:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.259+0000 7fa1bd7fa640 1 -- 172.21.15.31:0/349894771 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa1b00079b0 con 0x7fa1c01029c0 2024-04-25T15:59:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.260+0000 7fa1bd7fa640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1ac0afb80 0x7fa1ac0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.260+0000 7fa1bf7fe640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1ac0afb80 0x7fa1ac0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.260+0000 7fa1bd7fa640 1 -- 172.21.15.31:0/349894771 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fa1b00dedf0 con 0x7fa1c01029c0 2024-04-25T15:59:11.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.260+0000 7fa1bf7fe640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1ac0afb80 0x7fa1ac0b2090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa1c019b250 tx=0x7fa1a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:11.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.264+0000 7fa1bd7fa640 1 -- 172.21.15.31:0/349894771 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1b0014030 con 0x7fa1c01029c0 2024-04-25T15:59:11.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.452+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa1c002dda0 con 0x7fa1c01029c0 2024-04-25T15:59:11.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.453+0000 7fa1bd7fa640 1 -- 172.21.15.31:0/349894771 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+22183 (secure 0 0 0) 0x7fa1b00a72d0 con 0x7fa1c01029c0 2024-04-25T15:59:11.455 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:11.455 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":75,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:03.977416+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:59:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1ac0afb80 msgr2=0x7fa1ac0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1ac0afb80 0x7fa1ac0b2090 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa1c019b250 tx=0x7fa1a8005f90 comp rx=0 tx=0).stop 2024-04-25T15:59:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c01029c0 msgr2=0x7fa1c019a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c01029c0 0x7fa1c019a1b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fa1b000c970 tx=0x7fa1b000ce30 comp rx=0 tx=0).stop 2024-04-25T15:59:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 shutdown_connections 2024-04-25T15:59:11.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1ac0afb80 0x7fa1ac0b2090 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1c0108900 0x7fa1c01a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1c0103c00 0x7fa1c019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 --2- 172.21.15.31:0/349894771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa1c01029c0 0x7fa1c019a1b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 >> 172.21.15.31:0/349894771 conn(0x7fa1c00fdbd0 msgr2=0x7fa1c01048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:11.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 shutdown_connections 2024-04-25T15:59:11.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.455+0000 7fa1c63be640 1 -- 172.21.15.31:0/349894771 wait complete. 2024-04-25T15:59:11.469 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-25T15:59:11.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 -- 172.21.15.31:0/557105569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 msgr2=0x7f80401036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 --2- 172.21.15.31:0/557105569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 0x7f80401036c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8030009900 tx=0x7f803002a410 comp rx=0 tx=0).stop 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 -- 172.21.15.31:0/557105569 shutdown_connections 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 --2- 172.21.15.31:0/557105569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8040108900 0x7f804010ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 --2- 172.21.15.31:0/557105569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8040103c00 0x7f8040104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 --2- 172.21.15.31:0/557105569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 0x7f80401036c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 -- 172.21.15.31:0/557105569 >> 172.21.15.31:0/557105569 conn(0x7f80400fdbd0 msgr2=0x7f8040100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 -- 172.21.15.31:0/557105569 shutdown_connections 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 -- 172.21.15.31:0/557105569 wait complete. 2024-04-25T15:59:11.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.580+0000 7f8045a27640 1 Processor -- start 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f8045a27640 1 -- start start 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f8045a27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 0x7f804019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f8045a27640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8040103c00 0x7f804019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f8045a27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8040108900 0x7f80401a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f8045a27640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f804010f5f0 con 0x7f80401029c0 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f8045a27640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f804010f470 con 0x7f8040103c00 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f8045a27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f804010f770 con 0x7f8040108900 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 0x7f804019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 0x7f804019a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39660/0 (socket says 172.21.15.31:39660) 2024-04-25T15:59:11.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803effd640 1 -- 172.21.15.31:0/2588334115 learned_addr learned my addr 172.21.15.31:0/2588334115 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803e7fc640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8040103c00 0x7f804019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803f7fe640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8040108900 0x7f80401a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803effd640 1 -- 172.21.15.31:0/2588334115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8040108900 msgr2=0x7f80401a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803effd640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8040108900 0x7f80401a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803effd640 1 -- 172.21.15.31:0/2588334115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8040103c00 msgr2=0x7f804019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803effd640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8040103c00 0x7f804019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803effd640 1 -- 172.21.15.31:0/2588334115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80401a1d60 con 0x7f80401029c0 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803e7fc640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8040103c00 0x7f804019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.581+0000 7f803effd640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 0x7f804019a220 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8030006010 tx=0x7f8030031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.582+0000 7f8044a25640 1 -- 172.21.15.31:0/2588334115 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f803003f030 con 0x7f80401029c0 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.582+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80401a1ff0 con 0x7f80401029c0 2024-04-25T15:59:11.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.582+0000 7f8044a25640 1 -- 172.21.15.31:0/2588334115 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8030003aa0 con 0x7f80401029c0 2024-04-25T15:59:11.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.582+0000 7f8044a25640 1 -- 172.21.15.31:0/2588334115 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8030035070 con 0x7f80401029c0 2024-04-25T15:59:11.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.582+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80401a2450 con 0x7f80401029c0 2024-04-25T15:59:11.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.583+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f804006afa0 con 0x7f80401029c0 2024-04-25T15:59:11.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.584+0000 7f8044a25640 1 -- 172.21.15.31:0/2588334115 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f8030042030 con 0x7f80401029c0 2024-04-25T15:59:11.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.585+0000 7f8044a25640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80280afb80 0x7f80280b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.585+0000 7f803e7fc640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80280afb80 0x7f80280b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.585+0000 7f8044a25640 1 -- 172.21.15.31:0/2588334115 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f80300ff7a0 con 0x7f80401029c0 2024-04-25T15:59:11.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.585+0000 7f803e7fc640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80280afb80 0x7f80280b2090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f804019b2c0 tx=0x7f8034005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:11.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.589+0000 7f8044a25640 1 -- 172.21.15.31:0/2588334115 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80300c82d0 con 0x7f80401029c0 2024-04-25T15:59:11.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.770+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8040062fe0 con 0x7f80280afb80 2024-04-25T15:59:11.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.778+0000 7f8044a25640 1 -- 172.21.15.31:0/2588334115 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258916 (secure 0 0 0) 0x7f8040062fe0 con 0x7f80280afb80 2024-04-25T15:59:11.784 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80280afb80 msgr2=0x7f80280b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80280afb80 0x7f80280b2090 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f804019b2c0 tx=0x7f8034005e70 comp rx=0 tx=0).stop 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 msgr2=0x7f804019a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 0x7f804019a220 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8030006010 tx=0x7f8030031a50 comp rx=0 tx=0).stop 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 shutdown_connections 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80280afb80 0x7f80280b2090 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8040108900 0x7f80401a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8040103c00 0x7f804019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 --2- 172.21.15.31:0/2588334115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80401029c0 0x7f804019a220 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.781+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 >> 172.21.15.31:0/2588334115 conn(0x7f80400fdbd0 msgr2=0x7f80401048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:11.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.782+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 shutdown_connections 2024-04-25T15:59:11.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.782+0000 7f8045a27640 1 -- 172.21.15.31:0/2588334115 wait complete. 2024-04-25T15:59:11.794 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:11 smithi031 ceph-mon[29140]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:11.794 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:11 smithi031 ceph-mon[29140]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:11.794 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:11 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/274341499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:59:11.794 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:11 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/349894771' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:11.795 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-04-25T15:59:11.313746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594896,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":241,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354728,"kb_used_data":9716,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124587096,"statfs":{"total":1151940427776,"available":1151577186304,"internally_reserved":0,"allocated":9949184,"data_stored":6389040,"data_compressed":107478,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4528,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.335447"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981726+0000","last_change":"2024-04-25T15:59:02.997678+0000","last_active":"2024-04-25T15:59:03.981726+0000","last_peered":"2024-04-25T15:59:03.981726+0000","last_clean":"2024-04-25T15:59:03.981726+0000","last_became_active":"2024-04-25T15:59:02.997335+0000","last_became_peered":"2024-04-25T15:59:02.997335+0000","last_unstale":"2024-04-25T15:59:03.981726+0000","last_undegraded":"2024-04-25T15:59:03.981726+0000","last_fullsized":"2024-04-25T15:59:03.981726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:33.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279944+0000","last_change":"2024-04-25T15:59:03.001049+0000","last_active":"2024-04-25T15:59:04.279944+0000","last_peered":"2024-04-25T15:59:04.279944+0000","last_clean":"2024-04-25T15:59:04.279944+0000","last_became_active":"2024-04-25T15:59:03.000663+0000","last_became_peered":"2024-04-25T15:59:03.000663+0000","last_unstale":"2024-04-25T15:59:04.279944+0000","last_undegraded":"2024-04-25T15:59:04.279944+0000","last_fullsized":"2024-04-25T15:59:04.279944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:04:36.670519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279577+0000","last_change":"2024-04-25T15:59:03.003861+0000","last_active":"2024-04-25T15:59:04.279577+0000","last_peered":"2024-04-25T15:59:04.279577+0000","last_clean":"2024-04-25T15:59:04.279577+0000","last_became_active":"2024-04-25T15:59:03.003508+0000","last_became_peered":"2024-04-25T15:59:03.003508+0000","last_unstale":"2024-04-25T15:59:04.279577+0000","last_undegraded":"2024-04-25T15:59:04.279577+0000","last_fullsized":"2024-04-25T15:59:04.279577+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:25.679389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.384069+0000","last_change":"2024-04-25T15:59:03.003330+0000","last_active":"2024-04-25T15:59:04.384069+0000","last_peered":"2024-04-25T15:59:04.384069+0000","last_clean":"2024-04-25T15:59:04.384069+0000","last_became_active":"2024-04-25T15:59:03.002872+0000","last_became_peered":"2024-04-25T15:59:03.002872+0000","last_unstale":"2024-04-25T15:59:04.384069+0000","last_undegraded":"2024-04-25T15:59:04.384069+0000","last_fullsized":"2024-04-25T15:59:04.384069+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:02.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383605+0000","last_change":"2024-04-25T15:59:02.997684+0000","last_active":"2024-04-25T15:59:04.383605+0000","last_peered":"2024-04-25T15:59:04.383605+0000","last_clean":"2024-04-25T15:59:04.383605+0000","last_became_active":"2024-04-25T15:59:02.997370+0000","last_became_peered":"2024-04-25T15:59:02.997370+0000","last_unstale":"2024-04-25T15:59:04.383605+0000","last_undegraded":"2024-04-25T15:59:04.383605+0000","last_fullsized":"2024-04-25T15:59:04.383605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:24.349085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.980982+0000","last_change":"2024-04-25T15:59:03.003638+0000","last_active":"2024-04-25T15:59:03.980982+0000","last_peered":"2024-04-25T15:59:03.980982+0000","last_clean":"2024-04-25T15:59:03.980982+0000","last_became_active":"2024-04-25T15:59:03.003299+0000","last_became_peered":"2024-04-25T15:59:03.003299+0000","last_unstale":"2024-04-25T15:59:03.980982+0000","last_undegraded":"2024-04-25T15:59:03.980982+0000","last_fullsized":"2024-04-25T15:59:03.980982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:46.842345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382637+0000","last_change":"2024-04-25T15:59:02.998591+0000","last_active":"2024-04-25T15:59:04.382637+0000","last_peered":"2024-04-25T15:59:04.382637+0000","last_clean":"2024-04-25T15:59:04.382637+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T15:59:04.382637+0000","last_undegraded":"2024-04-25T15:59:04.382637+0000","last_fullsized":"2024-04-25T15:59:04.382637+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.444499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501385+0000","last_change":"2024-04-25T15:59:03.006225+0000","last_active":"2024-04-25T15:59:04.501385+0000","last_peered":"2024-04-25T15:59:04.501385+0000","last_clean":"2024-04-25T15:59:04.501385+0000","last_became_active":"2024-04-25T15:59:03.005883+0000","last_became_peered":"2024-04-25T15:59:03.005883+0000","last_unstale":"2024-04-25T15:59:04.501385+0000","last_undegraded":"2024-04-25T15:59:04.501385+0000","last_fullsized":"2024-04-25T15:59:04.501385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:17.773196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383942+0000","last_change":"2024-04-25T15:59:03.003605+0000","last_active":"2024-04-25T15:59:04.383942+0000","last_peered":"2024-04-25T15:59:04.383942+0000","last_clean":"2024-04-25T15:59:04.383942+0000","last_became_active":"2024-04-25T15:59:03.003271+0000","last_became_peered":"2024-04-25T15:59:03.003271+0000","last_unstale":"2024-04-25T15:59:04.383942+0000","last_undegraded":"2024-04-25T15:59:04.383942+0000","last_fullsized":"2024-04-25T15:59:04.383942+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:28.576407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981256+0000","last_change":"2024-04-25T15:59:03.006378+0000","last_active":"2024-04-25T15:59:03.981256+0000","last_peered":"2024-04-25T15:59:03.981256+0000","last_clean":"2024-04-25T15:59:03.981256+0000","last_became_active":"2024-04-25T15:59:03.006032+0000","last_became_peered":"2024-04-25T15:59:03.006032+0000","last_unstale":"2024-04-25T15:59:03.981256+0000","last_undegraded":"2024-04-25T15:59:03.981256+0000","last_fullsized":"2024-04-25T15:59:03.981256+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:41.473986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383595+0000","last_change":"2024-04-25T15:59:02.999307+0000","last_active":"2024-04-25T15:59:04.383595+0000","last_peered":"2024-04-25T15:59:04.383595+0000","last_clean":"2024-04-25T15:59:04.383595+0000","last_became_active":"2024-04-25T15:59:02.998976+0000","last_became_peered":"2024-04-25T15:59:02.998976+0000","last_unstale":"2024-04-25T15:59:04.383595+0000","last_undegraded":"2024-04-25T15:59:04.383595+0000","last_fullsized":"2024-04-25T15:59:04.383595+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:29.626440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383829+0000","last_change":"2024-04-25T15:59:03.005208+0000","last_active":"2024-04-25T15:59:04.383829+0000","last_peered":"2024-04-25T15:59:04.383829+0000","last_clean":"2024-04-25T15:59:04.383829+0000","last_became_active":"2024-04-25T15:59:03.004869+0000","last_became_peered":"2024-04-25T15:59:03.004869+0000","last_unstale":"2024-04-25T15:59:04.383829+0000","last_undegraded":"2024-04-25T15:59:04.383829+0000","last_fullsized":"2024-04-25T15:59:04.383829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:00:24.040763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382668+0000","last_change":"2024-04-25T15:59:02.999924+0000","last_active":"2024-04-25T15:59:04.382668+0000","last_peered":"2024-04-25T15:59:04.382668+0000","last_clean":"2024-04-25T15:59:04.382668+0000","last_became_active":"2024-04-25T15:59:02.999573+0000","last_became_peered":"2024-04-25T15:59:02.999573+0000","last_unstale":"2024-04-25T15:59:04.382668+0000","last_undegraded":"2024-04-25T15:59:04.382668+0000","last_fullsized":"2024-04-25T15:59:04.382668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:38.759429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279494+0000","last_change":"2024-04-25T15:59:03.002160+0000","last_active":"2024-04-25T15:59:04.279494+0000","last_peered":"2024-04-25T15:59:04.279494+0000","last_clean":"2024-04-25T15:59:04.279494+0000","last_became_active":"2024-04-25T15:59:03.001855+0000","last_became_peered":"2024-04-25T15:59:03.001855+0000","last_unstale":"2024-04-25T15:59:04.279494+0000","last_undegraded":"2024-04-25T15:59:04.279494+0000","last_fullsized":"2024-04-25T15:59:04.279494+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:43.383851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279727+0000","last_change":"2024-04-25T15:59:02.995318+0000","last_active":"2024-04-25T15:59:04.279727+0000","last_peered":"2024-04-25T15:59:04.279727+0000","last_clean":"2024-04-25T15:59:04.279727+0000","last_became_active":"2024-04-25T15:59:02.994956+0000","last_became_peered":"2024-04-25T15:59:02.994956+0000","last_unstale":"2024-04-25T15:59:04.279727+0000","last_undegraded":"2024-04-25T15:59:04.279727+0000","last_fullsized":"2024-04-25T15:59:04.279727+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:59.659400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382480+0000","last_change":"2024-04-25T15:59:02.996485+0000","last_active":"2024-04-25T15:59:04.382480+0000","last_peered":"2024-04-25T15:59:04.382480+0000","last_clean":"2024-04-25T15:59:04.382480+0000","last_became_active":"2024-04-25T15:59:02.996011+0000","last_became_peered":"2024-04-25T15:59:02.996011+0000","last_unstale":"2024-04-25T15:59:04.382480+0000","last_undegraded":"2024-04-25T15:59:04.382480+0000","last_fullsized":"2024-04-25T15:59:04.382480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:56.025254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981833+0000","last_change":"2024-04-25T15:59:02.998455+0000","last_active":"2024-04-25T15:59:03.981833+0000","last_peered":"2024-04-25T15:59:03.981833+0000","last_clean":"2024-04-25T15:59:03.981833+0000","last_became_active":"2024-04-25T15:59:02.998126+0000","last_became_peered":"2024-04-25T15:59:02.998126+0000","last_unstale":"2024-04-25T15:59:03.981833+0000","last_undegraded":"2024-04-25T15:59:03.981833+0000","last_fullsized":"2024-04-25T15:59:03.981833+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:20.342413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.982991+0000","last_change":"2024-04-25T15:59:02.998766+0000","last_active":"2024-04-25T15:59:03.982991+0000","last_peered":"2024-04-25T15:59:03.982991+0000","last_clean":"2024-04-25T15:59:03.982991+0000","last_became_active":"2024-04-25T15:59:02.998415+0000","last_became_peered":"2024-04-25T15:59:02.998415+0000","last_unstale":"2024-04-25T15:59:03.982991+0000","last_undegraded":"2024-04-25T15:59:03.982991+0000","last_fullsized":"2024-04-25T15:59:03.982991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:55:05.717847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383074+0000","last_change":"2024-04-25T15:59:02.994982+0000","last_active":"2024-04-25T15:59:04.383074+0000","last_peered":"2024-04-25T15:59:04.383074+0000","last_clean":"2024-04-25T15:59:04.383074+0000","last_became_active":"2024-04-25T15:59:02.993928+0000","last_became_peered":"2024-04-25T15:59:02.993928+0000","last_unstale":"2024-04-25T15:59:04.383074+0000","last_undegraded":"2024-04-25T15:59:04.383074+0000","last_fullsized":"2024-04-25T15:59:04.383074+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:22:25.346541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'11","reported_seq":26,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:08.560563+0000","last_change":"2024-04-25T15:59:02.512828+0000","last_active":"2024-04-25T15:59:08.560563+0000","last_peered":"2024-04-25T15:59:08.560563+0000","last_clean":"2024-04-25T15:59:08.560563+0000","last_became_active":"2024-04-25T15:59:02.512291+0000","last_became_peered":"2024-04-25T15:59:02.512291+0000","last_unstale":"2024-04-25T15:59:08.560563+0000","last_undegraded":"2024-04-25T15:59:08.560563+0000","last_fullsized":"2024-04-25T15:59:08.560563+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:02.372140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501589+0000","last_change":"2024-04-25T15:59:03.008472+0000","last_active":"2024-04-25T15:59:04.501589+0000","last_peered":"2024-04-25T15:59:04.501589+0000","last_clean":"2024-04-25T15:59:04.501589+0000","last_became_active":"2024-04-25T15:59:03.008108+0000","last_became_peered":"2024-04-25T15:59:03.008108+0000","last_unstale":"2024-04-25T15:59:04.501589+0000","last_undegraded":"2024-04-25T15:59:04.501589+0000","last_fullsized":"2024-04-25T15:59:04.501589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:40.087592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.113722+0000","last_change":"2024-04-25T15:59:02.517846+0000","last_active":"2024-04-25T15:59:06.113722+0000","last_peered":"2024-04-25T15:59:06.113722+0000","last_clean":"2024-04-25T15:59:06.113722+0000","last_became_active":"2024-04-25T15:59:02.517366+0000","last_became_peered":"2024-04-25T15:59:02.517366+0000","last_unstale":"2024-04-25T15:59:06.113722+0000","last_undegraded":"2024-04-25T15:59:06.113722+0000","last_fullsized":"2024-04-25T15:59:06.113722+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:29.620827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279820+0000","last_change":"2024-04-25T15:59:03.005202+0000","last_active":"2024-04-25T15:59:04.279820+0000","last_peered":"2024-04-25T15:59:04.279820+0000","last_clean":"2024-04-25T15:59:04.279820+0000","last_became_active":"2024-04-25T15:59:03.003447+0000","last_became_peered":"2024-04-25T15:59:03.003447+0000","last_unstale":"2024-04-25T15:59:04.279820+0000","last_undegraded":"2024-04-25T15:59:04.279820+0000","last_fullsized":"2024-04-25T15:59:04.279820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.740016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"74'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383447+0000","last_change":"2024-04-25T15:59:02.513701+0000","last_active":"2024-04-25T15:59:04.383447+0000","last_peered":"2024-04-25T15:59:04.383447+0000","last_clean":"2024-04-25T15:59:04.383447+0000","last_became_active":"2024-04-25T15:59:02.513361+0000","last_became_peered":"2024-04-25T15:59:02.513361+0000","last_unstale":"2024-04-25T15:59:04.383447+0000","last_undegraded":"2024-04-25T15:59:04.383447+0000","last_fullsized":"2024-04-25T15:59:04.383447+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:46.113568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383168+0000","last_change":"2024-04-25T15:59:03.004210+0000","last_active":"2024-04-25T15:59:04.383168+0000","last_peered":"2024-04-25T15:59:04.383168+0000","last_clean":"2024-04-25T15:59:04.383168+0000","last_became_active":"2024-04-25T15:59:03.002769+0000","last_became_peered":"2024-04-25T15:59:03.002769+0000","last_unstale":"2024-04-25T15:59:04.383168+0000","last_undegraded":"2024-04-25T15:59:04.383168+0000","last_fullsized":"2024-04-25T15:59:04.383168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:50.608327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"74'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383795+0000","last_change":"2024-04-25T15:59:02.506199+0000","last_active":"2024-04-25T15:59:04.383795+0000","last_peered":"2024-04-25T15:59:04.383795+0000","last_clean":"2024-04-25T15:59:04.383795+0000","last_became_active":"2024-04-25T15:59:02.505847+0000","last_became_peered":"2024-04-25T15:59:02.505847+0000","last_unstale":"2024-04-25T15:59:04.383795+0000","last_undegraded":"2024-04-25T15:59:04.383795+0000","last_fullsized":"2024-04-25T15:59:04.383795+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:35.227112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383483+0000","last_change":"2024-04-25T15:59:03.011367+0000","last_active":"2024-04-25T15:59:04.383483+0000","last_peered":"2024-04-25T15:59:04.383483+0000","last_clean":"2024-04-25T15:59:04.383483+0000","last_became_active":"2024-04-25T15:59:03.011012+0000","last_became_peered":"2024-04-25T15:59:03.011012+0000","last_unstale":"2024-04-25T15:59:04.383483+0000","last_undegraded":"2024-04-25T15:59:04.383483+0000","last_fullsized":"2024-04-25T15:59:04.383483+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:26.434883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'5","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.538819+0000","last_change":"2024-04-25T15:59:02.507763+0000","last_active":"2024-04-25T15:59:06.538819+0000","last_peered":"2024-04-25T15:59:06.538819+0000","last_clean":"2024-04-25T15:59:06.538819+0000","last_became_active":"2024-04-25T15:59:02.507338+0000","last_became_peered":"2024-04-25T15:59:02.507338+0000","last_unstale":"2024-04-25T15:59:06.538819+0000","last_undegraded":"2024-04-25T15:59:06.538819+0000","last_fullsized":"2024-04-25T15:59:06.538819+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:31.710709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383638+0000","last_change":"2024-04-25T15:59:03.002590+0000","last_active":"2024-04-25T15:59:04.383638+0000","last_peered":"2024-04-25T15:59:04.383638+0000","last_clean":"2024-04-25T15:59:04.383638+0000","last_became_active":"2024-04-25T15:59:03.002262+0000","last_became_peered":"2024-04-25T15:59:03.002262+0000","last_unstale":"2024-04-25T15:59:04.383638+0000","last_undegraded":"2024-04-25T15:59:04.383638+0000","last_fullsized":"2024-04-25T15:59:04.383638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:48.251484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"74'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.980758+0000","last_change":"2024-04-25T15:59:02.518853+0000","last_active":"2024-04-25T15:59:03.980758+0000","last_peered":"2024-04-25T15:59:03.980758+0000","last_clean":"2024-04-25T15:59:03.980758+0000","last_became_active":"2024-04-25T15:59:02.518505+0000","last_became_peered":"2024-04-25T15:59:02.518505+0000","last_unstale":"2024-04-25T15:59:03.980758+0000","last_undegraded":"2024-04-25T15:59:03.980758+0000","last_fullsized":"2024-04-25T15:59:03.980758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:01:56.282245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383040+0000","last_change":"2024-04-25T15:59:03.009733+0000","last_active":"2024-04-25T15:59:04.383040+0000","last_peered":"2024-04-25T15:59:04.383040+0000","last_clean":"2024-04-25T15:59:04.383040+0000","last_became_active":"2024-04-25T15:59:03.009251+0000","last_became_peered":"2024-04-25T15:59:03.009251+0000","last_unstale":"2024-04-25T15:59:04.383040+0000","last_undegraded":"2024-04-25T15:59:04.383040+0000","last_fullsized":"2024-04-25T15:59:04.383040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:22.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"75'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:07.552825+0000","last_change":"2024-04-25T15:59:02.512493+0000","last_active":"2024-04-25T15:59:07.552825+0000","last_peered":"2024-04-25T15:59:07.552825+0000","last_clean":"2024-04-25T15:59:07.552825+0000","last_became_active":"2024-04-25T15:59:02.512103+0000","last_became_peered":"2024-04-25T15:59:02.512103+0000","last_unstale":"2024-04-25T15:59:07.552825+0000","last_undegraded":"2024-04-25T15:59:07.552825+0000","last_fullsized":"2024-04-25T15:59:07.552825+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:29.457755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501458+0000","last_change":"2024-04-25T15:59:03.001406+0000","last_active":"2024-04-25T15:59:04.501458+0000","last_peered":"2024-04-25T15:59:04.501458+0000","last_clean":"2024-04-25T15:59:04.501458+0000","last_became_active":"2024-04-25T15:59:02.999038+0000","last_became_peered":"2024-04-25T15:59:02.999038+0000","last_unstale":"2024-04-25T15:59:04.501458+0000","last_undegraded":"2024-04-25T15:59:04.501458+0000","last_fullsized":"2024-04-25T15:59:04.501458+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:56.354821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"75'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382541+0000","last_change":"2024-04-25T15:59:02.509619+0000","last_active":"2024-04-25T15:59:04.382541+0000","last_peered":"2024-04-25T15:59:04.382541+0000","last_clean":"2024-04-25T15:59:04.382541+0000","last_became_active":"2024-04-25T15:59:02.509254+0000","last_became_peered":"2024-04-25T15:59:02.509254+0000","last_unstale":"2024-04-25T15:59:04.382541+0000","last_undegraded":"2024-04-25T15:59:04.382541+0000","last_fullsized":"2024-04-25T15:59:04.382541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:00.382581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.982919+0000","last_change":"2024-04-25T15:59:03.000009+0000","last_active":"2024-04-25T15:59:03.982919+0000","last_peered":"2024-04-25T15:59:03.982919+0000","last_clean":"2024-04-25T15:59:03.982919+0000","last_became_active":"2024-04-25T15:59:02.999662+0000","last_became_peered":"2024-04-25T15:59:02.999662+0000","last_unstale":"2024-04-25T15:59:03.982919+0000","last_undegraded":"2024-04-25T15:59:03.982919+0000","last_fullsized":"2024-04-25T15:59:03.982919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:05:38.004819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.538155+0000","last_change":"2024-04-25T15:59:02.514429+0000","last_active":"2024-04-25T15:59:06.538155+0000","last_peered":"2024-04-25T15:59:06.538155+0000","last_clean":"2024-04-25T15:59:06.538155+0000","last_became_active":"2024-04-25T15:59:02.514144+0000","last_became_peered":"2024-04-25T15:59:02.514144+0000","last_unstale":"2024-04-25T15:59:06.538155+0000","last_undegraded":"2024-04-25T15:59:06.538155+0000","last_fullsized":"2024-04-25T15:59:06.538155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:45.144630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.384252+0000","last_change":"2024-04-25T15:59:03.008460+0000","last_active":"2024-04-25T15:59:04.384252+0000","last_peered":"2024-04-25T15:59:04.384252+0000","last_clean":"2024-04-25T15:59:04.384252+0000","last_became_active":"2024-04-25T15:59:03.008024+0000","last_became_peered":"2024-04-25T15:59:03.008024+0000","last_unstale":"2024-04-25T15:59:04.384252+0000","last_undegraded":"2024-04-25T15:59:04.384252+0000","last_fullsized":"2024-04-25T15:59:04.384252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:10.288760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"75'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.536658+0000","last_change":"2024-04-25T15:59:02.512494+0000","last_active":"2024-04-25T15:59:06.536658+0000","last_peered":"2024-04-25T15:59:06.536658+0000","last_clean":"2024-04-25T15:59:06.536658+0000","last_became_active":"2024-04-25T15:59:02.511182+0000","last_became_peered":"2024-04-25T15:59:02.511182+0000","last_unstale":"2024-04-25T15:59:06.536658+0000","last_undegraded":"2024-04-25T15:59:06.536658+0000","last_fullsized":"2024-04-25T15:59:06.536658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:54.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279861+0000","last_change":"2024-04-25T15:59:03.003848+0000","last_active":"2024-04-25T15:59:04.279861+0000","last_peered":"2024-04-25T15:59:04.279861+0000","last_clean":"2024-04-25T15:59:04.279861+0000","last_became_active":"2024-04-25T15:59:03.003512+0000","last_became_peered":"2024-04-25T15:59:03.003512+0000","last_unstale":"2024-04-25T15:59:04.279861+0000","last_undegraded":"2024-04-25T15:59:04.279861+0000","last_fullsized":"2024-04-25T15:59:04.279861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:54.667054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383573+0000","last_change":"2024-04-25T15:59:02.509343+0000","last_active":"2024-04-25T15:59:04.383573+0000","last_peered":"2024-04-25T15:59:04.383573+0000","last_clean":"2024-04-25T15:59:04.383573+0000","last_became_active":"2024-04-25T15:59:02.508980+0000","last_became_peered":"2024-04-25T15:59:02.508980+0000","last_unstale":"2024-04-25T15:59:04.383573+0000","last_undegraded":"2024-04-25T15:59:04.383573+0000","last_fullsized":"2024-04-25T15:59:04.383573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:54.344717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383629+0000","last_change":"2024-04-25T15:59:02.999862+0000","last_active":"2024-04-25T15:59:04.383629+0000","last_peered":"2024-04-25T15:59:04.383629+0000","last_clean":"2024-04-25T15:59:04.383629+0000","last_became_active":"2024-04-25T15:59:02.999472+0000","last_became_peered":"2024-04-25T15:59:02.999472+0000","last_unstale":"2024-04-25T15:59:04.383629+0000","last_undegraded":"2024-04-25T15:59:04.383629+0000","last_fullsized":"2024-04-25T15:59:04.383629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:46.165045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"75'8","reported_seq":23,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.536992+0000","last_change":"2024-04-25T15:59:02.508949+0000","last_active":"2024-04-25T15:59:06.536992+0000","last_peered":"2024-04-25T15:59:06.536992+0000","last_clean":"2024-04-25T15:59:06.536992+0000","last_became_active":"2024-04-25T15:59:02.508564+0000","last_became_peered":"2024-04-25T15:59:02.508564+0000","last_unstale":"2024-04-25T15:59:06.536992+0000","last_undegraded":"2024-04-25T15:59:06.536992+0000","last_fullsized":"2024-04-25T15:59:06.536992+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:12.590193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383192+0000","last_change":"2024-04-25T15:59:02.998575+0000","last_active":"2024-04-25T15:59:04.383192+0000","last_peered":"2024-04-25T15:59:04.383192+0000","last_clean":"2024-04-25T15:59:04.383192+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T15:59:04.383192+0000","last_undegraded":"2024-04-25T15:59:04.383192+0000","last_fullsized":"2024-04-25T15:59:04.383192+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:40.640468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.537055+0000","last_change":"2024-04-25T15:59:02.519110+0000","last_active":"2024-04-25T15:59:06.537055+0000","last_peered":"2024-04-25T15:59:06.537055+0000","last_clean":"2024-04-25T15:59:06.537055+0000","last_became_active":"2024-04-25T15:59:02.518373+0000","last_became_peered":"2024-04-25T15:59:02.518373+0000","last_unstale":"2024-04-25T15:59:06.537055+0000","last_undegraded":"2024-04-25T15:59:06.537055+0000","last_fullsized":"2024-04-25T15:59:06.537055+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:27.840296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382488+0000","last_change":"2024-04-25T15:59:02.991002+0000","last_active":"2024-04-25T15:59:04.382488+0000","last_peered":"2024-04-25T15:59:04.382488+0000","last_clean":"2024-04-25T15:59:04.382488+0000","last_became_active":"2024-04-25T15:59:02.990605+0000","last_became_peered":"2024-04-25T15:59:02.990605+0000","last_unstale":"2024-04-25T15:59:04.382488+0000","last_undegraded":"2024-04-25T15:59:04.382488+0000","last_fullsized":"2024-04-25T15:59:04.382488+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:22:00.723255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.538706+0000","last_change":"2024-04-25T15:59:02.509809+0000","last_active":"2024-04-25T15:59:06.538706+0000","last_peered":"2024-04-25T15:59:06.538706+0000","last_clean":"2024-04-25T15:59:06.538706+0000","last_became_active":"2024-04-25T15:59:02.509370+0000","last_became_peered":"2024-04-25T15:59:02.509370+0000","last_unstale":"2024-04-25T15:59:06.538706+0000","last_undegraded":"2024-04-25T15:59:06.538706+0000","last_fullsized":"2024-04-25T15:59:06.538706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:24.416072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383094+0000","last_change":"2024-04-25T15:59:02.999836+0000","last_active":"2024-04-25T15:59:04.383094+0000","last_peered":"2024-04-25T15:59:04.383094+0000","last_clean":"2024-04-25T15:59:04.383094+0000","last_became_active":"2024-04-25T15:59:02.997770+0000","last_became_peered":"2024-04-25T15:59:02.997770+0000","last_unstale":"2024-04-25T15:59:04.383094+0000","last_undegraded":"2024-04-25T15:59:04.383094+0000","last_fullsized":"2024-04-25T15:59:04.383094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:30.953339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:07.551268+0000","last_change":"2024-04-25T15:59:02.510465+0000","last_active":"2024-04-25T15:59:07.551268+0000","last_peered":"2024-04-25T15:59:07.551268+0000","last_clean":"2024-04-25T15:59:07.551268+0000","last_became_active":"2024-04-25T15:59:02.510067+0000","last_became_peered":"2024-04-25T15:59:02.510067+0000","last_unstale":"2024-04-25T15:59:07.551268+0000","last_undegraded":"2024-04-25T15:59:07.551268+0000","last_fullsized":"2024-04-25T15:59:07.551268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:40.739358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382786+0000","last_change":"2024-04-25T15:59:03.000028+0000","last_active":"2024-04-25T15:59:04.382786+0000","last_peered":"2024-04-25T15:59:04.382786+0000","last_clean":"2024-04-25T15:59:04.382786+0000","last_became_active":"2024-04-25T15:59:02.999704+0000","last_became_peered":"2024-04-25T15:59:02.999704+0000","last_unstale":"2024-04-25T15:59:04.382786+0000","last_undegraded":"2024-04-25T15:59:04.382786+0000","last_fullsized":"2024-04-25T15:59:04.382786+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:55.015085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'10","reported_seq":25,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.536023+0000","last_change":"2024-04-25T15:59:02.514442+0000","last_active":"2024-04-25T15:59:06.536023+0000","last_peered":"2024-04-25T15:59:06.536023+0000","last_clean":"2024-04-25T15:59:06.536023+0000","last_became_active":"2024-04-25T15:59:02.513951+0000","last_became_peered":"2024-04-25T15:59:02.513951+0000","last_unstale":"2024-04-25T15:59:06.536023+0000","last_undegraded":"2024-04-25T15:59:06.536023+0000","last_fullsized":"2024-04-25T15:59:06.536023+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:56.443999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382416+0000","last_change":"2024-04-25T15:59:03.003327+0000","last_active":"2024-04-25T15:59:04.382416+0000","last_peered":"2024-04-25T15:59:04.382416+0000","last_clean":"2024-04-25T15:59:04.382416+0000","last_became_active":"2024-04-25T15:59:03.002781+0000","last_became_peered":"2024-04-25T15:59:03.002781+0000","last_unstale":"2024-04-25T15:59:04.382416+0000","last_undegraded":"2024-04-25T15:59:04.382416+0000","last_fullsized":"2024-04-25T15:59:04.382416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:01.816897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"74'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383817+0000","last_change":"2024-04-25T15:59:02.512836+0000","last_active":"2024-04-25T15:59:04.383817+0000","last_peered":"2024-04-25T15:59:04.383817+0000","last_clean":"2024-04-25T15:59:04.383817+0000","last_became_active":"2024-04-25T15:59:02.512294+0000","last_became_peered":"2024-04-25T15:59:02.512294+0000","last_unstale":"2024-04-25T15:59:04.383817+0000","last_undegraded":"2024-04-25T15:59:04.383817+0000","last_fullsized":"2024-04-25T15:59:04.383817+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:30:07.322880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981695+0000","last_change":"2024-04-25T15:59:03.004639+0000","last_active":"2024-04-25T15:59:03.981695+0000","last_peered":"2024-04-25T15:59:03.981695+0000","last_clean":"2024-04-25T15:59:03.981695+0000","last_became_active":"2024-04-25T15:59:03.004260+0000","last_became_peered":"2024-04-25T15:59:03.004260+0000","last_unstale":"2024-04-25T15:59:03.981695+0000","last_undegraded":"2024-04-25T15:59:03.981695+0000","last_fullsized":"2024-04-25T15:59:03.981695+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:00.450758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.115544+0000","last_change":"2024-04-25T15:59:02.509716+0000","last_active":"2024-04-25T15:59:04.115544+0000","last_peered":"2024-04-25T15:59:04.115544+0000","last_clean":"2024-04-25T15:59:04.115544+0000","last_became_active":"2024-04-25T15:59:02.509303+0000","last_became_peered":"2024-04-25T15:59:02.509303+0000","last_unstale":"2024-04-25T15:59:04.115544+0000","last_undegraded":"2024-04-25T15:59:04.115544+0000","last_fullsized":"2024-04-25T15:59:04.115544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:34.037590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981147+0000","last_change":"2024-04-25T15:59:03.005141+0000","last_active":"2024-04-25T15:59:03.981147+0000","last_peered":"2024-04-25T15:59:03.981147+0000","last_clean":"2024-04-25T15:59:03.981147+0000","last_became_active":"2024-04-25T15:59:03.004760+0000","last_became_peered":"2024-04-25T15:59:03.004760+0000","last_unstale":"2024-04-25T15:59:03.981147+0000","last_undegraded":"2024-04-25T15:59:03.981147+0000","last_fullsized":"2024-04-25T15:59:03.981147+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:14.923126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382710+0000","last_change":"2024-04-25T15:59:02.512779+0000","last_active":"2024-04-25T15:59:04.382710+0000","last_peered":"2024-04-25T15:59:04.382710+0000","last_clean":"2024-04-25T15:59:04.382710+0000","last_became_active":"2024-04-25T15:59:02.511240+0000","last_became_peered":"2024-04-25T15:59:02.511240+0000","last_unstale":"2024-04-25T15:59:04.382710+0000","last_undegraded":"2024-04-25T15:59:04.382710+0000","last_fullsized":"2024-04-25T15:59:04.382710+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:51.703751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981159+0000","last_change":"2024-04-25T15:59:03.004841+0000","last_active":"2024-04-25T15:59:03.981159+0000","last_peered":"2024-04-25T15:59:03.981159+0000","last_clean":"2024-04-25T15:59:03.981159+0000","last_became_active":"2024-04-25T15:59:03.004488+0000","last_became_peered":"2024-04-25T15:59:03.004488+0000","last_unstale":"2024-04-25T15:59:03.981159+0000","last_undegraded":"2024-04-25T15:59:03.981159+0000","last_fullsized":"2024-04-25T15:59:03.981159+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:07.672312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382598+0000","last_change":"2024-04-25T15:59:03.001296+0000","last_active":"2024-04-25T15:59:04.382598+0000","last_peered":"2024-04-25T15:59:04.382598+0000","last_clean":"2024-04-25T15:59:04.382598+0000","last_became_active":"2024-04-25T15:59:03.000568+0000","last_became_peered":"2024-04-25T15:59:03.000568+0000","last_unstale":"2024-04-25T15:59:04.382598+0000","last_undegraded":"2024-04-25T15:59:04.382598+0000","last_fullsized":"2024-04-25T15:59:04.382598+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:51.335849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:07.552145+0000","last_change":"2024-04-25T15:59:02.513943+0000","last_active":"2024-04-25T15:59:07.552145+0000","last_peered":"2024-04-25T15:59:07.552145+0000","last_clean":"2024-04-25T15:59:07.552145+0000","last_became_active":"2024-04-25T15:59:02.513595+0000","last_became_peered":"2024-04-25T15:59:02.513595+0000","last_unstale":"2024-04-25T15:59:07.552145+0000","last_undegraded":"2024-04-25T15:59:07.552145+0000","last_fullsized":"2024-04-25T15:59:07.552145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:50.720333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501669+0000","last_change":"2024-04-25T15:59:03.008561+0000","last_active":"2024-04-25T15:59:04.501669+0000","last_peered":"2024-04-25T15:59:04.501669+0000","last_clean":"2024-04-25T15:59:04.501669+0000","last_became_active":"2024-04-25T15:59:03.008134+0000","last_became_peered":"2024-04-25T15:59:03.008134+0000","last_unstale":"2024-04-25T15:59:04.501669+0000","last_undegraded":"2024-04-25T15:59:04.501669+0000","last_fullsized":"2024-04-25T15:59:04.501669+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:43.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.537970+0000","last_change":"2024-04-25T15:59:02.510731+0000","last_active":"2024-04-25T15:59:06.537970+0000","last_peered":"2024-04-25T15:59:06.537970+0000","last_clean":"2024-04-25T15:59:06.537970+0000","last_became_active":"2024-04-25T15:59:02.510356+0000","last_became_peered":"2024-04-25T15:59:02.510356+0000","last_unstale":"2024-04-25T15:59:06.537970+0000","last_undegraded":"2024-04-25T15:59:06.537970+0000","last_fullsized":"2024-04-25T15:59:06.537970+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:33.066858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":90,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383010+0000","last_change":"2024-04-25T15:57:47.988899+0000","last_active":"2024-04-25T15:59:04.383010+0000","last_peered":"2024-04-25T15:59:04.383010+0000","last_clean":"2024-04-25T15:59:04.383010+0000","last_became_active":"2024-04-25T15:57:47.982613+0000","last_became_peered":"2024-04-25T15:57:47.982613+0000","last_unstale":"2024-04-25T15:59:04.383010+0000","last_undegraded":"2024-04-25T15:59:04.383010+0000","last_fullsized":"2024-04-25T15:59:04.383010+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_clean_scrub_stamp":"2024-04-25T15:54:18.013489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:39.323767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382339+0000","last_change":"2024-04-25T15:59:02.998360+0000","last_active":"2024-04-25T15:59:04.382339+0000","last_peered":"2024-04-25T15:59:04.382339+0000","last_clean":"2024-04-25T15:59:04.382339+0000","last_became_active":"2024-04-25T15:59:02.998048+0000","last_became_peered":"2024-04-25T15:59:02.998048+0000","last_unstale":"2024-04-25T15:59:04.382339+0000","last_undegraded":"2024-04-25T15:59:04.382339+0000","last_fullsized":"2024-04-25T15:59:04.382339+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:01.524608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"75'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383319+0000","last_change":"2024-04-25T15:59:02.515769+0000","last_active":"2024-04-25T15:59:04.383319+0000","last_peered":"2024-04-25T15:59:04.383319+0000","last_clean":"2024-04-25T15:59:04.383319+0000","last_became_active":"2024-04-25T15:59:02.515467+0000","last_became_peered":"2024-04-25T15:59:02.515467+0000","last_unstale":"2024-04-25T15:59:04.383319+0000","last_undegraded":"2024-04-25T15:59:04.383319+0000","last_fullsized":"2024-04-25T15:59:04.383319+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:59:50.107475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"75'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383304+0000","last_change":"2024-04-25T15:59:02.512701+0000","last_active":"2024-04-25T15:59:04.383304+0000","last_peered":"2024-04-25T15:59:04.383304+0000","last_clean":"2024-04-25T15:59:04.383304+0000","last_became_active":"2024-04-25T15:59:02.512352+0000","last_became_peered":"2024-04-25T15:59:02.512352+0000","last_unstale":"2024-04-25T15:59:04.383304+0000","last_undegraded":"2024-04-25T15:59:04.383304+0000","last_fullsized":"2024-04-25T15:59:04.383304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:31.772382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382591+0000","last_change":"2024-04-25T15:59:02.989698+0000","last_active":"2024-04-25T15:59:04.382591+0000","last_peered":"2024-04-25T15:59:04.382591+0000","last_clean":"2024-04-25T15:59:04.382591+0000","last_became_active":"2024-04-25T15:59:02.989284+0000","last_became_peered":"2024-04-25T15:59:02.989284+0000","last_unstale":"2024-04-25T15:59:04.382591+0000","last_undegraded":"2024-04-25T15:59:04.382591+0000","last_fullsized":"2024-04-25T15:59:04.382591+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:12.501738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'7","reported_seq":22,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:08.561453+0000","last_change":"2024-04-25T15:59:02.517272+0000","last_active":"2024-04-25T15:59:08.561453+0000","last_peered":"2024-04-25T15:59:08.561453+0000","last_clean":"2024-04-25T15:59:08.561453+0000","last_became_active":"2024-04-25T15:59:02.516370+0000","last_became_peered":"2024-04-25T15:59:02.516370+0000","last_unstale":"2024-04-25T15:59:08.561453+0000","last_undegraded":"2024-04-25T15:59:08.561453+0000","last_fullsized":"2024-04-25T15:59:08.561453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:31.139689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501486+0000","last_change":"2024-04-25T15:59:03.003111+0000","last_active":"2024-04-25T15:59:04.501486+0000","last_peered":"2024-04-25T15:59:04.501486+0000","last_clean":"2024-04-25T15:59:04.501486+0000","last_became_active":"2024-04-25T15:59:03.002766+0000","last_became_peered":"2024-04-25T15:59:03.002766+0000","last_unstale":"2024-04-25T15:59:04.501486+0000","last_undegraded":"2024-04-25T15:59:04.501486+0000","last_fullsized":"2024-04-25T15:59:04.501486+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:39.866119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.537203+0000","last_change":"2024-04-25T15:59:02.521970+0000","last_active":"2024-04-25T15:59:06.537203+0000","last_peered":"2024-04-25T15:59:06.537203+0000","last_clean":"2024-04-25T15:59:06.537203+0000","last_became_active":"2024-04-25T15:59:02.521650+0000","last_became_peered":"2024-04-25T15:59:02.521650+0000","last_unstale":"2024-04-25T15:59:06.537203+0000","last_undegraded":"2024-04-25T15:59:06.537203+0000","last_fullsized":"2024-04-25T15:59:06.537203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:32.996868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279844+0000","last_change":"2024-04-25T15:59:02.996358+0000","last_active":"2024-04-25T15:59:04.279844+0000","last_peered":"2024-04-25T15:59:04.279844+0000","last_clean":"2024-04-25T15:59:04.279844+0000","last_became_active":"2024-04-25T15:59:02.995988+0000","last_became_peered":"2024-04-25T15:59:02.995988+0000","last_unstale":"2024-04-25T15:59:04.279844+0000","last_undegraded":"2024-04-25T15:59:04.279844+0000","last_fullsized":"2024-04-25T15:59:04.279844+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:51.021841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"75'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.536492+0000","last_change":"2024-04-25T15:59:02.518330+0000","last_active":"2024-04-25T15:59:06.536492+0000","last_peered":"2024-04-25T15:59:06.536492+0000","last_clean":"2024-04-25T15:59:06.536492+0000","last_became_active":"2024-04-25T15:59:02.517978+0000","last_became_peered":"2024-04-25T15:59:02.517978+0000","last_unstale":"2024-04-25T15:59:06.536492+0000","last_undegraded":"2024-04-25T15:59:06.536492+0000","last_fullsized":"2024-04-25T15:59:06.536492+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:25.443963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382788+0000","last_change":"2024-04-25T15:59:03.000198+0000","last_active":"2024-04-25T15:59:04.382788+0000","last_peered":"2024-04-25T15:59:04.382788+0000","last_clean":"2024-04-25T15:59:04.382788+0000","last_became_active":"2024-04-25T15:59:02.999878+0000","last_became_peered":"2024-04-25T15:59:02.999878+0000","last_unstale":"2024-04-25T15:59:04.382788+0000","last_undegraded":"2024-04-25T15:59:04.382788+0000","last_fullsized":"2024-04-25T15:59:04.382788+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:37.966595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"75'7","reported_seq":22,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:09.257733+0000","last_change":"2024-04-25T15:59:02.511298+0000","last_active":"2024-04-25T15:59:09.257733+0000","last_peered":"2024-04-25T15:59:09.257733+0000","last_clean":"2024-04-25T15:59:09.257733+0000","last_became_active":"2024-04-25T15:59:02.510521+0000","last_became_peered":"2024-04-25T15:59:02.510521+0000","last_unstale":"2024-04-25T15:59:09.257733+0000","last_undegraded":"2024-04-25T15:59:09.257733+0000","last_fullsized":"2024-04-25T15:59:09.257733+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:00.807919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981476+0000","last_change":"2024-04-25T15:59:02.999212+0000","last_active":"2024-04-25T15:59:03.981476+0000","last_peered":"2024-04-25T15:59:03.981476+0000","last_clean":"2024-04-25T15:59:03.981476+0000","last_became_active":"2024-04-25T15:59:02.998881+0000","last_became_peered":"2024-04-25T15:59:02.998881+0000","last_unstale":"2024-04-25T15:59:03.981476+0000","last_undegraded":"2024-04-25T15:59:03.981476+0000","last_fullsized":"2024-04-25T15:59:03.981476+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:09.092196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382729+0000","last_change":"2024-04-25T15:59:02.517412+0000","last_active":"2024-04-25T15:59:04.382729+0000","last_peered":"2024-04-25T15:59:04.382729+0000","last_clean":"2024-04-25T15:59:04.382729+0000","last_became_active":"2024-04-25T15:59:02.517077+0000","last_became_peered":"2024-04-25T15:59:02.517077+0000","last_unstale":"2024-04-25T15:59:04.382729+0000","last_undegraded":"2024-04-25T15:59:04.382729+0000","last_fullsized":"2024-04-25T15:59:04.382729+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:00.161199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501479+0000","last_change":"2024-04-25T15:59:02.999704+0000","last_active":"2024-04-25T15:59:04.501479+0000","last_peered":"2024-04-25T15:59:04.501479+0000","last_clean":"2024-04-25T15:59:04.501479+0000","last_became_active":"2024-04-25T15:59:02.999381+0000","last_became_peered":"2024-04-25T15:59:02.999381+0000","last_unstale":"2024-04-25T15:59:04.501479+0000","last_undegraded":"2024-04-25T15:59:04.501479+0000","last_fullsized":"2024-04-25T15:59:04.501479+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:34:43.572698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.535710+0000","last_change":"2024-04-25T15:59:02.515912+0000","last_active":"2024-04-25T15:59:06.535710+0000","last_peered":"2024-04-25T15:59:06.535710+0000","last_clean":"2024-04-25T15:59:06.535710+0000","last_became_active":"2024-04-25T15:59:02.514949+0000","last_became_peered":"2024-04-25T15:59:02.514949+0000","last_unstale":"2024-04-25T15:59:06.535710+0000","last_undegraded":"2024-04-25T15:59:06.535710+0000","last_fullsized":"2024-04-25T15:59:06.535710+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:33.467223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383268+0000","last_change":"2024-04-25T15:59:02.988896+0000","last_active":"2024-04-25T15:59:04.383268+0000","last_peered":"2024-04-25T15:59:04.383268+0000","last_clean":"2024-04-25T15:59:04.383268+0000","last_became_active":"2024-04-25T15:59:02.988478+0000","last_became_peered":"2024-04-25T15:59:02.988478+0000","last_unstale":"2024-04-25T15:59:04.383268+0000","last_undegraded":"2024-04-25T15:59:04.383268+0000","last_fullsized":"2024-04-25T15:59:04.383268+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:47.014678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383305+0000","last_change":"2024-04-25T15:59:02.523031+0000","last_active":"2024-04-25T15:59:04.383305+0000","last_peered":"2024-04-25T15:59:04.383305+0000","last_clean":"2024-04-25T15:59:04.383305+0000","last_became_active":"2024-04-25T15:59:02.522675+0000","last_became_peered":"2024-04-25T15:59:02.522675+0000","last_unstale":"2024-04-25T15:59:04.383305+0000","last_undegraded":"2024-04-25T15:59:04.383305+0000","last_fullsized":"2024-04-25T15:59:04.383305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:09:45.277456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383885+0000","last_change":"2024-04-25T15:59:02.990507+0000","last_active":"2024-04-25T15:59:04.383885+0000","last_peered":"2024-04-25T15:59:04.383885+0000","last_clean":"2024-04-25T15:59:04.383885+0000","last_became_active":"2024-04-25T15:59:02.990160+0000","last_became_peered":"2024-04-25T15:59:02.990160+0000","last_unstale":"2024-04-25T15:59:04.383885+0000","last_undegraded":"2024-04-25T15:59:04.383885+0000","last_fullsized":"2024-04-25T15:59:04.383885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:18:21.526155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"74'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383411+0000","last_change":"2024-04-25T15:59:02.510956+0000","last_active":"2024-04-25T15:59:04.383411+0000","last_peered":"2024-04-25T15:59:04.383411+0000","last_clean":"2024-04-25T15:59:04.383411+0000","last_became_active":"2024-04-25T15:59:02.510578+0000","last_became_peered":"2024-04-25T15:59:02.510578+0000","last_unstale":"2024-04-25T15:59:04.383411+0000","last_undegraded":"2024-04-25T15:59:04.383411+0000","last_fullsized":"2024-04-25T15:59:04.383411+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:11.512365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383323+0000","last_change":"2024-04-25T15:59:03.001122+0000","last_active":"2024-04-25T15:59:04.383323+0000","last_peered":"2024-04-25T15:59:04.383323+0000","last_clean":"2024-04-25T15:59:04.383323+0000","last_became_active":"2024-04-25T15:59:03.000672+0000","last_became_peered":"2024-04-25T15:59:03.000672+0000","last_unstale":"2024-04-25T15:59:04.383323+0000","last_undegraded":"2024-04-25T15:59:04.383323+0000","last_fullsized":"2024-04-25T15:59:04.383323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:33.216434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382609+0000","last_change":"2024-04-25T15:59:02.520645+0000","last_active":"2024-04-25T15:59:04.382609+0000","last_peered":"2024-04-25T15:59:04.382609+0000","last_clean":"2024-04-25T15:59:04.382609+0000","last_became_active":"2024-04-25T15:59:02.520197+0000","last_became_peered":"2024-04-25T15:59:02.520197+0000","last_unstale":"2024-04-25T15:59:04.382609+0000","last_undegraded":"2024-04-25T15:59:04.382609+0000","last_fullsized":"2024-04-25T15:59:04.382609+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:20.120554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383762+0000","last_change":"2024-04-25T15:59:03.001668+0000","last_active":"2024-04-25T15:59:04.383762+0000","last_peered":"2024-04-25T15:59:04.383762+0000","last_clean":"2024-04-25T15:59:04.383762+0000","last_became_active":"2024-04-25T15:59:03.001333+0000","last_became_peered":"2024-04-25T15:59:03.001333+0000","last_unstale":"2024-04-25T15:59:04.383762+0000","last_undegraded":"2024-04-25T15:59:04.383762+0000","last_fullsized":"2024-04-25T15:59:04.383762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:24.992476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981543+0000","last_change":"2024-04-25T15:59:03.005533+0000","last_active":"2024-04-25T15:59:03.981543+0000","last_peered":"2024-04-25T15:59:03.981543+0000","last_clean":"2024-04-25T15:59:03.981543+0000","last_became_active":"2024-04-25T15:59:03.004124+0000","last_became_peered":"2024-04-25T15:59:03.004124+0000","last_unstale":"2024-04-25T15:59:03.981543+0000","last_undegraded":"2024-04-25T15:59:03.981543+0000","last_fullsized":"2024-04-25T15:59:03.981543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:27:28.507065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382715+0000","last_change":"2024-04-25T15:59:02.993416+0000","last_active":"2024-04-25T15:59:04.382715+0000","last_peered":"2024-04-25T15:59:04.382715+0000","last_clean":"2024-04-25T15:59:04.382715+0000","last_became_active":"2024-04-25T15:59:02.993098+0000","last_became_peered":"2024-04-25T15:59:02.993098+0000","last_unstale":"2024-04-25T15:59:04.382715+0000","last_undegraded":"2024-04-25T15:59:04.382715+0000","last_fullsized":"2024-04-25T15:59:04.382715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:46.859932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382660+0000","last_change":"2024-04-25T15:59:02.996858+0000","last_active":"2024-04-25T15:59:04.382660+0000","last_peered":"2024-04-25T15:59:04.382660+0000","last_clean":"2024-04-25T15:59:04.382660+0000","last_became_active":"2024-04-25T15:59:02.996530+0000","last_became_peered":"2024-04-25T15:59:02.996530+0000","last_unstale":"2024-04-25T15:59:04.382660+0000","last_undegraded":"2024-04-25T15:59:04.382660+0000","last_fullsized":"2024-04-25T15:59:04.382660+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:36.723948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279720+0000","last_change":"2024-04-25T15:59:02.995942+0000","last_active":"2024-04-25T15:59:04.279720+0000","last_peered":"2024-04-25T15:59:04.279720+0000","last_clean":"2024-04-25T15:59:04.279720+0000","last_became_active":"2024-04-25T15:59:02.995567+0000","last_became_peered":"2024-04-25T15:59:02.995567+0000","last_unstale":"2024-04-25T15:59:04.279720+0000","last_undegraded":"2024-04-25T15:59:04.279720+0000","last_fullsized":"2024-04-25T15:59:04.279720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:32.692392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382613+0000","last_change":"2024-04-25T15:59:03.000326+0000","last_active":"2024-04-25T15:59:04.382613+0000","last_peered":"2024-04-25T15:59:04.382613+0000","last_clean":"2024-04-25T15:59:04.382613+0000","last_became_active":"2024-04-25T15:59:02.999961+0000","last_became_peered":"2024-04-25T15:59:02.999961+0000","last_unstale":"2024-04-25T15:59:04.382613+0000","last_undegraded":"2024-04-25T15:59:04.382613+0000","last_fullsized":"2024-04-25T15:59:04.382613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:10.631460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981619+0000","last_change":"2024-04-25T15:59:03.003625+0000","last_active":"2024-04-25T15:59:03.981619+0000","last_peered":"2024-04-25T15:59:03.981619+0000","last_clean":"2024-04-25T15:59:03.981619+0000","last_became_active":"2024-04-25T15:59:03.003280+0000","last_became_peered":"2024-04-25T15:59:03.003280+0000","last_unstale":"2024-04-25T15:59:03.981619+0000","last_undegraded":"2024-04-25T15:59:03.981619+0000","last_fullsized":"2024-04-25T15:59:03.981619+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:21:54.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383644+0000","last_change":"2024-04-25T15:59:02.994645+0000","last_active":"2024-04-25T15:59:04.383644+0000","last_peered":"2024-04-25T15:59:04.383644+0000","last_clean":"2024-04-25T15:59:04.383644+0000","last_became_active":"2024-04-25T15:59:02.993924+0000","last_became_peered":"2024-04-25T15:59:02.993924+0000","last_unstale":"2024-04-25T15:59:04.383644+0000","last_undegraded":"2024-04-25T15:59:04.383644+0000","last_fullsized":"2024-04-25T15:59:04.383644+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.644708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501509+0000","last_change":"2024-04-25T15:59:03.006156+0000","last_active":"2024-04-25T15:59:04.501509+0000","last_peered":"2024-04-25T15:59:04.501509+0000","last_clean":"2024-04-25T15:59:04.501509+0000","last_became_active":"2024-04-25T15:59:03.005795+0000","last_became_peered":"2024-04-25T15:59:03.005795+0000","last_unstale":"2024-04-25T15:59:04.501509+0000","last_undegraded":"2024-04-25T15:59:04.501509+0000","last_fullsized":"2024-04-25T15:59:04.501509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:23.523868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382418+0000","last_change":"2024-04-25T15:59:03.010404+0000","last_active":"2024-04-25T15:59:04.382418+0000","last_peered":"2024-04-25T15:59:04.382418+0000","last_clean":"2024-04-25T15:59:04.382418+0000","last_became_active":"2024-04-25T15:59:03.010100+0000","last_became_peered":"2024-04-25T15:59:03.010100+0000","last_unstale":"2024-04-25T15:59:04.382418+0000","last_undegraded":"2024-04-25T15:59:04.382418+0000","last_fullsized":"2024-04-25T15:59:04.382418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:22.718842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501455+0000","last_change":"2024-04-25T15:59:03.009240+0000","last_active":"2024-04-25T15:59:04.501455+0000","last_peered":"2024-04-25T15:59:04.501455+0000","last_clean":"2024-04-25T15:59:04.501455+0000","last_became_active":"2024-04-25T15:59:03.008723+0000","last_became_peered":"2024-04-25T15:59:03.008723+0000","last_unstale":"2024-04-25T15:59:04.501455+0000","last_undegraded":"2024-04-25T15:59:04.501455+0000","last_fullsized":"2024-04-25T15:59:04.501455+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:14.049934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383782+0000","last_change":"2024-04-25T15:59:02.994359+0000","last_active":"2024-04-25T15:59:04.383782+0000","last_peered":"2024-04-25T15:59:04.383782+0000","last_clean":"2024-04-25T15:59:04.383782+0000","last_became_active":"2024-04-25T15:59:02.993990+0000","last_became_peered":"2024-04-25T15:59:02.993990+0000","last_unstale":"2024-04-25T15:59:04.383782+0000","last_undegraded":"2024-04-25T15:59:04.383782+0000","last_fullsized":"2024-04-25T15:59:04.383782+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:55.477414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382793+0000","last_change":"2024-04-25T15:59:02.995510+0000","last_active":"2024-04-25T15:59:04.382793+0000","last_peered":"2024-04-25T15:59:04.382793+0000","last_clean":"2024-04-25T15:59:04.382793+0000","last_became_active":"2024-04-25T15:59:02.995008+0000","last_became_peered":"2024-04-25T15:59:02.995008+0000","last_unstale":"2024-04-25T15:59:04.382793+0000","last_undegraded":"2024-04-25T15:59:04.382793+0000","last_fullsized":"2024-04-25T15:59:04.382793+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:40.439615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981630+0000","last_change":"2024-04-25T15:59:03.005846+0000","last_active":"2024-04-25T15:59:03.981630+0000","last_peered":"2024-04-25T15:59:03.981630+0000","last_clean":"2024-04-25T15:59:03.981630+0000","last_became_active":"2024-04-25T15:59:03.003779+0000","last_became_peered":"2024-04-25T15:59:03.003779+0000","last_unstale":"2024-04-25T15:59:03.981630+0000","last_undegraded":"2024-04-25T15:59:03.981630+0000","last_fullsized":"2024-04-25T15:59:03.981630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.698405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247542,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4528,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":13914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710732,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":798720,"data_stored":383762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906963,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":884736,"data_stored":976698,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070488,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":384580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266718,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":811008,"data_stored":384388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462947,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":851968,"data_stored":974712,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626472,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":383852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":811008,"data_stored":385374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717324,"statfs":{"total":95995035648,"available":95966539776,"internally_reserved":0,"allocated":831488,"data_stored":385828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":815104,"data_stored":385408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":384426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986227,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":811008,"data_stored":384406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822701,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32032,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713120,"statfs":{"total":95995035648,"available":95962234880,"internally_reserved":0,"allocated":876544,"data_stored":975606,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:59:11.799 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T15:59:11.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:11 smithi063 ceph-mon[32413]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:11.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:11 smithi063 ceph-mon[32413]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:11.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:11 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/274341499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:59:11.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:11 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/349894771' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:11.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:11 smithi186 ceph-mon[33258]: from='client.14751 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:11.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:11 smithi186 ceph-mon[33258]: from='client.14757 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T15:59:11.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:11 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/274341499' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T15:59:11.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:11 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/349894771' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:11.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.946+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/194824154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac104d30 msgr2=0x7fb8ac105140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.946+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/194824154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac104d30 0x7fb8ac105140 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb89c009900 tx=0x7fb89c02a410 comp rx=0 tx=0).stop 2024-04-25T15:59:11.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.946+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/194824154 shutdown_connections 2024-04-25T15:59:11.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.946+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/194824154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8ac109a30 0x7fb8ac10be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.946+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/194824154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac104d30 0x7fb8ac105140 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.946+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/194824154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8ac103af0 0x7fb8ac1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.946+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/194824154 >> 172.21.15.31:0/194824154 conn(0x7fb8ac0fed40 msgr2=0x7fb8ac101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.946+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/194824154 shutdown_connections 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.946+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/194824154 wait complete. 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b3d5e640 1 Processor -- start 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b3d5e640 1 -- start start 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b3d5e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac103af0 0x7fb8ac077960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b3d5e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8ac104d30 0x7fb8ac076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b3d5e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8ac109a30 0x7fb8ac076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b3d5e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac10f5f0 con 0x7fb8ac103af0 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b3d5e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac10f470 con 0x7fb8ac104d30 2024-04-25T15:59:11.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b3d5e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8ac10f770 con 0x7fb8ac109a30 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b1afa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac103af0 0x7fb8ac077960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b1afa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac103af0 0x7fb8ac077960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39680/0 (socket says 172.21.15.31:39680) 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b1afa640 1 -- 172.21.15.31:0/1786009727 learned_addr learned my addr 172.21.15.31:0/1786009727 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b22fb640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8ac109a30 0x7fb8ac076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b12f9640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8ac104d30 0x7fb8ac076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b1afa640 1 -- 172.21.15.31:0/1786009727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8ac109a30 msgr2=0x7fb8ac076540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b1afa640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8ac109a30 0x7fb8ac076540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b1afa640 1 -- 172.21.15.31:0/1786009727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8ac104d30 msgr2=0x7fb8ac076000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b1afa640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8ac104d30 0x7fb8ac076000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b1afa640 1 -- 172.21.15.31:0/1786009727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ac076be0 con 0x7fb8ac103af0 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.947+0000 7fb8b12f9640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8ac104d30 0x7fb8ac076000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.948+0000 7fb8b1afa640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac103af0 0x7fb8ac077960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c970 tx=0x7fb8a800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.948+0000 7fb8a2ffd640 1 -- 172.21.15.31:0/1786009727 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a8007c70 con 0x7fb8ac103af0 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.948+0000 7fb8a2ffd640 1 -- 172.21.15.31:0/1786009727 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8a8007e10 con 0x7fb8ac103af0 2024-04-25T15:59:11.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.948+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8ac1a42e0 con 0x7fb8ac103af0 2024-04-25T15:59:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.948+0000 7fb8a2ffd640 1 -- 172.21.15.31:0/1786009727 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a80187b0 con 0x7fb8ac103af0 2024-04-25T15:59:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.948+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8ac1a46f0 con 0x7fb8ac103af0 2024-04-25T15:59:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.949+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8ac108840 con 0x7fb8ac103af0 2024-04-25T15:59:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.950+0000 7fb8a2ffd640 1 -- 172.21.15.31:0/1786009727 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fb8a800fab0 con 0x7fb8ac103af0 2024-04-25T15:59:11.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.951+0000 7fb8a2ffd640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8980afb80 0x7fb8980b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.951+0000 7fb8b12f9640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8980afb80 0x7fb8980b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.951+0000 7fb8a2ffd640 1 -- 172.21.15.31:0/1786009727 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fb8a80ded40 con 0x7fb8ac103af0 2024-04-25T15:59:11.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.952+0000 7fb8b12f9640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8980afb80 0x7fb8980b2090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb89c000c00 tx=0x7fb89c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:11.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:11.956+0000 7fb8a2ffd640 1 -- 172.21.15.31:0/1786009727 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8a8014030 con 0x7fb8ac103af0 2024-04-25T15:59:12.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.145+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb8ac062fe0 con 0x7fb8980afb80 2024-04-25T15:59:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.152+0000 7fb8a2ffd640 1 -- 172.21.15.31:0/1786009727 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258916 (secure 0 0 0) 0x7fb8ac062fe0 con 0x7fb8980afb80 2024-04-25T15:59:12.157 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T15:59:12.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8980afb80 msgr2=0x7fb8980b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8980afb80 0x7fb8980b2090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb89c000c00 tx=0x7fb89c005d50 comp rx=0 tx=0).stop 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac103af0 msgr2=0x7fb8ac077960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac103af0 0x7fb8ac077960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb8a800c970 tx=0x7fb8a800ce30 comp rx=0 tx=0).stop 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 shutdown_connections 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8980afb80 0x7fb8980b2090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8ac109a30 0x7fb8ac076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8ac104d30 0x7fb8ac076000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 --2- 172.21.15.31:0/1786009727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8ac103af0 0x7fb8ac077960 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 >> 172.21.15.31:0/1786009727 conn(0x7fb8ac0fed40 msgr2=0x7fb8ac105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 shutdown_connections 2024-04-25T15:59:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:12.155+0000 7fb8b3d5e640 1 -- 172.21.15.31:0/1786009727 wait complete. 2024-04-25T15:59:12.169 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":286,"stamp":"2024-04-25T15:59:11.313746+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594896,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":241,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":194,"ondisk_log_size":194,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354728,"kb_used_data":9716,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124587096,"statfs":{"total":1151940427776,"available":1151577186304,"internally_reserved":0,"allocated":9949184,"data_stored":6389040,"data_compressed":107478,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4528,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.335447"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981726+0000","last_change":"2024-04-25T15:59:02.997678+0000","last_active":"2024-04-25T15:59:03.981726+0000","last_peered":"2024-04-25T15:59:03.981726+0000","last_clean":"2024-04-25T15:59:03.981726+0000","last_became_active":"2024-04-25T15:59:02.997335+0000","last_became_peered":"2024-04-25T15:59:02.997335+0000","last_unstale":"2024-04-25T15:59:03.981726+0000","last_undegraded":"2024-04-25T15:59:03.981726+0000","last_fullsized":"2024-04-25T15:59:03.981726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:33.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279944+0000","last_change":"2024-04-25T15:59:03.001049+0000","last_active":"2024-04-25T15:59:04.279944+0000","last_peered":"2024-04-25T15:59:04.279944+0000","last_clean":"2024-04-25T15:59:04.279944+0000","last_became_active":"2024-04-25T15:59:03.000663+0000","last_became_peered":"2024-04-25T15:59:03.000663+0000","last_unstale":"2024-04-25T15:59:04.279944+0000","last_undegraded":"2024-04-25T15:59:04.279944+0000","last_fullsized":"2024-04-25T15:59:04.279944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:04:36.670519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279577+0000","last_change":"2024-04-25T15:59:03.003861+0000","last_active":"2024-04-25T15:59:04.279577+0000","last_peered":"2024-04-25T15:59:04.279577+0000","last_clean":"2024-04-25T15:59:04.279577+0000","last_became_active":"2024-04-25T15:59:03.003508+0000","last_became_peered":"2024-04-25T15:59:03.003508+0000","last_unstale":"2024-04-25T15:59:04.279577+0000","last_undegraded":"2024-04-25T15:59:04.279577+0000","last_fullsized":"2024-04-25T15:59:04.279577+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:25.679389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.384069+0000","last_change":"2024-04-25T15:59:03.003330+0000","last_active":"2024-04-25T15:59:04.384069+0000","last_peered":"2024-04-25T15:59:04.384069+0000","last_clean":"2024-04-25T15:59:04.384069+0000","last_became_active":"2024-04-25T15:59:03.002872+0000","last_became_peered":"2024-04-25T15:59:03.002872+0000","last_unstale":"2024-04-25T15:59:04.384069+0000","last_undegraded":"2024-04-25T15:59:04.384069+0000","last_fullsized":"2024-04-25T15:59:04.384069+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:02.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383605+0000","last_change":"2024-04-25T15:59:02.997684+0000","last_active":"2024-04-25T15:59:04.383605+0000","last_peered":"2024-04-25T15:59:04.383605+0000","last_clean":"2024-04-25T15:59:04.383605+0000","last_became_active":"2024-04-25T15:59:02.997370+0000","last_became_peered":"2024-04-25T15:59:02.997370+0000","last_unstale":"2024-04-25T15:59:04.383605+0000","last_undegraded":"2024-04-25T15:59:04.383605+0000","last_fullsized":"2024-04-25T15:59:04.383605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:24.349085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.980982+0000","last_change":"2024-04-25T15:59:03.003638+0000","last_active":"2024-04-25T15:59:03.980982+0000","last_peered":"2024-04-25T15:59:03.980982+0000","last_clean":"2024-04-25T15:59:03.980982+0000","last_became_active":"2024-04-25T15:59:03.003299+0000","last_became_peered":"2024-04-25T15:59:03.003299+0000","last_unstale":"2024-04-25T15:59:03.980982+0000","last_undegraded":"2024-04-25T15:59:03.980982+0000","last_fullsized":"2024-04-25T15:59:03.980982+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:46.842345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382637+0000","last_change":"2024-04-25T15:59:02.998591+0000","last_active":"2024-04-25T15:59:04.382637+0000","last_peered":"2024-04-25T15:59:04.382637+0000","last_clean":"2024-04-25T15:59:04.382637+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T15:59:04.382637+0000","last_undegraded":"2024-04-25T15:59:04.382637+0000","last_fullsized":"2024-04-25T15:59:04.382637+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.444499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501385+0000","last_change":"2024-04-25T15:59:03.006225+0000","last_active":"2024-04-25T15:59:04.501385+0000","last_peered":"2024-04-25T15:59:04.501385+0000","last_clean":"2024-04-25T15:59:04.501385+0000","last_became_active":"2024-04-25T15:59:03.005883+0000","last_became_peered":"2024-04-25T15:59:03.005883+0000","last_unstale":"2024-04-25T15:59:04.501385+0000","last_undegraded":"2024-04-25T15:59:04.501385+0000","last_fullsized":"2024-04-25T15:59:04.501385+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:17.773196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383942+0000","last_change":"2024-04-25T15:59:03.003605+0000","last_active":"2024-04-25T15:59:04.383942+0000","last_peered":"2024-04-25T15:59:04.383942+0000","last_clean":"2024-04-25T15:59:04.383942+0000","last_became_active":"2024-04-25T15:59:03.003271+0000","last_became_peered":"2024-04-25T15:59:03.003271+0000","last_unstale":"2024-04-25T15:59:04.383942+0000","last_undegraded":"2024-04-25T15:59:04.383942+0000","last_fullsized":"2024-04-25T15:59:04.383942+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:28.576407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981256+0000","last_change":"2024-04-25T15:59:03.006378+0000","last_active":"2024-04-25T15:59:03.981256+0000","last_peered":"2024-04-25T15:59:03.981256+0000","last_clean":"2024-04-25T15:59:03.981256+0000","last_became_active":"2024-04-25T15:59:03.006032+0000","last_became_peered":"2024-04-25T15:59:03.006032+0000","last_unstale":"2024-04-25T15:59:03.981256+0000","last_undegraded":"2024-04-25T15:59:03.981256+0000","last_fullsized":"2024-04-25T15:59:03.981256+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:22:41.473986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383595+0000","last_change":"2024-04-25T15:59:02.999307+0000","last_active":"2024-04-25T15:59:04.383595+0000","last_peered":"2024-04-25T15:59:04.383595+0000","last_clean":"2024-04-25T15:59:04.383595+0000","last_became_active":"2024-04-25T15:59:02.998976+0000","last_became_peered":"2024-04-25T15:59:02.998976+0000","last_unstale":"2024-04-25T15:59:04.383595+0000","last_undegraded":"2024-04-25T15:59:04.383595+0000","last_fullsized":"2024-04-25T15:59:04.383595+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:29.626440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383829+0000","last_change":"2024-04-25T15:59:03.005208+0000","last_active":"2024-04-25T15:59:04.383829+0000","last_peered":"2024-04-25T15:59:04.383829+0000","last_clean":"2024-04-25T15:59:04.383829+0000","last_became_active":"2024-04-25T15:59:03.004869+0000","last_became_peered":"2024-04-25T15:59:03.004869+0000","last_unstale":"2024-04-25T15:59:04.383829+0000","last_undegraded":"2024-04-25T15:59:04.383829+0000","last_fullsized":"2024-04-25T15:59:04.383829+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:00:24.040763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382668+0000","last_change":"2024-04-25T15:59:02.999924+0000","last_active":"2024-04-25T15:59:04.382668+0000","last_peered":"2024-04-25T15:59:04.382668+0000","last_clean":"2024-04-25T15:59:04.382668+0000","last_became_active":"2024-04-25T15:59:02.999573+0000","last_became_peered":"2024-04-25T15:59:02.999573+0000","last_unstale":"2024-04-25T15:59:04.382668+0000","last_undegraded":"2024-04-25T15:59:04.382668+0000","last_fullsized":"2024-04-25T15:59:04.382668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:24:38.759429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279494+0000","last_change":"2024-04-25T15:59:03.002160+0000","last_active":"2024-04-25T15:59:04.279494+0000","last_peered":"2024-04-25T15:59:04.279494+0000","last_clean":"2024-04-25T15:59:04.279494+0000","last_became_active":"2024-04-25T15:59:03.001855+0000","last_became_peered":"2024-04-25T15:59:03.001855+0000","last_unstale":"2024-04-25T15:59:04.279494+0000","last_undegraded":"2024-04-25T15:59:04.279494+0000","last_fullsized":"2024-04-25T15:59:04.279494+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:43.383851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279727+0000","last_change":"2024-04-25T15:59:02.995318+0000","last_active":"2024-04-25T15:59:04.279727+0000","last_peered":"2024-04-25T15:59:04.279727+0000","last_clean":"2024-04-25T15:59:04.279727+0000","last_became_active":"2024-04-25T15:59:02.994956+0000","last_became_peered":"2024-04-25T15:59:02.994956+0000","last_unstale":"2024-04-25T15:59:04.279727+0000","last_undegraded":"2024-04-25T15:59:04.279727+0000","last_fullsized":"2024-04-25T15:59:04.279727+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:59.659400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382480+0000","last_change":"2024-04-25T15:59:02.996485+0000","last_active":"2024-04-25T15:59:04.382480+0000","last_peered":"2024-04-25T15:59:04.382480+0000","last_clean":"2024-04-25T15:59:04.382480+0000","last_became_active":"2024-04-25T15:59:02.996011+0000","last_became_peered":"2024-04-25T15:59:02.996011+0000","last_unstale":"2024-04-25T15:59:04.382480+0000","last_undegraded":"2024-04-25T15:59:04.382480+0000","last_fullsized":"2024-04-25T15:59:04.382480+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:56.025254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981833+0000","last_change":"2024-04-25T15:59:02.998455+0000","last_active":"2024-04-25T15:59:03.981833+0000","last_peered":"2024-04-25T15:59:03.981833+0000","last_clean":"2024-04-25T15:59:03.981833+0000","last_became_active":"2024-04-25T15:59:02.998126+0000","last_became_peered":"2024-04-25T15:59:02.998126+0000","last_unstale":"2024-04-25T15:59:03.981833+0000","last_undegraded":"2024-04-25T15:59:03.981833+0000","last_fullsized":"2024-04-25T15:59:03.981833+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:20.342413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.982991+0000","last_change":"2024-04-25T15:59:02.998766+0000","last_active":"2024-04-25T15:59:03.982991+0000","last_peered":"2024-04-25T15:59:03.982991+0000","last_clean":"2024-04-25T15:59:03.982991+0000","last_became_active":"2024-04-25T15:59:02.998415+0000","last_became_peered":"2024-04-25T15:59:02.998415+0000","last_unstale":"2024-04-25T15:59:03.982991+0000","last_undegraded":"2024-04-25T15:59:03.982991+0000","last_fullsized":"2024-04-25T15:59:03.982991+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:55:05.717847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383074+0000","last_change":"2024-04-25T15:59:02.994982+0000","last_active":"2024-04-25T15:59:04.383074+0000","last_peered":"2024-04-25T15:59:04.383074+0000","last_clean":"2024-04-25T15:59:04.383074+0000","last_became_active":"2024-04-25T15:59:02.993928+0000","last_became_peered":"2024-04-25T15:59:02.993928+0000","last_unstale":"2024-04-25T15:59:04.383074+0000","last_undegraded":"2024-04-25T15:59:04.383074+0000","last_fullsized":"2024-04-25T15:59:04.383074+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:22:25.346541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'11","reported_seq":26,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:08.560563+0000","last_change":"2024-04-25T15:59:02.512828+0000","last_active":"2024-04-25T15:59:08.560563+0000","last_peered":"2024-04-25T15:59:08.560563+0000","last_clean":"2024-04-25T15:59:08.560563+0000","last_became_active":"2024-04-25T15:59:02.512291+0000","last_became_peered":"2024-04-25T15:59:02.512291+0000","last_unstale":"2024-04-25T15:59:08.560563+0000","last_undegraded":"2024-04-25T15:59:08.560563+0000","last_fullsized":"2024-04-25T15:59:08.560563+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:02.372140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501589+0000","last_change":"2024-04-25T15:59:03.008472+0000","last_active":"2024-04-25T15:59:04.501589+0000","last_peered":"2024-04-25T15:59:04.501589+0000","last_clean":"2024-04-25T15:59:04.501589+0000","last_became_active":"2024-04-25T15:59:03.008108+0000","last_became_peered":"2024-04-25T15:59:03.008108+0000","last_unstale":"2024-04-25T15:59:04.501589+0000","last_undegraded":"2024-04-25T15:59:04.501589+0000","last_fullsized":"2024-04-25T15:59:04.501589+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:40.087592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.113722+0000","last_change":"2024-04-25T15:59:02.517846+0000","last_active":"2024-04-25T15:59:06.113722+0000","last_peered":"2024-04-25T15:59:06.113722+0000","last_clean":"2024-04-25T15:59:06.113722+0000","last_became_active":"2024-04-25T15:59:02.517366+0000","last_became_peered":"2024-04-25T15:59:02.517366+0000","last_unstale":"2024-04-25T15:59:06.113722+0000","last_undegraded":"2024-04-25T15:59:06.113722+0000","last_fullsized":"2024-04-25T15:59:06.113722+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:29.620827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279820+0000","last_change":"2024-04-25T15:59:03.005202+0000","last_active":"2024-04-25T15:59:04.279820+0000","last_peered":"2024-04-25T15:59:04.279820+0000","last_clean":"2024-04-25T15:59:04.279820+0000","last_became_active":"2024-04-25T15:59:03.003447+0000","last_became_peered":"2024-04-25T15:59:03.003447+0000","last_unstale":"2024-04-25T15:59:04.279820+0000","last_undegraded":"2024-04-25T15:59:04.279820+0000","last_fullsized":"2024-04-25T15:59:04.279820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.740016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"74'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383447+0000","last_change":"2024-04-25T15:59:02.513701+0000","last_active":"2024-04-25T15:59:04.383447+0000","last_peered":"2024-04-25T15:59:04.383447+0000","last_clean":"2024-04-25T15:59:04.383447+0000","last_became_active":"2024-04-25T15:59:02.513361+0000","last_became_peered":"2024-04-25T15:59:02.513361+0000","last_unstale":"2024-04-25T15:59:04.383447+0000","last_undegraded":"2024-04-25T15:59:04.383447+0000","last_fullsized":"2024-04-25T15:59:04.383447+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:59:46.113568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383168+0000","last_change":"2024-04-25T15:59:03.004210+0000","last_active":"2024-04-25T15:59:04.383168+0000","last_peered":"2024-04-25T15:59:04.383168+0000","last_clean":"2024-04-25T15:59:04.383168+0000","last_became_active":"2024-04-25T15:59:03.002769+0000","last_became_peered":"2024-04-25T15:59:03.002769+0000","last_unstale":"2024-04-25T15:59:04.383168+0000","last_undegraded":"2024-04-25T15:59:04.383168+0000","last_fullsized":"2024-04-25T15:59:04.383168+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:50.608327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"74'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383795+0000","last_change":"2024-04-25T15:59:02.506199+0000","last_active":"2024-04-25T15:59:04.383795+0000","last_peered":"2024-04-25T15:59:04.383795+0000","last_clean":"2024-04-25T15:59:04.383795+0000","last_became_active":"2024-04-25T15:59:02.505847+0000","last_became_peered":"2024-04-25T15:59:02.505847+0000","last_unstale":"2024-04-25T15:59:04.383795+0000","last_undegraded":"2024-04-25T15:59:04.383795+0000","last_fullsized":"2024-04-25T15:59:04.383795+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:35.227112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383483+0000","last_change":"2024-04-25T15:59:03.011367+0000","last_active":"2024-04-25T15:59:04.383483+0000","last_peered":"2024-04-25T15:59:04.383483+0000","last_clean":"2024-04-25T15:59:04.383483+0000","last_became_active":"2024-04-25T15:59:03.011012+0000","last_became_peered":"2024-04-25T15:59:03.011012+0000","last_unstale":"2024-04-25T15:59:04.383483+0000","last_undegraded":"2024-04-25T15:59:04.383483+0000","last_fullsized":"2024-04-25T15:59:04.383483+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:52:26.434883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'5","reported_seq":20,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.538819+0000","last_change":"2024-04-25T15:59:02.507763+0000","last_active":"2024-04-25T15:59:06.538819+0000","last_peered":"2024-04-25T15:59:06.538819+0000","last_clean":"2024-04-25T15:59:06.538819+0000","last_became_active":"2024-04-25T15:59:02.507338+0000","last_became_peered":"2024-04-25T15:59:02.507338+0000","last_unstale":"2024-04-25T15:59:06.538819+0000","last_undegraded":"2024-04-25T15:59:06.538819+0000","last_fullsized":"2024-04-25T15:59:06.538819+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:31.710709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383638+0000","last_change":"2024-04-25T15:59:03.002590+0000","last_active":"2024-04-25T15:59:04.383638+0000","last_peered":"2024-04-25T15:59:04.383638+0000","last_clean":"2024-04-25T15:59:04.383638+0000","last_became_active":"2024-04-25T15:59:03.002262+0000","last_became_peered":"2024-04-25T15:59:03.002262+0000","last_unstale":"2024-04-25T15:59:04.383638+0000","last_undegraded":"2024-04-25T15:59:04.383638+0000","last_fullsized":"2024-04-25T15:59:04.383638+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:48.251484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"74'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.980758+0000","last_change":"2024-04-25T15:59:02.518853+0000","last_active":"2024-04-25T15:59:03.980758+0000","last_peered":"2024-04-25T15:59:03.980758+0000","last_clean":"2024-04-25T15:59:03.980758+0000","last_became_active":"2024-04-25T15:59:02.518505+0000","last_became_peered":"2024-04-25T15:59:02.518505+0000","last_unstale":"2024-04-25T15:59:03.980758+0000","last_undegraded":"2024-04-25T15:59:03.980758+0000","last_fullsized":"2024-04-25T15:59:03.980758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:01:56.282245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383040+0000","last_change":"2024-04-25T15:59:03.009733+0000","last_active":"2024-04-25T15:59:04.383040+0000","last_peered":"2024-04-25T15:59:04.383040+0000","last_clean":"2024-04-25T15:59:04.383040+0000","last_became_active":"2024-04-25T15:59:03.009251+0000","last_became_peered":"2024-04-25T15:59:03.009251+0000","last_unstale":"2024-04-25T15:59:04.383040+0000","last_undegraded":"2024-04-25T15:59:04.383040+0000","last_fullsized":"2024-04-25T15:59:04.383040+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:22.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"75'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:07.552825+0000","last_change":"2024-04-25T15:59:02.512493+0000","last_active":"2024-04-25T15:59:07.552825+0000","last_peered":"2024-04-25T15:59:07.552825+0000","last_clean":"2024-04-25T15:59:07.552825+0000","last_became_active":"2024-04-25T15:59:02.512103+0000","last_became_peered":"2024-04-25T15:59:02.512103+0000","last_unstale":"2024-04-25T15:59:07.552825+0000","last_undegraded":"2024-04-25T15:59:07.552825+0000","last_fullsized":"2024-04-25T15:59:07.552825+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:29.457755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501458+0000","last_change":"2024-04-25T15:59:03.001406+0000","last_active":"2024-04-25T15:59:04.501458+0000","last_peered":"2024-04-25T15:59:04.501458+0000","last_clean":"2024-04-25T15:59:04.501458+0000","last_became_active":"2024-04-25T15:59:02.999038+0000","last_became_peered":"2024-04-25T15:59:02.999038+0000","last_unstale":"2024-04-25T15:59:04.501458+0000","last_undegraded":"2024-04-25T15:59:04.501458+0000","last_fullsized":"2024-04-25T15:59:04.501458+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:56.354821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"75'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382541+0000","last_change":"2024-04-25T15:59:02.509619+0000","last_active":"2024-04-25T15:59:04.382541+0000","last_peered":"2024-04-25T15:59:04.382541+0000","last_clean":"2024-04-25T15:59:04.382541+0000","last_became_active":"2024-04-25T15:59:02.509254+0000","last_became_peered":"2024-04-25T15:59:02.509254+0000","last_unstale":"2024-04-25T15:59:04.382541+0000","last_undegraded":"2024-04-25T15:59:04.382541+0000","last_fullsized":"2024-04-25T15:59:04.382541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:00.382581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.982919+0000","last_change":"2024-04-25T15:59:03.000009+0000","last_active":"2024-04-25T15:59:03.982919+0000","last_peered":"2024-04-25T15:59:03.982919+0000","last_clean":"2024-04-25T15:59:03.982919+0000","last_became_active":"2024-04-25T15:59:02.999662+0000","last_became_peered":"2024-04-25T15:59:02.999662+0000","last_unstale":"2024-04-25T15:59:03.982919+0000","last_undegraded":"2024-04-25T15:59:03.982919+0000","last_fullsized":"2024-04-25T15:59:03.982919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:05:38.004819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.538155+0000","last_change":"2024-04-25T15:59:02.514429+0000","last_active":"2024-04-25T15:59:06.538155+0000","last_peered":"2024-04-25T15:59:06.538155+0000","last_clean":"2024-04-25T15:59:06.538155+0000","last_became_active":"2024-04-25T15:59:02.514144+0000","last_became_peered":"2024-04-25T15:59:02.514144+0000","last_unstale":"2024-04-25T15:59:06.538155+0000","last_undegraded":"2024-04-25T15:59:06.538155+0000","last_fullsized":"2024-04-25T15:59:06.538155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:45.144630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.384252+0000","last_change":"2024-04-25T15:59:03.008460+0000","last_active":"2024-04-25T15:59:04.384252+0000","last_peered":"2024-04-25T15:59:04.384252+0000","last_clean":"2024-04-25T15:59:04.384252+0000","last_became_active":"2024-04-25T15:59:03.008024+0000","last_became_peered":"2024-04-25T15:59:03.008024+0000","last_unstale":"2024-04-25T15:59:04.384252+0000","last_undegraded":"2024-04-25T15:59:04.384252+0000","last_fullsized":"2024-04-25T15:59:04.384252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:10.288760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"75'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.536658+0000","last_change":"2024-04-25T15:59:02.512494+0000","last_active":"2024-04-25T15:59:06.536658+0000","last_peered":"2024-04-25T15:59:06.536658+0000","last_clean":"2024-04-25T15:59:06.536658+0000","last_became_active":"2024-04-25T15:59:02.511182+0000","last_became_peered":"2024-04-25T15:59:02.511182+0000","last_unstale":"2024-04-25T15:59:06.536658+0000","last_undegraded":"2024-04-25T15:59:06.536658+0000","last_fullsized":"2024-04-25T15:59:06.536658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:54.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279861+0000","last_change":"2024-04-25T15:59:03.003848+0000","last_active":"2024-04-25T15:59:04.279861+0000","last_peered":"2024-04-25T15:59:04.279861+0000","last_clean":"2024-04-25T15:59:04.279861+0000","last_became_active":"2024-04-25T15:59:03.003512+0000","last_became_peered":"2024-04-25T15:59:03.003512+0000","last_unstale":"2024-04-25T15:59:04.279861+0000","last_undegraded":"2024-04-25T15:59:04.279861+0000","last_fullsized":"2024-04-25T15:59:04.279861+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:54.667054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383573+0000","last_change":"2024-04-25T15:59:02.509343+0000","last_active":"2024-04-25T15:59:04.383573+0000","last_peered":"2024-04-25T15:59:04.383573+0000","last_clean":"2024-04-25T15:59:04.383573+0000","last_became_active":"2024-04-25T15:59:02.508980+0000","last_became_peered":"2024-04-25T15:59:02.508980+0000","last_unstale":"2024-04-25T15:59:04.383573+0000","last_undegraded":"2024-04-25T15:59:04.383573+0000","last_fullsized":"2024-04-25T15:59:04.383573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:54.344717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383629+0000","last_change":"2024-04-25T15:59:02.999862+0000","last_active":"2024-04-25T15:59:04.383629+0000","last_peered":"2024-04-25T15:59:04.383629+0000","last_clean":"2024-04-25T15:59:04.383629+0000","last_became_active":"2024-04-25T15:59:02.999472+0000","last_became_peered":"2024-04-25T15:59:02.999472+0000","last_unstale":"2024-04-25T15:59:04.383629+0000","last_undegraded":"2024-04-25T15:59:04.383629+0000","last_fullsized":"2024-04-25T15:59:04.383629+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:46.165045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"75'8","reported_seq":23,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.536992+0000","last_change":"2024-04-25T15:59:02.508949+0000","last_active":"2024-04-25T15:59:06.536992+0000","last_peered":"2024-04-25T15:59:06.536992+0000","last_clean":"2024-04-25T15:59:06.536992+0000","last_became_active":"2024-04-25T15:59:02.508564+0000","last_became_peered":"2024-04-25T15:59:02.508564+0000","last_unstale":"2024-04-25T15:59:06.536992+0000","last_undegraded":"2024-04-25T15:59:06.536992+0000","last_fullsized":"2024-04-25T15:59:06.536992+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:12.590193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383192+0000","last_change":"2024-04-25T15:59:02.998575+0000","last_active":"2024-04-25T15:59:04.383192+0000","last_peered":"2024-04-25T15:59:04.383192+0000","last_clean":"2024-04-25T15:59:04.383192+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T15:59:04.383192+0000","last_undegraded":"2024-04-25T15:59:04.383192+0000","last_fullsized":"2024-04-25T15:59:04.383192+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:40.640468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.537055+0000","last_change":"2024-04-25T15:59:02.519110+0000","last_active":"2024-04-25T15:59:06.537055+0000","last_peered":"2024-04-25T15:59:06.537055+0000","last_clean":"2024-04-25T15:59:06.537055+0000","last_became_active":"2024-04-25T15:59:02.518373+0000","last_became_peered":"2024-04-25T15:59:02.518373+0000","last_unstale":"2024-04-25T15:59:06.537055+0000","last_undegraded":"2024-04-25T15:59:06.537055+0000","last_fullsized":"2024-04-25T15:59:06.537055+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:27.840296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382488+0000","last_change":"2024-04-25T15:59:02.991002+0000","last_active":"2024-04-25T15:59:04.382488+0000","last_peered":"2024-04-25T15:59:04.382488+0000","last_clean":"2024-04-25T15:59:04.382488+0000","last_became_active":"2024-04-25T15:59:02.990605+0000","last_became_peered":"2024-04-25T15:59:02.990605+0000","last_unstale":"2024-04-25T15:59:04.382488+0000","last_undegraded":"2024-04-25T15:59:04.382488+0000","last_fullsized":"2024-04-25T15:59:04.382488+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:22:00.723255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.538706+0000","last_change":"2024-04-25T15:59:02.509809+0000","last_active":"2024-04-25T15:59:06.538706+0000","last_peered":"2024-04-25T15:59:06.538706+0000","last_clean":"2024-04-25T15:59:06.538706+0000","last_became_active":"2024-04-25T15:59:02.509370+0000","last_became_peered":"2024-04-25T15:59:02.509370+0000","last_unstale":"2024-04-25T15:59:06.538706+0000","last_undegraded":"2024-04-25T15:59:06.538706+0000","last_fullsized":"2024-04-25T15:59:06.538706+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:24.416072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383094+0000","last_change":"2024-04-25T15:59:02.999836+0000","last_active":"2024-04-25T15:59:04.383094+0000","last_peered":"2024-04-25T15:59:04.383094+0000","last_clean":"2024-04-25T15:59:04.383094+0000","last_became_active":"2024-04-25T15:59:02.997770+0000","last_became_peered":"2024-04-25T15:59:02.997770+0000","last_unstale":"2024-04-25T15:59:04.383094+0000","last_undegraded":"2024-04-25T15:59:04.383094+0000","last_fullsized":"2024-04-25T15:59:04.383094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:30.953339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:07.551268+0000","last_change":"2024-04-25T15:59:02.510465+0000","last_active":"2024-04-25T15:59:07.551268+0000","last_peered":"2024-04-25T15:59:07.551268+0000","last_clean":"2024-04-25T15:59:07.551268+0000","last_became_active":"2024-04-25T15:59:02.510067+0000","last_became_peered":"2024-04-25T15:59:02.510067+0000","last_unstale":"2024-04-25T15:59:07.551268+0000","last_undegraded":"2024-04-25T15:59:07.551268+0000","last_fullsized":"2024-04-25T15:59:07.551268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:40.739358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382786+0000","last_change":"2024-04-25T15:59:03.000028+0000","last_active":"2024-04-25T15:59:04.382786+0000","last_peered":"2024-04-25T15:59:04.382786+0000","last_clean":"2024-04-25T15:59:04.382786+0000","last_became_active":"2024-04-25T15:59:02.999704+0000","last_became_peered":"2024-04-25T15:59:02.999704+0000","last_unstale":"2024-04-25T15:59:04.382786+0000","last_undegraded":"2024-04-25T15:59:04.382786+0000","last_fullsized":"2024-04-25T15:59:04.382786+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:55.015085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'10","reported_seq":25,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.536023+0000","last_change":"2024-04-25T15:59:02.514442+0000","last_active":"2024-04-25T15:59:06.536023+0000","last_peered":"2024-04-25T15:59:06.536023+0000","last_clean":"2024-04-25T15:59:06.536023+0000","last_became_active":"2024-04-25T15:59:02.513951+0000","last_became_peered":"2024-04-25T15:59:02.513951+0000","last_unstale":"2024-04-25T15:59:06.536023+0000","last_undegraded":"2024-04-25T15:59:06.536023+0000","last_fullsized":"2024-04-25T15:59:06.536023+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:56.443999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382416+0000","last_change":"2024-04-25T15:59:03.003327+0000","last_active":"2024-04-25T15:59:04.382416+0000","last_peered":"2024-04-25T15:59:04.382416+0000","last_clean":"2024-04-25T15:59:04.382416+0000","last_became_active":"2024-04-25T15:59:03.002781+0000","last_became_peered":"2024-04-25T15:59:03.002781+0000","last_unstale":"2024-04-25T15:59:04.382416+0000","last_undegraded":"2024-04-25T15:59:04.382416+0000","last_fullsized":"2024-04-25T15:59:04.382416+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:47:01.816897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"74'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383817+0000","last_change":"2024-04-25T15:59:02.512836+0000","last_active":"2024-04-25T15:59:04.383817+0000","last_peered":"2024-04-25T15:59:04.383817+0000","last_clean":"2024-04-25T15:59:04.383817+0000","last_became_active":"2024-04-25T15:59:02.512294+0000","last_became_peered":"2024-04-25T15:59:02.512294+0000","last_unstale":"2024-04-25T15:59:04.383817+0000","last_undegraded":"2024-04-25T15:59:04.383817+0000","last_fullsized":"2024-04-25T15:59:04.383817+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:30:07.322880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981695+0000","last_change":"2024-04-25T15:59:03.004639+0000","last_active":"2024-04-25T15:59:03.981695+0000","last_peered":"2024-04-25T15:59:03.981695+0000","last_clean":"2024-04-25T15:59:03.981695+0000","last_became_active":"2024-04-25T15:59:03.004260+0000","last_became_peered":"2024-04-25T15:59:03.004260+0000","last_unstale":"2024-04-25T15:59:03.981695+0000","last_undegraded":"2024-04-25T15:59:03.981695+0000","last_fullsized":"2024-04-25T15:59:03.981695+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:00.450758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.115544+0000","last_change":"2024-04-25T15:59:02.509716+0000","last_active":"2024-04-25T15:59:04.115544+0000","last_peered":"2024-04-25T15:59:04.115544+0000","last_clean":"2024-04-25T15:59:04.115544+0000","last_became_active":"2024-04-25T15:59:02.509303+0000","last_became_peered":"2024-04-25T15:59:02.509303+0000","last_unstale":"2024-04-25T15:59:04.115544+0000","last_undegraded":"2024-04-25T15:59:04.115544+0000","last_fullsized":"2024-04-25T15:59:04.115544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:34.037590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981147+0000","last_change":"2024-04-25T15:59:03.005141+0000","last_active":"2024-04-25T15:59:03.981147+0000","last_peered":"2024-04-25T15:59:03.981147+0000","last_clean":"2024-04-25T15:59:03.981147+0000","last_became_active":"2024-04-25T15:59:03.004760+0000","last_became_peered":"2024-04-25T15:59:03.004760+0000","last_unstale":"2024-04-25T15:59:03.981147+0000","last_undegraded":"2024-04-25T15:59:03.981147+0000","last_fullsized":"2024-04-25T15:59:03.981147+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:14.923126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382710+0000","last_change":"2024-04-25T15:59:02.512779+0000","last_active":"2024-04-25T15:59:04.382710+0000","last_peered":"2024-04-25T15:59:04.382710+0000","last_clean":"2024-04-25T15:59:04.382710+0000","last_became_active":"2024-04-25T15:59:02.511240+0000","last_became_peered":"2024-04-25T15:59:02.511240+0000","last_unstale":"2024-04-25T15:59:04.382710+0000","last_undegraded":"2024-04-25T15:59:04.382710+0000","last_fullsized":"2024-04-25T15:59:04.382710+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:51.703751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981159+0000","last_change":"2024-04-25T15:59:03.004841+0000","last_active":"2024-04-25T15:59:03.981159+0000","last_peered":"2024-04-25T15:59:03.981159+0000","last_clean":"2024-04-25T15:59:03.981159+0000","last_became_active":"2024-04-25T15:59:03.004488+0000","last_became_peered":"2024-04-25T15:59:03.004488+0000","last_unstale":"2024-04-25T15:59:03.981159+0000","last_undegraded":"2024-04-25T15:59:03.981159+0000","last_fullsized":"2024-04-25T15:59:03.981159+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:07.672312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382598+0000","last_change":"2024-04-25T15:59:03.001296+0000","last_active":"2024-04-25T15:59:04.382598+0000","last_peered":"2024-04-25T15:59:04.382598+0000","last_clean":"2024-04-25T15:59:04.382598+0000","last_became_active":"2024-04-25T15:59:03.000568+0000","last_became_peered":"2024-04-25T15:59:03.000568+0000","last_unstale":"2024-04-25T15:59:04.382598+0000","last_undegraded":"2024-04-25T15:59:04.382598+0000","last_fullsized":"2024-04-25T15:59:04.382598+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:51.335849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:07.552145+0000","last_change":"2024-04-25T15:59:02.513943+0000","last_active":"2024-04-25T15:59:07.552145+0000","last_peered":"2024-04-25T15:59:07.552145+0000","last_clean":"2024-04-25T15:59:07.552145+0000","last_became_active":"2024-04-25T15:59:02.513595+0000","last_became_peered":"2024-04-25T15:59:02.513595+0000","last_unstale":"2024-04-25T15:59:07.552145+0000","last_undegraded":"2024-04-25T15:59:07.552145+0000","last_fullsized":"2024-04-25T15:59:07.552145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:50.720333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501669+0000","last_change":"2024-04-25T15:59:03.008561+0000","last_active":"2024-04-25T15:59:04.501669+0000","last_peered":"2024-04-25T15:59:04.501669+0000","last_clean":"2024-04-25T15:59:04.501669+0000","last_became_active":"2024-04-25T15:59:03.008134+0000","last_became_peered":"2024-04-25T15:59:03.008134+0000","last_unstale":"2024-04-25T15:59:04.501669+0000","last_undegraded":"2024-04-25T15:59:04.501669+0000","last_fullsized":"2024-04-25T15:59:04.501669+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:43.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.537970+0000","last_change":"2024-04-25T15:59:02.510731+0000","last_active":"2024-04-25T15:59:06.537970+0000","last_peered":"2024-04-25T15:59:06.537970+0000","last_clean":"2024-04-25T15:59:06.537970+0000","last_became_active":"2024-04-25T15:59:02.510356+0000","last_became_peered":"2024-04-25T15:59:02.510356+0000","last_unstale":"2024-04-25T15:59:06.537970+0000","last_undegraded":"2024-04-25T15:59:06.537970+0000","last_fullsized":"2024-04-25T15:59:06.537970+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:33.066858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":90,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383010+0000","last_change":"2024-04-25T15:57:47.988899+0000","last_active":"2024-04-25T15:59:04.383010+0000","last_peered":"2024-04-25T15:59:04.383010+0000","last_clean":"2024-04-25T15:59:04.383010+0000","last_became_active":"2024-04-25T15:57:47.982613+0000","last_became_peered":"2024-04-25T15:57:47.982613+0000","last_unstale":"2024-04-25T15:59:04.383010+0000","last_undegraded":"2024-04-25T15:59:04.383010+0000","last_fullsized":"2024-04-25T15:59:04.383010+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_clean_scrub_stamp":"2024-04-25T15:54:18.013489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:39.323767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382339+0000","last_change":"2024-04-25T15:59:02.998360+0000","last_active":"2024-04-25T15:59:04.382339+0000","last_peered":"2024-04-25T15:59:04.382339+0000","last_clean":"2024-04-25T15:59:04.382339+0000","last_became_active":"2024-04-25T15:59:02.998048+0000","last_became_peered":"2024-04-25T15:59:02.998048+0000","last_unstale":"2024-04-25T15:59:04.382339+0000","last_undegraded":"2024-04-25T15:59:04.382339+0000","last_fullsized":"2024-04-25T15:59:04.382339+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:01.524608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"75'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383319+0000","last_change":"2024-04-25T15:59:02.515769+0000","last_active":"2024-04-25T15:59:04.383319+0000","last_peered":"2024-04-25T15:59:04.383319+0000","last_clean":"2024-04-25T15:59:04.383319+0000","last_became_active":"2024-04-25T15:59:02.515467+0000","last_became_peered":"2024-04-25T15:59:02.515467+0000","last_unstale":"2024-04-25T15:59:04.383319+0000","last_undegraded":"2024-04-25T15:59:04.383319+0000","last_fullsized":"2024-04-25T15:59:04.383319+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:59:50.107475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"75'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383304+0000","last_change":"2024-04-25T15:59:02.512701+0000","last_active":"2024-04-25T15:59:04.383304+0000","last_peered":"2024-04-25T15:59:04.383304+0000","last_clean":"2024-04-25T15:59:04.383304+0000","last_became_active":"2024-04-25T15:59:02.512352+0000","last_became_peered":"2024-04-25T15:59:02.512352+0000","last_unstale":"2024-04-25T15:59:04.383304+0000","last_undegraded":"2024-04-25T15:59:04.383304+0000","last_fullsized":"2024-04-25T15:59:04.383304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:31.772382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382591+0000","last_change":"2024-04-25T15:59:02.989698+0000","last_active":"2024-04-25T15:59:04.382591+0000","last_peered":"2024-04-25T15:59:04.382591+0000","last_clean":"2024-04-25T15:59:04.382591+0000","last_became_active":"2024-04-25T15:59:02.989284+0000","last_became_peered":"2024-04-25T15:59:02.989284+0000","last_unstale":"2024-04-25T15:59:04.382591+0000","last_undegraded":"2024-04-25T15:59:04.382591+0000","last_fullsized":"2024-04-25T15:59:04.382591+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:12.501738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"75'7","reported_seq":22,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:08.561453+0000","last_change":"2024-04-25T15:59:02.517272+0000","last_active":"2024-04-25T15:59:08.561453+0000","last_peered":"2024-04-25T15:59:08.561453+0000","last_clean":"2024-04-25T15:59:08.561453+0000","last_became_active":"2024-04-25T15:59:02.516370+0000","last_became_peered":"2024-04-25T15:59:02.516370+0000","last_unstale":"2024-04-25T15:59:08.561453+0000","last_undegraded":"2024-04-25T15:59:08.561453+0000","last_fullsized":"2024-04-25T15:59:08.561453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:31.139689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501486+0000","last_change":"2024-04-25T15:59:03.003111+0000","last_active":"2024-04-25T15:59:04.501486+0000","last_peered":"2024-04-25T15:59:04.501486+0000","last_clean":"2024-04-25T15:59:04.501486+0000","last_became_active":"2024-04-25T15:59:03.002766+0000","last_became_peered":"2024-04-25T15:59:03.002766+0000","last_unstale":"2024-04-25T15:59:04.501486+0000","last_undegraded":"2024-04-25T15:59:04.501486+0000","last_fullsized":"2024-04-25T15:59:04.501486+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:39.866119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.537203+0000","last_change":"2024-04-25T15:59:02.521970+0000","last_active":"2024-04-25T15:59:06.537203+0000","last_peered":"2024-04-25T15:59:06.537203+0000","last_clean":"2024-04-25T15:59:06.537203+0000","last_became_active":"2024-04-25T15:59:02.521650+0000","last_became_peered":"2024-04-25T15:59:02.521650+0000","last_unstale":"2024-04-25T15:59:06.537203+0000","last_undegraded":"2024-04-25T15:59:06.537203+0000","last_fullsized":"2024-04-25T15:59:06.537203+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:32.996868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279844+0000","last_change":"2024-04-25T15:59:02.996358+0000","last_active":"2024-04-25T15:59:04.279844+0000","last_peered":"2024-04-25T15:59:04.279844+0000","last_clean":"2024-04-25T15:59:04.279844+0000","last_became_active":"2024-04-25T15:59:02.995988+0000","last_became_peered":"2024-04-25T15:59:02.995988+0000","last_unstale":"2024-04-25T15:59:04.279844+0000","last_undegraded":"2024-04-25T15:59:04.279844+0000","last_fullsized":"2024-04-25T15:59:04.279844+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:51.021841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"75'2","reported_seq":17,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.536492+0000","last_change":"2024-04-25T15:59:02.518330+0000","last_active":"2024-04-25T15:59:06.536492+0000","last_peered":"2024-04-25T15:59:06.536492+0000","last_clean":"2024-04-25T15:59:06.536492+0000","last_became_active":"2024-04-25T15:59:02.517978+0000","last_became_peered":"2024-04-25T15:59:02.517978+0000","last_unstale":"2024-04-25T15:59:06.536492+0000","last_undegraded":"2024-04-25T15:59:06.536492+0000","last_fullsized":"2024-04-25T15:59:06.536492+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:25.443963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382788+0000","last_change":"2024-04-25T15:59:03.000198+0000","last_active":"2024-04-25T15:59:04.382788+0000","last_peered":"2024-04-25T15:59:04.382788+0000","last_clean":"2024-04-25T15:59:04.382788+0000","last_became_active":"2024-04-25T15:59:02.999878+0000","last_became_peered":"2024-04-25T15:59:02.999878+0000","last_unstale":"2024-04-25T15:59:04.382788+0000","last_undegraded":"2024-04-25T15:59:04.382788+0000","last_fullsized":"2024-04-25T15:59:04.382788+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:37.966595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"75'7","reported_seq":22,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:09.257733+0000","last_change":"2024-04-25T15:59:02.511298+0000","last_active":"2024-04-25T15:59:09.257733+0000","last_peered":"2024-04-25T15:59:09.257733+0000","last_clean":"2024-04-25T15:59:09.257733+0000","last_became_active":"2024-04-25T15:59:02.510521+0000","last_became_peered":"2024-04-25T15:59:02.510521+0000","last_unstale":"2024-04-25T15:59:09.257733+0000","last_undegraded":"2024-04-25T15:59:09.257733+0000","last_fullsized":"2024-04-25T15:59:09.257733+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:00.807919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981476+0000","last_change":"2024-04-25T15:59:02.999212+0000","last_active":"2024-04-25T15:59:03.981476+0000","last_peered":"2024-04-25T15:59:03.981476+0000","last_clean":"2024-04-25T15:59:03.981476+0000","last_became_active":"2024-04-25T15:59:02.998881+0000","last_became_peered":"2024-04-25T15:59:02.998881+0000","last_unstale":"2024-04-25T15:59:03.981476+0000","last_undegraded":"2024-04-25T15:59:03.981476+0000","last_fullsized":"2024-04-25T15:59:03.981476+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:09.092196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382729+0000","last_change":"2024-04-25T15:59:02.517412+0000","last_active":"2024-04-25T15:59:04.382729+0000","last_peered":"2024-04-25T15:59:04.382729+0000","last_clean":"2024-04-25T15:59:04.382729+0000","last_became_active":"2024-04-25T15:59:02.517077+0000","last_became_peered":"2024-04-25T15:59:02.517077+0000","last_unstale":"2024-04-25T15:59:04.382729+0000","last_undegraded":"2024-04-25T15:59:04.382729+0000","last_fullsized":"2024-04-25T15:59:04.382729+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:00.161199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501479+0000","last_change":"2024-04-25T15:59:02.999704+0000","last_active":"2024-04-25T15:59:04.501479+0000","last_peered":"2024-04-25T15:59:04.501479+0000","last_clean":"2024-04-25T15:59:04.501479+0000","last_became_active":"2024-04-25T15:59:02.999381+0000","last_became_peered":"2024-04-25T15:59:02.999381+0000","last_unstale":"2024-04-25T15:59:04.501479+0000","last_undegraded":"2024-04-25T15:59:04.501479+0000","last_fullsized":"2024-04-25T15:59:04.501479+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:34:43.572698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"75'4","reported_seq":19,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:06.535710+0000","last_change":"2024-04-25T15:59:02.515912+0000","last_active":"2024-04-25T15:59:06.535710+0000","last_peered":"2024-04-25T15:59:06.535710+0000","last_clean":"2024-04-25T15:59:06.535710+0000","last_became_active":"2024-04-25T15:59:02.514949+0000","last_became_peered":"2024-04-25T15:59:02.514949+0000","last_unstale":"2024-04-25T15:59:06.535710+0000","last_undegraded":"2024-04-25T15:59:06.535710+0000","last_fullsized":"2024-04-25T15:59:06.535710+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:33.467223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383268+0000","last_change":"2024-04-25T15:59:02.988896+0000","last_active":"2024-04-25T15:59:04.383268+0000","last_peered":"2024-04-25T15:59:04.383268+0000","last_clean":"2024-04-25T15:59:04.383268+0000","last_became_active":"2024-04-25T15:59:02.988478+0000","last_became_peered":"2024-04-25T15:59:02.988478+0000","last_unstale":"2024-04-25T15:59:04.383268+0000","last_undegraded":"2024-04-25T15:59:04.383268+0000","last_fullsized":"2024-04-25T15:59:04.383268+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:47.014678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":15,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383305+0000","last_change":"2024-04-25T15:59:02.523031+0000","last_active":"2024-04-25T15:59:04.383305+0000","last_peered":"2024-04-25T15:59:04.383305+0000","last_clean":"2024-04-25T15:59:04.383305+0000","last_became_active":"2024-04-25T15:59:02.522675+0000","last_became_peered":"2024-04-25T15:59:02.522675+0000","last_unstale":"2024-04-25T15:59:04.383305+0000","last_undegraded":"2024-04-25T15:59:04.383305+0000","last_fullsized":"2024-04-25T15:59:04.383305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:09:45.277456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383885+0000","last_change":"2024-04-25T15:59:02.990507+0000","last_active":"2024-04-25T15:59:04.383885+0000","last_peered":"2024-04-25T15:59:04.383885+0000","last_clean":"2024-04-25T15:59:04.383885+0000","last_became_active":"2024-04-25T15:59:02.990160+0000","last_became_peered":"2024-04-25T15:59:02.990160+0000","last_unstale":"2024-04-25T15:59:04.383885+0000","last_undegraded":"2024-04-25T15:59:04.383885+0000","last_fullsized":"2024-04-25T15:59:04.383885+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:18:21.526155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"74'1","reported_seq":16,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383411+0000","last_change":"2024-04-25T15:59:02.510956+0000","last_active":"2024-04-25T15:59:04.383411+0000","last_peered":"2024-04-25T15:59:04.383411+0000","last_clean":"2024-04-25T15:59:04.383411+0000","last_became_active":"2024-04-25T15:59:02.510578+0000","last_became_peered":"2024-04-25T15:59:02.510578+0000","last_unstale":"2024-04-25T15:59:04.383411+0000","last_undegraded":"2024-04-25T15:59:04.383411+0000","last_fullsized":"2024-04-25T15:59:04.383411+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:11.512365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383323+0000","last_change":"2024-04-25T15:59:03.001122+0000","last_active":"2024-04-25T15:59:04.383323+0000","last_peered":"2024-04-25T15:59:04.383323+0000","last_clean":"2024-04-25T15:59:04.383323+0000","last_became_active":"2024-04-25T15:59:03.000672+0000","last_became_peered":"2024-04-25T15:59:03.000672+0000","last_unstale":"2024-04-25T15:59:04.383323+0000","last_undegraded":"2024-04-25T15:59:04.383323+0000","last_fullsized":"2024-04-25T15:59:04.383323+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:33.216434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'3","reported_seq":18,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382609+0000","last_change":"2024-04-25T15:59:02.520645+0000","last_active":"2024-04-25T15:59:04.382609+0000","last_peered":"2024-04-25T15:59:04.382609+0000","last_clean":"2024-04-25T15:59:04.382609+0000","last_became_active":"2024-04-25T15:59:02.520197+0000","last_became_peered":"2024-04-25T15:59:02.520197+0000","last_unstale":"2024-04-25T15:59:04.382609+0000","last_undegraded":"2024-04-25T15:59:04.382609+0000","last_fullsized":"2024-04-25T15:59:04.382609+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:20.120554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383762+0000","last_change":"2024-04-25T15:59:03.001668+0000","last_active":"2024-04-25T15:59:04.383762+0000","last_peered":"2024-04-25T15:59:04.383762+0000","last_clean":"2024-04-25T15:59:04.383762+0000","last_became_active":"2024-04-25T15:59:03.001333+0000","last_became_peered":"2024-04-25T15:59:03.001333+0000","last_unstale":"2024-04-25T15:59:04.383762+0000","last_undegraded":"2024-04-25T15:59:04.383762+0000","last_fullsized":"2024-04-25T15:59:04.383762+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:24.992476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981543+0000","last_change":"2024-04-25T15:59:03.005533+0000","last_active":"2024-04-25T15:59:03.981543+0000","last_peered":"2024-04-25T15:59:03.981543+0000","last_clean":"2024-04-25T15:59:03.981543+0000","last_became_active":"2024-04-25T15:59:03.004124+0000","last_became_peered":"2024-04-25T15:59:03.004124+0000","last_unstale":"2024-04-25T15:59:03.981543+0000","last_undegraded":"2024-04-25T15:59:03.981543+0000","last_fullsized":"2024-04-25T15:59:03.981543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:27:28.507065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382715+0000","last_change":"2024-04-25T15:59:02.993416+0000","last_active":"2024-04-25T15:59:04.382715+0000","last_peered":"2024-04-25T15:59:04.382715+0000","last_clean":"2024-04-25T15:59:04.382715+0000","last_became_active":"2024-04-25T15:59:02.993098+0000","last_became_peered":"2024-04-25T15:59:02.993098+0000","last_unstale":"2024-04-25T15:59:04.382715+0000","last_undegraded":"2024-04-25T15:59:04.382715+0000","last_fullsized":"2024-04-25T15:59:04.382715+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:46.859932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382660+0000","last_change":"2024-04-25T15:59:02.996858+0000","last_active":"2024-04-25T15:59:04.382660+0000","last_peered":"2024-04-25T15:59:04.382660+0000","last_clean":"2024-04-25T15:59:04.382660+0000","last_became_active":"2024-04-25T15:59:02.996530+0000","last_became_peered":"2024-04-25T15:59:02.996530+0000","last_unstale":"2024-04-25T15:59:04.382660+0000","last_undegraded":"2024-04-25T15:59:04.382660+0000","last_fullsized":"2024-04-25T15:59:04.382660+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:36.723948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.279720+0000","last_change":"2024-04-25T15:59:02.995942+0000","last_active":"2024-04-25T15:59:04.279720+0000","last_peered":"2024-04-25T15:59:04.279720+0000","last_clean":"2024-04-25T15:59:04.279720+0000","last_became_active":"2024-04-25T15:59:02.995567+0000","last_became_peered":"2024-04-25T15:59:02.995567+0000","last_unstale":"2024-04-25T15:59:04.279720+0000","last_undegraded":"2024-04-25T15:59:04.279720+0000","last_fullsized":"2024-04-25T15:59:04.279720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:32.692392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382613+0000","last_change":"2024-04-25T15:59:03.000326+0000","last_active":"2024-04-25T15:59:04.382613+0000","last_peered":"2024-04-25T15:59:04.382613+0000","last_clean":"2024-04-25T15:59:04.382613+0000","last_became_active":"2024-04-25T15:59:02.999961+0000","last_became_peered":"2024-04-25T15:59:02.999961+0000","last_unstale":"2024-04-25T15:59:04.382613+0000","last_undegraded":"2024-04-25T15:59:04.382613+0000","last_fullsized":"2024-04-25T15:59:04.382613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:10.631460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981619+0000","last_change":"2024-04-25T15:59:03.003625+0000","last_active":"2024-04-25T15:59:03.981619+0000","last_peered":"2024-04-25T15:59:03.981619+0000","last_clean":"2024-04-25T15:59:03.981619+0000","last_became_active":"2024-04-25T15:59:03.003280+0000","last_became_peered":"2024-04-25T15:59:03.003280+0000","last_unstale":"2024-04-25T15:59:03.981619+0000","last_undegraded":"2024-04-25T15:59:03.981619+0000","last_fullsized":"2024-04-25T15:59:03.981619+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:21:54.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383644+0000","last_change":"2024-04-25T15:59:02.994645+0000","last_active":"2024-04-25T15:59:04.383644+0000","last_peered":"2024-04-25T15:59:04.383644+0000","last_clean":"2024-04-25T15:59:04.383644+0000","last_became_active":"2024-04-25T15:59:02.993924+0000","last_became_peered":"2024-04-25T15:59:02.993924+0000","last_unstale":"2024-04-25T15:59:04.383644+0000","last_undegraded":"2024-04-25T15:59:04.383644+0000","last_fullsized":"2024-04-25T15:59:04.383644+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.644708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501509+0000","last_change":"2024-04-25T15:59:03.006156+0000","last_active":"2024-04-25T15:59:04.501509+0000","last_peered":"2024-04-25T15:59:04.501509+0000","last_clean":"2024-04-25T15:59:04.501509+0000","last_became_active":"2024-04-25T15:59:03.005795+0000","last_became_peered":"2024-04-25T15:59:03.005795+0000","last_unstale":"2024-04-25T15:59:04.501509+0000","last_undegraded":"2024-04-25T15:59:04.501509+0000","last_fullsized":"2024-04-25T15:59:04.501509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:23.523868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382418+0000","last_change":"2024-04-25T15:59:03.010404+0000","last_active":"2024-04-25T15:59:04.382418+0000","last_peered":"2024-04-25T15:59:04.382418+0000","last_clean":"2024-04-25T15:59:04.382418+0000","last_became_active":"2024-04-25T15:59:03.010100+0000","last_became_peered":"2024-04-25T15:59:03.010100+0000","last_unstale":"2024-04-25T15:59:04.382418+0000","last_undegraded":"2024-04-25T15:59:04.382418+0000","last_fullsized":"2024-04-25T15:59:04.382418+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:22.718842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.501455+0000","last_change":"2024-04-25T15:59:03.009240+0000","last_active":"2024-04-25T15:59:04.501455+0000","last_peered":"2024-04-25T15:59:04.501455+0000","last_clean":"2024-04-25T15:59:04.501455+0000","last_became_active":"2024-04-25T15:59:03.008723+0000","last_became_peered":"2024-04-25T15:59:03.008723+0000","last_unstale":"2024-04-25T15:59:04.501455+0000","last_undegraded":"2024-04-25T15:59:04.501455+0000","last_fullsized":"2024-04-25T15:59:04.501455+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:14.049934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.383782+0000","last_change":"2024-04-25T15:59:02.994359+0000","last_active":"2024-04-25T15:59:04.383782+0000","last_peered":"2024-04-25T15:59:04.383782+0000","last_clean":"2024-04-25T15:59:04.383782+0000","last_became_active":"2024-04-25T15:59:02.993990+0000","last_became_peered":"2024-04-25T15:59:02.993990+0000","last_unstale":"2024-04-25T15:59:04.383782+0000","last_undegraded":"2024-04-25T15:59:04.383782+0000","last_fullsized":"2024-04-25T15:59:04.383782+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:55.477414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:04.382793+0000","last_change":"2024-04-25T15:59:02.995510+0000","last_active":"2024-04-25T15:59:04.382793+0000","last_peered":"2024-04-25T15:59:04.382793+0000","last_clean":"2024-04-25T15:59:04.382793+0000","last_became_active":"2024-04-25T15:59:02.995008+0000","last_became_peered":"2024-04-25T15:59:02.995008+0000","last_unstale":"2024-04-25T15:59:04.382793+0000","last_undegraded":"2024-04-25T15:59:04.382793+0000","last_fullsized":"2024-04-25T15:59:04.382793+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:40.439615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":75,"state":"active+clean","last_fresh":"2024-04-25T15:59:03.981630+0000","last_change":"2024-04-25T15:59:03.005846+0000","last_active":"2024-04-25T15:59:03.981630+0000","last_peered":"2024-04-25T15:59:03.981630+0000","last_clean":"2024-04-25T15:59:03.981630+0000","last_became_active":"2024-04-25T15:59:03.003779+0000","last_became_peered":"2024-04-25T15:59:03.003779+0000","last_unstale":"2024-04-25T15:59:03.981630+0000","last_undegraded":"2024-04-25T15:59:03.981630+0000","last_fullsized":"2024-04-25T15:59:03.981630+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.698405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247542,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4528,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":118,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":13914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710732,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":798720,"data_stored":383762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906963,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":884736,"data_stored":976698,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070488,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":384580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266718,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":811008,"data_stored":384388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462947,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32020,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713132,"statfs":{"total":95995035648,"available":95962247168,"internally_reserved":0,"allocated":851968,"data_stored":974712,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626472,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":383852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":811008,"data_stored":385374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717324,"statfs":{"total":95995035648,"available":95966539776,"internally_reserved":0,"allocated":831488,"data_stored":385828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":815104,"data_stored":385408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182455,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":384426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986227,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":811008,"data_stored":384406,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822701,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32032,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713120,"statfs":{"total":95995035648,"available":95962234880,"internally_reserved":0,"allocated":876544,"data_stored":975606,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T15:59:12.172 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-25T15:59:12.173 INFO:tasks.cephadm:Setting up client nodes... 2024-04-25T15:59:12.173 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-25T15:59:12.455 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:12.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:12 smithi031 ceph-mon[29140]: pgmap v286: 97 pgs: 97 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 14 op/s 2024-04-25T15:59:12.836 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:12 smithi031 ceph-mon[29140]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:59:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:12 smithi063 ceph-mon[32413]: pgmap v286: 97 pgs: 97 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 14 op/s 2024-04-25T15:59:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:12 smithi063 ceph-mon[32413]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:59:12.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:12 smithi186 ceph-mon[33258]: pgmap v286: 97 pgs: 97 active+clean; 581 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 14 op/s 2024-04-25T15:59:12.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:12 smithi186 ceph-mon[33258]: from='client.14775 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:59:13.848 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:13 smithi031 ceph-mon[29140]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:59:13.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:13 smithi063 ceph-mon[32413]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:59:13.869 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:59:13 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: 2024-04-25T15:59:13.398+0000 7ff2c1c23700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-25T15:59:13.869 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 15:59:13 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: 2024-04-25T15:59:13.398+0000 7ff2c1c23700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-04-25T15:59:13.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:13 smithi186 ceph-mon[33258]: from='client.14781 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T15:59:14.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.276+0000 7f1f12772700 1 -- 172.21.15.31:0/4112659487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c1030e0 msgr2=0x7f1f0c1034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.276+0000 7f1f12772700 1 --2- 172.21.15.31:0/4112659487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c1030e0 0x7f1f0c1034f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009990 tx=0x7f1ef4009ca0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.276+0000 7f1f12772700 1 -- 172.21.15.31:0/4112659487 shutdown_connections 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.276+0000 7f1f12772700 1 --2- 172.21.15.31:0/4112659487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f0c104bb0 0x7f1f0c10b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.276+0000 7f1f12772700 1 --2- 172.21.15.31:0/4112659487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c1030e0 0x7f1f0c1034f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.276+0000 7f1f12772700 1 --2- 172.21.15.31:0/4112659487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f0c104210 0x7f1f0c104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.276+0000 7f1f12772700 1 -- 172.21.15.31:0/4112659487 >> 172.21.15.31:0/4112659487 conn(0x7f1f0c0fe740 msgr2=0x7f1f0c100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.276+0000 7f1f12772700 1 -- 172.21.15.31:0/4112659487 shutdown_connections 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.276+0000 7f1f12772700 1 -- 172.21.15.31:0/4112659487 wait complete. 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.277+0000 7f1f12772700 1 Processor -- start 2024-04-25T15:59:14.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.277+0000 7f1f12772700 1 -- start start 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.277+0000 7f1f12772700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f0c1030e0 0x7f1f0c1996d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.277+0000 7f1f12772700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f0c104210 0x7f1f0c199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.277+0000 7f1f12772700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c104bb0 0x7f1f0c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.277+0000 7f1f12772700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c10eb30 con 0x7f1f0c104bb0 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.277+0000 7f1f12772700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c10e9b0 con 0x7f1f0c1030e0 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.277+0000 7f1f12772700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f0c10ecb0 con 0x7f1f0c104210 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f10d0f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c104bb0 0x7f1f0c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f10d0f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c104bb0 0x7f1f0c1a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58738/0 (socket says 172.21.15.31:58738) 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f10d0f700 1 -- 172.21.15.31:0/3047312023 learned_addr learned my addr 172.21.15.31:0/3047312023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:14.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f0b7fe700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f0c104210 0x7f1f0c199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f0bfff700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f0c1030e0 0x7f1f0c1996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f10d0f700 1 -- 172.21.15.31:0/3047312023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f0c104210 msgr2=0x7f1f0c199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f10d0f700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f0c104210 0x7f1f0c199c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f10d0f700 1 -- 172.21.15.31:0/3047312023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f0c1030e0 msgr2=0x7f1f0c1996d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f10d0f700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f0c1030e0 0x7f1f0c1996d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f10d0f700 1 -- 172.21.15.31:0/3047312023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f0c1a1230 con 0x7f1f0c104bb0 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f10d0f700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c104bb0 0x7f1f0c1a0bb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1f04007ee0 tx=0x7f1f0400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f097fa700 1 -- 172.21.15.31:0/3047312023 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04004bb0 con 0x7f1f0c104bb0 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f097fa700 1 -- 172.21.15.31:0/3047312023 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f04004d50 con 0x7f1f0c104bb0 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f0c1a1520 con 0x7f1f0c104bb0 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.278+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f0c1a1a10 con 0x7f1f0c104bb0 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.279+0000 7f1f097fa700 1 -- 172.21.15.31:0/3047312023 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f04005920 con 0x7f1f0c104bb0 2024-04-25T15:59:14.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.279+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eec005360 con 0x7f1f0c104bb0 2024-04-25T15:59:14.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.281+0000 7f1f097fa700 1 -- 172.21.15.31:0/3047312023 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f1f0401a020 con 0x7f1f0c104bb0 2024-04-25T15:59:14.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.281+0000 7f1f097fa700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ef80afc10 0x7f1ef80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:14.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.282+0000 7f1f0bfff700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ef80afc10 0x7f1ef80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:14.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.282+0000 7f1f097fa700 1 -- 172.21.15.31:0/3047312023 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f1f040de930 con 0x7f1f0c104bb0 2024-04-25T15:59:14.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.282+0000 7f1f0bfff700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ef80afc10 0x7f1ef80b2120 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40038a0 tx=0x7f1ef401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:14.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.285+0000 7f1f097fa700 1 -- 172.21.15.31:0/3047312023 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f04010070 con 0x7f1f0c104bb0 2024-04-25T15:59:14.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.587+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f1eec005650 con 0x7f1f0c104bb0 2024-04-25T15:59:14.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.590+0000 7f1f097fa700 1 -- 172.21.15.31:0/3047312023 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f1f040a7460 con 0x7f1f0c104bb0 2024-04-25T15:59:14.591 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2024-04-25T15:59:14.591 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDSfSpm3eAkIxAAS/2jupUM7P/fUNQloHKn1Q== 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ef80afc10 msgr2=0x7f1ef80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ef80afc10 0x7f1ef80b2120 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40038a0 tx=0x7f1ef401a040 comp rx=0 tx=0).stop 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c104bb0 msgr2=0x7f1f0c1a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c104bb0 0x7f1f0c1a0bb0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1f04007ee0 tx=0x7f1f0400d3b0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 shutdown_connections 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1ef80afc10 0x7f1ef80b2120 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f0c1030e0 0x7f1f0c1996d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1f0c104bb0 0x7f1f0c1a0bb0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 --2- 172.21.15.31:0/3047312023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f0c104210 0x7f1f0c199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 >> 172.21.15.31:0/3047312023 conn(0x7f1f0c0fe740 msgr2=0x7f1f0c107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:14.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 shutdown_connections 2024-04-25T15:59:14.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:14.592+0000 7f1f12772700 1 -- 172.21.15.31:0/3047312023 wait complete. 2024-04-25T15:59:14.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:14 smithi063 ceph-mon[32413]: pgmap v287: 97 pgs: 97 active+clean; 621 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 9.5 KiB/s wr, 26 op/s 2024-04-25T15:59:14.897 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:14 smithi031 ceph-mon[29140]: pgmap v287: 97 pgs: 97 active+clean; 621 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 9.5 KiB/s wr, 26 op/s 2024-04-25T15:59:14.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:14 smithi186 ceph-mon[33258]: pgmap v287: 97 pgs: 97 active+clean; 621 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 9.5 KiB/s wr, 26 op/s 2024-04-25T15:59:15.418 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T15:59:15.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-25T15:59:15.418 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-25T15:59:15.456 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-25T15:59:15.456 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-25T15:59:15.456 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-25T15:59:15.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:15 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3047312023' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T15:59:15.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:15 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3047312023' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T15:59:15.745 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:15.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:15 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3047312023' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T15:59:15.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:15 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3047312023' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T15:59:15.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:15 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3047312023' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-25T15:59:15.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:15 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3047312023' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-25T15:59:16.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:16 smithi063 ceph-mon[32413]: pgmap v288: 97 pgs: 97 active+clean; 621 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 9.4 KiB/s wr, 25 op/s 2024-04-25T15:59:16.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:16 smithi186 ceph-mon[33258]: pgmap v288: 97 pgs: 97 active+clean; 621 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 9.4 KiB/s wr, 25 op/s 2024-04-25T15:59:16.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:16 smithi031 ceph-mon[29140]: pgmap v288: 97 pgs: 97 active+clean; 621 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 9.4 KiB/s wr, 25 op/s 2024-04-25T15:59:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.740+0000 7f45db293700 1 -- 172.21.15.31:0/2241346594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4103420 msgr2=0x7f45d4103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.740+0000 7f45db293700 1 --2- 172.21.15.31:0/2241346594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4103420 0x7f45d4103830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f45d00099c0 tx=0x7f45d0009cd0 comp rx=0 tx=0).stop 2024-04-25T15:59:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.740+0000 7f45db293700 1 -- 172.21.15.31:0/2241346594 shutdown_connections 2024-04-25T15:59:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.740+0000 7f45db293700 1 --2- 172.21.15.31:0/2241346594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45d4108650 0x7f45d410ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.740+0000 7f45db293700 1 --2- 172.21.15.31:0/2241346594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4103420 0x7f45d4103830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.740+0000 7f45db293700 1 --2- 172.21.15.31:0/2241346594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4107d00 0x7f45d4108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.740+0000 7f45db293700 1 -- 172.21.15.31:0/2241346594 >> 172.21.15.31:0/2241346594 conn(0x7f45d4078cd0 msgr2=0x7f45d4079110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.740+0000 7f45db293700 1 -- 172.21.15.31:0/2241346594 shutdown_connections 2024-04-25T15:59:17.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.740+0000 7f45db293700 1 -- 172.21.15.31:0/2241346594 wait complete. 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45db293700 1 Processor -- start 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45db293700 1 -- start start 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45db293700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4103420 0x7f45d419cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45db293700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45d4107d00 0x7f45d419d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45db293700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4108650 0x7f45d41a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45db293700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4113280 con 0x7f45d4108650 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45db293700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4113100 con 0x7f45d4107d00 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45db293700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45d4113400 con 0x7f45d4103420 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45d9830700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4108650 0x7f45d41a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45d9830700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4108650 0x7f45d41a1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58754/0 (socket says 172.21.15.31:58754) 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45d9830700 1 -- 172.21.15.31:0/4243493180 learned_addr learned my addr 172.21.15.31:0/4243493180 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45d882e700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45d4107d00 0x7f45d419d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.741+0000 7f45d902f700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4103420 0x7f45d419cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:17.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45d9830700 1 -- 172.21.15.31:0/4243493180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4103420 msgr2=0x7f45d419cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45d9830700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4103420 0x7f45d419cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45d9830700 1 -- 172.21.15.31:0/4243493180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45d4107d00 msgr2=0x7f45d419d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45d9830700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45d4107d00 0x7f45d419d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45d9830700 1 -- 172.21.15.31:0/4243493180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d41a1e10 con 0x7f45d4108650 2024-04-25T15:59:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45d9830700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4108650 0x7f45d41a1770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00ada0 tx=0x7f45cc00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45ca7fc700 1 -- 172.21.15.31:0/4243493180 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc017020 con 0x7f45d4108650 2024-04-25T15:59:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d41a2100 con 0x7f45d4108650 2024-04-25T15:59:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45ca7fc700 1 -- 172.21.15.31:0/4243493180 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc00cd80 con 0x7f45d4108650 2024-04-25T15:59:17.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d41a9a20 con 0x7f45d4108650 2024-04-25T15:59:17.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.742+0000 7f45ca7fc700 1 -- 172.21.15.31:0/4243493180 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc005260 con 0x7f45d4108650 2024-04-25T15:59:17.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.743+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d4051190 con 0x7f45d4108650 2024-04-25T15:59:17.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.745+0000 7f45ca7fc700 1 -- 172.21.15.31:0/4243493180 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f45cc00a090 con 0x7f45d4108650 2024-04-25T15:59:17.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.746+0000 7f45ca7fc700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45c00afc10 0x7f45c00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:17.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.746+0000 7f45d902f700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45c00afc10 0x7f45c00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:17.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.746+0000 7f45ca7fc700 1 -- 172.21.15.31:0/4243493180 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f45cc0de920 con 0x7f45d4108650 2024-04-25T15:59:17.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.746+0000 7f45d902f700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45c00afc10 0x7f45c00b2120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009990 tx=0x7f45d00036e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:17.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:17.749+0000 7f45ca7fc700 1 -- 172.21.15.31:0/4243493180 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45cc021020 con 0x7f45d4108650 2024-04-25T15:59:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.015+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f45d402e510 con 0x7f45d4108650 2024-04-25T15:59:18.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.021+0000 7f45ca7fc700 1 -- 172.21.15.31:0/4243493180 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+284477 (secure 0 0 0) 0x7f45cc0a7520 con 0x7f45d4108650 2024-04-25T15:59:18.025 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:18.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45c00afc10 msgr2=0x7f45c00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45c00afc10 0x7f45c00b2120 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009990 tx=0x7f45d00036e0 comp rx=0 tx=0).stop 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4108650 msgr2=0x7f45d41a1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4108650 0x7f45d41a1770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00ada0 tx=0x7f45cc00c330 comp rx=0 tx=0).stop 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 shutdown_connections 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f45c00afc10 0x7f45c00b2120 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f45d4108650 0x7f45d41a1770 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45d4103420 0x7f45d419cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 --2- 172.21.15.31:0/4243493180 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45d4107d00 0x7f45d419d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 >> 172.21.15.31:0/4243493180 conn(0x7f45d4078cd0 msgr2=0x7f45d4075b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 shutdown_connections 2024-04-25T15:59:18.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:18.024+0000 7f45db293700 1 -- 172.21.15.31:0/4243493180 wait complete. 2024-04-25T15:59:18.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:18 smithi031 ceph-mon[29140]: pgmap v289: 97 pgs: 97 active+clean; 626 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.2 KiB/s wr, 26 op/s 2024-04-25T15:59:18.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:18 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4243493180' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T15:59:18.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:18 smithi063 ceph-mon[32413]: pgmap v289: 97 pgs: 97 active+clean; 626 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.2 KiB/s wr, 26 op/s 2024-04-25T15:59:18.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:18 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4243493180' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T15:59:18.873 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":3984865329},{"type":"v1","addr":"172.21.15.31:6801","nonce":3984865329}]},"active_addr":"172.21.15.31:6801/3984865329","active_change":"2024-04-25T15:51:49.220007+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.31:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":989398132}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2463152784}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2663702827}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":186916348}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":751618652}]}]} 2024-04-25T15:59:18.879 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-25T15:59:18.880 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-25T15:59:18.880 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T15:59:18.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:18 smithi186 ceph-mon[33258]: pgmap v289: 97 pgs: 97 active+clean; 626 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.2 KiB/s wr, 26 op/s 2024-04-25T15:59:18.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:18 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4243493180' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-25T15:59:19.138 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:20.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:20 smithi031 ceph-mon[29140]: pgmap v290: 97 pgs: 97 active+clean; 626 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-04-25T15:59:20.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:20 smithi063 ceph-mon[32413]: pgmap v290: 97 pgs: 97 active+clean; 626 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-04-25T15:59:20.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:20 smithi186 ceph-mon[33258]: pgmap v290: 97 pgs: 97 active+clean; 626 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-04-25T15:59:21.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.065+0000 7f3ba27e8700 1 -- 172.21.15.31:0/4289851927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c103360 msgr2=0x7f3b9c103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.065+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/4289851927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c103360 0x7f3b9c103770 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c0099c0 tx=0x7f3b8c009cd0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.065+0000 7f3ba27e8700 1 -- 172.21.15.31:0/4289851927 shutdown_connections 2024-04-25T15:59:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.065+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/4289851927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b9c1085e0 0x7f3b9c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.065+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/4289851927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c103360 0x7f3b9c103770 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.065+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/4289851927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b9c107c90 0x7f3b9c1080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.066+0000 7f3ba27e8700 1 -- 172.21.15.31:0/4289851927 >> 172.21.15.31:0/4289851927 conn(0x7f3b9c0fec90 msgr2=0x7f3b9c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.066+0000 7f3ba27e8700 1 -- 172.21.15.31:0/4289851927 shutdown_connections 2024-04-25T15:59:21.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.066+0000 7f3ba27e8700 1 -- 172.21.15.31:0/4289851927 wait complete. 2024-04-25T15:59:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.066+0000 7f3ba27e8700 1 Processor -- start 2024-04-25T15:59:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.066+0000 7f3ba27e8700 1 -- start start 2024-04-25T15:59:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3ba27e8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b9c103360 0x7f3b9c19bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3ba27e8700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c107c90 0x7f3b9c19c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3ba27e8700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b9c1085e0 0x7f3b9c19c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3ba27e8700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c113260 con 0x7f3b9c107c90 2024-04-25T15:59:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3ba27e8700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1130e0 con 0x7f3b9c1085e0 2024-04-25T15:59:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3ba27e8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9c1133e0 con 0x7f3b9c103360 2024-04-25T15:59:21.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c107c90 0x7f3b9c19c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9b7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c107c90 0x7f3b9c19c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58770/0 (socket says 172.21.15.31:58770) 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9b7fe700 1 -- 172.21.15.31:0/3011964559 learned_addr learned my addr 172.21.15.31:0/3011964559 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9bfff700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b9c103360 0x7f3b9c19bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3ba0d85700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b9c1085e0 0x7f3b9c19c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9b7fe700 1 -- 172.21.15.31:0/3011964559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b9c103360 msgr2=0x7f3b9c19bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9b7fe700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b9c103360 0x7f3b9c19bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9b7fe700 1 -- 172.21.15.31:0/3011964559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b9c1085e0 msgr2=0x7f3b9c19c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9b7fe700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b9c1085e0 0x7f3b9c19c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9b7fe700 1 -- 172.21.15.31:0/3011964559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b9c1a6e20 con 0x7f3b9c107c90 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.067+0000 7f3b9b7fe700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c107c90 0x7f3b9c19c080 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400d980 tx=0x7f3b8400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.068+0000 7f3b997fa700 1 -- 172.21.15.31:0/3011964559 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b84017070 con 0x7f3b9c107c90 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.068+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b9c1a7110 con 0x7f3b9c107c90 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.068+0000 7f3b997fa700 1 -- 172.21.15.31:0/3011964559 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b84010460 con 0x7f3b9c107c90 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.068+0000 7f3b997fa700 1 -- 172.21.15.31:0/3011964559 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8400de60 con 0x7f3b9c107c90 2024-04-25T15:59:21.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.068+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b9c1a7650 con 0x7f3b9c107c90 2024-04-25T15:59:21.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.069+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b9c051190 con 0x7f3b9c107c90 2024-04-25T15:59:21.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.071+0000 7f3b997fa700 1 -- 172.21.15.31:0/3011964559 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3b84010610 con 0x7f3b9c107c90 2024-04-25T15:59:21.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.072+0000 7f3b997fa700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b880afc10 0x7f3b880b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:21.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.072+0000 7f3b9bfff700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b880afc10 0x7f3b880b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:21.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.072+0000 7f3b997fa700 1 -- 172.21.15.31:0/3011964559 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f3b840de8b0 con 0x7f3b9c107c90 2024-04-25T15:59:21.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.072+0000 7f3b9bfff700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b880afc10 0x7f3b880b2120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c003800 tx=0x7f3b8c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:21.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.076+0000 7f3b997fa700 1 -- 172.21.15.31:0/3011964559 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b840a73e0 con 0x7f3b9c107c90 2024-04-25T15:59:21.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.283+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3b9c02e510 con 0x7f3b9c107c90 2024-04-25T15:59:21.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.284+0000 7f3b997fa700 1 -- 172.21.15.31:0/3011964559 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+22183 (secure 0 0 0) 0x7f3b840a6ce0 con 0x7f3b9c107c90 2024-04-25T15:59:21.286 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:21.286 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":75,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:03.977416+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:59:21.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b880afc10 msgr2=0x7f3b880b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:21.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b880afc10 0x7f3b880b2120 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c003800 tx=0x7f3b8c01a040 comp rx=0 tx=0).stop 2024-04-25T15:59:21.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c107c90 msgr2=0x7f3b9c19c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:21.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c107c90 0x7f3b9c19c080 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f3b8400d980 tx=0x7f3b8400dd40 comp rx=0 tx=0).stop 2024-04-25T15:59:21.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 shutdown_connections 2024-04-25T15:59:21.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b880afc10 0x7f3b880b2120 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b9c1085e0 0x7f3b9c19c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b9c103360 0x7f3b9c19bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 --2- 172.21.15.31:0/3011964559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b9c107c90 0x7f3b9c19c080 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:21.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 >> 172.21.15.31:0/3011964559 conn(0x7f3b9c0fec90 msgr2=0x7f3b9c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:21.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 shutdown_connections 2024-04-25T15:59:21.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:21.287+0000 7f3ba27e8700 1 -- 172.21.15.31:0/3011964559 wait complete. 2024-04-25T15:59:21.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:21 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3011964559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:21.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:21 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3011964559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:21.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:21 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3011964559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:21.996 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-25T15:59:21.996 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-25T15:59:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:22.630 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:22 smithi031 ceph-mon[29140]: pgmap v291: 97 pgs: 97 active+clean; 626 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.0 KiB/s wr, 20 op/s 2024-04-25T15:59:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:22 smithi063 ceph-mon[32413]: pgmap v291: 97 pgs: 97 active+clean; 626 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.0 KiB/s wr, 20 op/s 2024-04-25T15:59:22.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:22 smithi186 ceph-mon[33258]: pgmap v291: 97 pgs: 97 active+clean; 626 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 7.0 KiB/s wr, 20 op/s 2024-04-25T15:59:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.988+0000 7fb27991b700 1 -- 172.21.15.31:0/2416620776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27410e7a0 msgr2=0x7fb274110cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.988+0000 7fb27991b700 1 --2- 172.21.15.31:0/2416620776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27410e7a0 0x7fb274110cb0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb26c00b0a0 tx=0x7fb26c00b3b0 comp rx=0 tx=0).stop 2024-04-25T15:59:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.988+0000 7fb27991b700 1 -- 172.21.15.31:0/2416620776 shutdown_connections 2024-04-25T15:59:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.988+0000 7fb27991b700 1 --2- 172.21.15.31:0/2416620776 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb27410e7a0 0x7fb274110cb0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.988+0000 7fb27991b700 1 --2- 172.21.15.31:0/2416620776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb274105330 0x7fb274105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.988+0000 7fb27991b700 1 --2- 172.21.15.31:0/2416620776 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb274105c80 0x7fb27410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:23.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.988+0000 7fb27991b700 1 -- 172.21.15.31:0/2416620776 >> 172.21.15.31:0/2416620776 conn(0x7fb2740fecc0 msgr2=0x7fb2741010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.988+0000 7fb27991b700 1 -- 172.21.15.31:0/2416620776 shutdown_connections 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.988+0000 7fb27991b700 1 -- 172.21.15.31:0/2416620776 wait complete. 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.989+0000 7fb27991b700 1 Processor -- start 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.989+0000 7fb27991b700 1 -- start start 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb27991b700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb274105330 0x7fb2741a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb27991b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb274105c80 0x7fb2741a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb27991b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27410e7a0 0x7fb27419aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb27991b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb274114350 con 0x7fb274105c80 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb27991b700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb2741141d0 con 0x7fb274105330 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb27991b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb2741144d0 con 0x7fb27410e7a0 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2727fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb274105c80 0x7fb2741a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:23.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2727fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb274105c80 0x7fb2741a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45622/0 (socket says 172.21.15.31:45622) 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2727fc700 1 -- 172.21.15.31:0/1061322316 learned_addr learned my addr 172.21.15.31:0/1061322316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb272ffd700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb274105330 0x7fb2741a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2737fe700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27410e7a0 0x7fb27419aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2727fc700 1 -- 172.21.15.31:0/1061322316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27410e7a0 msgr2=0x7fb27419aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2727fc700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27410e7a0 0x7fb27419aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2727fc700 1 -- 172.21.15.31:0/1061322316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb274105330 msgr2=0x7fb2741a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2727fc700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb274105330 0x7fb2741a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2727fc700 1 -- 172.21.15.31:0/1061322316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27419b1d0 con 0x7fb274105c80 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2737fe700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27410e7a0 0x7fb27419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb2727fc700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb274105c80 0x7fb2741a0e10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb26800e970 tx=0x7fb26800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb278919700 1 -- 172.21.15.31:0/1061322316 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26800cbd0 con 0x7fb274105c80 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.990+0000 7fb278919700 1 -- 172.21.15.31:0/1061322316 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb26800cd70 con 0x7fb274105c80 2024-04-25T15:59:23.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.991+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27419b4c0 con 0x7fb274105c80 2024-04-25T15:59:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.991+0000 7fb278919700 1 -- 172.21.15.31:0/1061322316 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268005680 con 0x7fb274105c80 2024-04-25T15:59:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.991+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2741a7760 con 0x7fb274105c80 2024-04-25T15:59:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.992+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb274051190 con 0x7fb274105c80 2024-04-25T15:59:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.993+0000 7fb278919700 1 -- 172.21.15.31:0/1061322316 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fb268004750 con 0x7fb274105c80 2024-04-25T15:59:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.994+0000 7fb278919700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb25c0afc90 0x7fb25c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.994+0000 7fb272ffd700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb25c0afc90 0x7fb25c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.994+0000 7fb278919700 1 -- 172.21.15.31:0/1061322316 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fb2680ded30 con 0x7fb274105c80 2024-04-25T15:59:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.994+0000 7fb272ffd700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb25c0afc90 0x7fb25c0b21a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb27419bc30 tx=0x7fb264009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:24.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:23.998+0000 7fb278919700 1 -- 172.21.15.31:0/1061322316 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2680a7960 con 0x7fb274105c80 2024-04-25T15:59:24.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.206+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb27419bfe0 con 0x7fb274105c80 2024-04-25T15:59:24.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.208+0000 7fb278919700 1 -- 172.21.15.31:0/1061322316 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v75) ==== 74+0+22183 (secure 0 0 0) 0x7fb2680a7260 con 0x7fb274105c80 2024-04-25T15:59:24.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T15:59:24.209 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":75,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:03.977416+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T15:59:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb25c0afc90 msgr2=0x7fb25c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb25c0afc90 0x7fb25c0b21a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb27419bc30 tx=0x7fb264009260 comp rx=0 tx=0).stop 2024-04-25T15:59:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb274105c80 msgr2=0x7fb2741a0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb274105c80 0x7fb2741a0e10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fb26800e970 tx=0x7fb26800ec80 comp rx=0 tx=0).stop 2024-04-25T15:59:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 shutdown_connections 2024-04-25T15:59:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb25c0afc90 0x7fb25c0b21a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb274105330 0x7fb2741a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27410e7a0 0x7fb27419aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:24.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 --2- 172.21.15.31:0/1061322316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb274105c80 0x7fb2741a0e10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:24.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 >> 172.21.15.31:0/1061322316 conn(0x7fb2740fecc0 msgr2=0x7fb2740ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:24.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 shutdown_connections 2024-04-25T15:59:24.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:24.210+0000 7fb27991b700 1 -- 172.21.15.31:0/1061322316 wait complete. 2024-04-25T15:59:24.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:24 smithi063 ceph-mon[32413]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-04-25T15:59:24.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:24 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1061322316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:24.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:24 smithi186 ceph-mon[33258]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-04-25T15:59:24.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:24 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1061322316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:24.908 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:24 smithi031 ceph-mon[29140]: pgmap v292: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-04-25T15:59:24.908 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:24 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1061322316' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T15:59:24.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-25T15:59:24.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-25T15:59:24.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-25T15:59:24.942 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-25T15:59:24.943 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-25T15:59:24.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-25T15:59:24.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-25T15:59:24.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-25T15:59:24.946 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-25T15:59:24.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-25T15:59:24.947 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-25T15:59:24.948 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-25T15:59:25.569 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:26.425 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:26.427 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:26.434 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:26.437 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:26.440 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:26.440 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:26.441 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:26.454 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:26.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:26 smithi063 ceph-mon[32413]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T15:59:26.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:26 smithi186 ceph-mon[33258]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T15:59:26.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:26 smithi031 ceph-mon[29140]: pgmap v293: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 2.7 KiB/s wr, 11 op/s 2024-04-25T15:59:27.036 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:27.037 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:28.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:28 smithi186 ceph-mon[33258]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-25T15:59:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:28 smithi031 ceph-mon[29140]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-25T15:59:29.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:28 smithi063 ceph-mon[32413]: pgmap v294: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.8 KiB/s wr, 11 op/s 2024-04-25T15:59:30.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:30 smithi186 ceph-mon[33258]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T15:59:30.901 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:30 smithi031 ceph-mon[29140]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T15:59:31.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:30 smithi063 ceph-mon[32413]: pgmap v295: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T15:59:32.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.351+0000 7fd3253f4700 1 -- 172.21.15.31:0/267383013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd32010e770 msgr2=0x7fd320110c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:32.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.351+0000 7fd3253f4700 1 --2- 172.21.15.31:0/267383013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd32010e770 0x7fd320110c50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd31800b0a0 tx=0x7fd31800b3b0 comp rx=0 tx=0).stop 2024-04-25T15:59:32.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.351+0000 7fd3253f4700 1 -- 172.21.15.31:0/267383013 shutdown_connections 2024-04-25T15:59:32.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.351+0000 7fd3253f4700 1 --2- 172.21.15.31:0/267383013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd320105300 0x7fd320105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:32.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.351+0000 7fd3253f4700 1 --2- 172.21.15.31:0/267383013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd32010e770 0x7fd320110c50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:32.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.351+0000 7fd3253f4700 1 --2- 172.21.15.31:0/267383013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd320105c50 0x7fd32010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:32.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.351+0000 7fd3253f4700 1 -- 172.21.15.31:0/267383013 >> 172.21.15.31:0/267383013 conn(0x7fd3200fec90 msgr2=0x7fd3201010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:32.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.351+0000 7fd3253f4700 1 -- 172.21.15.31:0/267383013 shutdown_connections 2024-04-25T15:59:32.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.351+0000 7fd3253f4700 1 -- 172.21.15.31:0/267383013 wait complete. 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.352+0000 7fd3253f4700 1 Processor -- start 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.352+0000 7fd3253f4700 1 -- start start 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.352+0000 7fd3253f4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd320105300 0x7fd3201a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.352+0000 7fd3253f4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd320105c50 0x7fd3201a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.352+0000 7fd3253f4700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd32010e770 0x7fd32019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd3253f4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd3201142d0 con 0x7fd320105300 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd3253f4700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd320114150 con 0x7fd32010e770 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd3253f4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd320114450 con 0x7fd320105c50 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd320105300 0x7fd3201a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31effd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd320105300 0x7fd3201a07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45634/0 (socket says 172.21.15.31:45634) 2024-04-25T15:59:32.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31effd700 1 -- 172.21.15.31:0/1904594363 learned_addr learned my addr 172.21.15.31:0/1904594363 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31e7fc700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd320105c50 0x7fd3201a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31effd700 1 -- 172.21.15.31:0/1904594363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd320105c50 msgr2=0x7fd3201a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31f7fe700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd32010e770 0x7fd32019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31effd700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd320105c50 0x7fd3201a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31effd700 1 -- 172.21.15.31:0/1904594363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd32010e770 msgr2=0x7fd32019a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31effd700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd32010e770 0x7fd32019a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31effd700 1 -- 172.21.15.31:0/1904594363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd32019b0a0 con 0x7fd320105300 2024-04-25T15:59:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31f7fe700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd32010e770 0x7fd32019a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:59:32.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd31effd700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd320105300 0x7fd3201a07d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd31000d980 tx=0x7fd31000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:32.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.353+0000 7fd307fff700 1 -- 172.21.15.31:0/1904594363 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310016070 con 0x7fd320105300 2024-04-25T15:59:32.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.354+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd32019b390 con 0x7fd320105300 2024-04-25T15:59:32.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.354+0000 7fd307fff700 1 -- 172.21.15.31:0/1904594363 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd310004830 con 0x7fd320105300 2024-04-25T15:59:32.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.354+0000 7fd307fff700 1 -- 172.21.15.31:0/1904594363 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100054c0 con 0x7fd320105300 2024-04-25T15:59:32.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.354+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3201a7660 con 0x7fd320105300 2024-04-25T15:59:32.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.355+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd320063e20 con 0x7fd320105300 2024-04-25T15:59:32.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.356+0000 7fd307fff700 1 -- 172.21.15.31:0/1904594363 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd310004220 con 0x7fd320105300 2024-04-25T15:59:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.357+0000 7fd307fff700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd3080afc10 0x7fd3080b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.357+0000 7fd31e7fc700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd3080afc10 0x7fd3080b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:32.358 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.357+0000 7fd307fff700 1 -- 172.21.15.31:0/1904594363 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fd3100df110 con 0x7fd320105300 2024-04-25T15:59:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.358+0000 7fd307fff700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] conn(0x7fd3080b9440 0x7fd3080bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.358+0000 7fd307fff700 1 -- 172.21.15.31:0/1904594363 --> [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd31001ce70 con 0x7fd3080b9440 2024-04-25T15:59:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.358+0000 7fd307fff700 1 -- 172.21.15.31:0/1904594363 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7fd3100df5f0 con 0x7fd320105300 2024-04-25T15:59:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.358+0000 7fd31f7fe700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] conn(0x7fd3080b9440 0x7fd3080bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.358+0000 7fd31e7fc700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd3080afc10 0x7fd3080b2120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd320102140 tx=0x7fd31400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:32.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.358+0000 7fd31f7fe700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] conn(0x7fd3080b9440 0x7fd3080bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:32.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.359+0000 7fd307fff700 1 -- 172.21.15.31:0/1904594363 <== osd.0 v2:172.21.15.31:6802/2387250399 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd31001ce70 con 0x7fd3080b9440 2024-04-25T15:59:32.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.387+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 --> [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd320064220 con 0x7fd3080b9440 2024-04-25T15:59:32.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd307fff700 1 -- 172.21.15.31:0/1904594363 <== osd.0 v2:172.21.15.31:6802/2387250399 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd320064220 con 0x7fd3080b9440 2024-04-25T15:59:32.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] conn(0x7fd3080b9440 msgr2=0x7fd3080bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:32.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd3253f4700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] conn(0x7fd3080b9440 0x7fd3080bb890 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:32.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd3080afc10 msgr2=0x7fd3080b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:32.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd3253f4700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd3080afc10 0x7fd3080b2120 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd320102140 tx=0x7fd31400a300 comp rx=0 tx=0).stop 2024-04-25T15:59:32.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd320105300 msgr2=0x7fd3201a07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:32.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd3253f4700 1 --2- 172.21.15.31:0/1904594363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd320105300 0x7fd3201a07d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd31000d980 tx=0x7fd31000dc90 comp rx=0 tx=0).stop 2024-04-25T15:59:32.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd31f7fe700 1 -- 172.21.15.31:0/1904594363 reap_dead start 2024-04-25T15:59:32.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 shutdown_connections 2024-04-25T15:59:32.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.388+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 >> 172.21.15.31:0/1904594363 conn(0x7fd3200fec90 msgr2=0x7fd3200ff320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:32.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.389+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 shutdown_connections 2024-04-25T15:59:32.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:32.389+0000 7fd3253f4700 1 -- 172.21.15.31:0/1904594363 wait complete. 2024-04-25T15:59:32.897 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:32 smithi031 ceph-mon[29140]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-25T15:59:32.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:32 smithi186 ceph-mon[33258]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-25T15:59:32.921 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:32 smithi063 ceph-mon[32413]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-04-25T15:59:33.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.667+0000 7f645b67d700 1 -- 172.21.15.31:0/4114336464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6454108e10 msgr2=0x7f6454109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:33.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.667+0000 7f645b67d700 1 --2- 172.21.15.31:0/4114336464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6454108e10 0x7f6454109220 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f64440099c0 tx=0x7f6444009cd0 comp rx=0 tx=0).stop 2024-04-25T15:59:33.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.667+0000 7f645b67d700 1 -- 172.21.15.31:0/4114336464 shutdown_connections 2024-04-25T15:59:33.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.667+0000 7f645b67d700 1 --2- 172.21.15.31:0/4114336464 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64541044e0 0x7f64541048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.667+0000 7f645b67d700 1 --2- 172.21.15.31:0/4114336464 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6454109760 0x7f64541110c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.667+0000 7f645b67d700 1 --2- 172.21.15.31:0/4114336464 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6454108e10 0x7f6454109220 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.667+0000 7f645b67d700 1 -- 172.21.15.31:0/4114336464 >> 172.21.15.31:0/4114336464 conn(0x7f645407ae40 msgr2=0x7f645407b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.667+0000 7f645b67d700 1 -- 172.21.15.31:0/4114336464 shutdown_connections 2024-04-25T15:59:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.667+0000 7f645b67d700 1 -- 172.21.15.31:0/4114336464 wait complete. 2024-04-25T15:59:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f645b67d700 1 Processor -- start 2024-04-25T15:59:33.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f645b67d700 1 -- start start 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f645b67d700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64541044e0 0x7f6454075990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f645b67d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6454108e10 0x7f6454075ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f645b67d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6454109760 0x7f6454076410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f645b67d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f64541143e0 con 0x7f6454108e10 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f645b67d700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6454114260 con 0x7f64541044e0 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f645b67d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6454114560 con 0x7f6454109760 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f6458c18700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6454108e10 0x7f6454075ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f6458c18700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6454108e10 0x7f6454075ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36112/0 (socket says 172.21.15.31:36112) 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f6458c18700 1 -- 172.21.15.31:0/2599005995 learned_addr learned my addr 172.21.15.31:0/2599005995 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f6459c1a700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6454109760 0x7f6454076410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:33.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f6459419700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64541044e0 0x7f6454075990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.668+0000 7f6458c18700 1 -- 172.21.15.31:0/2599005995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6454109760 msgr2=0x7f6454076410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f6458c18700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6454109760 0x7f6454076410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f6458c18700 1 -- 172.21.15.31:0/2599005995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64541044e0 msgr2=0x7f6454075990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f6458c18700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f64541044e0 0x7f6454075990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f6458c18700 1 -- 172.21.15.31:0/2599005995 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64541a9150 con 0x7f6454108e10 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f6459c1a700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6454109760 0x7f6454076410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f6458c18700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6454108e10 0x7f6454075ed0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6444006010 tx=0x7f64440047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f644a7fc700 1 -- 172.21.15.31:0/2599005995 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444027070 con 0x7f6454108e10 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f644a7fc700 1 -- 172.21.15.31:0/2599005995 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6444003fa0 con 0x7f6454108e10 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64541a93e0 con 0x7f6454108e10 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f644a7fc700 1 -- 172.21.15.31:0/2599005995 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444022dd0 con 0x7f6454108e10 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.669+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64541a9830 con 0x7f6454108e10 2024-04-25T15:59:33.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.670+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6454063da0 con 0x7f6454108e10 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.671+0000 7f644a7fc700 1 -- 172.21.15.31:0/2599005995 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f644402a070 con 0x7f6454108e10 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.672+0000 7f644a7fc700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64400afc10 0x7f64400b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.672+0000 7f6459419700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64400afc10 0x7f64400b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.672+0000 7f644a7fc700 1 -- 172.21.15.31:0/2599005995 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f64440e0000 con 0x7f6454108e10 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.672+0000 7f644a7fc700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] conn(0x7f64400b9440 0x7f64400bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.672+0000 7f644a7fc700 1 -- 172.21.15.31:0/2599005995 --> [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6444001d30 con 0x7f64400b9440 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.672+0000 7f644a7fc700 1 -- 172.21.15.31:0/2599005995 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f64440e04e0 con 0x7f6454108e10 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.673+0000 7f6459419700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64400afc10 0x7f64400b2120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6450005950 tx=0x7f64500058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.673+0000 7f6459c1a700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] conn(0x7f64400b9440 0x7f64400bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:33.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.673+0000 7f6459c1a700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] conn(0x7f64400b9440 0x7f64400bb890 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:33.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.674+0000 7f644a7fc700 1 -- 172.21.15.31:0/2599005995 <== osd.5 v2:172.21.15.186:6808/1998951869 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6444001d30 con 0x7f64400b9440 2024-04-25T15:59:33.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.691+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 --> [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f64540641a0 con 0x7f64400b9440 2024-04-25T15:59:33.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.691+0000 7f644a7fc700 1 -- 172.21.15.31:0/2599005995 <== osd.5 v2:172.21.15.186:6808/1998951869 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f64540641a0 con 0x7f64400b9440 2024-04-25T15:59:33.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.691+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 >> [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] conn(0x7f64400b9440 msgr2=0x7f64400bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.691+0000 7f645b67d700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] conn(0x7f64400b9440 0x7f64400bb890 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.692+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64400afc10 msgr2=0x7f64400b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.692+0000 7f645b67d700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64400afc10 0x7f64400b2120 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6450005950 tx=0x7f64500058e0 comp rx=0 tx=0).stop 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.692+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6454108e10 msgr2=0x7f6454075ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.692+0000 7f645b67d700 1 --2- 172.21.15.31:0/2599005995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6454108e10 0x7f6454075ed0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6444006010 tx=0x7f64440047b0 comp rx=0 tx=0).stop 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.692+0000 7f6459c1a700 1 -- 172.21.15.31:0/2599005995 reap_dead start 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.692+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 shutdown_connections 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.692+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 >> 172.21.15.31:0/2599005995 conn(0x7f645407ae40 msgr2=0x7f64541018c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.692+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 shutdown_connections 2024-04-25T15:59:33.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:33.692+0000 7f645b67d700 1 -- 172.21.15.31:0/2599005995 wait complete. 2024-04-25T15:59:34.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:34 smithi186 ceph-mon[33258]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T15:59:34.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:34 smithi031 ceph-mon[29140]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T15:59:35.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:34 smithi063 ceph-mon[32413]: pgmap v297: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 12 op/s 2024-04-25T15:59:36.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:36 smithi031 ceph-mon[29140]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T15:59:36.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:36 smithi186 ceph-mon[33258]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T15:59:37.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:36 smithi063 ceph-mon[32413]: pgmap v298: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T15:59:38.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 -- 172.21.15.31:0/1232918770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f373c104160 msgr2=0x7f373c10ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 --2- 172.21.15.31:0/1232918770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f373c104160 0x7f373c10ff20 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f373400b0a0 tx=0x7f373400b3b0 comp rx=0 tx=0).stop 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 -- 172.21.15.31:0/1232918770 shutdown_connections 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 --2- 172.21.15.31:0/1232918770 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f373c1080b0 0x7f373c1084c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 --2- 172.21.15.31:0/1232918770 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f373c104160 0x7f373c10ff20 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 --2- 172.21.15.31:0/1232918770 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c103810 0x7f373c103c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 -- 172.21.15.31:0/1232918770 >> 172.21.15.31:0/1232918770 conn(0x7f373c0fe370 msgr2=0x7f373c1007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 -- 172.21.15.31:0/1232918770 shutdown_connections 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 -- 172.21.15.31:0/1232918770 wait complete. 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.572+0000 7f37419e1700 1 Processor -- start 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f37419e1700 1 -- start start 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f37419e1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f373c103810 0x7f373c1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f37419e1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104160 0x7f373c1a0d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:38.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f37419e1700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f373c1080b0 0x7f373c19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f37419e1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f373c113260 con 0x7f373c103810 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f37419e1700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f373c1130e0 con 0x7f373c1080b0 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f37419e1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f373c1133e0 con 0x7f373c104160 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f373c103810 0x7f373c1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f373c103810 0x7f373c1a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36140/0 (socket says 172.21.15.31:36140) 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373affd700 1 -- 172.21.15.31:0/4155259373 learned_addr learned my addr 172.21.15.31:0/4155259373 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373a7fc700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104160 0x7f373c1a0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373b7fe700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f373c1080b0 0x7f373c19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373affd700 1 -- 172.21.15.31:0/4155259373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104160 msgr2=0x7f373c1a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373affd700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f373c104160 0x7f373c1a0d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373affd700 1 -- 172.21.15.31:0/4155259373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f373c1080b0 msgr2=0x7f373c19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373affd700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f373c1080b0 0x7f373c19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.573+0000 7f373affd700 1 -- 172.21.15.31:0/4155259373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373c19b120 con 0x7f373c103810 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.574+0000 7f373affd700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f373c103810 0x7f373c1a0850 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f372c00d950 tx=0x7f372c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:38.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.574+0000 7f37409df700 1 -- 172.21.15.31:0/4155259373 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c017070 con 0x7f373c103810 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.574+0000 7f37409df700 1 -- 172.21.15.31:0/4155259373 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f372c009de0 con 0x7f373c103810 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.574+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373c19b410 con 0x7f373c103810 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.574+0000 7f37409df700 1 -- 172.21.15.31:0/4155259373 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f372c00f3c0 con 0x7f373c103810 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.574+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373c1a76e0 con 0x7f373c103810 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.575+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f373c0fead0 con 0x7f373c103810 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.576+0000 7f37409df700 1 -- 172.21.15.31:0/4155259373 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f372c01a070 con 0x7f373c103810 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.577+0000 7f37409df700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37280afc10 0x7f37280b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.577+0000 7f373a7fc700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37280afc10 0x7f37280b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.577+0000 7f37409df700 1 -- 172.21.15.31:0/4155259373 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f372c0de8a0 con 0x7f373c103810 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.577+0000 7f37409df700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] conn(0x7f37280b9440 0x7f37280bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.577+0000 7f37409df700 1 -- 172.21.15.31:0/4155259373 --> [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37280bbf60 con 0x7f37280b9440 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.577+0000 7f37409df700 1 -- 172.21.15.31:0/4155259373 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f372c0ded80 con 0x7f373c103810 2024-04-25T15:59:38.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.578+0000 7f373a7fc700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37280afc10 0x7f37280b2120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f373c1019f0 tx=0x7f372400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:38.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.578+0000 7f373b7fe700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] conn(0x7f37280b9440 0x7f37280bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:38.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.578+0000 7f373b7fe700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] conn(0x7f37280b9440 0x7f37280bb890 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:38.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.579+0000 7f37409df700 1 -- 172.21.15.31:0/4155259373 <== osd.8 v2:172.21.15.186:6816/82309382 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f37280bbf60 con 0x7f37280b9440 2024-04-25T15:59:38.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.596+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 --> [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f373c104a30 con 0x7f37280b9440 2024-04-25T15:59:38.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.596+0000 7f37409df700 1 -- 172.21.15.31:0/4155259373 <== osd.8 v2:172.21.15.186:6816/82309382 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f373c104a30 con 0x7f37280b9440 2024-04-25T15:59:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 >> [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] conn(0x7f37280b9440 msgr2=0x7f37280bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] conn(0x7f37280b9440 0x7f37280bb890 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37280afc10 msgr2=0x7f37280b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37280afc10 0x7f37280b2120 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f373c1019f0 tx=0x7f372400b410 comp rx=0 tx=0).stop 2024-04-25T15:59:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f373c103810 msgr2=0x7f373c1a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 --2- 172.21.15.31:0/4155259373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f373c103810 0x7f373c1a0850 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f372c00d950 tx=0x7f372c00dc60 comp rx=0 tx=0).stop 2024-04-25T15:59:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f373b7fe700 1 -- 172.21.15.31:0/4155259373 reap_dead start 2024-04-25T15:59:38.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 shutdown_connections 2024-04-25T15:59:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 >> 172.21.15.31:0/4155259373 conn(0x7f373c0fe370 msgr2=0x7f373c106f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 shutdown_connections 2024-04-25T15:59:38.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:38.597+0000 7f37419e1700 1 -- 172.21.15.31:0/4155259373 wait complete. 2024-04-25T15:59:38.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:38 smithi186 ceph-mon[33258]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T15:59:38.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:38 smithi031 ceph-mon[29140]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T15:59:39.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:38 smithi063 ceph-mon[32413]: pgmap v299: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T15:59:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.952+0000 7fbce204b700 1 -- 172.21.15.31:0/4121461467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcdc108d90 msgr2=0x7fbcdc1091a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.952+0000 7fbce204b700 1 --2- 172.21.15.31:0/4121461467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcdc108d90 0x7fbcdc1091a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fbcc4009a40 tx=0x7fbcc4009d50 comp rx=0 tx=0).stop 2024-04-25T15:59:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.952+0000 7fbce204b700 1 -- 172.21.15.31:0/4121461467 shutdown_connections 2024-04-25T15:59:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.952+0000 7fbce204b700 1 --2- 172.21.15.31:0/4121461467 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcdc1096e0 0x7fbcdc111040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.952+0000 7fbce204b700 1 --2- 172.21.15.31:0/4121461467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcdc104460 0x7fbcdc104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.952+0000 7fbce204b700 1 --2- 172.21.15.31:0/4121461467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcdc108d90 0x7fbcdc1091a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:40.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.952+0000 7fbce204b700 1 -- 172.21.15.31:0/4121461467 >> 172.21.15.31:0/4121461467 conn(0x7fbcdc0ffd40 msgr2=0x7fbcdc102160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.952+0000 7fbce204b700 1 -- 172.21.15.31:0/4121461467 shutdown_connections 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.952+0000 7fbce204b700 1 -- 172.21.15.31:0/4121461467 wait complete. 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbce204b700 1 Processor -- start 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbce204b700 1 -- start start 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbce204b700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcdc104460 0x7fbcdc19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbce204b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcdc108d90 0x7fbcdc19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbce204b700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcdc1096e0 0x7fbcdc1988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbce204b700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc112150 con 0x7fbcdc1096e0 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbce204b700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc111fd0 con 0x7fbcdc104460 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbce204b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbcdc1122d0 con 0x7fbcdc108d90 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbcdbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcdc1096e0 0x7fbcdc1988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbcdbfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcdc1096e0 0x7fbcdc1988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36122/0 (socket says 172.21.15.31:36122) 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.953+0000 7fbcdbfff700 1 -- 172.21.15.31:0/1033131608 learned_addr learned my addr 172.21.15.31:0/1033131608 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:40.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdb7fe700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcdc104460 0x7fbcdc19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:40.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdaffd700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcdc108d90 0x7fbcdc19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:40.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdbfff700 1 -- 172.21.15.31:0/1033131608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcdc108d90 msgr2=0x7fbcdc19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:40.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdbfff700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcdc108d90 0x7fbcdc19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:40.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdbfff700 1 -- 172.21.15.31:0/1033131608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcdc104460 msgr2=0x7fbcdc19e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:40.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdbfff700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcdc104460 0x7fbcdc19e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdbfff700 1 -- 172.21.15.31:0/1033131608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcdc198f80 con 0x7fbcdc1096e0 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdb7fe700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcdc104460 0x7fbcdc19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdaffd700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbcdc108d90 0x7fbcdc19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcdbfff700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcdc1096e0 0x7fbcdc1988d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400d980 tx=0x7fbcd400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcd8ff9700 1 -- 172.21.15.31:0/1033131608 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4017070 con 0x7fbcdc1096e0 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcdc199270 con 0x7fbcdc1096e0 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcd8ff9700 1 -- 172.21.15.31:0/1033131608 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcd4009df0 con 0x7fbcdc1096e0 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbcd8ff9700 1 -- 172.21.15.31:0/1033131608 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd40053a0 con 0x7fbcdc1096e0 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.954+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcdc1a5540 con 0x7fbcdc1096e0 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.955+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbcdc063e20 con 0x7fbcdc1096e0 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.957+0000 7fbcd8ff9700 1 -- 172.21.15.31:0/1033131608 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fbcd4004750 con 0x7fbcdc1096e0 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.957+0000 7fbcd8ff9700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcc80afc10 0x7fbcc80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:40.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.957+0000 7fbcdb7fe700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcc80afc10 0x7fbcc80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.957+0000 7fbcd8ff9700 1 -- 172.21.15.31:0/1033131608 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fbcd40def90 con 0x7fbcdc1096e0 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.957+0000 7fbcd8ff9700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] conn(0x7fbcc80b9440 0x7fbcc80bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.958+0000 7fbcd8ff9700 1 -- 172.21.15.31:0/1033131608 --> [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcd401ae70 con 0x7fbcc80b9440 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.958+0000 7fbcd8ff9700 1 -- 172.21.15.31:0/1033131608 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7fbcd40df470 con 0x7fbcdc1096e0 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.958+0000 7fbcdaffd700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] conn(0x7fbcc80b9440 0x7fbcc80bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.958+0000 7fbcdb7fe700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcc80afc10 0x7fbcc80b2120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc199a10 tx=0x7fbccc009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.958+0000 7fbcdaffd700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] conn(0x7fbcc80b9440 0x7fbcc80bb890 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.959+0000 7fbcd8ff9700 1 -- 172.21.15.31:0/1033131608 <== osd.9 v2:172.21.15.31:6826/3856015669 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbcd401ae70 con 0x7fbcc80b9440 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.975+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 --> [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbcdc0642a0 con 0x7fbcc80b9440 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbcd8ff9700 1 -- 172.21.15.31:0/1033131608 <== osd.9 v2:172.21.15.31:6826/3856015669 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbcdc0642a0 con 0x7fbcc80b9440 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] conn(0x7fbcc80b9440 msgr2=0x7fbcc80bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] conn(0x7fbcc80b9440 0x7fbcc80bb890 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:40.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcc80afc10 msgr2=0x7fbcc80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcc80afc10 0x7fbcc80b2120 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc199a10 tx=0x7fbccc009040 comp rx=0 tx=0).stop 2024-04-25T15:59:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcdc1096e0 msgr2=0x7fbcdc1988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 --2- 172.21.15.31:0/1033131608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbcdc1096e0 0x7fbcdc1988d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400d980 tx=0x7fbcd400dd40 comp rx=0 tx=0).stop 2024-04-25T15:59:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbcdbfff700 1 -- 172.21.15.31:0/1033131608 reap_dead start 2024-04-25T15:59:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 shutdown_connections 2024-04-25T15:59:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 >> 172.21.15.31:0/1033131608 conn(0x7fbcdc0ffd40 msgr2=0x7fbcdc102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 shutdown_connections 2024-04-25T15:59:40.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:37.976+0000 7fbce204b700 1 -- 172.21.15.31:0/1033131608 wait complete. 2024-04-25T15:59:40.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:40 smithi186 ceph-mon[33258]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T15:59:40.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:40 smithi031 ceph-mon[29140]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T15:59:41.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:40 smithi063 ceph-mon[32413]: pgmap v300: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T15:59:41.812 INFO:teuthology.orchestra.run.smithi031.stdout:253403070494 2024-04-25T15:59:41.813 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-25T15:59:42.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:42 smithi186 ceph-mon[33258]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T15:59:42.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:42 smithi031 ceph-mon[29140]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T15:59:43.014 INFO:teuthology.orchestra.run.smithi031.stdout:34359738444 2024-04-25T15:59:43.014 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-25T15:59:43.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:42 smithi063 ceph-mon[32413]: pgmap v301: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T15:59:43.538 INFO:teuthology.orchestra.run.smithi031.stdout:227633266725 2024-04-25T15:59:43.538 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-25T15:59:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.750+0000 7f4b9528a700 1 -- 172.21.15.31:0/281441452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b90105300 msgr2=0x7f4b90105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.750+0000 7f4b9528a700 1 --2- 172.21.15.31:0/281441452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b90105300 0x7f4b90105710 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4b80009990 tx=0x7f4b80009ca0 comp rx=0 tx=0).stop 2024-04-25T15:59:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.750+0000 7f4b9528a700 1 -- 172.21.15.31:0/281441452 shutdown_connections 2024-04-25T15:59:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.750+0000 7f4b9528a700 1 --2- 172.21.15.31:0/281441452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b9010e770 0x7f4b90110c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.750+0000 7f4b9528a700 1 --2- 172.21.15.31:0/281441452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b90105300 0x7f4b90105710 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.750+0000 7f4b9528a700 1 --2- 172.21.15.31:0/281441452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b90105c50 0x7f4b9010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.750+0000 7f4b9528a700 1 -- 172.21.15.31:0/281441452 >> 172.21.15.31:0/281441452 conn(0x7f4b900fec90 msgr2=0x7f4b901010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.750+0000 7f4b9528a700 1 -- 172.21.15.31:0/281441452 shutdown_connections 2024-04-25T15:59:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.750+0000 7f4b9528a700 1 -- 172.21.15.31:0/281441452 wait complete. 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.751+0000 7f4b9528a700 1 Processor -- start 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b9528a700 1 -- start start 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b9528a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b90105300 0x7f4b901a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b9528a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b90105c50 0x7f4b901a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b9528a700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b9010e770 0x7f4b9019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b9528a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b901142c0 con 0x7f4b90105c50 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b9528a700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b90114140 con 0x7f4b9010e770 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b9528a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b90114440 con 0x7f4b90105300 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b8e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b90105c50 0x7f4b901a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b8e7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b90105c50 0x7f4b901a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51452/0 (socket says 172.21.15.31:51452) 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b8e7fc700 1 -- 172.21.15.31:0/3042057279 learned_addr learned my addr 172.21.15.31:0/3042057279 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b8effd700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b90105300 0x7f4b901a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.752+0000 7f4b8f7fe700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b9010e770 0x7f4b9019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b8e7fc700 1 -- 172.21.15.31:0/3042057279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b90105300 msgr2=0x7f4b901a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:43.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b8e7fc700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b90105300 0x7f4b901a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b8e7fc700 1 -- 172.21.15.31:0/3042057279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b9010e770 msgr2=0x7f4b9019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b8e7fc700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b9010e770 0x7f4b9019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b8e7fc700 1 -- 172.21.15.31:0/3042057279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9019b0a0 con 0x7f4b90105c50 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b8effd700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b90105300 0x7f4b901a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b8e7fc700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b90105c50 0x7f4b901a0d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400d980 tx=0x7f4b8400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b77fff700 1 -- 172.21.15.31:0/3042057279 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84016070 con 0x7f4b90105c50 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b9019b390 con 0x7f4b90105c50 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b77fff700 1 -- 172.21.15.31:0/3042057279 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b84004830 con 0x7f4b90105c50 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b901a76d0 con 0x7f4b90105c50 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.753+0000 7f4b77fff700 1 -- 172.21.15.31:0/3042057279 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b840054c0 con 0x7f4b90105c50 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.754+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4b90063e20 con 0x7f4b90105c50 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.755+0000 7f4b77fff700 1 -- 172.21.15.31:0/3042057279 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f4b84004220 con 0x7f4b90105c50 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.756+0000 7f4b77fff700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b780afc10 0x7f4b780b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:43.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.756+0000 7f4b8effd700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b780afc10 0x7f4b780b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.756+0000 7f4b77fff700 1 -- 172.21.15.31:0/3042057279 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f4b840df0f0 con 0x7f4b90105c50 2024-04-25T15:59:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.756+0000 7f4b77fff700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] conn(0x7f4b780b9440 0x7f4b780bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.756+0000 7f4b77fff700 1 -- 172.21.15.31:0/3042057279 --> [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4b8401ce70 con 0x7f4b780b9440 2024-04-25T15:59:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.756+0000 7f4b77fff700 1 -- 172.21.15.31:0/3042057279 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f4b840df5d0 con 0x7f4b90105c50 2024-04-25T15:59:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.757+0000 7f4b8effd700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b780afc10 0x7f4b780b2120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4b80003800 tx=0x7f4b8001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.757+0000 7f4b8f7fe700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] conn(0x7f4b780b9440 0x7f4b780bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.757+0000 7f4b8f7fe700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] conn(0x7f4b780b9440 0x7f4b780bb890 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.758+0000 7f4b77fff700 1 -- 172.21.15.31:0/3042057279 <== osd.11 v2:172.21.15.186:6824/2008780285 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4b8401ce70 con 0x7f4b780b9440 2024-04-25T15:59:43.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.775+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 --> [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4b90064220 con 0x7f4b780b9440 2024-04-25T15:59:43.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.775+0000 7f4b77fff700 1 -- 172.21.15.31:0/3042057279 <== osd.11 v2:172.21.15.186:6824/2008780285 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4b90064220 con 0x7f4b780b9440 2024-04-25T15:59:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 >> [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] conn(0x7f4b780b9440 msgr2=0x7f4b780bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] conn(0x7f4b780b9440 0x7f4b780bb890 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b780afc10 msgr2=0x7f4b780b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b780afc10 0x7f4b780b2120 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4b80003800 tx=0x7f4b8001a040 comp rx=0 tx=0).stop 2024-04-25T15:59:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b90105c50 msgr2=0x7f4b901a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 --2- 172.21.15.31:0/3042057279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b90105c50 0x7f4b901a0d80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400d980 tx=0x7f4b8400dc90 comp rx=0 tx=0).stop 2024-04-25T15:59:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b8f7fe700 1 -- 172.21.15.31:0/3042057279 reap_dead start 2024-04-25T15:59:43.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 shutdown_connections 2024-04-25T15:59:43.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 >> 172.21.15.31:0/3042057279 conn(0x7f4b900fec90 msgr2=0x7f4b900ff340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:43.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 shutdown_connections 2024-04-25T15:59:43.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:43.776+0000 7f4b9528a700 1 -- 172.21.15.31:0/3042057279 wait complete. 2024-04-25T15:59:44.734 INFO:teuthology.orchestra.run.smithi031.stdout:154618822707 2024-04-25T15:59:44.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-25T15:59:44.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:44 smithi031 ceph-mon[29140]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T15:59:45.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:44 smithi063 ceph-mon[32413]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T15:59:45.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:44 smithi186 ceph-mon[33258]: pgmap v302: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T15:59:46.216 INFO:teuthology.orchestra.run.smithi031.stdout:300647710740 2024-04-25T15:59:46.216 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-25T15:59:46.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:46 smithi031 ceph-mon[29140]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T15:59:47.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:46 smithi063 ceph-mon[32413]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T15:59:47.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:46 smithi186 ceph-mon[33258]: pgmap v303: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T15:59:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.173+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3814661095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf7c105360 msgr2=0x7fbf7c105770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.173+0000 7fbf80ecb700 1 --2- 172.21.15.31:0/3814661095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf7c105360 0x7fbf7c105770 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400b0a0 tx=0x7fbf7400b3b0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.173+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3814661095 shutdown_connections 2024-04-25T15:59:47.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.173+0000 7fbf80ecb700 1 --2- 172.21.15.31:0/3814661095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf7c10e790 0x7fbf7c110c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.173+0000 7fbf80ecb700 1 --2- 172.21.15.31:0/3814661095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf7c105360 0x7fbf7c105770 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.173+0000 7fbf80ecb700 1 --2- 172.21.15.31:0/3814661095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf7c105cb0 0x7fbf7c10e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.173+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3814661095 >> 172.21.15.31:0/3814661095 conn(0x7fbf7c0fec90 msgr2=0x7fbf7c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.174+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3814661095 shutdown_connections 2024-04-25T15:59:47.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.174+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3814661095 wait complete. 2024-04-25T15:59:47.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.174+0000 7fbf80ecb700 1 Processor -- start 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.174+0000 7fbf80ecb700 1 -- start start 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf80ecb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf7c105360 0x7fbf7c1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf80ecb700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf7c105cb0 0x7fbf7c1a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf80ecb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf7c10e790 0x7fbf7c19aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf80ecb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7c1142e0 con 0x7fbf7c105360 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf80ecb700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7c114160 con 0x7fbf7c105cb0 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf80ecb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7c114460 con 0x7fbf7c10e790 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf7c105360 0x7fbf7c1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7a59c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf7c105360 0x7fbf7c1a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51534/0 (socket says 172.21.15.31:51534) 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7a59c700 1 -- 172.21.15.31:0/3570509615 learned_addr learned my addr 172.21.15.31:0/3570509615 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf79d9b700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf7c105cb0 0x7fbf7c1a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7ad9d700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf7c10e790 0x7fbf7c19aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7a59c700 1 -- 172.21.15.31:0/3570509615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf7c10e790 msgr2=0x7fbf7c19aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7a59c700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf7c10e790 0x7fbf7c19aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7a59c700 1 -- 172.21.15.31:0/3570509615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf7c105cb0 msgr2=0x7fbf7c1a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7a59c700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbf7c105cb0 0x7fbf7c1a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7a59c700 1 -- 172.21.15.31:0/3570509615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf7c19b150 con 0x7fbf7c105360 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf7a59c700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf7c105360 0x7fbf7c1a0880 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400b070 tx=0x7fbf74004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.175+0000 7fbf737fe700 1 -- 172.21.15.31:0/3570509615 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74021070 con 0x7fbf7c105360 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.176+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf7c19b3e0 con 0x7fbf7c105360 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.176+0000 7fbf737fe700 1 -- 172.21.15.31:0/3570509615 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf74007e30 con 0x7fbf7c105360 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.176+0000 7fbf737fe700 1 -- 172.21.15.31:0/3570509615 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf74004840 con 0x7fbf7c105360 2024-04-25T15:59:47.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.176+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf7c1a7710 con 0x7fbf7c105360 2024-04-25T15:59:47.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.176+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbf7c063e20 con 0x7fbf7c105360 2024-04-25T15:59:47.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.178+0000 7fbf737fe700 1 -- 172.21.15.31:0/3570509615 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fbf74007980 con 0x7fbf7c105360 2024-04-25T15:59:47.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.179+0000 7fbf737fe700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbf680afc10 0x7fbf680b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.179+0000 7fbf79d9b700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbf680afc10 0x7fbf680b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.179+0000 7fbf737fe700 1 -- 172.21.15.31:0/3570509615 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fbf7400f020 con 0x7fbf7c105360 2024-04-25T15:59:47.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.179+0000 7fbf737fe700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] conn(0x7fbf680b9310 0x7fbf680bb760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.179+0000 7fbf737fe700 1 -- 172.21.15.31:0/3570509615 --> [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf680bbe30 con 0x7fbf680b9310 2024-04-25T15:59:47.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.179+0000 7fbf737fe700 1 -- 172.21.15.31:0/3570509615 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7fbf740e0b30 con 0x7fbf7c105360 2024-04-25T15:59:47.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.179+0000 7fbf79d9b700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbf680afc10 0x7fbf680b2120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c1021d0 tx=0x7fbf6400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:47.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.179+0000 7fbf7ad9d700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] conn(0x7fbf680b9310 0x7fbf680bb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.180+0000 7fbf7ad9d700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] conn(0x7fbf680b9310 0x7fbf680bb760 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:47.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.181+0000 7fbf737fe700 1 -- 172.21.15.31:0/3570509615 <== osd.10 v2:172.21.15.63:6824/1440937910 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbf680bbe30 con 0x7fbf680b9310 2024-04-25T15:59:47.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.197+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 --> [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbf7c064220 con 0x7fbf680b9310 2024-04-25T15:59:47.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf737fe700 1 -- 172.21.15.31:0/3570509615 <== osd.10 v2:172.21.15.63:6824/1440937910 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbf7c064220 con 0x7fbf680b9310 2024-04-25T15:59:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 >> [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] conn(0x7fbf680b9310 msgr2=0x7fbf680bb760 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] conn(0x7fbf680b9310 0x7fbf680bb760 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbf680afc10 msgr2=0x7fbf680b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbf680afc10 0x7fbf680b2120 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c1021d0 tx=0x7fbf6400b410 comp rx=0 tx=0).stop 2024-04-25T15:59:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf7c105360 msgr2=0x7fbf7c1a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 --2- 172.21.15.31:0/3570509615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbf7c105360 0x7fbf7c1a0880 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbf7400b070 tx=0x7fbf74004d00 comp rx=0 tx=0).stop 2024-04-25T15:59:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf7ad9d700 1 -- 172.21.15.31:0/3570509615 reap_dead start 2024-04-25T15:59:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 shutdown_connections 2024-04-25T15:59:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 >> 172.21.15.31:0/3570509615 conn(0x7fbf7c0fec90 msgr2=0x7fbf7c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 shutdown_connections 2024-04-25T15:59:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.198+0000 7fbf80ecb700 1 -- 172.21.15.31:0/3570509615 wait complete. 2024-04-25T15:59:47.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.207+0000 7f2391d16700 1 -- 172.21.15.31:0/2152028868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f238c10e770 msgr2=0x7f238c110c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.207+0000 7f2391d16700 1 --2- 172.21.15.31:0/2152028868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f238c10e770 0x7f238c110c50 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f238400b0a0 tx=0x7f238400b3b0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.207+0000 7f2391d16700 1 -- 172.21.15.31:0/2152028868 shutdown_connections 2024-04-25T15:59:47.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.207+0000 7f2391d16700 1 --2- 172.21.15.31:0/2152028868 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f238c10e770 0x7f238c110c50 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.207+0000 7f2391d16700 1 --2- 172.21.15.31:0/2152028868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f238c105300 0x7f238c105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.207+0000 7f2391d16700 1 --2- 172.21.15.31:0/2152028868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f238c105c50 0x7f238c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.207+0000 7f2391d16700 1 -- 172.21.15.31:0/2152028868 >> 172.21.15.31:0/2152028868 conn(0x7f238c0fecb0 msgr2=0x7f238c1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.207+0000 7f2391d16700 1 -- 172.21.15.31:0/2152028868 shutdown_connections 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.207+0000 7f2391d16700 1 -- 172.21.15.31:0/2152028868 wait complete. 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f2391d16700 1 Processor -- start 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f2391d16700 1 -- start start 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f2391d16700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f238c105300 0x7f238c102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f2391d16700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f238c105c50 0x7f238c103530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f2391d16700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f238c10e770 0x7f238c103a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f2391d16700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f238c1142d0 con 0x7f238c10e770 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f2391d16700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f238c114150 con 0x7f238c105c50 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f2391d16700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f238c114450 con 0x7f238c105300 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f238bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f238c10e770 0x7f238c103a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f238bfff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f238c10e770 0x7f238c103a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51472/0 (socket says 172.21.15.31:51472) 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f238bfff700 1 -- 172.21.15.31:0/3579730667 learned_addr learned my addr 172.21.15.31:0/3579730667 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f238affd700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f238c105c50 0x7f238c103530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.208+0000 7f238b7fe700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f238c105300 0x7f238c102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f238bfff700 1 -- 172.21.15.31:0/3579730667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f238c105300 msgr2=0x7f238c102ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f238bfff700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f238c105300 0x7f238c102ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f238bfff700 1 -- 172.21.15.31:0/3579730667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f238c105c50 msgr2=0x7f238c103530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f238bfff700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f238c105c50 0x7f238c103530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f238bfff700 1 -- 172.21.15.31:0/3579730667 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f238c1a8ef0 con 0x7f238c10e770 2024-04-25T15:59:47.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f238bfff700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f238c10e770 0x7f238c103a70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f238400b830 tx=0x7f2384004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f2388ff9700 1 -- 172.21.15.31:0/3579730667 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2384021070 con 0x7f238c10e770 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f238c1a9180 con 0x7f238c10e770 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f2388ff9700 1 -- 172.21.15.31:0/3579730667 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2384007e30 con 0x7f238c10e770 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f2388ff9700 1 -- 172.21.15.31:0/3579730667 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2384004700 con 0x7f238c10e770 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.209+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f238c1a9680 con 0x7f238c10e770 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.210+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f238c063da0 con 0x7f238c10e770 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.212+0000 7f2388ff9700 1 -- 172.21.15.31:0/3579730667 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2384003d70 con 0x7f238c10e770 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.212+0000 7f2388ff9700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23740afc10 0x7f23740b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.213+0000 7f238b7fe700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23740afc10 0x7f23740b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.213+0000 7f2388ff9700 1 -- 172.21.15.31:0/3579730667 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f238400f020 con 0x7f238c10e770 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.213+0000 7f2388ff9700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] conn(0x7f23740b9440 0x7f23740bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.213+0000 7f2388ff9700 1 -- 172.21.15.31:0/3579730667 --> [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2384032ea0 con 0x7f23740b9440 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.213+0000 7f2388ff9700 1 -- 172.21.15.31:0/3579730667 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f23840e0a30 con 0x7f238c10e770 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.213+0000 7f238b7fe700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23740afc10 0x7f23740b2120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f237c005950 tx=0x7f237c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:47.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.213+0000 7f238affd700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] conn(0x7f23740b9440 0x7f23740bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.213+0000 7f238affd700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] conn(0x7f23740b9440 0x7f23740bb890 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.214+0000 7f2388ff9700 1 -- 172.21.15.31:0/3579730667 <== osd.4 v2:172.21.15.63:6808/2163634076 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2384032ea0 con 0x7f23740b9440 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.232+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 --> [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f238c0641d0 con 0x7f23740b9440 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f2388ff9700 1 -- 172.21.15.31:0/3579730667 <== osd.4 v2:172.21.15.63:6808/2163634076 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f238c0641d0 con 0x7f23740b9440 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 >> [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] conn(0x7f23740b9440 msgr2=0x7f23740bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f2391d16700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] conn(0x7f23740b9440 0x7f23740bb890 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23740afc10 msgr2=0x7f23740b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f2391d16700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23740afc10 0x7f23740b2120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f237c005950 tx=0x7f237c00b410 comp rx=0 tx=0).stop 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f238c10e770 msgr2=0x7f238c103a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f2391d16700 1 --2- 172.21.15.31:0/3579730667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f238c10e770 0x7f238c103a70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f238400b830 tx=0x7f2384004ce0 comp rx=0 tx=0).stop 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f238bfff700 1 -- 172.21.15.31:0/3579730667 reap_dead start 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 shutdown_connections 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.233+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 >> 172.21.15.31:0/3579730667 conn(0x7f238c0fecb0 msgr2=0x7f238c100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:47.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.234+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 shutdown_connections 2024-04-25T15:59:47.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:45.234+0000 7f2391d16700 1 -- 172.21.15.31:0/3579730667 wait complete. 2024-04-25T15:59:48.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.947+0000 7f757c9ba700 1 -- 172.21.15.31:0/1396993932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7578102640 msgr2=0x7f7578102a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.947+0000 7f757c9ba700 1 --2- 172.21.15.31:0/1396993932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7578102640 0x7f7578102a50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7560009990 tx=0x7f7560009ca0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.947+0000 7f757c9ba700 1 -- 172.21.15.31:0/1396993932 shutdown_connections 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.947+0000 7f757c9ba700 1 --2- 172.21.15.31:0/1396993932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f75781091b0 0x7f75781095c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.947+0000 7f757c9ba700 1 --2- 172.21.15.31:0/1396993932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7578102f90 0x7f7578111020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.947+0000 7f757c9ba700 1 --2- 172.21.15.31:0/1396993932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7578102640 0x7f7578102a50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.947+0000 7f757c9ba700 1 -- 172.21.15.31:0/1396993932 >> 172.21.15.31:0/1396993932 conn(0x7f75780fe370 msgr2=0x7f75781007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.947+0000 7f757c9ba700 1 -- 172.21.15.31:0/1396993932 shutdown_connections 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.947+0000 7f757c9ba700 1 -- 172.21.15.31:0/1396993932 wait complete. 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.948+0000 7f757c9ba700 1 Processor -- start 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.948+0000 7f757c9ba700 1 -- start start 2024-04-25T15:59:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.948+0000 7f757c9ba700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7578102640 0x7f757819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.948+0000 7f757c9ba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7578102f90 0x7f757819b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.948+0000 7f757c9ba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75781091b0 0x7f757819f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.948+0000 7f757c9ba700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7578114450 con 0x7f75781091b0 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.948+0000 7f757c9ba700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f75781142d0 con 0x7f7578102640 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.948+0000 7f757c9ba700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f75781145d0 con 0x7f7578102f90 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7576d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75781091b0 0x7f757819f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7576d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75781091b0 0x7f757819f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51494/0 (socket says 172.21.15.31:51494) 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7576d9d700 1 -- 172.21.15.31:0/2728371857 learned_addr learned my addr 172.21.15.31:0/2728371857 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f757659c700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7578102640 0x7f757819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7575d9b700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7578102f90 0x7f757819b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7576d9d700 1 -- 172.21.15.31:0/2728371857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7578102f90 msgr2=0x7f757819b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7576d9d700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7578102f90 0x7f757819b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7576d9d700 1 -- 172.21.15.31:0/2728371857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7578102640 msgr2=0x7f757819ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7576d9d700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7578102640 0x7f757819ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7576d9d700 1 -- 172.21.15.31:0/2728371857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f757819fc70 con 0x7f75781091b0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f7576d9d700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75781091b0 0x7f757819f5d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7570007ee0 tx=0x7f757000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f756f7fe700 1 -- 172.21.15.31:0/2728371857 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570004bb0 con 0x7f75781091b0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f756f7fe700 1 -- 172.21.15.31:0/2728371857 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7570004d50 con 0x7f75781091b0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f757819ff60 con 0x7f75781091b0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.949+0000 7f756f7fe700 1 -- 172.21.15.31:0/2728371857 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7570005920 con 0x7f75781091b0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.950+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75781a7880 con 0x7f75781091b0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.950+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7578063da0 con 0x7f75781091b0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.952+0000 7f756f7fe700 1 -- 172.21.15.31:0/2728371857 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f757001a020 con 0x7f75781091b0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.953+0000 7f756f7fe700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f75640afc10 0x7f75640b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.953+0000 7f757659c700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f75640afc10 0x7f75640b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.953+0000 7f756f7fe700 1 -- 172.21.15.31:0/2728371857 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f75700dea80 con 0x7f75781091b0 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.953+0000 7f756f7fe700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] conn(0x7f75640b9440 0x7f75640bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.953+0000 7f756f7fe700 1 -- 172.21.15.31:0/2728371857 --> [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7570024ea0 con 0x7f75640b9440 2024-04-25T15:59:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.953+0000 7f756f7fe700 1 -- 172.21.15.31:0/2728371857 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7f75700dee30 con 0x7f75781091b0 2024-04-25T15:59:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.953+0000 7f7575d9b700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] conn(0x7f75640b9440 0x7f75640bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.953+0000 7f757659c700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f75640afc10 0x7f75640b2120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f757806c940 tx=0x7f756800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.953+0000 7f7575d9b700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] conn(0x7f75640b9440 0x7f75640bb890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.954+0000 7f756f7fe700 1 -- 172.21.15.31:0/2728371857 <== osd.3 v2:172.21.15.31:6810/1212428989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7570024ea0 con 0x7f75640b9440 2024-04-25T15:59:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.971+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 --> [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f75780641a0 con 0x7f75640b9440 2024-04-25T15:59:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.971+0000 7f756f7fe700 1 -- 172.21.15.31:0/2728371857 <== osd.3 v2:172.21.15.31:6810/1212428989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f75780641a0 con 0x7f75640b9440 2024-04-25T15:59:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.971+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] conn(0x7f75640b9440 msgr2=0x7f75640bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.971+0000 7f757c9ba700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] conn(0x7f75640b9440 0x7f75640bb890 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.972+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f75640afc10 msgr2=0x7f75640b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.972+0000 7f757c9ba700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f75640afc10 0x7f75640b2120 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f757806c940 tx=0x7f756800a380 comp rx=0 tx=0).stop 2024-04-25T15:59:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.972+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75781091b0 msgr2=0x7f757819f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.972+0000 7f757c9ba700 1 --2- 172.21.15.31:0/2728371857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f75781091b0 0x7f757819f5d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7570007ee0 tx=0x7f757000d3b0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.972+0000 7f7576d9d700 1 -- 172.21.15.31:0/2728371857 reap_dead start 2024-04-25T15:59:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.972+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 shutdown_connections 2024-04-25T15:59:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.972+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 >> 172.21.15.31:0/2728371857 conn(0x7f75780fe370 msgr2=0x7f75780fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.972+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 shutdown_connections 2024-04-25T15:59:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:46.972+0000 7f757c9ba700 1 -- 172.21.15.31:0/2728371857 wait complete. 2024-04-25T15:59:48.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.065+0000 7fec9f6f9700 1 -- 172.21.15.31:0/1997408975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec9810e760 msgr2=0x7fec98110c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.065+0000 7fec9f6f9700 1 --2- 172.21.15.31:0/1997408975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec9810e760 0x7fec98110c40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fec9000b0a0 tx=0x7fec9000b3b0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.065+0000 7fec9f6f9700 1 -- 172.21.15.31:0/1997408975 shutdown_connections 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.065+0000 7fec9f6f9700 1 --2- 172.21.15.31:0/1997408975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec981052f0 0x7fec98105700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.065+0000 7fec9f6f9700 1 --2- 172.21.15.31:0/1997408975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec9810e760 0x7fec98110c40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.065+0000 7fec9f6f9700 1 --2- 172.21.15.31:0/1997408975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98105c40 0x7fec9810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.065+0000 7fec9f6f9700 1 -- 172.21.15.31:0/1997408975 >> 172.21.15.31:0/1997408975 conn(0x7fec980fec30 msgr2=0x7fec98101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.065+0000 7fec9f6f9700 1 -- 172.21.15.31:0/1997408975 shutdown_connections 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.065+0000 7fec9f6f9700 1 -- 172.21.15.31:0/1997408975 wait complete. 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9f6f9700 1 Processor -- start 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9f6f9700 1 -- start start 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9f6f9700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec981052f0 0x7fec981a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9f6f9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98105c40 0x7fec981a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9f6f9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec9810e760 0x7fec9819aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9f6f9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec981142c0 con 0x7fec9810e760 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9f6f9700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec98114140 con 0x7fec981052f0 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9f6f9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec98114440 con 0x7fec98105c40 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9dc96700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec9810e760 0x7fec9819aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9dc96700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec9810e760 0x7fec9819aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51514/0 (socket says 172.21.15.31:51514) 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9dc96700 1 -- 172.21.15.31:0/3862179601 learned_addr learned my addr 172.21.15.31:0/3862179601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9cc94700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98105c40 0x7fec981a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9d495700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec981052f0 0x7fec981a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9dc96700 1 -- 172.21.15.31:0/3862179601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98105c40 msgr2=0x7fec981a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9dc96700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98105c40 0x7fec981a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9dc96700 1 -- 172.21.15.31:0/3862179601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec981052f0 msgr2=0x7fec981a07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9dc96700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec981052f0 0x7fec981a07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9dc96700 1 -- 172.21.15.31:0/3862179601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9819b0b0 con 0x7fec9810e760 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.066+0000 7fec9cc94700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec98105c40 0x7fec981a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.067+0000 7fec9dc96700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec9810e760 0x7fec9819aa00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec900062a0 tx=0x7fec90004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.067+0000 7fec967fc700 1 -- 172.21.15.31:0/3862179601 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90021070 con 0x7fec9810e760 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.067+0000 7fec9f6f9700 1 -- 172.21.15.31:0/3862179601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9819b340 con 0x7fec9810e760 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.067+0000 7fec967fc700 1 -- 172.21.15.31:0/3862179601 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec90003c00 con 0x7fec9810e760 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.067+0000 7fec967fc700 1 -- 172.21.15.31:0/3862179601 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec90004830 con 0x7fec9810e760 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.067+0000 7fec9f6f9700 1 -- 172.21.15.31:0/3862179601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec981a7670 con 0x7fec9810e760 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.068+0000 7fec7ffff700 1 -- 172.21.15.31:0/3862179601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fec98063e20 con 0x7fec9810e760 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.070+0000 7fec967fc700 1 -- 172.21.15.31:0/3862179601 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fec90007a00 con 0x7fec9810e760 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.070+0000 7fec967fc700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec800afc10 0x7fec800b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.070+0000 7fec9d495700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec800afc10 0x7fec800b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.071+0000 7fec967fc700 1 -- 172.21.15.31:0/3862179601 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7fec9000f020 con 0x7fec9810e760 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.071+0000 7fec967fc700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] conn(0x7fec800b9440 0x7fec800bb890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:48.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.071+0000 7fec967fc700 1 -- 172.21.15.31:0/3862179601 --> [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec9001ce50 con 0x7fec800b9440 2024-04-25T15:59:48.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.071+0000 7fec967fc700 1 -- 172.21.15.31:0/3862179601 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=75) ==== 24+0+0 (secure 0 0 0) 0x7fec900e0870 con 0x7fec9810e760 2024-04-25T15:59:48.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.071+0000 7fec9d495700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec800afc10 0x7fec800b2120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fec88005950 tx=0x7fec880058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:48.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.071+0000 7fec9cc94700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] conn(0x7fec800b9440 0x7fec800bb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:48.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.072+0000 7fec9cc94700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] conn(0x7fec800b9440 0x7fec800bb890 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:48.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.072+0000 7fec967fc700 1 -- 172.21.15.31:0/3862179601 <== osd.7 v2:172.21.15.63:6816/4096527906 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fec9001ce50 con 0x7fec800b9440 2024-04-25T15:59:48.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.089+0000 7fec7ffff700 1 -- 172.21.15.31:0/3862179601 --> [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fec980642a0 con 0x7fec800b9440 2024-04-25T15:59:48.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec967fc700 1 -- 172.21.15.31:0/3862179601 <== osd.7 v2:172.21.15.63:6816/4096527906 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fec980642a0 con 0x7fec800b9440 2024-04-25T15:59:48.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec7ffff700 1 -- 172.21.15.31:0/3862179601 >> [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] conn(0x7fec800b9440 msgr2=0x7fec800bb890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec7ffff700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] conn(0x7fec800b9440 0x7fec800bb890 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec7ffff700 1 -- 172.21.15.31:0/3862179601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec800afc10 msgr2=0x7fec800b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec7ffff700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec800afc10 0x7fec800b2120 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fec88005950 tx=0x7fec880058e0 comp rx=0 tx=0).stop 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec7ffff700 1 -- 172.21.15.31:0/3862179601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec9810e760 msgr2=0x7fec9819aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec7ffff700 1 --2- 172.21.15.31:0/3862179601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec9810e760 0x7fec9819aa00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec900062a0 tx=0x7fec90004d30 comp rx=0 tx=0).stop 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec9dc96700 1 -- 172.21.15.31:0/3862179601 reap_dead start 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec7ffff700 1 -- 172.21.15.31:0/3862179601 shutdown_connections 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.090+0000 7fec7ffff700 1 -- 172.21.15.31:0/3862179601 >> 172.21.15.31:0/3862179601 conn(0x7fec980fec30 msgr2=0x7fec98100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.091+0000 7fec7ffff700 1 -- 172.21.15.31:0/3862179601 shutdown_connections 2024-04-25T15:59:48.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:47.091+0000 7fec7ffff700 1 -- 172.21.15.31:0/3862179601 wait complete. 2024-04-25T15:59:48.903 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:48 smithi031 ceph-mon[29140]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T15:59:49.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:48 smithi063 ceph-mon[32413]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T15:59:49.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:48 smithi186 ceph-mon[33258]: pgmap v304: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T15:59:49.409 INFO:teuthology.orchestra.run.smithi031.stdout:133143986235 2024-04-25T15:59:49.409 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-25T15:59:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-25T15:59:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-25T15:59:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-04-25T15:59:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-25T15:59:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-25T15:59:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-25T15:59:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-25T15:59:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-25T15:59:50.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-25T15:59:50.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-25T15:59:50.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-04-25T15:59:50.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-25T15:59:50.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-25T15:59:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-25T15:59:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-25T15:59:50.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-25T15:59:50.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]: dispatch 2024-04-25T15:59:50.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-04-25T15:59:50.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]: dispatch 2024-04-25T15:59:50.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]: dispatch 2024-04-25T15:59:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]: dispatch 2024-04-25T15:59:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2024-04-25T15:59:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-04-25T15:59:50.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-25T15:59:50.466 INFO:teuthology.orchestra.run.smithi031.stdout:274877906972 2024-04-25T15:59:50.466 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-25T15:59:50.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T15:59:50.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-25T15:59:50.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-25T15:59:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-04-25T15:59:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-25T15:59:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-25T15:59:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-25T15:59:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-25T15:59:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-25T15:59:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:50 smithi031 ceph-mon[29140]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T15:59:51.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T15:59:51.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-25T15:59:51.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-25T15:59:51.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-04-25T15:59:51.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-25T15:59:51.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-25T15:59:51.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-25T15:59:51.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-25T15:59:51.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-25T15:59:51.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:50 smithi063 ceph-mon[32413]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T15:59:51.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: pgmap v305: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T15:59:51.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 1]}]': finished 2024-04-25T15:59:51.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-04-25T15:59:51.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 8, 11]}]': finished 2024-04-25T15:59:51.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.b", "id": [3, 9]}]': finished 2024-04-25T15:59:51.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 6]}]': finished 2024-04-25T15:59:51.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2024-04-25T15:59:51.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-04-25T15:59:51.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-25T15:59:51.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:50 smithi186 ceph-mon[33258]: osdmap e76: 12 total, 12 up, 12 in 2024-04-25T15:59:51.320 INFO:teuthology.orchestra.run.smithi031.stdout:107374182463 2024-04-25T15:59:51.320 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-25T15:59:51.723 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:51.759 INFO:teuthology.orchestra.run.smithi031.stdout:201863462956 2024-04-25T15:59:51.760 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-25T15:59:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.729+0000 7f9968dec700 1 -- 172.21.15.31:0/1429004235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9964105c80 msgr2=0x7f996410e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.729+0000 7f9968dec700 1 --2- 172.21.15.31:0/1429004235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9964105c80 0x7f996410e1d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9958009990 tx=0x7f9958009ca0 comp rx=0 tx=0).stop 2024-04-25T15:59:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.729+0000 7f9968dec700 1 -- 172.21.15.31:0/1429004235 shutdown_connections 2024-04-25T15:59:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.729+0000 7f9968dec700 1 --2- 172.21.15.31:0/1429004235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9964105330 0x7f9964105740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.729+0000 7f9968dec700 1 --2- 172.21.15.31:0/1429004235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f996410e7a0 0x7f9964110cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.729+0000 7f9968dec700 1 --2- 172.21.15.31:0/1429004235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9964105c80 0x7f996410e1d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:51.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.729+0000 7f9968dec700 1 -- 172.21.15.31:0/1429004235 >> 172.21.15.31:0/1429004235 conn(0x7f99640fecc0 msgr2=0x7f99641010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.729+0000 7f9968dec700 1 -- 172.21.15.31:0/1429004235 shutdown_connections 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.729+0000 7f9968dec700 1 -- 172.21.15.31:0/1429004235 wait complete. 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.730+0000 7f9968dec700 1 Processor -- start 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.730+0000 7f9968dec700 1 -- start start 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.730+0000 7f9968dec700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9964105330 0x7f99641a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.730+0000 7f9968dec700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9964105c80 0x7f99641a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.730+0000 7f9968dec700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f996410e7a0 0x7f996419ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.730+0000 7f9968dec700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9964114330 con 0x7f9964105330 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.730+0000 7f9968dec700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f99641141b0 con 0x7f996410e7a0 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.730+0000 7f9968dec700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99641144b0 con 0x7f9964105c80 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f996259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9964105330 0x7f99641a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f996259c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9964105330 0x7f99641a09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51544/0 (socket says 172.21.15.31:51544) 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f996259c700 1 -- 172.21.15.31:0/1871768332 learned_addr learned my addr 172.21.15.31:0/1871768332 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:51.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f9962d9d700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f996410e7a0 0x7f996419ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f9961d9b700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9964105c80 0x7f99641a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f996259c700 1 -- 172.21.15.31:0/1871768332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9964105c80 msgr2=0x7f99641a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f996259c700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9964105c80 0x7f99641a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f996259c700 1 -- 172.21.15.31:0/1871768332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f996410e7a0 msgr2=0x7f996419ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f996259c700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f996410e7a0 0x7f996419ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f996259c700 1 -- 172.21.15.31:0/1871768332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f996419b1f0 con 0x7f9964105330 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f996259c700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9964105330 0x7f99641a09c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f995400d980 tx=0x7f995400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f99537fe700 1 -- 172.21.15.31:0/1871768332 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9954016070 con 0x7f9964105330 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f996419b4e0 con 0x7f9964105330 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f99537fe700 1 -- 172.21.15.31:0/1871768332 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f995400be00 con 0x7f9964105330 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f99537fe700 1 -- 172.21.15.31:0/1871768332 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995400f3c0 con 0x7f9964105330 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.731+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99641a7770 con 0x7f9964105330 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.732+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9964063e20 con 0x7f9964105330 2024-04-25T15:59:51.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.734+0000 7f99537fe700 1 -- 172.21.15.31:0/1871768332 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f9954019070 con 0x7f9964105330 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.734+0000 7f99537fe700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f994c0afc10 0x7f994c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.735+0000 7f9961d9b700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f994c0afc10 0x7f994c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.735+0000 7f99537fe700 1 -- 172.21.15.31:0/1871768332 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f99540df230 con 0x7f9964105330 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.735+0000 7f99537fe700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] conn(0x7f994c0b97a0 0x7f994c0bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.735+0000 7f99537fe700 1 -- 172.21.15.31:0/1871768332 --> [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f994c0bc360 con 0x7f994c0b97a0 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.735+0000 7f99537fe700 1 -- 172.21.15.31:0/1871768332 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f99540df680 con 0x7f9964105330 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.735+0000 7f9962d9d700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] conn(0x7f994c0b97a0 0x7f994c0bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.735+0000 7f9961d9b700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f994c0afc10 0x7f994c0b2120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9958000c00 tx=0x7f995801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.735+0000 7f9962d9d700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] conn(0x7f994c0b97a0 0x7f994c0bbc90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.736+0000 7f99537fe700 1 -- 172.21.15.31:0/1871768332 <== osd.6 v2:172.21.15.31:6818/3847071731 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f994c0bc360 con 0x7f994c0b97a0 2024-04-25T15:59:51.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.752+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 --> [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9964064250 con 0x7f994c0b97a0 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f99537fe700 1 -- 172.21.15.31:0/1871768332 <== osd.6 v2:172.21.15.31:6818/3847071731 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9964064250 con 0x7f994c0b97a0 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] conn(0x7f994c0b97a0 msgr2=0x7f994c0bbc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] conn(0x7f994c0b97a0 0x7f994c0bbc90 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f994c0afc10 msgr2=0x7f994c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f994c0afc10 0x7f994c0b2120 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9958000c00 tx=0x7f995801a040 comp rx=0 tx=0).stop 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9964105330 msgr2=0x7f99641a09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 --2- 172.21.15.31:0/1871768332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9964105330 0x7f99641a09c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f995400d980 tx=0x7f995400dc90 comp rx=0 tx=0).stop 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9962d9d700 1 -- 172.21.15.31:0/1871768332 reap_dead start 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 shutdown_connections 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 >> 172.21.15.31:0/1871768332 conn(0x7f99640fecc0 msgr2=0x7f9964100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 shutdown_connections 2024-04-25T15:59:51.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:50.753+0000 7f9968dec700 1 -- 172.21.15.31:0/1871768332 wait complete. 2024-04-25T15:59:51.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:51 smithi031 ceph-mon[29140]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T15:59:52.054 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:52.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:51 smithi063 ceph-mon[32413]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T15:59:52.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:51 smithi186 ceph-mon[33258]: osdmap e77: 12 total, 12 up, 12 in 2024-04-25T15:59:52.519 INFO:teuthology.orchestra.run.smithi031.stdout:180388626481 2024-04-25T15:59:52.519 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-25T15:59:52.784 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:52 smithi031 ceph-mon[29140]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-04-25T15:59:53.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:52 smithi063 ceph-mon[32413]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-04-25T15:59:53.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:52 smithi186 ceph-mon[33258]: pgmap v308: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 7 op/s 2024-04-25T15:59:54.721 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:54.724 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:54.725 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:54.846 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:54 smithi031 ceph-mon[29140]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 5 op/s; 148 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T15:59:54.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.962+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2466537355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74e810e7c0 msgr2=0x7f74e8110ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.962+0000 7f74ec8ee700 1 --2- 172.21.15.31:0/2466537355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74e810e7c0 0x7f74e8110ca0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f74e000b0a0 tx=0x7f74e000b3b0 comp rx=0 tx=0).stop 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.962+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2466537355 shutdown_connections 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.962+0000 7f74ec8ee700 1 --2- 172.21.15.31:0/2466537355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e8105390 0x7f74e81057a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.962+0000 7f74ec8ee700 1 --2- 172.21.15.31:0/2466537355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74e810e7c0 0x7f74e8110ca0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.962+0000 7f74ec8ee700 1 --2- 172.21.15.31:0/2466537355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e8105ce0 0x7f74e810e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.962+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2466537355 >> 172.21.15.31:0/2466537355 conn(0x7f74e80fecc0 msgr2=0x7f74e81010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.962+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2466537355 shutdown_connections 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.962+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2466537355 wait complete. 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74ec8ee700 1 Processor -- start 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74ec8ee700 1 -- start start 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74ec8ee700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e8105390 0x7f74e81a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74ec8ee700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e8105ce0 0x7f74e81a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:54.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74ec8ee700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74e810e7c0 0x7f74e819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74ec8ee700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f74e8114320 con 0x7f74e810e7c0 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74ec8ee700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f74e81141a0 con 0x7f74e8105390 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74ec8ee700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74e81144a0 con 0x7f74e8105ce0 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74e6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74e810e7c0 0x7f74e819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74e6d9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74e810e7c0 0x7f74e819aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59768/0 (socket says 172.21.15.31:59768) 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74e6d9d700 1 -- 172.21.15.31:0/2689477476 learned_addr learned my addr 172.21.15.31:0/2689477476 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.963+0000 7f74e659c700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e8105390 0x7f74e81a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74e5d9b700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e8105ce0 0x7f74e81a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74e6d9d700 1 -- 172.21.15.31:0/2689477476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e8105ce0 msgr2=0x7f74e81a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74e6d9d700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74e8105ce0 0x7f74e81a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74e6d9d700 1 -- 172.21.15.31:0/2689477476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e8105390 msgr2=0x7f74e81a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74e6d9d700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74e8105390 0x7f74e81a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74e6d9d700 1 -- 172.21.15.31:0/2689477476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74e819b110 con 0x7f74e810e7c0 2024-04-25T15:59:54.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74e6d9d700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74e810e7c0 0x7f74e819aa60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f74e000b830 tx=0x7f74e0004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74d77fe700 1 -- 172.21.15.31:0/2689477476 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e0021070 con 0x7f74e810e7c0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74e819b3a0 con 0x7f74e810e7c0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74d77fe700 1 -- 172.21.15.31:0/2689477476 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74e0007e30 con 0x7f74e810e7c0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74d77fe700 1 -- 172.21.15.31:0/2689477476 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e00047d0 con 0x7f74e810e7c0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.964+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74e81a76d0 con 0x7f74e810e7c0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.965+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f74e8063e20 con 0x7f74e810e7c0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.967+0000 7f74d77fe700 1 -- 172.21.15.31:0/2689477476 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f74e0007980 con 0x7f74e810e7c0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.967+0000 7f74d77fe700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74d00afc10 0x7f74d00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.967+0000 7f74e659c700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74d00afc10 0x7f74d00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.967+0000 7f74d77fe700 1 -- 172.21.15.31:0/2689477476 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f74e000f020 con 0x7f74e810e7c0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.968+0000 7f74d77fe700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] conn(0x7f74d00b97a0 0x7f74d00bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.968+0000 7f74d77fe700 1 -- 172.21.15.31:0/2689477476 --> [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f74d00bc360 con 0x7f74d00b97a0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.968+0000 7f74d77fe700 1 -- 172.21.15.31:0/2689477476 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7f74e00e09d0 con 0x7f74e810e7c0 2024-04-25T15:59:54.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.968+0000 7f74e659c700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74d00afc10 0x7f74d00b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f74e819bba0 tx=0x7f74d800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:54.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.968+0000 7f74e5d9b700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] conn(0x7f74d00b97a0 0x7f74d00bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:54.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.968+0000 7f74e5d9b700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] conn(0x7f74d00b97a0 0x7f74d00bbc90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:54.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.969+0000 7f74d77fe700 1 -- 172.21.15.31:0/2689477476 <== osd.1 v2:172.21.15.63:6800/1253577374 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f74d00bc360 con 0x7f74d00b97a0 2024-04-25T15:59:54.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.986+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 --> [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f74e80642a0 con 0x7f74d00b97a0 2024-04-25T15:59:54.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.987+0000 7f74d77fe700 1 -- 172.21.15.31:0/2689477476 <== osd.1 v2:172.21.15.63:6800/1253577374 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f74e80642a0 con 0x7f74d00b97a0 2024-04-25T15:59:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.987+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 >> [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] conn(0x7f74d00b97a0 msgr2=0x7f74d00bbc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.987+0000 7f74ec8ee700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] conn(0x7f74d00b97a0 0x7f74d00bbc90 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.988+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74d00afc10 msgr2=0x7f74d00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.988+0000 7f74ec8ee700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74d00afc10 0x7f74d00b2120 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f74e819bba0 tx=0x7f74d800b410 comp rx=0 tx=0).stop 2024-04-25T15:59:54.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.988+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74e810e7c0 msgr2=0x7f74e819aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.988+0000 7f74ec8ee700 1 --2- 172.21.15.31:0/2689477476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74e810e7c0 0x7f74e819aa60 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f74e000b830 tx=0x7f74e0004ce0 comp rx=0 tx=0).stop 2024-04-25T15:59:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.988+0000 7f74e6d9d700 1 -- 172.21.15.31:0/2689477476 reap_dead start 2024-04-25T15:59:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.988+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 shutdown_connections 2024-04-25T15:59:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.988+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 >> 172.21.15.31:0/2689477476 conn(0x7f74e80fecc0 msgr2=0x7f74e8100a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.988+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 shutdown_connections 2024-04-25T15:59:54.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:54.988+0000 7f74ec8ee700 1 -- 172.21.15.31:0/2689477476 wait complete. 2024-04-25T15:59:55.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:54 smithi063 ceph-mon[32413]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 5 op/s; 148 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T15:59:55.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:54 smithi186 ceph-mon[33258]: pgmap v309: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 5 op/s; 148 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T15:59:55.757 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:56.341 INFO:teuthology.orchestra.run.smithi031.stdout:55834574924 2024-04-25T15:59:56.341 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-25T15:59:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.553+0000 7fa49ba85700 1 -- 172.21.15.31:0/4142447584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa49410a2b0 msgr2=0x7fa49410a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.553+0000 7fa49ba85700 1 --2- 172.21.15.31:0/4142447584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa49410a2b0 0x7fa49410a6c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa484009990 tx=0x7fa484009ca0 comp rx=0 tx=0).stop 2024-04-25T15:59:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.553+0000 7fa49ba85700 1 -- 172.21.15.31:0/4142447584 shutdown_connections 2024-04-25T15:59:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.553+0000 7fa49ba85700 1 --2- 172.21.15.31:0/4142447584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa494103000 0x7fa494110ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.553+0000 7fa49ba85700 1 --2- 172.21.15.31:0/4142447584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa49410a2b0 0x7fa49410a6c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.553+0000 7fa49ba85700 1 --2- 172.21.15.31:0/4142447584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4941026b0 0x7fa494102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:56.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.553+0000 7fa49ba85700 1 -- 172.21.15.31:0/4142447584 >> 172.21.15.31:0/4142447584 conn(0x7fa4940fe5c0 msgr2=0x7fa4941009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.553+0000 7fa49ba85700 1 -- 172.21.15.31:0/4142447584 shutdown_connections 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.554+0000 7fa49ba85700 1 -- 172.21.15.31:0/4142447584 wait complete. 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.554+0000 7fa49ba85700 1 Processor -- start 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.554+0000 7fa49ba85700 1 -- start start 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa49ba85700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4941026b0 0x7fa4941a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa49ba85700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa494103000 0x7fa4941a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa49ba85700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa49410a2b0 0x7fa49419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa49ba85700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa494114350 con 0x7fa494103000 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa49ba85700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa4941141d0 con 0x7fa4941026b0 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa49ba85700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa4941144d0 con 0x7fa49410a2b0 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499020700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa494103000 0x7fa4941a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499020700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa494103000 0x7fa4941a0ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59774/0 (socket says 172.21.15.31:59774) 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499020700 1 -- 172.21.15.31:0/2706719542 learned_addr learned my addr 172.21.15.31:0/2706719542 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa49a022700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa49410a2b0 0x7fa49419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:56.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499821700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4941026b0 0x7fa4941a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499020700 1 -- 172.21.15.31:0/2706719542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa49410a2b0 msgr2=0x7fa49419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499020700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa49410a2b0 0x7fa49419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499020700 1 -- 172.21.15.31:0/2706719542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4941026b0 msgr2=0x7fa4941a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499020700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4941026b0 0x7fa4941a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499020700 1 -- 172.21.15.31:0/2706719542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49419b260 con 0x7fa494103000 2024-04-25T15:59:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.555+0000 7fa499821700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa4941026b0 0x7fa4941a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T15:59:56.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.556+0000 7fa499020700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa494103000 0x7fa4941a0ee0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa49000d950 tx=0x7fa49000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.556+0000 7fa48affd700 1 -- 172.21.15.31:0/2706719542 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490016070 con 0x7fa494103000 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.556+0000 7fa48affd700 1 -- 172.21.15.31:0/2706719542 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa49000de30 con 0x7fa494103000 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.556+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49419b550 con 0x7fa494103000 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.556+0000 7fa48affd700 1 -- 172.21.15.31:0/2706719542 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa490005350 con 0x7fa494103000 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.556+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4941037c0 con 0x7fa494103000 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.557+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa494063e20 con 0x7fa494103000 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.558+0000 7fa48affd700 1 -- 172.21.15.31:0/2706719542 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa4900054f0 con 0x7fa494103000 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.559+0000 7fa48affd700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4800afc10 0x7fa4800b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.559+0000 7fa499821700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4800afc10 0x7fa4800b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.559+0000 7fa48affd700 1 -- 172.21.15.31:0/2706719542 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa4900deda0 con 0x7fa494103000 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.559+0000 7fa48affd700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] conn(0x7fa4800b97a0 0x7fa4800bbc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.559+0000 7fa48affd700 1 -- 172.21.15.31:0/2706719542 --> [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4800bc360 con 0x7fa4800b97a0 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.559+0000 7fa48affd700 1 -- 172.21.15.31:0/2706719542 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_get_version_reply(handle=1 version=77) ==== 24+0+0 (secure 0 0 0) 0x7fa4900df1f0 con 0x7fa494103000 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.560+0000 7fa499821700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4800afc10 0x7fa4800b2120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa484003800 tx=0x7fa484005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.560+0000 7fa49a022700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] conn(0x7fa4800b97a0 0x7fa4800bbc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:56.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.560+0000 7fa49a022700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] conn(0x7fa4800b97a0 0x7fa4800bbc90 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.561+0000 7fa48affd700 1 -- 172.21.15.31:0/2706719542 <== osd.2 v2:172.21.15.186:6800/1381996284 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa4800bc360 con 0x7fa4800b97a0 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.579+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 --> [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa494064220 con 0x7fa4800b97a0 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa48affd700 1 -- 172.21.15.31:0/2706719542 <== osd.2 v2:172.21.15.186:6800/1381996284 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa494064220 con 0x7fa4800b97a0 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 >> [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] conn(0x7fa4800b97a0 msgr2=0x7fa4800bbc90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] conn(0x7fa4800b97a0 0x7fa4800bbc90 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4800afc10 msgr2=0x7fa4800b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4800afc10 0x7fa4800b2120 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa484003800 tx=0x7fa484005fb0 comp rx=0 tx=0).stop 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa494103000 msgr2=0x7fa4941a0ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 --2- 172.21.15.31:0/2706719542 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa494103000 0x7fa4941a0ee0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa49000d950 tx=0x7fa49000dd10 comp rx=0 tx=0).stop 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49a022700 1 -- 172.21.15.31:0/2706719542 reap_dead start 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 shutdown_connections 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 >> 172.21.15.31:0/2706719542 conn(0x7fa4940fe5c0 msgr2=0x7fa494100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 shutdown_connections 2024-04-25T15:59:56.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:56.580+0000 7fa49ba85700 1 -- 172.21.15.31:0/2706719542 wait complete. 2024-04-25T15:59:56.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:56 smithi031 ceph-mon[29140]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3 op/s; 148 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T15:59:57.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:56 smithi063 ceph-mon[32413]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3 op/s; 148 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T15:59:57.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:56 smithi186 ceph-mon[33258]: pgmap v310: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3 op/s; 148 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T15:59:57.355 INFO:teuthology.orchestra.run.smithi031.stdout:77309411399 2024-04-25T15:59:57.355 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-25T15:59:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.713+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/974456185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34108630 msgr2=0x7f1b3410ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.713+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/974456185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34108630 0x7f1b3410ff90 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00b0d0 tx=0x7f1b2c00b3e0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.714+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/974456185 shutdown_connections 2024-04-25T15:59:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.714+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/974456185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34108630 0x7f1b3410ff90 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.714+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/974456185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b341033b0 0x7f1b341037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.714+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/974456185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b34107ce0 0x7f1b341080f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.714+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/974456185 >> 172.21.15.31:0/974456185 conn(0x7f1b340fee00 msgr2=0x7f1b34101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.714+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/974456185 shutdown_connections 2024-04-25T15:59:58.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.714+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/974456185 wait complete. 2024-04-25T15:59:58.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.714+0000 7f1b3a0ba700 1 Processor -- start 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b3a0ba700 1 -- start start 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b3a0ba700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b341033b0 0x7f1b3419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b3a0ba700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34107ce0 0x7f1b3419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b3a0ba700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b34108630 0x7f1b34199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b3a0ba700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1b34113260 con 0x7f1b34107ce0 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b3a0ba700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341130e0 con 0x7f1b341033b0 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b3a0ba700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341133e0 con 0x7f1b34108630 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b32ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34107ce0 0x7f1b3419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b32ffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34107ce0 0x7f1b3419fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59790/0 (socket says 172.21.15.31:59790) 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b32ffd700 1 -- 172.21.15.31:0/2786978671 learned_addr learned my addr 172.21.15.31:0/2786978671 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b33fff700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b34108630 0x7f1b34199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b337fe700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b341033b0 0x7f1b3419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:58.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b32ffd700 1 -- 172.21.15.31:0/2786978671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b34108630 msgr2=0x7f1b34199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b32ffd700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b34108630 0x7f1b34199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b32ffd700 1 -- 172.21.15.31:0/2786978671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b341033b0 msgr2=0x7f1b3419f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.715+0000 7f1b32ffd700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b341033b0 0x7f1b3419f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.716+0000 7f1b32ffd700 1 -- 172.21.15.31:0/2786978671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3419a0b0 con 0x7f1b34107ce0 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.716+0000 7f1b337fe700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b341033b0 0x7f1b3419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.716+0000 7f1b32ffd700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34107ce0 0x7f1b3419fd20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800d980 tx=0x7f1b2800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.716+0000 7f1b30ff9700 1 -- 172.21.15.31:0/2786978671 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28017070 con 0x7f1b34107ce0 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.716+0000 7f1b30ff9700 1 -- 172.21.15.31:0/2786978671 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b28009de0 con 0x7f1b34107ce0 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.716+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3419a3a0 con 0x7f1b34107ce0 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.716+0000 7f1b30ff9700 1 -- 172.21.15.31:0/2786978671 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2800f3c0 con 0x7f1b34107ce0 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.716+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3406bf10 con 0x7f1b34107ce0 2024-04-25T15:59:58.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.717+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b34051190 con 0x7f1b34107ce0 2024-04-25T15:59:58.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.719+0000 7f1b30ff9700 1 -- 172.21.15.31:0/2786978671 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f1b28009930 con 0x7f1b34107ce0 2024-04-25T15:59:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.720+0000 7f1b30ff9700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1b1c0afc90 0x7f1b1c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.720+0000 7f1b337fe700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1b1c0afc90 0x7f1b1c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.720+0000 7f1b30ff9700 1 -- 172.21.15.31:0/2786978671 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1b280de800 con 0x7f1b34107ce0 2024-04-25T15:59:58.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.721+0000 7f1b337fe700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1b1c0afc90 0x7f1b1c0b21a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1b24005950 tx=0x7f1b240058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:58.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.724+0000 7f1b30ff9700 1 -- 172.21.15.31:0/2786978671 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b280a72b0 con 0x7f1b34107ce0 2024-04-25T15:59:58.799 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:58.900 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:58.900 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T15:59:58.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.915+0000 7f1812ebe700 1 -- 172.21.15.31:0/2721632292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c107c90 msgr2=0x7f180c1080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:58.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.915+0000 7f1812ebe700 1 --2- 172.21.15.31:0/2721632292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c107c90 0x7f180c1080a0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f17f80099c0 tx=0x7f17f8009cd0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.915+0000 7f1812ebe700 1 -- 172.21.15.31:0/2721632292 shutdown_connections 2024-04-25T15:59:58.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.915+0000 7f1812ebe700 1 --2- 172.21.15.31:0/2721632292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f180c103360 0x7f180c103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.915+0000 7f1812ebe700 1 --2- 172.21.15.31:0/2721632292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f180c1085e0 0x7f180c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.915+0000 7f1812ebe700 1 --2- 172.21.15.31:0/2721632292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c107c90 0x7f180c1080a0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.915+0000 7f1812ebe700 1 -- 172.21.15.31:0/2721632292 >> 172.21.15.31:0/2721632292 conn(0x7f180c0fecb0 msgr2=0x7f180c1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:58.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.916+0000 7f1812ebe700 1 -- 172.21.15.31:0/2721632292 shutdown_connections 2024-04-25T15:59:58.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.916+0000 7f1812ebe700 1 -- 172.21.15.31:0/2721632292 wait complete. 2024-04-25T15:59:58.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.917+0000 7f1812ebe700 1 Processor -- start 2024-04-25T15:59:58.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.917+0000 7f1812ebe700 1 -- start start 2024-04-25T15:59:58.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1812ebe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c103360 0x7f180c19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:58.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1812ebe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f180c107c90 0x7f180c19fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:58.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1812ebe700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f180c1085e0 0x7f180c199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:58.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1812ebe700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f180c1131f0 con 0x7f180c103360 2024-04-25T15:59:58.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1812ebe700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f180c113070 con 0x7f180c1085e0 2024-04-25T15:59:58.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1812ebe700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f180c113370 con 0x7f180c107c90 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1810c5a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c103360 0x7f180c19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1810c5a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c103360 0x7f180c19f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59812/0 (socket says 172.21.15.31:59812) 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1810c5a700 1 -- 172.21.15.31:0/285439010 learned_addr learned my addr 172.21.15.31:0/285439010 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f181145b700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f180c1085e0 0x7f180c199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f180bfff700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f180c107c90 0x7f180c19fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1810c5a700 1 -- 172.21.15.31:0/285439010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f180c107c90 msgr2=0x7f180c19fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1810c5a700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f180c107c90 0x7f180c19fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1810c5a700 1 -- 172.21.15.31:0/285439010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f180c1085e0 msgr2=0x7f180c199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1810c5a700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f180c1085e0 0x7f180c199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.918+0000 7f1810c5a700 1 -- 172.21.15.31:0/285439010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180c199f50 con 0x7f180c103360 2024-04-25T15:59:58.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.919+0000 7f1810c5a700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c103360 0x7f180c19f7b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f180000e9d0 tx=0x7f180000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:58.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.919+0000 7f1809ffb700 1 -- 172.21.15.31:0/285439010 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180000cbd0 con 0x7f180c103360 2024-04-25T15:59:58.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.919+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f180c19a1e0 con 0x7f180c103360 2024-04-25T15:59:58.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.919+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f180c19a740 con 0x7f180c103360 2024-04-25T15:59:58.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.919+0000 7f1809ffb700 1 -- 172.21.15.31:0/285439010 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f180000cd70 con 0x7f180c103360 2024-04-25T15:59:58.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.920+0000 7f1809ffb700 1 -- 172.21.15.31:0/285439010 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1800018710 con 0x7f180c103360 2024-04-25T15:59:58.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.921+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f180c051190 con 0x7f180c103360 2024-04-25T15:59:58.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.923+0000 7f1809ffb700 1 -- 172.21.15.31:0/285439010 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f1800010430 con 0x7f180c103360 2024-04-25T15:59:58.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.924+0000 7f1809ffb700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17fc0afe10 0x7f17fc0b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T15:59:58.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.925+0000 7f1809ffb700 1 -- 172.21.15.31:0/285439010 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f18000184b0 con 0x7f180c103360 2024-04-25T15:59:58.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.925+0000 7f180bfff700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17fc0afe10 0x7f17fc0b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T15:59:58.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.926+0000 7f180bfff700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17fc0afe10 0x7f17fc0b2320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f17f8000c00 tx=0x7f17f801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T15:59:58.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.932+0000 7f1809ffb700 1 -- 172.21.15.31:0/285439010 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f18000a7a80 con 0x7f180c103360 2024-04-25T15:59:58.934 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:58 smithi031 ceph-mon[29140]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4 op/s; 148 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T15:59:58.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.960+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f1b3406c940 con 0x7f1b34107ce0 2024-04-25T15:59:58.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.960+0000 7f1b30ff9700 1 -- 172.21.15.31:0/2786978671 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f1b280a6bb0 con 0x7f1b34107ce0 2024-04-25T15:59:58.961 INFO:teuthology.orchestra.run.smithi031.stdout:227633266729 2024-04-25T15:59:58.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1b1c0afc90 msgr2=0x7f1b1c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:58.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1b1c0afc90 0x7f1b1c0b21a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1b24005950 tx=0x7f1b240058e0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34107ce0 msgr2=0x7f1b3419fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:58.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34107ce0 0x7f1b3419fd20 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800d980 tx=0x7f1b2800dc90 comp rx=0 tx=0).stop 2024-04-25T15:59:58.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 shutdown_connections 2024-04-25T15:59:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1b1c0afc90 0x7f1b1c0b21a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1b341033b0 0x7f1b3419f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b34108630 0x7f1b34199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 --2- 172.21.15.31:0/2786978671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1b34107ce0 0x7f1b3419fd20 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 >> 172.21.15.31:0/2786978671 conn(0x7f1b340fee00 msgr2=0x7f1b34107450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.962+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 shutdown_connections 2024-04-25T15:59:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:58.963+0000 7f1b3a0ba700 1 -- 172.21.15.31:0/2786978671 wait complete. 2024-04-25T15:59:59.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:58 smithi063 ceph-mon[32413]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4 op/s; 148 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T15:59:59.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.147+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f180c10b070 con 0x7f180c103360 2024-04-25T15:59:59.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.147+0000 7f1809ffb700 1 -- 172.21.15.31:0/285439010 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f18000a7380 con 0x7f180c103360 2024-04-25T15:59:59.148 INFO:teuthology.orchestra.run.smithi031.stdout:154618822711 2024-04-25T15:59:59.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:58 smithi186 ceph-mon[33258]: pgmap v311: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 4 op/s; 148 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T15:59:59.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17fc0afe10 msgr2=0x7f17fc0b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:59.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17fc0afe10 0x7f17fc0b2320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f17f8000c00 tx=0x7f17f801a040 comp rx=0 tx=0).stop 2024-04-25T15:59:59.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c103360 msgr2=0x7f180c19f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T15:59:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c103360 0x7f180c19f7b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f180000e9d0 tx=0x7f180000ece0 comp rx=0 tx=0).stop 2024-04-25T15:59:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 shutdown_connections 2024-04-25T15:59:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17fc0afe10 0x7f17fc0b2320 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f180c1085e0 0x7f180c199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f180c103360 0x7f180c19f7b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 --2- 172.21.15.31:0/285439010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f180c107c90 0x7f180c19fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T15:59:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.149+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 >> 172.21.15.31:0/285439010 conn(0x7f180c0fecb0 msgr2=0x7f180c107430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T15:59:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.150+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 shutdown_connections 2024-04-25T15:59:59.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T15:59:59.150+0000 7f1812ebe700 1 -- 172.21.15.31:0/285439010 wait complete. 2024-04-25T15:59:59.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:59 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2786978671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T15:59:59.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 15:59:59 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/285439010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T16:00:00.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:59 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2786978671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T16:00:00.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 15:59:59 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/285439010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T16:00:00.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:59 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2786978671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-25T16:00:00.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 15:59:59 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/285439010' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-25T16:00:00.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:00 smithi031 ceph-mon[29140]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 5 op/s; 123 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:00:00.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:00 smithi031 ceph-mon[29140]: overall HEALTH_OK 2024-04-25T16:00:01.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:00 smithi063 ceph-mon[32413]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 5 op/s; 123 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:00:01.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:00 smithi063 ceph-mon[32413]: overall HEALTH_OK 2024-04-25T16:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:00 smithi186 ceph-mon[33258]: pgmap v312: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 5 op/s; 123 B/s, 1 keys/s, 1 objects/s recovering 2024-04-25T16:00:01.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:00 smithi186 ceph-mon[33258]: overall HEALTH_OK 2024-04-25T16:00:02.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:02 smithi031 ceph-mon[29140]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5 op/s; 111 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:00:02.950 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:02 smithi186 ceph-mon[33258]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5 op/s; 111 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:00:03.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:02 smithi063 ceph-mon[32413]: pgmap v313: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 5 op/s; 111 B/s, 0 keys/s, 1 objects/s recovering 2024-04-25T16:00:03.321 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822707 got 154618822711 for osd.5 2024-04-25T16:00:03.322 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:03.958 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:03 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:00:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.068+0000 7fdaa1961700 1 -- 172.21.15.31:0/3808479856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c107cb0 msgr2=0x7fda9c1080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.068+0000 7fdaa1961700 1 --2- 172.21.15.31:0/3808479856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c107cb0 0x7fda9c1080c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fda840099c0 tx=0x7fda84009cd0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.068+0000 7fdaa1961700 1 -- 172.21.15.31:0/3808479856 shutdown_connections 2024-04-25T16:00:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.068+0000 7fdaa1961700 1 --2- 172.21.15.31:0/3808479856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda9c108600 0x7fda9c10ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.068+0000 7fdaa1961700 1 --2- 172.21.15.31:0/3808479856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c103380 0x7fda9c103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.068+0000 7fdaa1961700 1 --2- 172.21.15.31:0/3808479856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c107cb0 0x7fda9c1080c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.068+0000 7fdaa1961700 1 -- 172.21.15.31:0/3808479856 >> 172.21.15.31:0/3808479856 conn(0x7fda9c0fecb0 msgr2=0x7fda9c1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.069+0000 7fdaa1961700 1 -- 172.21.15.31:0/3808479856 shutdown_connections 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.069+0000 7fdaa1961700 1 -- 172.21.15.31:0/3808479856 wait complete. 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fdaa1961700 1 Processor -- start 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fdaa1961700 1 -- start start 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fdaa1961700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c103380 0x7fda9c19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fdaa1961700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107cb0 0x7fda9c19b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fdaa1961700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda9c108600 0x7fda9c19f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fdaa1961700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113280 con 0x7fda9c103380 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fdaa1961700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113100 con 0x7fda9c108600 2024-04-25T16:00:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fdaa1961700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda9c113400 con 0x7fda9c107cb0 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fda9affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c103380 0x7fda9c19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fda9affd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c103380 0x7fda9c19ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51888/0 (socket says 172.21.15.31:51888) 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.070+0000 7fda9affd700 1 -- 172.21.15.31:0/770684966 learned_addr learned my addr 172.21.15.31:0/770684966 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fda9a7fc700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107cb0 0x7fda9c19b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fda9b7fe700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda9c108600 0x7fda9c19f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fda9affd700 1 -- 172.21.15.31:0/770684966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107cb0 msgr2=0x7fda9c19b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fda9affd700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107cb0 0x7fda9c19b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fda9affd700 1 -- 172.21.15.31:0/770684966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda9c108600 msgr2=0x7fda9c19f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fda9affd700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda9c108600 0x7fda9c19f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fda9affd700 1 -- 172.21.15.31:0/770684966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda9c19fbf0 con 0x7fda9c103380 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fda9a7fc700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107cb0 0x7fda9c19b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fda9affd700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c103380 0x7fda9c19ad20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fda9400d950 tx=0x7fda9400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fdaa095f700 1 -- 172.21.15.31:0/770684966 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94016070 con 0x7fda9c103380 2024-04-25T16:00:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fdaa095f700 1 -- 172.21.15.31:0/770684966 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda94009df0 con 0x7fda9c103380 2024-04-25T16:00:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda9c19fee0 con 0x7fda9c103380 2024-04-25T16:00:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fdaa095f700 1 -- 172.21.15.31:0/770684966 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda9400f3c0 con 0x7fda9c103380 2024-04-25T16:00:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.071+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda9c1a7800 con 0x7fda9c103380 2024-04-25T16:00:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.072+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda9c051190 con 0x7fda9c103380 2024-04-25T16:00:04.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.074+0000 7fdaa095f700 1 -- 172.21.15.31:0/770684966 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fda94019070 con 0x7fda9c103380 2024-04-25T16:00:04.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.074+0000 7fdaa095f700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda880afc10 0x7fda880b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:04.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.075+0000 7fda9a7fc700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda880afc10 0x7fda880b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:04.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.075+0000 7fdaa095f700 1 -- 172.21.15.31:0/770684966 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fda940de9d0 con 0x7fda9c103380 2024-04-25T16:00:04.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.075+0000 7fda9a7fc700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda880afc10 0x7fda880b2120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fda84000c00 tx=0x7fda8401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:04.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.078+0000 7fdaa095f700 1 -- 172.21.15.31:0/770684966 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fda940a7460 con 0x7fda9c103380 2024-04-25T16:00:04.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:03 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:00:04.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:03 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:00:04.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.287+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fda9c02e510 con 0x7fda9c103380 2024-04-25T16:00:04.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.288+0000 7fdaa095f700 1 -- 172.21.15.31:0/770684966 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fda940a6d60 con 0x7fda9c103380 2024-04-25T16:00:04.290 INFO:teuthology.orchestra.run.smithi031.stdout:300647710744 2024-04-25T16:00:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda880afc10 msgr2=0x7fda880b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda880afc10 0x7fda880b2120 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fda84000c00 tx=0x7fda8401a040 comp rx=0 tx=0).stop 2024-04-25T16:00:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c103380 msgr2=0x7fda9c19ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c103380 0x7fda9c19ad20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fda9400d950 tx=0x7fda9400dc60 comp rx=0 tx=0).stop 2024-04-25T16:00:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 shutdown_connections 2024-04-25T16:00:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda880afc10 0x7fda880b2120 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda9c108600 0x7fda9c19f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda9c103380 0x7fda9c19ad20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 --2- 172.21.15.31:0/770684966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda9c107cb0 0x7fda9c19b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:04.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 >> 172.21.15.31:0/770684966 conn(0x7fda9c0fecb0 msgr2=0x7fda9c075c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:04.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 shutdown_connections 2024-04-25T16:00:04.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:04.290+0000 7fdaa1961700 1 -- 172.21.15.31:0/770684966 wait complete. 2024-04-25T16:00:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:04 smithi063 ceph-mon[32413]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 7 op/s; 99 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:00:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:04 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/770684966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T16:00:05.050 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:04 smithi031 ceph-mon[29140]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 7 op/s; 99 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:00:05.051 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:04 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/770684966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T16:00:05.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:04 smithi186 ceph-mon[33258]: pgmap v314: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 7 op/s; 99 B/s, 0 keys/s, 0 objects/s recovering 2024-04-25T16:00:05.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:04 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/770684966' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-25T16:00:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:06.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:07.136 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:07 smithi031 ceph-mon[29140]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-25T16:00:07.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:07 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3114908785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T16:00:07.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:07 smithi063 ceph-mon[32413]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-25T16:00:07.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:07 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3114908785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T16:00:07.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:07 smithi186 ceph-mon[33258]: pgmap v315: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-25T16:00:07.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:07 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3114908785' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-25T16:00:07.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.734+0000 7f3e578f1700 1 -- 172.21.15.31:0/2533572201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 msgr2=0x7f3e5010e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:07.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.734+0000 7f3e578f1700 1 --2- 172.21.15.31:0/2533572201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 0x7f3e5010e1a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f3e40009990 tx=0x7f3e40009ca0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.734+0000 7f3e578f1700 1 -- 172.21.15.31:0/2533572201 shutdown_connections 2024-04-25T16:00:07.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.734+0000 7f3e578f1700 1 --2- 172.21.15.31:0/2533572201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e5010e770 0x7f3e50110c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.734+0000 7f3e578f1700 1 --2- 172.21.15.31:0/2533572201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e50105300 0x7f3e50105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.734+0000 7f3e578f1700 1 --2- 172.21.15.31:0/2533572201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 0x7f3e5010e1a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.734+0000 7f3e578f1700 1 -- 172.21.15.31:0/2533572201 >> 172.21.15.31:0/2533572201 conn(0x7f3e500fec90 msgr2=0x7f3e501010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:07.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.734+0000 7f3e578f1700 1 -- 172.21.15.31:0/2533572201 shutdown_connections 2024-04-25T16:00:07.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.734+0000 7f3e578f1700 1 -- 172.21.15.31:0/2533572201 wait complete. 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e578f1700 1 Processor -- start 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e578f1700 1 -- start start 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e578f1700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e50105300 0x7f3e5019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e578f1700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 0x7f3e5019d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e578f1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5010e770 0x7f3e501a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e578f1700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3e501142d0 con 0x7f3e50105c50 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e578f1700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50114150 con 0x7f3e50105300 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e578f1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3e50114450 con 0x7f3e5010e770 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e54e8c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 0x7f3e5019d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e54e8c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 0x7f3e5019d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51924/0 (socket says 172.21.15.31:51924) 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e54e8c700 1 -- 172.21.15.31:0/1567177390 learned_addr learned my addr 172.21.15.31:0/1567177390 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e5568d700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e50105300 0x7f3e5019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:07.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e55e8e700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5010e770 0x7f3e501a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e54e8c700 1 -- 172.21.15.31:0/1567177390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5010e770 msgr2=0x7f3e501a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e54e8c700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5010e770 0x7f3e501a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e54e8c700 1 -- 172.21.15.31:0/1567177390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e50105300 msgr2=0x7f3e5019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e54e8c700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e50105300 0x7f3e5019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.735+0000 7f3e54e8c700 1 -- 172.21.15.31:0/1567177390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e501a1d70 con 0x7f3e50105c50 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.736+0000 7f3e54e8c700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 0x7f3e5019d440 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e400057c0 tx=0x7f3e400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.736+0000 7f3e467fc700 1 -- 172.21.15.31:0/1567177390 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40027070 con 0x7f3e50105c50 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.736+0000 7f3e467fc700 1 -- 172.21.15.31:0/1567177390 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e40003f20 con 0x7f3e50105c50 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.736+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e501a2000 con 0x7f3e50105c50 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.736+0000 7f3e467fc700 1 -- 172.21.15.31:0/1567177390 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40022b30 con 0x7f3e50105c50 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.736+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e501a9980 con 0x7f3e50105c50 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.737+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e50051190 con 0x7f3e50105c50 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.738+0000 7f3e467fc700 1 -- 172.21.15.31:0/1567177390 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3e4002a070 con 0x7f3e50105c50 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.739+0000 7f3e467fc700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3e3c0afc90 0x7f3e3c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.739+0000 7f3e5568d700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3e3c0afc90 0x7f3e3c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:07.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.739+0000 7f3e467fc700 1 -- 172.21.15.31:0/1567177390 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3e400e01c0 con 0x7f3e50105c50 2024-04-25T16:00:07.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.740+0000 7f3e5568d700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3e3c0afc90 0x7f3e3c0b21a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c007be0 tx=0x7f3e4c0073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:07.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.743+0000 7f3e467fc700 1 -- 172.21.15.31:0/1567177390 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e40031080 con 0x7f3e50105c50 2024-04-25T16:00:07.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.966+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f3e5019e200 con 0x7f3e50105c50 2024-04-25T16:00:07.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.967+0000 7f3e467fc700 1 -- 172.21.15.31:0/1567177390 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3e5019e200 con 0x7f3e50105c50 2024-04-25T16:00:07.968 INFO:teuthology.orchestra.run.smithi031.stdout:253403070500 2024-04-25T16:00:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3e3c0afc90 msgr2=0x7f3e3c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:07.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3e3c0afc90 0x7f3e3c0b21a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c007be0 tx=0x7f3e4c0073d0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 msgr2=0x7f3e5019d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 0x7f3e5019d440 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3e400057c0 tx=0x7f3e400057f0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 shutdown_connections 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3e3c0afc90 0x7f3e3c0b21a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e50105300 0x7f3e5019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e5010e770 0x7f3e501a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 --2- 172.21.15.31:0/1567177390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3e50105c50 0x7f3e5019d440 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 >> 172.21.15.31:0/1567177390 conn(0x7f3e500fec90 msgr2=0x7f3e50102f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 shutdown_connections 2024-04-25T16:00:07.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:07.969+0000 7f3e578f1700 1 -- 172.21.15.31:0/1567177390 wait complete. 2024-04-25T16:00:08.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:08 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1567177390' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T16:00:08.392 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:08 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1567177390' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T16:00:08.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:08 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1567177390' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-25T16:00:08.421 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710740 got 300647710744 for osd.11 2024-04-25T16:00:08.421 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:09.102 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070494 got 253403070500 for osd.9 2024-04-25T16:00:09.103 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:09.324 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:09 smithi031 ceph-mon[29140]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:09.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:09 smithi063 ceph-mon[32413]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:09.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:09 smithi186 ceph-mon[33258]: pgmap v316: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.634+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3589611721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 msgr2=0x7fa6a410ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.634+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3589611721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 0x7fa6a410ff40 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009990 tx=0x7fa68c009ca0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.634+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3589611721 shutdown_connections 2024-04-25T16:00:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.634+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3589611721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a41080d0 0x7fa6a41084e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.634+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3589611721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 0x7fa6a410ff40 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.634+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3589611721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6a41037c0 0x7fa6a4103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.634+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3589611721 >> 172.21.15.31:0/3589611721 conn(0x7fa6a40fe5c0 msgr2=0x7fa6a41009e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.634+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3589611721 shutdown_connections 2024-04-25T16:00:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.634+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3589611721 wait complete. 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.635+0000 7fa6ab0f4700 1 Processor -- start 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.635+0000 7fa6ab0f4700 1 -- start start 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6ab0f4700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6a41037c0 0x7fa6a419f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6ab0f4700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 0x7fa6a419fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6ab0f4700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a41080d0 0x7fa6a41999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6ab0f4700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a41132d0 con 0x7fa6a4104110 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6ab0f4700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a4113150 con 0x7fa6a41080d0 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6ab0f4700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa6a4113450 con 0x7fa6a41037c0 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 0x7fa6a419fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:09.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 0x7fa6a419fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51940/0 (socket says 172.21.15.31:51940) 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a3fff700 1 -- 172.21.15.31:0/3199883948 learned_addr learned my addr 172.21.15.31:0/3199883948 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a9691700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a41080d0 0x7fa6a41999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a8e90700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6a41037c0 0x7fa6a419f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a3fff700 1 -- 172.21.15.31:0/3199883948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6a41037c0 msgr2=0x7fa6a419f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a3fff700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6a41037c0 0x7fa6a419f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a3fff700 1 -- 172.21.15.31:0/3199883948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a41080d0 msgr2=0x7fa6a41999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a3fff700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a41080d0 0x7fa6a41999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a3fff700 1 -- 172.21.15.31:0/3199883948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6a419a060 con 0x7fa6a4104110 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a3fff700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 0x7fa6a419fcd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa69400df40 tx=0x7fa69400df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:09.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a1ffb700 1 -- 172.21.15.31:0/3199883948 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69400e720 con 0x7fa6a4104110 2024-04-25T16:00:09.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6a1ffb700 1 -- 172.21.15.31:0/3199883948 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa69400ed70 con 0x7fa6a4104110 2024-04-25T16:00:09.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.636+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6a419a2f0 con 0x7fa6a4104110 2024-04-25T16:00:09.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.637+0000 7fa6a1ffb700 1 -- 172.21.15.31:0/3199883948 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6940163f0 con 0x7fa6a4104110 2024-04-25T16:00:09.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.637+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6a41026b0 con 0x7fa6a4104110 2024-04-25T16:00:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.638+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6a4051190 con 0x7fa6a4104110 2024-04-25T16:00:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.639+0000 7fa6a1ffb700 1 -- 172.21.15.31:0/3199883948 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa694009610 con 0x7fa6a4104110 2024-04-25T16:00:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.640+0000 7fa6a1ffb700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa6900afc10 0x7fa6900b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.640+0000 7fa6a8e90700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa6900afc10 0x7fa6900b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.640+0000 7fa6a1ffb700 1 -- 172.21.15.31:0/3199883948 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa6940e01a0 con 0x7fa6a4104110 2024-04-25T16:00:09.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.640+0000 7fa6a8e90700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa6900afc10 0x7fa6900b2120 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa6a419aaf0 tx=0x7fa69c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:09.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.645+0000 7fa6a1ffb700 1 -- 172.21.15.31:0/3199883948 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6940a8ce0 con 0x7fa6a4104110 2024-04-25T16:00:09.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.853+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fa6a41030e0 con 0x7fa6a4104110 2024-04-25T16:00:09.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.853+0000 7fa6a1ffb700 1 -- 172.21.15.31:0/3199883948 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa6940a85e0 con 0x7fa6a4104110 2024-04-25T16:00:09.855 INFO:teuthology.orchestra.run.smithi031.stdout:133143986239 2024-04-25T16:00:09.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa6900afc10 msgr2=0x7fa6900b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:09.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa6900afc10 0x7fa6900b2120 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa6a419aaf0 tx=0x7fa69c00b410 comp rx=0 tx=0).stop 2024-04-25T16:00:09.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 msgr2=0x7fa6a419fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:09.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 0x7fa6a419fcd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa69400df40 tx=0x7fa69400df70 comp rx=0 tx=0).stop 2024-04-25T16:00:09.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 shutdown_connections 2024-04-25T16:00:09.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa6900afc10 0x7fa6900b2120 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa6a41080d0 0x7fa6a41999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6a41037c0 0x7fa6a419f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 --2- 172.21.15.31:0/3199883948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa6a4104110 0x7fa6a419fcd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:09.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.855+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 >> 172.21.15.31:0/3199883948 conn(0x7fa6a40fe5c0 msgr2=0x7fa6a4107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:09.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.856+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 shutdown_connections 2024-04-25T16:00:09.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:09.856+0000 7fa6ab0f4700 1 -- 172.21.15.31:0/3199883948 wait complete. 2024-04-25T16:00:10.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:10 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3199883948' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T16:00:10.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:10 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3199883948' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T16:00:10.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:10 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3199883948' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-25T16:00:11.321 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266725 got 227633266729 for osd.8 2024-04-25T16:00:11.322 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:11.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:11 smithi063 ceph-mon[32413]: pgmap v317: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:11.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:11 smithi186 ceph-mon[33258]: pgmap v317: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:11.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:11 smithi031 ceph-mon[29140]: pgmap v317: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:11.685 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986235 got 133143986239 for osd.4 2024-04-25T16:00:11.685 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.658+0000 7f69f18f5700 1 -- 172.21.15.31:0/562414808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 msgr2=0x7f69ec103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.658+0000 7f69f18f5700 1 --2- 172.21.15.31:0/562414808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 0x7f69ec103770 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f69e40099c0 tx=0x7f69e4009cd0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.658+0000 7f69f18f5700 1 -- 172.21.15.31:0/562414808 shutdown_connections 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.658+0000 7f69f18f5700 1 --2- 172.21.15.31:0/562414808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69ec1085e0 0x7f69ec10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.658+0000 7f69f18f5700 1 --2- 172.21.15.31:0/562414808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 0x7f69ec103770 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.658+0000 7f69f18f5700 1 --2- 172.21.15.31:0/562414808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69ec107c90 0x7f69ec1080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.658+0000 7f69f18f5700 1 -- 172.21.15.31:0/562414808 >> 172.21.15.31:0/562414808 conn(0x7f69ec0fecb0 msgr2=0x7f69ec1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.658+0000 7f69f18f5700 1 -- 172.21.15.31:0/562414808 shutdown_connections 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.658+0000 7f69f18f5700 1 -- 172.21.15.31:0/562414808 wait complete. 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69f18f5700 1 Processor -- start 2024-04-25T16:00:11.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69f18f5700 1 -- start start 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69f18f5700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 0x7f69ec19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69f18f5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69ec107c90 0x7f69ec19c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69f18f5700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69ec1085e0 0x7f69ec19c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69f18f5700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec113210 con 0x7f69ec103360 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69f18f5700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec113090 con 0x7f69ec1085e0 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69f18f5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f69ec113390 con 0x7f69ec107c90 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69eaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 0x7f69ec19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69eaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 0x7f69ec19bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51906/0 (socket says 172.21.15.31:51906) 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69eaffd700 1 -- 172.21.15.31:0/3114908785 learned_addr learned my addr 172.21.15.31:0/3114908785 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69eb7fe700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69ec1085e0 0x7f69ec19c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.659+0000 7f69ea7fc700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69ec107c90 0x7f69ec19c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69eaffd700 1 -- 172.21.15.31:0/3114908785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69ec107c90 msgr2=0x7f69ec19c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69eaffd700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69ec107c90 0x7f69ec19c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69eaffd700 1 -- 172.21.15.31:0/3114908785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69ec1085e0 msgr2=0x7f69ec19c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69eaffd700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69ec1085e0 0x7f69ec19c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69eaffd700 1 -- 172.21.15.31:0/3114908785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69ec1a6db0 con 0x7f69ec103360 2024-04-25T16:00:11.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69ea7fc700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69ec107c90 0x7f69ec19c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:11.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69eaffd700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 0x7f69ec19bad0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f69e400b580 tx=0x7f69e40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:11.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69f08f3700 1 -- 172.21.15.31:0/3114908785 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4027070 con 0x7f69ec103360 2024-04-25T16:00:11.784 INFO:teuthology.orchestra.run.smithi031.stdout:107374182466 2024-04-25T16:00:11.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69f08f3700 1 -- 172.21.15.31:0/3114908785 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69e4004020 con 0x7f69ec103360 2024-04-25T16:00:11.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69f08f3700 1 -- 172.21.15.31:0/3114908785 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69e4022e30 con 0x7f69ec103360 2024-04-25T16:00:11.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69ec1a7040 con 0x7f69ec103360 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.660+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69ec1a7490 con 0x7f69ec103360 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.661+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69ec051190 con 0x7f69ec103360 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.662+0000 7f69f08f3700 1 -- 172.21.15.31:0/3114908785 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f69e4022590 con 0x7f69ec103360 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.663+0000 7f69f08f3700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69d80afc10 0x7f69d80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.663+0000 7f69ea7fc700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69d80afc10 0x7f69d80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.663+0000 7f69f08f3700 1 -- 172.21.15.31:0/3114908785 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f69e40dfb40 con 0x7f69ec103360 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.664+0000 7f69ea7fc700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69d80afc10 0x7f69d80b2120 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69d4005fd0 tx=0x7f69d4005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.668+0000 7f69f08f3700 1 -- 172.21.15.31:0/3114908785 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69e4013070 con 0x7f69ec103360 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.882+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f69ec02e6d0 con 0x7f69ec103360 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.882+0000 7f69f08f3700 1 -- 172.21.15.31:0/3114908785 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f69e40a8600 con 0x7f69ec103360 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69d80afc10 msgr2=0x7f69d80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69d80afc10 0x7f69d80b2120 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69d4005fd0 tx=0x7f69d4005ee0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 msgr2=0x7f69ec19bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 0x7f69ec19bad0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f69e400b580 tx=0x7f69e40047b0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 shutdown_connections 2024-04-25T16:00:11.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69d80afc10 0x7f69d80b2120 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f69ec1085e0 0x7f69ec19c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f69ec103360 0x7f69ec19bad0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 --2- 172.21.15.31:0/3114908785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69ec107c90 0x7f69ec19c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:11.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.884+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 >> 172.21.15.31:0/3114908785 conn(0x7f69ec0fecb0 msgr2=0x7f69ec107010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:11.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.885+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 shutdown_connections 2024-04-25T16:00:11.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:06.885+0000 7f69f18f5700 1 -- 172.21.15.31:0/3114908785 wait complete. 2024-04-25T16:00:12.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:12 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1935758052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T16:00:12.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:12 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1935758052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T16:00:12.411 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:12 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1935758052' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-25T16:00:12.446 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182463 got 107374182466 for osd.3 2024-04-25T16:00:12.446 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.900+0000 7f0b01d3d700 1 -- 172.21.15.31:0/962950958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 msgr2=0x7f0afc1037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.900+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/962950958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 0x7f0afc1037d0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0aec0099c0 tx=0x7f0aec009cd0 comp rx=0 tx=0).stop 2024-04-25T16:00:12.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.900+0000 7f0b01d3d700 1 -- 172.21.15.31:0/962950958 shutdown_connections 2024-04-25T16:00:12.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.900+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/962950958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc108640 0x7f0afc10ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:12.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.900+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/962950958 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 0x7f0afc1037d0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:12.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.900+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/962950958 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0afc107cf0 0x7f0afc108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:12.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.900+0000 7f0b01d3d700 1 -- 172.21.15.31:0/962950958 >> 172.21.15.31:0/962950958 conn(0x7f0afc0fedb0 msgr2=0x7f0afc1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:12.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.900+0000 7f0b01d3d700 1 -- 172.21.15.31:0/962950958 shutdown_connections 2024-04-25T16:00:12.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.900+0000 7f0b01d3d700 1 -- 172.21.15.31:0/962950958 wait complete. 2024-04-25T16:00:12.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.901+0000 7f0b01d3d700 1 Processor -- start 2024-04-25T16:00:12.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.901+0000 7f0b01d3d700 1 -- start start 2024-04-25T16:00:12.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0b01d3d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 0x7f0afc19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:12.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0b01d3d700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0afc107cf0 0x7f0afc19fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:12.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0b01d3d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc108640 0x7f0afc1999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:12.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0b01d3d700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0afc113270 con 0x7f0afc1033c0 2024-04-25T16:00:12.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0b01d3d700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0afc1130f0 con 0x7f0afc107cf0 2024-04-25T16:00:12.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0b01d3d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0afc1133f0 con 0x7f0afc108640 2024-04-25T16:00:12.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 0x7f0afc19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afb7fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 0x7f0afc19f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51972/0 (socket says 172.21.15.31:51972) 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afb7fe700 1 -- 172.21.15.31:0/2313179217 learned_addr learned my addr 172.21.15.31:0/2313179217 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afaffd700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0afc107cf0 0x7f0afc19fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afbfff700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc108640 0x7f0afc1999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afb7fe700 1 -- 172.21.15.31:0/2313179217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc108640 msgr2=0x7f0afc1999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afb7fe700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc108640 0x7f0afc1999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afb7fe700 1 -- 172.21.15.31:0/2313179217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0afc107cf0 msgr2=0x7f0afc19fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afb7fe700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0afc107cf0 0x7f0afc19fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afb7fe700 1 -- 172.21.15.31:0/2313179217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0afc19a0a0 con 0x7f0afc1033c0 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0afb7fe700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 0x7f0afc19f7d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0aec006010 tx=0x7f0aec0045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0af8ff9700 1 -- 172.21.15.31:0/2313179217 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec026070 con 0x7f0afc1033c0 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0af8ff9700 1 -- 172.21.15.31:0/2313179217 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0aec00f460 con 0x7f0afc1033c0 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0afc19a330 con 0x7f0afc1033c0 2024-04-25T16:00:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.902+0000 7f0af8ff9700 1 -- 172.21.15.31:0/2313179217 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0aec009e60 con 0x7f0afc1033c0 2024-04-25T16:00:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.903+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0afc0758c0 con 0x7f0afc1033c0 2024-04-25T16:00:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.903+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0afc104870 con 0x7f0afc1033c0 2024-04-25T16:00:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.905+0000 7f0af8ff9700 1 -- 172.21.15.31:0/2313179217 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f0aec029070 con 0x7f0afc1033c0 2024-04-25T16:00:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.906+0000 7f0af8ff9700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0ae40afc90 0x7f0ae40b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.906+0000 7f0afaffd700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0ae40afc90 0x7f0ae40b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.906+0000 7f0af8ff9700 1 -- 172.21.15.31:0/2313179217 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0aec0df8e0 con 0x7f0afc1033c0 2024-04-25T16:00:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.906+0000 7f0afaffd700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0ae40afc90 0x7f0ae40b21a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0afc102700 tx=0x7f0af000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:12.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:12.910+0000 7f0af8ff9700 1 -- 172.21.15.31:0/2313179217 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0aec013070 con 0x7f0afc1033c0 2024-04-25T16:00:13.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.122+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f0afc051190 con 0x7f0afc1033c0 2024-04-25T16:00:13.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.122+0000 7f0af8ff9700 1 -- 172.21.15.31:0/2313179217 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0aec0a8400 con 0x7f0afc1033c0 2024-04-25T16:00:13.124 INFO:teuthology.orchestra.run.smithi031.stdout:180388626485 2024-04-25T16:00:13.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0ae40afc90 msgr2=0x7f0ae40b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0ae40afc90 0x7f0ae40b21a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f0afc102700 tx=0x7f0af000b410 comp rx=0 tx=0).stop 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 msgr2=0x7f0afc19f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 0x7f0afc19f7d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f0aec006010 tx=0x7f0aec0045e0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 shutdown_connections 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0ae40afc90 0x7f0ae40b21a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0afc108640 0x7f0afc1999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0afc1033c0 0x7f0afc19f7d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 --2- 172.21.15.31:0/2313179217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0afc107cf0 0x7f0afc19fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 >> 172.21.15.31:0/2313179217 conn(0x7f0afc0fedb0 msgr2=0x7f0afc107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 shutdown_connections 2024-04-25T16:00:13.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:13.125+0000 7f0b01d3d700 1 -- 172.21.15.31:0/2313179217 wait complete. 2024-04-25T16:00:13.368 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:13 smithi031 ceph-mon[29140]: pgmap v318: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:13.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:13 smithi063 ceph-mon[32413]: pgmap v318: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:13.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:13 smithi186 ceph-mon[33258]: pgmap v318: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:13.621 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:13.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.878+0000 7ff51e709700 1 -- 172.21.15.31:0/2494047162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 msgr2=0x7ff5181048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:13.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.878+0000 7ff51e709700 1 --2- 172.21.15.31:0/2494047162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 0x7ff5181048a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff5080099c0 tx=0x7ff508009cd0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.878+0000 7ff51e709700 1 -- 172.21.15.31:0/2494047162 shutdown_connections 2024-04-25T16:00:13.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.878+0000 7ff51e709700 1 --2- 172.21.15.31:0/2494047162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff518109710 0x7ff518111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.878+0000 7ff51e709700 1 --2- 172.21.15.31:0/2494047162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 0x7ff5181048a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.878+0000 7ff51e709700 1 --2- 172.21.15.31:0/2494047162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff518108dc0 0x7ff5181091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.878+0000 7ff51e709700 1 -- 172.21.15.31:0/2494047162 >> 172.21.15.31:0/2494047162 conn(0x7ff5180ffda0 msgr2=0x7ff5181021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:13.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.878+0000 7ff51e709700 1 -- 172.21.15.31:0/2494047162 shutdown_connections 2024-04-25T16:00:13.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.878+0000 7ff51e709700 1 -- 172.21.15.31:0/2494047162 wait complete. 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.879+0000 7ff51e709700 1 Processor -- start 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.879+0000 7ff51e709700 1 -- start start 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.879+0000 7ff51e709700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 0x7ff5181a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.879+0000 7ff51e709700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff518108dc0 0x7ff5181a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.879+0000 7ff51e709700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff518109710 0x7ff51819ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.879+0000 7ff51e709700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff5181143c0 con 0x7ff518104490 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.879+0000 7ff51e709700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff518114240 con 0x7ff518109710 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.879+0000 7ff51e709700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff518114540 con 0x7ff518108dc0 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.879+0000 7ff517fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 0x7ff5181a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff517fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 0x7ff5181a09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:51956/0 (socket says 172.21.15.31:51956) 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff517fff700 1 -- 172.21.15.31:0/1935758052 learned_addr learned my addr 172.21.15.31:0/1935758052 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff51cca6700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff518109710 0x7ff51819ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:13.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff5177fe700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff518108dc0 0x7ff5181a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff517fff700 1 -- 172.21.15.31:0/1935758052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff518108dc0 msgr2=0x7ff5181a0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff517fff700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff518108dc0 0x7ff5181a0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff517fff700 1 -- 172.21.15.31:0/1935758052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff518109710 msgr2=0x7ff51819ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff517fff700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff518109710 0x7ff51819ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff517fff700 1 -- 172.21.15.31:0/1935758052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51819b1e0 con 0x7ff518104490 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff5177fe700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff518108dc0 0x7ff5181a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff517fff700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 0x7ff5181a09b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff5080098e0 tx=0x7ff508003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff5157fa700 1 -- 172.21.15.31:0/1935758052 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff508027070 con 0x7ff518104490 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff51819b470 con 0x7ff518104490 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff5157fa700 1 -- 172.21.15.31:0/1935758052 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5080040a0 con 0x7ff518104490 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff5157fa700 1 -- 172.21.15.31:0/1935758052 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50800f460 con 0x7ff518104490 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.880+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51819b9a0 con 0x7ff518104490 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.881+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff518105940 con 0x7ff518104490 2024-04-25T16:00:13.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.883+0000 7ff5157fa700 1 -- 172.21.15.31:0/1935758052 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7ff50802a070 con 0x7ff518104490 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.884+0000 7ff5157fa700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5000afc10 0x7ff5000b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.884+0000 7ff5177fe700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5000afc10 0x7ff5000b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.884+0000 7ff5157fa700 1 -- 172.21.15.31:0/1935758052 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff5080e0670 con 0x7ff518104490 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stdout:34359738452 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.885+0000 7ff5177fe700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5000afc10 0x7ff5000b2120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff50c005fd0 tx=0x7ff50c005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:10.888+0000 7ff5157fa700 1 -- 172.21.15.31:0/1935758052 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff508031080 con 0x7ff518104490 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.101+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff51802e510 con 0x7ff518104490 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.102+0000 7ff5157fa700 1 -- 172.21.15.31:0/1935758052 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff51819b9a0 con 0x7ff518104490 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5000afc10 msgr2=0x7ff5000b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:14.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5000afc10 0x7ff5000b2120 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7ff50c005fd0 tx=0x7ff50c005e40 comp rx=0 tx=0).stop 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 msgr2=0x7ff5181a09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 0x7ff5181a09b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ff5080098e0 tx=0x7ff508003680 comp rx=0 tx=0).stop 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 shutdown_connections 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5000afc10 0x7ff5000b2120 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff518109710 0x7ff51819ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff518104490 0x7ff5181a09b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 --2- 172.21.15.31:0/1935758052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff518108dc0 0x7ff5181a0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 >> 172.21.15.31:0/1935758052 conn(0x7ff5180ffda0 msgr2=0x7ff518108580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 shutdown_connections 2024-04-25T16:00:14.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:11.104+0000 7ff51e709700 1 -- 172.21.15.31:0/1935758052 wait complete. 2024-04-25T16:00:14.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:14 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2313179217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T16:00:14.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:14 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2313179217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T16:00:14.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:14 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2313179217' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-25T16:00:14.672 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738444 got 34359738452 for osd.0 2024-04-25T16:00:14.672 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.879+0000 7f2b3514c700 1 -- 172.21.15.31:0/2712527933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b301085e0 msgr2=0x7f2b3010ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.879+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2712527933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b301085e0 0x7f2b3010ff40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f2b2800b0d0 tx=0x7f2b2800b3e0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.879+0000 7f2b3514c700 1 -- 172.21.15.31:0/2712527933 shutdown_connections 2024-04-25T16:00:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.879+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2712527933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b301085e0 0x7f2b3010ff40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.879+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2712527933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b30103360 0x7f2b30103770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.879+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2712527933 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b30107c90 0x7f2b301080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.879+0000 7f2b3514c700 1 -- 172.21.15.31:0/2712527933 >> 172.21.15.31:0/2712527933 conn(0x7f2b300fec90 msgr2=0x7f2b301010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:14.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.879+0000 7f2b3514c700 1 -- 172.21.15.31:0/2712527933 shutdown_connections 2024-04-25T16:00:14.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.879+0000 7f2b3514c700 1 -- 172.21.15.31:0/2712527933 wait complete. 2024-04-25T16:00:14.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.880+0000 7f2b3514c700 1 Processor -- start 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.880+0000 7f2b3514c700 1 -- start start 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b3514c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b30103360 0x7f2b3019bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b3514c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b30107c90 0x7f2b3019c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b3514c700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b301085e0 0x7f2b3019c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b3514c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b30113210 con 0x7f2b30103360 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b3514c700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2b30113090 con 0x7f2b301085e0 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b3514c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b30113390 con 0x7f2b30107c90 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b30103360 0x7f2b3019bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2f59e700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b301085e0 0x7f2b3019c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2ed9d700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b30103360 0x7f2b3019bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52486/0 (socket says 172.21.15.31:52486) 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2ed9d700 1 -- 172.21.15.31:0/2574871533 learned_addr learned my addr 172.21.15.31:0/2574871533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2659c700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b30107c90 0x7f2b3019c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:14.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2ed9d700 1 -- 172.21.15.31:0/2574871533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b30107c90 msgr2=0x7f2b3019c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2ed9d700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b30107c90 0x7f2b3019c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2ed9d700 1 -- 172.21.15.31:0/2574871533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b301085e0 msgr2=0x7f2b3019c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2ed9d700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b301085e0 0x7f2b3019c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2ed9d700 1 -- 172.21.15.31:0/2574871533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b301a6d90 con 0x7f2b30103360 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.881+0000 7f2b2ed9d700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b30103360 0x7f2b3019bad0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c9b0 tx=0x7f2b2000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.882+0000 7f2b2cd99700 1 -- 172.21.15.31:0/2574871533 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20007b00 con 0x7f2b30103360 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.882+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b301a7080 con 0x7f2b30103360 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.882+0000 7f2b2cd99700 1 -- 172.21.15.31:0/2574871533 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b2000ce50 con 0x7f2b30103360 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.882+0000 7f2b2cd99700 1 -- 172.21.15.31:0/2574871533 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20018660 con 0x7f2b30103360 2024-04-25T16:00:14.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.882+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b301a75c0 con 0x7f2b30103360 2024-04-25T16:00:14.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.883+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b30051190 con 0x7f2b30103360 2024-04-25T16:00:14.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.884+0000 7f2b2cd99700 1 -- 172.21.15.31:0/2574871533 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2b2000f450 con 0x7f2b30103360 2024-04-25T16:00:14.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.885+0000 7f2b2cd99700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b1c0afc10 0x7f2b1c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:14.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.885+0000 7f2b2659c700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b1c0afc10 0x7f2b1c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:14.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.885+0000 7f2b2cd99700 1 -- 172.21.15.31:0/2574871533 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2b200ded50 con 0x7f2b30103360 2024-04-25T16:00:14.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.885+0000 7f2b2659c700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b1c0afc10 0x7f2b1c0b2120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2b301027c0 tx=0x7f2b1800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:14.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:14.890+0000 7f2b2cd99700 1 -- 172.21.15.31:0/2574871533 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b20014030 con 0x7f2b30103360 2024-04-25T16:00:15.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.104+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f2b3002e510 con 0x7f2b30103360 2024-04-25T16:00:15.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.107+0000 7f2b2cd99700 1 -- 172.21.15.31:0/2574871533 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2b200a7780 con 0x7f2b30103360 2024-04-25T16:00:15.108 INFO:teuthology.orchestra.run.smithi031.stdout:201863462961 2024-04-25T16:00:15.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.109+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b1c0afc10 msgr2=0x7f2b1c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.109+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b1c0afc10 0x7f2b1c0b2120 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2b301027c0 tx=0x7f2b1800a300 comp rx=0 tx=0).stop 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.109+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b30103360 msgr2=0x7f2b3019bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.109+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b30103360 0x7f2b3019bad0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2b2000c9b0 tx=0x7f2b2000ccc0 comp rx=0 tx=0).stop 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.109+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 shutdown_connections 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.110+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b1c0afc10 0x7f2b1c0b2120 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.110+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b301085e0 0x7f2b3019c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.110+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b30103360 0x7f2b3019bad0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.110+0000 7f2b3514c700 1 --2- 172.21.15.31:0/2574871533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b30107c90 0x7f2b3019c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.110+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 >> 172.21.15.31:0/2574871533 conn(0x7f2b300fec90 msgr2=0x7f2b30107400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:15.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.110+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 shutdown_connections 2024-04-25T16:00:15.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:15.110+0000 7f2b3514c700 1 -- 172.21.15.31:0/2574871533 wait complete. 2024-04-25T16:00:15.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:15 smithi063 ceph-mon[32413]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:00:15.391 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:15.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:15 smithi186 ceph-mon[33258]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:00:15.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:15 smithi031 ceph-mon[29140]: pgmap v319: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:00:16.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:16 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2574871533' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T16:00:16.271 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:16.301 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626481 got 180388626485 for osd.6 2024-04-25T16:00:16.302 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:16.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:16 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2574871533' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T16:00:16.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:16 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2574871533' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-25T16:00:16.698 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462956 got 201863462961 for osd.7 2024-04-25T16:00:16.699 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:17.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:17 smithi063 ceph-mon[32413]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:00:17.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:17 smithi186 ceph-mon[33258]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:00:17.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:17 smithi031 ceph-mon[29140]: pgmap v320: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:00:17.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.929+0000 7fee312e7700 1 -- 172.21.15.31:0/1728858961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 msgr2=0x7fee2c10e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:17.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.929+0000 7fee312e7700 1 --2- 172.21.15.31:0/1728858961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 0x7fee2c10e220 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fee20009990 tx=0x7fee20009ca0 comp rx=0 tx=0).stop 2024-04-25T16:00:17.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.929+0000 7fee312e7700 1 -- 172.21.15.31:0/1728858961 shutdown_connections 2024-04-25T16:00:17.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.929+0000 7fee312e7700 1 --2- 172.21.15.31:0/1728858961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c10e7b0 0x7fee2c110c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:17.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.929+0000 7fee312e7700 1 --2- 172.21.15.31:0/1728858961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee2c105380 0x7fee2c105790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:17.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.929+0000 7fee312e7700 1 --2- 172.21.15.31:0/1728858961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 0x7fee2c10e220 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:17.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.929+0000 7fee312e7700 1 -- 172.21.15.31:0/1728858961 >> 172.21.15.31:0/1728858961 conn(0x7fee2c0fec90 msgr2=0x7fee2c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:17.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.929+0000 7fee312e7700 1 -- 172.21.15.31:0/1728858961 shutdown_connections 2024-04-25T16:00:17.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.929+0000 7fee312e7700 1 -- 172.21.15.31:0/1728858961 wait complete. 2024-04-25T16:00:17.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.930+0000 7fee312e7700 1 Processor -- start 2024-04-25T16:00:17.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.930+0000 7fee312e7700 1 -- start start 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee312e7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee2c105380 0x7fee2c1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee312e7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 0x7fee2c1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee312e7700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c10e7b0 0x7fee2c19aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee312e7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c1142e0 con 0x7fee2c105cd0 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee312e7700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c114160 con 0x7fee2c10e7b0 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee312e7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c114460 con 0x7fee2c105380 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2a7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 0x7fee2c1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2a7fc700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 0x7fee2c1a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52510/0 (socket says 172.21.15.31:52510) 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2a7fc700 1 -- 172.21.15.31:0/1503500905 learned_addr learned my addr 172.21.15.31:0/1503500905 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2b7fe700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c10e7b0 0x7fee2c19aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2affd700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee2c105380 0x7fee2c1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2a7fc700 1 -- 172.21.15.31:0/1503500905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee2c105380 msgr2=0x7fee2c1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:17.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2a7fc700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee2c105380 0x7fee2c1a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2a7fc700 1 -- 172.21.15.31:0/1503500905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c10e7b0 msgr2=0x7fee2c19aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2a7fc700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c10e7b0 0x7fee2c19aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2a7fc700 1 -- 172.21.15.31:0/1503500905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c19b170 con 0x7fee2c105cd0 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.931+0000 7fee2a7fc700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 0x7fee2c1a0de0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fee20000c00 tx=0x7fee200047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.932+0000 7fee13fff700 1 -- 172.21.15.31:0/1503500905 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20027070 con 0x7fee2c105cd0 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.932+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2c19b400 con 0x7fee2c105cd0 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.932+0000 7fee13fff700 1 -- 172.21.15.31:0/1503500905 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee20003f20 con 0x7fee2c105cd0 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.932+0000 7fee13fff700 1 -- 172.21.15.31:0/1503500905 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee20022cf0 con 0x7fee2c105cd0 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.932+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2c1a7730 con 0x7fee2c105cd0 2024-04-25T16:00:17.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.933+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2c051190 con 0x7fee2c105cd0 2024-04-25T16:00:17.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.934+0000 7fee13fff700 1 -- 172.21.15.31:0/1503500905 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fee2002a070 con 0x7fee2c105cd0 2024-04-25T16:00:17.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.935+0000 7fee13fff700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fee140afc10 0x7fee140b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:17.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.935+0000 7fee2affd700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fee140afc10 0x7fee140b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:17.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.935+0000 7fee13fff700 1 -- 172.21.15.31:0/1503500905 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fee200dffd0 con 0x7fee2c105cd0 2024-04-25T16:00:17.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.936+0000 7fee2affd700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fee140afc10 0x7fee140b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fee1c005950 tx=0x7fee1c005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:17.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:17.939+0000 7fee13fff700 1 -- 172.21.15.31:0/1503500905 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fee20013070 con 0x7fee2c105cd0 2024-04-25T16:00:18.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.150+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fee2c02e510 con 0x7fee2c105cd0 2024-04-25T16:00:18.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.151+0000 7fee13fff700 1 -- 172.21.15.31:0/1503500905 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fee200a8af0 con 0x7fee2c105cd0 2024-04-25T16:00:18.152 INFO:teuthology.orchestra.run.smithi031.stdout:274877906978 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fee140afc10 msgr2=0x7fee140b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fee140afc10 0x7fee140b2120 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fee1c005950 tx=0x7fee1c005ed0 comp rx=0 tx=0).stop 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 msgr2=0x7fee2c1a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 0x7fee2c1a0de0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fee20000c00 tx=0x7fee200047b0 comp rx=0 tx=0).stop 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 shutdown_connections 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fee140afc10 0x7fee140b2120 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c10e7b0 0x7fee2c19aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee2c105380 0x7fee2c1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 --2- 172.21.15.31:0/1503500905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fee2c105cd0 0x7fee2c1a0de0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:18.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 >> 172.21.15.31:0/1503500905 conn(0x7fee2c0fec90 msgr2=0x7fee2c0ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:18.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 shutdown_connections 2024-04-25T16:00:18.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:18.153+0000 7fee312e7700 1 -- 172.21.15.31:0/1503500905 wait complete. 2024-04-25T16:00:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:19 smithi063 ceph-mon[32413]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:19 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1503500905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T16:00:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:19 smithi186 ceph-mon[33258]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:19 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1503500905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T16:00:19.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:19 smithi031 ceph-mon[29140]: pgmap v321: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:19.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:19 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1503500905' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-25T16:00:19.905 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906972 got 274877906978 for osd.10 2024-04-25T16:00:19.906 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:21.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:21 smithi031 ceph-mon[29140]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:21.396 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:21 smithi031 ceph-mon[29140]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T16:00:21.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:21 smithi186 ceph-mon[33258]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:21.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:21 smithi186 ceph-mon[33258]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T16:00:21.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:21 smithi063 ceph-mon[32413]: pgmap v322: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:21.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:21 smithi063 ceph-mon[32413]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-04-25T16:00:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.979+0000 7f387f8be700 1 -- 172.21.15.31:0/142030152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878108dc0 msgr2=0x7f38781091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.979+0000 7f387f8be700 1 --2- 172.21.15.31:0/142030152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878108dc0 0x7f38781091d0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f38740099c0 tx=0x7f3874009cd0 comp rx=0 tx=0).stop 2024-04-25T16:00:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.979+0000 7f387f8be700 1 -- 172.21.15.31:0/142030152 shutdown_connections 2024-04-25T16:00:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.979+0000 7f387f8be700 1 --2- 172.21.15.31:0/142030152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3878104490 0x7f38781048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.979+0000 7f387f8be700 1 --2- 172.21.15.31:0/142030152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3878109710 0x7f3878111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.979+0000 7f387f8be700 1 --2- 172.21.15.31:0/142030152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878108dc0 0x7f38781091d0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.979+0000 7f387f8be700 1 -- 172.21.15.31:0/142030152 >> 172.21.15.31:0/142030152 conn(0x7f38780ffda0 msgr2=0x7f38781021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.980+0000 7f387f8be700 1 -- 172.21.15.31:0/142030152 shutdown_connections 2024-04-25T16:00:21.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.980+0000 7f387f8be700 1 -- 172.21.15.31:0/142030152 wait complete. 2024-04-25T16:00:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387f8be700 1 Processor -- start 2024-04-25T16:00:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387f8be700 1 -- start start 2024-04-25T16:00:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387f8be700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878104490 0x7f38781a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387f8be700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3878108dc0 0x7f38781a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387f8be700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3878109710 0x7f387819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:21.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387f8be700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3878114370 con 0x7f3878104490 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387f8be700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f38781141f0 con 0x7f3878108dc0 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387f8be700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f38781144f0 con 0x7f3878109710 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387d65a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878104490 0x7f38781a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387d65a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878104490 0x7f38781a08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52526/0 (socket says 172.21.15.31:52526) 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387d65a700 1 -- 172.21.15.31:0/493357767 learned_addr learned my addr 172.21.15.31:0/493357767 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387de5b700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3878109710 0x7f387819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387ce59700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3878108dc0 0x7f38781a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387d65a700 1 -- 172.21.15.31:0/493357767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3878109710 msgr2=0x7f387819aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387d65a700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3878109710 0x7f387819aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387d65a700 1 -- 172.21.15.31:0/493357767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3878108dc0 msgr2=0x7f38781a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387d65a700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3878108dc0 0x7f38781a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387d65a700 1 -- 172.21.15.31:0/493357767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387819b160 con 0x7f3878104490 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.981+0000 7f387ce59700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3878108dc0 0x7f38781a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:21.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.982+0000 7f387d65a700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878104490 0x7f38781a08b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f386800c9b0 tx=0x7f386800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:21.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.982+0000 7f386e7fc700 1 -- 172.21.15.31:0/493357767 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868007b00 con 0x7f3878104490 2024-04-25T16:00:21.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.982+0000 7f386e7fc700 1 -- 172.21.15.31:0/493357767 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3868007ca0 con 0x7f3878104490 2024-04-25T16:00:21.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.982+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387819b450 con 0x7f3878104490 2024-04-25T16:00:21.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.982+0000 7f386e7fc700 1 -- 172.21.15.31:0/493357767 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3868018730 con 0x7f3878104490 2024-04-25T16:00:21.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.982+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38781a7760 con 0x7f3878104490 2024-04-25T16:00:21.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.983+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3878051190 con 0x7f3878104490 2024-04-25T16:00:21.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.984+0000 7f386e7fc700 1 -- 172.21.15.31:0/493357767 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f386800f450 con 0x7f3878104490 2024-04-25T16:00:21.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.985+0000 7f386e7fc700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38640afc00 0x7f38640b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:21.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.985+0000 7f387ce59700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38640afc00 0x7f38640b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:21.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.985+0000 7f386e7fc700 1 -- 172.21.15.31:0/493357767 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f38680def10 con 0x7f3878104490 2024-04-25T16:00:21.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.985+0000 7f387ce59700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38640afc00 0x7f38640b2110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3874003800 tx=0x7f3874000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:21.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:21.988+0000 7f386e7fc700 1 -- 172.21.15.31:0/493357767 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3868014030 con 0x7f3878104490 2024-04-25T16:00:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.202+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f387819bfe0 con 0x7f3878104490 2024-04-25T16:00:22.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.203+0000 7f386e7fc700 1 -- 172.21.15.31:0/493357767 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f38680a7c10 con 0x7f3878104490 2024-04-25T16:00:22.204 INFO:teuthology.orchestra.run.smithi031.stdout:77309411404 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38640afc00 msgr2=0x7f38640b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38640afc00 0x7f38640b2110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3874003800 tx=0x7f3874000bc0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878104490 msgr2=0x7f38781a08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878104490 0x7f38781a08b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f386800c9b0 tx=0x7f386800cd70 comp rx=0 tx=0).stop 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 shutdown_connections 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38640afc00 0x7f38640b2110 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3878109710 0x7f387819aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3878104490 0x7f38781a08b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 --2- 172.21.15.31:0/493357767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3878108dc0 0x7f38781a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 >> 172.21.15.31:0/493357767 conn(0x7f38780ffda0 msgr2=0x7f3878108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 shutdown_connections 2024-04-25T16:00:22.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.205+0000 7f387f8be700 1 -- 172.21.15.31:0/493357767 wait complete. 2024-04-25T16:00:22.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.686+0000 7f2349bdd700 1 -- 172.21.15.31:0/1954339155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344103390 msgr2=0x7f23441037a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:22.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.686+0000 7f2349bdd700 1 --2- 172.21.15.31:0/1954339155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344103390 0x7f23441037a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f23340099c0 tx=0x7f2334009cd0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.686+0000 7f2349bdd700 1 -- 172.21.15.31:0/1954339155 shutdown_connections 2024-04-25T16:00:22.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.686+0000 7f2349bdd700 1 --2- 172.21.15.31:0/1954339155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2344108610 0x7f234410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.686+0000 7f2349bdd700 1 --2- 172.21.15.31:0/1954339155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344103390 0x7f23441037a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.686+0000 7f2349bdd700 1 --2- 172.21.15.31:0/1954339155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2344107cc0 0x7f23441080d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.686+0000 7f2349bdd700 1 -- 172.21.15.31:0/1954339155 >> 172.21.15.31:0/1954339155 conn(0x7f23440fec90 msgr2=0x7f23441010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:22.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.686+0000 7f2349bdd700 1 -- 172.21.15.31:0/1954339155 shutdown_connections 2024-04-25T16:00:22.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.686+0000 7f2349bdd700 1 -- 172.21.15.31:0/1954339155 wait complete. 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.687+0000 7f2349bdd700 1 Processor -- start 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.687+0000 7f2349bdd700 1 -- start start 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.687+0000 7f2349bdd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2344103390 0x7f234419bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2349bdd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2344107cc0 0x7f234419c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2349bdd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344108610 0x7f234419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2349bdd700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2344113290 con 0x7f2344108610 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2349bdd700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2344113110 con 0x7f2344107cc0 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2349bdd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2344113410 con 0x7f2344103390 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2343fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344108610 0x7f234419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2343fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344108610 0x7f234419c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52554/0 (socket says 172.21.15.31:52554) 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2342ffd700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2344107cc0 0x7f234419c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:22.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2343fff700 1 -- 172.21.15.31:0/3490365338 learned_addr learned my addr 172.21.15.31:0/3490365338 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f23437fe700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2344103390 0x7f234419bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2343fff700 1 -- 172.21.15.31:0/3490365338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2344103390 msgr2=0x7f234419bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2343fff700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2344103390 0x7f234419bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2343fff700 1 -- 172.21.15.31:0/3490365338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2344107cc0 msgr2=0x7f234419c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2343fff700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2344107cc0 0x7f234419c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2343fff700 1 -- 172.21.15.31:0/3490365338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23441a6f20 con 0x7f2344108610 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.688+0000 7f2343fff700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344108610 0x7f234419c670 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f233c00ada0 tx=0x7f233c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.689+0000 7f2340ff9700 1 -- 172.21.15.31:0/3490365338 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c017020 con 0x7f2344108610 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.689+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23441a7210 con 0x7f2344108610 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.689+0000 7f2340ff9700 1 -- 172.21.15.31:0/3490365338 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f233c00cd80 con 0x7f2344108610 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.689+0000 7f2340ff9700 1 -- 172.21.15.31:0/3490365338 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c005260 con 0x7f2344108610 2024-04-25T16:00:22.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.689+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23441a7700 con 0x7f2344108610 2024-04-25T16:00:22.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.690+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2344051190 con 0x7f2344108610 2024-04-25T16:00:22.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.692+0000 7f2340ff9700 1 -- 172.21.15.31:0/3490365338 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f233c00a090 con 0x7f2344108610 2024-04-25T16:00:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.693+0000 7f2340ff9700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f232c0afb80 0x7f232c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.693+0000 7f23437fe700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f232c0afb80 0x7f232c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.693+0000 7f2340ff9700 1 -- 172.21.15.31:0/3490365338 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f233c0de8b0 con 0x7f2344108610 2024-04-25T16:00:22.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.694+0000 7f23437fe700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f232c0afb80 0x7f232c0b2090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2334009990 tx=0x7f23340058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:22.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.697+0000 7f2340ff9700 1 -- 172.21.15.31:0/3490365338 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f233c021020 con 0x7f2344108610 2024-04-25T16:00:22.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.909+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f234402e510 con 0x7f2344108610 2024-04-25T16:00:22.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.909+0000 7f2340ff9700 1 -- 172.21.15.31:0/3490365338 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f233c0a74d0 con 0x7f2344108610 2024-04-25T16:00:22.910 INFO:teuthology.orchestra.run.smithi031.stdout:55834574929 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f232c0afb80 msgr2=0x7f232c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f232c0afb80 0x7f232c0b2090 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2334009990 tx=0x7f23340058e0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344108610 msgr2=0x7f234419c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344108610 0x7f234419c670 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f233c00ada0 tx=0x7f233c00c330 comp rx=0 tx=0).stop 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 shutdown_connections 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f232c0afb80 0x7f232c0b2090 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2344108610 0x7f234419c670 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2344103390 0x7f234419bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 --2- 172.21.15.31:0/3490365338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2344107cc0 0x7f234419c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 >> 172.21.15.31:0/3490365338 conn(0x7f23440fec90 msgr2=0x7f23441010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:22.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 shutdown_connections 2024-04-25T16:00:22.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:22.912+0000 7f2349bdd700 1 -- 172.21.15.31:0/3490365338 wait complete. 2024-04-25T16:00:23.157 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411399 got 77309411404 for osd.2 2024-04-25T16:00:23.171 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:23.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:23 smithi031 ceph-mon[29140]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:23.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:23 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/493357767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T16:00:23.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:23 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3490365338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T16:00:23.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:23 smithi186 ceph-mon[33258]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:23.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:23 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/493357767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T16:00:23.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:23 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3490365338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T16:00:23.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:23 smithi063 ceph-mon[32413]: pgmap v323: 97 pgs: 97 active+clean; 639 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:23.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:23 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/493357767' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-25T16:00:23.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:23 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3490365338' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-25T16:00:23.871 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574924 got 55834574929 for osd.1 2024-04-25T16:00:23.872 DEBUG:teuthology.parallel:result is None 2024-04-25T16:00:23.872 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-25T16:00:23.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T16:00:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:25.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:25 smithi186 ceph-mon[33258]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 12 op/s 2024-04-25T16:00:25.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:25 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:25.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:25 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:25.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:25 smithi031 ceph-mon[29140]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 12 op/s 2024-04-25T16:00:25.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:25 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:25.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:25 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:25.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:25 smithi063 ceph-mon[32413]: pgmap v324: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 341 B/s wr, 12 op/s 2024-04-25T16:00:25.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:25 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:25.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:25 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.976+0000 7fe62b7d9700 1 -- 172.21.15.31:0/1986764662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624109720 msgr2=0x7fe624111080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.976+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/1986764662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624109720 0x7fe624111080 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe61c00b0d0 tx=0x7fe61c00b3e0 comp rx=0 tx=0).stop 2024-04-25T16:00:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.976+0000 7fe62b7d9700 1 -- 172.21.15.31:0/1986764662 shutdown_connections 2024-04-25T16:00:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.976+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/1986764662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6241044a0 0x7fe6241048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.976+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/1986764662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624109720 0x7fe624111080 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.976+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/1986764662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe624108dd0 0x7fe6241091e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.976+0000 7fe62b7d9700 1 -- 172.21.15.31:0/1986764662 >> 172.21.15.31:0/1986764662 conn(0x7fe6240ffd70 msgr2=0x7fe6241021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.976+0000 7fe62b7d9700 1 -- 172.21.15.31:0/1986764662 shutdown_connections 2024-04-25T16:00:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.976+0000 7fe62b7d9700 1 -- 172.21.15.31:0/1986764662 wait complete. 2024-04-25T16:00:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.977+0000 7fe62b7d9700 1 Processor -- start 2024-04-25T16:00:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.977+0000 7fe62b7d9700 1 -- start start 2024-04-25T16:00:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.977+0000 7fe62b7d9700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6241044a0 0x7fe6241a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.977+0000 7fe62b7d9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624108dd0 0x7fe6241a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.977+0000 7fe62b7d9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe624109720 0x7fe62419aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.977+0000 7fe62b7d9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe624114350 con 0x7fe624108dd0 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.977+0000 7fe62b7d9700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe6241141d0 con 0x7fe6241044a0 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.977+0000 7fe62b7d9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe6241144d0 con 0x7fe624109720 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe628d74700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624108dd0 0x7fe6241a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe628d74700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624108dd0 0x7fe6241a0d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53466/0 (socket says 172.21.15.31:53466) 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe628d74700 1 -- 172.21.15.31:0/2244132541 learned_addr learned my addr 172.21.15.31:0/2244132541 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe629575700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6241044a0 0x7fe6241a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe629d76700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe624109720 0x7fe62419aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe628d74700 1 -- 172.21.15.31:0/2244132541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe624109720 msgr2=0x7fe62419aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe628d74700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe624109720 0x7fe62419aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe628d74700 1 -- 172.21.15.31:0/2244132541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6241044a0 msgr2=0x7fe6241a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe628d74700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6241044a0 0x7fe6241a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:25.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe628d74700 1 -- 172.21.15.31:0/2244132541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe62419b0f0 con 0x7fe624108dd0 2024-04-25T16:00:25.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe629d76700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe624109720 0x7fe62419aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:25.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe628d74700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624108dd0 0x7fe6241a0d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe61400c9e0 tx=0x7fe61400ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:25.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe61a7fc700 1 -- 172.21.15.31:0/2244132541 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe614007aa0 con 0x7fe624108dd0 2024-04-25T16:00:25.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe61a7fc700 1 -- 172.21.15.31:0/2244132541 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe614007c40 con 0x7fe624108dd0 2024-04-25T16:00:25.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe62419b3e0 con 0x7fe624108dd0 2024-04-25T16:00:25.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe61a7fc700 1 -- 172.21.15.31:0/2244132541 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe614018660 con 0x7fe624108dd0 2024-04-25T16:00:25.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.978+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6241a76d0 con 0x7fe624108dd0 2024-04-25T16:00:25.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.979+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe624051190 con 0x7fe624108dd0 2024-04-25T16:00:25.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.981+0000 7fe61a7fc700 1 -- 172.21.15.31:0/2244132541 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe614018800 con 0x7fe624108dd0 2024-04-25T16:00:25.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.981+0000 7fe61a7fc700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6100afc00 0x7fe6100b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:25.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.981+0000 7fe61a7fc700 1 -- 172.21.15.31:0/2244132541 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe6140ded30 con 0x7fe624108dd0 2024-04-25T16:00:25.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.981+0000 7fe629575700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6100afc00 0x7fe6100b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:25.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.982+0000 7fe629575700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6100afc00 0x7fe6100b2110 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe62406cdf0 tx=0x7fe6200073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:25.985+0000 7fe61a7fc700 1 -- 172.21.15.31:0/2244132541 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe614014030 con 0x7fe624108dd0 2024-04-25T16:00:26.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.191+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe624063ad0 con 0x7fe6100afc00 2024-04-25T16:00:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.198+0000 7fe61a7fc700 1 -- 172.21.15.31:0/2244132541 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259221 (secure 0 0 0) 0x7fe624063ad0 con 0x7fe6100afc00 2024-04-25T16:00:26.203 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6100afc00 msgr2=0x7fe6100b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6100afc00 0x7fe6100b2110 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe62406cdf0 tx=0x7fe6200073d0 comp rx=0 tx=0).stop 2024-04-25T16:00:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624108dd0 msgr2=0x7fe6241a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624108dd0 0x7fe6241a0d80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe61400c9e0 tx=0x7fe61400ccf0 comp rx=0 tx=0).stop 2024-04-25T16:00:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 shutdown_connections 2024-04-25T16:00:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6100afc00 0x7fe6100b2110 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6241044a0 0x7fe6241a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe624109720 0x7fe62419aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:26.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 --2- 172.21.15.31:0/2244132541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe624108dd0 0x7fe6241a0d80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:26.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.202+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 >> 172.21.15.31:0/2244132541 conn(0x7fe6240ffd70 msgr2=0x7fe624108510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:26.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.203+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 shutdown_connections 2024-04-25T16:00:26.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:26.203+0000 7fe62b7d9700 1 -- 172.21.15.31:0/2244132541 wait complete. 2024-04-25T16:00:26.206 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T16:00:26.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:00:26.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:00:26.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:26.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:00:26.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:00:26.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:26.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:00:26.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:00:26.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:00:26.961 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-04-25T16:00:25.334836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654990,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":806,"num_read_kb":521,"num_write":367,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2953032,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":804,"ondisk_log_size":804,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355336,"kb_used_data":10344,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124586488,"statfs":{"total":1151940427776,"available":1151576563712,"internally_reserved":0,"allocated":10592256,"data_stored":6775626,"data_compressed":113085,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":112,"num_read_kb":62,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003648"},"pg_stats":[{"pgid":"3.3f","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354813+0000","last_change":"2024-04-25T15:59:02.997678+0000","last_active":"2024-04-25T16:00:19.354813+0000","last_peered":"2024-04-25T16:00:19.354813+0000","last_clean":"2024-04-25T16:00:19.354813+0000","last_became_active":"2024-04-25T15:59:02.997335+0000","last_became_peered":"2024-04-25T15:59:02.997335+0000","last_unstale":"2024-04-25T16:00:19.354813+0000","last_undegraded":"2024-04-25T16:00:19.354813+0000","last_fullsized":"2024-04-25T16:00:19.354813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:33.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349727+0000","last_change":"2024-04-25T15:59:03.001049+0000","last_active":"2024-04-25T16:00:19.349727+0000","last_peered":"2024-04-25T16:00:19.349727+0000","last_clean":"2024-04-25T16:00:19.349727+0000","last_became_active":"2024-04-25T15:59:03.000663+0000","last_became_peered":"2024-04-25T15:59:03.000663+0000","last_unstale":"2024-04-25T16:00:19.349727+0000","last_undegraded":"2024-04-25T16:00:19.349727+0000","last_fullsized":"2024-04-25T16:00:19.349727+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:04:36.670519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354156+0000","last_change":"2024-04-25T15:59:03.003861+0000","last_active":"2024-04-25T16:00:19.354156+0000","last_peered":"2024-04-25T16:00:19.354156+0000","last_clean":"2024-04-25T16:00:19.354156+0000","last_became_active":"2024-04-25T15:59:03.003508+0000","last_became_peered":"2024-04-25T15:59:03.003508+0000","last_unstale":"2024-04-25T16:00:19.354156+0000","last_undegraded":"2024-04-25T16:00:19.354156+0000","last_fullsized":"2024-04-25T16:00:19.354156+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:25.679389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337216+0000","last_change":"2024-04-25T15:59:03.003330+0000","last_active":"2024-04-25T16:00:19.337216+0000","last_peered":"2024-04-25T16:00:19.337216+0000","last_clean":"2024-04-25T16:00:19.337216+0000","last_became_active":"2024-04-25T15:59:03.002872+0000","last_became_peered":"2024-04-25T15:59:03.002872+0000","last_unstale":"2024-04-25T16:00:19.337216+0000","last_undegraded":"2024-04-25T16:00:19.337216+0000","last_fullsized":"2024-04-25T16:00:19.337216+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:02.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.324906+0000","last_change":"2024-04-25T15:59:02.997684+0000","last_active":"2024-04-25T16:00:19.324906+0000","last_peered":"2024-04-25T16:00:19.324906+0000","last_clean":"2024-04-25T16:00:19.324906+0000","last_became_active":"2024-04-25T15:59:02.997370+0000","last_became_peered":"2024-04-25T15:59:02.997370+0000","last_unstale":"2024-04-25T16:00:19.324906+0000","last_undegraded":"2024-04-25T16:00:19.324906+0000","last_fullsized":"2024-04-25T16:00:19.324906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:24.349085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348756+0000","last_change":"2024-04-25T15:59:03.003638+0000","last_active":"2024-04-25T16:00:19.348756+0000","last_peered":"2024-04-25T16:00:19.348756+0000","last_clean":"2024-04-25T16:00:19.348756+0000","last_became_active":"2024-04-25T15:59:03.003299+0000","last_became_peered":"2024-04-25T15:59:03.003299+0000","last_unstale":"2024-04-25T16:00:19.348756+0000","last_undegraded":"2024-04-25T16:00:19.348756+0000","last_fullsized":"2024-04-25T16:00:19.348756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:46.842345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.338851+0000","last_change":"2024-04-25T15:59:02.998591+0000","last_active":"2024-04-25T16:00:19.338851+0000","last_peered":"2024-04-25T16:00:19.338851+0000","last_clean":"2024-04-25T16:00:19.338851+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.338851+0000","last_undegraded":"2024-04-25T16:00:19.338851+0000","last_fullsized":"2024-04-25T16:00:19.338851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.444499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343142+0000","last_change":"2024-04-25T15:59:03.006225+0000","last_active":"2024-04-25T16:00:19.343142+0000","last_peered":"2024-04-25T16:00:19.343142+0000","last_clean":"2024-04-25T16:00:19.343142+0000","last_became_active":"2024-04-25T15:59:03.005883+0000","last_became_peered":"2024-04-25T15:59:03.005883+0000","last_unstale":"2024-04-25T16:00:19.343142+0000","last_undegraded":"2024-04-25T16:00:19.343142+0000","last_fullsized":"2024-04-25T16:00:19.343142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:17.773196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359702+0000","last_change":"2024-04-25T15:59:03.003605+0000","last_active":"2024-04-25T16:00:19.359702+0000","last_peered":"2024-04-25T16:00:19.359702+0000","last_clean":"2024-04-25T16:00:19.359702+0000","last_became_active":"2024-04-25T15:59:03.003271+0000","last_became_peered":"2024-04-25T15:59:03.003271+0000","last_unstale":"2024-04-25T16:00:19.359702+0000","last_undegraded":"2024-04-25T16:00:19.359702+0000","last_fullsized":"2024-04-25T16:00:19.359702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:28.576407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'16","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348846+0000","last_change":"2024-04-25T15:59:50.679099+0000","last_active":"2024-04-25T16:00:19.348846+0000","last_peered":"2024-04-25T16:00:19.348846+0000","last_clean":"2024-04-25T16:00:19.348846+0000","last_became_active":"2024-04-25T15:59:50.678558+0000","last_became_peered":"2024-04-25T15:59:50.678558+0000","last_unstale":"2024-04-25T16:00:19.348846+0000","last_undegraded":"2024-04-25T16:00:19.348846+0000","last_fullsized":"2024-04-25T16:00:19.348846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:40.691050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359322+0000","last_change":"2024-04-25T15:59:02.999307+0000","last_active":"2024-04-25T16:00:19.359322+0000","last_peered":"2024-04-25T16:00:19.359322+0000","last_clean":"2024-04-25T16:00:19.359322+0000","last_became_active":"2024-04-25T15:59:02.998976+0000","last_became_peered":"2024-04-25T15:59:02.998976+0000","last_unstale":"2024-04-25T16:00:19.359322+0000","last_undegraded":"2024-04-25T16:00:19.359322+0000","last_fullsized":"2024-04-25T16:00:19.359322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:29.626440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347700+0000","last_change":"2024-04-25T15:59:03.005208+0000","last_active":"2024-04-25T16:00:19.347700+0000","last_peered":"2024-04-25T16:00:19.347700+0000","last_clean":"2024-04-25T16:00:19.347700+0000","last_became_active":"2024-04-25T15:59:03.004869+0000","last_became_peered":"2024-04-25T15:59:03.004869+0000","last_unstale":"2024-04-25T16:00:19.347700+0000","last_undegraded":"2024-04-25T16:00:19.347700+0000","last_fullsized":"2024-04-25T16:00:19.347700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:00:24.040763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"77'10","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349448+0000","last_change":"2024-04-25T15:59:50.673556+0000","last_active":"2024-04-25T16:00:19.349448+0000","last_peered":"2024-04-25T16:00:19.349448+0000","last_clean":"2024-04-25T16:00:19.349448+0000","last_became_active":"2024-04-25T15:59:50.673119+0000","last_became_peered":"2024-04-25T15:59:50.673119+0000","last_unstale":"2024-04-25T16:00:19.349448+0000","last_undegraded":"2024-04-25T16:00:19.349448+0000","last_fullsized":"2024-04-25T16:00:19.349448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:09:50.286968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350359+0000","last_change":"2024-04-25T15:59:03.002160+0000","last_active":"2024-04-25T16:00:19.350359+0000","last_peered":"2024-04-25T16:00:19.350359+0000","last_clean":"2024-04-25T16:00:19.350359+0000","last_became_active":"2024-04-25T15:59:03.001855+0000","last_became_peered":"2024-04-25T15:59:03.001855+0000","last_unstale":"2024-04-25T16:00:19.350359+0000","last_undegraded":"2024-04-25T16:00:19.350359+0000","last_fullsized":"2024-04-25T16:00:19.350359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:43.383851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350513+0000","last_change":"2024-04-25T15:59:02.995318+0000","last_active":"2024-04-25T16:00:19.350513+0000","last_peered":"2024-04-25T16:00:19.350513+0000","last_clean":"2024-04-25T16:00:19.350513+0000","last_became_active":"2024-04-25T15:59:02.994956+0000","last_became_peered":"2024-04-25T15:59:02.994956+0000","last_unstale":"2024-04-25T16:00:19.350513+0000","last_undegraded":"2024-04-25T16:00:19.350513+0000","last_fullsized":"2024-04-25T16:00:19.350513+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:59.659400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359091+0000","last_change":"2024-04-25T15:59:02.996485+0000","last_active":"2024-04-25T16:00:19.359091+0000","last_peered":"2024-04-25T16:00:19.359091+0000","last_clean":"2024-04-25T16:00:19.359091+0000","last_became_active":"2024-04-25T15:59:02.996011+0000","last_became_peered":"2024-04-25T15:59:02.996011+0000","last_unstale":"2024-04-25T16:00:19.359091+0000","last_undegraded":"2024-04-25T16:00:19.359091+0000","last_fullsized":"2024-04-25T16:00:19.359091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:56.025254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.355086+0000","last_change":"2024-04-25T15:59:02.998455+0000","last_active":"2024-04-25T16:00:19.355086+0000","last_peered":"2024-04-25T16:00:19.355086+0000","last_clean":"2024-04-25T16:00:19.355086+0000","last_became_active":"2024-04-25T15:59:02.998126+0000","last_became_peered":"2024-04-25T15:59:02.998126+0000","last_unstale":"2024-04-25T16:00:19.355086+0000","last_undegraded":"2024-04-25T16:00:19.355086+0000","last_fullsized":"2024-04-25T16:00:19.355086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:20.342413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350929+0000","last_change":"2024-04-25T15:59:02.998766+0000","last_active":"2024-04-25T16:00:19.350929+0000","last_peered":"2024-04-25T16:00:19.350929+0000","last_clean":"2024-04-25T16:00:19.350929+0000","last_became_active":"2024-04-25T15:59:02.998415+0000","last_became_peered":"2024-04-25T15:59:02.998415+0000","last_unstale":"2024-04-25T16:00:19.350929+0000","last_undegraded":"2024-04-25T16:00:19.350929+0000","last_fullsized":"2024-04-25T16:00:19.350929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:55:05.717847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340892+0000","last_change":"2024-04-25T15:59:02.994982+0000","last_active":"2024-04-25T16:00:19.340892+0000","last_peered":"2024-04-25T16:00:19.340892+0000","last_clean":"2024-04-25T16:00:19.340892+0000","last_became_active":"2024-04-25T15:59:02.993928+0000","last_became_peered":"2024-04-25T15:59:02.993928+0000","last_unstale":"2024-04-25T16:00:19.340892+0000","last_undegraded":"2024-04-25T16:00:19.340892+0000","last_fullsized":"2024-04-25T16:00:19.340892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:22:25.346541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'13","reported_seq":107,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:22.382837+0000","last_change":"2024-04-25T15:59:02.512828+0000","last_active":"2024-04-25T16:00:22.382837+0000","last_peered":"2024-04-25T16:00:22.382837+0000","last_clean":"2024-04-25T16:00:22.382837+0000","last_became_active":"2024-04-25T15:59:02.512291+0000","last_became_peered":"2024-04-25T15:59:02.512291+0000","last_unstale":"2024-04-25T16:00:22.382837+0000","last_undegraded":"2024-04-25T16:00:22.382837+0000","last_fullsized":"2024-04-25T16:00:22.382837+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:02.372140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.339895+0000","last_change":"2024-04-25T15:59:03.008472+0000","last_active":"2024-04-25T16:00:19.339895+0000","last_peered":"2024-04-25T16:00:19.339895+0000","last_clean":"2024-04-25T16:00:19.339895+0000","last_became_active":"2024-04-25T15:59:03.008108+0000","last_became_peered":"2024-04-25T15:59:03.008108+0000","last_unstale":"2024-04-25T16:00:19.339895+0000","last_undegraded":"2024-04-25T16:00:19.339895+0000","last_fullsized":"2024-04-25T16:00:19.339895+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:40.087592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'2","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807892+0000","last_change":"2024-04-25T15:59:02.517846+0000","last_active":"2024-04-25T15:59:50.807892+0000","last_peered":"2024-04-25T15:59:50.807892+0000","last_clean":"2024-04-25T15:59:50.807892+0000","last_became_active":"2024-04-25T15:59:02.517366+0000","last_became_peered":"2024-04-25T15:59:02.517366+0000","last_unstale":"2024-04-25T15:59:50.807892+0000","last_undegraded":"2024-04-25T15:59:50.807892+0000","last_fullsized":"2024-04-25T15:59:50.807892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:29.620827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354150+0000","last_change":"2024-04-25T15:59:03.005202+0000","last_active":"2024-04-25T16:00:19.354150+0000","last_peered":"2024-04-25T16:00:19.354150+0000","last_clean":"2024-04-25T16:00:19.354150+0000","last_became_active":"2024-04-25T15:59:03.003447+0000","last_became_peered":"2024-04-25T15:59:03.003447+0000","last_unstale":"2024-04-25T16:00:19.354150+0000","last_undegraded":"2024-04-25T16:00:19.354150+0000","last_fullsized":"2024-04-25T16:00:19.354150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.740016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'12","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.703214+0000","last_change":"2024-04-25T15:59:50.702764+0000","last_active":"2024-04-25T15:59:50.703214+0000","last_peered":"2024-04-25T15:59:50.703214+0000","last_clean":"2024-04-25T15:59:50.703214+0000","last_became_active":"2024-04-25T15:59:50.679231+0000","last_became_peered":"2024-04-25T15:59:50.679231+0000","last_unstale":"2024-04-25T15:59:50.703214+0000","last_undegraded":"2024-04-25T15:59:50.703214+0000","last_fullsized":"2024-04-25T15:59:50.703214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:37.561992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360223+0000","last_change":"2024-04-25T15:59:03.004210+0000","last_active":"2024-04-25T16:00:19.360223+0000","last_peered":"2024-04-25T16:00:19.360223+0000","last_clean":"2024-04-25T16:00:19.360223+0000","last_became_active":"2024-04-25T15:59:03.002769+0000","last_became_peered":"2024-04-25T15:59:03.002769+0000","last_unstale":"2024-04-25T16:00:19.360223+0000","last_undegraded":"2024-04-25T16:00:19.360223+0000","last_fullsized":"2024-04-25T16:00:19.360223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:50.608327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669960+0000","last_change":"2024-04-25T15:59:02.506199+0000","last_active":"2024-04-25T15:59:50.669960+0000","last_peered":"2024-04-25T15:59:50.669960+0000","last_clean":"2024-04-25T15:59:50.669960+0000","last_became_active":"2024-04-25T15:59:02.505847+0000","last_became_peered":"2024-04-25T15:59:02.505847+0000","last_unstale":"2024-04-25T15:59:50.669960+0000","last_undegraded":"2024-04-25T15:59:50.669960+0000","last_fullsized":"2024-04-25T15:59:50.669960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:35.227112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"77'15","reported_seq":38,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359258+0000","last_change":"2024-04-25T15:59:50.679040+0000","last_active":"2024-04-25T16:00:19.359258+0000","last_peered":"2024-04-25T16:00:19.359258+0000","last_clean":"2024-04-25T16:00:19.359258+0000","last_became_active":"2024-04-25T15:59:50.678094+0000","last_became_peered":"2024-04-25T15:59:50.678094+0000","last_unstale":"2024-04-25T16:00:19.359258+0000","last_undegraded":"2024-04-25T16:00:19.359258+0000","last_fullsized":"2024-04-25T16:00:19.359258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:11.957966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'5","reported_seq":100,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:21.374575+0000","last_change":"2024-04-25T15:59:02.507763+0000","last_active":"2024-04-25T16:00:21.374575+0000","last_peered":"2024-04-25T16:00:21.374575+0000","last_clean":"2024-04-25T16:00:21.374575+0000","last_became_active":"2024-04-25T15:59:02.507338+0000","last_became_peered":"2024-04-25T15:59:02.507338+0000","last_unstale":"2024-04-25T16:00:21.374575+0000","last_undegraded":"2024-04-25T16:00:21.374575+0000","last_fullsized":"2024-04-25T16:00:21.374575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:31.710709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.326104+0000","last_change":"2024-04-25T15:59:03.002590+0000","last_active":"2024-04-25T16:00:19.326104+0000","last_peered":"2024-04-25T16:00:19.326104+0000","last_clean":"2024-04-25T16:00:19.326104+0000","last_became_active":"2024-04-25T15:59:03.002262+0000","last_became_peered":"2024-04-25T15:59:03.002262+0000","last_unstale":"2024-04-25T16:00:19.326104+0000","last_undegraded":"2024-04-25T16:00:19.326104+0000","last_fullsized":"2024-04-25T16:00:19.326104+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:48.251484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671899+0000","last_change":"2024-04-25T15:59:02.518853+0000","last_active":"2024-04-25T15:59:50.671899+0000","last_peered":"2024-04-25T15:59:50.671899+0000","last_clean":"2024-04-25T15:59:50.671899+0000","last_became_active":"2024-04-25T15:59:02.518505+0000","last_became_peered":"2024-04-25T15:59:02.518505+0000","last_unstale":"2024-04-25T15:59:50.671899+0000","last_undegraded":"2024-04-25T15:59:50.671899+0000","last_fullsized":"2024-04-25T15:59:50.671899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:01:56.282245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345600+0000","last_change":"2024-04-25T15:59:03.009733+0000","last_active":"2024-04-25T16:00:19.345600+0000","last_peered":"2024-04-25T16:00:19.345600+0000","last_clean":"2024-04-25T16:00:19.345600+0000","last_became_active":"2024-04-25T15:59:03.009251+0000","last_became_peered":"2024-04-25T15:59:03.009251+0000","last_unstale":"2024-04-25T16:00:19.345600+0000","last_undegraded":"2024-04-25T16:00:19.345600+0000","last_fullsized":"2024-04-25T16:00:19.345600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:22.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"75'3","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670602+0000","last_change":"2024-04-25T15:59:02.512493+0000","last_active":"2024-04-25T15:59:50.670602+0000","last_peered":"2024-04-25T15:59:50.670602+0000","last_clean":"2024-04-25T15:59:50.670602+0000","last_became_active":"2024-04-25T15:59:02.512103+0000","last_became_peered":"2024-04-25T15:59:02.512103+0000","last_unstale":"2024-04-25T15:59:50.670602+0000","last_undegraded":"2024-04-25T15:59:50.670602+0000","last_fullsized":"2024-04-25T15:59:50.670602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:29.457755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342733+0000","last_change":"2024-04-25T15:59:03.001406+0000","last_active":"2024-04-25T16:00:19.342733+0000","last_peered":"2024-04-25T16:00:19.342733+0000","last_clean":"2024-04-25T16:00:19.342733+0000","last_became_active":"2024-04-25T15:59:02.999038+0000","last_became_peered":"2024-04-25T15:59:02.999038+0000","last_unstale":"2024-04-25T16:00:19.342733+0000","last_undegraded":"2024-04-25T16:00:19.342733+0000","last_fullsized":"2024-04-25T16:00:19.342733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:56.354821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671531+0000","last_change":"2024-04-25T15:59:02.509619+0000","last_active":"2024-04-25T15:59:50.671531+0000","last_peered":"2024-04-25T15:59:50.671531+0000","last_clean":"2024-04-25T15:59:50.671531+0000","last_became_active":"2024-04-25T15:59:02.509254+0000","last_became_peered":"2024-04-25T15:59:02.509254+0000","last_unstale":"2024-04-25T15:59:50.671531+0000","last_undegraded":"2024-04-25T15:59:50.671531+0000","last_fullsized":"2024-04-25T15:59:50.671531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:00.382581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347276+0000","last_change":"2024-04-25T15:59:03.000009+0000","last_active":"2024-04-25T16:00:19.347276+0000","last_peered":"2024-04-25T16:00:19.347276+0000","last_clean":"2024-04-25T16:00:19.347276+0000","last_became_active":"2024-04-25T15:59:02.999662+0000","last_became_peered":"2024-04-25T15:59:02.999662+0000","last_unstale":"2024-04-25T16:00:19.347276+0000","last_undegraded":"2024-04-25T16:00:19.347276+0000","last_fullsized":"2024-04-25T16:00:19.347276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:05:38.004819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"75'4","reported_seq":169,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:21.373660+0000","last_change":"2024-04-25T15:59:02.514429+0000","last_active":"2024-04-25T16:00:21.373660+0000","last_peered":"2024-04-25T16:00:21.373660+0000","last_clean":"2024-04-25T16:00:21.373660+0000","last_became_active":"2024-04-25T15:59:02.514144+0000","last_became_peered":"2024-04-25T15:59:02.514144+0000","last_unstale":"2024-04-25T16:00:21.373660+0000","last_undegraded":"2024-04-25T16:00:21.373660+0000","last_fullsized":"2024-04-25T16:00:21.373660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:45.144630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":146,"num_read_kb":146,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360393+0000","last_change":"2024-04-25T15:59:03.008460+0000","last_active":"2024-04-25T16:00:19.360393+0000","last_peered":"2024-04-25T16:00:19.360393+0000","last_clean":"2024-04-25T16:00:19.360393+0000","last_became_active":"2024-04-25T15:59:03.008024+0000","last_became_peered":"2024-04-25T15:59:03.008024+0000","last_unstale":"2024-04-25T16:00:19.360393+0000","last_undegraded":"2024-04-25T16:00:19.360393+0000","last_fullsized":"2024-04-25T16:00:19.360393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:10.288760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"75'4","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676828+0000","last_change":"2024-04-25T15:59:02.512494+0000","last_active":"2024-04-25T15:59:50.676828+0000","last_peered":"2024-04-25T15:59:50.676828+0000","last_clean":"2024-04-25T15:59:50.676828+0000","last_became_active":"2024-04-25T15:59:02.511182+0000","last_became_peered":"2024-04-25T15:59:02.511182+0000","last_unstale":"2024-04-25T15:59:50.676828+0000","last_undegraded":"2024-04-25T15:59:50.676828+0000","last_fullsized":"2024-04-25T15:59:50.676828+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:54.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350559+0000","last_change":"2024-04-25T15:59:03.003848+0000","last_active":"2024-04-25T16:00:19.350559+0000","last_peered":"2024-04-25T16:00:19.350559+0000","last_clean":"2024-04-25T16:00:19.350559+0000","last_became_active":"2024-04-25T15:59:03.003512+0000","last_became_peered":"2024-04-25T15:59:03.003512+0000","last_unstale":"2024-04-25T16:00:19.350559+0000","last_undegraded":"2024-04-25T16:00:19.350559+0000","last_fullsized":"2024-04-25T16:00:19.350559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:54.667054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672694+0000","last_change":"2024-04-25T15:59:02.509343+0000","last_active":"2024-04-25T15:59:50.672694+0000","last_peered":"2024-04-25T15:59:50.672694+0000","last_clean":"2024-04-25T15:59:50.672694+0000","last_became_active":"2024-04-25T15:59:02.508980+0000","last_became_peered":"2024-04-25T15:59:02.508980+0000","last_unstale":"2024-04-25T15:59:50.672694+0000","last_undegraded":"2024-04-25T15:59:50.672694+0000","last_fullsized":"2024-04-25T15:59:50.672694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:54.344717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359525+0000","last_change":"2024-04-25T15:59:02.999862+0000","last_active":"2024-04-25T16:00:19.359525+0000","last_peered":"2024-04-25T16:00:19.359525+0000","last_clean":"2024-04-25T16:00:19.359525+0000","last_became_active":"2024-04-25T15:59:02.999472+0000","last_became_peered":"2024-04-25T15:59:02.999472+0000","last_unstale":"2024-04-25T16:00:19.359525+0000","last_undegraded":"2024-04-25T16:00:19.359525+0000","last_fullsized":"2024-04-25T16:00:19.359525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:46.165045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"77'32","reported_seq":205,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:21.376772+0000","last_change":"2024-04-25T15:59:02.508949+0000","last_active":"2024-04-25T16:00:21.376772+0000","last_peered":"2024-04-25T16:00:21.376772+0000","last_clean":"2024-04-25T16:00:21.376772+0000","last_became_active":"2024-04-25T15:59:02.508564+0000","last_became_peered":"2024-04-25T15:59:02.508564+0000","last_unstale":"2024-04-25T16:00:21.376772+0000","last_undegraded":"2024-04-25T16:00:21.376772+0000","last_fullsized":"2024-04-25T16:00:21.376772+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:12.590193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57434,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":154,"num_read_kb":134,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344171+0000","last_change":"2024-04-25T15:59:02.998575+0000","last_active":"2024-04-25T16:00:19.344171+0000","last_peered":"2024-04-25T16:00:19.344171+0000","last_clean":"2024-04-25T16:00:19.344171+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.344171+0000","last_undegraded":"2024-04-25T16:00:19.344171+0000","last_fullsized":"2024-04-25T16:00:19.344171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:40.640468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'9","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677368+0000","last_change":"2024-04-25T15:59:02.519110+0000","last_active":"2024-04-25T15:59:50.677368+0000","last_peered":"2024-04-25T15:59:50.677368+0000","last_clean":"2024-04-25T15:59:50.677368+0000","last_became_active":"2024-04-25T15:59:02.518373+0000","last_became_peered":"2024-04-25T15:59:02.518373+0000","last_unstale":"2024-04-25T15:59:50.677368+0000","last_undegraded":"2024-04-25T15:59:50.677368+0000","last_fullsized":"2024-04-25T15:59:50.677368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:27.840296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344110+0000","last_change":"2024-04-25T15:59:02.991002+0000","last_active":"2024-04-25T16:00:19.344110+0000","last_peered":"2024-04-25T16:00:19.344110+0000","last_clean":"2024-04-25T16:00:19.344110+0000","last_became_active":"2024-04-25T15:59:02.990605+0000","last_became_peered":"2024-04-25T15:59:02.990605+0000","last_unstale":"2024-04-25T16:00:19.344110+0000","last_undegraded":"2024-04-25T16:00:19.344110+0000","last_fullsized":"2024-04-25T16:00:19.344110+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:22:00.723255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671723+0000","last_change":"2024-04-25T15:59:02.509809+0000","last_active":"2024-04-25T15:59:50.671723+0000","last_peered":"2024-04-25T15:59:50.671723+0000","last_clean":"2024-04-25T15:59:50.671723+0000","last_became_active":"2024-04-25T15:59:02.509370+0000","last_became_peered":"2024-04-25T15:59:02.509370+0000","last_unstale":"2024-04-25T15:59:50.671723+0000","last_undegraded":"2024-04-25T15:59:50.671723+0000","last_fullsized":"2024-04-25T15:59:50.671723+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:24.416072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343672+0000","last_change":"2024-04-25T15:59:02.999836+0000","last_active":"2024-04-25T16:00:19.343672+0000","last_peered":"2024-04-25T16:00:19.343672+0000","last_clean":"2024-04-25T16:00:19.343672+0000","last_became_active":"2024-04-25T15:59:02.997770+0000","last_became_peered":"2024-04-25T15:59:02.997770+0000","last_unstale":"2024-04-25T16:00:19.343672+0000","last_undegraded":"2024-04-25T16:00:19.343672+0000","last_fullsized":"2024-04-25T16:00:19.343672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:30.953339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"77'7","reported_seq":103,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:21.386288+0000","last_change":"2024-04-25T15:59:02.510465+0000","last_active":"2024-04-25T16:00:21.386288+0000","last_peered":"2024-04-25T16:00:21.386288+0000","last_clean":"2024-04-25T16:00:21.386288+0000","last_became_active":"2024-04-25T15:59:02.510067+0000","last_became_peered":"2024-04-25T15:59:02.510067+0000","last_unstale":"2024-04-25T16:00:21.386288+0000","last_undegraded":"2024-04-25T16:00:21.386288+0000","last_fullsized":"2024-04-25T16:00:21.386288+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:40.739358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346234+0000","last_change":"2024-04-25T15:59:03.000028+0000","last_active":"2024-04-25T16:00:19.346234+0000","last_peered":"2024-04-25T16:00:19.346234+0000","last_clean":"2024-04-25T16:00:19.346234+0000","last_became_active":"2024-04-25T15:59:02.999704+0000","last_became_peered":"2024-04-25T15:59:02.999704+0000","last_unstale":"2024-04-25T16:00:19.346234+0000","last_undegraded":"2024-04-25T16:00:19.346234+0000","last_fullsized":"2024-04-25T16:00:19.346234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:55.015085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'17","reported_seq":37,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669662+0000","last_change":"2024-04-25T15:59:02.514442+0000","last_active":"2024-04-25T15:59:50.669662+0000","last_peered":"2024-04-25T15:59:50.669662+0000","last_clean":"2024-04-25T15:59:50.669662+0000","last_became_active":"2024-04-25T15:59:02.513951+0000","last_became_peered":"2024-04-25T15:59:02.513951+0000","last_unstale":"2024-04-25T15:59:50.669662+0000","last_undegraded":"2024-04-25T15:59:50.669662+0000","last_fullsized":"2024-04-25T15:59:50.669662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:56.443999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"77'6","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.358196+0000","last_change":"2024-04-25T15:59:50.677155+0000","last_active":"2024-04-25T16:00:19.358196+0000","last_peered":"2024-04-25T16:00:19.358196+0000","last_clean":"2024-04-25T16:00:19.358196+0000","last_became_active":"2024-04-25T15:59:50.676721+0000","last_became_peered":"2024-04-25T15:59:50.676721+0000","last_unstale":"2024-04-25T16:00:19.358196+0000","last_undegraded":"2024-04-25T16:00:19.358196+0000","last_fullsized":"2024-04-25T16:00:19.358196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:04.138269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.683753+0000","last_change":"2024-04-25T15:59:50.683228+0000","last_active":"2024-04-25T15:59:50.683753+0000","last_peered":"2024-04-25T15:59:50.683753+0000","last_clean":"2024-04-25T15:59:50.683753+0000","last_became_active":"2024-04-25T15:59:50.675414+0000","last_became_peered":"2024-04-25T15:59:50.675414+0000","last_unstale":"2024-04-25T15:59:50.683753+0000","last_undegraded":"2024-04-25T15:59:50.683753+0000","last_fullsized":"2024-04-25T15:59:50.683753+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:46.577883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341606+0000","last_change":"2024-04-25T15:59:03.004639+0000","last_active":"2024-04-25T16:00:19.341606+0000","last_peered":"2024-04-25T16:00:19.341606+0000","last_clean":"2024-04-25T16:00:19.341606+0000","last_became_active":"2024-04-25T15:59:03.004260+0000","last_became_peered":"2024-04-25T15:59:03.004260+0000","last_unstale":"2024-04-25T16:00:19.341606+0000","last_undegraded":"2024-04-25T16:00:19.341606+0000","last_fullsized":"2024-04-25T16:00:19.341606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:00.450758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666336+0000","last_change":"2024-04-25T15:59:02.509716+0000","last_active":"2024-04-25T15:59:50.666336+0000","last_peered":"2024-04-25T15:59:50.666336+0000","last_clean":"2024-04-25T15:59:50.666336+0000","last_became_active":"2024-04-25T15:59:02.509303+0000","last_became_peered":"2024-04-25T15:59:02.509303+0000","last_unstale":"2024-04-25T15:59:50.666336+0000","last_undegraded":"2024-04-25T15:59:50.666336+0000","last_fullsized":"2024-04-25T15:59:50.666336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:34.037590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349639+0000","last_change":"2024-04-25T15:59:03.005141+0000","last_active":"2024-04-25T16:00:19.349639+0000","last_peered":"2024-04-25T16:00:19.349639+0000","last_clean":"2024-04-25T16:00:19.349639+0000","last_became_active":"2024-04-25T15:59:03.004760+0000","last_became_peered":"2024-04-25T15:59:03.004760+0000","last_unstale":"2024-04-25T16:00:19.349639+0000","last_undegraded":"2024-04-25T16:00:19.349639+0000","last_fullsized":"2024-04-25T16:00:19.349639+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:14.923126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"75'5","reported_seq":97,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:20.380650+0000","last_change":"2024-04-25T15:59:02.512779+0000","last_active":"2024-04-25T16:00:20.380650+0000","last_peered":"2024-04-25T16:00:20.380650+0000","last_clean":"2024-04-25T16:00:20.380650+0000","last_became_active":"2024-04-25T15:59:02.511240+0000","last_became_peered":"2024-04-25T15:59:02.511240+0000","last_unstale":"2024-04-25T16:00:20.380650+0000","last_undegraded":"2024-04-25T16:00:20.380650+0000","last_fullsized":"2024-04-25T16:00:20.380650+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:51.703751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351726+0000","last_change":"2024-04-25T15:59:03.004841+0000","last_active":"2024-04-25T16:00:19.351726+0000","last_peered":"2024-04-25T16:00:19.351726+0000","last_clean":"2024-04-25T16:00:19.351726+0000","last_became_active":"2024-04-25T15:59:03.004488+0000","last_became_peered":"2024-04-25T15:59:03.004488+0000","last_unstale":"2024-04-25T16:00:19.351726+0000","last_undegraded":"2024-04-25T16:00:19.351726+0000","last_fullsized":"2024-04-25T16:00:19.351726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:07.672312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345631+0000","last_change":"2024-04-25T15:59:03.001296+0000","last_active":"2024-04-25T16:00:19.345631+0000","last_peered":"2024-04-25T16:00:19.345631+0000","last_clean":"2024-04-25T16:00:19.345631+0000","last_became_active":"2024-04-25T15:59:03.000568+0000","last_became_peered":"2024-04-25T15:59:03.000568+0000","last_unstale":"2024-04-25T16:00:19.345631+0000","last_undegraded":"2024-04-25T16:00:19.345631+0000","last_fullsized":"2024-04-25T16:00:19.345631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:51.335849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"77'6","reported_seq":101,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:22.387281+0000","last_change":"2024-04-25T15:59:02.513943+0000","last_active":"2024-04-25T16:00:22.387281+0000","last_peered":"2024-04-25T16:00:22.387281+0000","last_clean":"2024-04-25T16:00:22.387281+0000","last_became_active":"2024-04-25T15:59:02.513595+0000","last_became_peered":"2024-04-25T15:59:02.513595+0000","last_unstale":"2024-04-25T16:00:22.387281+0000","last_undegraded":"2024-04-25T16:00:22.387281+0000","last_fullsized":"2024-04-25T16:00:22.387281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:50.720333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.352846+0000","last_change":"2024-04-25T15:59:03.008561+0000","last_active":"2024-04-25T16:00:19.352846+0000","last_peered":"2024-04-25T16:00:19.352846+0000","last_clean":"2024-04-25T16:00:19.352846+0000","last_became_active":"2024-04-25T15:59:03.008134+0000","last_became_peered":"2024-04-25T15:59:03.008134+0000","last_unstale":"2024-04-25T16:00:19.352846+0000","last_undegraded":"2024-04-25T16:00:19.352846+0000","last_fullsized":"2024-04-25T16:00:19.352846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:43.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"75'6","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669568+0000","last_change":"2024-04-25T15:59:02.510731+0000","last_active":"2024-04-25T15:59:50.669568+0000","last_peered":"2024-04-25T15:59:50.669568+0000","last_clean":"2024-04-25T15:59:50.669568+0000","last_became_active":"2024-04-25T15:59:02.510356+0000","last_became_peered":"2024-04-25T15:59:02.510356+0000","last_unstale":"2024-04-25T15:59:50.669568+0000","last_undegraded":"2024-04-25T15:59:50.669568+0000","last_fullsized":"2024-04-25T15:59:50.669568+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:33.066858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":94,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676873+0000","last_change":"2024-04-25T15:57:47.988899+0000","last_active":"2024-04-25T15:59:50.676873+0000","last_peered":"2024-04-25T15:59:50.676873+0000","last_clean":"2024-04-25T15:59:50.676873+0000","last_became_active":"2024-04-25T15:57:47.982613+0000","last_became_peered":"2024-04-25T15:57:47.982613+0000","last_unstale":"2024-04-25T15:59:50.676873+0000","last_undegraded":"2024-04-25T15:59:50.676873+0000","last_fullsized":"2024-04-25T15:59:50.676873+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_clean_scrub_stamp":"2024-04-25T15:54:18.013489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:39.323767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.334020+0000","last_change":"2024-04-25T15:59:02.998360+0000","last_active":"2024-04-25T16:00:19.334020+0000","last_peered":"2024-04-25T16:00:19.334020+0000","last_clean":"2024-04-25T16:00:19.334020+0000","last_became_active":"2024-04-25T15:59:02.998048+0000","last_became_peered":"2024-04-25T15:59:02.998048+0000","last_unstale":"2024-04-25T16:00:19.334020+0000","last_undegraded":"2024-04-25T16:00:19.334020+0000","last_fullsized":"2024-04-25T16:00:19.334020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:01.524608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"75'4","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677261+0000","last_change":"2024-04-25T15:59:02.515769+0000","last_active":"2024-04-25T15:59:50.677261+0000","last_peered":"2024-04-25T15:59:50.677261+0000","last_clean":"2024-04-25T15:59:50.677261+0000","last_became_active":"2024-04-25T15:59:02.515467+0000","last_became_peered":"2024-04-25T15:59:02.515467+0000","last_unstale":"2024-04-25T15:59:50.677261+0000","last_undegraded":"2024-04-25T15:59:50.677261+0000","last_fullsized":"2024-04-25T15:59:50.677261+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:59:50.107475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670090+0000","last_change":"2024-04-25T15:59:02.512701+0000","last_active":"2024-04-25T15:59:50.670090+0000","last_peered":"2024-04-25T15:59:50.670090+0000","last_clean":"2024-04-25T15:59:50.670090+0000","last_became_active":"2024-04-25T15:59:02.512352+0000","last_became_peered":"2024-04-25T15:59:02.512352+0000","last_unstale":"2024-04-25T15:59:50.670090+0000","last_undegraded":"2024-04-25T15:59:50.670090+0000","last_fullsized":"2024-04-25T15:59:50.670090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:31.772382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.329902+0000","last_change":"2024-04-25T15:59:02.989698+0000","last_active":"2024-04-25T16:00:19.329902+0000","last_peered":"2024-04-25T16:00:19.329902+0000","last_clean":"2024-04-25T16:00:19.329902+0000","last_became_active":"2024-04-25T15:59:02.989284+0000","last_became_peered":"2024-04-25T15:59:02.989284+0000","last_unstale":"2024-04-25T16:00:19.329902+0000","last_undegraded":"2024-04-25T16:00:19.329902+0000","last_fullsized":"2024-04-25T16:00:19.329902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:12.501738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"77'13","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666592+0000","last_change":"2024-04-25T15:59:02.517272+0000","last_active":"2024-04-25T15:59:50.666592+0000","last_peered":"2024-04-25T15:59:50.666592+0000","last_clean":"2024-04-25T15:59:50.666592+0000","last_became_active":"2024-04-25T15:59:02.516370+0000","last_became_peered":"2024-04-25T15:59:02.516370+0000","last_unstale":"2024-04-25T15:59:50.666592+0000","last_undegraded":"2024-04-25T15:59:50.666592+0000","last_fullsized":"2024-04-25T15:59:50.666592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:31.139689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345612+0000","last_change":"2024-04-25T15:59:03.003111+0000","last_active":"2024-04-25T16:00:19.345612+0000","last_peered":"2024-04-25T16:00:19.345612+0000","last_clean":"2024-04-25T16:00:19.345612+0000","last_became_active":"2024-04-25T15:59:03.002766+0000","last_became_peered":"2024-04-25T15:59:03.002766+0000","last_unstale":"2024-04-25T16:00:19.345612+0000","last_undegraded":"2024-04-25T16:00:19.345612+0000","last_fullsized":"2024-04-25T16:00:19.345612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:39.866119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'7","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670151+0000","last_change":"2024-04-25T15:59:02.521970+0000","last_active":"2024-04-25T15:59:50.670151+0000","last_peered":"2024-04-25T15:59:50.670151+0000","last_clean":"2024-04-25T15:59:50.670151+0000","last_became_active":"2024-04-25T15:59:02.521650+0000","last_became_peered":"2024-04-25T15:59:02.521650+0000","last_unstale":"2024-04-25T15:59:50.670151+0000","last_undegraded":"2024-04-25T15:59:50.670151+0000","last_fullsized":"2024-04-25T15:59:50.670151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:32.996868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351310+0000","last_change":"2024-04-25T15:59:02.996358+0000","last_active":"2024-04-25T16:00:19.351310+0000","last_peered":"2024-04-25T16:00:19.351310+0000","last_clean":"2024-04-25T16:00:19.351310+0000","last_became_active":"2024-04-25T15:59:02.995988+0000","last_became_peered":"2024-04-25T15:59:02.995988+0000","last_unstale":"2024-04-25T16:00:19.351310+0000","last_undegraded":"2024-04-25T16:00:19.351310+0000","last_fullsized":"2024-04-25T16:00:19.351310+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:51.021841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"75'2","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.674464+0000","last_change":"2024-04-25T15:59:02.518330+0000","last_active":"2024-04-25T15:59:50.674464+0000","last_peered":"2024-04-25T15:59:50.674464+0000","last_clean":"2024-04-25T15:59:50.674464+0000","last_became_active":"2024-04-25T15:59:02.517978+0000","last_became_peered":"2024-04-25T15:59:02.517978+0000","last_unstale":"2024-04-25T15:59:50.674464+0000","last_undegraded":"2024-04-25T15:59:50.674464+0000","last_fullsized":"2024-04-25T15:59:50.674464+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:25.443963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.302105+0000","last_change":"2024-04-25T15:59:03.000198+0000","last_active":"2024-04-25T16:00:19.302105+0000","last_peered":"2024-04-25T16:00:19.302105+0000","last_clean":"2024-04-25T16:00:19.302105+0000","last_became_active":"2024-04-25T15:59:02.999878+0000","last_became_peered":"2024-04-25T15:59:02.999878+0000","last_unstale":"2024-04-25T16:00:19.302105+0000","last_undegraded":"2024-04-25T16:00:19.302105+0000","last_fullsized":"2024-04-25T16:00:19.302105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:37.966595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807467+0000","last_change":"2024-04-25T15:59:02.511298+0000","last_active":"2024-04-25T15:59:50.807467+0000","last_peered":"2024-04-25T15:59:50.807467+0000","last_clean":"2024-04-25T15:59:50.807467+0000","last_became_active":"2024-04-25T15:59:02.510521+0000","last_became_peered":"2024-04-25T15:59:02.510521+0000","last_unstale":"2024-04-25T15:59:50.807467+0000","last_undegraded":"2024-04-25T15:59:50.807467+0000","last_fullsized":"2024-04-25T15:59:50.807467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:00.807919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347593+0000","last_change":"2024-04-25T15:59:02.999212+0000","last_active":"2024-04-25T16:00:19.347593+0000","last_peered":"2024-04-25T16:00:19.347593+0000","last_clean":"2024-04-25T16:00:19.347593+0000","last_became_active":"2024-04-25T15:59:02.998881+0000","last_became_peered":"2024-04-25T15:59:02.998881+0000","last_unstale":"2024-04-25T16:00:19.347593+0000","last_undegraded":"2024-04-25T16:00:19.347593+0000","last_fullsized":"2024-04-25T16:00:19.347593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:09.092196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"75'3","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676880+0000","last_change":"2024-04-25T15:59:02.517412+0000","last_active":"2024-04-25T15:59:50.676880+0000","last_peered":"2024-04-25T15:59:50.676880+0000","last_clean":"2024-04-25T15:59:50.676880+0000","last_became_active":"2024-04-25T15:59:02.517077+0000","last_became_peered":"2024-04-25T15:59:02.517077+0000","last_unstale":"2024-04-25T15:59:50.676880+0000","last_undegraded":"2024-04-25T15:59:50.676880+0000","last_fullsized":"2024-04-25T15:59:50.676880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:00.161199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"77'8","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342365+0000","last_change":"2024-04-25T15:59:50.678844+0000","last_active":"2024-04-25T16:00:19.342365+0000","last_peered":"2024-04-25T16:00:19.342365+0000","last_clean":"2024-04-25T16:00:19.342365+0000","last_became_active":"2024-04-25T15:59:50.678411+0000","last_became_peered":"2024-04-25T15:59:50.678411+0000","last_unstale":"2024-04-25T16:00:19.342365+0000","last_undegraded":"2024-04-25T16:00:19.342365+0000","last_fullsized":"2024-04-25T16:00:19.342365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:56:48.018678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669652+0000","last_change":"2024-04-25T15:59:02.515912+0000","last_active":"2024-04-25T15:59:50.669652+0000","last_peered":"2024-04-25T15:59:50.669652+0000","last_clean":"2024-04-25T15:59:50.669652+0000","last_became_active":"2024-04-25T15:59:02.514949+0000","last_became_peered":"2024-04-25T15:59:02.514949+0000","last_unstale":"2024-04-25T15:59:50.669652+0000","last_undegraded":"2024-04-25T15:59:50.669652+0000","last_fullsized":"2024-04-25T15:59:50.669652+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:33.467223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360414+0000","last_change":"2024-04-25T15:59:02.988896+0000","last_active":"2024-04-25T16:00:19.360414+0000","last_peered":"2024-04-25T16:00:19.360414+0000","last_clean":"2024-04-25T16:00:19.360414+0000","last_became_active":"2024-04-25T15:59:02.988478+0000","last_became_peered":"2024-04-25T15:59:02.988478+0000","last_unstale":"2024-04-25T16:00:19.360414+0000","last_undegraded":"2024-04-25T16:00:19.360414+0000","last_fullsized":"2024-04-25T16:00:19.360414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:47.014678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"75'5","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:21.722005+0000","last_change":"2024-04-25T15:59:50.687632+0000","last_active":"2024-04-25T16:00:21.722005+0000","last_peered":"2024-04-25T16:00:21.722005+0000","last_clean":"2024-04-25T16:00:21.722005+0000","last_became_active":"2024-04-25T15:59:50.680402+0000","last_became_peered":"2024-04-25T15:59:50.680402+0000","last_unstale":"2024-04-25T16:00:21.722005+0000","last_undegraded":"2024-04-25T16:00:21.722005+0000","last_fullsized":"2024-04-25T16:00:21.722005+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:12.544489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":33,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359317+0000","last_change":"2024-04-25T15:59:02.990507+0000","last_active":"2024-04-25T16:00:19.359317+0000","last_peered":"2024-04-25T16:00:19.359317+0000","last_clean":"2024-04-25T16:00:19.359317+0000","last_became_active":"2024-04-25T15:59:02.990160+0000","last_became_peered":"2024-04-25T15:59:02.990160+0000","last_unstale":"2024-04-25T16:00:19.359317+0000","last_undegraded":"2024-04-25T16:00:19.359317+0000","last_fullsized":"2024-04-25T16:00:19.359317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:18:21.526155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672520+0000","last_change":"2024-04-25T15:59:02.510956+0000","last_active":"2024-04-25T15:59:50.672520+0000","last_peered":"2024-04-25T15:59:50.672520+0000","last_clean":"2024-04-25T15:59:50.672520+0000","last_became_active":"2024-04-25T15:59:02.510578+0000","last_became_peered":"2024-04-25T15:59:02.510578+0000","last_unstale":"2024-04-25T15:59:50.672520+0000","last_undegraded":"2024-04-25T15:59:50.672520+0000","last_fullsized":"2024-04-25T15:59:50.672520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:11.512365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337102+0000","last_change":"2024-04-25T15:59:03.001122+0000","last_active":"2024-04-25T16:00:19.337102+0000","last_peered":"2024-04-25T16:00:19.337102+0000","last_clean":"2024-04-25T16:00:19.337102+0000","last_became_active":"2024-04-25T15:59:03.000672+0000","last_became_peered":"2024-04-25T15:59:03.000672+0000","last_unstale":"2024-04-25T16:00:19.337102+0000","last_undegraded":"2024-04-25T16:00:19.337102+0000","last_fullsized":"2024-04-25T16:00:19.337102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:33.216434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'5","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670676+0000","last_change":"2024-04-25T15:59:02.520645+0000","last_active":"2024-04-25T15:59:50.670676+0000","last_peered":"2024-04-25T15:59:50.670676+0000","last_clean":"2024-04-25T15:59:50.670676+0000","last_became_active":"2024-04-25T15:59:02.520197+0000","last_became_peered":"2024-04-25T15:59:02.520197+0000","last_unstale":"2024-04-25T15:59:50.670676+0000","last_undegraded":"2024-04-25T15:59:50.670676+0000","last_fullsized":"2024-04-25T15:59:50.670676+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:20.120554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347691+0000","last_change":"2024-04-25T15:59:03.001668+0000","last_active":"2024-04-25T16:00:19.347691+0000","last_peered":"2024-04-25T16:00:19.347691+0000","last_clean":"2024-04-25T16:00:19.347691+0000","last_became_active":"2024-04-25T15:59:03.001333+0000","last_became_peered":"2024-04-25T15:59:03.001333+0000","last_unstale":"2024-04-25T16:00:19.347691+0000","last_undegraded":"2024-04-25T16:00:19.347691+0000","last_fullsized":"2024-04-25T16:00:19.347691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:24.992476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346081+0000","last_change":"2024-04-25T15:59:03.005533+0000","last_active":"2024-04-25T16:00:19.346081+0000","last_peered":"2024-04-25T16:00:19.346081+0000","last_clean":"2024-04-25T16:00:19.346081+0000","last_became_active":"2024-04-25T15:59:03.004124+0000","last_became_peered":"2024-04-25T15:59:03.004124+0000","last_unstale":"2024-04-25T16:00:19.346081+0000","last_undegraded":"2024-04-25T16:00:19.346081+0000","last_fullsized":"2024-04-25T16:00:19.346081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:27:28.507065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"77'12","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.331274+0000","last_change":"2024-04-25T15:59:02.993416+0000","last_active":"2024-04-25T16:00:19.331274+0000","last_peered":"2024-04-25T16:00:19.331274+0000","last_clean":"2024-04-25T16:00:19.331274+0000","last_became_active":"2024-04-25T15:59:02.993098+0000","last_became_peered":"2024-04-25T15:59:02.993098+0000","last_unstale":"2024-04-25T16:00:19.331274+0000","last_undegraded":"2024-04-25T16:00:19.331274+0000","last_fullsized":"2024-04-25T16:00:19.331274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:46.859932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350142+0000","last_change":"2024-04-25T15:59:02.996858+0000","last_active":"2024-04-25T16:00:19.350142+0000","last_peered":"2024-04-25T16:00:19.350142+0000","last_clean":"2024-04-25T16:00:19.350142+0000","last_became_active":"2024-04-25T15:59:02.996530+0000","last_became_peered":"2024-04-25T15:59:02.996530+0000","last_unstale":"2024-04-25T16:00:19.350142+0000","last_undegraded":"2024-04-25T16:00:19.350142+0000","last_fullsized":"2024-04-25T16:00:19.350142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:36.723948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350470+0000","last_change":"2024-04-25T15:59:02.995942+0000","last_active":"2024-04-25T16:00:19.350470+0000","last_peered":"2024-04-25T16:00:19.350470+0000","last_clean":"2024-04-25T16:00:19.350470+0000","last_became_active":"2024-04-25T15:59:02.995567+0000","last_became_peered":"2024-04-25T15:59:02.995567+0000","last_unstale":"2024-04-25T16:00:19.350470+0000","last_undegraded":"2024-04-25T16:00:19.350470+0000","last_fullsized":"2024-04-25T16:00:19.350470+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:32.692392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350261+0000","last_change":"2024-04-25T15:59:03.000326+0000","last_active":"2024-04-25T16:00:19.350261+0000","last_peered":"2024-04-25T16:00:19.350261+0000","last_clean":"2024-04-25T16:00:19.350261+0000","last_became_active":"2024-04-25T15:59:02.999961+0000","last_became_peered":"2024-04-25T15:59:02.999961+0000","last_unstale":"2024-04-25T16:00:19.350261+0000","last_undegraded":"2024-04-25T16:00:19.350261+0000","last_fullsized":"2024-04-25T16:00:19.350261+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:10.631460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341696+0000","last_change":"2024-04-25T15:59:03.003625+0000","last_active":"2024-04-25T16:00:19.341696+0000","last_peered":"2024-04-25T16:00:19.341696+0000","last_clean":"2024-04-25T16:00:19.341696+0000","last_became_active":"2024-04-25T15:59:03.003280+0000","last_became_peered":"2024-04-25T15:59:03.003280+0000","last_unstale":"2024-04-25T16:00:19.341696+0000","last_undegraded":"2024-04-25T16:00:19.341696+0000","last_fullsized":"2024-04-25T16:00:19.341696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:21:54.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346578+0000","last_change":"2024-04-25T15:59:02.994645+0000","last_active":"2024-04-25T16:00:19.346578+0000","last_peered":"2024-04-25T16:00:19.346578+0000","last_clean":"2024-04-25T16:00:19.346578+0000","last_became_active":"2024-04-25T15:59:02.993924+0000","last_became_peered":"2024-04-25T15:59:02.993924+0000","last_unstale":"2024-04-25T16:00:19.346578+0000","last_undegraded":"2024-04-25T16:00:19.346578+0000","last_fullsized":"2024-04-25T16:00:19.346578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.644708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340331+0000","last_change":"2024-04-25T15:59:03.006156+0000","last_active":"2024-04-25T16:00:19.340331+0000","last_peered":"2024-04-25T16:00:19.340331+0000","last_clean":"2024-04-25T16:00:19.340331+0000","last_became_active":"2024-04-25T15:59:03.005795+0000","last_became_peered":"2024-04-25T15:59:03.005795+0000","last_unstale":"2024-04-25T16:00:19.340331+0000","last_undegraded":"2024-04-25T16:00:19.340331+0000","last_fullsized":"2024-04-25T16:00:19.340331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:23.523868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337613+0000","last_change":"2024-04-25T15:59:03.010404+0000","last_active":"2024-04-25T16:00:19.337613+0000","last_peered":"2024-04-25T16:00:19.337613+0000","last_clean":"2024-04-25T16:00:19.337613+0000","last_became_active":"2024-04-25T15:59:03.010100+0000","last_became_peered":"2024-04-25T15:59:03.010100+0000","last_unstale":"2024-04-25T16:00:19.337613+0000","last_undegraded":"2024-04-25T16:00:19.337613+0000","last_fullsized":"2024-04-25T16:00:19.337613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:22.718842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.335498+0000","last_change":"2024-04-25T15:59:03.009240+0000","last_active":"2024-04-25T16:00:19.335498+0000","last_peered":"2024-04-25T16:00:19.335498+0000","last_clean":"2024-04-25T16:00:19.335498+0000","last_became_active":"2024-04-25T15:59:03.008723+0000","last_became_peered":"2024-04-25T15:59:03.008723+0000","last_unstale":"2024-04-25T16:00:19.335498+0000","last_undegraded":"2024-04-25T16:00:19.335498+0000","last_fullsized":"2024-04-25T16:00:19.335498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:14.049934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359019+0000","last_change":"2024-04-25T15:59:02.994359+0000","last_active":"2024-04-25T16:00:19.359019+0000","last_peered":"2024-04-25T16:00:19.359019+0000","last_clean":"2024-04-25T16:00:19.359019+0000","last_became_active":"2024-04-25T15:59:02.993990+0000","last_became_peered":"2024-04-25T15:59:02.993990+0000","last_unstale":"2024-04-25T16:00:19.359019+0000","last_undegraded":"2024-04-25T16:00:19.359019+0000","last_fullsized":"2024-04-25T16:00:19.359019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:55.477414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350058+0000","last_change":"2024-04-25T15:59:02.995510+0000","last_active":"2024-04-25T16:00:19.350058+0000","last_peered":"2024-04-25T16:00:19.350058+0000","last_clean":"2024-04-25T16:00:19.350058+0000","last_became_active":"2024-04-25T15:59:02.995008+0000","last_became_peered":"2024-04-25T15:59:02.995008+0000","last_unstale":"2024-04-25T16:00:19.350058+0000","last_undegraded":"2024-04-25T16:00:19.350058+0000","last_fullsized":"2024-04-25T16:00:19.350058+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:40.439615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341453+0000","last_change":"2024-04-25T15:59:03.005846+0000","last_active":"2024-04-25T16:00:19.341453+0000","last_peered":"2024-04-25T16:00:19.341453+0000","last_clean":"2024-04-25T16:00:19.341453+0000","last_became_active":"2024-04-25T15:59:03.003779+0000","last_became_peered":"2024-04-25T15:59:03.003779+0000","last_unstale":"2024-04-25T16:00:19.341453+0000","last_undegraded":"2024-04-25T16:00:19.341453+0000","last_fullsized":"2024-04-25T16:00:19.341453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.698405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247542,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64388,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":730,"num_read_kb":457,"num_write":242,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1192,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":193164,"data_compressed":5607,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710748,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":839680,"data_stored":402102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906979,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":1051075,"data_compressed":37695,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070504,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":402010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266734,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":851968,"data_stored":402366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462963,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":995047,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626488,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":401727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":461244,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":403066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":459124,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":401552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986243,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":402120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822717,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":994193,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60356,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58236,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":59702,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:00:26.966 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-25T16:00:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:27 smithi031 ceph-mon[29140]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T16:00:27.233 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:27.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:27 smithi186 ceph-mon[33258]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T16:00:27.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:27 smithi063 ceph-mon[32413]: pgmap v325: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T16:00:28.390 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:28 smithi031 ceph-mon[29140]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:00:28.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:28 smithi186 ceph-mon[33258]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:00:28.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:28 smithi063 ceph-mon[32413]: from='client.14955 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:00:28.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.967+0000 7fc7646c7700 1 -- 172.21.15.31:0/720748456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 msgr2=0x7fc75c103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:28.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.967+0000 7fc7646c7700 1 --2- 172.21.15.31:0/720748456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 0x7fc75c103770 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc7580099c0 tx=0x7fc758009cd0 comp rx=0 tx=0).stop 2024-04-25T16:00:28.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.967+0000 7fc7646c7700 1 -- 172.21.15.31:0/720748456 shutdown_connections 2024-04-25T16:00:28.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.967+0000 7fc7646c7700 1 --2- 172.21.15.31:0/720748456 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc75c1085e0 0x7fc75c10ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:28.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.967+0000 7fc7646c7700 1 --2- 172.21.15.31:0/720748456 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 0x7fc75c103770 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:28.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.967+0000 7fc7646c7700 1 --2- 172.21.15.31:0/720748456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc75c107c90 0x7fc75c1080a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:28.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.967+0000 7fc7646c7700 1 -- 172.21.15.31:0/720748456 >> 172.21.15.31:0/720748456 conn(0x7fc75c0fecb0 msgr2=0x7fc75c1010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:28.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.967+0000 7fc7646c7700 1 -- 172.21.15.31:0/720748456 shutdown_connections 2024-04-25T16:00:28.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.967+0000 7fc7646c7700 1 -- 172.21.15.31:0/720748456 wait complete. 2024-04-25T16:00:28.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.968+0000 7fc7646c7700 1 Processor -- start 2024-04-25T16:00:28.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.968+0000 7fc7646c7700 1 -- start start 2024-04-25T16:00:28.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc7646c7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 0x7fc75c19ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:28.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc7646c7700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc75c107c90 0x7fc75c19b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:28.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc7646c7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc75c1085e0 0x7fc75c19f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:28.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc7646c7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c114360 con 0x7fc75c103360 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc7646c7700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c1141e0 con 0x7fc75c107c90 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc7646c7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c1144e0 con 0x7fc75c1085e0 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762463700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 0x7fc75c19ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762463700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 0x7fc75c19ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53480/0 (socket says 172.21.15.31:53480) 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762463700 1 -- 172.21.15.31:0/1958004274 learned_addr learned my addr 172.21.15.31:0/1958004274 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762c64700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc75c1085e0 0x7fc75c19f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc761c62700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc75c107c90 0x7fc75c19b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762463700 1 -- 172.21.15.31:0/1958004274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc75c1085e0 msgr2=0x7fc75c19f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762463700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc75c1085e0 0x7fc75c19f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762463700 1 -- 172.21.15.31:0/1958004274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc75c107c90 msgr2=0x7fc75c19b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762463700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc75c107c90 0x7fc75c19b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762463700 1 -- 172.21.15.31:0/1958004274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75c19fc00 con 0x7fc75c103360 2024-04-25T16:00:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc761c62700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc75c107c90 0x7fc75c19b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc762463700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 0x7fc75c19ad30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc75800b580 tx=0x7fc7580047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc7537fe700 1 -- 172.21.15.31:0/1958004274 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758027070 con 0x7fc75c103360 2024-04-25T16:00:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.969+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75c19fe90 con 0x7fc75c103360 2024-04-25T16:00:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.970+0000 7fc7537fe700 1 -- 172.21.15.31:0/1958004274 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc758003f20 con 0x7fc75c103360 2024-04-25T16:00:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.970+0000 7fc7537fe700 1 -- 172.21.15.31:0/1958004274 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758022d70 con 0x7fc75c103360 2024-04-25T16:00:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.970+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75c1a7810 con 0x7fc75c103360 2024-04-25T16:00:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.971+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc75c104810 con 0x7fc75c103360 2024-04-25T16:00:28.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.973+0000 7fc7537fe700 1 -- 172.21.15.31:0/1958004274 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc758022500 con 0x7fc75c103360 2024-04-25T16:00:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.973+0000 7fc7537fe700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7480afb80 0x7fc7480b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.974+0000 7fc761c62700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7480afb80 0x7fc7480b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:28.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.974+0000 7fc7537fe700 1 -- 172.21.15.31:0/1958004274 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc7580dfd70 con 0x7fc75c103360 2024-04-25T16:00:28.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.974+0000 7fc761c62700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7480afb80 0x7fc7480b2090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc74c005950 tx=0x7fc74c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:28.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:28.977+0000 7fc7537fe700 1 -- 172.21.15.31:0/1958004274 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc758013070 con 0x7fc75c103360 2024-04-25T16:00:29.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.179+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc75c051190 con 0x7fc7480afb80 2024-04-25T16:00:29.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.187+0000 7fc7537fe700 1 -- 172.21.15.31:0/1958004274 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259146 (secure 0 0 0) 0x7fc75c051190 con 0x7fc7480afb80 2024-04-25T16:00:29.192 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7480afb80 msgr2=0x7fc7480b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7480afb80 0x7fc7480b2090 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc74c005950 tx=0x7fc74c0058e0 comp rx=0 tx=0).stop 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 msgr2=0x7fc75c19ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 0x7fc75c19ad30 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc75800b580 tx=0x7fc7580047b0 comp rx=0 tx=0).stop 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 shutdown_connections 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7480afb80 0x7fc7480b2090 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc75c1085e0 0x7fc75c19f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc75c103360 0x7fc75c19ad30 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 --2- 172.21.15.31:0/1958004274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc75c107c90 0x7fc75c19b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 >> 172.21.15.31:0/1958004274 conn(0x7fc75c0fecb0 msgr2=0x7fc75c101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 shutdown_connections 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:29.190+0000 7fc7646c7700 1 -- 172.21.15.31:0/1958004274 wait complete. 2024-04-25T16:00:29.194 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T16:00:29.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:29 smithi186 ceph-mon[33258]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:00:29.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:29 smithi031 ceph-mon[29140]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:00:29.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:29 smithi063 ceph-mon[32413]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:00:29.909 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-04-25T16:00:27.335515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654990,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":831,"num_read_kb":536,"num_write":367,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2953032,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":804,"ondisk_log_size":804,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355336,"kb_used_data":10344,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124586488,"statfs":{"total":1151940427776,"available":1151576563712,"internally_reserved":0,"allocated":10592256,"data_stored":6775626,"data_compressed":113085,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":127,"num_read_kb":67,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003554"},"pg_stats":[{"pgid":"3.3f","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354813+0000","last_change":"2024-04-25T15:59:02.997678+0000","last_active":"2024-04-25T16:00:19.354813+0000","last_peered":"2024-04-25T16:00:19.354813+0000","last_clean":"2024-04-25T16:00:19.354813+0000","last_became_active":"2024-04-25T15:59:02.997335+0000","last_became_peered":"2024-04-25T15:59:02.997335+0000","last_unstale":"2024-04-25T16:00:19.354813+0000","last_undegraded":"2024-04-25T16:00:19.354813+0000","last_fullsized":"2024-04-25T16:00:19.354813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:33.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349727+0000","last_change":"2024-04-25T15:59:03.001049+0000","last_active":"2024-04-25T16:00:19.349727+0000","last_peered":"2024-04-25T16:00:19.349727+0000","last_clean":"2024-04-25T16:00:19.349727+0000","last_became_active":"2024-04-25T15:59:03.000663+0000","last_became_peered":"2024-04-25T15:59:03.000663+0000","last_unstale":"2024-04-25T16:00:19.349727+0000","last_undegraded":"2024-04-25T16:00:19.349727+0000","last_fullsized":"2024-04-25T16:00:19.349727+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:04:36.670519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354156+0000","last_change":"2024-04-25T15:59:03.003861+0000","last_active":"2024-04-25T16:00:19.354156+0000","last_peered":"2024-04-25T16:00:19.354156+0000","last_clean":"2024-04-25T16:00:19.354156+0000","last_became_active":"2024-04-25T15:59:03.003508+0000","last_became_peered":"2024-04-25T15:59:03.003508+0000","last_unstale":"2024-04-25T16:00:19.354156+0000","last_undegraded":"2024-04-25T16:00:19.354156+0000","last_fullsized":"2024-04-25T16:00:19.354156+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:25.679389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337216+0000","last_change":"2024-04-25T15:59:03.003330+0000","last_active":"2024-04-25T16:00:19.337216+0000","last_peered":"2024-04-25T16:00:19.337216+0000","last_clean":"2024-04-25T16:00:19.337216+0000","last_became_active":"2024-04-25T15:59:03.002872+0000","last_became_peered":"2024-04-25T15:59:03.002872+0000","last_unstale":"2024-04-25T16:00:19.337216+0000","last_undegraded":"2024-04-25T16:00:19.337216+0000","last_fullsized":"2024-04-25T16:00:19.337216+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:02.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.324906+0000","last_change":"2024-04-25T15:59:02.997684+0000","last_active":"2024-04-25T16:00:19.324906+0000","last_peered":"2024-04-25T16:00:19.324906+0000","last_clean":"2024-04-25T16:00:19.324906+0000","last_became_active":"2024-04-25T15:59:02.997370+0000","last_became_peered":"2024-04-25T15:59:02.997370+0000","last_unstale":"2024-04-25T16:00:19.324906+0000","last_undegraded":"2024-04-25T16:00:19.324906+0000","last_fullsized":"2024-04-25T16:00:19.324906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:24.349085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348756+0000","last_change":"2024-04-25T15:59:03.003638+0000","last_active":"2024-04-25T16:00:19.348756+0000","last_peered":"2024-04-25T16:00:19.348756+0000","last_clean":"2024-04-25T16:00:19.348756+0000","last_became_active":"2024-04-25T15:59:03.003299+0000","last_became_peered":"2024-04-25T15:59:03.003299+0000","last_unstale":"2024-04-25T16:00:19.348756+0000","last_undegraded":"2024-04-25T16:00:19.348756+0000","last_fullsized":"2024-04-25T16:00:19.348756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:46.842345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.338851+0000","last_change":"2024-04-25T15:59:02.998591+0000","last_active":"2024-04-25T16:00:19.338851+0000","last_peered":"2024-04-25T16:00:19.338851+0000","last_clean":"2024-04-25T16:00:19.338851+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.338851+0000","last_undegraded":"2024-04-25T16:00:19.338851+0000","last_fullsized":"2024-04-25T16:00:19.338851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.444499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343142+0000","last_change":"2024-04-25T15:59:03.006225+0000","last_active":"2024-04-25T16:00:19.343142+0000","last_peered":"2024-04-25T16:00:19.343142+0000","last_clean":"2024-04-25T16:00:19.343142+0000","last_became_active":"2024-04-25T15:59:03.005883+0000","last_became_peered":"2024-04-25T15:59:03.005883+0000","last_unstale":"2024-04-25T16:00:19.343142+0000","last_undegraded":"2024-04-25T16:00:19.343142+0000","last_fullsized":"2024-04-25T16:00:19.343142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:17.773196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359702+0000","last_change":"2024-04-25T15:59:03.003605+0000","last_active":"2024-04-25T16:00:19.359702+0000","last_peered":"2024-04-25T16:00:19.359702+0000","last_clean":"2024-04-25T16:00:19.359702+0000","last_became_active":"2024-04-25T15:59:03.003271+0000","last_became_peered":"2024-04-25T15:59:03.003271+0000","last_unstale":"2024-04-25T16:00:19.359702+0000","last_undegraded":"2024-04-25T16:00:19.359702+0000","last_fullsized":"2024-04-25T16:00:19.359702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:28.576407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'16","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348846+0000","last_change":"2024-04-25T15:59:50.679099+0000","last_active":"2024-04-25T16:00:19.348846+0000","last_peered":"2024-04-25T16:00:19.348846+0000","last_clean":"2024-04-25T16:00:19.348846+0000","last_became_active":"2024-04-25T15:59:50.678558+0000","last_became_peered":"2024-04-25T15:59:50.678558+0000","last_unstale":"2024-04-25T16:00:19.348846+0000","last_undegraded":"2024-04-25T16:00:19.348846+0000","last_fullsized":"2024-04-25T16:00:19.348846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:40.691050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359322+0000","last_change":"2024-04-25T15:59:02.999307+0000","last_active":"2024-04-25T16:00:19.359322+0000","last_peered":"2024-04-25T16:00:19.359322+0000","last_clean":"2024-04-25T16:00:19.359322+0000","last_became_active":"2024-04-25T15:59:02.998976+0000","last_became_peered":"2024-04-25T15:59:02.998976+0000","last_unstale":"2024-04-25T16:00:19.359322+0000","last_undegraded":"2024-04-25T16:00:19.359322+0000","last_fullsized":"2024-04-25T16:00:19.359322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:29.626440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347700+0000","last_change":"2024-04-25T15:59:03.005208+0000","last_active":"2024-04-25T16:00:19.347700+0000","last_peered":"2024-04-25T16:00:19.347700+0000","last_clean":"2024-04-25T16:00:19.347700+0000","last_became_active":"2024-04-25T15:59:03.004869+0000","last_became_peered":"2024-04-25T15:59:03.004869+0000","last_unstale":"2024-04-25T16:00:19.347700+0000","last_undegraded":"2024-04-25T16:00:19.347700+0000","last_fullsized":"2024-04-25T16:00:19.347700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:00:24.040763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"77'10","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349448+0000","last_change":"2024-04-25T15:59:50.673556+0000","last_active":"2024-04-25T16:00:19.349448+0000","last_peered":"2024-04-25T16:00:19.349448+0000","last_clean":"2024-04-25T16:00:19.349448+0000","last_became_active":"2024-04-25T15:59:50.673119+0000","last_became_peered":"2024-04-25T15:59:50.673119+0000","last_unstale":"2024-04-25T16:00:19.349448+0000","last_undegraded":"2024-04-25T16:00:19.349448+0000","last_fullsized":"2024-04-25T16:00:19.349448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:09:50.286968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350359+0000","last_change":"2024-04-25T15:59:03.002160+0000","last_active":"2024-04-25T16:00:19.350359+0000","last_peered":"2024-04-25T16:00:19.350359+0000","last_clean":"2024-04-25T16:00:19.350359+0000","last_became_active":"2024-04-25T15:59:03.001855+0000","last_became_peered":"2024-04-25T15:59:03.001855+0000","last_unstale":"2024-04-25T16:00:19.350359+0000","last_undegraded":"2024-04-25T16:00:19.350359+0000","last_fullsized":"2024-04-25T16:00:19.350359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:43.383851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350513+0000","last_change":"2024-04-25T15:59:02.995318+0000","last_active":"2024-04-25T16:00:19.350513+0000","last_peered":"2024-04-25T16:00:19.350513+0000","last_clean":"2024-04-25T16:00:19.350513+0000","last_became_active":"2024-04-25T15:59:02.994956+0000","last_became_peered":"2024-04-25T15:59:02.994956+0000","last_unstale":"2024-04-25T16:00:19.350513+0000","last_undegraded":"2024-04-25T16:00:19.350513+0000","last_fullsized":"2024-04-25T16:00:19.350513+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:59.659400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359091+0000","last_change":"2024-04-25T15:59:02.996485+0000","last_active":"2024-04-25T16:00:19.359091+0000","last_peered":"2024-04-25T16:00:19.359091+0000","last_clean":"2024-04-25T16:00:19.359091+0000","last_became_active":"2024-04-25T15:59:02.996011+0000","last_became_peered":"2024-04-25T15:59:02.996011+0000","last_unstale":"2024-04-25T16:00:19.359091+0000","last_undegraded":"2024-04-25T16:00:19.359091+0000","last_fullsized":"2024-04-25T16:00:19.359091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:56.025254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.355086+0000","last_change":"2024-04-25T15:59:02.998455+0000","last_active":"2024-04-25T16:00:19.355086+0000","last_peered":"2024-04-25T16:00:19.355086+0000","last_clean":"2024-04-25T16:00:19.355086+0000","last_became_active":"2024-04-25T15:59:02.998126+0000","last_became_peered":"2024-04-25T15:59:02.998126+0000","last_unstale":"2024-04-25T16:00:19.355086+0000","last_undegraded":"2024-04-25T16:00:19.355086+0000","last_fullsized":"2024-04-25T16:00:19.355086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:20.342413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350929+0000","last_change":"2024-04-25T15:59:02.998766+0000","last_active":"2024-04-25T16:00:19.350929+0000","last_peered":"2024-04-25T16:00:19.350929+0000","last_clean":"2024-04-25T16:00:19.350929+0000","last_became_active":"2024-04-25T15:59:02.998415+0000","last_became_peered":"2024-04-25T15:59:02.998415+0000","last_unstale":"2024-04-25T16:00:19.350929+0000","last_undegraded":"2024-04-25T16:00:19.350929+0000","last_fullsized":"2024-04-25T16:00:19.350929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:55:05.717847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340892+0000","last_change":"2024-04-25T15:59:02.994982+0000","last_active":"2024-04-25T16:00:19.340892+0000","last_peered":"2024-04-25T16:00:19.340892+0000","last_clean":"2024-04-25T16:00:19.340892+0000","last_became_active":"2024-04-25T15:59:02.993928+0000","last_became_peered":"2024-04-25T15:59:02.993928+0000","last_unstale":"2024-04-25T16:00:19.340892+0000","last_undegraded":"2024-04-25T16:00:19.340892+0000","last_fullsized":"2024-04-25T16:00:19.340892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:22:25.346541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'13","reported_seq":107,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:22.382837+0000","last_change":"2024-04-25T15:59:02.512828+0000","last_active":"2024-04-25T16:00:22.382837+0000","last_peered":"2024-04-25T16:00:22.382837+0000","last_clean":"2024-04-25T16:00:22.382837+0000","last_became_active":"2024-04-25T15:59:02.512291+0000","last_became_peered":"2024-04-25T15:59:02.512291+0000","last_unstale":"2024-04-25T16:00:22.382837+0000","last_undegraded":"2024-04-25T16:00:22.382837+0000","last_fullsized":"2024-04-25T16:00:22.382837+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:02.372140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":75,"num_read_kb":75,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.339895+0000","last_change":"2024-04-25T15:59:03.008472+0000","last_active":"2024-04-25T16:00:19.339895+0000","last_peered":"2024-04-25T16:00:19.339895+0000","last_clean":"2024-04-25T16:00:19.339895+0000","last_became_active":"2024-04-25T15:59:03.008108+0000","last_became_peered":"2024-04-25T15:59:03.008108+0000","last_unstale":"2024-04-25T16:00:19.339895+0000","last_undegraded":"2024-04-25T16:00:19.339895+0000","last_fullsized":"2024-04-25T16:00:19.339895+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:40.087592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'2","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807892+0000","last_change":"2024-04-25T15:59:02.517846+0000","last_active":"2024-04-25T15:59:50.807892+0000","last_peered":"2024-04-25T15:59:50.807892+0000","last_clean":"2024-04-25T15:59:50.807892+0000","last_became_active":"2024-04-25T15:59:02.517366+0000","last_became_peered":"2024-04-25T15:59:02.517366+0000","last_unstale":"2024-04-25T15:59:50.807892+0000","last_undegraded":"2024-04-25T15:59:50.807892+0000","last_fullsized":"2024-04-25T15:59:50.807892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:29.620827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354150+0000","last_change":"2024-04-25T15:59:03.005202+0000","last_active":"2024-04-25T16:00:19.354150+0000","last_peered":"2024-04-25T16:00:19.354150+0000","last_clean":"2024-04-25T16:00:19.354150+0000","last_became_active":"2024-04-25T15:59:03.003447+0000","last_became_peered":"2024-04-25T15:59:03.003447+0000","last_unstale":"2024-04-25T16:00:19.354150+0000","last_undegraded":"2024-04-25T16:00:19.354150+0000","last_fullsized":"2024-04-25T16:00:19.354150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.740016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'12","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.703214+0000","last_change":"2024-04-25T15:59:50.702764+0000","last_active":"2024-04-25T15:59:50.703214+0000","last_peered":"2024-04-25T15:59:50.703214+0000","last_clean":"2024-04-25T15:59:50.703214+0000","last_became_active":"2024-04-25T15:59:50.679231+0000","last_became_peered":"2024-04-25T15:59:50.679231+0000","last_unstale":"2024-04-25T15:59:50.703214+0000","last_undegraded":"2024-04-25T15:59:50.703214+0000","last_fullsized":"2024-04-25T15:59:50.703214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:37.561992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360223+0000","last_change":"2024-04-25T15:59:03.004210+0000","last_active":"2024-04-25T16:00:19.360223+0000","last_peered":"2024-04-25T16:00:19.360223+0000","last_clean":"2024-04-25T16:00:19.360223+0000","last_became_active":"2024-04-25T15:59:03.002769+0000","last_became_peered":"2024-04-25T15:59:03.002769+0000","last_unstale":"2024-04-25T16:00:19.360223+0000","last_undegraded":"2024-04-25T16:00:19.360223+0000","last_fullsized":"2024-04-25T16:00:19.360223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:50.608327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669960+0000","last_change":"2024-04-25T15:59:02.506199+0000","last_active":"2024-04-25T15:59:50.669960+0000","last_peered":"2024-04-25T15:59:50.669960+0000","last_clean":"2024-04-25T15:59:50.669960+0000","last_became_active":"2024-04-25T15:59:02.505847+0000","last_became_peered":"2024-04-25T15:59:02.505847+0000","last_unstale":"2024-04-25T15:59:50.669960+0000","last_undegraded":"2024-04-25T15:59:50.669960+0000","last_fullsized":"2024-04-25T15:59:50.669960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:35.227112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"77'15","reported_seq":38,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359258+0000","last_change":"2024-04-25T15:59:50.679040+0000","last_active":"2024-04-25T16:00:19.359258+0000","last_peered":"2024-04-25T16:00:19.359258+0000","last_clean":"2024-04-25T16:00:19.359258+0000","last_became_active":"2024-04-25T15:59:50.678094+0000","last_became_peered":"2024-04-25T15:59:50.678094+0000","last_unstale":"2024-04-25T16:00:19.359258+0000","last_undegraded":"2024-04-25T16:00:19.359258+0000","last_fullsized":"2024-04-25T16:00:19.359258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:11.957966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'5","reported_seq":100,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:21.374575+0000","last_change":"2024-04-25T15:59:02.507763+0000","last_active":"2024-04-25T16:00:21.374575+0000","last_peered":"2024-04-25T16:00:21.374575+0000","last_clean":"2024-04-25T16:00:21.374575+0000","last_became_active":"2024-04-25T15:59:02.507338+0000","last_became_peered":"2024-04-25T15:59:02.507338+0000","last_unstale":"2024-04-25T16:00:21.374575+0000","last_undegraded":"2024-04-25T16:00:21.374575+0000","last_fullsized":"2024-04-25T16:00:21.374575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:31.710709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.326104+0000","last_change":"2024-04-25T15:59:03.002590+0000","last_active":"2024-04-25T16:00:19.326104+0000","last_peered":"2024-04-25T16:00:19.326104+0000","last_clean":"2024-04-25T16:00:19.326104+0000","last_became_active":"2024-04-25T15:59:03.002262+0000","last_became_peered":"2024-04-25T15:59:03.002262+0000","last_unstale":"2024-04-25T16:00:19.326104+0000","last_undegraded":"2024-04-25T16:00:19.326104+0000","last_fullsized":"2024-04-25T16:00:19.326104+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:48.251484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671899+0000","last_change":"2024-04-25T15:59:02.518853+0000","last_active":"2024-04-25T15:59:50.671899+0000","last_peered":"2024-04-25T15:59:50.671899+0000","last_clean":"2024-04-25T15:59:50.671899+0000","last_became_active":"2024-04-25T15:59:02.518505+0000","last_became_peered":"2024-04-25T15:59:02.518505+0000","last_unstale":"2024-04-25T15:59:50.671899+0000","last_undegraded":"2024-04-25T15:59:50.671899+0000","last_fullsized":"2024-04-25T15:59:50.671899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:01:56.282245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345600+0000","last_change":"2024-04-25T15:59:03.009733+0000","last_active":"2024-04-25T16:00:19.345600+0000","last_peered":"2024-04-25T16:00:19.345600+0000","last_clean":"2024-04-25T16:00:19.345600+0000","last_became_active":"2024-04-25T15:59:03.009251+0000","last_became_peered":"2024-04-25T15:59:03.009251+0000","last_unstale":"2024-04-25T16:00:19.345600+0000","last_undegraded":"2024-04-25T16:00:19.345600+0000","last_fullsized":"2024-04-25T16:00:19.345600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:22.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"75'3","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670602+0000","last_change":"2024-04-25T15:59:02.512493+0000","last_active":"2024-04-25T15:59:50.670602+0000","last_peered":"2024-04-25T15:59:50.670602+0000","last_clean":"2024-04-25T15:59:50.670602+0000","last_became_active":"2024-04-25T15:59:02.512103+0000","last_became_peered":"2024-04-25T15:59:02.512103+0000","last_unstale":"2024-04-25T15:59:50.670602+0000","last_undegraded":"2024-04-25T15:59:50.670602+0000","last_fullsized":"2024-04-25T15:59:50.670602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:29.457755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342733+0000","last_change":"2024-04-25T15:59:03.001406+0000","last_active":"2024-04-25T16:00:19.342733+0000","last_peered":"2024-04-25T16:00:19.342733+0000","last_clean":"2024-04-25T16:00:19.342733+0000","last_became_active":"2024-04-25T15:59:02.999038+0000","last_became_peered":"2024-04-25T15:59:02.999038+0000","last_unstale":"2024-04-25T16:00:19.342733+0000","last_undegraded":"2024-04-25T16:00:19.342733+0000","last_fullsized":"2024-04-25T16:00:19.342733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:56.354821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671531+0000","last_change":"2024-04-25T15:59:02.509619+0000","last_active":"2024-04-25T15:59:50.671531+0000","last_peered":"2024-04-25T15:59:50.671531+0000","last_clean":"2024-04-25T15:59:50.671531+0000","last_became_active":"2024-04-25T15:59:02.509254+0000","last_became_peered":"2024-04-25T15:59:02.509254+0000","last_unstale":"2024-04-25T15:59:50.671531+0000","last_undegraded":"2024-04-25T15:59:50.671531+0000","last_fullsized":"2024-04-25T15:59:50.671531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:00.382581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347276+0000","last_change":"2024-04-25T15:59:03.000009+0000","last_active":"2024-04-25T16:00:19.347276+0000","last_peered":"2024-04-25T16:00:19.347276+0000","last_clean":"2024-04-25T16:00:19.347276+0000","last_became_active":"2024-04-25T15:59:02.999662+0000","last_became_peered":"2024-04-25T15:59:02.999662+0000","last_unstale":"2024-04-25T16:00:19.347276+0000","last_undegraded":"2024-04-25T16:00:19.347276+0000","last_fullsized":"2024-04-25T16:00:19.347276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:05:38.004819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"75'4","reported_seq":179,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:26.382476+0000","last_change":"2024-04-25T15:59:02.514429+0000","last_active":"2024-04-25T16:00:26.382476+0000","last_peered":"2024-04-25T16:00:26.382476+0000","last_clean":"2024-04-25T16:00:26.382476+0000","last_became_active":"2024-04-25T15:59:02.514144+0000","last_became_peered":"2024-04-25T15:59:02.514144+0000","last_unstale":"2024-04-25T16:00:26.382476+0000","last_undegraded":"2024-04-25T16:00:26.382476+0000","last_fullsized":"2024-04-25T16:00:26.382476+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:45.144630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":156,"num_read_kb":156,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360393+0000","last_change":"2024-04-25T15:59:03.008460+0000","last_active":"2024-04-25T16:00:19.360393+0000","last_peered":"2024-04-25T16:00:19.360393+0000","last_clean":"2024-04-25T16:00:19.360393+0000","last_became_active":"2024-04-25T15:59:03.008024+0000","last_became_peered":"2024-04-25T15:59:03.008024+0000","last_unstale":"2024-04-25T16:00:19.360393+0000","last_undegraded":"2024-04-25T16:00:19.360393+0000","last_fullsized":"2024-04-25T16:00:19.360393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:10.288760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"75'4","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676828+0000","last_change":"2024-04-25T15:59:02.512494+0000","last_active":"2024-04-25T15:59:50.676828+0000","last_peered":"2024-04-25T15:59:50.676828+0000","last_clean":"2024-04-25T15:59:50.676828+0000","last_became_active":"2024-04-25T15:59:02.511182+0000","last_became_peered":"2024-04-25T15:59:02.511182+0000","last_unstale":"2024-04-25T15:59:50.676828+0000","last_undegraded":"2024-04-25T15:59:50.676828+0000","last_fullsized":"2024-04-25T15:59:50.676828+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:54.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350559+0000","last_change":"2024-04-25T15:59:03.003848+0000","last_active":"2024-04-25T16:00:19.350559+0000","last_peered":"2024-04-25T16:00:19.350559+0000","last_clean":"2024-04-25T16:00:19.350559+0000","last_became_active":"2024-04-25T15:59:03.003512+0000","last_became_peered":"2024-04-25T15:59:03.003512+0000","last_unstale":"2024-04-25T16:00:19.350559+0000","last_undegraded":"2024-04-25T16:00:19.350559+0000","last_fullsized":"2024-04-25T16:00:19.350559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:54.667054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672694+0000","last_change":"2024-04-25T15:59:02.509343+0000","last_active":"2024-04-25T15:59:50.672694+0000","last_peered":"2024-04-25T15:59:50.672694+0000","last_clean":"2024-04-25T15:59:50.672694+0000","last_became_active":"2024-04-25T15:59:02.508980+0000","last_became_peered":"2024-04-25T15:59:02.508980+0000","last_unstale":"2024-04-25T15:59:50.672694+0000","last_undegraded":"2024-04-25T15:59:50.672694+0000","last_fullsized":"2024-04-25T15:59:50.672694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:54.344717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359525+0000","last_change":"2024-04-25T15:59:02.999862+0000","last_active":"2024-04-25T16:00:19.359525+0000","last_peered":"2024-04-25T16:00:19.359525+0000","last_clean":"2024-04-25T16:00:19.359525+0000","last_became_active":"2024-04-25T15:59:02.999472+0000","last_became_peered":"2024-04-25T15:59:02.999472+0000","last_unstale":"2024-04-25T16:00:19.359525+0000","last_undegraded":"2024-04-25T16:00:19.359525+0000","last_fullsized":"2024-04-25T16:00:19.359525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:46.165045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"77'32","reported_seq":215,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:26.384688+0000","last_change":"2024-04-25T15:59:02.508949+0000","last_active":"2024-04-25T16:00:26.384688+0000","last_peered":"2024-04-25T16:00:26.384688+0000","last_clean":"2024-04-25T16:00:26.384688+0000","last_became_active":"2024-04-25T15:59:02.508564+0000","last_became_peered":"2024-04-25T15:59:02.508564+0000","last_unstale":"2024-04-25T16:00:26.384688+0000","last_undegraded":"2024-04-25T16:00:26.384688+0000","last_fullsized":"2024-04-25T16:00:26.384688+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:12.590193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57434,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":164,"num_read_kb":139,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344171+0000","last_change":"2024-04-25T15:59:02.998575+0000","last_active":"2024-04-25T16:00:19.344171+0000","last_peered":"2024-04-25T16:00:19.344171+0000","last_clean":"2024-04-25T16:00:19.344171+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.344171+0000","last_undegraded":"2024-04-25T16:00:19.344171+0000","last_fullsized":"2024-04-25T16:00:19.344171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:40.640468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'9","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677368+0000","last_change":"2024-04-25T15:59:02.519110+0000","last_active":"2024-04-25T15:59:50.677368+0000","last_peered":"2024-04-25T15:59:50.677368+0000","last_clean":"2024-04-25T15:59:50.677368+0000","last_became_active":"2024-04-25T15:59:02.518373+0000","last_became_peered":"2024-04-25T15:59:02.518373+0000","last_unstale":"2024-04-25T15:59:50.677368+0000","last_undegraded":"2024-04-25T15:59:50.677368+0000","last_fullsized":"2024-04-25T15:59:50.677368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:27.840296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344110+0000","last_change":"2024-04-25T15:59:02.991002+0000","last_active":"2024-04-25T16:00:19.344110+0000","last_peered":"2024-04-25T16:00:19.344110+0000","last_clean":"2024-04-25T16:00:19.344110+0000","last_became_active":"2024-04-25T15:59:02.990605+0000","last_became_peered":"2024-04-25T15:59:02.990605+0000","last_unstale":"2024-04-25T16:00:19.344110+0000","last_undegraded":"2024-04-25T16:00:19.344110+0000","last_fullsized":"2024-04-25T16:00:19.344110+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:22:00.723255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671723+0000","last_change":"2024-04-25T15:59:02.509809+0000","last_active":"2024-04-25T15:59:50.671723+0000","last_peered":"2024-04-25T15:59:50.671723+0000","last_clean":"2024-04-25T15:59:50.671723+0000","last_became_active":"2024-04-25T15:59:02.509370+0000","last_became_peered":"2024-04-25T15:59:02.509370+0000","last_unstale":"2024-04-25T15:59:50.671723+0000","last_undegraded":"2024-04-25T15:59:50.671723+0000","last_fullsized":"2024-04-25T15:59:50.671723+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:24.416072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343672+0000","last_change":"2024-04-25T15:59:02.999836+0000","last_active":"2024-04-25T16:00:19.343672+0000","last_peered":"2024-04-25T16:00:19.343672+0000","last_clean":"2024-04-25T16:00:19.343672+0000","last_became_active":"2024-04-25T15:59:02.997770+0000","last_became_peered":"2024-04-25T15:59:02.997770+0000","last_unstale":"2024-04-25T16:00:19.343672+0000","last_undegraded":"2024-04-25T16:00:19.343672+0000","last_fullsized":"2024-04-25T16:00:19.343672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:30.953339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"77'7","reported_seq":103,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:21.386288+0000","last_change":"2024-04-25T15:59:02.510465+0000","last_active":"2024-04-25T16:00:21.386288+0000","last_peered":"2024-04-25T16:00:21.386288+0000","last_clean":"2024-04-25T16:00:21.386288+0000","last_became_active":"2024-04-25T15:59:02.510067+0000","last_became_peered":"2024-04-25T15:59:02.510067+0000","last_unstale":"2024-04-25T16:00:21.386288+0000","last_undegraded":"2024-04-25T16:00:21.386288+0000","last_fullsized":"2024-04-25T16:00:21.386288+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:40.739358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346234+0000","last_change":"2024-04-25T15:59:03.000028+0000","last_active":"2024-04-25T16:00:19.346234+0000","last_peered":"2024-04-25T16:00:19.346234+0000","last_clean":"2024-04-25T16:00:19.346234+0000","last_became_active":"2024-04-25T15:59:02.999704+0000","last_became_peered":"2024-04-25T15:59:02.999704+0000","last_unstale":"2024-04-25T16:00:19.346234+0000","last_undegraded":"2024-04-25T16:00:19.346234+0000","last_fullsized":"2024-04-25T16:00:19.346234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:55.015085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'17","reported_seq":37,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669662+0000","last_change":"2024-04-25T15:59:02.514442+0000","last_active":"2024-04-25T15:59:50.669662+0000","last_peered":"2024-04-25T15:59:50.669662+0000","last_clean":"2024-04-25T15:59:50.669662+0000","last_became_active":"2024-04-25T15:59:02.513951+0000","last_became_peered":"2024-04-25T15:59:02.513951+0000","last_unstale":"2024-04-25T15:59:50.669662+0000","last_undegraded":"2024-04-25T15:59:50.669662+0000","last_fullsized":"2024-04-25T15:59:50.669662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:56.443999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"77'6","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.358196+0000","last_change":"2024-04-25T15:59:50.677155+0000","last_active":"2024-04-25T16:00:19.358196+0000","last_peered":"2024-04-25T16:00:19.358196+0000","last_clean":"2024-04-25T16:00:19.358196+0000","last_became_active":"2024-04-25T15:59:50.676721+0000","last_became_peered":"2024-04-25T15:59:50.676721+0000","last_unstale":"2024-04-25T16:00:19.358196+0000","last_undegraded":"2024-04-25T16:00:19.358196+0000","last_fullsized":"2024-04-25T16:00:19.358196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:04.138269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.683753+0000","last_change":"2024-04-25T15:59:50.683228+0000","last_active":"2024-04-25T15:59:50.683753+0000","last_peered":"2024-04-25T15:59:50.683753+0000","last_clean":"2024-04-25T15:59:50.683753+0000","last_became_active":"2024-04-25T15:59:50.675414+0000","last_became_peered":"2024-04-25T15:59:50.675414+0000","last_unstale":"2024-04-25T15:59:50.683753+0000","last_undegraded":"2024-04-25T15:59:50.683753+0000","last_fullsized":"2024-04-25T15:59:50.683753+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:46.577883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341606+0000","last_change":"2024-04-25T15:59:03.004639+0000","last_active":"2024-04-25T16:00:19.341606+0000","last_peered":"2024-04-25T16:00:19.341606+0000","last_clean":"2024-04-25T16:00:19.341606+0000","last_became_active":"2024-04-25T15:59:03.004260+0000","last_became_peered":"2024-04-25T15:59:03.004260+0000","last_unstale":"2024-04-25T16:00:19.341606+0000","last_undegraded":"2024-04-25T16:00:19.341606+0000","last_fullsized":"2024-04-25T16:00:19.341606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:00.450758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":20,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666336+0000","last_change":"2024-04-25T15:59:02.509716+0000","last_active":"2024-04-25T15:59:50.666336+0000","last_peered":"2024-04-25T15:59:50.666336+0000","last_clean":"2024-04-25T15:59:50.666336+0000","last_became_active":"2024-04-25T15:59:02.509303+0000","last_became_peered":"2024-04-25T15:59:02.509303+0000","last_unstale":"2024-04-25T15:59:50.666336+0000","last_undegraded":"2024-04-25T15:59:50.666336+0000","last_fullsized":"2024-04-25T15:59:50.666336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:34.037590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349639+0000","last_change":"2024-04-25T15:59:03.005141+0000","last_active":"2024-04-25T16:00:19.349639+0000","last_peered":"2024-04-25T16:00:19.349639+0000","last_clean":"2024-04-25T16:00:19.349639+0000","last_became_active":"2024-04-25T15:59:03.004760+0000","last_became_peered":"2024-04-25T15:59:03.004760+0000","last_unstale":"2024-04-25T16:00:19.349639+0000","last_undegraded":"2024-04-25T16:00:19.349639+0000","last_fullsized":"2024-04-25T16:00:19.349639+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:14.923126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"75'5","reported_seq":102,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:25.387529+0000","last_change":"2024-04-25T15:59:02.512779+0000","last_active":"2024-04-25T16:00:25.387529+0000","last_peered":"2024-04-25T16:00:25.387529+0000","last_clean":"2024-04-25T16:00:25.387529+0000","last_became_active":"2024-04-25T15:59:02.511240+0000","last_became_peered":"2024-04-25T15:59:02.511240+0000","last_unstale":"2024-04-25T16:00:25.387529+0000","last_undegraded":"2024-04-25T16:00:25.387529+0000","last_fullsized":"2024-04-25T16:00:25.387529+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:51.703751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351726+0000","last_change":"2024-04-25T15:59:03.004841+0000","last_active":"2024-04-25T16:00:19.351726+0000","last_peered":"2024-04-25T16:00:19.351726+0000","last_clean":"2024-04-25T16:00:19.351726+0000","last_became_active":"2024-04-25T15:59:03.004488+0000","last_became_peered":"2024-04-25T15:59:03.004488+0000","last_unstale":"2024-04-25T16:00:19.351726+0000","last_undegraded":"2024-04-25T16:00:19.351726+0000","last_fullsized":"2024-04-25T16:00:19.351726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:07.672312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345631+0000","last_change":"2024-04-25T15:59:03.001296+0000","last_active":"2024-04-25T16:00:19.345631+0000","last_peered":"2024-04-25T16:00:19.345631+0000","last_clean":"2024-04-25T16:00:19.345631+0000","last_became_active":"2024-04-25T15:59:03.000568+0000","last_became_peered":"2024-04-25T15:59:03.000568+0000","last_unstale":"2024-04-25T16:00:19.345631+0000","last_undegraded":"2024-04-25T16:00:19.345631+0000","last_fullsized":"2024-04-25T16:00:19.345631+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:51.335849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"77'6","reported_seq":101,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:22.387281+0000","last_change":"2024-04-25T15:59:02.513943+0000","last_active":"2024-04-25T16:00:22.387281+0000","last_peered":"2024-04-25T16:00:22.387281+0000","last_clean":"2024-04-25T16:00:22.387281+0000","last_became_active":"2024-04-25T15:59:02.513595+0000","last_became_peered":"2024-04-25T15:59:02.513595+0000","last_unstale":"2024-04-25T16:00:22.387281+0000","last_undegraded":"2024-04-25T16:00:22.387281+0000","last_fullsized":"2024-04-25T16:00:22.387281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:50.720333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.352846+0000","last_change":"2024-04-25T15:59:03.008561+0000","last_active":"2024-04-25T16:00:19.352846+0000","last_peered":"2024-04-25T16:00:19.352846+0000","last_clean":"2024-04-25T16:00:19.352846+0000","last_became_active":"2024-04-25T15:59:03.008134+0000","last_became_peered":"2024-04-25T15:59:03.008134+0000","last_unstale":"2024-04-25T16:00:19.352846+0000","last_undegraded":"2024-04-25T16:00:19.352846+0000","last_fullsized":"2024-04-25T16:00:19.352846+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:43.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"75'6","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669568+0000","last_change":"2024-04-25T15:59:02.510731+0000","last_active":"2024-04-25T15:59:50.669568+0000","last_peered":"2024-04-25T15:59:50.669568+0000","last_clean":"2024-04-25T15:59:50.669568+0000","last_became_active":"2024-04-25T15:59:02.510356+0000","last_became_peered":"2024-04-25T15:59:02.510356+0000","last_unstale":"2024-04-25T15:59:50.669568+0000","last_undegraded":"2024-04-25T15:59:50.669568+0000","last_fullsized":"2024-04-25T15:59:50.669568+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:33.066858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":94,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676873+0000","last_change":"2024-04-25T15:57:47.988899+0000","last_active":"2024-04-25T15:59:50.676873+0000","last_peered":"2024-04-25T15:59:50.676873+0000","last_clean":"2024-04-25T15:59:50.676873+0000","last_became_active":"2024-04-25T15:57:47.982613+0000","last_became_peered":"2024-04-25T15:57:47.982613+0000","last_unstale":"2024-04-25T15:59:50.676873+0000","last_undegraded":"2024-04-25T15:59:50.676873+0000","last_fullsized":"2024-04-25T15:59:50.676873+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_clean_scrub_stamp":"2024-04-25T15:54:18.013489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:39.323767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.334020+0000","last_change":"2024-04-25T15:59:02.998360+0000","last_active":"2024-04-25T16:00:19.334020+0000","last_peered":"2024-04-25T16:00:19.334020+0000","last_clean":"2024-04-25T16:00:19.334020+0000","last_became_active":"2024-04-25T15:59:02.998048+0000","last_became_peered":"2024-04-25T15:59:02.998048+0000","last_unstale":"2024-04-25T16:00:19.334020+0000","last_undegraded":"2024-04-25T16:00:19.334020+0000","last_fullsized":"2024-04-25T16:00:19.334020+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:01.524608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"75'4","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677261+0000","last_change":"2024-04-25T15:59:02.515769+0000","last_active":"2024-04-25T15:59:50.677261+0000","last_peered":"2024-04-25T15:59:50.677261+0000","last_clean":"2024-04-25T15:59:50.677261+0000","last_became_active":"2024-04-25T15:59:02.515467+0000","last_became_peered":"2024-04-25T15:59:02.515467+0000","last_unstale":"2024-04-25T15:59:50.677261+0000","last_undegraded":"2024-04-25T15:59:50.677261+0000","last_fullsized":"2024-04-25T15:59:50.677261+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:59:50.107475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670090+0000","last_change":"2024-04-25T15:59:02.512701+0000","last_active":"2024-04-25T15:59:50.670090+0000","last_peered":"2024-04-25T15:59:50.670090+0000","last_clean":"2024-04-25T15:59:50.670090+0000","last_became_active":"2024-04-25T15:59:02.512352+0000","last_became_peered":"2024-04-25T15:59:02.512352+0000","last_unstale":"2024-04-25T15:59:50.670090+0000","last_undegraded":"2024-04-25T15:59:50.670090+0000","last_fullsized":"2024-04-25T15:59:50.670090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:31.772382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.329902+0000","last_change":"2024-04-25T15:59:02.989698+0000","last_active":"2024-04-25T16:00:19.329902+0000","last_peered":"2024-04-25T16:00:19.329902+0000","last_clean":"2024-04-25T16:00:19.329902+0000","last_became_active":"2024-04-25T15:59:02.989284+0000","last_became_peered":"2024-04-25T15:59:02.989284+0000","last_unstale":"2024-04-25T16:00:19.329902+0000","last_undegraded":"2024-04-25T16:00:19.329902+0000","last_fullsized":"2024-04-25T16:00:19.329902+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:12.501738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"77'13","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666592+0000","last_change":"2024-04-25T15:59:02.517272+0000","last_active":"2024-04-25T15:59:50.666592+0000","last_peered":"2024-04-25T15:59:50.666592+0000","last_clean":"2024-04-25T15:59:50.666592+0000","last_became_active":"2024-04-25T15:59:02.516370+0000","last_became_peered":"2024-04-25T15:59:02.516370+0000","last_unstale":"2024-04-25T15:59:50.666592+0000","last_undegraded":"2024-04-25T15:59:50.666592+0000","last_fullsized":"2024-04-25T15:59:50.666592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:31.139689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345612+0000","last_change":"2024-04-25T15:59:03.003111+0000","last_active":"2024-04-25T16:00:19.345612+0000","last_peered":"2024-04-25T16:00:19.345612+0000","last_clean":"2024-04-25T16:00:19.345612+0000","last_became_active":"2024-04-25T15:59:03.002766+0000","last_became_peered":"2024-04-25T15:59:03.002766+0000","last_unstale":"2024-04-25T16:00:19.345612+0000","last_undegraded":"2024-04-25T16:00:19.345612+0000","last_fullsized":"2024-04-25T16:00:19.345612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:39.866119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'7","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670151+0000","last_change":"2024-04-25T15:59:02.521970+0000","last_active":"2024-04-25T15:59:50.670151+0000","last_peered":"2024-04-25T15:59:50.670151+0000","last_clean":"2024-04-25T15:59:50.670151+0000","last_became_active":"2024-04-25T15:59:02.521650+0000","last_became_peered":"2024-04-25T15:59:02.521650+0000","last_unstale":"2024-04-25T15:59:50.670151+0000","last_undegraded":"2024-04-25T15:59:50.670151+0000","last_fullsized":"2024-04-25T15:59:50.670151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:32.996868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351310+0000","last_change":"2024-04-25T15:59:02.996358+0000","last_active":"2024-04-25T16:00:19.351310+0000","last_peered":"2024-04-25T16:00:19.351310+0000","last_clean":"2024-04-25T16:00:19.351310+0000","last_became_active":"2024-04-25T15:59:02.995988+0000","last_became_peered":"2024-04-25T15:59:02.995988+0000","last_unstale":"2024-04-25T16:00:19.351310+0000","last_undegraded":"2024-04-25T16:00:19.351310+0000","last_fullsized":"2024-04-25T16:00:19.351310+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:51.021841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"75'2","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.674464+0000","last_change":"2024-04-25T15:59:02.518330+0000","last_active":"2024-04-25T15:59:50.674464+0000","last_peered":"2024-04-25T15:59:50.674464+0000","last_clean":"2024-04-25T15:59:50.674464+0000","last_became_active":"2024-04-25T15:59:02.517978+0000","last_became_peered":"2024-04-25T15:59:02.517978+0000","last_unstale":"2024-04-25T15:59:50.674464+0000","last_undegraded":"2024-04-25T15:59:50.674464+0000","last_fullsized":"2024-04-25T15:59:50.674464+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:25.443963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.302105+0000","last_change":"2024-04-25T15:59:03.000198+0000","last_active":"2024-04-25T16:00:19.302105+0000","last_peered":"2024-04-25T16:00:19.302105+0000","last_clean":"2024-04-25T16:00:19.302105+0000","last_became_active":"2024-04-25T15:59:02.999878+0000","last_became_peered":"2024-04-25T15:59:02.999878+0000","last_unstale":"2024-04-25T16:00:19.302105+0000","last_undegraded":"2024-04-25T16:00:19.302105+0000","last_fullsized":"2024-04-25T16:00:19.302105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:37.966595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807467+0000","last_change":"2024-04-25T15:59:02.511298+0000","last_active":"2024-04-25T15:59:50.807467+0000","last_peered":"2024-04-25T15:59:50.807467+0000","last_clean":"2024-04-25T15:59:50.807467+0000","last_became_active":"2024-04-25T15:59:02.510521+0000","last_became_peered":"2024-04-25T15:59:02.510521+0000","last_unstale":"2024-04-25T15:59:50.807467+0000","last_undegraded":"2024-04-25T15:59:50.807467+0000","last_fullsized":"2024-04-25T15:59:50.807467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:00.807919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347593+0000","last_change":"2024-04-25T15:59:02.999212+0000","last_active":"2024-04-25T16:00:19.347593+0000","last_peered":"2024-04-25T16:00:19.347593+0000","last_clean":"2024-04-25T16:00:19.347593+0000","last_became_active":"2024-04-25T15:59:02.998881+0000","last_became_peered":"2024-04-25T15:59:02.998881+0000","last_unstale":"2024-04-25T16:00:19.347593+0000","last_undegraded":"2024-04-25T16:00:19.347593+0000","last_fullsized":"2024-04-25T16:00:19.347593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:09.092196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"75'3","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676880+0000","last_change":"2024-04-25T15:59:02.517412+0000","last_active":"2024-04-25T15:59:50.676880+0000","last_peered":"2024-04-25T15:59:50.676880+0000","last_clean":"2024-04-25T15:59:50.676880+0000","last_became_active":"2024-04-25T15:59:02.517077+0000","last_became_peered":"2024-04-25T15:59:02.517077+0000","last_unstale":"2024-04-25T15:59:50.676880+0000","last_undegraded":"2024-04-25T15:59:50.676880+0000","last_fullsized":"2024-04-25T15:59:50.676880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:00.161199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"77'8","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342365+0000","last_change":"2024-04-25T15:59:50.678844+0000","last_active":"2024-04-25T16:00:19.342365+0000","last_peered":"2024-04-25T16:00:19.342365+0000","last_clean":"2024-04-25T16:00:19.342365+0000","last_became_active":"2024-04-25T15:59:50.678411+0000","last_became_peered":"2024-04-25T15:59:50.678411+0000","last_unstale":"2024-04-25T16:00:19.342365+0000","last_undegraded":"2024-04-25T16:00:19.342365+0000","last_fullsized":"2024-04-25T16:00:19.342365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:56:48.018678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669652+0000","last_change":"2024-04-25T15:59:02.515912+0000","last_active":"2024-04-25T15:59:50.669652+0000","last_peered":"2024-04-25T15:59:50.669652+0000","last_clean":"2024-04-25T15:59:50.669652+0000","last_became_active":"2024-04-25T15:59:02.514949+0000","last_became_peered":"2024-04-25T15:59:02.514949+0000","last_unstale":"2024-04-25T15:59:50.669652+0000","last_undegraded":"2024-04-25T15:59:50.669652+0000","last_fullsized":"2024-04-25T15:59:50.669652+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:33.467223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360414+0000","last_change":"2024-04-25T15:59:02.988896+0000","last_active":"2024-04-25T16:00:19.360414+0000","last_peered":"2024-04-25T16:00:19.360414+0000","last_clean":"2024-04-25T16:00:19.360414+0000","last_became_active":"2024-04-25T15:59:02.988478+0000","last_became_peered":"2024-04-25T15:59:02.988478+0000","last_unstale":"2024-04-25T16:00:19.360414+0000","last_undegraded":"2024-04-25T16:00:19.360414+0000","last_fullsized":"2024-04-25T16:00:19.360414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:47.014678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"75'5","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:21.722005+0000","last_change":"2024-04-25T15:59:50.687632+0000","last_active":"2024-04-25T16:00:21.722005+0000","last_peered":"2024-04-25T16:00:21.722005+0000","last_clean":"2024-04-25T16:00:21.722005+0000","last_became_active":"2024-04-25T15:59:50.680402+0000","last_became_peered":"2024-04-25T15:59:50.680402+0000","last_unstale":"2024-04-25T16:00:21.722005+0000","last_undegraded":"2024-04-25T16:00:21.722005+0000","last_fullsized":"2024-04-25T16:00:21.722005+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:12.544489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":33,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359317+0000","last_change":"2024-04-25T15:59:02.990507+0000","last_active":"2024-04-25T16:00:19.359317+0000","last_peered":"2024-04-25T16:00:19.359317+0000","last_clean":"2024-04-25T16:00:19.359317+0000","last_became_active":"2024-04-25T15:59:02.990160+0000","last_became_peered":"2024-04-25T15:59:02.990160+0000","last_unstale":"2024-04-25T16:00:19.359317+0000","last_undegraded":"2024-04-25T16:00:19.359317+0000","last_fullsized":"2024-04-25T16:00:19.359317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:18:21.526155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672520+0000","last_change":"2024-04-25T15:59:02.510956+0000","last_active":"2024-04-25T15:59:50.672520+0000","last_peered":"2024-04-25T15:59:50.672520+0000","last_clean":"2024-04-25T15:59:50.672520+0000","last_became_active":"2024-04-25T15:59:02.510578+0000","last_became_peered":"2024-04-25T15:59:02.510578+0000","last_unstale":"2024-04-25T15:59:50.672520+0000","last_undegraded":"2024-04-25T15:59:50.672520+0000","last_fullsized":"2024-04-25T15:59:50.672520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:11.512365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337102+0000","last_change":"2024-04-25T15:59:03.001122+0000","last_active":"2024-04-25T16:00:19.337102+0000","last_peered":"2024-04-25T16:00:19.337102+0000","last_clean":"2024-04-25T16:00:19.337102+0000","last_became_active":"2024-04-25T15:59:03.000672+0000","last_became_peered":"2024-04-25T15:59:03.000672+0000","last_unstale":"2024-04-25T16:00:19.337102+0000","last_undegraded":"2024-04-25T16:00:19.337102+0000","last_fullsized":"2024-04-25T16:00:19.337102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:33.216434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'5","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670676+0000","last_change":"2024-04-25T15:59:02.520645+0000","last_active":"2024-04-25T15:59:50.670676+0000","last_peered":"2024-04-25T15:59:50.670676+0000","last_clean":"2024-04-25T15:59:50.670676+0000","last_became_active":"2024-04-25T15:59:02.520197+0000","last_became_peered":"2024-04-25T15:59:02.520197+0000","last_unstale":"2024-04-25T15:59:50.670676+0000","last_undegraded":"2024-04-25T15:59:50.670676+0000","last_fullsized":"2024-04-25T15:59:50.670676+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:20.120554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347691+0000","last_change":"2024-04-25T15:59:03.001668+0000","last_active":"2024-04-25T16:00:19.347691+0000","last_peered":"2024-04-25T16:00:19.347691+0000","last_clean":"2024-04-25T16:00:19.347691+0000","last_became_active":"2024-04-25T15:59:03.001333+0000","last_became_peered":"2024-04-25T15:59:03.001333+0000","last_unstale":"2024-04-25T16:00:19.347691+0000","last_undegraded":"2024-04-25T16:00:19.347691+0000","last_fullsized":"2024-04-25T16:00:19.347691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:24.992476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346081+0000","last_change":"2024-04-25T15:59:03.005533+0000","last_active":"2024-04-25T16:00:19.346081+0000","last_peered":"2024-04-25T16:00:19.346081+0000","last_clean":"2024-04-25T16:00:19.346081+0000","last_became_active":"2024-04-25T15:59:03.004124+0000","last_became_peered":"2024-04-25T15:59:03.004124+0000","last_unstale":"2024-04-25T16:00:19.346081+0000","last_undegraded":"2024-04-25T16:00:19.346081+0000","last_fullsized":"2024-04-25T16:00:19.346081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:27:28.507065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"77'12","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.331274+0000","last_change":"2024-04-25T15:59:02.993416+0000","last_active":"2024-04-25T16:00:19.331274+0000","last_peered":"2024-04-25T16:00:19.331274+0000","last_clean":"2024-04-25T16:00:19.331274+0000","last_became_active":"2024-04-25T15:59:02.993098+0000","last_became_peered":"2024-04-25T15:59:02.993098+0000","last_unstale":"2024-04-25T16:00:19.331274+0000","last_undegraded":"2024-04-25T16:00:19.331274+0000","last_fullsized":"2024-04-25T16:00:19.331274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:46.859932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350142+0000","last_change":"2024-04-25T15:59:02.996858+0000","last_active":"2024-04-25T16:00:19.350142+0000","last_peered":"2024-04-25T16:00:19.350142+0000","last_clean":"2024-04-25T16:00:19.350142+0000","last_became_active":"2024-04-25T15:59:02.996530+0000","last_became_peered":"2024-04-25T15:59:02.996530+0000","last_unstale":"2024-04-25T16:00:19.350142+0000","last_undegraded":"2024-04-25T16:00:19.350142+0000","last_fullsized":"2024-04-25T16:00:19.350142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:36.723948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350470+0000","last_change":"2024-04-25T15:59:02.995942+0000","last_active":"2024-04-25T16:00:19.350470+0000","last_peered":"2024-04-25T16:00:19.350470+0000","last_clean":"2024-04-25T16:00:19.350470+0000","last_became_active":"2024-04-25T15:59:02.995567+0000","last_became_peered":"2024-04-25T15:59:02.995567+0000","last_unstale":"2024-04-25T16:00:19.350470+0000","last_undegraded":"2024-04-25T16:00:19.350470+0000","last_fullsized":"2024-04-25T16:00:19.350470+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:32.692392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350261+0000","last_change":"2024-04-25T15:59:03.000326+0000","last_active":"2024-04-25T16:00:19.350261+0000","last_peered":"2024-04-25T16:00:19.350261+0000","last_clean":"2024-04-25T16:00:19.350261+0000","last_became_active":"2024-04-25T15:59:02.999961+0000","last_became_peered":"2024-04-25T15:59:02.999961+0000","last_unstale":"2024-04-25T16:00:19.350261+0000","last_undegraded":"2024-04-25T16:00:19.350261+0000","last_fullsized":"2024-04-25T16:00:19.350261+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:10.631460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341696+0000","last_change":"2024-04-25T15:59:03.003625+0000","last_active":"2024-04-25T16:00:19.341696+0000","last_peered":"2024-04-25T16:00:19.341696+0000","last_clean":"2024-04-25T16:00:19.341696+0000","last_became_active":"2024-04-25T15:59:03.003280+0000","last_became_peered":"2024-04-25T15:59:03.003280+0000","last_unstale":"2024-04-25T16:00:19.341696+0000","last_undegraded":"2024-04-25T16:00:19.341696+0000","last_fullsized":"2024-04-25T16:00:19.341696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:21:54.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346578+0000","last_change":"2024-04-25T15:59:02.994645+0000","last_active":"2024-04-25T16:00:19.346578+0000","last_peered":"2024-04-25T16:00:19.346578+0000","last_clean":"2024-04-25T16:00:19.346578+0000","last_became_active":"2024-04-25T15:59:02.993924+0000","last_became_peered":"2024-04-25T15:59:02.993924+0000","last_unstale":"2024-04-25T16:00:19.346578+0000","last_undegraded":"2024-04-25T16:00:19.346578+0000","last_fullsized":"2024-04-25T16:00:19.346578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.644708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340331+0000","last_change":"2024-04-25T15:59:03.006156+0000","last_active":"2024-04-25T16:00:19.340331+0000","last_peered":"2024-04-25T16:00:19.340331+0000","last_clean":"2024-04-25T16:00:19.340331+0000","last_became_active":"2024-04-25T15:59:03.005795+0000","last_became_peered":"2024-04-25T15:59:03.005795+0000","last_unstale":"2024-04-25T16:00:19.340331+0000","last_undegraded":"2024-04-25T16:00:19.340331+0000","last_fullsized":"2024-04-25T16:00:19.340331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:23.523868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337613+0000","last_change":"2024-04-25T15:59:03.010404+0000","last_active":"2024-04-25T16:00:19.337613+0000","last_peered":"2024-04-25T16:00:19.337613+0000","last_clean":"2024-04-25T16:00:19.337613+0000","last_became_active":"2024-04-25T15:59:03.010100+0000","last_became_peered":"2024-04-25T15:59:03.010100+0000","last_unstale":"2024-04-25T16:00:19.337613+0000","last_undegraded":"2024-04-25T16:00:19.337613+0000","last_fullsized":"2024-04-25T16:00:19.337613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:22.718842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.335498+0000","last_change":"2024-04-25T15:59:03.009240+0000","last_active":"2024-04-25T16:00:19.335498+0000","last_peered":"2024-04-25T16:00:19.335498+0000","last_clean":"2024-04-25T16:00:19.335498+0000","last_became_active":"2024-04-25T15:59:03.008723+0000","last_became_peered":"2024-04-25T15:59:03.008723+0000","last_unstale":"2024-04-25T16:00:19.335498+0000","last_undegraded":"2024-04-25T16:00:19.335498+0000","last_fullsized":"2024-04-25T16:00:19.335498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:14.049934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359019+0000","last_change":"2024-04-25T15:59:02.994359+0000","last_active":"2024-04-25T16:00:19.359019+0000","last_peered":"2024-04-25T16:00:19.359019+0000","last_clean":"2024-04-25T16:00:19.359019+0000","last_became_active":"2024-04-25T15:59:02.993990+0000","last_became_peered":"2024-04-25T15:59:02.993990+0000","last_unstale":"2024-04-25T16:00:19.359019+0000","last_undegraded":"2024-04-25T16:00:19.359019+0000","last_fullsized":"2024-04-25T16:00:19.359019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:55.477414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350058+0000","last_change":"2024-04-25T15:59:02.995510+0000","last_active":"2024-04-25T16:00:19.350058+0000","last_peered":"2024-04-25T16:00:19.350058+0000","last_clean":"2024-04-25T16:00:19.350058+0000","last_became_active":"2024-04-25T15:59:02.995008+0000","last_became_peered":"2024-04-25T15:59:02.995008+0000","last_unstale":"2024-04-25T16:00:19.350058+0000","last_undegraded":"2024-04-25T16:00:19.350058+0000","last_fullsized":"2024-04-25T16:00:19.350058+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:40.439615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341453+0000","last_change":"2024-04-25T15:59:03.005846+0000","last_active":"2024-04-25T16:00:19.341453+0000","last_peered":"2024-04-25T16:00:19.341453+0000","last_clean":"2024-04-25T16:00:19.341453+0000","last_became_active":"2024-04-25T15:59:03.003779+0000","last_became_peered":"2024-04-25T15:59:03.003779+0000","last_unstale":"2024-04-25T16:00:19.341453+0000","last_undegraded":"2024-04-25T16:00:19.341453+0000","last_fullsized":"2024-04-25T16:00:19.341453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.698405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247542,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64388,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":755,"num_read_kb":472,"num_write":242,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1192,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":193164,"data_compressed":5607,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710749,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":839680,"data_stored":402102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906980,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":1051075,"data_compressed":37695,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070504,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":402010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266735,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":851968,"data_stored":402366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462964,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":995047,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626488,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":401727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":461244,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":403066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":459124,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":401552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986243,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":402120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822717,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":994193,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60356,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58236,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":59702,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:00:29.913 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-25T16:00:29.913 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-25T16:00:29.913 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-25T16:00:29.913 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-25T16:00:30.163 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:30.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:30 smithi186 ceph-mon[33258]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:00:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:30 smithi186 ceph-mon[33258]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:00:30.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:30 smithi031 ceph-mon[29140]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:00:30.424 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:30 smithi031 ceph-mon[29140]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:00:30.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:30 smithi063 ceph-mon[32413]: from='client.14961 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:00:30.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:30 smithi063 ceph-mon[32413]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:00:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.965+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/582149580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 msgr2=0x7f3fd4111090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.965+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/582149580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 0x7f3fd4111090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc00b0d0 tx=0x7f3fcc00b3e0 comp rx=0 tx=0).stop 2024-04-25T16:00:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.965+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/582149580 shutdown_connections 2024-04-25T16:00:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.965+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/582149580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fd41044b0 0x7f3fd41048c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.965+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/582149580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 0x7f3fd4111090 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.965+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/582149580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fd4108de0 0x7f3fd41091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.965+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/582149580 >> 172.21.15.31:0/582149580 conn(0x7f3fd40ffdc0 msgr2=0x7f3fd4102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.965+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/582149580 shutdown_connections 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.965+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/582149580 wait complete. 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdc7fb700 1 Processor -- start 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdc7fb700 1 -- start start 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdc7fb700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fd41044b0 0x7f3fd41a09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdc7fb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fd4108de0 0x7f3fd41a0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdc7fb700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 0x7f3fd419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdc7fb700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd41143d0 con 0x7f3fd4109730 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdc7fb700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd4114250 con 0x7f3fd41044b0 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdc7fb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3fd4114550 con 0x7f3fd4108de0 2024-04-25T16:00:31.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdad98700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 0x7f3fd419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdad98700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 0x7f3fd419ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53498/0 (socket says 172.21.15.31:53498) 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdad98700 1 -- 172.21.15.31:0/297929188 learned_addr learned my addr 172.21.15.31:0/297929188 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fda597700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fd41044b0 0x7f3fd41a09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fd9d96700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fd4108de0 0x7f3fd41a0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdad98700 1 -- 172.21.15.31:0/297929188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fd4108de0 msgr2=0x7f3fd41a0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdad98700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fd4108de0 0x7f3fd41a0ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdad98700 1 -- 172.21.15.31:0/297929188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fd41044b0 msgr2=0x7f3fd41a09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdad98700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fd41044b0 0x7f3fd41a09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.966+0000 7f3fdad98700 1 -- 172.21.15.31:0/297929188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fd419b260 con 0x7f3fd4109730 2024-04-25T16:00:31.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.967+0000 7f3fdad98700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 0x7f3fd419ab30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0095c0 tx=0x7f3fcc007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:31.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.967+0000 7f3fcb7fe700 1 -- 172.21.15.31:0/297929188 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc01f070 con 0x7f3fd4109730 2024-04-25T16:00:31.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.967+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fd419b4f0 con 0x7f3fd4109730 2024-04-25T16:00:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.967+0000 7f3fcb7fe700 1 -- 172.21.15.31:0/297929188 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fcc003c00 con 0x7f3fd4109730 2024-04-25T16:00:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.967+0000 7f3fcb7fe700 1 -- 172.21.15.31:0/297929188 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fcc01b6a0 con 0x7f3fd4109730 2024-04-25T16:00:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.967+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fd41a7750 con 0x7f3fd4109730 2024-04-25T16:00:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.968+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fd4105960 con 0x7f3fd4109730 2024-04-25T16:00:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.969+0000 7f3fcb7fe700 1 -- 172.21.15.31:0/297929188 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3fcc022070 con 0x7f3fd4109730 2024-04-25T16:00:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.970+0000 7f3fcb7fe700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fc00afb80 0x7f3fc00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.970+0000 7f3fda597700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fc00afb80 0x7f3fc00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.970+0000 7f3fcb7fe700 1 -- 172.21.15.31:0/297929188 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3fcc00f020 con 0x7f3fd4109730 2024-04-25T16:00:31.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.970+0000 7f3fda597700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fc00afb80 0x7f3fc00b2090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0005950 tx=0x7f3fd000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:31.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:31.975+0000 7f3fcb7fe700 1 -- 172.21.15.31:0/297929188 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3fcc029020 con 0x7f3fd4109730 2024-04-25T16:00:32.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.228+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f3fd419bfd0 con 0x7f3fd4109730 2024-04-25T16:00:32.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.229+0000 7f3fcb7fe700 1 -- 172.21.15.31:0/297929188 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f3fcc0a8b00 con 0x7f3fd4109730 2024-04-25T16:00:32.230 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:32.230 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fc00afb80 msgr2=0x7f3fc00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fc00afb80 0x7f3fc00b2090 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0005950 tx=0x7f3fd000b410 comp rx=0 tx=0).stop 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 msgr2=0x7f3fd419ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 0x7f3fd419ab30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc0095c0 tx=0x7f3fcc007a20 comp rx=0 tx=0).stop 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 shutdown_connections 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fc00afb80 0x7f3fc00b2090 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fd41044b0 0x7f3fd41a09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fd4109730 0x7f3fd419ab30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 --2- 172.21.15.31:0/297929188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fd4108de0 0x7f3fd41a0ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 >> 172.21.15.31:0/297929188 conn(0x7f3fd40ffdc0 msgr2=0x7f3fd4108550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 shutdown_connections 2024-04-25T16:00:32.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:32.233+0000 7f3fdc7fb700 1 -- 172.21.15.31:0/297929188 wait complete. 2024-04-25T16:00:32.604 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:32 smithi031 ceph-mon[29140]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T16:00:32.604 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:32 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/297929188' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T16:00:32.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:32 smithi186 ceph-mon[33258]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T16:00:32.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:32 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/297929188' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T16:00:32.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:32 smithi063 ceph-mon[32413]: pgmap v328: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-25T16:00:32.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:32 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/297929188' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-25T16:00:33.053 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-25T16:00:33.053 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:00:33.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.171+0000 7f11a4534640 1 -- 172.21.15.31:0/2376839752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c1029e0 msgr2=0x7f119c102eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.171+0000 7f11a4534640 1 --2- 172.21.15.31:0/2376839752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c1029e0 0x7f119c102eb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1190009900 tx=0x7f119002a400 comp rx=0 tx=0).stop 2024-04-25T16:00:33.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.171+0000 7f11a4534640 1 -- 172.21.15.31:0/2376839752 shutdown_connections 2024-04-25T16:00:33.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.171+0000 7f11a4534640 1 --2- 172.21.15.31:0/2376839752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119c103540 0x7f119c103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.171+0000 7f11a4534640 1 --2- 172.21.15.31:0/2376839752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c1029e0 0x7f119c102eb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.171+0000 7f11a4534640 1 --2- 172.21.15.31:0/2376839752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c102000 0x7f119c102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.171+0000 7f11a4534640 1 -- 172.21.15.31:0/2376839752 >> 172.21.15.31:0/2376839752 conn(0x7f119c0fdbd0 msgr2=0x7f119c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.171+0000 7f11a4534640 1 -- 172.21.15.31:0/2376839752 shutdown_connections 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.171+0000 7f11a4534640 1 -- 172.21.15.31:0/2376839752 wait complete. 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a4534640 1 Processor -- start 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a4534640 1 -- start start 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a4534640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c102000 0x7f119c19c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a4534640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c1029e0 0x7f119c19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a4534640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119c103540 0x7f119c1a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a4534640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f119c1117c0 con 0x7f119c102000 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a4534640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f119c111640 con 0x7f119c103540 2024-04-25T16:00:33.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a4534640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f119c111940 con 0x7f119c1029e0 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a22d0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c102000 0x7f119c19c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a22d0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c102000 0x7f119c19c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53520/0 (socket says 172.21.15.31:53520) 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a22d0640 1 -- 172.21.15.31:0/128569095 learned_addr learned my addr 172.21.15.31:0/128569095 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a1acf640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c1029e0 0x7f119c19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a2ad1640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119c103540 0x7f119c1a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a22d0640 1 -- 172.21.15.31:0/128569095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c1029e0 msgr2=0x7f119c19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a22d0640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c1029e0 0x7f119c19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a22d0640 1 -- 172.21.15.31:0/128569095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119c103540 msgr2=0x7f119c1a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a22d0640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119c103540 0x7f119c1a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a22d0640 1 -- 172.21.15.31:0/128569095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c1a3f60 con 0x7f119c102000 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.172+0000 7f11a2ad1640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119c103540 0x7f119c1a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.173+0000 7f11a22d0640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c102000 0x7f119c19c420 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f119400e9f0 tx=0x7f119400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:33.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.173+0000 7f119b7fe640 1 -- 172.21.15.31:0/128569095 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119400cd80 con 0x7f119c102000 2024-04-25T16:00:33.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.173+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119c1a4250 con 0x7f119c102000 2024-04-25T16:00:33.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.173+0000 7f119b7fe640 1 -- 172.21.15.31:0/128569095 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11940054d0 con 0x7f119c102000 2024-04-25T16:00:33.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.173+0000 7f119b7fe640 1 -- 172.21.15.31:0/128569095 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1194010690 con 0x7f119c102000 2024-04-25T16:00:33.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.173+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119c1a47a0 con 0x7f119c102000 2024-04-25T16:00:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.174+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f119c10d460 con 0x7f119c102000 2024-04-25T16:00:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.175+0000 7f119b7fe640 1 -- 172.21.15.31:0/128569095 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1194004e30 con 0x7f119c102000 2024-04-25T16:00:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.176+0000 7f119b7fe640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f11840afaf0 0x7f11840b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.176+0000 7f11a1acf640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f11840afaf0 0x7f11840b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.176+0000 7f119b7fe640 1 -- 172.21.15.31:0/128569095 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f11940df310 con 0x7f119c102000 2024-04-25T16:00:33.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.176+0000 7f11a1acf640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f11840afaf0 0x7f11840b2000 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1190009230 tx=0x7f1190031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:33.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.179+0000 7f119b7fe640 1 -- 172.21.15.31:0/128569095 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1194053ff0 con 0x7f119c102000 2024-04-25T16:00:33.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.369+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f119c0637a0 con 0x7f119c102000 2024-04-25T16:00:33.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.370+0000 7f119b7fe640 1 -- 172.21.15.31:0/128569095 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f11940a7ea0 con 0x7f119c102000 2024-04-25T16:00:33.371 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:33.372 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:00:33.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f11840afaf0 msgr2=0x7f11840b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f11840afaf0 0x7f11840b2000 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1190009230 tx=0x7f1190031400 comp rx=0 tx=0).stop 2024-04-25T16:00:33.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c102000 msgr2=0x7f119c19c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c102000 0x7f119c19c420 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f119400e9f0 tx=0x7f119400edf0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 shutdown_connections 2024-04-25T16:00:33.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f11840afaf0 0x7f11840b2000 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f119c103540 0x7f119c1a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c1029e0 0x7f119c19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 --2- 172.21.15.31:0/128569095 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f119c102000 0x7f119c19c420 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 >> 172.21.15.31:0/128569095 conn(0x7f119c0fdbd0 msgr2=0x7f119c10a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:33.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 shutdown_connections 2024-04-25T16:00:33.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.373+0000 7f11a4534640 1 -- 172.21.15.31:0/128569095 wait complete. 2024-04-25T16:00:33.386 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:59:00.978038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:59:01.829535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:00:33.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:00:33.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.502+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/4194021235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f4102180 msgr2=0x7fd8f4102e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.502+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/4194021235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f4102180 0x7fd8f4102e80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0009900 tx=0x7fd8f002a410 comp rx=0 tx=0).stop 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.502+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/4194021235 shutdown_connections 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.502+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/4194021235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f41080c0 0x7fd8f410a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.502+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/4194021235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f41033c0 0x7fd8f41037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.502+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/4194021235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f4102180 0x7fd8f4102e80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.502+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/4194021235 >> 172.21.15.31:0/4194021235 conn(0x7fd8f40fd350 msgr2=0x7fd8f40ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.502+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/4194021235 shutdown_connections 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.502+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/4194021235 wait complete. 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8fbaaa640 1 Processor -- start 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8fbaaa640 1 -- start start 2024-04-25T16:00:33.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8fbaaa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f4102180 0x7fd8f40ff3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8fbaaa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f41033c0 0x7fd8f40fda60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8fbaaa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f41080c0 0x7fd8f40fdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8fbaaa640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f410d450 con 0x7fd8f41033c0 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8fbaaa640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f410d2d0 con 0x7fd8f41080c0 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8fbaaa640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f410d5d0 con 0x7fd8f4102180 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8f9045640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f41033c0 0x7fd8f40fda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8f9045640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f41033c0 0x7fd8f40fda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53542/0 (socket says 172.21.15.31:53542) 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8f9045640 1 -- 172.21.15.31:0/592166287 learned_addr learned my addr 172.21.15.31:0/592166287 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8f9846640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f4102180 0x7fd8f40ff3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8f9045640 1 -- 172.21.15.31:0/592166287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f4102180 msgr2=0x7fd8f40ff3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8f9045640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f4102180 0x7fd8f40ff3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8f9045640 1 -- 172.21.15.31:0/592166287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f41080c0 msgr2=0x7fd8f40fdfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8f9045640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f41080c0 0x7fd8f40fdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.503+0000 7fd8f9045640 1 -- 172.21.15.31:0/592166287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f40fe6d0 con 0x7fd8f41033c0 2024-04-25T16:00:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.504+0000 7fd8f9045640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f41033c0 0x7fd8f40fda60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400ddc0 tx=0x7fd8e400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.504+0000 7fd8eaffd640 1 -- 172.21.15.31:0/592166287 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e400be60 con 0x7fd8f41033c0 2024-04-25T16:00:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.504+0000 7fd8eaffd640 1 -- 172.21.15.31:0/592166287 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8e40054d0 con 0x7fd8f41033c0 2024-04-25T16:00:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.504+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f41a4360 con 0x7fd8f41033c0 2024-04-25T16:00:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.504+0000 7fd8eaffd640 1 -- 172.21.15.31:0/592166287 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e4004e40 con 0x7fd8f41033c0 2024-04-25T16:00:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.504+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f41a4820 con 0x7fd8f41033c0 2024-04-25T16:00:33.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.505+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8f41a4a80 con 0x7fd8f41033c0 2024-04-25T16:00:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.507+0000 7fd8eaffd640 1 -- 172.21.15.31:0/592166287 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd8e401d020 con 0x7fd8f41033c0 2024-04-25T16:00:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.507+0000 7fd8eaffd640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afaf0 0x7fd8e00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.507+0000 7fd8eaffd640 1 -- 172.21.15.31:0/592166287 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd8e40df150 con 0x7fd8f41033c0 2024-04-25T16:00:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.507+0000 7fd8f9846640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afaf0 0x7fd8e00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.508+0000 7fd8f9846640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afaf0 0x7fd8e00b2000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0005ab0 tx=0x7fd8f0031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:33.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.511+0000 7fd8eaffd640 1 -- 172.21.15.31:0/592166287 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8e4024070 con 0x7fd8f41033c0 2024-04-25T16:00:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.695+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd8f41a4c50 con 0x7fd8f41033c0 2024-04-25T16:00:33.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.696+0000 7fd8eaffd640 1 -- 172.21.15.31:0/592166287 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7fd8e40a7d40 con 0x7fd8f41033c0 2024-04-25T16:00:33.697 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T16:00:33.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afaf0 msgr2=0x7fd8e00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afaf0 0x7fd8e00b2000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0005ab0 tx=0x7fd8f0031400 comp rx=0 tx=0).stop 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f41033c0 msgr2=0x7fd8f40fda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f41033c0 0x7fd8f40fda60 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd8e400ddc0 tx=0x7fd8e400a620 comp rx=0 tx=0).stop 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 shutdown_connections 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afaf0 0x7fd8e00b2000 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f41080c0 0x7fd8f40fdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f41033c0 0x7fd8f40fda60 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 --2- 172.21.15.31:0/592166287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f4102180 0x7fd8f40ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 >> 172.21.15.31:0/592166287 conn(0x7fd8f40fd350 msgr2=0x7fd8f41060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 shutdown_connections 2024-04-25T16:00:33.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.698+0000 7fd8fbaaa640 1 -- 172.21.15.31:0/592166287 wait complete. 2024-04-25T16:00:33.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:33 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/128569095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:00:33.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:00:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.825+0000 7f9d28b3f640 1 -- 172.21.15.31:0/4185142291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 msgr2=0x7f9d24105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.825+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/4185142291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 0x7f9d24105150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009900 tx=0x7f9d1802a410 comp rx=0 tx=0).stop 2024-04-25T16:00:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.825+0000 7f9d28b3f640 1 -- 172.21.15.31:0/4185142291 shutdown_connections 2024-04-25T16:00:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.825+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/4185142291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d24109a40 0x7f9d2410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.825+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/4185142291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 0x7f9d24105150 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.825+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/4185142291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d24103b00 0x7f9d24104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.825+0000 7f9d28b3f640 1 -- 172.21.15.31:0/4185142291 >> 172.21.15.31:0/4185142291 conn(0x7f9d240fed10 msgr2=0x7f9d241011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:33.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.825+0000 7f9d28b3f640 1 -- 172.21.15.31:0/4185142291 shutdown_connections 2024-04-25T16:00:33.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.825+0000 7f9d28b3f640 1 -- 172.21.15.31:0/4185142291 wait complete. 2024-04-25T16:00:33.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d28b3f640 1 Processor -- start 2024-04-25T16:00:33.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d28b3f640 1 -- start start 2024-04-25T16:00:33.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d28b3f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d24103b00 0x7f9d2419a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d28b3f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 0x7f9d2419a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d28b3f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d24109a40 0x7f9d241a1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d28b3f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2410f600 con 0x7f9d24104d40 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d28b3f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2410f480 con 0x7f9d24109a40 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d28b3f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9d2410f780 con 0x7f9d24103b00 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d21d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 0x7f9d2419a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d21d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 0x7f9d2419a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45618/0 (socket says 172.21.15.31:45618) 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d21d9b640 1 -- 172.21.15.31:0/382836061 learned_addr learned my addr 172.21.15.31:0/382836061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d2259c640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d24103b00 0x7f9d2419a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d22d9d640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d24109a40 0x7f9d241a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d21d9b640 1 -- 172.21.15.31:0/382836061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d24103b00 msgr2=0x7f9d2419a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d21d9b640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d24103b00 0x7f9d2419a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d21d9b640 1 -- 172.21.15.31:0/382836061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d24109a40 msgr2=0x7f9d241a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d21d9b640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d24109a40 0x7f9d241a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:33.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.826+0000 7f9d21d9b640 1 -- 172.21.15.31:0/382836061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d241a1d00 con 0x7f9d24104d40 2024-04-25T16:00:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.827+0000 7f9d21d9b640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 0x7f9d2419a800 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9d18006010 tx=0x7f9d18004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.827+0000 7f9d137fe640 1 -- 172.21.15.31:0/382836061 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1803f030 con 0x7f9d24104d40 2024-04-25T16:00:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.827+0000 7f9d137fe640 1 -- 172.21.15.31:0/382836061 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d18003cb0 con 0x7f9d24104d40 2024-04-25T16:00:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.827+0000 7f9d137fe640 1 -- 172.21.15.31:0/382836061 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d18035070 con 0x7f9d24104d40 2024-04-25T16:00:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.827+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d241a1f90 con 0x7f9d24104d40 2024-04-25T16:00:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.827+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d241a23a0 con 0x7f9d24104d40 2024-04-25T16:00:33.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.828+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d24108850 con 0x7f9d24104d40 2024-04-25T16:00:33.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.830+0000 7f9d137fe640 1 -- 172.21.15.31:0/382836061 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9d18042030 con 0x7f9d24104d40 2024-04-25T16:00:33.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.830+0000 7f9d137fe640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9d0c0afaf0 0x7f9d0c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:33.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.831+0000 7f9d137fe640 1 -- 172.21.15.31:0/382836061 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9d180ff930 con 0x7f9d24104d40 2024-04-25T16:00:33.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.831+0000 7f9d2259c640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9d0c0afaf0 0x7f9d0c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:33.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.831+0000 7f9d2259c640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9d0c0afaf0 0x7f9d0c0b2000 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9d14005fd0 tx=0x7f9d1400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:33.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:33.834+0000 7f9d137fe640 1 -- 172.21.15.31:0/382836061 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d180c84c0 con 0x7f9d24104d40 2024-04-25T16:00:33.854 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:33 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/128569095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:00:33.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:33 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/128569095' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:00:34.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.020+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9d240637d0 con 0x7f9d24104d40 2024-04-25T16:00:34.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.020+0000 7f9d137fe640 1 -- 172.21.15.31:0/382836061 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7f9d180c7dc0 con 0x7f9d24104d40 2024-04-25T16:00:34.021 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T16:00:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9d0c0afaf0 msgr2=0x7f9d0c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9d0c0afaf0 0x7f9d0c0b2000 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9d14005fd0 tx=0x7f9d1400b410 comp rx=0 tx=0).stop 2024-04-25T16:00:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 msgr2=0x7f9d2419a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 0x7f9d2419a800 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9d18006010 tx=0x7f9d18004340 comp rx=0 tx=0).stop 2024-04-25T16:00:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 shutdown_connections 2024-04-25T16:00:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9d0c0afaf0 0x7f9d0c0b2000 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9d24109a40 0x7f9d241a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9d24104d40 0x7f9d2419a800 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 --2- 172.21.15.31:0/382836061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d24103b00 0x7f9d2419a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 >> 172.21.15.31:0/382836061 conn(0x7f9d240fed10 msgr2=0x7f9d24105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:34.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 shutdown_connections 2024-04-25T16:00:34.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.022+0000 7f9d28b3f640 1 -- 172.21.15.31:0/382836061 wait complete. 2024-04-25T16:00:34.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:00:34.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.148+0000 7f20fec30640 1 -- 172.21.15.31:0/2380219645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 msgr2=0x7f20f810bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.148+0000 7f20fec30640 1 --2- 172.21.15.31:0/2380219645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 0x7f20f810bcb0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f20f0009ca0 tx=0x7f20f002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.148+0000 7f20fec30640 1 -- 172.21.15.31:0/2380219645 shutdown_connections 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.148+0000 7f20fec30640 1 --2- 172.21.15.31:0/2380219645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 0x7f20f810bcb0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.148+0000 7f20fec30640 1 --2- 172.21.15.31:0/2380219645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20f8106f30 0x7f20f8109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.148+0000 7f20fec30640 1 --2- 172.21.15.31:0/2380219645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f81045d0 0x7f20f81069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.148+0000 7f20fec30640 1 -- 172.21.15.31:0/2380219645 >> 172.21.15.31:0/2380219645 conn(0x7f20f80fdbd0 msgr2=0x7f20f8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.148+0000 7f20fec30640 1 -- 172.21.15.31:0/2380219645 shutdown_connections 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fec30640 1 -- 172.21.15.31:0/2380219645 wait complete. 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fec30640 1 Processor -- start 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fec30640 1 -- start start 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fec30640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20f81045d0 0x7f20f8103a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fec30640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f8106f30 0x7f20f81020c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fec30640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 0x7f20f8102600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fec30640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f20f810f5f0 con 0x7f20f8109890 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fec30640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f20f810f470 con 0x7f20f8106f30 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fec30640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f20f810f770 con 0x7f20f81045d0 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fd1cd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 0x7f20f8102600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.149+0000 7f20fd1cd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 0x7f20f8102600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45636/0 (socket says 172.21.15.31:45636) 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fd1cd640 1 -- 172.21.15.31:0/379456291 learned_addr learned my addr 172.21.15.31:0/379456291 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fc9cc640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20f81045d0 0x7f20f8103a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20f7fff640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f8106f30 0x7f20f81020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fd1cd640 1 -- 172.21.15.31:0/379456291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20f81045d0 msgr2=0x7f20f8103a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fd1cd640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20f81045d0 0x7f20f8103a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fd1cd640 1 -- 172.21.15.31:0/379456291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f8106f30 msgr2=0x7f20f81020c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fd1cd640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f8106f30 0x7f20f81020c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fd1cd640 1 -- 172.21.15.31:0/379456291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f8102d30 con 0x7f20f8109890 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fd1cd640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 0x7f20f8102600 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f20f0033040 tx=0x7f20f00319f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:34.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20f5ffb640 1 -- 172.21.15.31:0/379456291 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f0003c70 con 0x7f20f8109890 2024-04-25T16:00:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20f5ffb640 1 -- 172.21.15.31:0/379456291 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f20f0003e10 con 0x7f20f8109890 2024-04-25T16:00:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f81a42c0 con 0x7f20f8109890 2024-04-25T16:00:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20f5ffb640 1 -- 172.21.15.31:0/379456291 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20f002fc70 con 0x7f20f8109890 2024-04-25T16:00:34.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.150+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f81a4720 con 0x7f20f8109890 2024-04-25T16:00:34.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.151+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f806a160 con 0x7f20f8109890 2024-04-25T16:00:34.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.153+0000 7f20f5ffb640 1 -- 172.21.15.31:0/379456291 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f20f0004b40 con 0x7f20f8109890 2024-04-25T16:00:34.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.153+0000 7f20f5ffb640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f20e80afaf0 0x7f20e80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.154+0000 7f20fc9cc640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f20e80afaf0 0x7f20e80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.154+0000 7f20f5ffb640 1 -- 172.21.15.31:0/379456291 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f20f0100360 con 0x7f20f8109890 2024-04-25T16:00:34.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.154+0000 7f20fc9cc640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f20e80afaf0 0x7f20e80b2000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0058c0 tx=0x7f20ec00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:34.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.157+0000 7f20f5ffb640 1 -- 172.21.15.31:0/379456291 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20f00c8f50 con 0x7f20f8109890 2024-04-25T16:00:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.343+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f20f80ff6f0 con 0x7f20f8109890 2024-04-25T16:00:34.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.344+0000 7f20f5ffb640 1 -- 172.21.15.31:0/379456291 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7f20f00c8850 con 0x7f20f8109890 2024-04-25T16:00:34.345 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T16:00:34.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f20e80afaf0 msgr2=0x7f20e80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f20e80afaf0 0x7f20e80b2000 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f20ec0058c0 tx=0x7f20ec00a2d0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 msgr2=0x7f20f8102600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 0x7f20f8102600 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f20f0033040 tx=0x7f20f00319f0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 shutdown_connections 2024-04-25T16:00:34.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f20e80afaf0 0x7f20e80b2000 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f20f8109890 0x7f20f8102600 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f20f8106f30 0x7f20f81020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 --2- 172.21.15.31:0/379456291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f20f81045d0 0x7f20f8103a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 >> 172.21.15.31:0/379456291 conn(0x7f20f80fdbd0 msgr2=0x7f20f81051a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:34.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 shutdown_connections 2024-04-25T16:00:34.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.346+0000 7f20fec30640 1 -- 172.21.15.31:0/379456291 wait complete. 2024-04-25T16:00:34.359 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:00:34.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.470+0000 7f554c649640 1 -- 172.21.15.31:0/2532593340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5544103c20 msgr2=0x7f5544104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.470+0000 7f554c649640 1 --2- 172.21.15.31:0/2532593340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5544103c20 0x7f5544104030 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5534009900 tx=0x7f553402a410 comp rx=0 tx=0).stop 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.470+0000 7f554c649640 1 -- 172.21.15.31:0/2532593340 shutdown_connections 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.470+0000 7f554c649640 1 --2- 172.21.15.31:0/2532593340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5544108920 0x7f554410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.470+0000 7f554c649640 1 --2- 172.21.15.31:0/2532593340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5544103c20 0x7f5544104030 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.470+0000 7f554c649640 1 --2- 172.21.15.31:0/2532593340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55441029e0 0x7f55441036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.470+0000 7f554c649640 1 -- 172.21.15.31:0/2532593340 >> 172.21.15.31:0/2532593340 conn(0x7f55440fdbd0 msgr2=0x7f5544100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.470+0000 7f554c649640 1 -- 172.21.15.31:0/2532593340 shutdown_connections 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.470+0000 7f554c649640 1 -- 172.21.15.31:0/2532593340 wait complete. 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554c649640 1 Processor -- start 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554c649640 1 -- start start 2024-04-25T16:00:34.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554c649640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55441029e0 0x7f55440ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554c649640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5544103c20 0x7f55440fe100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554c649640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5544108920 0x7f55440fe640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554c649640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f554410e560 con 0x7f55441029e0 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554c649640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f554410e3e0 con 0x7f5544108920 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554c649640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f554410e6e0 con 0x7f5544103c20 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554a3e5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55441029e0 0x7f55440ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554a3e5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55441029e0 0x7f55440ffa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45656/0 (socket says 172.21.15.31:45656) 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.471+0000 7f554a3e5640 1 -- 172.21.15.31:0/843702014 learned_addr learned my addr 172.21.15.31:0/843702014 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f5549be4640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5544103c20 0x7f55440fe100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f554a3e5640 1 -- 172.21.15.31:0/843702014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5544103c20 msgr2=0x7f55440fe100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f554a3e5640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5544103c20 0x7f55440fe100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f554a3e5640 1 -- 172.21.15.31:0/843702014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5544108920 msgr2=0x7f55440fe640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f554a3e5640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5544108920 0x7f55440fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f554a3e5640 1 -- 172.21.15.31:0/843702014 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55440fece0 con 0x7f55441029e0 2024-04-25T16:00:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f5549be4640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5544103c20 0x7f55440fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f554a3e5640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55441029e0 0x7f55440ffa60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5540006fd0 tx=0x7f554000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f553b7fe640 1 -- 172.21.15.31:0/843702014 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5540011830 con 0x7f55441029e0 2024-04-25T16:00:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f553b7fe640 1 -- 172.21.15.31:0/843702014 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5540016070 con 0x7f55441029e0 2024-04-25T16:00:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55441a43d0 con 0x7f55441029e0 2024-04-25T16:00:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f553b7fe640 1 -- 172.21.15.31:0/843702014 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5540010e00 con 0x7f55441029e0 2024-04-25T16:00:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.472+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55441a4860 con 0x7f55441029e0 2024-04-25T16:00:34.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.473+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55441a4af0 con 0x7f55441029e0 2024-04-25T16:00:34.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.474+0000 7f553b7fe640 1 -- 172.21.15.31:0/843702014 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f55400119d0 con 0x7f55441029e0 2024-04-25T16:00:34.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.475+0000 7f553b7fe640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55300afaf0 0x7f55300b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.475+0000 7f5549be4640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55300afaf0 0x7f55300b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.475+0000 7f553b7fe640 1 -- 172.21.15.31:0/843702014 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f55400dea00 con 0x7f55441029e0 2024-04-25T16:00:34.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.476+0000 7f5549be4640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55300afaf0 0x7f55300b2000 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5534005f70 tx=0x7f5534031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:34.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.479+0000 7f553b7fe640 1 -- 172.21.15.31:0/843702014 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f55400a75f0 con 0x7f55441029e0 2024-04-25T16:00:34.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:34 smithi031 ceph-mon[29140]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-25T16:00:34.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:34 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/592166287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:00:34.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:34 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/382836061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:00:34.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:34 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/379456291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:00:34.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.705+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55441a4cc0 con 0x7f55441029e0 2024-04-25T16:00:34.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.706+0000 7f553b7fe640 1 -- 172.21.15.31:0/843702014 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11917 (secure 0 0 0) 0x7f55400a6ef0 con 0x7f55441029e0 2024-04-25T16:00:34.707 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:34.707 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T15:59:10.264996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[14631,24553,24563,24521,24535]},"id":1}]} 2024-04-25T16:00:34.707 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55300afaf0 msgr2=0x7f55300b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55300afaf0 0x7f55300b2000 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f5534005f70 tx=0x7f5534031400 comp rx=0 tx=0).stop 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55441029e0 msgr2=0x7f55440ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55441029e0 0x7f55440ffa60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5540006fd0 tx=0x7f554000b8e0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 shutdown_connections 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55300afaf0 0x7f55300b2000 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5544108920 0x7f55440fe640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5544103c20 0x7f55440fe100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 --2- 172.21.15.31:0/843702014 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55441029e0 0x7f55440ffa60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 >> 172.21.15.31:0/843702014 conn(0x7f55440fdbd0 msgr2=0x7f5544109ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 shutdown_connections 2024-04-25T16:00:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.708+0000 7f554c649640 1 -- 172.21.15.31:0/843702014 wait complete. 2024-04-25T16:00:34.721 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:00:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.831+0000 7f2be1534640 1 -- 172.21.15.31:0/3577367649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc103c00 msgr2=0x7f2bdc104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.831+0000 7f2be1534640 1 --2- 172.21.15.31:0/3577367649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc103c00 0x7f2bdc104010 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2bd0009900 tx=0x7f2bd002a410 comp rx=0 tx=0).stop 2024-04-25T16:00:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.832+0000 7f2be1534640 1 -- 172.21.15.31:0/3577367649 shutdown_connections 2024-04-25T16:00:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.832+0000 7f2be1534640 1 --2- 172.21.15.31:0/3577367649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc108900 0x7f2bdc10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.832+0000 7f2be1534640 1 --2- 172.21.15.31:0/3577367649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc103c00 0x7f2bdc104010 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.832+0000 7f2be1534640 1 --2- 172.21.15.31:0/3577367649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bdc1029c0 0x7f2bdc1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.832+0000 7f2be1534640 1 -- 172.21.15.31:0/3577367649 >> 172.21.15.31:0/3577367649 conn(0x7f2bdc0fdbd0 msgr2=0x7f2bdc100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:34.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.832+0000 7f2be1534640 1 -- 172.21.15.31:0/3577367649 shutdown_connections 2024-04-25T16:00:34.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.832+0000 7f2be1534640 1 -- 172.21.15.31:0/3577367649 wait complete. 2024-04-25T16:00:34.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.832+0000 7f2be1534640 1 Processor -- start 2024-04-25T16:00:34.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.832+0000 7f2be1534640 1 -- start start 2024-04-25T16:00:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2be1534640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc1029c0 0x7f2bdc06a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2be1534640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bdc103c00 0x7f2bdc19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2be1534640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc108900 0x7f2bdc1a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2be1534640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2bdc10e4c0 con 0x7f2bdc108900 2024-04-25T16:00:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2be1534640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2bdc10e340 con 0x7f2bdc1029c0 2024-04-25T16:00:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2be1534640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2bdc10e640 con 0x7f2bdc103c00 2024-04-25T16:00:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc108900 0x7f2bdc1a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc108900 0x7f2bdc1a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45688/0 (socket says 172.21.15.31:45688) 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdb7fe640 1 -- 172.21.15.31:0/3379898933 learned_addr learned my addr 172.21.15.31:0/3379898933 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdaffd640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc1029c0 0x7f2bdc06a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bda7fc640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bdc103c00 0x7f2bdc19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdb7fe640 1 -- 172.21.15.31:0/3379898933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bdc103c00 msgr2=0x7f2bdc19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdb7fe640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bdc103c00 0x7f2bdc19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdb7fe640 1 -- 172.21.15.31:0/3379898933 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc1029c0 msgr2=0x7f2bdc06a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdb7fe640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc1029c0 0x7f2bdc06a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdb7fe640 1 -- 172.21.15.31:0/3379898933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bdc1a1d40 con 0x7f2bdc108900 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bda7fc640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bdc103c00 0x7f2bdc19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bdb7fe640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc108900 0x7f2bdc1a1520 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00d950 tx=0x7f2bcc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.833+0000 7f2bc3fff640 1 -- 172.21.15.31:0/3379898933 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc017030 con 0x7f2bdc108900 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.834+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bdc1a1fd0 con 0x7f2bdc108900 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.834+0000 7f2bc3fff640 1 -- 172.21.15.31:0/3379898933 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2bcc004020 con 0x7f2bdc108900 2024-04-25T16:00:34.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.834+0000 7f2bc3fff640 1 -- 172.21.15.31:0/3379898933 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc003b50 con 0x7f2bdc108900 2024-04-25T16:00:34.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.834+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bdc1a2440 con 0x7f2bdc108900 2024-04-25T16:00:34.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.835+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bdc107710 con 0x7f2bdc108900 2024-04-25T16:00:34.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.836+0000 7f2bc3fff640 1 -- 172.21.15.31:0/3379898933 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2bcc01a070 con 0x7f2bdc108900 2024-04-25T16:00:34.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.837+0000 7f2bc3fff640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2bc40afaf0 0x7f2bc40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:34.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.837+0000 7f2bdaffd640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2bc40afaf0 0x7f2bc40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:34.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.837+0000 7f2bc3fff640 1 -- 172.21.15.31:0/3379898933 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2bcc0de6c0 con 0x7f2bdc108900 2024-04-25T16:00:34.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.838+0000 7f2bdaffd640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2bc40afaf0 0x7f2bc40b2000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc1035c0 tx=0x7f2bd400a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:34.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:34.841+0000 7f2bc3fff640 1 -- 172.21.15.31:0/3379898933 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2bcc021070 con 0x7f2bdc108900 2024-04-25T16:00:34.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:34 smithi063 ceph-mon[32413]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-25T16:00:34.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:34 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/592166287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:00:34.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:34 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/382836061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:00:34.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:34 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/379456291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:00:34.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:34 smithi186 ceph-mon[33258]: pgmap v329: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-25T16:00:34.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:34 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/592166287' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:00:34.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:34 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/382836061' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:00:34.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:34 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/379456291' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:00:35.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.027+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2bdc02dd20 con 0x7f2bdc108900 2024-04-25T16:00:35.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.029+0000 7f2bc3fff640 1 -- 172.21.15.31:0/3379898933 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f2bcc0a72b0 con 0x7f2bdc108900 2024-04-25T16:00:35.030 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:35.030 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2bc40afaf0 msgr2=0x7f2bc40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2bc40afaf0 0x7f2bc40b2000 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2bdc1035c0 tx=0x7f2bd400a590 comp rx=0 tx=0).stop 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc108900 msgr2=0x7f2bdc1a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc108900 0x7f2bdc1a1520 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc00d950 tx=0x7f2bcc00de10 comp rx=0 tx=0).stop 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 shutdown_connections 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2bc40afaf0 0x7f2bc40b2000 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2bdc108900 0x7f2bdc1a1520 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bdc103c00 0x7f2bdc19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 --2- 172.21.15.31:0/3379898933 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2bdc1029c0 0x7f2bdc06a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 >> 172.21.15.31:0/3379898933 conn(0x7f2bdc0fdbd0 msgr2=0x7f2bdc1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 shutdown_connections 2024-04-25T16:00:35.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.031+0000 7f2be1534640 1 -- 172.21.15.31:0/3379898933 wait complete. 2024-04-25T16:00:35.044 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:59:00.978038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:59:01.829535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:00:35.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:00:35.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.158+0000 7fb66812e640 1 -- 172.21.15.31:0/3579451896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 msgr2=0x7fb66010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.158+0000 7fb66812e640 1 --2- 172.21.15.31:0/3579451896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 0x7fb66010ad50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb658009ca0 tx=0x7fb65802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.158+0000 7fb66812e640 1 -- 172.21.15.31:0/3579451896 shutdown_connections 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.158+0000 7fb66812e640 1 --2- 172.21.15.31:0/3579451896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 0x7fb66010ad50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.158+0000 7fb66812e640 1 --2- 172.21.15.31:0/3579451896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb660103c30 0x7fb660104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.158+0000 7fb66812e640 1 --2- 172.21.15.31:0/3579451896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb6601029f0 0x7fb6601036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.158+0000 7fb66812e640 1 -- 172.21.15.31:0/3579451896 >> 172.21.15.31:0/3579451896 conn(0x7fb6600fdbc0 msgr2=0x7fb660100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 -- 172.21.15.31:0/3579451896 shutdown_connections 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 -- 172.21.15.31:0/3579451896 wait complete. 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 Processor -- start 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 -- start start 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6601029f0 0x7fb66019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb660103c30 0x7fb66019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 0x7fb6601a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb66010f620 con 0x7fb660108930 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb66010f4a0 con 0x7fb660103c30 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.159+0000 7fb66812e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb66010f7a0 con 0x7fb6601029f0 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6666cb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 0x7fb6601a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6666cb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 0x7fb6601a1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45698/0 (socket says 172.21.15.31:45698) 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6666cb640 1 -- 172.21.15.31:0/1390478326 learned_addr learned my addr 172.21.15.31:0/1390478326 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6656c9640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb660103c30 0x7fb66019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb665eca640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6601029f0 0x7fb66019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6666cb640 1 -- 172.21.15.31:0/1390478326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6601029f0 msgr2=0x7fb66019a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6666cb640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6601029f0 0x7fb66019a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6666cb640 1 -- 172.21.15.31:0/1390478326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb660103c30 msgr2=0x7fb66019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6666cb640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb660103c30 0x7fb66019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6666cb640 1 -- 172.21.15.31:0/1390478326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6601a1c30 con 0x7fb660108930 2024-04-25T16:00:35.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6656c9640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb660103c30 0x7fb66019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:35.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb665eca640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6601029f0 0x7fb66019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:35.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb6666cb640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 0x7fb6601a1670 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb6580097b0 tx=0x7fb658003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:35.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb656ffd640 1 -- 172.21.15.31:0/1390478326 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb658004240 con 0x7fb660108930 2024-04-25T16:00:35.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb656ffd640 1 -- 172.21.15.31:0/1390478326 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6580317c0 con 0x7fb660108930 2024-04-25T16:00:35.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6601a1ec0 con 0x7fb660108930 2024-04-25T16:00:35.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb656ffd640 1 -- 172.21.15.31:0/1390478326 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65802fde0 con 0x7fb660108930 2024-04-25T16:00:35.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.160+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6601a2320 con 0x7fb660108930 2024-04-25T16:00:35.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.161+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb66006a190 con 0x7fb660108930 2024-04-25T16:00:35.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.163+0000 7fb656ffd640 1 -- 172.21.15.31:0/1390478326 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb6580043e0 con 0x7fb660108930 2024-04-25T16:00:35.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.164+0000 7fb656ffd640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb64c0afaf0 0x7fb64c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.164+0000 7fb665eca640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb64c0afaf0 0x7fb64c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.164+0000 7fb656ffd640 1 -- 172.21.15.31:0/1390478326 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb658100970 con 0x7fb660108930 2024-04-25T16:00:35.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.164+0000 7fb665eca640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb64c0afaf0 0x7fb64c0b2000 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb6601035f0 tx=0x7fb65c00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:35.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.168+0000 7fb656ffd640 1 -- 172.21.15.31:0/1390478326 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb66006a190 con 0x7fb660108930 2024-04-25T16:00:35.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.355+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb66002dc10 con 0x7fb660108930 2024-04-25T16:00:35.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.355+0000 7fb656ffd640 1 -- 172.21.15.31:0/1390478326 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7fb6580c9500 con 0x7fb660108930 2024-04-25T16:00:35.359 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T16:00:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb64c0afaf0 msgr2=0x7fb64c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb64c0afaf0 0x7fb64c0b2000 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb6601035f0 tx=0x7fb65c00a560 comp rx=0 tx=0).stop 2024-04-25T16:00:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 msgr2=0x7fb6601a1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 0x7fb6601a1670 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb6580097b0 tx=0x7fb658003c20 comp rx=0 tx=0).stop 2024-04-25T16:00:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 shutdown_connections 2024-04-25T16:00:35.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb64c0afaf0 0x7fb64c0b2000 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb660108930 0x7fb6601a1670 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb660103c30 0x7fb66019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 --2- 172.21.15.31:0/1390478326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6601029f0 0x7fb66019a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 >> 172.21.15.31:0/1390478326 conn(0x7fb6600fdbc0 msgr2=0x7fb660104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 shutdown_connections 2024-04-25T16:00:35.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.357+0000 7fb66812e640 1 -- 172.21.15.31:0/1390478326 wait complete. 2024-04-25T16:00:35.370 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:00:35.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.485+0000 7f42fe60c640 1 -- 172.21.15.31:0/2342477454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f8108920 msgr2=0x7f42f810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.485+0000 7f42fe60c640 1 --2- 172.21.15.31:0/2342477454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f8108920 0x7f42f810ad40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f42f0009ca0 tx=0x7f42f002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.485+0000 7f42fe60c640 1 -- 172.21.15.31:0/2342477454 shutdown_connections 2024-04-25T16:00:35.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.485+0000 7f42fe60c640 1 --2- 172.21.15.31:0/2342477454 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f8108920 0x7f42f810ad40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.485+0000 7f42fe60c640 1 --2- 172.21.15.31:0/2342477454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8103c20 0x7f42f8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.485+0000 7f42fe60c640 1 --2- 172.21.15.31:0/2342477454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42f81029e0 0x7f42f81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.485+0000 7f42fe60c640 1 -- 172.21.15.31:0/2342477454 >> 172.21.15.31:0/2342477454 conn(0x7f42f80fdbd0 msgr2=0x7f42f8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.485+0000 7f42fe60c640 1 -- 172.21.15.31:0/2342477454 shutdown_connections 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.485+0000 7f42fe60c640 1 -- 172.21.15.31:0/2342477454 wait complete. 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42fe60c640 1 Processor -- start 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42fe60c640 1 -- start start 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42fe60c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f81029e0 0x7f42f819a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42fe60c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42f8103c20 0x7f42f819a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42fe60c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8108920 0x7f42f81a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42fe60c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f42f810e4f0 con 0x7f42f81029e0 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42fe60c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f42f810e370 con 0x7f42f8103c20 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42fe60c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f42f810e670 con 0x7f42f8108920 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42f7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f81029e0 0x7f42f819a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42f7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f81029e0 0x7f42f819a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45722/0 (socket says 172.21.15.31:45722) 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42f7fff640 1 -- 172.21.15.31:0/637818239 learned_addr learned my addr 172.21.15.31:0/637818239 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.486+0000 7f42fcba9640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8108920 0x7f42f81a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f77fe640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42f8103c20 0x7f42f819a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f7fff640 1 -- 172.21.15.31:0/637818239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8108920 msgr2=0x7f42f81a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f7fff640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8108920 0x7f42f81a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f7fff640 1 -- 172.21.15.31:0/637818239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42f8103c20 msgr2=0x7f42f819a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f7fff640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42f8103c20 0x7f42f819a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f7fff640 1 -- 172.21.15.31:0/637818239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f81a1d60 con 0x7f42f81029e0 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f7fff640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f81029e0 0x7f42f819a220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f42e800d740 tx=0x7f42e800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f57fa640 1 -- 172.21.15.31:0/637818239 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8017070 con 0x7f42f81029e0 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f57fa640 1 -- 172.21.15.31:0/637818239 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42e800de50 con 0x7f42f81029e0 2024-04-25T16:00:35.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f81a1ff0 con 0x7f42f81029e0 2024-04-25T16:00:35.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42f57fa640 1 -- 172.21.15.31:0/637818239 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e8003c20 con 0x7f42f81029e0 2024-04-25T16:00:35.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.487+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f81a24b0 con 0x7f42f81029e0 2024-04-25T16:00:35.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.488+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f81a2710 con 0x7f42f81029e0 2024-04-25T16:00:35.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.489+0000 7f42f57fa640 1 -- 172.21.15.31:0/637818239 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f42e8003dc0 con 0x7f42f81029e0 2024-04-25T16:00:35.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.490+0000 7f42f57fa640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f42e00afaf0 0x7f42e00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.490+0000 7f42f77fe640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f42e00afaf0 0x7f42e00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.490+0000 7f42f57fa640 1 -- 172.21.15.31:0/637818239 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f42e80de660 con 0x7f42f81029e0 2024-04-25T16:00:35.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.491+0000 7f42f77fe640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f42e00afaf0 0x7f42e00b2000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f42ec005fd0 tx=0x7f42ec009490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:35.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.495+0000 7f42f57fa640 1 -- 172.21.15.31:0/637818239 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f42e80a71f0 con 0x7f42f81029e0 2024-04-25T16:00:35.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.680+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f42f81a2a00 con 0x7f42f81029e0 2024-04-25T16:00:35.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.681+0000 7f42f57fa640 1 -- 172.21.15.31:0/637818239 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7f42e80af270 con 0x7f42f81029e0 2024-04-25T16:00:35.682 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f42e00afaf0 msgr2=0x7f42e00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f42e00afaf0 0x7f42e00b2000 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f42ec005fd0 tx=0x7f42ec009490 comp rx=0 tx=0).stop 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f81029e0 msgr2=0x7f42f819a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f81029e0 0x7f42f819a220 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f42e800d740 tx=0x7f42e800dc00 comp rx=0 tx=0).stop 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 shutdown_connections 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f42e00afaf0 0x7f42e00b2000 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42f8108920 0x7f42f81a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f42f8103c20 0x7f42f819a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 --2- 172.21.15.31:0/637818239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f42f81029e0 0x7f42f819a220 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 >> 172.21.15.31:0/637818239 conn(0x7f42f80fdbd0 msgr2=0x7f42f80ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 shutdown_connections 2024-04-25T16:00:35.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.683+0000 7f42fe60c640 1 -- 172.21.15.31:0/637818239 wait complete. 2024-04-25T16:00:35.694 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:35 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/843702014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:00:35.694 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:35 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3379898933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:00:35.694 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:35 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1390478326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:00:35.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:00:35.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 -- 172.21.15.31:0/284575959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 msgr2=0x7f771010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 --2- 172.21.15.31:0/284575959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 0x7f771010ad40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7708009ca0 tx=0x7f770802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 -- 172.21.15.31:0/284575959 shutdown_connections 2024-04-25T16:00:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 --2- 172.21.15.31:0/284575959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 0x7f771010ad40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 --2- 172.21.15.31:0/284575959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7710103c20 0x7f7710104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 --2- 172.21.15.31:0/284575959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77101029e0 0x7f77101036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 -- 172.21.15.31:0/284575959 >> 172.21.15.31:0/284575959 conn(0x7f77100fdbd0 msgr2=0x7f7710100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 -- 172.21.15.31:0/284575959 shutdown_connections 2024-04-25T16:00:35.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 -- 172.21.15.31:0/284575959 wait complete. 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 Processor -- start 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.809+0000 7f7716a0c640 1 -- start start 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7716a0c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77101029e0 0x7f77101993f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7716a0c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7710103c20 0x7f7710199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7716a0c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 0x7f77101a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7716a0c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f771010e4f0 con 0x7f7710108920 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7716a0c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f771010e370 con 0x7f77101029e0 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7716a0c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f771010e670 con 0x7f7710103c20 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7714fa9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 0x7f77101a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7714fa9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 0x7f77101a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45746/0 (socket says 172.21.15.31:45746) 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7714fa9640 1 -- 172.21.15.31:0/1932559865 learned_addr learned my addr 172.21.15.31:0/1932559865 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f770ffff640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77101029e0 0x7f77101993f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f770f7fe640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7710103c20 0x7f7710199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7714fa9640 1 -- 172.21.15.31:0/1932559865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7710103c20 msgr2=0x7f7710199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7714fa9640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7710103c20 0x7f7710199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7714fa9640 1 -- 172.21.15.31:0/1932559865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77101029e0 msgr2=0x7f77101993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7714fa9640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77101029e0 0x7f77101993f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7714fa9640 1 -- 172.21.15.31:0/1932559865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77101a0e80 con 0x7f7710108920 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f7714fa9640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 0x7f77101a08c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f77080097b0 tx=0x7f7708003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.810+0000 7f770d7fa640 1 -- 172.21.15.31:0/1932559865 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770804a070 con 0x7f7710108920 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.811+0000 7f770d7fa640 1 -- 172.21.15.31:0/1932559865 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77080317c0 con 0x7f7710108920 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.811+0000 7f770d7fa640 1 -- 172.21.15.31:0/1932559865 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770802fcc0 con 0x7f7710108920 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.811+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7710069e90 con 0x7f7710108920 2024-04-25T16:00:35.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.811+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f771006a430 con 0x7f7710108920 2024-04-25T16:00:35.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.812+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f771006c2a0 con 0x7f7710108920 2024-04-25T16:00:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.813+0000 7f770d7fa640 1 -- 172.21.15.31:0/1932559865 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f770804d070 con 0x7f7710108920 2024-04-25T16:00:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.814+0000 7f770d7fa640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f76f80afb70 0x7f76f80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.814+0000 7f770d7fa640 1 -- 172.21.15.31:0/1932559865 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7708101860 con 0x7f7710108920 2024-04-25T16:00:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.814+0000 7f770ffff640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f76f80afb70 0x7f76f80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:35.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.814+0000 7f770ffff640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f76f80afb70 0x7f76f80b2080 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f77101035e0 tx=0x7f7700005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:35.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:35.818+0000 7f770d7fa640 1 -- 172.21.15.31:0/1932559865 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7708081ff0 con 0x7f7710108920 2024-04-25T16:00:35.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:35 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/843702014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:00:35.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:35 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3379898933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:00:35.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:35 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1390478326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:00:35.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:35 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/843702014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:00:35.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:35 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3379898933' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:00:35.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:35 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1390478326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:00:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.003+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f77101074a0 con 0x7f7710108920 2024-04-25T16:00:36.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.004+0000 7f770d7fa640 1 -- 172.21.15.31:0/1932559865 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7f77080ca450 con 0x7f7710108920 2024-04-25T16:00:36.004 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T16:00:36.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f76f80afb70 msgr2=0x7f76f80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f76f80afb70 0x7f76f80b2080 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f77101035e0 tx=0x7f7700005f90 comp rx=0 tx=0).stop 2024-04-25T16:00:36.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 msgr2=0x7f77101a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 0x7f77101a08c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f77080097b0 tx=0x7f7708003ba0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 shutdown_connections 2024-04-25T16:00:36.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f76f80afb70 0x7f76f80b2080 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7710108920 0x7f77101a08c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7710103c20 0x7f7710199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 --2- 172.21.15.31:0/1932559865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f77101029e0 0x7f77101993f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 >> 172.21.15.31:0/1932559865 conn(0x7f77100fdbd0 msgr2=0x7f7710104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:36.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 shutdown_connections 2024-04-25T16:00:36.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.006+0000 7f7716a0c640 1 -- 172.21.15.31:0/1932559865 wait complete. 2024-04-25T16:00:36.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:00:36.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.129+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/1486915531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24103b00 msgr2=0x7f8a24104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.129+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/1486915531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24103b00 0x7f8a24104800 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009900 tx=0x7f8a1802a410 comp rx=0 tx=0).stop 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.129+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/1486915531 shutdown_connections 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.129+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/1486915531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a24109a40 0x7f8a2410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.129+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/1486915531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a24104d40 0x7f8a24105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.129+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/1486915531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24103b00 0x7f8a24104800 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.129+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/1486915531 >> 172.21.15.31:0/1486915531 conn(0x7f8a240fed10 msgr2=0x7f8a241011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.129+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/1486915531 shutdown_connections 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.129+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/1486915531 wait complete. 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2ab8f640 1 Processor -- start 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2ab8f640 1 -- start start 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2ab8f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a24103b00 0x7f8a2419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2ab8f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a24104d40 0x7f8a2419a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2ab8f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24109a40 0x7f8a241a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2ab8f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2410f600 con 0x7f8a24109a40 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2ab8f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2410f480 con 0x7f8a24103b00 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2ab8f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2410f780 con 0x7f8a24104d40 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2912c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24109a40 0x7f8a241a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2912c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24109a40 0x7f8a241a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45758/0 (socket says 172.21.15.31:45758) 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2912c640 1 -- 172.21.15.31:0/691491239 learned_addr learned my addr 172.21.15.31:0/691491239 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a2892b640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a24103b00 0x7f8a2419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.130+0000 7f8a23fff640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a24104d40 0x7f8a2419a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a2912c640 1 -- 172.21.15.31:0/691491239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a24104d40 msgr2=0x7f8a2419a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a2912c640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a24104d40 0x7f8a2419a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a2912c640 1 -- 172.21.15.31:0/691491239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a24103b00 msgr2=0x7f8a2419a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a2912c640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a24103b00 0x7f8a2419a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a2912c640 1 -- 172.21.15.31:0/691491239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a241a1da0 con 0x7f8a24109a40 2024-04-25T16:00:36.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a2892b640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a24103b00 0x7f8a2419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:36.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a2912c640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24109a40 0x7f8a241a1730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00eb10 tx=0x7f8a1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:36.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a21ffb640 1 -- 172.21.15.31:0/691491239 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c004b20 con 0x7f8a24109a40 2024-04-25T16:00:36.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a241a2090 con 0x7f8a24109a40 2024-04-25T16:00:36.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a21ffb640 1 -- 172.21.15.31:0/691491239 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a1c004cc0 con 0x7f8a24109a40 2024-04-25T16:00:36.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a21ffb640 1 -- 172.21.15.31:0/691491239 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c010670 con 0x7f8a24109a40 2024-04-25T16:00:36.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.131+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a241a2590 con 0x7f8a24109a40 2024-04-25T16:00:36.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.132+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a24108850 con 0x7f8a24109a40 2024-04-25T16:00:36.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.133+0000 7f8a21ffb640 1 -- 172.21.15.31:0/691491239 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8a1c005020 con 0x7f8a24109a40 2024-04-25T16:00:36.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.134+0000 7f8a21ffb640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a140afaf0 0x7f8a140b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.134+0000 7f8a2892b640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a140afaf0 0x7f8a140b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.134+0000 7f8a21ffb640 1 -- 172.21.15.31:0/691491239 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8a1c0df370 con 0x7f8a24109a40 2024-04-25T16:00:36.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.135+0000 7f8a2892b640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a140afaf0 0x7f8a140b2000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8a18005f70 tx=0x7f8a18031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:36.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.139+0000 7f8a21ffb640 1 -- 172.21.15.31:0/691491239 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8a1c0a7f00 con 0x7f8a24109a40 2024-04-25T16:00:36.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.362+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a2402db50 con 0x7f8a24109a40 2024-04-25T16:00:36.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.363+0000 7f8a21ffb640 1 -- 172.21.15.31:0/691491239 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11917 (secure 0 0 0) 0x7f8a1c0a7800 con 0x7f8a24109a40 2024-04-25T16:00:36.364 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:36.364 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T15:59:10.264996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[14631,24553,24563,24521,24535]},"id":1}]} 2024-04-25T16:00:36.364 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T16:00:36.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.365+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a140afaf0 msgr2=0x7f8a140b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.365+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a140afaf0 0x7f8a140b2000 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8a18005f70 tx=0x7f8a18031400 comp rx=0 tx=0).stop 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.365+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24109a40 msgr2=0x7f8a241a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.365+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24109a40 0x7f8a241a1730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00eb10 tx=0x7f8a1c00c2d0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.366+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 shutdown_connections 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.366+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a140afaf0 0x7f8a140b2000 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.366+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a24109a40 0x7f8a241a1730 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.366+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a24104d40 0x7f8a2419a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.366+0000 7f8a2ab8f640 1 --2- 172.21.15.31:0/691491239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a24103b00 0x7f8a2419a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.366+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 >> 172.21.15.31:0/691491239 conn(0x7f8a240fed10 msgr2=0x7f8a24105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.366+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 shutdown_connections 2024-04-25T16:00:36.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.366+0000 7f8a2ab8f640 1 -- 172.21.15.31:0/691491239 wait complete. 2024-04-25T16:00:36.378 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:00:36.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.491+0000 7f19bfb8f640 1 -- 172.21.15.31:0/615217167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8108e80 msgr2=0x7f19b810b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.491+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/615217167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8108e80 0x7f19b810b320 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f19a8009900 tx=0x7f19a802a410 comp rx=0 tx=0).stop 2024-04-25T16:00:36.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.491+0000 7f19bfb8f640 1 -- 172.21.15.31:0/615217167 shutdown_connections 2024-04-25T16:00:36.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.491+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/615217167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19b810b9b0 0x7f19b810dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.491+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/615217167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8108e80 0x7f19b810b320 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.491+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/615217167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19b8103fd0 0x7f19b81043e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.491+0000 7f19bfb8f640 1 -- 172.21.15.31:0/615217167 >> 172.21.15.31:0/615217167 conn(0x7f19b80fd370 msgr2=0x7f19b80ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:36.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.491+0000 7f19bfb8f640 1 -- 172.21.15.31:0/615217167 shutdown_connections 2024-04-25T16:00:36.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.491+0000 7f19bfb8f640 1 -- 172.21.15.31:0/615217167 wait complete. 2024-04-25T16:00:36.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bfb8f640 1 Processor -- start 2024-04-25T16:00:36.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bfb8f640 1 -- start start 2024-04-25T16:00:36.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bfb8f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8103fd0 0x7f19b819c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bfb8f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19b8108e80 0x7f19b819c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bfb8f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19b810b9b0 0x7f19b81a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bfb8f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f19b8111830 con 0x7f19b8103fd0 2024-04-25T16:00:36.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bfb8f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f19b81116b0 con 0x7f19b810b9b0 2024-04-25T16:00:36.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bfb8f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f19b81119b0 con 0x7f19b8108e80 2024-04-25T16:00:36.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bd92b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8103fd0 0x7f19b819c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bd92b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8103fd0 0x7f19b819c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45774/0 (socket says 172.21.15.31:45774) 2024-04-25T16:00:36.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bd92b640 1 -- 172.21.15.31:0/1733652766 learned_addr learned my addr 172.21.15.31:0/1733652766 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:36.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bd12a640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19b8108e80 0x7f19b819c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19be12c640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19b810b9b0 0x7f19b81a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bd92b640 1 -- 172.21.15.31:0/1733652766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19b8108e80 msgr2=0x7f19b819c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bd92b640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19b8108e80 0x7f19b819c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bd92b640 1 -- 172.21.15.31:0/1733652766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19b810b9b0 msgr2=0x7f19b81a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bd92b640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19b810b9b0 0x7f19b81a38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19bd92b640 1 -- 172.21.15.31:0/1733652766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19b81a3f40 con 0x7f19b8103fd0 2024-04-25T16:00:36.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.492+0000 7f19be12c640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19b810b9b0 0x7f19b81a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:36.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.493+0000 7f19bd92b640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8103fd0 0x7f19b819c400 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f19b400d740 tx=0x7f19b400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:36.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.493+0000 7f19aeffd640 1 -- 172.21.15.31:0/1733652766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4017070 con 0x7f19b8103fd0 2024-04-25T16:00:36.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.493+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19b81a4230 con 0x7f19b8103fd0 2024-04-25T16:00:36.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.493+0000 7f19aeffd640 1 -- 172.21.15.31:0/1733652766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b400de50 con 0x7f19b8103fd0 2024-04-25T16:00:36.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.493+0000 7f19aeffd640 1 -- 172.21.15.31:0/1733652766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b400f620 con 0x7f19b8103fd0 2024-04-25T16:00:36.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.493+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19b81a4780 con 0x7f19b8103fd0 2024-04-25T16:00:36.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.494+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19b80fee90 con 0x7f19b8103fd0 2024-04-25T16:00:36.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.496+0000 7f19aeffd640 1 -- 172.21.15.31:0/1733652766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f19b401a070 con 0x7f19b8103fd0 2024-04-25T16:00:36.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.497+0000 7f19aeffd640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f19a40afaf0 0x7f19a40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.497+0000 7f19bd12a640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f19a40afaf0 0x7f19a40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.497+0000 7f19aeffd640 1 -- 172.21.15.31:0/1733652766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f19b40de670 con 0x7f19b8103fd0 2024-04-25T16:00:36.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.497+0000 7f19bd12a640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f19a40afaf0 0x7f19a40b2000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f19a8009230 tx=0x7f19a8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:36.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.504+0000 7f19aeffd640 1 -- 172.21.15.31:0/1733652766 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f19b40a7200 con 0x7f19b8103fd0 2024-04-25T16:00:36.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.693+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19b802dd40 con 0x7f19b8103fd0 2024-04-25T16:00:36.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.694+0000 7f19aeffd640 1 -- 172.21.15.31:0/1733652766 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f19b40a6b00 con 0x7f19b8103fd0 2024-04-25T16:00:36.696 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:36.696 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:00:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f19a40afaf0 msgr2=0x7f19a40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f19a40afaf0 0x7f19a40b2000 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f19a8009230 tx=0x7f19a8031400 comp rx=0 tx=0).stop 2024-04-25T16:00:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8103fd0 msgr2=0x7f19b819c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8103fd0 0x7f19b819c400 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f19b400d740 tx=0x7f19b400dc00 comp rx=0 tx=0).stop 2024-04-25T16:00:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 shutdown_connections 2024-04-25T16:00:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f19a40afaf0 0x7f19a40b2000 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f19b810b9b0 0x7f19b81a38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19b8108e80 0x7f19b819c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 --2- 172.21.15.31:0/1733652766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f19b8103fd0 0x7f19b819c400 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 >> 172.21.15.31:0/1733652766 conn(0x7f19b80fd370 msgr2=0x7f19b810c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 shutdown_connections 2024-04-25T16:00:36.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.697+0000 7f19bfb8f640 1 -- 172.21.15.31:0/1733652766 wait complete. 2024-04-25T16:00:36.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:36 smithi031 ceph-mon[29140]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T16:00:36.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:36 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/637818239' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:00:36.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:36 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1932559865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:00:36.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:36 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/691491239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:00:36.710 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:00:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 -- 172.21.15.31:0/1955482587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac1029e0 msgr2=0x7f65ac1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 --2- 172.21.15.31:0/1955482587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac1029e0 0x7f65ac1036e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f659c009900 tx=0x7f659c02a410 comp rx=0 tx=0).stop 2024-04-25T16:00:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 -- 172.21.15.31:0/1955482587 shutdown_connections 2024-04-25T16:00:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 --2- 172.21.15.31:0/1955482587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65ac108920 0x7f65ac10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 --2- 172.21.15.31:0/1955482587 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac103c20 0x7f65ac104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 --2- 172.21.15.31:0/1955482587 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac1029e0 0x7f65ac1036e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 -- 172.21.15.31:0/1955482587 >> 172.21.15.31:0/1955482587 conn(0x7f65ac0fdbd0 msgr2=0x7f65ac100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 -- 172.21.15.31:0/1955482587 shutdown_connections 2024-04-25T16:00:36.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 -- 172.21.15.31:0/1955482587 wait complete. 2024-04-25T16:00:36.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.821+0000 7f65b17f8640 1 Processor -- start 2024-04-25T16:00:36.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65b17f8640 1 -- start start 2024-04-25T16:00:36.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65b17f8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65ac1029e0 0x7f65ac199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65b17f8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac103c20 0x7f65ac1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65b17f8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac108920 0x7f65ac1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65b17f8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac10e4e0 con 0x7f65ac108920 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65b17f8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac10e360 con 0x7f65ac103c20 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65b17f8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac10e660 con 0x7f65ac1029e0 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac108920 0x7f65ac1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65ab7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac108920 0x7f65ac1a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45790/0 (socket says 172.21.15.31:45790) 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65ab7fe640 1 -- 172.21.15.31:0/3620016299 learned_addr learned my addr 172.21.15.31:0/3620016299 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65a27fc640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac103c20 0x7f65ac1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65aaffd640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65ac1029e0 0x7f65ac199180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65ab7fe640 1 -- 172.21.15.31:0/3620016299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65ac1029e0 msgr2=0x7f65ac199180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65ab7fe640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65ac1029e0 0x7f65ac199180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65ab7fe640 1 -- 172.21.15.31:0/3620016299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac103c20 msgr2=0x7f65ac1996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65ab7fe640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac103c20 0x7f65ac1996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65ab7fe640 1 -- 172.21.15.31:0/3620016299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ac1a0c10 con 0x7f65ac108920 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.822+0000 7f65ab7fe640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac108920 0x7f65ac1a0650 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f65a400eaa0 tx=0x7f65a400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.823+0000 7f65a8ff9640 1 -- 172.21.15.31:0/3620016299 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a4004d20 con 0x7f65ac108920 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.823+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ac1a0f00 con 0x7f65ac108920 2024-04-25T16:00:36.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.823+0000 7f65a8ff9640 1 -- 172.21.15.31:0/3620016299 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65a4005020 con 0x7f65ac108920 2024-04-25T16:00:36.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.823+0000 7f65a8ff9640 1 -- 172.21.15.31:0/3620016299 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a40116d0 con 0x7f65ac108920 2024-04-25T16:00:36.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.823+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ac1a1360 con 0x7f65ac108920 2024-04-25T16:00:36.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.824+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ac107730 con 0x7f65ac108920 2024-04-25T16:00:36.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.825+0000 7f65a8ff9640 1 -- 172.21.15.31:0/3620016299 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f65a400caf0 con 0x7f65ac108920 2024-04-25T16:00:36.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.826+0000 7f65a8ff9640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f65980afaf0 0x7f65980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:36.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.826+0000 7f65aaffd640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f65980afaf0 0x7f65980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:36.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.826+0000 7f65a8ff9640 1 -- 172.21.15.31:0/3620016299 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f65a40df0b0 con 0x7f65ac108920 2024-04-25T16:00:36.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.826+0000 7f65aaffd640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f65980afaf0 0x7f65980b2000 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f659c009230 tx=0x7f659c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:36.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:36.829+0000 7f65a8ff9640 1 -- 172.21.15.31:0/3620016299 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f65a4016030 con 0x7f65ac108920 2024-04-25T16:00:36.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:36 smithi063 ceph-mon[32413]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T16:00:36.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:36 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/637818239' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:00:36.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:36 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1932559865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:00:36.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:36 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/691491239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:00:36.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:36 smithi186 ceph-mon[33258]: pgmap v330: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T16:00:36.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:36 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/637818239' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:00:36.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:36 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1932559865' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:00:36.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:36 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/691491239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:00:37.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.065+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f65ac02db50 con 0x7f65ac108920 2024-04-25T16:00:37.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.065+0000 7f65a8ff9640 1 -- 172.21.15.31:0/3620016299 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11917 (secure 0 0 0) 0x7f65a40a7ca0 con 0x7f65ac108920 2024-04-25T16:00:37.066 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:37.067 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T15:59:10.264996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[14631,24553,24563,24521,24535]},"id":1}]} 2024-04-25T16:00:37.067 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T16:00:37.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f65980afaf0 msgr2=0x7f65980b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f65980afaf0 0x7f65980b2000 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f659c009230 tx=0x7f659c031400 comp rx=0 tx=0).stop 2024-04-25T16:00:37.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac108920 msgr2=0x7f65ac1a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac108920 0x7f65ac1a0650 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f65a400eaa0 tx=0x7f65a400eea0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 shutdown_connections 2024-04-25T16:00:37.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f65980afaf0 0x7f65980b2000 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f65ac108920 0x7f65ac1a0650 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f65ac103c20 0x7f65ac1996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 --2- 172.21.15.31:0/3620016299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65ac1029e0 0x7f65ac199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 >> 172.21.15.31:0/3620016299 conn(0x7f65ac0fdbd0 msgr2=0x7f65ac104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 shutdown_connections 2024-04-25T16:00:37.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.069+0000 7f65b17f8640 1 -- 172.21.15.31:0/3620016299 wait complete. 2024-04-25T16:00:37.088 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T15:59:02.905418+0000', 'modified': '2024-04-25T15:59:10.264996+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24563, 'mds_1': 24553, 'mds_2': 24535, 'mds_3': 24521, 'mds_4': 14631}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14613': {'gid': 14613, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.31:6839/3019802236', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6838', 'nonce': 3019802236}, {'type': 'v1', 'addr': '172.21.15.31:6839', 'nonce': 3019802236}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.31:6841/2149984264', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6840', 'nonce': 2149984264}, {'type': 'v1', 'addr': '172.21.15.31:6841', 'nonce': 2149984264}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6833/927313408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6832', 'nonce': 927313408}, {'type': 'v1', 'addr': '172.21.15.63:6833', 'nonce': 927313408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6833/1701597397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6832', 'nonce': 1701597397}, {'type': 'v1', 'addr': '172.21.15.186:6833', 'nonce': 1701597397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6835/1584006856', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6834', 'nonce': 1584006856}, {'type': 'v1', 'addr': '172.21.15.63:6835', 'nonce': 1584006856}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6835/1885074975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 1885074975}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 1885074975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'h', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.63:6837/546178402', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6836', 'nonce': 546178402}, {'type': 'v1', 'addr': '172.21.15.63:6837', 'nonce': 546178402}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.186:6837/509868288', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 509868288}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 509868288}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.186:6839/2846674462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 2846674462}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 2846674462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24563': {'gid': 24563, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.63:6839/495769167', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6838', 'nonce': 495769167}, {'type': 'v1', 'addr': '172.21.15.63:6839', 'nonce': 495769167}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24563, 'qdb_cluster': [14631, 24553, 24563, 24521, 24535]} 2024-04-25T16:00:37.088 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T16:00:37.088 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T16:00:37.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 -- 172.21.15.31:0/4280957243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 msgr2=0x7f415c104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 --2- 172.21.15.31:0/4280957243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 0x7f415c104800 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f414c009ca0 tx=0x7f414c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 -- 172.21.15.31:0/4280957243 shutdown_connections 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 --2- 172.21.15.31:0/4280957243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f415c109a40 0x7f415c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 --2- 172.21.15.31:0/4280957243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f415c104d40 0x7f415c105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 --2- 172.21.15.31:0/4280957243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 0x7f415c104800 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 -- 172.21.15.31:0/4280957243 >> 172.21.15.31:0/4280957243 conn(0x7f415c0fecf0 msgr2=0x7f415c1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 -- 172.21.15.31:0/4280957243 shutdown_connections 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 -- 172.21.15.31:0/4280957243 wait complete. 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 Processor -- start 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.201+0000 7f416409f640 1 -- start start 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416409f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 0x7f415c076850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416409f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f415c104d40 0x7f415c074ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416409f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f415c109a40 0x7f415c075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416409f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f415c10f5e0 con 0x7f415c103b00 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416409f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f415c10f460 con 0x7f415c109a40 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416409f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f415c10f760 con 0x7f415c104d40 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f4161e3b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 0x7f415c076850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f4161e3b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 0x7f415c076850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45810/0 (socket says 172.21.15.31:45810) 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f4161e3b640 1 -- 172.21.15.31:0/3887999822 learned_addr learned my addr 172.21.15.31:0/3887999822 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416263c640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f415c109a40 0x7f415c075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416163a640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f415c104d40 0x7f415c074ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f4161e3b640 1 -- 172.21.15.31:0/3887999822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f415c104d40 msgr2=0x7f415c074ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f4161e3b640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f415c104d40 0x7f415c074ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f4161e3b640 1 -- 172.21.15.31:0/3887999822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f415c109a40 msgr2=0x7f415c075430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f4161e3b640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f415c109a40 0x7f415c075430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f4161e3b640 1 -- 172.21.15.31:0/3887999822 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f415c075ad0 con 0x7f415c103b00 2024-04-25T16:00:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416263c640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f415c109a40 0x7f415c075430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f4161e3b640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 0x7f415c076850 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f414c033040 tx=0x7f414c003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f415affd640 1 -- 172.21.15.31:0/3887999822 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c004240 con 0x7f415c103b00 2024-04-25T16:00:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416409f640 1 -- 172.21.15.31:0/3887999822 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f415c1a4270 con 0x7f415c103b00 2024-04-25T16:00:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f415affd640 1 -- 172.21.15.31:0/3887999822 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f414c003e40 con 0x7f415c103b00 2024-04-25T16:00:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f415affd640 1 -- 172.21.15.31:0/3887999822 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c02fde0 con 0x7f415c103b00 2024-04-25T16:00:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.202+0000 7f416409f640 1 -- 172.21.15.31:0/3887999822 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f415c1a4620 con 0x7f415c103b00 2024-04-25T16:00:37.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.205+0000 7f416b02d740 1 Processor -- start 2024-04-25T16:00:37.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416b02d740 1 -- start start 2024-04-25T16:00:37.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416b02d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8db2270 0x55cca9153370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416b02d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8d893b0 0x55cca91538f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416b02d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 0x55cca915a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416b02d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cca90b8740 con 0x55cca8db2270 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416b02d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cca90543a0 con 0x55cca8d893b0 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416b02d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cca90b4700 con 0x55cca8d91590 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416263c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8db2270 0x55cca9153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f415affd640 1 -- 172.21.15.31:0/3887999822 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f414c0043e0 con 0x7f415c103b00 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416263c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8db2270 0x55cca9153370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45812/0 (socket says 172.21.15.31:45812) 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416263c640 1 -- 172.21.15.31:0/2300693058 learned_addr learned my addr 172.21.15.31:0/2300693058 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416163a640 1 --2- 172.21.15.31:0/2300693058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 0x55cca915a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416263c640 1 -- 172.21.15.31:0/2300693058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 msgr2=0x55cca915a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416263c640 1 --2- 172.21.15.31:0/2300693058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 0x55cca915a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416263c640 1 -- 172.21.15.31:0/2300693058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8d893b0 msgr2=0x55cca91538f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416263c640 1 --2- 172.21.15.31:0/2300693058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8d893b0 0x55cca91538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.206+0000 7f416263c640 1 -- 172.21.15.31:0/2300693058 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cca9070750 con 0x55cca8db2270 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.207+0000 7f416263c640 1 --2- 172.21.15.31:0/2300693058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8db2270 0x55cca9153370 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f415400ac90 tx=0x7f415400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247163058e7318c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.207+0000 7f4143fff640 1 -- 172.21.15.31:0/2300693058 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4154005220 con 0x55cca8db2270 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.207+0000 7f415affd640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41440afaf0 0x7f41440b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.207+0000 7f4143fff640 1 -- 172.21.15.31:0/2300693058 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4154004c50 con 0x55cca8db2270 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.207+0000 7f4143fff640 1 -- 172.21.15.31:0/2300693058 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41540107f0 con 0x55cca8db2270 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.207+0000 7f4161e3b640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41440afaf0 0x7f41440b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.207+0000 7f415affd640 1 -- 172.21.15.31:0/3887999822 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f414c100670 con 0x7f415c103b00 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f4161e3b640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41440afaf0 0x7f41440b2000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f414c030040 tx=0x7f414c02a8c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 -- 172.21.15.31:0/2300693058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8db2270 msgr2=0x55cca9153370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 --2- 172.21.15.31:0/2300693058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8db2270 0x55cca9153370 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f415400ac90 tx=0x7f415400e720 comp rx=0 tx=0).stop 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 -- 172.21.15.31:0/2300693058 shutdown_connections 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 --2- 172.21.15.31:0/2300693058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 0x55cca915a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 --2- 172.21.15.31:0/2300693058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8d893b0 0x55cca91538f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 --2- 172.21.15.31:0/2300693058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8db2270 0x55cca9153370 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 -- 172.21.15.31:0/2300693058 >> 172.21.15.31:0/2300693058 conn(0x55cca8da63c0 msgr2=0x55cca914ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 -- 172.21.15.31:0/2300693058 shutdown_connections 2024-04-25T16:00:37.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 -- 172.21.15.31:0/2300693058 wait complete. 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 -- start start 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.208+0000 7f416b02d740 1 Processor -- start 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416b02d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d893b0 0x55cca915d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416b02d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 0x55cca915d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416b02d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 0x55cca91646e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416b02d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cca915b240 con 0x55cca8d893b0 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416b02d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cca903ffe0 con 0x55cca8db2270 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416b02d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cca915b3c0 con 0x55cca8d91590 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416b02d740 10 client.0 _ll_register_callbacks cb 0x55cca9056150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d893b0 0x55cca915d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d893b0 0x55cca915d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45820/0 (socket says 172.21.15.31:45820) 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 -- 172.21.15.31:0/909355594 learned_addr learned my addr 172.21.15.31:0/909355594 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416163a640 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 0x55cca915d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 0x55cca91646e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 -- 172.21.15.31:0/909355594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 msgr2=0x55cca915d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 0x55cca915d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 -- 172.21.15.31:0/909355594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 msgr2=0x55cca91646e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 0x55cca91646e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 -- 172.21.15.31:0/909355594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cca8dca2d0 con 0x55cca8d893b0 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416263c640 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d893b0 0x55cca915d240 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4154005dd0 tx=0x7f415400af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9360ff803ccaff5d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f41417fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f41417fa640 1 -- 172.21.15.31:0/909355594 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41540107d0 con 0x55cca8d893b0 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f41417fa640 1 -- 172.21.15.31:0/909355594 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4154010970 con 0x55cca8d893b0 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.209+0000 7f416b02d740 1 -- 172.21.15.31:0/909355594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4154004c50 con 0x55cca8d893b0 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f41417fa640 1 -- 172.21.15.31:0/909355594 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4154005390 con 0x55cca8d893b0 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f416b02d740 10 client.15051 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f416b02d740 1 -- 172.21.15.31:0/909355594 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cca8e946b0 con 0x55cca8d893b0 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f41417fa640 1 -- 172.21.15.31:0/909355594 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4154004c50 con 0x55cca8d893b0 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f41417fa640 1 -- 172.21.15.31:0/909355594 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f4154028020 con 0x55cca8d893b0 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f416b02d740 10 client.15051 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f416b02d740 10 client.15051 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f416b02d740 20 client.15051 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f416b02d740 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55cca8d70e40 0x55cca916aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f416b02d740 4 client.15051 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f416b02d740 1 -- 172.21.15.31:0/909355594 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cca90c6350 con 0x55cca8d70e40 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.210+0000 7f416163a640 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55cca8d70e40 0x55cca916aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.211+0000 7f41417fa640 10 client.15051 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:00:37.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.211+0000 7f416163a640 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55cca8d70e40 0x55cca916aac0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9d531264fc6f6d1d server_cookie=14d06dacac5f505d in_seq=0 out_seq=0 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f41417fa640 1 -- 172.21.15.31:0/909355594 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cca90c6350 con 0x55cca8d70e40 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f41417fa640 10 client.15051 handle_command_reply: tid=0 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 client.15051 shutdown 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 -- 172.21.15.31:0/909355594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d893b0 msgr2=0x55cca915d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d893b0 0x55cca915d240 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f4154005dd0 tx=0x7f415400af90 comp rx=0 tx=0).stop 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 -- 172.21.15.31:0/909355594 shutdown_connections 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55cca8d70e40 0x55cca916aac0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 0x55cca91646e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d91590 0x55cca915d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 --2- 172.21.15.31:0/909355594 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d893b0 0x55cca915d240 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f41417fa640 0 client.15051 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:00:37.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 -- 172.21.15.31:0/909355594 >> 172.21.15.31:0/909355594 conn(0x55cca8da63c0 msgr2=0x55cca914dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 -- 172.21.15.31:0/909355594 shutdown_connections 2024-04-25T16:00:37.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 1 -- 172.21.15.31:0/909355594 wait complete. 2024-04-25T16:00:37.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.212+0000 7f416b02d740 20 client.15051 trim_cache size 0 max 16384 2024-04-25T16:00:37.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.230+0000 7f416b02d740 1 Processor -- start 2024-04-25T16:00:37.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416b02d740 1 -- start start 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416b02d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8d893b0 0x55cca915ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416b02d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca915d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416b02d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8db2270 0x55cca9164670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416b02d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cca9168e00 con 0x55cca8d91590 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416b02d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cca91694d0 con 0x55cca8d893b0 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416b02d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cca9169ba0 con 0x55cca8db2270 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416163a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca915d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416163a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca915d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45828/0 (socket says 172.21.15.31:45828) 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416163a640 1 -- 172.21.15.31:0/3803959412 learned_addr learned my addr 172.21.15.31:0/3803959412 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416163a640 1 -- 172.21.15.31:0/3803959412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8db2270 msgr2=0x55cca9164670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416163a640 1 --2- 172.21.15.31:0/3803959412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8db2270 0x55cca9164670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416163a640 1 -- 172.21.15.31:0/3803959412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8d893b0 msgr2=0x55cca915ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416163a640 1 --2- 172.21.15.31:0/3803959412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8d893b0 0x55cca915ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416163a640 1 -- 172.21.15.31:0/3803959412 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cca9157da0 con 0x55cca8d91590 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f416163a640 1 --2- 172.21.15.31:0/3803959412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca915d350 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f415000c6d0 tx=0x7f415000cdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a578f7723b9993e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f41437fe640 1 -- 172.21.15.31:0/3803959412 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150020030 con 0x55cca8d91590 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f41437fe640 1 -- 172.21.15.31:0/3803959412 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4150005020 con 0x55cca8d91590 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.231+0000 7f41437fe640 1 -- 172.21.15.31:0/3803959412 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415001c820 con 0x55cca8d91590 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 -- 172.21.15.31:0/3803959412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 msgr2=0x55cca915d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 --2- 172.21.15.31:0/3803959412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca915d350 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f415000c6d0 tx=0x7f415000cdb0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 -- 172.21.15.31:0/3803959412 shutdown_connections 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 --2- 172.21.15.31:0/3803959412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8db2270 0x55cca9164670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 --2- 172.21.15.31:0/3803959412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca915d350 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 --2- 172.21.15.31:0/3803959412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8d893b0 0x55cca915ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 -- 172.21.15.31:0/3803959412 >> 172.21.15.31:0/3803959412 conn(0x55cca8da63c0 msgr2=0x55cca915b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 -- 172.21.15.31:0/3803959412 shutdown_connections 2024-04-25T16:00:37.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 -- 172.21.15.31:0/3803959412 wait complete. 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.232+0000 7f416b02d740 1 -- start start 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416b02d740 1 Processor -- start 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416b02d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d893b0 0x55cca9160e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416b02d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca9161380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416b02d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 0x55cca91642c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416b02d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cca916a940 con 0x55cca8d91590 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416b02d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cca916a270 con 0x55cca8db2270 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416b02d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cca916b010 con 0x55cca8d893b0 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416b02d740 10 client.0 _ll_register_callbacks cb 0x7f4154014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416163a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca9161380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416163a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca9161380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45830/0 (socket says 172.21.15.31:45830) 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416163a640 1 -- 172.21.15.31:0/211887608 learned_addr learned my addr 172.21.15.31:0/211887608 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416263c640 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d893b0 0x55cca9160e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416163a640 1 -- 172.21.15.31:0/211887608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d893b0 msgr2=0x55cca9160e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416163a640 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d893b0 0x55cca9160e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416263c640 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 0x55cca91642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416163a640 1 -- 172.21.15.31:0/211887608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 msgr2=0x55cca91642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416163a640 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 0x55cca91642c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.233+0000 7f416163a640 1 -- 172.21.15.31:0/211887608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cca9164880 con 0x55cca8d91590 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416163a640 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca9161380 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f41500051d0 tx=0x7f4150005200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623b22cf27e9a9d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f41417fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f41417fa640 1 -- 172.21.15.31:0/211887608 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150020040 con 0x55cca8d91590 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f41417fa640 1 -- 172.21.15.31:0/211887608 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f415001c5a0 con 0x55cca8d91590 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416b02d740 1 -- 172.21.15.31:0/211887608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4150005020 con 0x55cca8d91590 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f41417fa640 1 -- 172.21.15.31:0/211887608 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4150005620 con 0x55cca8d91590 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416b02d740 10 client.15057 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416b02d740 1 -- 172.21.15.31:0/211887608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cca91d4090 con 0x55cca8d91590 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f41417fa640 1 -- 172.21.15.31:0/211887608 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4150005020 con 0x55cca8d91590 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f41417fa640 1 -- 172.21.15.31:0/211887608 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f41500254d0 con 0x55cca8d91590 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416b02d740 10 client.15057 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416b02d740 10 client.15057 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416b02d740 20 client.15057 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416b02d740 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55cca8d46280 0x55cca91d7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416b02d740 4 client.15057 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.234+0000 7f416b02d740 1 -- 172.21.15.31:0/211887608 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "status"}) -- 0x55cca90c01c0 con 0x55cca8d46280 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416263c640 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55cca8d46280 0x55cca91d7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f41417fa640 10 client.15057 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416263c640 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55cca8d46280 0x55cca91d7690 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=305a0bae8e0260d6 server_cookie=44c263fcd95c699d in_seq=0 out_seq=0 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f41417fa640 1 -- 172.21.15.31:0/211887608 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55cca90c01c0 con 0x55cca8d46280 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f41417fa640 10 client.15057 handle_command_reply: tid=0 2024-04-25T16:00:37.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416b02d740 1 client.15057 shutdown 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416b02d740 1 -- 172.21.15.31:0/211887608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 msgr2=0x55cca9161380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416b02d740 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca9161380 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f41500051d0 tx=0x7f4150005200 comp rx=0 tx=0).stop 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416b02d740 1 -- 172.21.15.31:0/211887608 shutdown_connections 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416b02d740 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55cca8d46280 0x55cca91d7690 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416b02d740 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cca8db2270 0x55cca91642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416b02d740 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cca8d91590 0x55cca9161380 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f41417fa640 0 client.15057 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.235+0000 7f416b02d740 1 --2- 172.21.15.31:0/211887608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cca8d893b0 0x55cca9160e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416b02d740 1 -- 172.21.15.31:0/211887608 >> 172.21.15.31:0/211887608 conn(0x55cca8da63c0 msgr2=0x55cca915ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416b02d740 1 -- 172.21.15.31:0/211887608 shutdown_connections 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416b02d740 1 -- 172.21.15.31:0/211887608 wait complete. 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416b02d740 20 client.15057 trim_cache size 0 max 16384 2024-04-25T16:00:37.243 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 4, 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 14631, 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 87.990032154999994, 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 102.289366187 2024-04-25T16:00:37.244 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:00:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 -- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41440afaf0 msgr2=0x7f41440b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41440afaf0 0x7f41440b2000 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f414c030040 tx=0x7f414c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 -- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 msgr2=0x7f415c076850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 0x7f415c076850 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f414c033040 tx=0x7f414c003c20 comp rx=0 tx=0).stop 2024-04-25T16:00:37.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 -- 172.21.15.31:0/3887999822 shutdown_connections 2024-04-25T16:00:37.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41440afaf0 0x7f41440b2000 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f415c109a40 0x7f415c075430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f415c104d40 0x7f415c074ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 --2- 172.21.15.31:0/3887999822 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f415c103b00 0x7f415c076850 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 -- 172.21.15.31:0/3887999822 >> 172.21.15.31:0/3887999822 conn(0x7f415c0fecf0 msgr2=0x7f415c105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 -- 172.21.15.31:0/3887999822 shutdown_connections 2024-04-25T16:00:37.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.236+0000 7f416409f640 1 -- 172.21.15.31:0/3887999822 wait complete. 2024-04-25T16:00:37.249 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.361+0000 7f80c79d6640 1 -- 172.21.15.31:0/3227689512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c01029f0 msgr2=0x7f80c01036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.361+0000 7f80c79d6640 1 --2- 172.21.15.31:0/3227689512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c01029f0 0x7f80c01036f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f80bc009900 tx=0x7f80bc02a410 comp rx=0 tx=0).stop 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.361+0000 7f80c79d6640 1 -- 172.21.15.31:0/3227689512 shutdown_connections 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.361+0000 7f80c79d6640 1 --2- 172.21.15.31:0/3227689512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0108930 0x7f80c010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.361+0000 7f80c79d6640 1 --2- 172.21.15.31:0/3227689512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80c0103c30 0x7f80c0104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.361+0000 7f80c79d6640 1 --2- 172.21.15.31:0/3227689512 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c01029f0 0x7f80c01036f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.361+0000 7f80c79d6640 1 -- 172.21.15.31:0/3227689512 >> 172.21.15.31:0/3227689512 conn(0x7f80c00fdbc0 msgr2=0x7f80c0100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 -- 172.21.15.31:0/3227689512 shutdown_connections 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 -- 172.21.15.31:0/3227689512 wait complete. 2024-04-25T16:00:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 Processor -- start 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 -- start start 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80c01029f0 0x7f80c019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0103c30 0x7f80c019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c0108930 0x7f80c01a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f80c010f620 con 0x7f80c0108930 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f80c010f4a0 con 0x7f80c01029f0 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.362+0000 7f80c79d6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f80c010f7a0 con 0x7f80c0103c30 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5f73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c0108930 0x7f80c01a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5f73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c0108930 0x7f80c01a1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45848/0 (socket says 172.21.15.31:45848) 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5f73640 1 -- 172.21.15.31:0/2896163079 learned_addr learned my addr 172.21.15.31:0/2896163079 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5772640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80c01029f0 0x7f80c019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c4f71640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0103c30 0x7f80c019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5f73640 1 -- 172.21.15.31:0/2896163079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0103c30 msgr2=0x7f80c019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5f73640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0103c30 0x7f80c019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5f73640 1 -- 172.21.15.31:0/2896163079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80c01029f0 msgr2=0x7f80c019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5f73640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80c01029f0 0x7f80c019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5f73640 1 -- 172.21.15.31:0/2896163079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c01a1cc0 con 0x7f80c0108930 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c4f71640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0103c30 0x7f80c019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c5f73640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c0108930 0x7f80c01a1700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f80b800eaa0 tx=0x7f80b800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80b67fc640 1 -- 172.21.15.31:0/2896163079 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b800cdb0 con 0x7f80c0108930 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80b67fc640 1 -- 172.21.15.31:0/2896163079 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b8005020 con 0x7f80c0108930 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c79d6640 1 -- 172.21.15.31:0/2896163079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c01a1fb0 con 0x7f80c0108930 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80b67fc640 1 -- 172.21.15.31:0/2896163079 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b80116d0 con 0x7f80c0108930 2024-04-25T16:00:37.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.363+0000 7f80c79d6640 1 -- 172.21.15.31:0/2896163079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c01a2410 con 0x7f80c0108930 2024-04-25T16:00:37.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80ce885740 1 Processor -- start 2024-04-25T16:00:37.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80b67fc640 1 -- 172.21.15.31:0/2896163079 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f80b80051d0 con 0x7f80c0108930 2024-04-25T16:00:37.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80ce885740 1 -- start start 2024-04-25T16:00:37.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80ce885740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716a84780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80ce885740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 0x562716a84d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80ce885740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80ce885740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5627169e5740 con 0x5627166be590 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80ce885740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5627169813a0 con 0x5627166b63b0 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80ce885740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5627169e1700 con 0x5627166df270 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80c4f71640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80c4f71640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45856/0 (socket says 172.21.15.31:45856) 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80c4f71640 1 -- 172.21.15.31:0/854788524 learned_addr learned my addr 172.21.15.31:0/854788524 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80c4f71640 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716a84780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.366+0000 7f80c5772640 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 0x562716a84d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80c4f71640 1 -- 172.21.15.31:0/854788524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 msgr2=0x562716a84780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80c4f71640 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716a84780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80c4f71640 1 -- 172.21.15.31:0/854788524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 msgr2=0x562716a84d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80c4f71640 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 0x562716a84d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80c4f71640 1 -- 172.21.15.31:0/854788524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56271699d750 con 0x5627166be590 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80c5772640 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 0x562716a84d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:37.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80b67fc640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80ac0afaf0 0x7f80ac0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80c4f71640 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8bc00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f80b0012830 tx=0x7f80b000d400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfbf3c50e9284800 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80c5772640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80ac0afaf0 0x7f80ac0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80ab7fe640 1 -- 172.21.15.31:0/854788524 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0005220 con 0x5627166be590 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80b67fc640 1 -- 172.21.15.31:0/2896163079 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f80b80ded70 con 0x7f80c0108930 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80ab7fe640 1 -- 172.21.15.31:0/854788524 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b0004c50 con 0x5627166be590 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.367+0000 7f80ab7fe640 1 -- 172.21.15.31:0/854788524 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b000aa10 con 0x5627166be590 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 -- 172.21.15.31:0/854788524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 msgr2=0x562716a8bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80c5772640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80ac0afaf0 0x7f80ac0b2000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f80bc005ab0 tx=0x7f80bc031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8bc00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f80b0012830 tx=0x7f80b000d400 comp rx=0 tx=0).stop 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 -- 172.21.15.31:0/854788524 shutdown_connections 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8bc00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 0x562716a84d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 --2- 172.21.15.31:0/854788524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716a84780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 -- 172.21.15.31:0/854788524 >> 172.21.15.31:0/854788524 conn(0x5627166d33c0 msgr2=0x562716a7bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 -- 172.21.15.31:0/854788524 shutdown_connections 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 -- 172.21.15.31:0/854788524 wait complete. 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 -- start start 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.368+0000 7f80ce885740 1 Processor -- start 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80ce885740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166b63b0 0x562716a8e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80ce885740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80ce885740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166df270 0x562716a95c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80ce885740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562716a8c650 con 0x5627166be590 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80ce885740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56271696cfe0 con 0x5627166df270 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80ce885740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562716a8c7d0 con 0x5627166b63b0 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80ce885740 10 client.0 _ll_register_callbacks cb 0x562716983150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c4f71640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c4f71640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45864/0 (socket says 172.21.15.31:45864) 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c4f71640 1 -- 172.21.15.31:0/564289361 learned_addr learned my addr 172.21.15.31:0/564289361 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c5f73640 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166df270 0x562716a95c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c5772640 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166b63b0 0x562716a8e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c4f71640 1 -- 172.21.15.31:0/564289361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166b63b0 msgr2=0x562716a8e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c4f71640 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166b63b0 0x562716a8e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c4f71640 1 -- 172.21.15.31:0/564289361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166df270 msgr2=0x562716a95c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c4f71640 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166df270 0x562716a95c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c4f71640 1 -- 172.21.15.31:0/564289361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627166f72d0 con 0x5627166be590 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.369+0000 7f80c4f71640 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8eca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f80b000dc90 tx=0x7f80b0005d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2649546f1767dfa8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80a8ff9640 1 -- 172.21.15.31:0/564289361 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b002a060 con 0x5627166be590 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80a8ff9640 1 -- 172.21.15.31:0/564289361 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f80b0004fc0 con 0x5627166be590 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80ce885740 1 -- 172.21.15.31:0/564289361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80b0004c50 con 0x5627166be590 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80a8ff9640 1 -- 172.21.15.31:0/564289361 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f80b0005200 con 0x5627166be590 2024-04-25T16:00:37.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80ce885740 10 client.15069 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80ce885740 1 -- 172.21.15.31:0/564289361 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5627167c16b0 con 0x5627166be590 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80a8ff9640 1 -- 172.21.15.31:0/564289361 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b0004c50 con 0x5627166be590 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80a8ff9640 1 -- 172.21.15.31:0/564289361 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f80b000ac20 con 0x5627166be590 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80ce885740 10 client.15069 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80ce885740 10 client.15069 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80ce885740 20 client.15069 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80ce885740 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x56271669de40 0x562716a9c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80ce885740 4 client.15069 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.370+0000 7f80ce885740 1 -- 172.21.15.31:0/564289361 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627169f3350 con 0x56271669de40 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.371+0000 7f80c5772640 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x56271669de40 0x562716a9c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.371+0000 7f80a8ff9640 10 client.15069 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.371+0000 7f80c5772640 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x56271669de40 0x562716a9c090 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3f99a2bb00af7f97 server_cookie=6c7da9b5aa2ca67e in_seq=0 out_seq=0 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80a8ff9640 1 -- 172.21.15.31:0/564289361 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5627169f3350 con 0x56271669de40 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80a8ff9640 10 client.15069 handle_command_reply: tid=0 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 client.15069 shutdown 2024-04-25T16:00:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 -- 172.21.15.31:0/564289361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 msgr2=0x562716a8eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8eca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f80b000dc90 tx=0x7f80b0005d00 comp rx=0 tx=0).stop 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 -- 172.21.15.31:0/564289361 shutdown_connections 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166df270 0x562716a95c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8eca0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x56271669de40 0x562716a9c090 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 --2- 172.21.15.31:0/564289361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166b63b0 0x562716a8e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 -- 172.21.15.31:0/564289361 >> 172.21.15.31:0/564289361 conn(0x5627166d33c0 msgr2=0x562716a7ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80a8ff9640 0 client.15069 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 -- 172.21.15.31:0/564289361 shutdown_connections 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 1 -- 172.21.15.31:0/564289361 wait complete. 2024-04-25T16:00:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.372+0000 7f80ce885740 20 client.15069 trim_cache size 0 max 16384 2024-04-25T16:00:37.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.391+0000 7f80ce885740 1 Processor -- start 2024-04-25T16:00:37.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.391+0000 7f80ce885740 1 -- start start 2024-04-25T16:00:37.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.391+0000 7f80ce885740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 0x562716a8cd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.391+0000 7f80ce885740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.391+0000 7f80ce885740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716a7b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.391+0000 7f80ce885740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562716a9a3d0 con 0x5627166be590 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.391+0000 7f80ce885740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562716a9aaa0 con 0x5627166b63b0 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.391+0000 7f80ce885740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562716a9b170 con 0x5627166df270 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.391+0000 7f80c4f71640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80c4f71640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45870/0 (socket says 172.21.15.31:45870) 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80c4f71640 1 -- 172.21.15.31:0/1827851983 learned_addr learned my addr 172.21.15.31:0/1827851983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80c5772640 1 --2- 172.21.15.31:0/1827851983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 0x562716a8cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80c4f71640 1 -- 172.21.15.31:0/1827851983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 msgr2=0x562716a7b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80c4f71640 1 --2- 172.21.15.31:0/1827851983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716a7b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80c4f71640 1 -- 172.21.15.31:0/1827851983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 msgr2=0x562716a8cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80c4f71640 1 --2- 172.21.15.31:0/1827851983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 0x562716a8cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80c4f71640 1 -- 172.21.15.31:0/1827851983 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562716a895a0 con 0x5627166be590 2024-04-25T16:00:37.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80c4f71640 1 --2- 172.21.15.31:0/1827851983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8d2c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f80b0012610 tx=0x7f80b000db00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e336ed40c9734bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80aaffd640 1 -- 172.21.15.31:0/1827851983 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b002a070 con 0x5627166be590 2024-04-25T16:00:37.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80aaffd640 1 -- 172.21.15.31:0/1827851983 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b00118b0 con 0x5627166be590 2024-04-25T16:00:37.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.392+0000 7f80aaffd640 1 -- 172.21.15.31:0/1827851983 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0034070 con 0x5627166be590 2024-04-25T16:00:37.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 -- 172.21.15.31:0/1827851983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 msgr2=0x562716a8d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 --2- 172.21.15.31:0/1827851983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8d2c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f80b0012610 tx=0x7f80b000db00 comp rx=0 tx=0).stop 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 -- 172.21.15.31:0/1827851983 shutdown_connections 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 --2- 172.21.15.31:0/1827851983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716a7b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 --2- 172.21.15.31:0/1827851983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166be590 0x562716a8d2c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 --2- 172.21.15.31:0/1827851983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166b63b0 0x562716a8cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 -- 172.21.15.31:0/1827851983 >> 172.21.15.31:0/1827851983 conn(0x5627166d33c0 msgr2=0x562716a87e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 -- 172.21.15.31:0/1827851983 shutdown_connections 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 -- 172.21.15.31:0/1827851983 wait complete. 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 -- start start 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.393+0000 7f80ce885740 1 Processor -- start 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80ce885740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166b63b0 0x562716a901c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80ce885740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166be590 0x562716a90700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80ce885740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716b02f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80ce885740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562716a9bf10 con 0x5627166b63b0 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80ce885740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562716a9b840 con 0x5627166be590 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80ce885740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562716a9c5e0 con 0x5627166df270 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80ce885740 10 client.0 _ll_register_callbacks cb 0x7f80b000de20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5772640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166b63b0 0x562716a901c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5772640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166b63b0 0x562716a901c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45872/0 (socket says 172.21.15.31:45872) 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5772640 1 -- 172.21.15.31:0/3828885159 learned_addr learned my addr 172.21.15.31:0/3828885159 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c4f71640 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166be590 0x562716a90700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5f73640 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716b02f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5772640 1 -- 172.21.15.31:0/3828885159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 msgr2=0x562716b02f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5772640 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716b02f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5772640 1 -- 172.21.15.31:0/3828885159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166be590 msgr2=0x562716a90700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5772640 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166be590 0x562716a90700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5772640 1 -- 172.21.15.31:0/3828885159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562716b03440 con 0x5627166b63b0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5f73640 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716b02f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80c5772640 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166b63b0 0x562716a901c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0096c0 tx=0x7f80bc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f03e5ea95e20f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.394+0000 7f80a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80a8ff9640 1 -- 172.21.15.31:0/3828885159 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc031e60 con 0x5627166b63b0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80a8ff9640 1 -- 172.21.15.31:0/3828885159 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f80bc0035f0 con 0x5627166b63b0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80ce885740 1 -- 172.21.15.31:0/3828885159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f80b00118b0 con 0x5627166b63b0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80a8ff9640 1 -- 172.21.15.31:0/3828885159 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f80bc0037f0 con 0x5627166b63b0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80ce885740 10 client.15075 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80ce885740 1 -- 172.21.15.31:0/3828885159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562716b046d0 con 0x5627166b63b0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80a8ff9640 1 -- 172.21.15.31:0/3828885159 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b00118b0 con 0x5627166b63b0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80a8ff9640 1 -- 172.21.15.31:0/3828885159 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f80bc052ce0 con 0x5627166b63b0 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80ce885740 10 client.15075 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80ce885740 10 client.15075 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T16:00:37.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80ce885740 20 client.15075 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80ce885740 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562716673280 0x562716b07f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80ce885740 4 client.15075 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:00:37.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80ce885740 1 -- 172.21.15.31:0/3828885159 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "status"}) -- 0x5627169ed1c0 con 0x562716673280 2024-04-25T16:00:37.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.395+0000 7f80c4f71640 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562716673280 0x562716b07f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80c4f71640 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562716673280 0x562716b07f90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4641cf07fc34c5 server_cookie=90e6a6b7ea01084 in_seq=0 out_seq=0 2024-04-25T16:00:37.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80a8ff9640 10 client.15075 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:00:37.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80a8ff9640 1 -- 172.21.15.31:0/3828885159 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5627169ed1c0 con 0x562716673280 2024-04-25T16:00:37.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80a8ff9640 10 client.15075 handle_command_reply: tid=0 2024-04-25T16:00:37.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 client.15075 shutdown 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 -- 172.21.15.31:0/3828885159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166b63b0 msgr2=0x562716a901c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166b63b0 0x562716a901c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0096c0 tx=0x7f80bc02a410 comp rx=0 tx=0).stop 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 -- 172.21.15.31:0/3828885159 shutdown_connections 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562716673280 0x562716b07f90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5627166df270 0x562716b02f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80a8ff9640 0 client.15075 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5627166be590 0x562716a90700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 --2- 172.21.15.31:0/3828885159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5627166b63b0 0x562716a901c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 -- 172.21.15.31:0/3828885159 >> 172.21.15.31:0/3828885159 conn(0x5627166d33c0 msgr2=0x562716a8cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 -- 172.21.15.31:0/3828885159 shutdown_connections 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.396+0000 7f80ce885740 1 -- 172.21.15.31:0/3828885159 wait complete. 2024-04-25T16:00:37.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80ce885740 20 client.15075 trim_cache size 0 max 16384 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 3, 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24521, 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 88.846277944999997, 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 111.068671157 2024-04-25T16:00:37.407 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:00:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 -- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80ac0afaf0 msgr2=0x7f80ac0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80ac0afaf0 0x7f80ac0b2000 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f80bc005ab0 tx=0x7f80bc031400 comp rx=0 tx=0).stop 2024-04-25T16:00:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 -- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c0108930 msgr2=0x7f80c01a1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c0108930 0x7f80c01a1700 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f80b800eaa0 tx=0x7f80b800ef60 comp rx=0 tx=0).stop 2024-04-25T16:00:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 -- 172.21.15.31:0/2896163079 shutdown_connections 2024-04-25T16:00:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80ac0afaf0 0x7f80ac0b2000 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80c0108930 0x7f80c01a1700 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80c0103c30 0x7f80c019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 --2- 172.21.15.31:0/2896163079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80c01029f0 0x7f80c019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 -- 172.21.15.31:0/2896163079 >> 172.21.15.31:0/2896163079 conn(0x7f80c00fdbc0 msgr2=0x7f80c0104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 -- 172.21.15.31:0/2896163079 shutdown_connections 2024-04-25T16:00:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.397+0000 7f80c79d6640 1 -- 172.21.15.31:0/2896163079 wait complete. 2024-04-25T16:00:37.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T16:00:37.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.522+0000 7f3921f96640 1 -- 172.21.15.31:0/802964391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 msgr2=0x7f391c10be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.522+0000 7f3921f96640 1 --2- 172.21.15.31:0/802964391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 0x7f391c10be50 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3914009ca0 tx=0x7f391402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.522+0000 7f3921f96640 1 -- 172.21.15.31:0/802964391 shutdown_connections 2024-04-25T16:00:37.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.522+0000 7f3921f96640 1 --2- 172.21.15.31:0/802964391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 0x7f391c10be50 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.522+0000 7f3921f96640 1 --2- 172.21.15.31:0/802964391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f391c104d30 0x7f391c105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.522+0000 7f3921f96640 1 --2- 172.21.15.31:0/802964391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f391c103af0 0x7f391c1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.522+0000 7f3921f96640 1 -- 172.21.15.31:0/802964391 >> 172.21.15.31:0/802964391 conn(0x7f391c0fece0 msgr2=0x7f391c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.522+0000 7f3921f96640 1 -- 172.21.15.31:0/802964391 shutdown_connections 2024-04-25T16:00:37.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.522+0000 7f3921f96640 1 -- 172.21.15.31:0/802964391 wait complete. 2024-04-25T16:00:37.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f3921f96640 1 Processor -- start 2024-04-25T16:00:37.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f3921f96640 1 -- start start 2024-04-25T16:00:37.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f3921f96640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f391c103af0 0x7f391c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f3921f96640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f391c104d30 0x7f391c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f3921f96640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 0x7f391c1a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f3921f96640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f391c10f620 con 0x7f391c109a30 2024-04-25T16:00:37.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f3921f96640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f391c10f4a0 con 0x7f391c103af0 2024-04-25T16:00:37.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f3921f96640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f391c10f7a0 con 0x7f391c104d30 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 0x7f391c1a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 0x7f391c1a16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45896/0 (socket says 172.21.15.31:45896) 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391bfff640 1 -- 172.21.15.31:0/2374854219 learned_addr learned my addr 172.21.15.31:0/2374854219 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391b7fe640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f391c103af0 0x7f391c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391affd640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f391c104d30 0x7f391c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391bfff640 1 -- 172.21.15.31:0/2374854219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f391c104d30 msgr2=0x7f391c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391bfff640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f391c104d30 0x7f391c19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391bfff640 1 -- 172.21.15.31:0/2374854219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f391c103af0 msgr2=0x7f391c19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391bfff640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f391c103af0 0x7f391c19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.523+0000 7f391bfff640 1 -- 172.21.15.31:0/2374854219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391c1a1d30 con 0x7f391c109a30 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.524+0000 7f391bfff640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 0x7f391c1a16c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f39140097b0 tx=0x7f3914003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.524+0000 7f3918ff9640 1 -- 172.21.15.31:0/2374854219 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914004240 con 0x7f391c109a30 2024-04-25T16:00:37.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.524+0000 7f3918ff9640 1 -- 172.21.15.31:0/2374854219 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39140317c0 con 0x7f391c109a30 2024-04-25T16:00:37.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.524+0000 7f3921f96640 1 -- 172.21.15.31:0/2374854219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f391c1a1fc0 con 0x7f391c109a30 2024-04-25T16:00:37.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.524+0000 7f3918ff9640 1 -- 172.21.15.31:0/2374854219 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391402fde0 con 0x7f391c109a30 2024-04-25T16:00:37.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.524+0000 7f3921f96640 1 -- 172.21.15.31:0/2374854219 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f391c1a2420 con 0x7f391c109a30 2024-04-25T16:00:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.526+0000 7f3918ff9640 1 -- 172.21.15.31:0/2374854219 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f39140043e0 con 0x7f391c109a30 2024-04-25T16:00:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3928e48740 1 Processor -- start 2024-04-25T16:00:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3928e48740 1 -- start start 2024-04-25T16:00:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3928e48740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e699370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3928e48740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6998f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3928e48740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 0x55c42e6a07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3928e48740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c42e5fe740 con 0x55c42e2cf3b0 2024-04-25T16:00:37.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3928e48740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c42e59a3a0 con 0x55c42e2d7590 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3928e48740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c42e5fa700 con 0x55c42e2f8270 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45902/0 (socket says 172.21.15.31:45902) 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391b7fe640 1 -- 172.21.15.31:0/3677130400 learned_addr learned my addr 172.21.15.31:0/3677130400 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3918ff9640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39040afaf0 0x7f39040b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391affd640 1 --2- 172.21.15.31:0/3677130400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e699370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391bfff640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39040afaf0 0x7f39040b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391affd640 1 --2- 172.21.15.31:0/3677130400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 0x55c42e6a07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391b7fe640 1 -- 172.21.15.31:0/3677130400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 msgr2=0x55c42e699370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391b7fe640 1 --2- 172.21.15.31:0/3677130400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e699370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391b7fe640 1 -- 172.21.15.31:0/3677130400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 msgr2=0x55c42e6a07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391b7fe640 1 --2- 172.21.15.31:0/3677130400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 0x55c42e6a07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f391b7fe640 1 -- 172.21.15.31:0/3677130400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c42e5b6750 con 0x55c42e2cf3b0 2024-04-25T16:00:37.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.527+0000 7f3918ff9640 1 -- 172.21.15.31:0/2374854219 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f39141005b0 con 0x7f391c109a30 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.528+0000 7f391b7fe640 1 --2- 172.21.15.31:0/3677130400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6998f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f390c00abb0 tx=0x7f390c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15848dbf0d16d4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.528+0000 7f391bfff640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39040afaf0 0x7f39040b2000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f391404d000 tx=0x7f39140f7190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.528+0000 7f3901ffb640 1 -- 172.21.15.31:0/3677130400 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c005220 con 0x55c42e2cf3b0 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.528+0000 7f3901ffb640 1 -- 172.21.15.31:0/3677130400 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f390c004c50 con 0x55c42e2cf3b0 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.528+0000 7f3901ffb640 1 -- 172.21.15.31:0/3677130400 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c0186c0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 -- 172.21.15.31:0/3677130400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 msgr2=0x55c42e6998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 --2- 172.21.15.31:0/3677130400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6998f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f390c00abb0 tx=0x7f390c00f790 comp rx=0 tx=0).stop 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 -- 172.21.15.31:0/3677130400 shutdown_connections 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 --2- 172.21.15.31:0/3677130400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 0x55c42e6a07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 --2- 172.21.15.31:0/3677130400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6998f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 --2- 172.21.15.31:0/3677130400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e699370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 -- 172.21.15.31:0/3677130400 >> 172.21.15.31:0/3677130400 conn(0x55c42e2ec3c0 msgr2=0x55c42e694d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 -- 172.21.15.31:0/3677130400 shutdown_connections 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 -- 172.21.15.31:0/3677130400 wait complete. 2024-04-25T16:00:37.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 -- start start 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 Processor -- start 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.529+0000 7f3928e48740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f3928e48740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 0x55c42e6a3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f3928e48740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e6aa6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f3928e48740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c42e6a1240 con 0x55c42e2cf3b0 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f3928e48740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c42e585fe0 con 0x55c42e2d7590 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f3928e48740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c42e6a13c0 con 0x55c42e2f8270 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f3928e48740 10 client.0 _ll_register_callbacks cb 0x55c42e59c150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45916/0 (socket says 172.21.15.31:45916) 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 -- 172.21.15.31:0/716170487 learned_addr learned my addr 172.21.15.31:0/716170487 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391b7fe640 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 0x55c42e6a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e6aa6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 -- 172.21.15.31:0/716170487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 msgr2=0x55c42e6aa6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e6aa6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 -- 172.21.15.31:0/716170487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 msgr2=0x55c42e6a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 0x55c42e6a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 -- 172.21.15.31:0/716170487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c42e3102d0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f391affd640 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a3220 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f391c19b290 tx=0x7f39100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52410ce9906a92f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f38faffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f38faffd640 1 -- 172.21.15.31:0/716170487 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3910004020 con 0x55c42e2cf3b0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f38faffd640 1 -- 172.21.15.31:0/716170487 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f39100041c0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.530+0000 7f3928e48740 1 -- 172.21.15.31:0/716170487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f390c004c50 con 0x55c42e2cf3b0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f38faffd640 1 -- 172.21.15.31:0/716170487 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f39100043c0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f3928e48740 10 client.15087 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f3928e48740 1 -- 172.21.15.31:0/716170487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c42e3da6b0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f38faffd640 1 -- 172.21.15.31:0/716170487 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f390c004c50 con 0x55c42e2cf3b0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f38faffd640 1 -- 172.21.15.31:0/716170487 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f391001f4d0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f3928e48740 10 client.15087 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f3928e48740 10 client.15087 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f3928e48740 20 client.15087 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f3928e48740 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55c42e2b6e40 0x55c42e6b1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f3928e48740 4 client.15087 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f3928e48740 1 -- 172.21.15.31:0/716170487 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c42e60c350 con 0x55c42e2b6e40 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.531+0000 7f391b7fe640 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55c42e2b6e40 0x55c42e6b1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.532+0000 7f391b7fe640 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55c42e2b6e40 0x55c42e6b1000 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f2a33c3fab093dd server_cookie=d5e01eea029e1983 in_seq=0 out_seq=0 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.532+0000 7f38faffd640 10 client.15087 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f38faffd640 1 -- 172.21.15.31:0/716170487 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c42e60c350 con 0x55c42e2b6e40 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f38faffd640 10 client.15087 handle_command_reply: tid=0 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 client.15087 shutdown 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 -- 172.21.15.31:0/716170487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 msgr2=0x55c42e6a3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a3220 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f391c19b290 tx=0x7f39100046c0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 -- 172.21.15.31:0/716170487 shutdown_connections 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e6aa6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55c42e2b6e40 0x55c42e6b1000 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2d7590 0x55c42e6a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 --2- 172.21.15.31:0/716170487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a3220 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f38faffd640 0 client.15087 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:00:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 -- 172.21.15.31:0/716170487 >> 172.21.15.31:0/716170487 conn(0x55c42e2ec3c0 msgr2=0x55c42e693d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 -- 172.21.15.31:0/716170487 shutdown_connections 2024-04-25T16:00:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 1 -- 172.21.15.31:0/716170487 wait complete. 2024-04-25T16:00:37.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.533+0000 7f3928e48740 20 client.15087 trim_cache size 0 max 16384 2024-04-25T16:00:37.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f3928e48740 1 Processor -- start 2024-04-25T16:00:37.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f3928e48740 1 -- start start 2024-04-25T16:00:37.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f3928e48740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2cf3b0 0x55c42e68fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f3928e48740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2d7590 0x55c42e6946b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f3928e48740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e694bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f3928e48740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c42e6af340 con 0x55c42e2d7590 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f3928e48740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c42e6afa10 con 0x55c42e2cf3b0 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f3928e48740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c42e6b00e0 con 0x55c42e2f8270 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f391b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2d7590 0x55c42e6946b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f391b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2d7590 0x55c42e6946b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45924/0 (socket says 172.21.15.31:45924) 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f391b7fe640 1 -- 172.21.15.31:0/4290101839 learned_addr learned my addr 172.21.15.31:0/4290101839 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f391b7fe640 1 -- 172.21.15.31:0/4290101839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 msgr2=0x55c42e694bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f391b7fe640 1 --2- 172.21.15.31:0/4290101839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e694bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f391b7fe640 1 -- 172.21.15.31:0/4290101839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2cf3b0 msgr2=0x55c42e68fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f391b7fe640 1 --2- 172.21.15.31:0/4290101839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2cf3b0 0x55c42e68fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f391b7fe640 1 -- 172.21.15.31:0/4290101839 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c42e69e030 con 0x55c42e2d7590 2024-04-25T16:00:37.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.552+0000 7f391b7fe640 1 --2- 172.21.15.31:0/4290101839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2d7590 0x55c42e6946b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f390c004880 tx=0x7f390c00a8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad1bca1d458a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f39017fa640 1 -- 172.21.15.31:0/4290101839 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c004040 con 0x55c42e2d7590 2024-04-25T16:00:37.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f39017fa640 1 -- 172.21.15.31:0/4290101839 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f390c0041e0 con 0x55c42e2d7590 2024-04-25T16:00:37.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f39017fa640 1 -- 172.21.15.31:0/4290101839 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f390c003d40 con 0x55c42e2d7590 2024-04-25T16:00:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f3928e48740 1 -- 172.21.15.31:0/4290101839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2d7590 msgr2=0x55c42e6946b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f3928e48740 1 --2- 172.21.15.31:0/4290101839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2d7590 0x55c42e6946b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f390c004880 tx=0x7f390c00a8a0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f3928e48740 1 -- 172.21.15.31:0/4290101839 shutdown_connections 2024-04-25T16:00:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f3928e48740 1 --2- 172.21.15.31:0/4290101839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2f8270 0x55c42e694bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f3928e48740 1 --2- 172.21.15.31:0/4290101839 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2d7590 0x55c42e6946b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f3928e48740 1 --2- 172.21.15.31:0/4290101839 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2cf3b0 0x55c42e68fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f3928e48740 1 -- 172.21.15.31:0/4290101839 >> 172.21.15.31:0/4290101839 conn(0x55c42e2ec3c0 msgr2=0x55c42e69c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f3928e48740 1 -- 172.21.15.31:0/4290101839 shutdown_connections 2024-04-25T16:00:37.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.553+0000 7f3928e48740 1 -- 172.21.15.31:0/4290101839 wait complete. 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f3928e48740 1 -- start start 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f3928e48740 1 Processor -- start 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f3928e48740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f3928e48740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2d7590 0x55c42e6a5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f3928e48740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2f8270 0x55c42e6a87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f3928e48740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c42e6b0e80 con 0x55c42e2cf3b0 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f3928e48740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c42e6b07b0 con 0x55c42e2f8270 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f3928e48740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c42e6b1550 con 0x55c42e2d7590 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f3928e48740 10 client.0 _ll_register_callbacks cb 0x7f391000e350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f391affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f391affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45928/0 (socket says 172.21.15.31:45928) 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f391affd640 1 -- 172.21.15.31:0/821616640 learned_addr learned my addr 172.21.15.31:0/821616640 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.554+0000 7f391b7fe640 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2d7590 0x55c42e6a5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f391affd640 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2f8270 0x55c42e6a87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f391affd640 1 -- 172.21.15.31:0/821616640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2d7590 msgr2=0x55c42e6a5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f391affd640 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2d7590 0x55c42e6a5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f391affd640 1 -- 172.21.15.31:0/821616640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2f8270 msgr2=0x55c42e6a87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f391affd640 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2f8270 0x55c42e6a87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f391affd640 1 -- 172.21.15.31:0/821616640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c42e6a8db0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f391b7fe640 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2d7590 0x55c42e6a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f391affd640 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a53e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f391000e930 tx=0x7f3910001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f7ca3fba23ecd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f38faffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f38faffd640 1 -- 172.21.15.31:0/821616640 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391001fcc0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f38faffd640 1 -- 172.21.15.31:0/821616640 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f391002c450 con 0x55c42e2cf3b0 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f3928e48740 1 -- 172.21.15.31:0/821616640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f390c0041e0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f38faffd640 1 -- 172.21.15.31:0/821616640 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f391001fe60 con 0x55c42e2cf3b0 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f3928e48740 10 client.15093 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f3928e48740 1 -- 172.21.15.31:0/821616640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c42e6aa0d0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.555+0000 7f38faffd640 1 -- 172.21.15.31:0/821616640 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f390c0041e0 con 0x55c42e2cf3b0 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.556+0000 7f38faffd640 1 -- 172.21.15.31:0/821616640 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3910038580 con 0x55c42e2cf3b0 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.556+0000 7f3928e48740 10 client.15093 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.556+0000 7f3928e48740 10 client.15093 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.556+0000 7f3928e48740 20 client.15093 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.556+0000 7f3928e48740 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55c42e28c280 0x55c42e71d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.556+0000 7f3928e48740 4 client.15093 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:00:37.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.556+0000 7f3928e48740 1 -- 172.21.15.31:0/821616640 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "status"}) -- 0x55c42e6061c0 con 0x55c42e28c280 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.556+0000 7f391b7fe640 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55c42e28c280 0x55c42e71d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f38faffd640 10 client.15093 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f391b7fe640 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55c42e28c280 0x55c42e71d5d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a85f4a67eea173f6 server_cookie=3e36cb0605bb84d9 in_seq=0 out_seq=0 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f38faffd640 1 -- 172.21.15.31:0/821616640 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55c42e6061c0 con 0x55c42e28c280 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f38faffd640 10 client.15093 handle_command_reply: tid=0 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 client.15093 shutdown 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 -- 172.21.15.31:0/821616640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 msgr2=0x55c42e6a53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a53e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f391000e930 tx=0x7f3910001e50 comp rx=0 tx=0).stop 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 -- 172.21.15.31:0/821616640 shutdown_connections 2024-04-25T16:00:37.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55c42e28c280 0x55c42e71d5d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c42e2f8270 0x55c42e6a87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c42e2d7590 0x55c42e6a5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f38faffd640 0 client.15093 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:00:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 --2- 172.21.15.31:0/821616640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c42e2cf3b0 0x55c42e6a53e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 -- 172.21.15.31:0/821616640 >> 172.21.15.31:0/821616640 conn(0x55c42e2ec3c0 msgr2=0x55c42e68fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 -- 172.21.15.31:0/821616640 shutdown_connections 2024-04-25T16:00:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 1 -- 172.21.15.31:0/821616640 wait complete. 2024-04-25T16:00:37.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.557+0000 7f3928e48740 20 client.15093 trim_cache size 0 max 16384 2024-04-25T16:00:37.565 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24535, 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 90.015389553999995, 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:00:37.566 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 114.376237895 2024-04-25T16:00:37.567 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:00:37.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 -- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39040afaf0 msgr2=0x7f39040b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39040afaf0 0x7f39040b2000 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f391404d000 tx=0x7f39140f7190 comp rx=0 tx=0).stop 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 -- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 msgr2=0x7f391c1a16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 0x7f391c1a16c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f39140097b0 tx=0x7f3914003c20 comp rx=0 tx=0).stop 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 -- 172.21.15.31:0/2374854219 shutdown_connections 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39040afaf0 0x7f39040b2000 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f391c109a30 0x7f391c1a16c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f391c104d30 0x7f391c19a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 --2- 172.21.15.31:0/2374854219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f391c103af0 0x7f391c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 -- 172.21.15.31:0/2374854219 >> 172.21.15.31:0/2374854219 conn(0x7f391c0fece0 msgr2=0x7f391c105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 -- 172.21.15.31:0/2374854219 shutdown_connections 2024-04-25T16:00:37.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.558+0000 7f3921f96640 1 -- 172.21.15.31:0/2374854219 wait complete. 2024-04-25T16:00:37.570 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T16:00:37.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.682+0000 7f3a7ff57640 1 -- 172.21.15.31:0/1963483986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78103af0 msgr2=0x7f3a781047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.682+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/1963483986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78103af0 0x7f3a781047f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009900 tx=0x7f3a7002a410 comp rx=0 tx=0).stop 2024-04-25T16:00:37.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.682+0000 7f3a7ff57640 1 -- 172.21.15.31:0/1963483986 shutdown_connections 2024-04-25T16:00:37.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.682+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/1963483986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78109a30 0x7f3a7810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.682+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/1963483986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a78104d30 0x7f3a78105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.682+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/1963483986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78103af0 0x7f3a781047f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.682+0000 7f3a7ff57640 1 -- 172.21.15.31:0/1963483986 >> 172.21.15.31:0/1963483986 conn(0x7f3a780fece0 msgr2=0x7f3a78101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.682+0000 7f3a7ff57640 1 -- 172.21.15.31:0/1963483986 shutdown_connections 2024-04-25T16:00:37.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.682+0000 7f3a7ff57640 1 -- 172.21.15.31:0/1963483986 wait complete. 2024-04-25T16:00:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7ff57640 1 Processor -- start 2024-04-25T16:00:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7ff57640 1 -- start start 2024-04-25T16:00:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7ff57640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78103af0 0x7f3a7819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7ff57640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78104d30 0x7f3a7819a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7ff57640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a78109a30 0x7f3a781a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7ff57640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7810f5f0 con 0x7f3a78104d30 2024-04-25T16:00:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7ff57640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7810f470 con 0x7f3a78103af0 2024-04-25T16:00:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7ff57640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7810f770 con 0x7f3a78109a30 2024-04-25T16:00:37.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7d4f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78104d30 0x7f3a7819a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7d4f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78104d30 0x7f3a7819a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45940/0 (socket says 172.21.15.31:45940) 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/3147165257 learned_addr learned my addr 172.21.15.31:0/3147165257 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7e4f4640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a78109a30 0x7f3a781a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.683+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78103af0 0x7f3a7819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/3147165257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a78109a30 msgr2=0x7f3a781a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a78109a30 0x7f3a781a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/3147165257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78103af0 msgr2=0x7f3a7819a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78103af0 0x7f3a7819a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/3147165257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a781a1d90 con 0x7f3a78104d30 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78103af0 0x7f3a7819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:37.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78104d30 0x7f3a7819a790 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d770 tx=0x7f3a6c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a76ffd640 1 -- 172.21.15.31:0/3147165257 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c017070 con 0x7f3a78104d30 2024-04-25T16:00:37.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a7ff57640 1 -- 172.21.15.31:0/3147165257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a781a2080 con 0x7f3a78104d30 2024-04-25T16:00:37.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a76ffd640 1 -- 172.21.15.31:0/3147165257 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a6c0054d0 con 0x7f3a78104d30 2024-04-25T16:00:37.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a76ffd640 1 -- 172.21.15.31:0/3147165257 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c004c30 con 0x7f3a78104d30 2024-04-25T16:00:37.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.684+0000 7f3a7ff57640 1 -- 172.21.15.31:0/3147165257 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a781a25d0 con 0x7f3a78104d30 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a86e07740 1 Processor -- start 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a76ffd640 1 -- 172.21.15.31:0/3147165257 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3a6c004e10 con 0x7f3a78104d30 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a86e07740 1 -- start start 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 0x5604d8f40370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 0x5604d8f408f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f477f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a86e07740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5604d8ea5740 con 0x5604d8b7e590 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a86e07740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5604d8e413a0 con 0x5604d8b763b0 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a86e07740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604d8ea1700 con 0x5604d8b9f270 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a7d4f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a7d4f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f477f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45950/0 (socket says 172.21.15.31:45950) 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/79270 learned_addr learned my addr 172.21.15.31:0/79270 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a7e4f4640 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 0x5604d8f40370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.687+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 0x5604d8f408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/79270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 msgr2=0x5604d8f40370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 0x5604d8f40370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/79270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 msgr2=0x5604d8f408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 0x5604d8f408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/79270 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604d8e5d750 con 0x5604d8b7e590 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a7e4f4640 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 0x5604d8f40370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a76ffd640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3a600afaf0 0x7f3a600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a7e4f4640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3a600afaf0 0x7f3a600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f477f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0e4c10 tx=0x7f3a6c0e6690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e66e74d621bb065c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a76ffd640 1 -- 172.21.15.31:0/3147165257 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3a6c0de6d0 con 0x7f3a78104d30 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a5ffff640 1 -- 172.21.15.31:0/79270 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c020070 con 0x5604d8b7e590 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a5ffff640 1 -- 172.21.15.31:0/79270 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a6c0a66d0 con 0x5604d8b7e590 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.688+0000 7f3a5ffff640 1 -- 172.21.15.31:0/79270 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0a2a80 con 0x5604d8b7e590 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a86e07740 1 -- 172.21.15.31:0/79270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 msgr2=0x5604d8f477f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a86e07740 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f477f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0e4c10 tx=0x7f3a6c0e6690 comp rx=0 tx=0).stop 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a7e4f4640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3a600afaf0 0x7f3a600b2000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x5604d8f41540 tx=0x7f3a68009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a86e07740 1 -- 172.21.15.31:0/79270 shutdown_connections 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a86e07740 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f477f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a86e07740 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 0x5604d8f408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a86e07740 1 --2- 172.21.15.31:0/79270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 0x5604d8f40370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a86e07740 1 -- 172.21.15.31:0/79270 >> 172.21.15.31:0/79270 conn(0x5604d8b933c0 msgr2=0x5604d8f3bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a86e07740 1 -- 172.21.15.31:0/79270 shutdown_connections 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.689+0000 7f3a86e07740 1 -- 172.21.15.31:0/79270 wait complete. 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a86e07740 1 -- start start 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a86e07740 1 Processor -- start 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 0x5604d8f4a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f4a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 0x5604d8f51680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a86e07740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5604d8f48240 con 0x5604d8b7e590 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a86e07740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5604d8e2cfe0 con 0x5604d8b763b0 2024-04-25T16:00:37.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a86e07740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604d8f483c0 con 0x5604d8b9f270 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a86e07740 10 client.0 _ll_register_callbacks cb 0x5604d8e43150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a7d4f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f4a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a7d4f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f4a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45952/0 (socket says 172.21.15.31:45952) 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/4136359363 learned_addr learned my addr 172.21.15.31:0/4136359363 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a7e4f4640 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 0x5604d8f51680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.690+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 0x5604d8f4a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/4136359363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 msgr2=0x5604d8f51680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 0x5604d8f51680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/4136359363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 msgr2=0x5604d8f4a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 0x5604d8f4a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/4136359363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604d8bb72d0 con 0x5604d8b7e590 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f4a790 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0dc230 tx=0x7f3a6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858c89ea853531d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/4136359363 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c020040 con 0x5604d8b7e590 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/4136359363 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3a6c0c11c0 con 0x5604d8b7e590 2024-04-25T16:00:37.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a86e07740 1 -- 172.21.15.31:0/4136359363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a6c0a66d0 con 0x5604d8b7e590 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/4136359363 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3a6c0a6330 con 0x5604d8b7e590 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a86e07740 10 client.15105 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a86e07740 1 -- 172.21.15.31:0/4136359363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5604d8c816b0 con 0x5604d8b7e590 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/4136359363 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a6c0a66d0 con 0x5604d8b7e590 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/4136359363 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3a6c0af070 con 0x5604d8b7e590 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a86e07740 10 client.15105 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a86e07740 10 client.15105 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a86e07740 20 client.15105 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.691+0000 7f3a86e07740 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5604d8b5de40 0x5604d8f579f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.692+0000 7f3a86e07740 4 client.15105 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.692+0000 7f3a86e07740 1 -- 172.21.15.31:0/4136359363 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604d8eb3350 con 0x5604d8b5de40 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.692+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5604d8b5de40 0x5604d8f579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.692+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5604d8b5de40 0x5604d8f579f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6fe524e41a1f10ed server_cookie=5c883f04f7a32bf5 in_seq=0 out_seq=0 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.692+0000 7f3a5d7fa640 10 client.15105 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:00:37.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.693+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/4136359363 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5604d8eb3350 con 0x5604d8b5de40 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.693+0000 7f3a5d7fa640 10 client.15105 handle_command_reply: tid=0 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.693+0000 7f3a86e07740 1 client.15105 shutdown 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.693+0000 7f3a86e07740 1 -- 172.21.15.31:0/4136359363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 msgr2=0x5604d8f4a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.693+0000 7f3a86e07740 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f4a790 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0dc230 tx=0x7f3a6c0046c0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a86e07740 1 -- 172.21.15.31:0/4136359363 shutdown_connections 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a86e07740 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b9f270 0x5604d8f51680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a86e07740 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5604d8b5de40 0x5604d8f579f0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a86e07740 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f4a790 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a5d7fa640 0 client.15105 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a86e07740 1 --2- 172.21.15.31:0/4136359363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b763b0 0x5604d8f4a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a86e07740 1 -- 172.21.15.31:0/4136359363 >> 172.21.15.31:0/4136359363 conn(0x5604d8b933c0 msgr2=0x5604d8f3ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a86e07740 1 -- 172.21.15.31:0/4136359363 shutdown_connections 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a86e07740 1 -- 172.21.15.31:0/4136359363 wait complete. 2024-04-25T16:00:37.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.694+0000 7f3a86e07740 20 client.15105 trim_cache size 0 max 16384 2024-04-25T16:00:37.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:37 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1733652766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:00:37.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:37 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3620016299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:00:37.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a86e07740 1 Processor -- start 2024-04-25T16:00:37.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a86e07740 1 -- start start 2024-04-25T16:00:37.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b763b0 0x5604d8f36a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f515a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 0x5604d8f51ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a86e07740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5604d8f55d30 con 0x5604d8b7e590 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a86e07740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5604d8f56400 con 0x5604d8b9f270 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a86e07740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604d8f56ad0 con 0x5604d8b763b0 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7d4f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7d4f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45956/0 (socket says 172.21.15.31:45956) 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/2515064846 learned_addr learned my addr 172.21.15.31:0/2515064846 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/2515064846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b763b0 0x5604d8f36a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/2515064846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b763b0 msgr2=0x5604d8f36a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/2515064846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b763b0 0x5604d8f36a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7e4f4640 1 --2- 172.21.15.31:0/2515064846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 0x5604d8f51ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/2515064846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 msgr2=0x5604d8f51ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/2515064846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 0x5604d8f51ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7d4f2640 1 -- 172.21.15.31:0/2515064846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604d8f4ac60 con 0x5604d8b7e590 2024-04-25T16:00:37.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.713+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/2515064846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f515a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009610 tx=0x7f3a6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41252efcdc043cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a5f7fe640 1 -- 172.21.15.31:0/2515064846 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c020070 con 0x5604d8b7e590 2024-04-25T16:00:37.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a5f7fe640 1 -- 172.21.15.31:0/2515064846 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a6c0af360 con 0x5604d8b7e590 2024-04-25T16:00:37.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a5f7fe640 1 -- 172.21.15.31:0/2515064846 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c0a6510 con 0x5604d8b7e590 2024-04-25T16:00:37.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a86e07740 1 -- 172.21.15.31:0/2515064846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 msgr2=0x5604d8f515a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a86e07740 1 --2- 172.21.15.31:0/2515064846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f515a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009610 tx=0x7f3a6c004730 comp rx=0 tx=0).stop 2024-04-25T16:00:37.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a86e07740 1 -- 172.21.15.31:0/2515064846 shutdown_connections 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a86e07740 1 --2- 172.21.15.31:0/2515064846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 0x5604d8f51ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a86e07740 1 --2- 172.21.15.31:0/2515064846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b7e590 0x5604d8f515a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a86e07740 1 --2- 172.21.15.31:0/2515064846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b763b0 0x5604d8f36a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a86e07740 1 -- 172.21.15.31:0/2515064846 >> 172.21.15.31:0/2515064846 conn(0x5604d8b933c0 msgr2=0x5604d8f446a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a86e07740 1 -- 172.21.15.31:0/2515064846 shutdown_connections 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.714+0000 7f3a86e07740 1 -- 172.21.15.31:0/2515064846 wait complete. 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a86e07740 1 -- start start 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a86e07740 1 Processor -- start 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b763b0 0x5604d8f4cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b7e590 0x5604d8f4d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a86e07740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 0x5604d8f503a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a86e07740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5604d8f57870 con 0x5604d8b763b0 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a86e07740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5604d8f571a0 con 0x5604d8b9f270 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a86e07740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5604d8f57f40 con 0x5604d8b7e590 2024-04-25T16:00:37.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a86e07740 10 client.0 _ll_register_callbacks cb 0x7f3a70009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a7dcf3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b763b0 0x5604d8f4cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a7dcf3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b763b0 0x5604d8f4cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45972/0 (socket says 172.21.15.31:45972) 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a7dcf3640 1 -- 172.21.15.31:0/1700852119 learned_addr learned my addr 172.21.15.31:0/1700852119 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b7e590 0x5604d8f4d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.715+0000 7f3a7e4f4640 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 0x5604d8f503a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a7dcf3640 1 -- 172.21.15.31:0/1700852119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b7e590 msgr2=0x5604d8f4d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b7e590 0x5604d8f4d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a7dcf3640 1 -- 172.21.15.31:0/1700852119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 msgr2=0x5604d8f503a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 0x5604d8f503a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a7dcf3640 1 -- 172.21.15.31:0/1700852119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604d8f50960 con 0x5604d8b763b0 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b7e590 0x5604d8f4d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a7dcf3640 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b763b0 0x5604d8f4cf20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3a70005ab0 tx=0x7f3a700041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8583b1fefc54188 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a5d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/1700852119 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a70049070 con 0x5604d8b763b0 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/1700852119 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3a70005240 con 0x5604d8b763b0 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a86e07740 1 -- 172.21.15.31:0/1700852119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5604d8ead1c0 con 0x5604d8b763b0 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/1700852119 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3a70035040 con 0x5604d8b763b0 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a86e07740 10 client.15111 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a86e07740 1 -- 172.21.15.31:0/1700852119 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5604d8f51cb0 con 0x5604d8b763b0 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.716+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/1700852119 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5604d8ead1c0 con 0x5604d8b763b0 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/1700852119 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3a7004c030 con 0x5604d8b763b0 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a86e07740 10 client.15111 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a86e07740 10 client.15111 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a86e07740 20 client.15111 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a86e07740 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5604d8b33280 0x5604d8fc3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a86e07740 4 client.15111 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a86e07740 1 -- 172.21.15.31:0/1700852119 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "status"}) -- 0x5604d8e289a0 con 0x5604d8b33280 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5604d8b33280 0x5604d8fc3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a5d7fa640 10 client.15111 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:00:37.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.717+0000 7f3a7d4f2640 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5604d8b33280 0x5604d8fc3a80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bcbcf20e3e7e327b server_cookie=edc217eb8c5c3297 in_seq=0 out_seq=0 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a5d7fa640 1 -- 172.21.15.31:0/1700852119 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5604d8e289a0 con 0x5604d8b33280 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a5d7fa640 10 client.15111 handle_command_reply: tid=0 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 client.15111 shutdown 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 -- 172.21.15.31:0/1700852119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b763b0 msgr2=0x5604d8f4cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b763b0 0x5604d8f4cf20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3a70005ab0 tx=0x7f3a700041a0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 -- 172.21.15.31:0/1700852119 shutdown_connections 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5604d8b9f270 0x5604d8f503a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5604d8b7e590 0x5604d8f4d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5604d8b33280 0x5604d8fc3a80 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 --2- 172.21.15.31:0/1700852119 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5604d8b763b0 0x5604d8f4cf20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 -- 172.21.15.31:0/1700852119 >> 172.21.15.31:0/1700852119 conn(0x5604d8b933c0 msgr2=0x5604d8f3b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a5d7fa640 0 client.15111 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 -- 172.21.15.31:0/1700852119 shutdown_connections 2024-04-25T16:00:37.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 1 -- 172.21.15.31:0/1700852119 wait complete. 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a86e07740 20 client.15111 trim_cache size 0 max 16384 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24553, 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 91.192421503000006, 2024-04-25T16:00:37.725 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:00:37.726 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:00:37.726 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:00:37.726 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 108.618865915 2024-04-25T16:00:37.726 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a7ff57640 1 -- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3a600afaf0 msgr2=0x7f3a600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3a600afaf0 0x7f3a600b2000 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x5604d8f41540 tx=0x7f3a68009600 comp rx=0 tx=0).stop 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a7ff57640 1 -- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78104d30 msgr2=0x7f3a7819a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78104d30 0x7f3a7819a790 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c00d770 tx=0x7f3a6c00dc30 comp rx=0 tx=0).stop 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.718+0000 7f3a7ff57640 1 -- 172.21.15.31:0/3147165257 shutdown_connections 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.719+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3a600afaf0 0x7f3a600b2000 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.719+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a78109a30 0x7f3a781a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.719+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a78104d30 0x7f3a7819a790 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.719+0000 7f3a7ff57640 1 --2- 172.21.15.31:0/3147165257 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a78103af0 0x7f3a7819a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.719+0000 7f3a7ff57640 1 -- 172.21.15.31:0/3147165257 >> 172.21.15.31:0/3147165257 conn(0x7f3a780fece0 msgr2=0x7f3a78105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.719+0000 7f3a7ff57640 1 -- 172.21.15.31:0/3147165257 shutdown_connections 2024-04-25T16:00:37.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.719+0000 7f3a7ff57640 1 -- 172.21.15.31:0/3147165257 wait complete. 2024-04-25T16:00:37.731 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.842+0000 7f3b23b1c640 1 -- 172.21.15.31:0/821235539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c109a40 msgr2=0x7f3b1c10be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.842+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/821235539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c109a40 0x7f3b1c10be60 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009ca0 tx=0x7f3b1402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.842+0000 7f3b23b1c640 1 -- 172.21.15.31:0/821235539 shutdown_connections 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.842+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/821235539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c109a40 0x7f3b1c10be60 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.842+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/821235539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b1c104d40 0x7f3b1c105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.842+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/821235539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b1c103b00 0x7f3b1c104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.842+0000 7f3b23b1c640 1 -- 172.21.15.31:0/821235539 >> 172.21.15.31:0/821235539 conn(0x7f3b1c0fecf0 msgr2=0x7f3b1c1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.842+0000 7f3b23b1c640 1 -- 172.21.15.31:0/821235539 shutdown_connections 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.842+0000 7f3b23b1c640 1 -- 172.21.15.31:0/821235539 wait complete. 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b23b1c640 1 Processor -- start 2024-04-25T16:00:37.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b23b1c640 1 -- start start 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b23b1c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b1c103b00 0x7f3b1c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b23b1c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c104d40 0x7f3b1c19a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b23b1c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b1c109a40 0x7f3b1c1a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b23b1c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c10f630 con 0x7f3b1c104d40 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b23b1c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c10f4b0 con 0x7f3b1c109a40 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b23b1c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3b1c10f7b0 con 0x7f3b1c103b00 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b210b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c104d40 0x7f3b1c19a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b210b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c104d40 0x7f3b1c19a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45988/0 (socket says 172.21.15.31:45988) 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b210b7640 1 -- 172.21.15.31:0/1048087031 learned_addr learned my addr 172.21.15.31:0/1048087031 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b220b9640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b1c109a40 0x7f3b1c1a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b218b8640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b1c103b00 0x7f3b1c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b210b7640 1 -- 172.21.15.31:0/1048087031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b1c103b00 msgr2=0x7f3b1c19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b210b7640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b1c103b00 0x7f3b1c19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b210b7640 1 -- 172.21.15.31:0/1048087031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b1c109a40 msgr2=0x7f3b1c1a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b210b7640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b1c109a40 0x7f3b1c1a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b210b7640 1 -- 172.21.15.31:0/1048087031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b1c1a1ce0 con 0x7f3b1c104d40 2024-04-25T16:00:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.843+0000 7f3b210b7640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c104d40 0x7f3b1c19a6e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00d770 tx=0x7f3b0c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.844+0000 7f3b12ffd640 1 -- 172.21.15.31:0/1048087031 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c017070 con 0x7f3b1c104d40 2024-04-25T16:00:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.844+0000 7f3b12ffd640 1 -- 172.21.15.31:0/1048087031 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b0c0054d0 con 0x7f3b1c104d40 2024-04-25T16:00:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.844+0000 7f3b23b1c640 1 -- 172.21.15.31:0/1048087031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b1c1a1fd0 con 0x7f3b1c104d40 2024-04-25T16:00:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.844+0000 7f3b12ffd640 1 -- 172.21.15.31:0/1048087031 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c004c30 con 0x7f3b1c104d40 2024-04-25T16:00:37.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.844+0000 7f3b23b1c640 1 -- 172.21.15.31:0/1048087031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b1c1a2430 con 0x7f3b1c104d40 2024-04-25T16:00:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.846+0000 7f3b2a9cc740 1 Processor -- start 2024-04-25T16:00:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b2a9cc740 1 -- start start 2024-04-25T16:00:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b12ffd640 1 -- 172.21.15.31:0/1048087031 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3b0c004e10 con 0x7f3b1c104d40 2024-04-25T16:00:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50d2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d083b0 0x5574a50d28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d10590 0x5574a50d97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5574a5037740 con 0x5574a4d31270 2024-04-25T16:00:37.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5574a4fd33a0 con 0x5574a4d10590 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5574a5033700 con 0x5574a4d083b0 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b220b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50d2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b220b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50d2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46000/0 (socket says 172.21.15.31:46000) 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b220b9640 1 -- 172.21.15.31:0/313647233 learned_addr learned my addr 172.21.15.31:0/313647233 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b218b8640 1 --2- 172.21.15.31:0/313647233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d083b0 0x5574a50d28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b210b7640 1 --2- 172.21.15.31:0/313647233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d10590 0x5574a50d97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b220b9640 1 -- 172.21.15.31:0/313647233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d083b0 msgr2=0x5574a50d28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b220b9640 1 --2- 172.21.15.31:0/313647233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d083b0 0x5574a50d28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b220b9640 1 -- 172.21.15.31:0/313647233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d10590 msgr2=0x5574a50d97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b220b9640 1 --2- 172.21.15.31:0/313647233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d10590 0x5574a50d97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.847+0000 7f3b220b9640 1 -- 172.21.15.31:0/313647233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574a4fef750 con 0x5574a4d31270 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.848+0000 7f3b12ffd640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b080afaf0 0x7f3b080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.848+0000 7f3b218b8640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b080afaf0 0x7f3b080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.848+0000 7f3b220b9640 1 --2- 172.21.15.31:0/313647233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50d2370 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3b14035000 tx=0x7f3b1402f4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b5d514609b7568 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.848+0000 7f3b12ffd640 1 -- 172.21.15.31:0/1048087031 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3b0c0de6d0 con 0x7f3b1c104d40 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.848+0000 7f3b07fff640 1 -- 172.21.15.31:0/313647233 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b14031650 con 0x5574a4d31270 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.848+0000 7f3b07fff640 1 -- 172.21.15.31:0/313647233 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b14036030 con 0x5574a4d31270 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.848+0000 7f3b07fff640 1 -- 172.21.15.31:0/313647233 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b140047a0 con 0x5574a4d31270 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/313647233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 msgr2=0x5574a50d2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/313647233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50d2370 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3b14035000 tx=0x7f3b1402f4c0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b218b8640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b080afaf0 0x7f3b080b2000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x5574a50d3750 tx=0x7f3b18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/313647233 shutdown_connections 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/313647233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d10590 0x5574a50d97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/313647233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d083b0 0x5574a50d28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/313647233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50d2370 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/313647233 >> 172.21.15.31:0/313647233 conn(0x5574a4d253c0 msgr2=0x5574a50cdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/313647233 shutdown_connections 2024-04-25T16:00:37.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/313647233 wait complete. 2024-04-25T16:00:37.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.849+0000 7f3b2a9cc740 1 -- start start 2024-04-25T16:00:37.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b2a9cc740 1 Processor -- start 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50dc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d10590 0x5574a50dc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50e35e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5574a50da060 con 0x5574a4d31270 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5574a4fbefe0 con 0x5574a4d083b0 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5574a50da1e0 con 0x5574a4d10590 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b2a9cc740 10 client.0 _ll_register_callbacks cb 0x5574a4fd5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b220b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50e35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b220b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50e35e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46014/0 (socket says 172.21.15.31:46014) 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b220b9640 1 -- 172.21.15.31:0/3152500480 learned_addr learned my addr 172.21.15.31:0/3152500480 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b218b8640 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50dc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b210b7640 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d10590 0x5574a50dc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b220b9640 1 -- 172.21.15.31:0/3152500480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d10590 msgr2=0x5574a50dc680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b220b9640 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d10590 0x5574a50dc680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b220b9640 1 -- 172.21.15.31:0/3152500480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 msgr2=0x5574a50dc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b220b9640 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50dc140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b220b9640 1 -- 172.21.15.31:0/3152500480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574a4d492d0 con 0x5574a4d31270 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.850+0000 7f3b218b8640 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50dc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b220b9640 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50e35e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3b1402ade0 tx=0x7f3b14031780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c471345cba10cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b057fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b057fa640 1 -- 172.21.15.31:0/3152500480 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1404a040 con 0x5574a4d31270 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b057fa640 1 -- 172.21.15.31:0/3152500480 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3b14003b70 con 0x5574a4d31270 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3152500480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b14036030 con 0x5574a4d31270 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b057fa640 1 -- 172.21.15.31:0/3152500480 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3b14004810 con 0x5574a4d31270 2024-04-25T16:00:37.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b2a9cc740 10 client.15123 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3152500480 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5574a4e136b0 con 0x5574a4d31270 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b057fa640 1 -- 172.21.15.31:0/3152500480 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b14036030 con 0x5574a4d31270 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b057fa640 1 -- 172.21.15.31:0/3152500480 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3b1404d030 con 0x5574a4d31270 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b2a9cc740 10 client.15123 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b2a9cc740 10 client.15123 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b2a9cc740 20 client.15123 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5574a4cefe40 0x5574a50e9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b2a9cc740 4 client.15123 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.851+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3152500480 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5574a5045350 con 0x5574a4cefe40 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.852+0000 7f3b218b8640 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5574a4cefe40 0x5574a50e9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.852+0000 7f3b218b8640 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5574a4cefe40 0x5574a50e9b60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ebd581346cdd152 server_cookie=348bd9f3c0f595bb in_seq=0 out_seq=0 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.852+0000 7f3b057fa640 10 client.15123 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.853+0000 7f3b057fa640 1 -- 172.21.15.31:0/3152500480 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5574a5045350 con 0x5574a4cefe40 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.853+0000 7f3b057fa640 10 client.15123 handle_command_reply: tid=0 2024-04-25T16:00:37.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.853+0000 7f3b2a9cc740 1 client.15123 shutdown 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3152500480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 msgr2=0x5574a50e35e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50e35e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3b1402ade0 tx=0x7f3b14031780 comp rx=0 tx=0).stop 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3152500480 shutdown_connections 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5574a4cefe40 0x5574a50e9b60 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50e35e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d10590 0x5574a50dc680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3152500480 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50dc140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b057fa640 0 client.15123 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3152500480 >> 172.21.15.31:0/3152500480 conn(0x5574a4d253c0 msgr2=0x5574a50ccd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3152500480 shutdown_connections 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3152500480 wait complete. 2024-04-25T16:00:37.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.854+0000 7f3b2a9cc740 20 client.15123 trim_cache size 0 max 16384 2024-04-25T16:00:37.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:37 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1733652766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:00:37.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:37 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3620016299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:00:37.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b2a9cc740 1 Processor -- start 2024-04-25T16:00:37.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b2a9cc740 1 -- start start 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50da760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d10590 0x5574a50e3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50daca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5574a50e7ea0 con 0x5574a4d31270 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5574a50e8570 con 0x5574a4d083b0 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5574a50e8c40 con 0x5574a4d10590 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b220b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50daca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b220b9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50daca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46030/0 (socket says 172.21.15.31:46030) 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b220b9640 1 -- 172.21.15.31:0/3958098068 learned_addr learned my addr 172.21.15.31:0/3958098068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b218b8640 1 --2- 172.21.15.31:0/3958098068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50da760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b220b9640 1 -- 172.21.15.31:0/3958098068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d10590 msgr2=0x5574a50e3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b220b9640 1 --2- 172.21.15.31:0/3958098068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d10590 0x5574a50e3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b220b9640 1 -- 172.21.15.31:0/3958098068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 msgr2=0x5574a50da760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b220b9640 1 --2- 172.21.15.31:0/3958098068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50da760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b220b9640 1 -- 172.21.15.31:0/3958098068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574a50d7190 con 0x5574a4d31270 2024-04-25T16:00:37.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.873+0000 7f3b220b9640 1 --2- 172.21.15.31:0/3958098068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50daca0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3b1402fd90 tx=0x7f3b14000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1b97340859b08a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b077fe640 1 -- 172.21.15.31:0/3958098068 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b1402f660 con 0x5574a4d31270 2024-04-25T16:00:37.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b077fe640 1 -- 172.21.15.31:0/3958098068 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b14036070 con 0x5574a4d31270 2024-04-25T16:00:37.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b077fe640 1 -- 172.21.15.31:0/3958098068 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b14003970 con 0x5574a4d31270 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3958098068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 msgr2=0x5574a50daca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3958098068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50daca0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f3b1402fd90 tx=0x7f3b14000f40 comp rx=0 tx=0).stop 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3958098068 shutdown_connections 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3958098068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d31270 0x5574a50daca0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3958098068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d10590 0x5574a50e3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/3958098068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50da760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3958098068 >> 172.21.15.31:0/3958098068 conn(0x5574a4d253c0 msgr2=0x5574a50d5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3958098068 shutdown_connections 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.874+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/3958098068 wait complete. 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b2a9cc740 1 -- start start 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b2a9cc740 1 Processor -- start 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50e0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d10590 0x5574a50e1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b2a9cc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d31270 0x5574a50e1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5574a50e99e0 con 0x5574a4d10590 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5574a50e9310 con 0x5574a4d083b0 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b2a9cc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5574a50ea0b0 con 0x5574a4d31270 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b2a9cc740 10 client.0 _ll_register_callbacks cb 0x7f3b0c0e4420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b210b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d10590 0x5574a50e1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b210b7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d10590 0x5574a50e1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46040/0 (socket says 172.21.15.31:46040) 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.875+0000 7f3b210b7640 1 -- 172.21.15.31:0/2946785714 learned_addr learned my addr 172.21.15.31:0/2946785714 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b220b9640 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d31270 0x5574a50e1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b218b8640 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50e0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b210b7640 1 -- 172.21.15.31:0/2946785714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d31270 msgr2=0x5574a50e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b210b7640 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d31270 0x5574a50e1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b210b7640 1 -- 172.21.15.31:0/2946785714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 msgr2=0x5574a50e0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:00:37.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b210b7640 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50e0ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b210b7640 1 -- 172.21.15.31:0/2946785714 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5574a50e2490 con 0x5574a4d10590 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b210b7640 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d10590 0x5574a50e1010 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0a6b50 tx=0x7f3b0c0a6990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc96f8bcc4f072dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b057fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b057fa640 1 -- 172.21.15.31:0/2946785714 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c020070 con 0x5574a4d10590 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b057fa640 1 -- 172.21.15.31:0/2946785714 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3b0c0af590 con 0x5574a4d10590 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/2946785714 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3b14036070 con 0x5574a4d10590 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b057fa640 1 -- 172.21.15.31:0/2946785714 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f3b0c0af360 con 0x5574a4d10590 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b2a9cc740 10 client.15129 fetch_fsmap learned FSMap version 26 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/2946785714 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5574a50e37e0 con 0x5574a4d10590 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b057fa640 1 -- 172.21.15.31:0/2946785714 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b14036070 con 0x5574a4d10590 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.876+0000 7f3b057fa640 1 -- 172.21.15.31:0/2946785714 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f3b0c0af040 con 0x5574a4d10590 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b2a9cc740 10 client.15129 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:00:37.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b2a9cc740 10 client.15129 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b2a9cc740 20 client.15129 populate_metadata read hostname 'smithi031' 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5574a4cc5280 0x5574a5156090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b2a9cc740 4 client.15129 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/2946785714 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "status"}) -- 0x5574a503f1c0 con 0x5574a4cc5280 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b218b8640 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5574a4cc5280 0x5574a5156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b218b8640 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5574a4cc5280 0x5574a5156090 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feb85023f24d6dd2 server_cookie=d6d3a704e7a9434b in_seq=0 out_seq=0 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b057fa640 10 client.15129 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b057fa640 1 -- 172.21.15.31:0/2946785714 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5574a503f1c0 con 0x5574a4cc5280 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.877+0000 7f3b057fa640 10 client.15129 handle_command_reply: tid=0 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 client.15129 shutdown 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/2946785714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d10590 msgr2=0x5574a50e1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d10590 0x5574a50e1010 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c0a6b50 tx=0x7f3b0c0a6990 comp rx=0 tx=0).stop 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/2946785714 shutdown_connections 2024-04-25T16:00:37.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5574a4cc5280 0x5574a5156090 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5574a4d31270 0x5574a50e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b057fa640 0 client.15129 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5574a4d10590 0x5574a50e1010 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 --2- 172.21.15.31:0/2946785714 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5574a4d083b0 0x5574a50e0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/2946785714 >> 172.21.15.31:0/2946785714 conn(0x5574a4d253c0 msgr2=0x5574a50da6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/2946785714 shutdown_connections 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 1 -- 172.21.15.31:0/2946785714 wait complete. 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b2a9cc740 20 client.15129 trim_cache size 0 max 16384 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24563, 2024-04-25T16:00:37.885 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:00:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:00:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:00:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 94.381843450000005, 2024-04-25T16:00:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:00:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:00:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:00:37.886 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 99.595367230999997 2024-04-25T16:00:37.886 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:00:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 -- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b080afaf0 msgr2=0x7f3b080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b080afaf0 0x7f3b080b2000 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x5574a50d3750 tx=0x7f3b18005f90 comp rx=0 tx=0).stop 2024-04-25T16:00:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 -- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c104d40 msgr2=0x7f3b1c19a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c104d40 0x7f3b1c19a6e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00d770 tx=0x7f3b0c00dc30 comp rx=0 tx=0).stop 2024-04-25T16:00:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 -- 172.21.15.31:0/1048087031 shutdown_connections 2024-04-25T16:00:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3b080afaf0 0x7f3b080b2000 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3b1c109a40 0x7f3b1c1a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3b1c104d40 0x7f3b1c19a6e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 --2- 172.21.15.31:0/1048087031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b1c103b00 0x7f3b1c19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.878+0000 7f3b23b1c640 1 -- 172.21.15.31:0/1048087031 >> 172.21.15.31:0/1048087031 conn(0x7f3b1c0fecf0 msgr2=0x7f3b1c105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.879+0000 7f3b23b1c640 1 -- 172.21.15.31:0/1048087031 shutdown_connections 2024-04-25T16:00:37.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:37.879+0000 7f3b23b1c640 1 -- 172.21.15.31:0/1048087031 wait complete. 2024-04-25T16:00:37.891 INFO:tasks.cephadm:Setup complete, yielding 2024-04-25T16:00:37.891 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T16:00:37.902 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T16:00:37.902 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-25T16:00:37.903 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:37 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1733652766' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:00:37.903 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:37 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3620016299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:00:38.149 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:38.534 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:38 smithi031 ceph-mon[29140]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:38.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:38 smithi063 ceph-mon[32413]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:38.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:38 smithi186 ceph-mon[33258]: pgmap v331: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:39.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.972+0000 7f28a3616700 1 -- 172.21.15.31:0/3370447850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c107320 msgr2=0x7f289c109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:39.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.972+0000 7f28a3616700 1 --2- 172.21.15.31:0/3370447850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c107320 0x7f289c109800 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f288c009990 tx=0x7f288c009ca0 comp rx=0 tx=0).stop 2024-04-25T16:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.972+0000 7f28a3616700 1 -- 172.21.15.31:0/3370447850 shutdown_connections 2024-04-25T16:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.972+0000 7f28a3616700 1 --2- 172.21.15.31:0/3370447850 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f289c076580 0x7f289c106d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.972+0000 7f28a3616700 1 --2- 172.21.15.31:0/3370447850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f289c109e90 0x7f289c10c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.972+0000 7f28a3616700 1 --2- 172.21.15.31:0/3370447850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c107320 0x7f289c109800 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.972+0000 7f28a3616700 1 -- 172.21.15.31:0/3370447850 >> 172.21.15.31:0/3370447850 conn(0x7f289c0fe740 msgr2=0x7f289c100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.972+0000 7f28a3616700 1 -- 172.21.15.31:0/3370447850 shutdown_connections 2024-04-25T16:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.972+0000 7f28a3616700 1 -- 172.21.15.31:0/3370447850 wait complete. 2024-04-25T16:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a3616700 1 Processor -- start 2024-04-25T16:00:39.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a3616700 1 -- start start 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a3616700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c076580 0x7f289c1045a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a3616700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f289c107320 0x7f289c102c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a3616700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f289c109e90 0x7f289c103190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a3616700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f289c10fc30 con 0x7f289c076580 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a3616700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f289c10fab0 con 0x7f289c107320 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a3616700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f289c10fdb0 con 0x7f289c109e90 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a13b2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c076580 0x7f289c1045a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a13b2700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c076580 0x7f289c1045a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46052/0 (socket says 172.21.15.31:46052) 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a13b2700 1 -- 172.21.15.31:0/2806175283 learned_addr learned my addr 172.21.15.31:0/2806175283 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a0bb1700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f289c107320 0x7f289c102c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a13b2700 1 -- 172.21.15.31:0/2806175283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f289c109e90 msgr2=0x7f289c103190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:39.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.973+0000 7f28a1bb3700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f289c109e90 0x7f289c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28a13b2700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f289c109e90 0x7f289c103190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28a13b2700 1 -- 172.21.15.31:0/2806175283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f289c107320 msgr2=0x7f289c102c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28a13b2700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f289c107320 0x7f289c102c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28a13b2700 1 -- 172.21.15.31:0/2806175283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f289c103860 con 0x7f289c076580 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28a0bb1700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f289c107320 0x7f289c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28a1bb3700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f289c109e90 0x7f289c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28a13b2700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c076580 0x7f289c1045a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f289800c9b0 tx=0x7f289800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28927fc700 1 -- 172.21.15.31:0/2806175283 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898007b00 con 0x7f289c076580 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28927fc700 1 -- 172.21.15.31:0/2806175283 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2898007ca0 con 0x7f289c076580 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28927fc700 1 -- 172.21.15.31:0/2806175283 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2898018730 con 0x7f289c076580 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f289c1a4850 con 0x7f289c076580 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.974+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f289c1a4c50 con 0x7f289c076580 2024-04-25T16:00:39.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.975+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f289c051190 con 0x7f289c076580 2024-04-25T16:00:39.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.977+0000 7f28927fc700 1 -- 172.21.15.31:0/2806175283 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f289800f450 con 0x7f289c076580 2024-04-25T16:00:39.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.977+0000 7f28927fc700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28880afb80 0x7f28880b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:39.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.978+0000 7f28a0bb1700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28880afb80 0x7f28880b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:39.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.978+0000 7f28927fc700 1 -- 172.21.15.31:0/2806175283 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f28980decc0 con 0x7f289c076580 2024-04-25T16:00:39.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.978+0000 7f28a0bb1700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28880afb80 0x7f28880b2090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f288c003800 tx=0x7f288c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:39.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:39.982+0000 7f28927fc700 1 -- 172.21.15.31:0/2806175283 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2898014030 con 0x7f289c076580 2024-04-25T16:00:40.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.210+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f289c063ad0 con 0x7f28880afb80 2024-04-25T16:00:40.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.212+0000 7f28927fc700 1 -- 172.21.15.31:0/2806175283 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f289c063ad0 con 0x7f28880afb80 2024-04-25T16:00:40.212 INFO:teuthology.orchestra.run.smithi031.stdout:Backend: cephadm 2024-04-25T16:00:40.212 INFO:teuthology.orchestra.run.smithi031.stdout:Available: Yes 2024-04-25T16:00:40.212 INFO:teuthology.orchestra.run.smithi031.stdout:Paused: No 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28880afb80 msgr2=0x7f28880b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28880afb80 0x7f28880b2090 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f288c003800 tx=0x7f288c01a040 comp rx=0 tx=0).stop 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c076580 msgr2=0x7f289c1045a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c076580 0x7f289c1045a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f289800c9b0 tx=0x7f289800cd70 comp rx=0 tx=0).stop 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 shutdown_connections 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28880afb80 0x7f28880b2090 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f289c109e90 0x7f289c103190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f289c076580 0x7f289c1045a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 --2- 172.21.15.31:0/2806175283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f289c107320 0x7f289c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 >> 172.21.15.31:0/2806175283 conn(0x7f289c0fe740 msgr2=0x7f289c105400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:40.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 shutdown_connections 2024-04-25T16:00:40.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:40.214+0000 7f28a3616700 1 -- 172.21.15.31:0/2806175283 wait complete. 2024-04-25T16:00:40.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:40 smithi031 ceph-mon[29140]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:40.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:40 smithi063 ceph-mon[32413]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:40.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:40 smithi186 ceph-mon[33258]: pgmap v332: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:40.924 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-25T16:00:41.173 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:41.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:41 smithi031 ceph-mon[29140]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:41.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:41 smithi063 ceph-mon[32413]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:41.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:41 smithi186 ceph-mon[33258]: from='client.15135 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:42.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:42 smithi031 ceph-mon[29140]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:42.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:42 smithi063 ceph-mon[32413]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:42.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:42 smithi186 ceph-mon[33258]: pgmap v333: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:43.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.079+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/2392497292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 msgr2=0x7fb2f4103520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:43.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.079+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/2392497292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 0x7fb2f4103520 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4009990 tx=0x7fb2e4009ca0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.079+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/2392497292 shutdown_connections 2024-04-25T16:00:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.079+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/2392497292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2f4104be0 0x7fb2f410b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.079+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/2392497292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 0x7fb2f4103520 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.079+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/2392497292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4104240 0x7fb2f4104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.079+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/2392497292 >> 172.21.15.31:0/2392497292 conn(0x7fb2f40fe750 msgr2=0x7fb2f4100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.079+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/2392497292 shutdown_connections 2024-04-25T16:00:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.080+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/2392497292 wait complete. 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.080+0000 7fb2fa3ae700 1 Processor -- start 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.080+0000 7fb2fa3ae700 1 -- start start 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2fa3ae700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 0x7fb2f419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2fa3ae700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2f4104240 0x7fb2f419ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2fa3ae700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4104be0 0x7fb2f41a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2fa3ae700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f410eb70 con 0x7fb2f4103110 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2fa3ae700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f410e9f0 con 0x7fb2f4104240 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2fa3ae700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb2f410ecf0 con 0x7fb2f4104be0 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 0x7fb2f419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f3fff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 0x7fb2f419a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46074/0 (socket says 172.21.15.31:46074) 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f3fff700 1 -- 172.21.15.31:0/3422938584 learned_addr learned my addr 172.21.15.31:0/3422938584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:43.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f894b700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4104be0 0x7fb2f41a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f37fe700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2f4104240 0x7fb2f419ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f3fff700 1 -- 172.21.15.31:0/3422938584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4104be0 msgr2=0x7fb2f41a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f3fff700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4104be0 0x7fb2f41a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f3fff700 1 -- 172.21.15.31:0/3422938584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2f4104240 msgr2=0x7fb2f419ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f3fff700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2f4104240 0x7fb2f419ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f3fff700 1 -- 172.21.15.31:0/3422938584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2f41a2340 con 0x7fb2f4103110 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.081+0000 7fb2f3fff700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 0x7fb2f419a7e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4000c00 tx=0x7fb2e40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.082+0000 7fb2f17fa700 1 -- 172.21.15.31:0/3422938584 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4004830 con 0x7fb2f4103110 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.082+0000 7fb2f17fa700 1 -- 172.21.15.31:0/3422938584 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2e4005020 con 0x7fb2f4103110 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.082+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2f41a25d0 con 0x7fb2f4103110 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.082+0000 7fb2f17fa700 1 -- 172.21.15.31:0/3422938584 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2e4003d90 con 0x7fb2f4103110 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.082+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2f41a29d0 con 0x7fb2f4103110 2024-04-25T16:00:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.083+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2f4051190 con 0x7fb2f4103110 2024-04-25T16:00:43.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.084+0000 7fb2f17fa700 1 -- 172.21.15.31:0/3422938584 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb2e40051d0 con 0x7fb2f4103110 2024-04-25T16:00:43.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.085+0000 7fb2f17fa700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2dc0afc00 0x7fb2dc0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:43.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.085+0000 7fb2f37fe700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2dc0afc00 0x7fb2dc0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:43.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.085+0000 7fb2f17fa700 1 -- 172.21.15.31:0/3422938584 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb2e40dfa50 con 0x7fb2f4103110 2024-04-25T16:00:43.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.086+0000 7fb2f37fe700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2dc0afc00 0x7fb2dc0b2110 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8005950 tx=0x7fb2e800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:43.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.090+0000 7fb2f17fa700 1 -- 172.21.15.31:0/3422938584 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2e4013070 con 0x7fb2f4103110 2024-04-25T16:00:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.307+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fb2f41007e0 con 0x7fb2dc0afc00 2024-04-25T16:00:43.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.321+0000 7fb2f17fa700 1 -- 172.21.15.31:0/3422938584 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fb2f41007e0 con 0x7fb2dc0afc00 2024-04-25T16:00:43.322 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-25T16:00:43.322 INFO:teuthology.orchestra.run.smithi031.stdout:mds.a smithi031 running (108s) 19s ago 108s 87.7M - 19.0.0-3291-g994e2f20 d53c96779a29 98c0e7e3c52e 2024-04-25T16:00:43.322 INFO:teuthology.orchestra.run.smithi031.stdout:mds.b smithi063 running (105s) 38s ago 104s 92.9M - 19.0.0-3291-g994e2f20 d53c96779a29 497d4c2b60ca 2024-04-25T16:00:43.322 INFO:teuthology.orchestra.run.smithi031.stdout:mds.c smithi186 running (2m) 38s ago 2m 88.2M - 19.0.0-3291-g994e2f20 d53c96779a29 0d441945cd33 2024-04-25T16:00:43.322 INFO:teuthology.orchestra.run.smithi031.stdout:mds.d smithi031 running (2m) 19s ago 2m 83.1M - 19.0.0-3291-g994e2f20 d53c96779a29 625af22b8d0c 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mds.e smithi063 running (2m) 38s ago 2m 83.5M - 19.0.0-3291-g994e2f20 d53c96779a29 5d6e749ec588 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mds.f smithi186 running (2m) 38s ago 2m 85.7M - 19.0.0-3291-g994e2f20 d53c96779a29 2d7978c8c71f 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mds.g smithi031 running (111s) 19s ago 111s 86.5M - 19.0.0-3291-g994e2f20 d53c96779a29 ad720772e5c1 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mds.h smithi063 running (117s) 38s ago 116s 84.6M - 19.0.0-3291-g994e2f20 d53c96779a29 09bf47fc2733 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mds.i smithi186 running (114s) 38s ago 114s 85.4M - 19.0.0-3291-g994e2f20 d53c96779a29 55b78effe522 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mds.j smithi031 running (2m) 19s ago 2m 83.8M - 19.0.0-3291-g994e2f20 d53c96779a29 43107973a76c 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mds.k smithi063 running (2m) 38s ago 2m 83.4M - 19.0.0-3291-g994e2f20 d53c96779a29 184ad245f379 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mds.l smithi186 running (2m) 38s ago 2m 84.7M - 19.0.0-3291-g994e2f20 d53c96779a29 6116e03527e8 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.x smithi031 *:9283,8765 running (9m) 19s ago 9m 583M - 19.0.0-3291-g994e2f20 d53c96779a29 8ac56c1f2c76 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.y smithi063 *:8443,8765 running (7m) 38s ago 7m 523M - 19.0.0-3291-g994e2f20 d53c96779a29 1513da908ef8 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.z smithi186 *:8443,8765 running (7m) 38s ago 7m 523M - 19.0.0-3291-g994e2f20 d53c96779a29 5ec702eb857b 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a smithi031 running (9m) 19s ago 9m 117M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 3c77fdc22f43 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b smithi063 running (7m) 38s ago 7m 106M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 c5ed416e9877 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c smithi186 running (8m) 38s ago 8m 109M 2048M 19.0.0-3291-g994e2f20 d53c96779a29 ef9abd0df575 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (7m) 19s ago 7m 111M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 2d80d52e1ea0 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi063 running (6m) 38s ago 6m 112M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 ce26214f0cda 2024-04-25T16:00:43.323 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi186 running (6m) 38s ago 6m 123M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 4b192dd9dbd4 2024-04-25T16:00:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi031 running (6m) 19s ago 6m 109M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 c678ce22756e 2024-04-25T16:00:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi063 running (5m) 38s ago 5m 112M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 a504b7666b84 2024-04-25T16:00:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi186 running (5m) 38s ago 5m 121M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 f35558d125f2 2024-04-25T16:00:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi031 running (4m) 19s ago 4m 110M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 0e1d38009e03 2024-04-25T16:00:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi063 running (4m) 38s ago 4m 121M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 0a9359540769 2024-04-25T16:00:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 smithi186 running (3m) 38s ago 3m 120M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 27c75e9fe869 2024-04-25T16:00:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 smithi031 running (3m) 19s ago 3m 109M 4273M 19.0.0-3291-g994e2f20 d53c96779a29 1bf01118a0ce 2024-04-25T16:00:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 smithi063 running (3m) 38s ago 3m 121M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 c71d238c17e7 2024-04-25T16:00:43.324 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 smithi186 running (2m) 38s ago 2m 111M 4272M 19.0.0-3291-g994e2f20 d53c96779a29 6b73f091e8d9 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2dc0afc00 msgr2=0x7fb2dc0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2dc0afc00 0x7fb2dc0b2110 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb2e8005950 tx=0x7fb2e800a300 comp rx=0 tx=0).stop 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 msgr2=0x7fb2f419a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 0x7fb2f419a7e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb2e4000c00 tx=0x7fb2e40047b0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 shutdown_connections 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2dc0afc00 0x7fb2dc0b2110 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2f4104be0 0x7fb2f41a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2f4103110 0x7fb2f419a7e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 --2- 172.21.15.31:0/3422938584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2f4104240 0x7fb2f419ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.323+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 >> 172.21.15.31:0/3422938584 conn(0x7fb2f40fe750 msgr2=0x7fb2f41000d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.324+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 shutdown_connections 2024-04-25T16:00:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:43.324+0000 7fb2fa3ae700 1 -- 172.21.15.31:0/3422938584 wait complete. 2024-04-25T16:00:44.032 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-25T16:00:44.289 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:44.664 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:44 smithi031 ceph-mon[29140]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:44.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:44 smithi031 ceph-mon[29140]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:00:44.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:44 smithi063 ceph-mon[32413]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:44.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:44 smithi063 ceph-mon[32413]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:00:44.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:44 smithi186 ceph-mon[33258]: from='client.15141 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:44.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:44 smithi186 ceph-mon[33258]: pgmap v334: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:00:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.099+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/2874139291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 msgr2=0x7fc6bc1034f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.100+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/2874139291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 0x7fc6bc1034f0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8009990 tx=0x7fc6b8009ca0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.100+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/2874139291 shutdown_connections 2024-04-25T16:00:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.100+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/2874139291 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6bc104bb0 0x7fc6bc10b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.100+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/2874139291 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 0x7fc6bc1034f0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.100+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/2874139291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6bc104210 0x7fc6bc104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.100+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/2874139291 >> 172.21.15.31:0/2874139291 conn(0x7fc6bc0fe740 msgr2=0x7fc6bc100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.100+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/2874139291 shutdown_connections 2024-04-25T16:00:46.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.100+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/2874139291 wait complete. 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c3c3a700 1 Processor -- start 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c3c3a700 1 -- start start 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c3c3a700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 0x7fc6bc06bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c3c3a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6bc104210 0x7fc6bc19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c3c3a700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6bc104bb0 0x7fc6bc1a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c19d6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 0x7fc6bc06bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c3c3a700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc10eb30 con 0x7fc6bc1030e0 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c3c3a700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc10e9b0 con 0x7fc6bc104bb0 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c3c3a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc6bc10ecb0 con 0x7fc6bc104210 2024-04-25T16:00:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c19d6700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 0x7fc6bc06bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50374/0 (socket says 172.21.15.31:50374) 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c19d6700 1 -- 172.21.15.31:0/1975200073 learned_addr learned my addr 172.21.15.31:0/1975200073 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.101+0000 7fc6c11d5700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6bc104210 0x7fc6bc19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6c19d6700 1 -- 172.21.15.31:0/1975200073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6bc104210 msgr2=0x7fc6bc19f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6c19d6700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6bc104210 0x7fc6bc19f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6c19d6700 1 -- 172.21.15.31:0/1975200073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6bc104bb0 msgr2=0x7fc6bc1a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6c19d6700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6bc104bb0 0x7fc6bc1a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6c19d6700 1 -- 172.21.15.31:0/1975200073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6bc1a2340 con 0x7fc6bc1030e0 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6c11d5700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6bc104210 0x7fc6bc19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6c19d6700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 0x7fc6bc06bfa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8000c00 tx=0x7fc6b8005b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6b2ffd700 1 -- 172.21.15.31:0/1975200073 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b800b890 con 0x7fc6bc1030e0 2024-04-25T16:00:46.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6bc199310 con 0x7fc6bc1030e0 2024-04-25T16:00:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6b2ffd700 1 -- 172.21.15.31:0/1975200073 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6b8004830 con 0x7fc6bc1030e0 2024-04-25T16:00:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6bc1997e0 con 0x7fc6bc1030e0 2024-04-25T16:00:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.102+0000 7fc6b2ffd700 1 -- 172.21.15.31:0/1975200073 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6b8005400 con 0x7fc6bc1030e0 2024-04-25T16:00:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.103+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6bc051190 con 0x7fc6bc1030e0 2024-04-25T16:00:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.106+0000 7fc6b2ffd700 1 -- 172.21.15.31:0/1975200073 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc6b800ba30 con 0x7fc6bc1030e0 2024-04-25T16:00:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.107+0000 7fc6b2ffd700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6a80afc00 0x7fc6a80b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.107+0000 7fc6c11d5700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6a80afc00 0x7fc6a80b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.107+0000 7fc6b2ffd700 1 -- 172.21.15.31:0/1975200073 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc6b80e45b0 con 0x7fc6bc1030e0 2024-04-25T16:00:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.107+0000 7fc6c11d5700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6a80afc00 0x7fc6a80b2110 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc06b950 tx=0x7fc6ac009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.113+0000 7fc6b2ffd700 1 -- 172.21.15.31:0/1975200073 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc6b8013070 con 0x7fc6bc1030e0 2024-04-25T16:00:46.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.331+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fc6bc063b90 con 0x7fc6a80afc00 2024-04-25T16:00:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.337+0000 7fc6b2ffd700 1 -- 172.21.15.31:0/1975200073 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fc6bc063b90 con 0x7fc6a80afc00 2024-04-25T16:00:46.338 INFO:teuthology.orchestra.run.smithi031.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-25T16:00:46.338 INFO:teuthology.orchestra.run.smithi031.stdout:mds.all 12/12 41s ago 2m smithi031=a;smithi031=d;smithi031=g;smithi031=j;smithi063=b;smithi063=e;smithi063=h;smithi063=k;smithi186=c;smithi186=f;smithi186=i;smithi186=l;count:12 2024-04-25T16:00:46.338 INFO:teuthology.orchestra.run.smithi031.stdout:mgr 3/3 41s ago 7m smithi031=x;smithi063=y;smithi186=z;count:3 2024-04-25T16:00:46.338 INFO:teuthology.orchestra.run.smithi031.stdout:mon 3/3 41s ago 8m smithi031:172.21.15.31=a;smithi063:172.21.15.63=b;smithi186:172.21.15.186=c;count:3 2024-04-25T16:00:46.338 INFO:teuthology.orchestra.run.smithi031.stdout:osd 12 41s ago - 2024-04-25T16:00:46.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.339+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6a80afc00 msgr2=0x7fc6a80b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:46.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.339+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6a80afc00 0x7fc6a80b2110 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc06b950 tx=0x7fc6ac009670 comp rx=0 tx=0).stop 2024-04-25T16:00:46.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.339+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 msgr2=0x7fc6bc06bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:46.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.339+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 0x7fc6bc06bfa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fc6b8000c00 tx=0x7fc6b8005b00 comp rx=0 tx=0).stop 2024-04-25T16:00:46.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.340+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 shutdown_connections 2024-04-25T16:00:46.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.340+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6a80afc00 0x7fc6a80b2110 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.340+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc6bc104bb0 0x7fc6bc1a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.340+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc6bc1030e0 0x7fc6bc06bfa0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.340+0000 7fc6c3c3a700 1 --2- 172.21.15.31:0/1975200073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6bc104210 0x7fc6bc19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:46.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.340+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 >> 172.21.15.31:0/1975200073 conn(0x7fc6bc0fe740 msgr2=0x7fc6bc107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:46.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.340+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 shutdown_connections 2024-04-25T16:00:46.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:46.340+0000 7fc6c3c3a700 1 -- 172.21.15.31:0/1975200073 wait complete. 2024-04-25T16:00:46.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:46 smithi031 ceph-mon[29140]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:00:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:46 smithi063 ceph-mon[32413]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:00:46.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:46 smithi186 ceph-mon[33258]: pgmap v335: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:00:47.128 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-25T16:00:47.380 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:47.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:47 smithi031 ceph-mon[29140]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:47.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:47 smithi063 ceph-mon[32413]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:47.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:47 smithi186 ceph-mon[33258]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:48.601 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:48 smithi031 ceph-mon[29140]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:48.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:48 smithi063 ceph-mon[32413]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:48.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:48 smithi186 ceph-mon[33258]: pgmap v336: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:49.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.317+0000 7fc9873b9700 1 -- 172.21.15.31:0/2621414922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 msgr2=0x7fc980104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:49.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.317+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2621414922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 0x7fc980104640 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fc970009990 tx=0x7fc970009ca0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.317+0000 7fc9873b9700 1 -- 172.21.15.31:0/2621414922 shutdown_connections 2024-04-25T16:00:49.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.317+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2621414922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc980104bd0 0x7fc98010b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.317+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2621414922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc980103100 0x7fc980103510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.317+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2621414922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 0x7fc980104640 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.317+0000 7fc9873b9700 1 -- 172.21.15.31:0/2621414922 >> 172.21.15.31:0/2621414922 conn(0x7fc9800fe760 msgr2=0x7fc980100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:49.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.317+0000 7fc9873b9700 1 -- 172.21.15.31:0/2621414922 shutdown_connections 2024-04-25T16:00:49.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.317+0000 7fc9873b9700 1 -- 172.21.15.31:0/2621414922 wait complete. 2024-04-25T16:00:49.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc9873b9700 1 Processor -- start 2024-04-25T16:00:49.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc9873b9700 1 -- start start 2024-04-25T16:00:49.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc9873b9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc980103100 0x7fc980106ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:49.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc9873b9700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 0x7fc980105570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:49.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc9873b9700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc980104bd0 0x7fc980105ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc9873b9700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc98010eb70 con 0x7fc980104230 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc9873b9700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc98010e9f0 con 0x7fc980104bd0 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc9873b9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc98010ecf0 con 0x7fc980103100 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc984954700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 0x7fc980105570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc984954700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 0x7fc980105570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50396/0 (socket says 172.21.15.31:50396) 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc984954700 1 -- 172.21.15.31:0/2232185963 learned_addr learned my addr 172.21.15.31:0/2232185963 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc985155700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc980103100 0x7fc980106ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.318+0000 7fc985956700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc980104bd0 0x7fc980105ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc984954700 1 -- 172.21.15.31:0/2232185963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc980103100 msgr2=0x7fc980106ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc984954700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc980103100 0x7fc980106ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc984954700 1 -- 172.21.15.31:0/2232185963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc980104bd0 msgr2=0x7fc980105ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc984954700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc980104bd0 0x7fc980105ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc984954700 1 -- 172.21.15.31:0/2232185963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc980106210 con 0x7fc980104230 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc985956700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc980104bd0 0x7fc980105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc985155700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc980103100 0x7fc980106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:00:49.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc984954700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 0x7fc980105570 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc9700098e0 tx=0x7fc9700047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:49.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc9767fc700 1 -- 172.21.15.31:0/2232185963 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970026070 con 0x7fc980104230 2024-04-25T16:00:49.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9801a48b0 con 0x7fc980104230 2024-04-25T16:00:49.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc9767fc700 1 -- 172.21.15.31:0/2232185963 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc970021410 con 0x7fc980104230 2024-04-25T16:00:49.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc9767fc700 1 -- 172.21.15.31:0/2232185963 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc970017650 con 0x7fc980104230 2024-04-25T16:00:49.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.319+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9801a4c50 con 0x7fc980104230 2024-04-25T16:00:49.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.320+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc980051190 con 0x7fc980104230 2024-04-25T16:00:49.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.322+0000 7fc9767fc700 1 -- 172.21.15.31:0/2232185963 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc970029070 con 0x7fc980104230 2024-04-25T16:00:49.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.322+0000 7fc9767fc700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc96c0afb80 0x7fc96c0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:49.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.323+0000 7fc985155700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc96c0afb80 0x7fc96c0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:49.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.323+0000 7fc9767fc700 1 -- 172.21.15.31:0/2232185963 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc9700df9e0 con 0x7fc980104230 2024-04-25T16:00:49.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.323+0000 7fc985155700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc96c0afb80 0x7fc96c0b2090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc980104130 tx=0x7fc97c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:49.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.328+0000 7fc9767fc700 1 -- 172.21.15.31:0/2232185963 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc970013070 con 0x7fc980104230 2024-04-25T16:00:49.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.541+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fc980100770 con 0x7fc96c0afb80 2024-04-25T16:00:49.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.543+0000 7fc9767fc700 1 -- 172.21.15.31:0/2232185963 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fc980100770 con 0x7fc96c0afb80 2024-04-25T16:00:49.544 INFO:teuthology.orchestra.run.smithi031.stdout:HOST ADDR LABELS STATUS 2024-04-25T16:00:49.544 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 172.21.15.31 2024-04-25T16:00:49.544 INFO:teuthology.orchestra.run.smithi031.stdout:smithi063 172.21.15.63 2024-04-25T16:00:49.544 INFO:teuthology.orchestra.run.smithi031.stdout:smithi186 172.21.15.186 2024-04-25T16:00:49.544 INFO:teuthology.orchestra.run.smithi031.stdout:3 hosts in cluster 2024-04-25T16:00:49.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc96c0afb80 msgr2=0x7fc96c0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:49.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc96c0afb80 0x7fc96c0b2090 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc980104130 tx=0x7fc97c009260 comp rx=0 tx=0).stop 2024-04-25T16:00:49.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 msgr2=0x7fc980105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:49.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 0x7fc980105570 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fc9700098e0 tx=0x7fc9700047b0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 shutdown_connections 2024-04-25T16:00:49.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc96c0afb80 0x7fc96c0b2090 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc980104bd0 0x7fc980105ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc980103100 0x7fc980106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 --2- 172.21.15.31:0/2232185963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc980104230 0x7fc980105570 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 >> 172.21.15.31:0/2232185963 conn(0x7fc9800fe760 msgr2=0x7fc980100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 shutdown_connections 2024-04-25T16:00:49.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:49.545+0000 7fc9873b9700 1 -- 172.21.15.31:0/2232185963 wait complete. 2024-04-25T16:00:50.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-25T16:00:50.571 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:50.693 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:50 smithi031 ceph-mon[29140]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:50.693 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:50 smithi031 ceph-mon[29140]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:50.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:50 smithi063 ceph-mon[32413]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:50.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:50 smithi063 ceph-mon[32413]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:50.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:50 smithi186 ceph-mon[33258]: pgmap v337: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:00:50.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:50 smithi186 ceph-mon[33258]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:52.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.413+0000 7fed16e6f700 1 -- 172.21.15.31:0/2214789352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 msgr2=0x7fed1010b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:52.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.413+0000 7fed16e6f700 1 --2- 172.21.15.31:0/2214789352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 0x7fed1010b460 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fed0800b0a0 tx=0x7fed0800b3b0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.413+0000 7fed16e6f700 1 -- 172.21.15.31:0/2214789352 shutdown_connections 2024-04-25T16:00:52.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.413+0000 7fed16e6f700 1 --2- 172.21.15.31:0/2214789352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 0x7fed1010b460 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.413+0000 7fed16e6f700 1 --2- 172.21.15.31:0/2214789352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed101030e0 0x7fed101034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.413+0000 7fed16e6f700 1 --2- 172.21.15.31:0/2214789352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed10104210 0x7fed10104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.413+0000 7fed16e6f700 1 -- 172.21.15.31:0/2214789352 >> 172.21.15.31:0/2214789352 conn(0x7fed100fe740 msgr2=0x7fed10100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.413+0000 7fed16e6f700 1 -- 172.21.15.31:0/2214789352 shutdown_connections 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.413+0000 7fed16e6f700 1 -- 172.21.15.31:0/2214789352 wait complete. 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed16e6f700 1 Processor -- start 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed16e6f700 1 -- start start 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed16e6f700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed101030e0 0x7fed1006bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed16e6f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed10104210 0x7fed1019f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed16e6f700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 0x7fed101a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed16e6f700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed1010eb60 con 0x7fed10104bb0 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed16e6f700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fed1010e9e0 con 0x7fed101030e0 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed16e6f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed1010ece0 con 0x7fed10104210 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed1540c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 0x7fed101a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed14c0b700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed101030e0 0x7fed1006bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed1540c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 0x7fed101a1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:50422/0 (socket says 172.21.15.31:50422) 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed1540c700 1 -- 172.21.15.31:0/1337563647 learned_addr learned my addr 172.21.15.31:0/1337563647 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.414+0000 7fed0ffff700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed10104210 0x7fed1019f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:52.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed1540c700 1 -- 172.21.15.31:0/1337563647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed10104210 msgr2=0x7fed1019f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed1540c700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed10104210 0x7fed1019f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed1540c700 1 -- 172.21.15.31:0/1337563647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed101030e0 msgr2=0x7fed1006bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed1540c700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed101030e0 0x7fed1006bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed1540c700 1 -- 172.21.15.31:0/1337563647 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed101a2280 con 0x7fed10104bb0 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed1540c700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 0x7fed101a1a50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fed0800b070 tx=0x7fed08004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed0dffb700 1 -- 172.21.15.31:0/1337563647 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08021070 con 0x7fed10104bb0 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed0dffb700 1 -- 172.21.15.31:0/1337563647 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed08007c80 con 0x7fed10104bb0 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed101a2510 con 0x7fed10104bb0 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed0dffb700 1 -- 172.21.15.31:0/1337563647 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed08005280 con 0x7fed10104bb0 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.415+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed101a2a50 con 0x7fed10104bb0 2024-04-25T16:00:52.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.416+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed10051190 con 0x7fed10104bb0 2024-04-25T16:00:52.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.419+0000 7fed0dffb700 1 -- 172.21.15.31:0/1337563647 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fed08024070 con 0x7fed10104bb0 2024-04-25T16:00:52.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.419+0000 7fed0dffb700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecf80afb80 0x7fecf80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:52.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.419+0000 7fed14c0b700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecf80afb80 0x7fecf80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:52.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.419+0000 7fed0dffb700 1 -- 172.21.15.31:0/1337563647 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fed0800f020 con 0x7fed10104bb0 2024-04-25T16:00:52.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.420+0000 7fed14c0b700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecf80afb80 0x7fecf80b2090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fed10104110 tx=0x7fed0000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:52.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.423+0000 7fed0dffb700 1 -- 172.21.15.31:0/1337563647 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed0802b020 con 0x7fed10104bb0 2024-04-25T16:00:52.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.641+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7fed10063ad0 con 0x7fecf80afb80 2024-04-25T16:00:52.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.643+0000 7fed0dffb700 1 -- 172.21.15.31:0/1337563647 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7fed10063ad0 con 0x7fecf80afb80 2024-04-25T16:00:52.644 INFO:teuthology.orchestra.run.smithi031.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-25T16:00:52.644 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53310008400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T16:00:52.644 INFO:teuthology.orchestra.run.smithi031.stdout:smithi063 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5344005B400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-25T16:00:52.644 INFO:teuthology.orchestra.run.smithi031.stdout:smithi186 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CB400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-25T16:00:52.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecf80afb80 msgr2=0x7fecf80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:52.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecf80afb80 0x7fecf80b2090 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fed10104110 tx=0x7fed0000a380 comp rx=0 tx=0).stop 2024-04-25T16:00:52.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 msgr2=0x7fed101a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:52.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 0x7fed101a1a50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fed0800b070 tx=0x7fed08004e40 comp rx=0 tx=0).stop 2024-04-25T16:00:52.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 shutdown_connections 2024-04-25T16:00:52.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecf80afb80 0x7fecf80b2090 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed101030e0 0x7fed1006bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed10104bb0 0x7fed101a1a50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 --2- 172.21.15.31:0/1337563647 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed10104210 0x7fed1019f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:52.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 >> 172.21.15.31:0/1337563647 conn(0x7fed100fe740 msgr2=0x7fed10107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:52.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 shutdown_connections 2024-04-25T16:00:52.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:52.645+0000 7fed16e6f700 1 -- 172.21.15.31:0/1337563647 wait complete. 2024-04-25T16:00:52.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:52 smithi031 ceph-mon[29140]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:52.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:52 smithi063 ceph-mon[32413]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:52.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:52 smithi186 ceph-mon[33258]: pgmap v338: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:00:53.489 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-25T16:00:53.500 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T16:00:53.501 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-25T16:00:53.753 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:53.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:53 smithi063 ceph-mon[32413]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:53.959 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:53 smithi186 ceph-mon[33258]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:53.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:53 smithi031 ceph-mon[29140]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-25T16:00:54.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:54 smithi063 ceph-mon[32413]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:00:54.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:54 smithi186 ceph-mon[33258]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:00:54.932 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:54 smithi031 ceph-mon[29140]: pgmap v339: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.480+0000 7fa176fa7700 1 -- 172.21.15.31:0/22542191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104bb0 msgr2=0x7fa17010b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.480+0000 7fa176fa7700 1 --2- 172.21.15.31:0/22542191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104bb0 0x7fa17010b460 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa16800b0a0 tx=0x7fa16800b3b0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.480+0000 7fa176fa7700 1 -- 172.21.15.31:0/22542191 shutdown_connections 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.480+0000 7fa176fa7700 1 --2- 172.21.15.31:0/22542191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104bb0 0x7fa17010b460 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.480+0000 7fa176fa7700 1 --2- 172.21.15.31:0/22542191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1701030e0 0x7fa1701034f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.480+0000 7fa176fa7700 1 --2- 172.21.15.31:0/22542191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa170104210 0x7fa170104620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.480+0000 7fa176fa7700 1 -- 172.21.15.31:0/22542191 >> 172.21.15.31:0/22542191 conn(0x7fa1700fe740 msgr2=0x7fa170100bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.480+0000 7fa176fa7700 1 -- 172.21.15.31:0/22542191 shutdown_connections 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.480+0000 7fa176fa7700 1 -- 172.21.15.31:0/22542191 wait complete. 2024-04-25T16:00:55.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa176fa7700 1 Processor -- start 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa176fa7700 1 -- start start 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa176fa7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1701030e0 0x7fa17010d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa176fa7700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104210 0x7fa17010bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa176fa7700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa170104bb0 0x7fa17010c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa176fa7700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa17010fc40 con 0x7fa170104210 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa176fa7700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa17010fac0 con 0x7fa170104bb0 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa176fa7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa17010fdc0 con 0x7fa1701030e0 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104210 0x7fa17010bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104210 0x7fa17010bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40170/0 (socket says 172.21.15.31:40170) 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa174d43700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1701030e0 0x7fa17010d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa16ffff700 1 -- 172.21.15.31:0/1584696177 learned_addr learned my addr 172.21.15.31:0/1584696177 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa175544700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa170104bb0 0x7fa17010c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa16ffff700 1 -- 172.21.15.31:0/1584696177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1701030e0 msgr2=0x7fa17010d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa16ffff700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1701030e0 0x7fa17010d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa16ffff700 1 -- 172.21.15.31:0/1584696177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa170104bb0 msgr2=0x7fa17010c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.481+0000 7fa16ffff700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa170104bb0 0x7fa17010c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.482+0000 7fa16ffff700 1 -- 172.21.15.31:0/1584696177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa17010cb70 con 0x7fa170104210 2024-04-25T16:00:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.482+0000 7fa16ffff700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104210 0x7fa17010bed0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00d950 tx=0x7fa15c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.482+0000 7fa16dffb700 1 -- 172.21.15.31:0/1584696177 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c016070 con 0x7fa170104210 2024-04-25T16:00:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.482+0000 7fa16dffb700 1 -- 172.21.15.31:0/1584696177 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa15c009df0 con 0x7fa170104210 2024-04-25T16:00:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.482+0000 7fa16dffb700 1 -- 172.21.15.31:0/1584696177 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c00f3c0 con 0x7fa170104210 2024-04-25T16:00:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.482+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1701a47f0 con 0x7fa170104210 2024-04-25T16:00:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.482+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1701a4cd0 con 0x7fa170104210 2024-04-25T16:00:55.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.483+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa170051190 con 0x7fa170104210 2024-04-25T16:00:55.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.485+0000 7fa16dffb700 1 -- 172.21.15.31:0/1584696177 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa15c019070 con 0x7fa170104210 2024-04-25T16:00:55.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.486+0000 7fa16dffb700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1600afb80 0x7fa1600b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:55.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.486+0000 7fa174d43700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1600afb80 0x7fa1600b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:55.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.486+0000 7fa16dffb700 1 -- 172.21.15.31:0/1584696177 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa15c0de860 con 0x7fa170104210 2024-04-25T16:00:55.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.487+0000 7fa174d43700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1600afb80 0x7fa1600b2090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa170104110 tx=0x7fa16400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:55.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.490+0000 7fa16dffb700 1 -- 172.21.15.31:0/1584696177 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa15c0a73f0 con 0x7fa170104210 2024-04-25T16:00:55.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.744+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fa1700642a0 con 0x7fa170104210 2024-04-25T16:00:55.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.745+0000 7fa16dffb700 1 -- 172.21.15.31:0/1584696177 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 26 v26) ==== 76+0+3080 (secure 0 0 0) 0x7fa15c0a6cf0 con 0x7fa170104210 2024-04-25T16:00:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:e26 2024-04-25T16:00:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-25T16:00:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T16:00:55.746 INFO:teuthology.orchestra.run.smithi031.stdout:legacy client fscid: 1 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:Filesystem 'cephfs' (1) 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:fs_name cephfs 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 26 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-04-25T15:59:02.905418+0000 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-04-25T15:59:10.264996+0000 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:tableserver 0 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:root 0 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:session_timeout 300 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:session_autoclose 300 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:max_file_size 1099511627776 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:max_xattr_size 65536 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:required_client_features {} 2024-04-25T16:00:55.747 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure 0 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:last_failure_osd_epoch 0 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:max_mds 5 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:in 0,1,2,3,4 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:up {0=24563,1=24553,2=24535,3=24521,4=14631} 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:failed 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:damaged 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:stopped 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:data_pools [3] 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:metadata_pool 2 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:inline_data disabled 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:balancer 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:bal_rank_mask -1 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:standby_count_wanted 0 2024-04-25T16:00:55.748 INFO:teuthology.orchestra.run.smithi031.stdout:qdb_cluster leader: 24563 members: 14631,24553,24563,24521,24535 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.b{0:24563} state up:active seq 3 addr [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.f{0:24517} state up:standby-replay seq 1 addr [v2:172.21.15.186:6834/1885074975,v1:172.21.15.186:6835/1885074975] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.i{1:24553} state up:active seq 6 addr [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.e{1:24488} state up:standby-replay seq 1 addr [v2:172.21.15.63:6834/1584006856,v1:172.21.15.63:6835/1584006856] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.c{2:24535} state up:active seq 8 addr [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.l{2:24460} state up:standby-replay seq 1 addr [v2:172.21.15.186:6832/1701597397,v1:172.21.15.186:6833/1701597397] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.h{3:24521} state up:active seq 7 addr [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.k{3:24455} state up:standby-replay seq 1 addr [v2:172.21.15.63:6832/927313408,v1:172.21.15.63:6833/927313408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.a{4:14631} state up:active seq 5 addr [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.g{4:14613} state up:standby-replay seq 1 addr [v2:172.21.15.31:6838/3019802236,v1:172.21.15.31:6839/3019802236] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout:Standby daemons: 2024-04-25T16:00:55.749 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:00:55.750 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.j{-1:14538} state up:standby seq 1 addr [v2:172.21.15.31:6834/3224554516,v1:172.21.15.31:6835/3224554516] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.750 INFO:teuthology.orchestra.run.smithi031.stdout:[mds.d{-1:14550} state up:standby seq 1 addr [v2:172.21.15.31:6836/533105912,v1:172.21.15.31:6837/533105912] compat {c=[1],r=[1],i=[1fff]}] 2024-04-25T16:00:55.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1600afb80 msgr2=0x7fa1600b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1600afb80 0x7fa1600b2090 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa170104110 tx=0x7fa16400b410 comp rx=0 tx=0).stop 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104210 msgr2=0x7fa17010bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104210 0x7fa17010bed0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00d950 tx=0x7fa15c00dc60 comp rx=0 tx=0).stop 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 shutdown_connections 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa1600afb80 0x7fa1600b2090 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa170104bb0 0x7fa17010c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1701030e0 0x7fa17010d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 --2- 172.21.15.31:0/1584696177 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa170104210 0x7fa17010bed0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 >> 172.21.15.31:0/1584696177 conn(0x7fa1700fe740 msgr2=0x7fa170107440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 shutdown_connections 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:55.748+0000 7fa176fa7700 1 -- 172.21.15.31:0/1584696177 wait complete. 2024-04-25T16:00:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T16:00:56.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-25T16:00:56.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:56 smithi031 ceph-mon[29140]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:00:56.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:56 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1584696177' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T16:00:56.788 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:00:56.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:56 smithi063 ceph-mon[32413]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:00:56.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:56 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1584696177' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T16:00:56.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:56 smithi186 ceph-mon[33258]: pgmap v340: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:00:56.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:56 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1584696177' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-25T16:00:58.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.653+0000 7f6a9a090700 1 -- 172.21.15.31:0/3976463509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 msgr2=0x7f6a94106da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:58.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.653+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3976463509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 0x7f6a94106da0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009990 tx=0x7f6a84009ca0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.653+0000 7f6a9a090700 1 -- 172.21.15.31:0/3976463509 shutdown_connections 2024-04-25T16:00:58.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.653+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3976463509 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a94109ec0 0x7f6a9410c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.653+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3976463509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 0x7f6a94106da0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.653+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3976463509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a94107370 0x7f6a94109830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.653+0000 7f6a9a090700 1 -- 172.21.15.31:0/3976463509 >> 172.21.15.31:0/3976463509 conn(0x7f6a940fe750 msgr2=0x7f6a94100c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:58.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.654+0000 7f6a9a090700 1 -- 172.21.15.31:0/3976463509 shutdown_connections 2024-04-25T16:00:58.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.654+0000 7f6a9a090700 1 -- 172.21.15.31:0/3976463509 wait complete. 2024-04-25T16:00:58.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.654+0000 7f6a9a090700 1 Processor -- start 2024-04-25T16:00:58.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.654+0000 7f6a9a090700 1 -- start start 2024-04-25T16:00:58.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a9a090700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 0x7f6a9419a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:58.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a9a090700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a94107370 0x7f6a9419adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:58.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a9a090700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a94109ec0 0x7f6a941a1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:58.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a9a090700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9410fc50 con 0x7f6a9406c180 2024-04-25T16:00:58.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a9a090700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9410fad0 con 0x7f6a94107370 2024-04-25T16:00:58.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a9a090700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6a9410fdd0 con 0x7f6a94109ec0 2024-04-25T16:00:58.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a937fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 0x7f6a9419a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:58.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a937fe700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 0x7f6a9419a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40196/0 (socket says 172.21.15.31:40196) 2024-04-25T16:00:58.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a937fe700 1 -- 172.21.15.31:0/3490735013 learned_addr learned my addr 172.21.15.31:0/3490735013 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:58.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a92ffd700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a94107370 0x7f6a9419adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:58.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a93fff700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a94109ec0 0x7f6a941a1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:58.661 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:58 smithi031 ceph-mon[29140]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:58.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a937fe700 1 -- 172.21.15.31:0/3490735013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a94109ec0 msgr2=0x7f6a941a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:58.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a937fe700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a94109ec0 0x7f6a941a1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a937fe700 1 -- 172.21.15.31:0/3490735013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a94107370 msgr2=0x7f6a9419adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:58.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a937fe700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a94107370 0x7f6a9419adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.655+0000 7f6a937fe700 1 -- 172.21.15.31:0/3490735013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a941a2370 con 0x7f6a9406c180 2024-04-25T16:00:58.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.656+0000 7f6a937fe700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 0x7f6a9419a880 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6a84003800 tx=0x7f6a840047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.656+0000 7f6a90ff9700 1 -- 172.21.15.31:0/3490735013 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a84027070 con 0x7f6a9406c180 2024-04-25T16:00:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.656+0000 7f6a90ff9700 1 -- 172.21.15.31:0/3490735013 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a840040c0 con 0x7f6a9406c180 2024-04-25T16:00:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.656+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a94102b80 con 0x7f6a9406c180 2024-04-25T16:00:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.656+0000 7f6a90ff9700 1 -- 172.21.15.31:0/3490735013 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a84022d90 con 0x7f6a9406c180 2024-04-25T16:00:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.656+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a94103080 con 0x7f6a9406c180 2024-04-25T16:00:58.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.657+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a94051190 con 0x7f6a9406c180 2024-04-25T16:00:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.658+0000 7f6a90ff9700 1 -- 172.21.15.31:0/3490735013 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6a8402a070 con 0x7f6a9406c180 2024-04-25T16:00:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.659+0000 7f6a90ff9700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6a7c0afc00 0x7f6a7c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.659+0000 7f6a92ffd700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6a7c0afc00 0x7f6a7c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.659+0000 7f6a90ff9700 1 -- 172.21.15.31:0/3490735013 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6a840df970 con 0x7f6a9406c180 2024-04-25T16:00:58.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.660+0000 7f6a92ffd700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6a7c0afc00 0x7f6a7c0b2110 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a9419b930 tx=0x7f6a88013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:58.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.664+0000 7f6a90ff9700 1 -- 172.21.15.31:0/3490735013 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a84013070 con 0x7f6a9406c180 2024-04-25T16:00:58.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:58 smithi063 ceph-mon[32413]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.869+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f6a94104520 con 0x7f6a9406c180 2024-04-25T16:00:58.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.870+0000 7f6a90ff9700 1 -- 172.21.15.31:0/3490735013 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v77) ==== 56+0+5275 (secure 0 0 0) 0x7f6a840a85f0 con 0x7f6a9406c180 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:epoch 77 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:fsid 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:created 2024-04-25T15:50:52.065833+0000 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:modified 2024-04-25T15:59:50.662616+0000 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:crush_version 26 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:full_ratio 0.95 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:backfillfull_ratio 0.9 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:nearfull_ratio 0.85 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:require_min_compat_client luminous 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:min_compat_client luminous 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:require_osd_release squid 2024-04-25T16:00:58.871 INFO:teuthology.orchestra.run.smithi031.stdout:stretch_mode_enabled false 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:max_osd 12 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 up in weight 1 up_from 8 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6802/2387250399,v1:172.21.15.31:6803/2387250399] [v2:172.21.15.31:6804/2387250399,v1:172.21.15.31:6805/2387250399] exists,up cd266854-c19f-4bb8-9461-5b329b2aa0b7 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 up in weight 1 up_from 13 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6800/1253577374,v1:172.21.15.63:6801/1253577374] [v2:172.21.15.63:6802/1253577374,v1:172.21.15.63:6803/1253577374] exists,up 4b7ee32d-a410-468a-ad35-a2113b0bc4d5 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 up in weight 1 up_from 18 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6800/1381996284,v1:172.21.15.186:6801/1381996284] [v2:172.21.15.186:6802/1381996284,v1:172.21.15.186:6803/1381996284] exists,up e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 up in weight 1 up_from 25 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6810/1212428989,v1:172.21.15.31:6811/1212428989] [v2:172.21.15.31:6812/1212428989,v1:172.21.15.31:6813/1212428989] exists,up 5c56990d-8026-4d79-bb6f-f1ee793e6a66 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 up in weight 1 up_from 31 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6808/2163634076,v1:172.21.15.63:6809/2163634076] [v2:172.21.15.63:6810/2163634076,v1:172.21.15.63:6811/2163634076] exists,up e948c581-682c-4d78-9e92-d70304b5076e 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 up in weight 1 up_from 36 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6808/1998951869,v1:172.21.15.186:6809/1998951869] [v2:172.21.15.186:6810/1998951869,v1:172.21.15.186:6811/1998951869] exists,up 79c9c971-2a3e-4117-843f-efd5270a8de3 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 up in weight 1 up_from 42 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6818/3847071731,v1:172.21.15.31:6819/3847071731] [v2:172.21.15.31:6820/3847071731,v1:172.21.15.31:6821/3847071731] exists,up b8955799-23fe-4161-890c-c77eb55760b2 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 up in weight 1 up_from 47 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6816/4096527906,v1:172.21.15.63:6817/4096527906] [v2:172.21.15.63:6818/4096527906,v1:172.21.15.63:6819/4096527906] exists,up 33003c67-ad68-41a1-98eb-aea9138f1432 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.8 up in weight 1 up_from 53 up_thru 73 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6816/82309382,v1:172.21.15.186:6817/82309382] [v2:172.21.15.186:6818/82309382,v1:172.21.15.186:6819/82309382] exists,up 75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.9 up in weight 1 up_from 59 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.31:6826/3856015669,v1:172.21.15.31:6827/3856015669] [v2:172.21.15.31:6828/3856015669,v1:172.21.15.31:6829/3856015669] exists,up 37886022-d7c3-400c-aa91-02eedf2a12ee 2024-04-25T16:00:58.872 INFO:teuthology.orchestra.run.smithi031.stdout:osd.10 up in weight 1 up_from 64 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.63:6824/1440937910,v1:172.21.15.63:6825/1440937910] [v2:172.21.15.63:6826/1440937910,v1:172.21.15.63:6827/1440937910] exists,up 97d9d39e-2bd0-47a2-b5a4-cca5bde83613 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:osd.11 up in weight 1 up_from 70 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6824/2008780285,v1:172.21.15.186:6825/2008780285] [v2:172.21.15.186:6826/2008780285,v1:172.21.15.186:6827/2008780285] exists,up ba832dc5-0d0e-420c-9430-12fdc737a86a 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.c [2,1] 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.f [8,11] 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 2.1d [2,4,8,11] 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.b [3,9] 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.11 [4,6] 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.1d [4,8] 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.33 [5,9] 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:pg_upmap_items 3.36 [4,9] 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/2182379244 expires 2024-04-26T15:51:49.219867+0000 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3143860956 expires 2024-04-26T15:51:49.219867+0000 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/3323797811 expires 2024-04-26T15:51:49.219867+0000 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/3023595010 expires 2024-04-26T15:51:49.219867+0000 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/3549555101 expires 2024-04-26T15:51:21.857593+0000 2024-04-25T16:00:58.873 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6801/3549555101 expires 2024-04-26T15:51:21.857593+0000 2024-04-25T16:00:58.874 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/650992382 expires 2024-04-26T15:51:21.857593+0000 2024-04-25T16:00:58.874 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:6800/3023595010 expires 2024-04-26T15:51:49.219867+0000 2024-04-25T16:00:58.874 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/106399623 expires 2024-04-26T15:51:21.857593+0000 2024-04-25T16:00:58.874 INFO:teuthology.orchestra.run.smithi031.stdout:blocklist 172.21.15.31:0/2862878115 expires 2024-04-26T15:51:21.857593+0000 2024-04-25T16:00:58.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6a7c0afc00 msgr2=0x7f6a7c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:58.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6a7c0afc00 0x7f6a7c0b2110 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f6a9419b930 tx=0x7f6a88013040 comp rx=0 tx=0).stop 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 msgr2=0x7f6a9419a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 0x7f6a9419a880 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6a84003800 tx=0x7f6a840047b0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 shutdown_connections 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6a7c0afc00 0x7f6a7c0b2110 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a94109ec0 0x7f6a941a1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6a9406c180 0x7f6a9419a880 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 --2- 172.21.15.31:0/3490735013 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a94107370 0x7f6a9419adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 >> 172.21.15.31:0/3490735013 conn(0x7f6a940fe750 msgr2=0x7f6a941058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 shutdown_connections 2024-04-25T16:00:58.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:58.873+0000 7f6a9a090700 1 -- 172.21.15.31:0/3490735013 wait complete. 2024-04-25T16:00:58.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:58 smithi186 ceph-mon[33258]: pgmap v341: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:00:59.684 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:00:59 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3490735013' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T16:00:59.717 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-25T16:00:59.728 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:00:59.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.848+0000 7f0615f1f640 1 -- 172.21.15.31:0/3754553807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610109a40 msgr2=0x7f061010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:59.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.848+0000 7f0615f1f640 1 --2- 172.21.15.31:0/3754553807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610109a40 0x7f061010be60 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0608009ca0 tx=0x7f060802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:00:59.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.849+0000 7f0615f1f640 1 -- 172.21.15.31:0/3754553807 shutdown_connections 2024-04-25T16:00:59.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.849+0000 7f0615f1f640 1 --2- 172.21.15.31:0/3754553807 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610109a40 0x7f061010be60 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:59.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.849+0000 7f0615f1f640 1 --2- 172.21.15.31:0/3754553807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0610104d40 0x7f0610105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:59.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.849+0000 7f0615f1f640 1 --2- 172.21.15.31:0/3754553807 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0610103b00 0x7f0610104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:59.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.849+0000 7f0615f1f640 1 -- 172.21.15.31:0/3754553807 >> 172.21.15.31:0/3754553807 conn(0x7f06100fed10 msgr2=0x7f06101011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:00:59.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.849+0000 7f0615f1f640 1 -- 172.21.15.31:0/3754553807 shutdown_connections 2024-04-25T16:00:59.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.849+0000 7f0615f1f640 1 -- 172.21.15.31:0/3754553807 wait complete. 2024-04-25T16:00:59.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.849+0000 7f0615f1f640 1 Processor -- start 2024-04-25T16:00:59.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.849+0000 7f0615f1f640 1 -- start start 2024-04-25T16:00:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f0615f1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610103b00 0x7f061019a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f0615f1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0610104d40 0x7f061019a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f0615f1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0610109a40 0x7f06101a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f0615f1f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f061010f610 con 0x7f0610103b00 2024-04-25T16:00:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f0615f1f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f061010f490 con 0x7f0610104d40 2024-04-25T16:00:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f0615f1f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f061010f790 con 0x7f0610109a40 2024-04-25T16:00:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610103b00 0x7f061019a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:59.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610103b00 0x7f061019a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40214/0 (socket says 172.21.15.31:40214) 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060f7fe640 1 -- 172.21.15.31:0/1557280111 learned_addr learned my addr 172.21.15.31:0/1557280111 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060effd640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0610104d40 0x7f061019a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060f7fe640 1 -- 172.21.15.31:0/1557280111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0610109a40 msgr2=0x7f06101a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060ffff640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0610109a40 0x7f06101a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060f7fe640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0610109a40 0x7f06101a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060f7fe640 1 -- 172.21.15.31:0/1557280111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0610104d40 msgr2=0x7f061019a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060f7fe640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0610104d40 0x7f061019a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060f7fe640 1 -- 172.21.15.31:0/1557280111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06101a1d50 con 0x7f0610103b00 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060effd640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0610104d40 0x7f061019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060ffff640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0610109a40 0x7f06101a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060f7fe640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610103b00 0x7f061019a210 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f060000c970 tx=0x7f060000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060cff9640 1 -- 172.21.15.31:0/1557280111 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600004220 con 0x7f0610103b00 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060cff9640 1 -- 172.21.15.31:0/1557280111 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f06000043c0 con 0x7f0610103b00 2024-04-25T16:00:59.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.850+0000 7f060cff9640 1 -- 172.21.15.31:0/1557280111 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060000f450 con 0x7f0610103b00 2024-04-25T16:00:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.851+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06101a1fe0 con 0x7f0610103b00 2024-04-25T16:00:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.851+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06101a24f0 con 0x7f0610103b00 2024-04-25T16:00:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.852+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0610108850 con 0x7f0610103b00 2024-04-25T16:00:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.853+0000 7f060cff9640 1 -- 172.21.15.31:0/1557280111 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f06000079b0 con 0x7f0610103b00 2024-04-25T16:00:59.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.854+0000 7f060cff9640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f05f80afb70 0x7f05f80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:00:59.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.854+0000 7f060cff9640 1 -- 172.21.15.31:0/1557280111 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f06000ded50 con 0x7f0610103b00 2024-04-25T16:00:59.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.854+0000 7f060effd640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f05f80afb70 0x7f05f80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:00:59.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.854+0000 7f060effd640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f05f80afb70 0x7f05f80b2080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0604005fd0 tx=0x7f0604005ec0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:00:59.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:00:59.858+0000 7f060cff9640 1 -- 172.21.15.31:0/1557280111 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0600014030 con 0x7f0610103b00 2024-04-25T16:00:59.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:00:59 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3490735013' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T16:00:59.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:00:59 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3490735013' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-25T16:01:00.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.044+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f061002dd20 con 0x7f0610103b00 2024-04-25T16:01:00.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.046+0000 7f060cff9640 1 -- 172.21.15.31:0/1557280111 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f06000a79c0 con 0x7f0610103b00 2024-04-25T16:01:00.047 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:00.047 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f05f80afb70 msgr2=0x7f05f80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f05f80afb70 0x7f05f80b2080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0604005fd0 tx=0x7f0604005ec0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610103b00 msgr2=0x7f061019a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610103b00 0x7f061019a210 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f060000c970 tx=0x7f060000ce30 comp rx=0 tx=0).stop 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 shutdown_connections 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f05f80afb70 0x7f05f80b2080 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0610109a40 0x7f06101a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0610104d40 0x7f061019a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 --2- 172.21.15.31:0/1557280111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0610103b00 0x7f061019a210 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 >> 172.21.15.31:0/1557280111 conn(0x7f06100fed10 msgr2=0x7f0610105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 shutdown_connections 2024-04-25T16:01:00.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.048+0000 7f0615f1f640 1 -- 172.21.15.31:0/1557280111 wait complete. 2024-04-25T16:01:00.060 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:59:00.978038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:59:01.829535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:01:00.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:01:00.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.176+0000 7fec74af2640 1 -- 172.21.15.31:0/572293283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec701029e0 msgr2=0x7fec701036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.176+0000 7fec74af2640 1 --2- 172.21.15.31:0/572293283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec701029e0 0x7fec701036e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fec60009900 tx=0x7fec6002a410 comp rx=0 tx=0).stop 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.176+0000 7fec74af2640 1 -- 172.21.15.31:0/572293283 shutdown_connections 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.176+0000 7fec74af2640 1 --2- 172.21.15.31:0/572293283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec70108920 0x7fec7010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.176+0000 7fec74af2640 1 --2- 172.21.15.31:0/572293283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec70103c20 0x7fec70104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.176+0000 7fec74af2640 1 --2- 172.21.15.31:0/572293283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec701029e0 0x7fec701036e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.176+0000 7fec74af2640 1 -- 172.21.15.31:0/572293283 >> 172.21.15.31:0/572293283 conn(0x7fec700fdbd0 msgr2=0x7fec70100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.176+0000 7fec74af2640 1 -- 172.21.15.31:0/572293283 shutdown_connections 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.176+0000 7fec74af2640 1 -- 172.21.15.31:0/572293283 wait complete. 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec74af2640 1 Processor -- start 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec74af2640 1 -- start start 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec74af2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec701029e0 0x7fec7019a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec74af2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec70103c20 0x7fec7019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec74af2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec70108920 0x7fec701a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec74af2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec7010f5d0 con 0x7fec70103c20 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec74af2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec7010f450 con 0x7fec70108920 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec74af2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec7010f750 con 0x7fec701029e0 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec70103c20 0x7fec7019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec70103c20 0x7fec7019aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40238/0 (socket says 172.21.15.31:40238) 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6dd9b640 1 -- 172.21.15.31:0/1960209806 learned_addr learned my addr 172.21.15.31:0/1960209806 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6ed9d640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec70108920 0x7fec701a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6e59c640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec701029e0 0x7fec7019a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6dd9b640 1 -- 172.21.15.31:0/1960209806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec701029e0 msgr2=0x7fec7019a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6dd9b640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec701029e0 0x7fec7019a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6dd9b640 1 -- 172.21.15.31:0/1960209806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec70108920 msgr2=0x7fec701a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6dd9b640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec70108920 0x7fec701a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6dd9b640 1 -- 172.21.15.31:0/1960209806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec701a1fe0 con 0x7fec70103c20 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.177+0000 7fec6e59c640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec701029e0 0x7fec7019a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.178+0000 7fec6dd9b640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec70103c20 0x7fec7019aa90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fec6400c920 tx=0x7fec6400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:00.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.178+0000 7fec5f7fe640 1 -- 172.21.15.31:0/1960209806 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec64007c70 con 0x7fec70103c20 2024-04-25T16:01:00.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.178+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec701a22d0 con 0x7fec70103c20 2024-04-25T16:01:00.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.178+0000 7fec5f7fe640 1 -- 172.21.15.31:0/1960209806 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec64007e10 con 0x7fec70103c20 2024-04-25T16:01:00.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.178+0000 7fec5f7fe640 1 -- 172.21.15.31:0/1960209806 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec64018820 con 0x7fec70103c20 2024-04-25T16:01:00.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.178+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec701a2870 con 0x7fec70103c20 2024-04-25T16:01:00.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.179+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec70107730 con 0x7fec70103c20 2024-04-25T16:01:00.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.180+0000 7fec5f7fe640 1 -- 172.21.15.31:0/1960209806 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fec6400f450 con 0x7fec70103c20 2024-04-25T16:01:00.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.181+0000 7fec5f7fe640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec580afb70 0x7fec580b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.181+0000 7fec6e59c640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec580afb70 0x7fec580b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.181+0000 7fec5f7fe640 1 -- 172.21.15.31:0/1960209806 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fec640def60 con 0x7fec70103c20 2024-04-25T16:01:00.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.181+0000 7fec6e59c640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec580afb70 0x7fec580b2080 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fec60009230 tx=0x7fec60000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:00.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.185+0000 7fec5f7fe640 1 -- 172.21.15.31:0/1960209806 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fec64014030 con 0x7fec70103c20 2024-04-25T16:01:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.370+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fec7002dc70 con 0x7fec70103c20 2024-04-25T16:01:00.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.370+0000 7fec5f7fe640 1 -- 172.21.15.31:0/1960209806 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7fec640a7c60 con 0x7fec70103c20 2024-04-25T16:01:00.372 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.372+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec580afb70 msgr2=0x7fec580b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.372+0000 7fec74af2640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec580afb70 0x7fec580b2080 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fec60009230 tx=0x7fec60000bc0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.372+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec70103c20 msgr2=0x7fec7019aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.372+0000 7fec74af2640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec70103c20 0x7fec7019aa90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fec6400c920 tx=0x7fec6400cde0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.373+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 shutdown_connections 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.373+0000 7fec74af2640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec580afb70 0x7fec580b2080 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.373+0000 7fec74af2640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec70108920 0x7fec701a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.373+0000 7fec74af2640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec70103c20 0x7fec7019aa90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.373+0000 7fec74af2640 1 --2- 172.21.15.31:0/1960209806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec701029e0 0x7fec7019a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.373+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 >> 172.21.15.31:0/1960209806 conn(0x7fec700fdbd0 msgr2=0x7fec70104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.373+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 shutdown_connections 2024-04-25T16:01:00.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.373+0000 7fec74af2640 1 -- 172.21.15.31:0/1960209806 wait complete. 2024-04-25T16:01:00.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:01:00.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.500+0000 7fb3ed106640 1 -- 172.21.15.31:0/1232018846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e81029f0 msgr2=0x7fb3e81036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.500+0000 7fb3ed106640 1 --2- 172.21.15.31:0/1232018846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e81029f0 0x7fb3e81036f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009900 tx=0x7fb3d802a410 comp rx=0 tx=0).stop 2024-04-25T16:01:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.500+0000 7fb3ed106640 1 -- 172.21.15.31:0/1232018846 shutdown_connections 2024-04-25T16:01:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.500+0000 7fb3ed106640 1 --2- 172.21.15.31:0/1232018846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3e8108930 0x7fb3e810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.500+0000 7fb3ed106640 1 --2- 172.21.15.31:0/1232018846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3e8103c30 0x7fb3e8104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.500+0000 7fb3ed106640 1 --2- 172.21.15.31:0/1232018846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e81029f0 0x7fb3e81036f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.500+0000 7fb3ed106640 1 -- 172.21.15.31:0/1232018846 >> 172.21.15.31:0/1232018846 conn(0x7fb3e80fdbc0 msgr2=0x7fb3e8100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.500+0000 7fb3ed106640 1 -- 172.21.15.31:0/1232018846 shutdown_connections 2024-04-25T16:01:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.500+0000 7fb3ed106640 1 -- 172.21.15.31:0/1232018846 wait complete. 2024-04-25T16:01:00.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.501+0000 7fb3ed106640 1 Processor -- start 2024-04-25T16:01:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.501+0000 7fb3ed106640 1 -- start start 2024-04-25T16:01:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.501+0000 7fb3ed106640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3e81029f0 0x7fb3e8075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.501+0000 7fb3ed106640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e8103c30 0x7fb3e819f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.501+0000 7fb3ed106640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3e8108930 0x7fb3e81a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.501+0000 7fb3ed106640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e810e5a0 con 0x7fb3e8103c30 2024-04-25T16:01:00.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.501+0000 7fb3ed106640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e810e420 con 0x7fb3e8108930 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.501+0000 7fb3ed106640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb3e810e720 con 0x7fb3e81029f0 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e8103c30 0x7fb3e819f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e8103c30 0x7fb3e819f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40242/0 (socket says 172.21.15.31:40242) 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e659c640 1 -- 172.21.15.31:0/258684718 learned_addr learned my addr 172.21.15.31:0/258684718 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e759e640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3e8108930 0x7fb3e81a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e6d9d640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3e81029f0 0x7fb3e8075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e659c640 1 -- 172.21.15.31:0/258684718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3e81029f0 msgr2=0x7fb3e8075860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e659c640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3e81029f0 0x7fb3e8075860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e659c640 1 -- 172.21.15.31:0/258684718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3e8108930 msgr2=0x7fb3e81a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e659c640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3e8108930 0x7fb3e81a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e659c640 1 -- 172.21.15.31:0/258684718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3e81a1d30 con 0x7fb3e8103c30 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3e659c640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e8103c30 0x7fb3e819f030 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000c920 tx=0x7fb3d000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:00.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3dffff640 1 -- 172.21.15.31:0/258684718 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0007c70 con 0x7fb3e8103c30 2024-04-25T16:01:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3e81a1fc0 con 0x7fb3e8103c30 2024-04-25T16:01:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3e81a2550 con 0x7fb3e8103c30 2024-04-25T16:01:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3dffff640 1 -- 172.21.15.31:0/258684718 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3d0007e10 con 0x7fb3e8103c30 2024-04-25T16:01:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.502+0000 7fb3dffff640 1 -- 172.21.15.31:0/258684718 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3d0018820 con 0x7fb3e8103c30 2024-04-25T16:01:00.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.503+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3c80052c0 con 0x7fb3e8103c30 2024-04-25T16:01:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.505+0000 7fb3dffff640 1 -- 172.21.15.31:0/258684718 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb3d00189c0 con 0x7fb3e8103c30 2024-04-25T16:01:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.506+0000 7fb3dffff640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb3d40afaf0 0x7fb3d40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.506+0000 7fb3e6d9d640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb3d40afaf0 0x7fb3d40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.506+0000 7fb3dffff640 1 -- 172.21.15.31:0/258684718 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb3d00def20 con 0x7fb3e8103c30 2024-04-25T16:01:00.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.506+0000 7fb3e6d9d640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb3d40afaf0 0x7fb3d40b2000 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb3d802aa00 tx=0x7fb3d8031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:00.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.509+0000 7fb3dffff640 1 -- 172.21.15.31:0/258684718 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3d0014030 con 0x7fb3e8103c30 2024-04-25T16:01:00.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.695+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb3c8005880 con 0x7fb3e8103c30 2024-04-25T16:01:00.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.695+0000 7fb3dffff640 1 -- 172.21.15.31:0/258684718 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7fb3d00a7b40 con 0x7fb3e8103c30 2024-04-25T16:01:00.697 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb3d40afaf0 msgr2=0x7fb3d40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb3d40afaf0 0x7fb3d40b2000 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb3d802aa00 tx=0x7fb3d8031400 comp rx=0 tx=0).stop 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e8103c30 msgr2=0x7fb3e819f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e8103c30 0x7fb3e819f030 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb3d000c920 tx=0x7fb3d000cde0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 shutdown_connections 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb3d40afaf0 0x7fb3d40b2000 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb3e8108930 0x7fb3e81a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb3e8103c30 0x7fb3e819f030 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 --2- 172.21.15.31:0/258684718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3e81029f0 0x7fb3e8075860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.697+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 >> 172.21.15.31:0/258684718 conn(0x7fb3e80fdbc0 msgr2=0x7fb3e8104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.698+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 shutdown_connections 2024-04-25T16:01:00.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.698+0000 7fb3ed106640 1 -- 172.21.15.31:0/258684718 wait complete. 2024-04-25T16:01:00.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:00 smithi031 ceph-mon[29140]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:01:00.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:00 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1557280111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:00.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:00 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1960209806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:00.711 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:01:00.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.824+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4268314046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db4108920 msgr2=0x7f6db410ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.824+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4268314046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db4108920 0x7f6db410ad40 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6dac009ca0 tx=0x7f6dac02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.824+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4268314046 shutdown_connections 2024-04-25T16:01:00.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.824+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4268314046 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db4108920 0x7f6db410ad40 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.824+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4268314046 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6db4103c20 0x7f6db4104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.824+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4268314046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db41029e0 0x7f6db41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.824+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4268314046 >> 172.21.15.31:0/4268314046 conn(0x7f6db40fdbd0 msgr2=0x7f6db4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:00.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.824+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4268314046 shutdown_connections 2024-04-25T16:01:00.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.824+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4268314046 wait complete. 2024-04-25T16:01:00.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db8ebd640 1 Processor -- start 2024-04-25T16:01:00.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db8ebd640 1 -- start start 2024-04-25T16:01:00.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db8ebd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db41029e0 0x7f6db4199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db8ebd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6db4103c20 0x7f6db4199950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db8ebd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db4108920 0x7f6db41a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db8ebd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6db410e510 con 0x7f6db41029e0 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db8ebd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6db410e390 con 0x7f6db4103c20 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db8ebd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6db410e690 con 0x7f6db4108920 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db41029e0 0x7f6db4199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db41029e0 0x7f6db4199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40262/0 (socket says 172.21.15.31:40262) 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db259c640 1 -- 172.21.15.31:0/4059070532 learned_addr learned my addr 172.21.15.31:0/4059070532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db1d9b640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6db4103c20 0x7f6db4199950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db2d9d640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db4108920 0x7f6db41a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db259c640 1 -- 172.21.15.31:0/4059070532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db4108920 msgr2=0x7f6db41a08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db259c640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db4108920 0x7f6db41a08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db259c640 1 -- 172.21.15.31:0/4059070532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6db4103c20 msgr2=0x7f6db4199950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db259c640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6db4103c20 0x7f6db4199950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:00.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db259c640 1 -- 172.21.15.31:0/4059070532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6db41a0f50 con 0x7f6db41029e0 2024-04-25T16:01:00.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db2d9d640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db4108920 0x7f6db41a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:00.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.825+0000 7f6db259c640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db41029e0 0x7f6db4199410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005d00 tx=0x7f6da400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:00.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.826+0000 7f6da37fe640 1 -- 172.21.15.31:0/4059070532 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da4017070 con 0x7f6db41029e0 2024-04-25T16:01:00.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.826+0000 7f6da37fe640 1 -- 172.21.15.31:0/4059070532 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6da4009c30 con 0x7f6db41029e0 2024-04-25T16:01:00.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.826+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6db41a1240 con 0x7f6db41029e0 2024-04-25T16:01:00.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.826+0000 7f6da37fe640 1 -- 172.21.15.31:0/4059070532 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6da400f620 con 0x7f6db41029e0 2024-04-25T16:01:00.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.826+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6db41a1790 con 0x7f6db41029e0 2024-04-25T16:01:00.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.827+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6db4107730 con 0x7f6db41029e0 2024-04-25T16:01:00.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.828+0000 7f6da37fe640 1 -- 172.21.15.31:0/4059070532 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6da401a070 con 0x7f6db41029e0 2024-04-25T16:01:00.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.829+0000 7f6da37fe640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d9c0afb70 0x7f6d9c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:00.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.829+0000 7f6db1d9b640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d9c0afb70 0x7f6d9c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:00.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.829+0000 7f6da37fe640 1 -- 172.21.15.31:0/4059070532 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6da40de5f0 con 0x7f6db41029e0 2024-04-25T16:01:00.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.829+0000 7f6db1d9b640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d9c0afb70 0x7f6d9c0b2080 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6db419a4b0 tx=0x7f6da800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:00.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:00.833+0000 7f6da37fe640 1 -- 172.21.15.31:0/4059070532 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6da4009de0 con 0x7f6db41029e0 2024-04-25T16:01:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:00 smithi063 ceph-mon[32413]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:01:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:00 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1557280111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:00 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1960209806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:00 smithi186 ceph-mon[33258]: pgmap v342: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:01:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:00 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1557280111' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:00 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1960209806' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:01.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.018+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6db402dd30 con 0x7f6db41029e0 2024-04-25T16:01:01.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.019+0000 7f6da37fe640 1 -- 172.21.15.31:0/4059070532 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7f6da40a71e0 con 0x7f6db41029e0 2024-04-25T16:01:01.021 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d9c0afb70 msgr2=0x7f6d9c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d9c0afb70 0x7f6d9c0b2080 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6db419a4b0 tx=0x7f6da800b390 comp rx=0 tx=0).stop 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db41029e0 msgr2=0x7f6db4199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db41029e0 0x7f6db4199410 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005d00 tx=0x7f6da400dde0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 shutdown_connections 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d9c0afb70 0x7f6d9c0b2080 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db4108920 0x7f6db41a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6db4103c20 0x7f6db4199950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 --2- 172.21.15.31:0/4059070532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6db41029e0 0x7f6db4199410 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 >> 172.21.15.31:0/4059070532 conn(0x7f6db40fdbd0 msgr2=0x7f6db4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:01.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 shutdown_connections 2024-04-25T16:01:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.021+0000 7f6db8ebd640 1 -- 172.21.15.31:0/4059070532 wait complete. 2024-04-25T16:01:01.035 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:01:01.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.144+0000 7f652a985640 1 -- 172.21.15.31:0/4186345972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524103ad0 msgr2=0x7f65241047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.144+0000 7f652a985640 1 --2- 172.21.15.31:0/4186345972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524103ad0 0x7f65241047d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f6514009900 tx=0x7f651402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:01.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.144+0000 7f652a985640 1 -- 172.21.15.31:0/4186345972 shutdown_connections 2024-04-25T16:01:01.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.144+0000 7f652a985640 1 --2- 172.21.15.31:0/4186345972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6524109a10 0x7f652410be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.144+0000 7f652a985640 1 --2- 172.21.15.31:0/4186345972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6524104d10 0x7f6524105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.144+0000 7f652a985640 1 --2- 172.21.15.31:0/4186345972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524103ad0 0x7f65241047d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.144+0000 7f652a985640 1 -- 172.21.15.31:0/4186345972 >> 172.21.15.31:0/4186345972 conn(0x7f65240fece0 msgr2=0x7f6524101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.144+0000 7f652a985640 1 -- 172.21.15.31:0/4186345972 shutdown_connections 2024-04-25T16:01:01.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.144+0000 7f652a985640 1 -- 172.21.15.31:0/4186345972 wait complete. 2024-04-25T16:01:01.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f652a985640 1 Processor -- start 2024-04-25T16:01:01.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f652a985640 1 -- start start 2024-04-25T16:01:01.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f652a985640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6524103ad0 0x7f6524077960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f652a985640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6524104d10 0x7f6524076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f652a985640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524109a10 0x7f6524076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f652a985640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f652410f5d0 con 0x7f6524109a10 2024-04-25T16:01:01.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f652a985640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f652410f450 con 0x7f6524103ad0 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f652a985640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f652410f750 con 0x7f6524104d10 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f6528f22640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524109a10 0x7f6524076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.145+0000 7f6528f22640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524109a10 0x7f6524076540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40278/0 (socket says 172.21.15.31:40278) 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f6528f22640 1 -- 172.21.15.31:0/1559338528 learned_addr learned my addr 172.21.15.31:0/1559338528 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f65237fe640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6524104d10 0x7f6524076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f6523fff640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6524103ad0 0x7f6524077960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f6528f22640 1 -- 172.21.15.31:0/1559338528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6524104d10 msgr2=0x7f6524076000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f6528f22640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6524104d10 0x7f6524076000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f6528f22640 1 -- 172.21.15.31:0/1559338528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6524103ad0 msgr2=0x7f6524077960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f6528f22640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6524103ad0 0x7f6524077960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f6528f22640 1 -- 172.21.15.31:0/1559338528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6524076be0 con 0x7f6524109a10 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f6523fff640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6524103ad0 0x7f6524077960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f6528f22640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524109a10 0x7f6524076540 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f651c00eaa0 tx=0x7f651c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f65217fa640 1 -- 172.21.15.31:0/1559338528 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c00cdb0 con 0x7f6524109a10 2024-04-25T16:01:01.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65241a42d0 con 0x7f6524109a10 2024-04-25T16:01:01.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f65217fa640 1 -- 172.21.15.31:0/1559338528 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f651c005020 con 0x7f6524109a10 2024-04-25T16:01:01.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f65217fa640 1 -- 172.21.15.31:0/1559338528 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651c0116d0 con 0x7f6524109a10 2024-04-25T16:01:01.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.146+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65241a46e0 con 0x7f6524109a10 2024-04-25T16:01:01.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.147+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6524108820 con 0x7f6524109a10 2024-04-25T16:01:01.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.149+0000 7f65217fa640 1 -- 172.21.15.31:0/1559338528 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f651c0054a0 con 0x7f6524109a10 2024-04-25T16:01:01.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.149+0000 7f65217fa640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f650c0afaf0 0x7f650c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.150+0000 7f6523fff640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f650c0afaf0 0x7f650c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.150+0000 7f65217fa640 1 -- 172.21.15.31:0/1559338528 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f651c0df280 con 0x7f6524109a10 2024-04-25T16:01:01.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.150+0000 7f6523fff640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f650c0afaf0 0x7f650c0b2000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6514005f70 tx=0x7f6514000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:01.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.153+0000 7f65217fa640 1 -- 172.21.15.31:0/1559338528 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f651c016030 con 0x7f6524109a10 2024-04-25T16:01:01.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.383+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f652402db50 con 0x7f6524109a10 2024-04-25T16:01:01.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.383+0000 7f65217fa640 1 -- 172.21.15.31:0/1559338528 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11917 (secure 0 0 0) 0x7f651c0a7e10 con 0x7f6524109a10 2024-04-25T16:01:01.385 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:01.385 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T15:59:10.264996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[14631,24553,24563,24521,24535]},"id":1}]} 2024-04-25T16:01:01.386 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T16:01:01.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.385+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f650c0afaf0 msgr2=0x7f650c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.385+0000 7f652a985640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f650c0afaf0 0x7f650c0b2000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6514005f70 tx=0x7f6514000bc0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.385+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524109a10 msgr2=0x7f6524076540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.385+0000 7f652a985640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524109a10 0x7f6524076540 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f651c00eaa0 tx=0x7f651c00ef60 comp rx=0 tx=0).stop 2024-04-25T16:01:01.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.385+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 shutdown_connections 2024-04-25T16:01:01.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.385+0000 7f652a985640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f650c0afaf0 0x7f650c0b2000 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.385+0000 7f652a985640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6524109a10 0x7f6524076540 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.385+0000 7f652a985640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6524104d10 0x7f6524076000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.385+0000 7f652a985640 1 --2- 172.21.15.31:0/1559338528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6524103ad0 0x7f6524077960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.386+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 >> 172.21.15.31:0/1559338528 conn(0x7f65240fece0 msgr2=0x7f65241059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:01.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.386+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 shutdown_connections 2024-04-25T16:01:01.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.386+0000 7f652a985640 1 -- 172.21.15.31:0/1559338528 wait complete. 2024-04-25T16:01:01.399 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:01:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.514+0000 7fce8cbf3640 1 -- 172.21.15.31:0/2242230718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88109a50 msgr2=0x7fce8810be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.514+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/2242230718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88109a50 0x7fce8810be70 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fce80009ca0 tx=0x7fce8002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.514+0000 7fce8cbf3640 1 -- 172.21.15.31:0/2242230718 shutdown_connections 2024-04-25T16:01:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.514+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/2242230718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88109a50 0x7fce8810be70 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.514+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/2242230718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce88104d50 0x7fce88105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.514+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/2242230718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce88103b10 0x7fce88104810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.514+0000 7fce8cbf3640 1 -- 172.21.15.31:0/2242230718 >> 172.21.15.31:0/2242230718 conn(0x7fce880fed40 msgr2=0x7fce881011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.514+0000 7fce8cbf3640 1 -- 172.21.15.31:0/2242230718 shutdown_connections 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.514+0000 7fce8cbf3640 1 -- 172.21.15.31:0/2242230718 wait complete. 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.515+0000 7fce8cbf3640 1 Processor -- start 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.515+0000 7fce8cbf3640 1 -- start start 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.515+0000 7fce8cbf3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce88103b10 0x7fce8819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.515+0000 7fce8cbf3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88104d50 0x7fce8819a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.515+0000 7fce8cbf3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce88109a50 0x7fce881a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.515+0000 7fce8cbf3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fce8810f640 con 0x7fce88104d50 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.515+0000 7fce8cbf3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fce8810f4c0 con 0x7fce88103b10 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.515+0000 7fce8cbf3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce8810f7c0 con 0x7fce88109a50 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce85d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88104d50 0x7fce8819a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce85d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88104d50 0x7fce8819a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40304/0 (socket says 172.21.15.31:40304) 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce85d9b640 1 -- 172.21.15.31:0/1762554584 learned_addr learned my addr 172.21.15.31:0/1762554584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce8659c640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce88103b10 0x7fce8819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce86d9d640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce88109a50 0x7fce881a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce85d9b640 1 -- 172.21.15.31:0/1762554584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce88109a50 msgr2=0x7fce881a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce85d9b640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce88109a50 0x7fce881a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce85d9b640 1 -- 172.21.15.31:0/1762554584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce88103b10 msgr2=0x7fce8819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce85d9b640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce88103b10 0x7fce8819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce85d9b640 1 -- 172.21.15.31:0/1762554584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce881a1d00 con 0x7fce88104d50 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce8659c640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce88103b10 0x7fce8819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce85d9b640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88104d50 0x7fce8819a700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00e9f0 tx=0x7fce7c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce777fe640 1 -- 172.21.15.31:0/1762554584 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7c00cd80 con 0x7fce88104d50 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce777fe640 1 -- 172.21.15.31:0/1762554584 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce7c0054d0 con 0x7fce88104d50 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce881a1ff0 con 0x7fce88104d50 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce777fe640 1 -- 172.21.15.31:0/1762554584 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7c010690 con 0x7fce88104d50 2024-04-25T16:01:01.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.516+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce881a2460 con 0x7fce88104d50 2024-04-25T16:01:01.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.517+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce88108860 con 0x7fce88104d50 2024-04-25T16:01:01.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.519+0000 7fce777fe640 1 -- 172.21.15.31:0/1762554584 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fce7c004e90 con 0x7fce88104d50 2024-04-25T16:01:01.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.519+0000 7fce777fe640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce700afaf0 0x7fce700b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.519+0000 7fce8659c640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce700afaf0 0x7fce700b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.519+0000 7fce777fe640 1 -- 172.21.15.31:0/1762554584 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fce7c0df010 con 0x7fce88104d50 2024-04-25T16:01:01.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.520+0000 7fce8659c640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce700afaf0 0x7fce700b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fce78005d00 tx=0x7fce78005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:01.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.524+0000 7fce777fe640 1 -- 172.21.15.31:0/1762554584 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce7c0a7c00 con 0x7fce88104d50 2024-04-25T16:01:01.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:01 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/258684718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:01.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:01 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4059070532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:01.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:01 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1559338528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:01.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.713+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce880637a0 con 0x7fce88104d50 2024-04-25T16:01:01.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.715+0000 7fce777fe640 1 -- 172.21.15.31:0/1762554584 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7fce7c0a7500 con 0x7fce88104d50 2024-04-25T16:01:01.717 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:01.717 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:01:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce700afaf0 msgr2=0x7fce700b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce700afaf0 0x7fce700b2000 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fce78005d00 tx=0x7fce78005c90 comp rx=0 tx=0).stop 2024-04-25T16:01:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88104d50 msgr2=0x7fce8819a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88104d50 0x7fce8819a700 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00e9f0 tx=0x7fce7c00edf0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 shutdown_connections 2024-04-25T16:01:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce700afaf0 0x7fce700b2000 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce88109a50 0x7fce881a1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce88104d50 0x7fce8819a700 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 --2- 172.21.15.31:0/1762554584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce88103b10 0x7fce8819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 >> 172.21.15.31:0/1762554584 conn(0x7fce880fed40 msgr2=0x7fce88105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:01.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 shutdown_connections 2024-04-25T16:01:01.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.717+0000 7fce8cbf3640 1 -- 172.21.15.31:0/1762554584 wait complete. 2024-04-25T16:01:01.731 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:59:00.978038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:59:01.829535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:01:01.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:01:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.845+0000 7ff16c98f640 1 -- 172.21.15.31:0/1929130933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 msgr2=0x7ff168102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.845+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1929130933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 0x7ff168102460 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff160009930 tx=0x7ff16002a420 comp rx=0 tx=0).stop 2024-04-25T16:01:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.845+0000 7ff16c98f640 1 -- 172.21.15.31:0/1929130933 shutdown_connections 2024-04-25T16:01:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.845+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1929130933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1681035b0 0x7ff168103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.845+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1929130933 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff168102a50 0x7ff168102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.845+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1929130933 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 0x7ff168102460 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.845+0000 7ff16c98f640 1 -- 172.21.15.31:0/1929130933 >> 172.21.15.31:0/1929130933 conn(0x7ff1680fdc40 msgr2=0x7ff1681000f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.845+0000 7ff16c98f640 1 -- 172.21.15.31:0/1929130933 shutdown_connections 2024-04-25T16:01:01.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.845+0000 7ff16c98f640 1 -- 172.21.15.31:0/1929130933 wait complete. 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16c98f640 1 Processor -- start 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16c98f640 1 -- start start 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16c98f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 0x7ff16819c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16c98f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff168102a50 0x7ff16819c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16c98f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1681035b0 0x7ff1681a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16c98f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff168111830 con 0x7ff168102050 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16c98f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681116b0 con 0x7ff168102a50 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16c98f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1681119b0 con 0x7ff1681035b0 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 0x7ff16819c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 0x7ff16819c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40320/0 (socket says 172.21.15.31:40320) 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff16659c640 1 -- 172.21.15.31:0/1694927143 learned_addr learned my addr 172.21.15.31:0/1694927143 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.846+0000 7ff166d9d640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1681035b0 0x7ff1681a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff16659c640 1 -- 172.21.15.31:0/1694927143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1681035b0 msgr2=0x7ff1681a3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff16659c640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1681035b0 0x7ff1681a3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff16659c640 1 -- 172.21.15.31:0/1694927143 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff168102a50 msgr2=0x7ff16819c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff16659c640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff168102a50 0x7ff16819c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff16659c640 1 -- 172.21.15.31:0/1694927143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1681a3f70 con 0x7ff168102050 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff16659c640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 0x7ff16819c430 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff160031440 tx=0x7ff160003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff1577fe640 1 -- 172.21.15.31:0/1694927143 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff16003f070 con 0x7ff168102050 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff1577fe640 1 -- 172.21.15.31:0/1694927143 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff16003b460 con 0x7ff168102050 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1681a4200 con 0x7ff168102050 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff1577fe640 1 -- 172.21.15.31:0/1694927143 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff160035030 con 0x7ff168102050 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.847+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1681a4740 con 0x7ff168102050 2024-04-25T16:01:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.848+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff16810d4b0 con 0x7ff168102050 2024-04-25T16:01:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.850+0000 7ff1577fe640 1 -- 172.21.15.31:0/1694927143 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff160042070 con 0x7ff168102050 2024-04-25T16:01:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.850+0000 7ff1577fe640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff1500afb70 0x7ff1500b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.850+0000 7ff165d9b640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff1500afb70 0x7ff1500b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:01.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.850+0000 7ff1577fe640 1 -- 172.21.15.31:0/1694927143 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff1600ff920 con 0x7ff168102050 2024-04-25T16:01:01.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.851+0000 7ff165d9b640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff1500afb70 0x7ff1500b2080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff15c005fd0 tx=0x7ff15c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:01.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:01.854+0000 7ff1577fe640 1 -- 172.21.15.31:0/1694927143 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1600c8510 con 0x7ff168102050 2024-04-25T16:01:01.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:01 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/258684718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:01.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:01 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4059070532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:01.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:01 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1559338528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:01 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/258684718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:01 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4059070532' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:01.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:01 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1559338528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:02.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.039+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1680637a0 con 0x7ff168102050 2024-04-25T16:01:02.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.040+0000 7ff1577fe640 1 -- 172.21.15.31:0/1694927143 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7ff1600c7e10 con 0x7ff168102050 2024-04-25T16:01:02.042 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.041+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff1500afb70 msgr2=0x7ff1500b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.041+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff1500afb70 0x7ff1500b2080 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff15c005fd0 tx=0x7ff15c005850 comp rx=0 tx=0).stop 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.041+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 msgr2=0x7ff16819c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.041+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 0x7ff16819c430 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff160031440 tx=0x7ff160003d50 comp rx=0 tx=0).stop 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.041+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 shutdown_connections 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.041+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff1500afb70 0x7ff1500b2080 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.041+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1681035b0 0x7ff1681a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.041+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff168102a50 0x7ff16819c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.042+0000 7ff16c98f640 1 --2- 172.21.15.31:0/1694927143 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff168102050 0x7ff16819c430 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.042+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 >> 172.21.15.31:0/1694927143 conn(0x7ff1680fdc40 msgr2=0x7ff16810a680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.042+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 shutdown_connections 2024-04-25T16:01:02.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.042+0000 7ff16c98f640 1 -- 172.21.15.31:0/1694927143 wait complete. 2024-04-25T16:01:02.055 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.168+0000 7f24d53d6640 1 -- 172.21.15.31:0/3379181201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0102260 msgr2=0x7f24d010bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.168+0000 7f24d53d6640 1 --2- 172.21.15.31:0/3379181201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0102260 0x7f24d010bcc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f24c0009ca0 tx=0x7f24c002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.168+0000 7f24d53d6640 1 -- 172.21.15.31:0/3379181201 shutdown_connections 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.168+0000 7f24d53d6640 1 --2- 172.21.15.31:0/3379181201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0102260 0x7f24d010bcc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.168+0000 7f24d53d6640 1 --2- 172.21.15.31:0/3379181201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d0101760 0x7f24d0101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.168+0000 7f24d53d6640 1 --2- 172.21.15.31:0/3379181201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24d01050b0 0x7f24d01054c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.168+0000 7f24d53d6640 1 -- 172.21.15.31:0/3379181201 >> 172.21.15.31:0/3379181201 conn(0x7f24d00fd380 msgr2=0x7f24d00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.169+0000 7f24d53d6640 1 -- 172.21.15.31:0/3379181201 shutdown_connections 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.169+0000 7f24d53d6640 1 -- 172.21.15.31:0/3379181201 wait complete. 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.169+0000 7f24d53d6640 1 Processor -- start 2024-04-25T16:01:02.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.169+0000 7f24d53d6640 1 -- start start 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24d53d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0101760 0x7f24d019c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24d53d6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d0102260 0x7f24d019c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24d53d6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24d01050b0 0x7f24d01a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24d53d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f24d010f620 con 0x7f24d0101760 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24d53d6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f24d010f4a0 con 0x7f24d0102260 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24d53d6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f24d010f7a0 con 0x7f24d01050b0 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ceffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0101760 0x7f24d019c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ceffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0101760 0x7f24d019c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40330/0 (socket says 172.21.15.31:40330) 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ceffd640 1 -- 172.21.15.31:0/1868715297 learned_addr learned my addr 172.21.15.31:0/1868715297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ce7fc640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d0102260 0x7f24d019c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24cf7fe640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24d01050b0 0x7f24d01a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ceffd640 1 -- 172.21.15.31:0/1868715297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24d01050b0 msgr2=0x7f24d01a3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ceffd640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24d01050b0 0x7f24d01a3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ceffd640 1 -- 172.21.15.31:0/1868715297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d0102260 msgr2=0x7f24d019c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ceffd640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d0102260 0x7f24d019c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ceffd640 1 -- 172.21.15.31:0/1868715297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24d01a3ec0 con 0x7f24d0101760 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24ceffd640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0101760 0x7f24d019c430 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f24c8005fd0 tx=0x7f24c800dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24c7fff640 1 -- 172.21.15.31:0/1868715297 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8017070 con 0x7f24d0101760 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24d01a41b0 con 0x7f24d0101760 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.170+0000 7f24c7fff640 1 -- 172.21.15.31:0/1868715297 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f24c800de50 con 0x7f24d0101760 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.171+0000 7f24c7fff640 1 -- 172.21.15.31:0/1868715297 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c8003c50 con 0x7f24d0101760 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.171+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24d01a4700 con 0x7f24d0101760 2024-04-25T16:01:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.171+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24d01a4960 con 0x7f24d0101760 2024-04-25T16:01:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.173+0000 7f24c7fff640 1 -- 172.21.15.31:0/1868715297 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f24c8003df0 con 0x7f24d0101760 2024-04-25T16:01:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.173+0000 7f24c7fff640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24bc0afaf0 0x7f24bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.174+0000 7f24ce7fc640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24bc0afaf0 0x7f24bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.174+0000 7f24c7fff640 1 -- 172.21.15.31:0/1868715297 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f24c80def90 con 0x7f24d0101760 2024-04-25T16:01:02.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.174+0000 7f24ce7fc640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24bc0afaf0 0x7f24bc0b2000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f24d019d4d0 tx=0x7f24b800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:02.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.177+0000 7f24c7fff640 1 -- 172.21.15.31:0/1868715297 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24c8020030 con 0x7f24d0101760 2024-04-25T16:01:02.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.364+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f24d01a4c90 con 0x7f24d0101760 2024-04-25T16:01:02.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.364+0000 7f24c7fff640 1 -- 172.21.15.31:0/1868715297 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7f24c80a7c20 con 0x7f24d0101760 2024-04-25T16:01:02.366 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24bc0afaf0 msgr2=0x7f24bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24bc0afaf0 0x7f24bc0b2000 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f24d019d4d0 tx=0x7f24b800a2d0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0101760 msgr2=0x7f24d019c430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0101760 0x7f24d019c430 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f24c8005fd0 tx=0x7f24c800dd20 comp rx=0 tx=0).stop 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 shutdown_connections 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24bc0afaf0 0x7f24bc0b2000 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24d01050b0 0x7f24d01a3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24d0102260 0x7f24d019c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 --2- 172.21.15.31:0/1868715297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24d0101760 0x7f24d019c430 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 >> 172.21.15.31:0/1868715297 conn(0x7f24d00fd380 msgr2=0x7f24d01027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:02.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 shutdown_connections 2024-04-25T16:01:02.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.366+0000 7f24d53d6640 1 -- 172.21.15.31:0/1868715297 wait complete. 2024-04-25T16:01:02.380 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:01:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.493+0000 7f4882c15640 1 -- 172.21.15.31:0/2542849785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 msgr2=0x7f487c104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.493+0000 7f4882c15640 1 --2- 172.21.15.31:0/2542849785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 0x7f487c104030 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f486c009900 tx=0x7f486c02a410 comp rx=0 tx=0).stop 2024-04-25T16:01:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.493+0000 7f4882c15640 1 -- 172.21.15.31:0/2542849785 shutdown_connections 2024-04-25T16:01:02.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.493+0000 7f4882c15640 1 --2- 172.21.15.31:0/2542849785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487c108920 0x7f487c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.493+0000 7f4882c15640 1 --2- 172.21.15.31:0/2542849785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 0x7f487c104030 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.493+0000 7f4882c15640 1 --2- 172.21.15.31:0/2542849785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f487c1029e0 0x7f487c1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.493+0000 7f4882c15640 1 -- 172.21.15.31:0/2542849785 >> 172.21.15.31:0/2542849785 conn(0x7f487c0fdbd0 msgr2=0x7f487c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.493+0000 7f4882c15640 1 -- 172.21.15.31:0/2542849785 shutdown_connections 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.493+0000 7f4882c15640 1 -- 172.21.15.31:0/2542849785 wait complete. 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4882c15640 1 Processor -- start 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4882c15640 1 -- start start 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4882c15640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f487c1029e0 0x7f487c19a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4882c15640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 0x7f487c19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4882c15640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487c108920 0x7f487c1a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4882c15640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f487c10f610 con 0x7f487c103c20 2024-04-25T16:01:02.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4882c15640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f487c10f490 con 0x7f487c1029e0 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4882c15640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f487c10f790 con 0x7f487c108920 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4873fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 0x7f487c19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4873fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 0x7f487c19aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40344/0 (socket says 172.21.15.31:40344) 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.494+0000 7f4873fff640 1 -- 172.21.15.31:0/3815729319 learned_addr learned my addr 172.21.15.31:0/3815729319 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f48811b2640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487c108920 0x7f487c1a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f48809b1640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f487c1029e0 0x7f487c19a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4873fff640 1 -- 172.21.15.31:0/3815729319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487c108920 msgr2=0x7f487c1a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4873fff640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487c108920 0x7f487c1a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4873fff640 1 -- 172.21.15.31:0/3815729319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f487c1029e0 msgr2=0x7f487c19a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4873fff640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f487c1029e0 0x7f487c19a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4873fff640 1 -- 172.21.15.31:0/3815729319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f487c1a2080 con 0x7f487c103c20 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4873fff640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 0x7f487c19aa80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f486c006010 tx=0x7f486c004340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4871ffb640 1 -- 172.21.15.31:0/3815729319 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c03f030 con 0x7f487c103c20 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f487c1a2310 con 0x7f487c103c20 2024-04-25T16:01:02.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4871ffb640 1 -- 172.21.15.31:0/3815729319 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f486c003cb0 con 0x7f487c103c20 2024-04-25T16:01:02.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4871ffb640 1 -- 172.21.15.31:0/3815729319 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f486c035070 con 0x7f487c103c20 2024-04-25T16:01:02.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.495+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f487c1a2850 con 0x7f487c103c20 2024-04-25T16:01:02.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.496+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f487c107730 con 0x7f487c103c20 2024-04-25T16:01:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.498+0000 7f4871ffb640 1 -- 172.21.15.31:0/3815729319 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f486c042030 con 0x7f487c103c20 2024-04-25T16:01:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.498+0000 7f4871ffb640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f48640afaf0 0x7f48640b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.499+0000 7f48809b1640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f48640afaf0 0x7f48640b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.499+0000 7f4871ffb640 1 -- 172.21.15.31:0/3815729319 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f486c0ff960 con 0x7f487c103c20 2024-04-25T16:01:02.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.499+0000 7f48809b1640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f48640afaf0 0x7f48640b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4878005fd0 tx=0x7f4878005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:02.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.502+0000 7f4871ffb640 1 -- 172.21.15.31:0/3815729319 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f486c0c84f0 con 0x7f487c103c20 2024-04-25T16:01:02.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.690+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f487c02dcd0 con 0x7f487c103c20 2024-04-25T16:01:02.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.691+0000 7f4871ffb640 1 -- 172.21.15.31:0/3815729319 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7f486c003e60 con 0x7f487c103c20 2024-04-25T16:01:02.693 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f48640afaf0 msgr2=0x7f48640b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f48640afaf0 0x7f48640b2000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f4878005fd0 tx=0x7f4878005850 comp rx=0 tx=0).stop 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 msgr2=0x7f487c19aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 0x7f487c19aa80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f486c006010 tx=0x7f486c004340 comp rx=0 tx=0).stop 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 shutdown_connections 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f48640afaf0 0x7f48640b2000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f487c108920 0x7f487c1a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f487c103c20 0x7f487c19aa80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 --2- 172.21.15.31:0/3815729319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f487c1029e0 0x7f487c19a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 >> 172.21.15.31:0/3815729319 conn(0x7f487c0fdbd0 msgr2=0x7f487c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:02.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 shutdown_connections 2024-04-25T16:01:02.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.693+0000 7f4882c15640 1 -- 172.21.15.31:0/3815729319 wait complete. 2024-04-25T16:01:02.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:02 smithi031 ceph-mon[29140]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:01:02.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:02 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1762554584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:02.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:02 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1694927143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:02.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:02 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1868715297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:02.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:01:02.771 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:02 smithi063 ceph-mon[32413]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:01:02.771 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:02 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1762554584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:02.771 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:02 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1694927143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:02.771 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:02 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1868715297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:02.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:02 smithi186 ceph-mon[33258]: pgmap v343: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:01:02.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:02 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1762554584' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:02.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:02 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1694927143' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:02.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:02 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1868715297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.816+0000 7fb3160c6640 1 -- 172.21.15.31:0/3987377243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310103b00 msgr2=0x7fb310104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.816+0000 7fb3160c6640 1 --2- 172.21.15.31:0/3987377243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310103b00 0x7fb310104800 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fb300009900 tx=0x7fb30002a410 comp rx=0 tx=0).stop 2024-04-25T16:01:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.816+0000 7fb3160c6640 1 -- 172.21.15.31:0/3987377243 shutdown_connections 2024-04-25T16:01:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.816+0000 7fb3160c6640 1 --2- 172.21.15.31:0/3987377243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb310109a40 0x7fb31010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.816+0000 7fb3160c6640 1 --2- 172.21.15.31:0/3987377243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb310104d40 0x7fb310105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.816+0000 7fb3160c6640 1 --2- 172.21.15.31:0/3987377243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310103b00 0x7fb310104800 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.816+0000 7fb3160c6640 1 -- 172.21.15.31:0/3987377243 >> 172.21.15.31:0/3987377243 conn(0x7fb3100fed10 msgr2=0x7fb3101011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.816+0000 7fb3160c6640 1 -- 172.21.15.31:0/3987377243 shutdown_connections 2024-04-25T16:01:02.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.816+0000 7fb3160c6640 1 -- 172.21.15.31:0/3987377243 wait complete. 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb3160c6640 1 Processor -- start 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb3160c6640 1 -- start start 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb3160c6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb310103b00 0x7fb310077960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb3160c6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310104d40 0x7fb310076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb3160c6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb310109a40 0x7fb310076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb3160c6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb31010f5e0 con 0x7fb310104d40 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb3160c6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb31010f460 con 0x7fb310109a40 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb3160c6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb31010f760 con 0x7fb310103b00 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb30effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310104d40 0x7fb310076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb30effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310104d40 0x7fb310076000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40362/0 (socket says 172.21.15.31:40362) 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb30effd640 1 -- 172.21.15.31:0/832658896 learned_addr learned my addr 172.21.15.31:0/832658896 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb30ffff640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb310109a40 0x7fb310076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.817+0000 7fb30f7fe640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb310103b00 0x7fb310077960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb30effd640 1 -- 172.21.15.31:0/832658896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb310103b00 msgr2=0x7fb310077960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb30effd640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb310103b00 0x7fb310077960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb30effd640 1 -- 172.21.15.31:0/832658896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb310109a40 msgr2=0x7fb310076540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb30effd640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb310109a40 0x7fb310076540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb30effd640 1 -- 172.21.15.31:0/832658896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb310076be0 con 0x7fb310104d40 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb30effd640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310104d40 0x7fb310076000 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb30400d950 tx=0x7fb30400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb30cff9640 1 -- 172.21.15.31:0/832658896 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb304017070 con 0x7fb310104d40 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb30cff9640 1 -- 172.21.15.31:0/832658896 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb304009c30 con 0x7fb310104d40 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3101a4300 con 0x7fb310104d40 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb30cff9640 1 -- 172.21.15.31:0/832658896 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb30400f620 con 0x7fb310104d40 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.818+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3101a4710 con 0x7fb310104d40 2024-04-25T16:01:02.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.819+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb310108850 con 0x7fb310104d40 2024-04-25T16:01:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.820+0000 7fb30cff9640 1 -- 172.21.15.31:0/832658896 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb304010460 con 0x7fb310104d40 2024-04-25T16:01:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.821+0000 7fb30cff9640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2f80afaf0 0x7fb2f80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.821+0000 7fb30f7fe640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2f80afaf0 0x7fb2f80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.822+0000 7fb30cff9640 1 -- 172.21.15.31:0/832658896 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb3040de7f0 con 0x7fb310104d40 2024-04-25T16:01:02.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.822+0000 7fb30f7fe640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2f80afaf0 0x7fb2f80b2000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb300009230 tx=0x7fb300031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:02.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:02.825+0000 7fb30cff9640 1 -- 172.21.15.31:0/832658896 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3040a73e0 con 0x7fb310104d40 2024-04-25T16:01:03.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.061+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb31002dbf0 con 0x7fb310104d40 2024-04-25T16:01:03.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.061+0000 7fb30cff9640 1 -- 172.21.15.31:0/832658896 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11917 (secure 0 0 0) 0x7fb304009de0 con 0x7fb310104d40 2024-04-25T16:01:03.063 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:03.063 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T15:59:10.264996+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[14631,24553,24563,24521,24535]},"id":1}]} 2024-04-25T16:01:03.064 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 26 2024-04-25T16:01:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2f80afaf0 msgr2=0x7fb2f80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2f80afaf0 0x7fb2f80b2000 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb300009230 tx=0x7fb300031400 comp rx=0 tx=0).stop 2024-04-25T16:01:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310104d40 msgr2=0x7fb310076000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310104d40 0x7fb310076000 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb30400d950 tx=0x7fb30400de10 comp rx=0 tx=0).stop 2024-04-25T16:01:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 shutdown_connections 2024-04-25T16:01:03.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2f80afaf0 0x7fb2f80b2000 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb310109a40 0x7fb310076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb310104d40 0x7fb310076000 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 --2- 172.21.15.31:0/832658896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb310103b00 0x7fb310077960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 >> 172.21.15.31:0/832658896 conn(0x7fb3100fed10 msgr2=0x7fb310105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 shutdown_connections 2024-04-25T16:01:03.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.064+0000 7fb3160c6640 1 -- 172.21.15.31:0/832658896 wait complete. 2024-04-25T16:01:03.077 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:01:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.188+0000 7f219ca49640 1 -- 172.21.15.31:0/3985529639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2198103c20 msgr2=0x7f2198104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.188+0000 7f219ca49640 1 --2- 172.21.15.31:0/3985529639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2198103c20 0x7f2198104030 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2180009900 tx=0x7f218002a410 comp rx=0 tx=0).stop 2024-04-25T16:01:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.188+0000 7f219ca49640 1 -- 172.21.15.31:0/3985529639 shutdown_connections 2024-04-25T16:01:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.188+0000 7f219ca49640 1 --2- 172.21.15.31:0/3985529639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2198108920 0x7f219810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.188+0000 7f219ca49640 1 --2- 172.21.15.31:0/3985529639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2198103c20 0x7f2198104030 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.188+0000 7f219ca49640 1 --2- 172.21.15.31:0/3985529639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f21981029e0 0x7f21981036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.188+0000 7f219ca49640 1 -- 172.21.15.31:0/3985529639 >> 172.21.15.31:0/3985529639 conn(0x7f21980fdbd0 msgr2=0x7f2198100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.188+0000 7f219ca49640 1 -- 172.21.15.31:0/3985529639 shutdown_connections 2024-04-25T16:01:03.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.188+0000 7f219ca49640 1 -- 172.21.15.31:0/3985529639 wait complete. 2024-04-25T16:01:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219ca49640 1 Processor -- start 2024-04-25T16:01:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219ca49640 1 -- start start 2024-04-25T16:01:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219ca49640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21981029e0 0x7f2198075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219ca49640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2198103c20 0x7f219819f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219ca49640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2198108920 0x7f21981a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219ca49640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f219810e560 con 0x7f21981029e0 2024-04-25T16:01:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219ca49640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f219810e3e0 con 0x7f2198108920 2024-04-25T16:01:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219ca49640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f219810e6e0 con 0x7f2198103c20 2024-04-25T16:01:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21981029e0 0x7f2198075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21981029e0 0x7f2198075860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40382/0 (socket says 172.21.15.31:40382) 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f219659c640 1 -- 172.21.15.31:0/3088932414 learned_addr learned my addr 172.21.15.31:0/3088932414 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.189+0000 7f2196d9d640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2198108920 0x7f21981a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f218ffff640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2198103c20 0x7f219819f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f219659c640 1 -- 172.21.15.31:0/3088932414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2198103c20 msgr2=0x7f219819f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f219659c640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2198103c20 0x7f219819f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f219659c640 1 -- 172.21.15.31:0/3088932414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2198108920 msgr2=0x7f21981a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f219659c640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2198108920 0x7f21981a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f219659c640 1 -- 172.21.15.31:0/3088932414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21981a1db0 con 0x7f21981029e0 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f219659c640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21981029e0 0x7f2198075860 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f218800e9f0 tx=0x7f218800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f218f7fe640 1 -- 172.21.15.31:0/3088932414 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218800ccc0 con 0x7f21981029e0 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f218f7fe640 1 -- 172.21.15.31:0/3088932414 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f21880035f0 con 0x7f21981029e0 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f218f7fe640 1 -- 172.21.15.31:0/3088932414 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f218800ce60 con 0x7f21981029e0 2024-04-25T16:01:03.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21981a20a0 con 0x7f21981029e0 2024-04-25T16:01:03.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.190+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21981a25f0 con 0x7f21981029e0 2024-04-25T16:01:03.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.191+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2198107730 con 0x7f21981029e0 2024-04-25T16:01:03.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.192+0000 7f218f7fe640 1 -- 172.21.15.31:0/3088932414 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2188003aa0 con 0x7f21981029e0 2024-04-25T16:01:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.193+0000 7f218f7fe640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f21840afaf0 0x7f21840b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.193+0000 7f218f7fe640 1 -- 172.21.15.31:0/3088932414 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f21880defc0 con 0x7f21981029e0 2024-04-25T16:01:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.193+0000 7f218ffff640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f21840afaf0 0x7f21840b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.194+0000 7f218ffff640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f21840afaf0 0x7f21840b2000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2180006010 tx=0x7f2180031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.197+0000 7f218f7fe640 1 -- 172.21.15.31:0/3088932414 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f21880a7c50 con 0x7f21981029e0 2024-04-25T16:01:03.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.385+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f219802dda0 con 0x7f21981029e0 2024-04-25T16:01:03.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.387+0000 7f218f7fe640 1 -- 172.21.15.31:0/3088932414 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f21880a7550 con 0x7f21981029e0 2024-04-25T16:01:03.389 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:03.389 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:01:03.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.388+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f21840afaf0 msgr2=0x7f21840b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.388+0000 7f219ca49640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f21840afaf0 0x7f21840b2000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f2180006010 tx=0x7f2180031400 comp rx=0 tx=0).stop 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.388+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21981029e0 msgr2=0x7f2198075860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.388+0000 7f219ca49640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21981029e0 0x7f2198075860 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f218800e9f0 tx=0x7f218800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.389+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 shutdown_connections 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.389+0000 7f219ca49640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f21840afaf0 0x7f21840b2000 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.389+0000 7f219ca49640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2198108920 0x7f21981a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.389+0000 7f219ca49640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2198103c20 0x7f219819f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.389+0000 7f219ca49640 1 --2- 172.21.15.31:0/3088932414 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f21981029e0 0x7f2198075860 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.389+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 >> 172.21.15.31:0/3088932414 conn(0x7f21980fdbd0 msgr2=0x7f2198104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.389+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 shutdown_connections 2024-04-25T16:01:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.389+0000 7f219ca49640 1 -- 172.21.15.31:0/3088932414 wait complete. 2024-04-25T16:01:03.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 26, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-25T15:59:02.905418+0000', 'modified': '2024-04-25T15:59:10.264996+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24563, 'mds_1': 24553, 'mds_2': 24535, 'mds_3': 24521, 'mds_4': 14631}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14613': {'gid': 14613, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.31:6839/3019802236', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6838', 'nonce': 3019802236}, {'type': 'v1', 'addr': '172.21.15.31:6839', 'nonce': 3019802236}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14631': {'gid': 14631, 'name': 'a', 'rank': 4, 'incarnation': 24, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.31:6841/2149984264', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.31:6840', 'nonce': 2149984264}, {'type': 'v1', 'addr': '172.21.15.31:6841', 'nonce': 2149984264}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24455': {'gid': 24455, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6833/927313408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6832', 'nonce': 927313408}, {'type': 'v1', 'addr': '172.21.15.63:6833', 'nonce': 927313408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24460': {'gid': 24460, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6833/1701597397', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6832', 'nonce': 1701597397}, {'type': 'v1', 'addr': '172.21.15.186:6833', 'nonce': 1701597397}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24488': {'gid': 24488, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.63:6835/1584006856', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6834', 'nonce': 1584006856}, {'type': 'v1', 'addr': '172.21.15.63:6835', 'nonce': 1584006856}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.186:6835/1885074975', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 1885074975}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 1885074975}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24521': {'gid': 24521, 'name': 'h', 'rank': 3, 'incarnation': 22, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.63:6837/546178402', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6836', 'nonce': 546178402}, {'type': 'v1', 'addr': '172.21.15.63:6837', 'nonce': 546178402}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 2, 'incarnation': 20, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.186:6837/509868288', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 509868288}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 509868288}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24553': {'gid': 24553, 'name': 'i', 'rank': 1, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.186:6839/2846674462', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 2846674462}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 2846674462}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24563': {'gid': 24563, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.63:6839/495769167', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.63:6838', 'nonce': 495769167}, {'type': 'v1', 'addr': '172.21.15.63:6839', 'nonce': 495769167}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24563, 'qdb_cluster': [14631, 24553, 24563, 24521, 24535]} 2024-04-25T16:01:03.403 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-25T16:01:03.403 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-25T16:01:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.515+0000 7faf7a938640 1 -- 172.21.15.31:0/3325365645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74103af0 msgr2=0x7faf741047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.515+0000 7faf7a938640 1 --2- 172.21.15.31:0/3325365645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74103af0 0x7faf741047f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7faf64009900 tx=0x7faf6402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.515+0000 7faf7a938640 1 -- 172.21.15.31:0/3325365645 shutdown_connections 2024-04-25T16:01:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.515+0000 7faf7a938640 1 --2- 172.21.15.31:0/3325365645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf74109a30 0x7faf7410be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.515+0000 7faf7a938640 1 --2- 172.21.15.31:0/3325365645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf74104d30 0x7faf74105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.515+0000 7faf7a938640 1 --2- 172.21.15.31:0/3325365645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74103af0 0x7faf741047f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.515+0000 7faf7a938640 1 -- 172.21.15.31:0/3325365645 >> 172.21.15.31:0/3325365645 conn(0x7faf740fece0 msgr2=0x7faf74101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.515+0000 7faf7a938640 1 -- 172.21.15.31:0/3325365645 shutdown_connections 2024-04-25T16:01:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.515+0000 7faf7a938640 1 -- 172.21.15.31:0/3325365645 wait complete. 2024-04-25T16:01:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf7a938640 1 Processor -- start 2024-04-25T16:01:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf7a938640 1 -- start start 2024-04-25T16:01:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf7a938640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf74103af0 0x7faf74077960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf7a938640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74104d30 0x7faf74076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf7a938640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf74109a30 0x7faf74076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf7a938640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faf7410f6a0 con 0x7faf74104d30 2024-04-25T16:01:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf7a938640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faf7410f520 con 0x7faf74109a30 2024-04-25T16:01:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf7a938640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faf7410f820 con 0x7faf74103af0 2024-04-25T16:01:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74104d30 0x7faf74076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74104d30 0x7faf74076000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40404/0 (socket says 172.21.15.31:40404) 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.516+0000 7faf737fe640 1 -- 172.21.15.31:0/454292911 learned_addr learned my addr 172.21.15.31:0/454292911 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf78ed5640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf74109a30 0x7faf74076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf73fff640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf74103af0 0x7faf74077960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf737fe640 1 -- 172.21.15.31:0/454292911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf74103af0 msgr2=0x7faf74077960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf737fe640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf74103af0 0x7faf74077960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf737fe640 1 -- 172.21.15.31:0/454292911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf74109a30 msgr2=0x7faf74076540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf737fe640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf74109a30 0x7faf74076540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf737fe640 1 -- 172.21.15.31:0/454292911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf74076c70 con 0x7faf74104d30 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf737fe640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74104d30 0x7faf74076000 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7faf6800e9f0 tx=0x7faf6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf717fa640 1 -- 172.21.15.31:0/454292911 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6800cd80 con 0x7faf74104d30 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf717fa640 1 -- 172.21.15.31:0/454292911 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf680108e0 con 0x7faf74104d30 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf7a938640 1 -- 172.21.15.31:0/454292911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf741a4330 con 0x7faf74104d30 2024-04-25T16:01:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf717fa640 1 -- 172.21.15.31:0/454292911 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf68018980 con 0x7faf74104d30 2024-04-25T16:01:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.517+0000 7faf7a938640 1 -- 172.21.15.31:0/454292911 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf741a4870 con 0x7faf74104d30 2024-04-25T16:01:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf717fa640 1 -- 172.21.15.31:0/454292911 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7faf68010430 con 0x7faf74104d30 2024-04-25T16:01:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf817ef740 1 Processor -- start 2024-04-25T16:01:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf817ef740 1 -- start start 2024-04-25T16:01:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf817ef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d540270 0x55926d8e1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf817ef740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d5173b0 0x55926d8e18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf817ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 0x55926d8e87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf817ef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55926d846740 con 0x55926d540270 2024-04-25T16:01:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf817ef740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55926d7e23a0 con 0x55926d5173b0 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf817ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55926d842700 con 0x55926d51f590 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf78ed5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d540270 0x55926d8e1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf78ed5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d540270 0x55926d8e1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40418/0 (socket says 172.21.15.31:40418) 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf78ed5640 1 -- 172.21.15.31:0/1196483897 learned_addr learned my addr 172.21.15.31:0/1196483897 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.520+0000 7faf73fff640 1 --2- 172.21.15.31:0/1196483897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d5173b0 0x55926d8e18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf737fe640 1 --2- 172.21.15.31:0/1196483897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 0x55926d8e87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf78ed5640 1 -- 172.21.15.31:0/1196483897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 msgr2=0x55926d8e87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf78ed5640 1 --2- 172.21.15.31:0/1196483897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 0x55926d8e87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf78ed5640 1 -- 172.21.15.31:0/1196483897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d5173b0 msgr2=0x55926d8e18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf78ed5640 1 --2- 172.21.15.31:0/1196483897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d5173b0 0x55926d8e18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf78ed5640 1 -- 172.21.15.31:0/1196483897 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55926d7fe750 con 0x55926d540270 2024-04-25T16:01:03.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf717fa640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faf5c0afaf0 0x7faf5c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf73fff640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faf5c0afaf0 0x7faf5c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf78ed5640 1 --2- 172.21.15.31:0/1196483897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d540270 0x55926d8e1370 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00a430 tx=0x7faf6c009650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd45ff0747c00221 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf717fa640 1 -- 172.21.15.31:0/454292911 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faf680deee0 con 0x7faf74104d30 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf5a7fc640 1 -- 172.21.15.31:0/1196483897 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c013790 con 0x55926d540270 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf5a7fc640 1 -- 172.21.15.31:0/1196483897 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf6c019070 con 0x55926d540270 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.521+0000 7faf5a7fc640 1 -- 172.21.15.31:0/1196483897 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf6c014d00 con 0x55926d540270 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 -- 172.21.15.31:0/1196483897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d540270 msgr2=0x55926d8e1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 --2- 172.21.15.31:0/1196483897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d540270 0x55926d8e1370 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7faf6c00a430 tx=0x7faf6c009650 comp rx=0 tx=0).stop 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf73fff640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faf5c0afaf0 0x7faf5c0b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7faf6402aa00 tx=0x7faf640314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 -- 172.21.15.31:0/1196483897 shutdown_connections 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 --2- 172.21.15.31:0/1196483897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 0x55926d8e87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 --2- 172.21.15.31:0/1196483897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d5173b0 0x55926d8e18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 --2- 172.21.15.31:0/1196483897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d540270 0x55926d8e1370 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 -- 172.21.15.31:0/1196483897 >> 172.21.15.31:0/1196483897 conn(0x55926d5343c0 msgr2=0x55926d8dcd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 -- 172.21.15.31:0/1196483897 shutdown_connections 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 -- 172.21.15.31:0/1196483897 wait complete. 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.522+0000 7faf817ef740 1 -- start start 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf817ef740 1 Processor -- start 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf817ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 0x55926d8eb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf817ef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf817ef740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf817ef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55926d8e9240 con 0x55926d51f590 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf817ef740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55926d7cdfe0 con 0x55926d540270 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf817ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55926d8e93c0 con 0x55926d5173b0 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf817ef740 10 client.0 _ll_register_callbacks cb 0x55926d7e4150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:03.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40422/0 (socket says 172.21.15.31:40422) 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf737fe640 1 -- 172.21.15.31:0/4257694084 learned_addr learned my addr 172.21.15.31:0/4257694084 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf78ed5640 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf73fff640 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 0x55926d8eb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf737fe640 1 -- 172.21.15.31:0/4257694084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 msgr2=0x55926d8eb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf737fe640 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 0x55926d8eb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf737fe640 1 -- 172.21.15.31:0/4257694084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 msgr2=0x55926d8f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf737fe640 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf737fe640 1 -- 172.21.15.31:0/4257694084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55926d5582d0 con 0x55926d51f590 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.523+0000 7faf78ed5640 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf737fe640 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb860 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7faf680a3500 tx=0x7faf680a33c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adf48e3bed7c90f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf537fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:03.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf537fe640 1 -- 172.21.15.31:0/4257694084 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf68010720 con 0x55926d51f590 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf537fe640 1 -- 172.21.15.31:0/4257694084 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faf680a66e0 con 0x55926d51f590 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf817ef740 1 -- 172.21.15.31:0/4257694084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf6c019070 con 0x55926d51f590 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf537fe640 1 -- 172.21.15.31:0/4257694084 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faf680a6f90 con 0x55926d51f590 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf817ef740 10 client.15249 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf817ef740 1 -- 172.21.15.31:0/4257694084 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55926d6226b0 con 0x55926d51f590 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf537fe640 1 -- 172.21.15.31:0/4257694084 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf6c019070 con 0x55926d51f590 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf537fe640 1 -- 172.21.15.31:0/4257694084 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faf680aeb00 con 0x55926d51f590 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf817ef740 10 client.15249 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf817ef740 10 client.15249 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf817ef740 20 client.15249 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf817ef740 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55926d4fee40 0x55926d8f8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf817ef740 4 client.15249 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.524+0000 7faf817ef740 1 -- 172.21.15.31:0/4257694084 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55926d854350 con 0x55926d4fee40 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.525+0000 7faf73fff640 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55926d4fee40 0x55926d8f8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.525+0000 7faf537fe640 10 client.15249 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.525+0000 7faf73fff640 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55926d4fee40 0x55926d8f8fe0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=46a50a2b5fea407d server_cookie=42d7b67c34bd7f01 in_seq=0 out_seq=0 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf537fe640 1 -- 172.21.15.31:0/4257694084 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55926d854350 con 0x55926d4fee40 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf537fe640 10 client.15249 handle_command_reply: tid=0 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 client.15249 shutdown 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 -- 172.21.15.31:0/4257694084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 msgr2=0x55926d8eb860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb860 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7faf680a3500 tx=0x7faf680a33c0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 -- 172.21.15.31:0/4257694084 shutdown_connections 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55926d4fee40 0x55926d8f8fe0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb860 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf537fe640 0 client.15249 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:01:03.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 --2- 172.21.15.31:0/4257694084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 0x55926d8eb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 -- 172.21.15.31:0/4257694084 >> 172.21.15.31:0/4257694084 conn(0x55926d5343c0 msgr2=0x55926d8dbd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 -- 172.21.15.31:0/4257694084 shutdown_connections 2024-04-25T16:01:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 1 -- 172.21.15.31:0/4257694084 wait complete. 2024-04-25T16:01:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.526+0000 7faf817ef740 20 client.15249 trim_cache size 0 max 16384 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf817ef740 1 Processor -- start 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf817ef740 1 -- start start 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf817ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 0x55926d8eaef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf817ef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf817ef740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf817ef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55926d8f7320 con 0x55926d51f590 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf817ef740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55926d8f79f0 con 0x55926d540270 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf817ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55926d8f80c0 con 0x55926d5173b0 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf737fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40438/0 (socket says 172.21.15.31:40438) 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf737fe640 1 -- 172.21.15.31:0/223046870 learned_addr learned my addr 172.21.15.31:0/223046870 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf78ed5640 1 --2- 172.21.15.31:0/223046870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf73fff640 1 --2- 172.21.15.31:0/223046870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 0x55926d8eaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf737fe640 1 -- 172.21.15.31:0/223046870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 msgr2=0x55926d8eaef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf737fe640 1 --2- 172.21.15.31:0/223046870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 0x55926d8eaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf737fe640 1 -- 172.21.15.31:0/223046870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 msgr2=0x55926d8f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.545+0000 7faf737fe640 1 --2- 172.21.15.31:0/223046870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.546+0000 7faf737fe640 1 -- 172.21.15.31:0/223046870 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55926d8e61a0 con 0x55926d51f590 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.546+0000 7faf737fe640 1 --2- 172.21.15.31:0/223046870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb430 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7faf680097b0 tx=0x7faf680a2df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8600ca85241610e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.546+0000 7faf59ffb640 1 -- 172.21.15.31:0/223046870 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf68010720 con 0x55926d51f590 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.546+0000 7faf59ffb640 1 -- 172.21.15.31:0/223046870 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf680ab780 con 0x55926d51f590 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.546+0000 7faf59ffb640 1 -- 172.21.15.31:0/223046870 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf680a2f10 con 0x55926d51f590 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 -- 172.21.15.31:0/223046870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 msgr2=0x55926d8eb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 --2- 172.21.15.31:0/223046870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb430 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7faf680097b0 tx=0x7faf680a2df0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 -- 172.21.15.31:0/223046870 shutdown_connections 2024-04-25T16:01:03.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 --2- 172.21.15.31:0/223046870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 --2- 172.21.15.31:0/223046870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d51f590 0x55926d8eb430 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 --2- 172.21.15.31:0/223046870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d5173b0 0x55926d8eaef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 -- 172.21.15.31:0/223046870 >> 172.21.15.31:0/223046870 conn(0x55926d5343c0 msgr2=0x55926d8e9940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 -- 172.21.15.31:0/223046870 shutdown_connections 2024-04-25T16:01:03.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 -- 172.21.15.31:0/223046870 wait complete. 2024-04-25T16:01:03.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 -- start start 2024-04-25T16:01:03.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 Processor -- start 2024-04-25T16:01:03.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d5173b0 0x55926d8f0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.547+0000 7faf817ef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 0x55926d8f0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf817ef740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf817ef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55926d8f8e60 con 0x55926d5173b0 2024-04-25T16:01:03.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf817ef740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55926d8f8790 con 0x55926d540270 2024-04-25T16:01:03.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf817ef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55926d8f9530 con 0x55926d51f590 2024-04-25T16:01:03.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf817ef740 10 client.0 _ll_register_callbacks cb 0x7faf68009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:03.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf73fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d5173b0 0x55926d8f0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf73fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d5173b0 0x55926d8f0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40440/0 (socket says 172.21.15.31:40440) 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf73fff640 1 -- 172.21.15.31:0/892620633 learned_addr learned my addr 172.21.15.31:0/892620633 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf78ed5640 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf737fe640 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 0x55926d8f0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf73fff640 1 -- 172.21.15.31:0/892620633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 msgr2=0x55926d8f0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf73fff640 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 0x55926d8f0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf73fff640 1 -- 172.21.15.31:0/892620633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 msgr2=0x55926d8f19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf73fff640 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f19e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf73fff640 1 -- 172.21.15.31:0/892620633 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55926d8f1fa0 con 0x55926d5173b0 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf737fe640 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 0x55926d8f0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf73fff640 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d5173b0 0x55926d8f0570 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7faf640096c0 tx=0x7faf64009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b42027d885b21409 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf537fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf537fe640 1 -- 172.21.15.31:0/892620633 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf64059060 con 0x55926d5173b0 2024-04-25T16:01:03.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf537fe640 1 -- 172.21.15.31:0/892620633 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faf64003de0 con 0x55926d5173b0 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf817ef740 1 -- 172.21.15.31:0/892620633 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faf680ab780 con 0x55926d5173b0 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf537fe640 1 -- 172.21.15.31:0/892620633 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7faf64035040 con 0x55926d5173b0 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf817ef740 10 client.15255 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.548+0000 7faf817ef740 1 -- 172.21.15.31:0/892620633 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55926d8f32f0 con 0x55926d5173b0 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf537fe640 1 -- 172.21.15.31:0/892620633 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf680ab780 con 0x55926d5173b0 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf537fe640 1 -- 172.21.15.31:0/892620633 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7faf64044040 con 0x55926d5173b0 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf817ef740 10 client.15255 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf817ef740 10 client.15255 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf817ef740 20 client.15255 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf817ef740 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55926d4d4280 0x55926d965550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf817ef740 4 client.15255 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf817ef740 1 -- 172.21.15.31:0/892620633 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "status"}) -- 0x55926d84e1c0 con 0x55926d4d4280 2024-04-25T16:01:03.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf737fe640 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55926d4d4280 0x55926d965550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf537fe640 10 client.15255 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.549+0000 7faf737fe640 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55926d4d4280 0x55926d965550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=518b4daf5b069d79 server_cookie=3e0a0c5513381427 in_seq=0 out_seq=0 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf537fe640 1 -- 172.21.15.31:0/892620633 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55926d84e1c0 con 0x55926d4d4280 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf537fe640 10 client.15255 handle_command_reply: tid=0 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 client.15255 shutdown 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 -- 172.21.15.31:0/892620633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d5173b0 msgr2=0x55926d8f0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d5173b0 0x55926d8f0570 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7faf640096c0 tx=0x7faf64009260 comp rx=0 tx=0).stop 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 -- 172.21.15.31:0/892620633 shutdown_connections 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55926d4d4280 0x55926d965550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55926d540270 0x55926d8f19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55926d51f590 0x55926d8f0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf537fe640 0 client.15255 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 --2- 172.21.15.31:0/892620633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55926d5173b0 0x55926d8f0570 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 -- 172.21.15.31:0/892620633 >> 172.21.15.31:0/892620633 conn(0x55926d5343c0 msgr2=0x55926d8eaef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 -- 172.21.15.31:0/892620633 shutdown_connections 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 1 -- 172.21.15.31:0/892620633 wait complete. 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.550+0000 7faf817ef740 20 client.15255 trim_cache size 0 max 16384 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 4, 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 14631, 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:01:03.559 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:01:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 114.30543310500001, 2024-04-25T16:01:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:01:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:01:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:01:03.560 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 128.604768353 2024-04-25T16:01:03.560 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 -- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faf5c0afaf0 msgr2=0x7faf5c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faf5c0afaf0 0x7faf5c0b2000 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7faf6402aa00 tx=0x7faf640314a0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 -- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74104d30 msgr2=0x7faf74076000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74104d30 0x7faf74076000 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7faf6800e9f0 tx=0x7faf6800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 -- 172.21.15.31:0/454292911 shutdown_connections 2024-04-25T16:01:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faf5c0afaf0 0x7faf5c0b2000 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faf74109a30 0x7faf74076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faf74104d30 0x7faf74076000 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 --2- 172.21.15.31:0/454292911 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf74103af0 0x7faf74077960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 -- 172.21.15.31:0/454292911 >> 172.21.15.31:0/454292911 conn(0x7faf740fece0 msgr2=0x7faf74105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 -- 172.21.15.31:0/454292911 shutdown_connections 2024-04-25T16:01:03.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.551+0000 7faf7a938640 1 -- 172.21.15.31:0/454292911 wait complete. 2024-04-25T16:01:03.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-25T16:01:03.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.674+0000 7fe3118ee640 1 -- 172.21.15.31:0/3079329053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c102250 msgr2=0x7fe30c10bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.674+0000 7fe3118ee640 1 --2- 172.21.15.31:0/3079329053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c102250 0x7fe30c10bcb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe304009ca0 tx=0x7fe30402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.674+0000 7fe3118ee640 1 -- 172.21.15.31:0/3079329053 shutdown_connections 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.674+0000 7fe3118ee640 1 --2- 172.21.15.31:0/3079329053 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c102250 0x7fe30c10bcb0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.674+0000 7fe3118ee640 1 --2- 172.21.15.31:0/3079329053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c101750 0x7fe30c101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.674+0000 7fe3118ee640 1 --2- 172.21.15.31:0/3079329053 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe30c1050a0 0x7fe30c1054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.674+0000 7fe3118ee640 1 -- 172.21.15.31:0/3079329053 >> 172.21.15.31:0/3079329053 conn(0x7fe30c0fd350 msgr2=0x7fe30c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.674+0000 7fe3118ee640 1 -- 172.21.15.31:0/3079329053 shutdown_connections 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.674+0000 7fe3118ee640 1 -- 172.21.15.31:0/3079329053 wait complete. 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe3118ee640 1 Processor -- start 2024-04-25T16:01:03.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe3118ee640 1 -- start start 2024-04-25T16:01:03.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe3118ee640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe30c101750 0x7fe30c19a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe3118ee640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c102250 0x7fe30c19a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe3118ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c1050a0 0x7fe30c1a1660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe3118ee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c10f610 con 0x7fe30c1050a0 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe3118ee640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c10f490 con 0x7fe30c101750 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe3118ee640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe30c10f790 con 0x7fe30c102250 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe30b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c1050a0 0x7fe30c1a1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe30b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c1050a0 0x7fe30c1a1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55910/0 (socket says 172.21.15.31:55910) 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe30b7fe640 1 -- 172.21.15.31:0/2034427856 learned_addr learned my addr 172.21.15.31:0/2034427856 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.675+0000 7fe30affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe30c101750 0x7fe30c19a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c102250 0x7fe30c19a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe30b7fe640 1 -- 172.21.15.31:0/2034427856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c102250 msgr2=0x7fe30c19a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe30b7fe640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c102250 0x7fe30c19a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe30b7fe640 1 -- 172.21.15.31:0/2034427856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe30c101750 msgr2=0x7fe30c19a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe30b7fe640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe30c101750 0x7fe30c19a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe30b7fe640 1 -- 172.21.15.31:0/2034427856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe30c1a1cd0 con 0x7fe30c1050a0 2024-04-25T16:01:03.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe30affd640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe30c101750 0x7fe30c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe30b7fe640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c1050a0 0x7fe30c1a1660 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe3040097b0 tx=0x7fe304031840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe3108ec640 1 -- 172.21.15.31:0/2034427856 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe304003bf0 con 0x7fe30c1050a0 2024-04-25T16:01:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe3108ec640 1 -- 172.21.15.31:0/2034427856 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe304003d90 con 0x7fe30c1050a0 2024-04-25T16:01:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe3118ee640 1 -- 172.21.15.31:0/2034427856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe30c1a1f60 con 0x7fe30c1050a0 2024-04-25T16:01:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe3108ec640 1 -- 172.21.15.31:0/2034427856 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30402fc30 con 0x7fe30c1050a0 2024-04-25T16:01:03.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.676+0000 7fe3118ee640 1 -- 172.21.15.31:0/2034427856 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe30c1a23c0 con 0x7fe30c1050a0 2024-04-25T16:01:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe31879e740 1 Processor -- start 2024-04-25T16:01:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe3108ec640 1 -- 172.21.15.31:0/2034427856 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe304004040 con 0x7fe30c1050a0 2024-04-25T16:01:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe31879e740 1 -- start start 2024-04-25T16:01:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe31879e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 0x5607ec02b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe31879e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec02b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe31879e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc69590 0x5607ec0327f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe31879e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5607ebf90740 con 0x5607ebc613b0 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe31879e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5607ebf2c3a0 con 0x5607ebc8a270 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe31879e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5607ebf8c700 con 0x5607ebc69590 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec02b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec02b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55912/0 (socket says 172.21.15.31:55912) 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30affd640 1 -- 172.21.15.31:0/813963507 learned_addr learned my addr 172.21.15.31:0/813963507 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 0x5607ec02b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc69590 0x5607ec0327f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30affd640 1 -- 172.21.15.31:0/813963507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc69590 msgr2=0x5607ec0327f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30affd640 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc69590 0x5607ec0327f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30affd640 1 -- 172.21.15.31:0/813963507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 msgr2=0x5607ec02b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30affd640 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 0x5607ec02b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30affd640 1 -- 172.21.15.31:0/813963507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607ebf48750 con 0x5607ebc613b0 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.679+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 0x5607ec02b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.680+0000 7fe3108ec640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2f40afaf0 0x7fe2f40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.680+0000 7fe30affd640 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec02b8f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009fd0 tx=0x7fe2fc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bca9c355681382f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.680+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2f40afaf0 0x7fe2f40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.680+0000 7fe3108ec640 1 -- 172.21.15.31:0/2034427856 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe304100340 con 0x7fe30c1050a0 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.680+0000 7fe2f1ffb640 1 -- 172.21.15.31:0/813963507 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc004310 con 0x5607ebc613b0 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.680+0000 7fe2f1ffb640 1 -- 172.21.15.31:0/813963507 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2fc0044b0 con 0x5607ebc613b0 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.680+0000 7fe2f1ffb640 1 -- 172.21.15.31:0/813963507 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc0035f0 con 0x5607ebc613b0 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2f40afaf0 0x7fe2f40b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x5607ec02c540 tx=0x7fe30000c470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 -- 172.21.15.31:0/813963507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 msgr2=0x5607ec02b8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec02b8f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009fd0 tx=0x7fe2fc0118d0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 -- 172.21.15.31:0/813963507 shutdown_connections 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc69590 0x5607ec0327f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec02b8f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 --2- 172.21.15.31:0/813963507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 0x5607ec02b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 -- 172.21.15.31:0/813963507 >> 172.21.15.31:0/813963507 conn(0x5607ebc7e3c0 msgr2=0x5607ec026d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 -- 172.21.15.31:0/813963507 shutdown_connections 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 -- 172.21.15.31:0/813963507 wait complete. 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 -- start start 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.681+0000 7fe31879e740 1 Processor -- start 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe31879e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc613b0 0x5607ec035200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe31879e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec035740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe31879e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 0x5607ec03c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe31879e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5607ec033240 con 0x5607ebc69590 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe31879e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5607ebf17fe0 con 0x5607ebc8a270 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe31879e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5607ec0333c0 con 0x5607ebc613b0 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe31879e740 10 client.0 _ll_register_callbacks cb 0x5607ebf2e150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:03.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec035740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec035740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55928/0 (socket says 172.21.15.31:55928) 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30a7fc640 1 -- 172.21.15.31:0/919342595 learned_addr learned my addr 172.21.15.31:0/919342595 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30b7fe640 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 0x5607ec03c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30affd640 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc613b0 0x5607ec035200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30a7fc640 1 -- 172.21.15.31:0/919342595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc613b0 msgr2=0x5607ec035200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc613b0 0x5607ec035200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30a7fc640 1 -- 172.21.15.31:0/919342595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 msgr2=0x5607ec03c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 0x5607ec03c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30a7fc640 1 -- 172.21.15.31:0/919342595 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607ebca22d0 con 0x5607ebc69590 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec035740 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe30000c8e0 tx=0x7fe300004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d063e215d994029b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe2eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe2eaffd640 1 -- 172.21.15.31:0/919342595 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30000fd10 con 0x5607ebc69590 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe2eaffd640 1 -- 172.21.15.31:0/919342595 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe300004030 con 0x5607ebc69590 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.682+0000 7fe31879e740 1 -- 172.21.15.31:0/919342595 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2fc0044b0 con 0x5607ebc69590 2024-04-25T16:01:03.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe2eaffd640 1 -- 172.21.15.31:0/919342595 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe300004230 con 0x5607ebc69590 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe31879e740 10 client.15267 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe31879e740 1 -- 172.21.15.31:0/919342595 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5607ebd6c6b0 con 0x5607ebc69590 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe2eaffd640 1 -- 172.21.15.31:0/919342595 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe2fc0044b0 con 0x5607ebc69590 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe2eaffd640 1 -- 172.21.15.31:0/919342595 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe30001d030 con 0x5607ebc69590 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe31879e740 10 client.15267 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe31879e740 10 client.15267 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe31879e740 20 client.15267 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe31879e740 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5607ebc48e40 0x5607ec042fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe31879e740 4 client.15267 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe31879e740 1 -- 172.21.15.31:0/919342595 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5607ebf9e350 con 0x5607ebc48e40 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.683+0000 7fe30affd640 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5607ebc48e40 0x5607ec042fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.684+0000 7fe2eaffd640 10 client.15267 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.684+0000 7fe30affd640 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5607ebc48e40 0x5607ec042fc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bbbf5423ad5f8c2e server_cookie=36c494326ca96993 in_seq=0 out_seq=0 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.684+0000 7fe2eaffd640 1 -- 172.21.15.31:0/919342595 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5607ebf9e350 con 0x5607ebc48e40 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.684+0000 7fe2eaffd640 10 client.15267 handle_command_reply: tid=0 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 client.15267 shutdown 2024-04-25T16:01:03.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 -- 172.21.15.31:0/919342595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 msgr2=0x5607ec035740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec035740 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe30000c8e0 tx=0x7fe300004730 comp rx=0 tx=0).stop 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 -- 172.21.15.31:0/919342595 shutdown_connections 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc8a270 0x5607ec03c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec035740 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5607ebc48e40 0x5607ec042fc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 --2- 172.21.15.31:0/919342595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc613b0 0x5607ec035200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe2eaffd640 0 client.15267 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 -- 172.21.15.31:0/919342595 >> 172.21.15.31:0/919342595 conn(0x5607ebc7e3c0 msgr2=0x5607ec025d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 -- 172.21.15.31:0/919342595 shutdown_connections 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 1 -- 172.21.15.31:0/919342595 wait complete. 2024-04-25T16:01:03.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.685+0000 7fe31879e740 20 client.15267 trim_cache size 0 max 16384 2024-04-25T16:01:03.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:03 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3815729319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:03.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:03 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/832658896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:03.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:03 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3088932414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe31879e740 1 Processor -- start 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe31879e740 1 -- start start 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe31879e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc613b0 0x5607ec021a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe31879e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec03c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe31879e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc8a270 0x5607ec03cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe31879e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5607ec041300 con 0x5607ebc69590 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe31879e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5607ec0419d0 con 0x5607ebc613b0 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe31879e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5607ec0420a0 con 0x5607ebc8a270 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec03c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec03c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55944/0 (socket says 172.21.15.31:55944) 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30a7fc640 1 -- 172.21.15.31:0/1889192383 learned_addr learned my addr 172.21.15.31:0/1889192383 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30affd640 1 --2- 172.21.15.31:0/1889192383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc613b0 0x5607ec021a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30a7fc640 1 -- 172.21.15.31:0/1889192383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc8a270 msgr2=0x5607ec03cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/1889192383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc8a270 0x5607ec03cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30a7fc640 1 -- 172.21.15.31:0/1889192383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc613b0 msgr2=0x5607ec021a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/1889192383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc613b0 0x5607ec021a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30a7fc640 1 -- 172.21.15.31:0/1889192383 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5607ec02af20 con 0x5607ebc69590 2024-04-25T16:01:03.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.704+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/1889192383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec03c5c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe300006010 tx=0x7fe30000f850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad6a567f8b26205 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe2f17fa640 1 -- 172.21.15.31:0/1889192383 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe300029530 con 0x5607ebc69590 2024-04-25T16:01:03.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe2f17fa640 1 -- 172.21.15.31:0/1889192383 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe300025070 con 0x5607ebc69590 2024-04-25T16:01:03.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe2f17fa640 1 -- 172.21.15.31:0/1889192383 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3000035f0 con 0x5607ebc69590 2024-04-25T16:01:03.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe31879e740 1 -- 172.21.15.31:0/1889192383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 msgr2=0x5607ec03c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe31879e740 1 --2- 172.21.15.31:0/1889192383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec03c5c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fe300006010 tx=0x7fe30000f850 comp rx=0 tx=0).stop 2024-04-25T16:01:03.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe31879e740 1 -- 172.21.15.31:0/1889192383 shutdown_connections 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe31879e740 1 --2- 172.21.15.31:0/1889192383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc8a270 0x5607ec03cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe31879e740 1 --2- 172.21.15.31:0/1889192383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc69590 0x5607ec03c5c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe31879e740 1 --2- 172.21.15.31:0/1889192383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc613b0 0x5607ec021a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.705+0000 7fe31879e740 1 -- 172.21.15.31:0/1889192383 >> 172.21.15.31:0/1889192383 conn(0x5607ebc7e3c0 msgr2=0x5607ec02f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 -- 172.21.15.31:0/1889192383 shutdown_connections 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 -- 172.21.15.31:0/1889192383 wait complete. 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 -- start start 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 Processor -- start 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec038770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc69590 0x5607ec038cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc8a270 0x5607ec03bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5607ec042e40 con 0x5607ebc613b0 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5607ec042770 con 0x5607ebc69590 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5607ec043510 con 0x5607ebc8a270 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe31879e740 10 client.0 _ll_register_callbacks cb 0x7fe30000c830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:03.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.706+0000 7fe30affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec038770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec038770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55958/0 (socket says 172.21.15.31:55958) 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30affd640 1 -- 172.21.15.31:0/3675387522 learned_addr learned my addr 172.21.15.31:0/3675387522 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc69590 0x5607ec038cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30b7fe640 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc8a270 0x5607ec03bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30affd640 1 -- 172.21.15.31:0/3675387522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc8a270 msgr2=0x5607ec03bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30affd640 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc8a270 0x5607ec03bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30affd640 1 -- 172.21.15.31:0/3675387522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc69590 msgr2=0x5607ec038cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30affd640 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc69590 0x5607ec038cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30affd640 1 -- 172.21.15.31:0/3675387522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5607ec03c160 con 0x5607ebc613b0 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe30affd640 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec038770 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009730 tx=0x7fe2fc0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff021aa4ca41fa68 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe2eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe2eaffd640 1 -- 172.21.15.31:0/3675387522 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc004490 con 0x5607ebc613b0 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe2eaffd640 1 -- 172.21.15.31:0/3675387522 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe2fc02a220 con 0x5607ebc613b0 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe31879e740 1 -- 172.21.15.31:0/3675387522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe300025070 con 0x5607ebc613b0 2024-04-25T16:01:03.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe2eaffd640 1 -- 172.21.15.31:0/3675387522 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe2fc02a460 con 0x5607ebc613b0 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe31879e740 10 client.15273 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.707+0000 7fe31879e740 1 -- 172.21.15.31:0/3675387522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5607ec0abea0 con 0x5607ebc613b0 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe2eaffd640 1 -- 172.21.15.31:0/3675387522 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe300025070 con 0x5607ebc613b0 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe2eaffd640 1 -- 172.21.15.31:0/3675387522 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fe2fc02ace0 con 0x5607ebc613b0 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe31879e740 10 client.15273 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe31879e740 10 client.15273 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe31879e740 20 client.15273 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe31879e740 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5607ebc1e280 0x5607ec0ad380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe31879e740 4 client.15273 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe31879e740 1 -- 172.21.15.31:0/3675387522 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "status"}) -- 0x5607ebf981c0 con 0x5607ebc1e280 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5607ebc1e280 0x5607ec0ad380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe2eaffd640 10 client.15273 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.708+0000 7fe30a7fc640 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5607ebc1e280 0x5607ec0ad380 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c17a2a83f114eb75 server_cookie=5ca6b42f96660b83 in_seq=0 out_seq=0 2024-04-25T16:01:03.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe2eaffd640 1 -- 172.21.15.31:0/3675387522 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x5607ebf981c0 con 0x5607ebc1e280 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe2eaffd640 10 client.15273 handle_command_reply: tid=0 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 client.15273 shutdown 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 -- 172.21.15.31:0/3675387522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 msgr2=0x5607ec038770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec038770 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc009730 tx=0x7fe2fc0096c0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 -- 172.21.15.31:0/3675387522 shutdown_connections 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5607ebc1e280 0x5607ec0ad380 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5607ebc8a270 0x5607ec03bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5607ebc69590 0x5607ec038cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe2eaffd640 0 client.15273 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 --2- 172.21.15.31:0/3675387522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5607ebc613b0 0x5607ec038770 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 -- 172.21.15.31:0/3675387522 >> 172.21.15.31:0/3675387522 conn(0x5607ebc7e3c0 msgr2=0x5607ec0337c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 -- 172.21.15.31:0/3675387522 shutdown_connections 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 1 -- 172.21.15.31:0/3675387522 wait complete. 2024-04-25T16:01:03.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe31879e740 20 client.15273 trim_cache size 0 max 16384 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 3, 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24521, 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 115.15980073, 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:01:03.717 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 137.38219213900001 2024-04-25T16:01:03.718 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe3118ee640 1 -- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2f40afaf0 msgr2=0x7fe2f40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe3118ee640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2f40afaf0 0x7fe2f40b2000 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x5607ec02c540 tx=0x7fe30000c470 comp rx=0 tx=0).stop 2024-04-25T16:01:03.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe3118ee640 1 -- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c1050a0 msgr2=0x7fe30c1a1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.709+0000 7fe3118ee640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c1050a0 0x7fe30c1a1660 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe3040097b0 tx=0x7fe304031840 comp rx=0 tx=0).stop 2024-04-25T16:01:03.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.710+0000 7fe3118ee640 1 -- 172.21.15.31:0/2034427856 shutdown_connections 2024-04-25T16:01:03.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.710+0000 7fe3118ee640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2f40afaf0 0x7fe2f40b2000 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.710+0000 7fe3118ee640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe30c1050a0 0x7fe30c1a1660 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.710+0000 7fe3118ee640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c102250 0x7fe30c19a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.710+0000 7fe3118ee640 1 --2- 172.21.15.31:0/2034427856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe30c101750 0x7fe30c19a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.710+0000 7fe3118ee640 1 -- 172.21.15.31:0/2034427856 >> 172.21.15.31:0/2034427856 conn(0x7fe30c0fd350 msgr2=0x7fe30c107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.710+0000 7fe3118ee640 1 -- 172.21.15.31:0/2034427856 shutdown_connections 2024-04-25T16:01:03.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.710+0000 7fe3118ee640 1 -- 172.21.15.31:0/2034427856 wait complete. 2024-04-25T16:01:03.723 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.833+0000 7f5da86d8640 1 -- 172.21.15.31:0/898477176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5da0109a40 msgr2=0x7f5da010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.833+0000 7f5da86d8640 1 --2- 172.21.15.31:0/898477176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5da0109a40 0x7f5da010be60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009be0 tx=0x7f5d9002a6c0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 -- 172.21.15.31:0/898477176 shutdown_connections 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 --2- 172.21.15.31:0/898477176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5da0109a40 0x7f5da010be60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 --2- 172.21.15.31:0/898477176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5da0104d40 0x7f5da0105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 --2- 172.21.15.31:0/898477176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5da0103b00 0x7f5da0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 -- 172.21.15.31:0/898477176 >> 172.21.15.31:0/898477176 conn(0x7f5da00fed10 msgr2=0x7f5da01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 -- 172.21.15.31:0/898477176 shutdown_connections 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 -- 172.21.15.31:0/898477176 wait complete. 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 Processor -- start 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 -- start start 2024-04-25T16:01:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.834+0000 7f5da86d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5da0103b00 0x7f5da0077960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da86d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5da0104d40 0x7f5da0076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da86d8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5da0109a40 0x7f5da0076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da86d8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5da010f630 con 0x7f5da0104d40 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da86d8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5da010f4b0 con 0x7f5da0103b00 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da86d8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5da010f7b0 con 0x7f5da0109a40 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da5c73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5da0104d40 0x7f5da0076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da5c73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5da0104d40 0x7f5da0076000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55974/0 (socket says 172.21.15.31:55974) 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da5c73640 1 -- 172.21.15.31:0/723928849 learned_addr learned my addr 172.21.15.31:0/723928849 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da6474640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5da0103b00 0x7f5da0077960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da6c75640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5da0109a40 0x7f5da0076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da5c73640 1 -- 172.21.15.31:0/723928849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5da0109a40 msgr2=0x7f5da0076540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da5c73640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5da0109a40 0x7f5da0076540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da5c73640 1 -- 172.21.15.31:0/723928849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5da0103b00 msgr2=0x7f5da0077960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da5c73640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5da0103b00 0x7f5da0077960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da5c73640 1 -- 172.21.15.31:0/723928849 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5da0076c70 con 0x7f5da0104d40 2024-04-25T16:01:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da6c75640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5da0109a40 0x7f5da0076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da5c73640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5da0104d40 0x7f5da0076000 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400e9f0 tx=0x7f5d9400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5d9f7fe640 1 -- 172.21.15.31:0/723928849 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9400cd80 con 0x7f5da0104d40 2024-04-25T16:01:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da86d8640 1 -- 172.21.15.31:0/723928849 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5da01a42f0 con 0x7f5da0104d40 2024-04-25T16:01:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5d9f7fe640 1 -- 172.21.15.31:0/723928849 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d940054d0 con 0x7f5da0104d40 2024-04-25T16:01:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5d9f7fe640 1 -- 172.21.15.31:0/723928849 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d94010690 con 0x7f5da0104d40 2024-04-25T16:01:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.835+0000 7f5da86d8640 1 -- 172.21.15.31:0/723928849 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5da01a4760 con 0x7f5da0104d40 2024-04-25T16:01:03.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5daf586740 1 Processor -- start 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5d9f7fe640 1 -- 172.21.15.31:0/723928849 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5d94004e90 con 0x7f5da0104d40 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5daf586740 1 -- start start 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5daf586740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab81370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5daf586740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab818f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5daf586740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 0x561d2ab887f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5daf586740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d2aae6740 con 0x561d2a7e0270 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5daf586740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561d2aa823a0 con 0x561d2a7b73b0 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5daf586740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561d2aae2700 con 0x561d2a7bf590 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5da6c75640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab81370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5da6474640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab818f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5da6c75640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab81370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55982/0 (socket says 172.21.15.31:55982) 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5da6474640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab818f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52074/0 (socket says 172.21.15.31:52074) 2024-04-25T16:01:03.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5da6c75640 1 -- 172.21.15.31:0/3227319138 learned_addr learned my addr 172.21.15.31:0/3227319138 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.838+0000 7f5da5c73640 1 --2- 172.21.15.31:0/3227319138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 0x561d2ab887f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5da6474640 1 -- 172.21.15.31:0/3227319138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 msgr2=0x561d2ab887f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5da6474640 1 --2- 172.21.15.31:0/3227319138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 0x561d2ab887f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5da6474640 1 -- 172.21.15.31:0/3227319138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 msgr2=0x561d2ab81370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5da6474640 1 --2- 172.21.15.31:0/3227319138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab81370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5da6474640 1 -- 172.21.15.31:0/3227319138 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d2aa9e750 con 0x561d2a7b73b0 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5da5c73640 1 --2- 172.21.15.31:0/3227319138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 0x561d2ab887f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5d9f7fe640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5d880afaf0 0x7f5d880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5da6c75640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5d880afaf0 0x7f5d880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5da6474640 1 --2- 172.21.15.31:0/3227319138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab818f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5da0104700 tx=0x7f5d98011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25a42f75e5c0476 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5d9f7fe640 1 -- 172.21.15.31:0/723928849 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5d940df0d0 con 0x7f5da0104d40 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5d9cff9640 1 -- 172.21.15.31:0/3227319138 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9801e070 con 0x561d2a7b73b0 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5d9cff9640 1 -- 172.21.15.31:0/3227319138 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d980198e0 con 0x561d2a7b73b0 2024-04-25T16:01:03.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.839+0000 7f5d9cff9640 1 -- 172.21.15.31:0/3227319138 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9800e9c0 con 0x561d2a7b73b0 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 -- 172.21.15.31:0/3227319138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 msgr2=0x561d2ab818f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 --2- 172.21.15.31:0/3227319138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab818f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5da0104700 tx=0x7f5d98011810 comp rx=0 tx=0).stop 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5da6c75640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5d880afaf0 0x7f5d880b2000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5d90005cc0 tx=0x7f5d90005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 -- 172.21.15.31:0/3227319138 shutdown_connections 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 --2- 172.21.15.31:0/3227319138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 0x561d2ab887f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 --2- 172.21.15.31:0/3227319138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab818f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 --2- 172.21.15.31:0/3227319138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab81370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 -- 172.21.15.31:0/3227319138 >> 172.21.15.31:0/3227319138 conn(0x561d2a7d43c0 msgr2=0x561d2ab7cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 -- 172.21.15.31:0/3227319138 shutdown_connections 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 -- 172.21.15.31:0/3227319138 wait complete. 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.840+0000 7f5daf586740 1 -- start start 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5daf586740 1 Processor -- start 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5daf586740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7b73b0 0x561d2ab8b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5daf586740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7bf590 0x561d2ab8b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5daf586740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7e0270 0x561d2ab92720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5daf586740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d2ab89240 con 0x561d2a7b73b0 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5daf586740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561d2aa6dfe0 con 0x561d2a7bf590 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5daf586740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561d2ab893c0 con 0x561d2a7e0270 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5daf586740 10 client.0 _ll_register_callbacks cb 0x561d2aa84150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6474640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7b73b0 0x561d2ab8b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6474640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7b73b0 0x561d2ab8b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55994/0 (socket says 172.21.15.31:55994) 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6474640 1 -- 172.21.15.31:0/1720953908 learned_addr learned my addr 172.21.15.31:0/1720953908 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da5c73640 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7bf590 0x561d2ab8b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6c75640 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7e0270 0x561d2ab92720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6474640 1 -- 172.21.15.31:0/1720953908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7e0270 msgr2=0x561d2ab92720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6474640 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7e0270 0x561d2ab92720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6474640 1 -- 172.21.15.31:0/1720953908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7bf590 msgr2=0x561d2ab8b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6474640 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7bf590 0x561d2ab8b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6474640 1 -- 172.21.15.31:0/1720953908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d2a7f82d0 con 0x561d2a7b73b0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5da6474640 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7b73b0 0x561d2ab8b2f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5d98000c00 tx=0x7f5d980098f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17dbf341f30f7134 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.841+0000 7f5d85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5d85ffb640 1 -- 172.21.15.31:0/1720953908 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9801e050 con 0x561d2a7b73b0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5d85ffb640 1 -- 172.21.15.31:0/1720953908 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5d98005040 con 0x561d2a7b73b0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5daf586740 1 -- 172.21.15.31:0/1720953908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d980198e0 con 0x561d2a7b73b0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5d85ffb640 1 -- 172.21.15.31:0/1720953908 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5d98005240 con 0x561d2a7b73b0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5daf586740 10 client.15285 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5daf586740 1 -- 172.21.15.31:0/1720953908 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561d2a8c26b0 con 0x561d2a7b73b0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5d85ffb640 1 -- 172.21.15.31:0/1720953908 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d980198e0 con 0x561d2a7b73b0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5d85ffb640 1 -- 172.21.15.31:0/1720953908 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5d980047e0 con 0x561d2a7b73b0 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5daf586740 10 client.15285 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5daf586740 10 client.15285 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5daf586740 20 client.15285 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5daf586740 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561d2a79ee40 0x561d2ab98d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5daf586740 4 client.15285 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.842+0000 7f5daf586740 1 -- 172.21.15.31:0/1720953908 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d2aaf4350 con 0x561d2a79ee40 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.843+0000 7f5da5c73640 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561d2a79ee40 0x561d2ab98d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.843+0000 7f5d85ffb640 10 client.15285 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.843+0000 7f5da5c73640 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561d2a79ee40 0x561d2ab98d40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e65d3d0c4d54960 server_cookie=fdde2b8612823eec in_seq=0 out_seq=0 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5d85ffb640 1 -- 172.21.15.31:0/1720953908 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561d2aaf4350 con 0x561d2a79ee40 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5d85ffb640 10 client.15285 handle_command_reply: tid=0 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 client.15285 shutdown 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 -- 172.21.15.31:0/1720953908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7b73b0 msgr2=0x561d2ab8b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7b73b0 0x561d2ab8b2f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5d98000c00 tx=0x7f5d980098f0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 -- 172.21.15.31:0/1720953908 shutdown_connections 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7e0270 0x561d2ab92720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561d2a79ee40 0x561d2ab98d40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7bf590 0x561d2ab8b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 --2- 172.21.15.31:0/1720953908 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7b73b0 0x561d2ab8b2f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5d85ffb640 0 client.15285 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:01:03.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 -- 172.21.15.31:0/1720953908 >> 172.21.15.31:0/1720953908 conn(0x561d2a7d43c0 msgr2=0x561d2ab7bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 -- 172.21.15.31:0/1720953908 shutdown_connections 2024-04-25T16:01:03.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 1 -- 172.21.15.31:0/1720953908 wait complete. 2024-04-25T16:01:03.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.844+0000 7f5daf586740 20 client.15285 trim_cache size 0 max 16384 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5daf586740 1 Processor -- start 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5daf586740 1 -- start start 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5daf586740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab968e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5daf586740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 0x561d2ab89910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5daf586740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab89e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5daf586740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d2ab97080 con 0x561d2a7e0270 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5daf586740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561d2ab97750 con 0x561d2a7b73b0 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5daf586740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561d2ab97e20 con 0x561d2a7bf590 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6c75640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab89e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6c75640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab89e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56002/0 (socket says 172.21.15.31:56002) 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6c75640 1 -- 172.21.15.31:0/3920194962 learned_addr learned my addr 172.21.15.31:0/3920194962 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6474640 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab968e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da5c73640 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 0x561d2ab89910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6c75640 1 -- 172.21.15.31:0/3920194962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 msgr2=0x561d2ab89910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6c75640 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 0x561d2ab89910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6c75640 1 -- 172.21.15.31:0/3920194962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 msgr2=0x561d2ab968e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6c75640 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab968e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6c75640 1 -- 172.21.15.31:0/3920194962 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d2ab86110 con 0x561d2a7e0270 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.863+0000 7f5da6474640 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.864+0000 7f5da6c75640 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab89e50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5d90030840 tx=0x7f5d9003e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c4d8049374256f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.864+0000 7f5d87fff640 1 -- 172.21.15.31:0/3920194962 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d90036050 con 0x561d2a7e0270 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.864+0000 7f5d87fff640 1 -- 172.21.15.31:0/3920194962 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d90051070 con 0x561d2a7e0270 2024-04-25T16:01:03.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.864+0000 7f5d87fff640 1 -- 172.21.15.31:0/3920194962 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9004ca10 con 0x561d2a7e0270 2024-04-25T16:01:03.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.864+0000 7f5daf586740 1 -- 172.21.15.31:0/3920194962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 msgr2=0x561d2ab89e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.864+0000 7f5daf586740 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab89e50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5d90030840 tx=0x7f5d9003e000 comp rx=0 tx=0).stop 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 -- 172.21.15.31:0/3920194962 shutdown_connections 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7e0270 0x561d2ab89e50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7bf590 0x561d2ab89910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 --2- 172.21.15.31:0/3920194962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7b73b0 0x561d2ab968e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 -- 172.21.15.31:0/3920194962 >> 172.21.15.31:0/3920194962 conn(0x561d2a7d43c0 msgr2=0x561d2ab77a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 -- 172.21.15.31:0/3920194962 shutdown_connections 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 -- 172.21.15.31:0/3920194962 wait complete. 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 -- start start 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 Processor -- start 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7b73b0 0x561d2ab8d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7bf590 0x561d2ab8d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7e0270 0x561d2ab90770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d2ab98bc0 con 0x561d2a7bf590 2024-04-25T16:01:03.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561d2ab984f0 con 0x561d2a7e0270 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561d2ab99290 con 0x561d2a7b73b0 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5daf586740 10 client.0 _ll_register_callbacks cb 0x7f5d9801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5da5c73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7bf590 0x561d2ab8d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5da5c73640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7bf590 0x561d2ab8d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56008/0 (socket says 172.21.15.31:56008) 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5da5c73640 1 -- 172.21.15.31:0/2212627859 learned_addr learned my addr 172.21.15.31:0/2212627859 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5da6c75640 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7e0270 0x561d2ab90770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.865+0000 7f5da6474640 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7b73b0 0x561d2ab8d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5da5c73640 1 -- 172.21.15.31:0/2212627859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7b73b0 msgr2=0x561d2ab8d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5da5c73640 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7b73b0 0x561d2ab8d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5da5c73640 1 -- 172.21.15.31:0/2212627859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7e0270 msgr2=0x561d2ab90770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5da5c73640 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7e0270 0x561d2ab90770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5da5c73640 1 -- 172.21.15.31:0/2212627859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d2ab90d30 con 0x561d2a7bf590 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5da6c75640 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7e0270 0x561d2ab90770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5da5c73640 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7bf590 0x561d2ab8d880 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5d940dcbe0 tx=0x7f5d94004a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c15ec1e962adee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5d85ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5d85ffb640 1 -- 172.21.15.31:0/2212627859 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d940a7cc0 con 0x561d2a7bf590 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5d85ffb640 1 -- 172.21.15.31:0/2212627859 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5d940900b0 con 0x561d2a7bf590 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5daf586740 1 -- 172.21.15.31:0/2212627859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d90051070 con 0x561d2a7bf590 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5d85ffb640 1 -- 172.21.15.31:0/2212627859 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5d940a75c0 con 0x561d2a7bf590 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5daf586740 10 client.15291 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5daf586740 1 -- 172.21.15.31:0/2212627859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561d2ab92080 con 0x561d2a7bf590 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.866+0000 7f5d85ffb640 1 -- 172.21.15.31:0/2212627859 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d90051070 con 0x561d2a7bf590 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5d85ffb640 1 -- 172.21.15.31:0/2212627859 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5d940b0020 con 0x561d2a7bf590 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5daf586740 10 client.15291 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5daf586740 10 client.15291 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5daf586740 20 client.15291 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5daf586740 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561d2a774280 0x561d2ac05230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5daf586740 4 client.15291 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:01:03.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5daf586740 1 -- 172.21.15.31:0/2212627859 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "status"}) -- 0x561d2aaee1c0 con 0x561d2a774280 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5da6474640 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561d2a774280 0x561d2ac05230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5da6474640 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561d2a774280 0x561d2ac05230 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ef8e7bb0287e98bc server_cookie=977be4c4afe356e0 in_seq=0 out_seq=0 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.867+0000 7f5d85ffb640 10 client.15291 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5d85ffb640 1 -- 172.21.15.31:0/2212627859 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x561d2aaee1c0 con 0x561d2a774280 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5d85ffb640 10 client.15291 handle_command_reply: tid=0 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 client.15291 shutdown 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 -- 172.21.15.31:0/2212627859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7bf590 msgr2=0x561d2ab8d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7bf590 0x561d2ab8d880 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5d940dcbe0 tx=0x7f5d94004a90 comp rx=0 tx=0).stop 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 -- 172.21.15.31:0/2212627859 shutdown_connections 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561d2a774280 0x561d2ac05230 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d2a7e0270 0x561d2ab90770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d2a7bf590 0x561d2ab8d880 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 --2- 172.21.15.31:0/2212627859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d2a7b73b0 0x561d2ab8d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5d85ffb640 0 client.15291 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 -- 172.21.15.31:0/2212627859 >> 172.21.15.31:0/2212627859 conn(0x561d2a7d43c0 msgr2=0x561d2ab89850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 -- 172.21.15.31:0/2212627859 shutdown_connections 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 1 -- 172.21.15.31:0/2212627859 wait complete. 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5daf586740 20 client.15291 trim_cache size 0 max 16384 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 2, 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24535, 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:01:03.876 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:01:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 116.32722993, 2024-04-25T16:01:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:01:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:01:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:01:03.877 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 140.68807687099999 2024-04-25T16:01:03.877 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 -- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5d880afaf0 msgr2=0x7f5d880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5d880afaf0 0x7f5d880b2000 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5d90005cc0 tx=0x7f5d90005bf0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 -- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5da0104d40 msgr2=0x7f5da0076000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5da0104d40 0x7f5da0076000 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5d9400e9f0 tx=0x7f5d9400edf0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 -- 172.21.15.31:0/723928849 shutdown_connections 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5d880afaf0 0x7f5d880b2000 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5da0109a40 0x7f5da0076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5da0104d40 0x7f5da0076000 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 --2- 172.21.15.31:0/723928849 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5da0103b00 0x7f5da0077960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.868+0000 7f5da86d8640 1 -- 172.21.15.31:0/723928849 >> 172.21.15.31:0/723928849 conn(0x7f5da00fed10 msgr2=0x7f5da0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.869+0000 7f5da86d8640 1 -- 172.21.15.31:0/723928849 shutdown_connections 2024-04-25T16:01:03.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.869+0000 7f5da86d8640 1 -- 172.21.15.31:0/723928849 wait complete. 2024-04-25T16:01:03.879 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:03 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3815729319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:03.879 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:03 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/832658896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:03.879 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:03 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3088932414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:03.882 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-25T16:01:03.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:03 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3815729319' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:03.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:03 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/832658896' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:03.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:03 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3088932414' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.992+0000 7f5878c67640 1 -- 172.21.15.31:0/1642315057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 msgr2=0x7f587410be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.992+0000 7f5878c67640 1 --2- 172.21.15.31:0/1642315057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 0x7f587410be50 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f586c009ca0 tx=0x7f586c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.992+0000 7f5878c67640 1 -- 172.21.15.31:0/1642315057 shutdown_connections 2024-04-25T16:01:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.992+0000 7f5878c67640 1 --2- 172.21.15.31:0/1642315057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 0x7f587410be50 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.992+0000 7f5878c67640 1 --2- 172.21.15.31:0/1642315057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874104d30 0x7f5874105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.992+0000 7f5878c67640 1 --2- 172.21.15.31:0/1642315057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5874103af0 0x7f58741047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.992+0000 7f5878c67640 1 -- 172.21.15.31:0/1642315057 >> 172.21.15.31:0/1642315057 conn(0x7f58740fece0 msgr2=0x7f5874101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:03.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.992+0000 7f5878c67640 1 -- 172.21.15.31:0/1642315057 shutdown_connections 2024-04-25T16:01:03.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.992+0000 7f5878c67640 1 -- 172.21.15.31:0/1642315057 wait complete. 2024-04-25T16:01:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5878c67640 1 Processor -- start 2024-04-25T16:01:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5878c67640 1 -- start start 2024-04-25T16:01:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5878c67640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5874103af0 0x7f587419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5878c67640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874104d30 0x7f587419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5878c67640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 0x7f58741a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5878c67640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f587410f620 con 0x7f5874109a30 2024-04-25T16:01:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5878c67640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f587410f4a0 con 0x7f5874104d30 2024-04-25T16:01:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5878c67640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f587410f7a0 con 0x7f5874103af0 2024-04-25T16:01:03.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5872d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 0x7f58741a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5872d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 0x7f58741a16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56020/0 (socket says 172.21.15.31:56020) 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5872d9d640 1 -- 172.21.15.31:0/599209604 learned_addr learned my addr 172.21.15.31:0/599209604 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f5869d9b640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874104d30 0x7f587419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.993+0000 7f587259c640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5874103af0 0x7f587419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f5872d9d640 1 -- 172.21.15.31:0/599209604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5874103af0 msgr2=0x7f587419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f5872d9d640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5874103af0 0x7f587419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f5872d9d640 1 -- 172.21.15.31:0/599209604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874104d30 msgr2=0x7f587419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f5872d9d640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874104d30 0x7f587419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f5872d9d640 1 -- 172.21.15.31:0/599209604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58741a1d30 con 0x7f5874109a30 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f5869d9b640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874104d30 0x7f587419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f5872d9d640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 0x7f58741a16c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f586c004af0 tx=0x7f586c031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f586bfff640 1 -- 172.21.15.31:0/599209604 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c003b70 con 0x7f5874109a30 2024-04-25T16:01:03.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f586bfff640 1 -- 172.21.15.31:0/599209604 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f586c003d10 con 0x7f5874109a30 2024-04-25T16:01:03.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f5878c67640 1 -- 172.21.15.31:0/599209604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58741a1fc0 con 0x7f5874109a30 2024-04-25T16:01:03.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f586bfff640 1 -- 172.21.15.31:0/599209604 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f586c02fb20 con 0x7f5874109a30 2024-04-25T16:01:03.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.994+0000 7f5878c67640 1 -- 172.21.15.31:0/599209604 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58741a2420 con 0x7f5874109a30 2024-04-25T16:01:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f586bfff640 1 -- 172.21.15.31:0/599209604 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f586c004580 con 0x7f5874109a30 2024-04-25T16:01:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f587fbfa740 1 Processor -- start 2024-04-25T16:01:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f587fbfa740 1 -- start start 2024-04-25T16:01:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e840683370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 0x55e8406838f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f587fbfa740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e8405e8740 con 0x55e8402c1590 2024-04-25T16:01:04.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f587fbfa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e8405843a0 con 0x55e8402e2270 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f587fbfa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e8405e4700 con 0x55e8402b93b0 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f586bfff640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f58600afaf0 0x7f58600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56028/0 (socket says 172.21.15.31:56028) 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.997+0000 7f5869d9b640 1 -- 172.21.15.31:0/1517163325 learned_addr learned my addr 172.21.15.31:0/1517163325 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5869d9b640 1 --2- 172.21.15.31:0/1517163325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e840683370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f587259c640 1 --2- 172.21.15.31:0/1517163325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 0x55e8406838f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5872d9d640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f58600afaf0 0x7f58600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f586bfff640 1 -- 172.21.15.31:0/599209604 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f586c1001d0 con 0x7f5874109a30 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5869d9b640 1 -- 172.21.15.31:0/1517163325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 msgr2=0x55e8406838f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5869d9b640 1 --2- 172.21.15.31:0/1517163325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 0x55e8406838f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5869d9b640 1 -- 172.21.15.31:0/1517163325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 msgr2=0x55e84068a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5869d9b640 1 --2- 172.21.15.31:0/1517163325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5869d9b640 1 -- 172.21.15.31:0/1517163325 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e8405a0750 con 0x55e8402e2270 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5869d9b640 1 --2- 172.21.15.31:0/1517163325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e840683370 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f585c011c10 tx=0x7f585c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b50578102007ab7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5872d9d640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f58600afaf0 0x7f58600b2000 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f586c0c58a0 tx=0x7f586c08eb50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5868d99640 1 -- 172.21.15.31:0/1517163325 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c011dc0 con 0x55e8402e2270 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5868d99640 1 -- 172.21.15.31:0/1517163325 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f585c0118a0 con 0x55e8402e2270 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.998+0000 7f5868d99640 1 -- 172.21.15.31:0/1517163325 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c003a00 con 0x55e8402e2270 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.999+0000 7f587fbfa740 1 -- 172.21.15.31:0/1517163325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 msgr2=0x55e840683370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.999+0000 7f587fbfa740 1 --2- 172.21.15.31:0/1517163325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e840683370 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f585c011c10 tx=0x7f585c005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.999+0000 7f587fbfa740 1 -- 172.21.15.31:0/1517163325 shutdown_connections 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.999+0000 7f587fbfa740 1 --2- 172.21.15.31:0/1517163325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.999+0000 7f587fbfa740 1 --2- 172.21.15.31:0/1517163325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 0x55e8406838f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.999+0000 7f587fbfa740 1 --2- 172.21.15.31:0/1517163325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e840683370 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.999+0000 7f587fbfa740 1 -- 172.21.15.31:0/1517163325 >> 172.21.15.31:0/1517163325 conn(0x55e8402d63c0 msgr2=0x55e84067ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.999+0000 7f587fbfa740 1 -- 172.21.15.31:0/1517163325 shutdown_connections 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:03.999+0000 7f587fbfa740 1 -- 172.21.15.31:0/1517163325 wait complete. 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587fbfa740 1 -- start start 2024-04-25T16:01:04.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587fbfa740 1 Processor -- start 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 0x55e84068d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e8406947a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587fbfa740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e84068b240 con 0x55e8402c1590 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587fbfa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e84056ffe0 con 0x55e8402e2270 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587fbfa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e84068b3c0 con 0x55e8402b93b0 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587fbfa740 10 client.0 _ll_register_callbacks cb 0x55e840586150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56036/0 (socket says 172.21.15.31:56036) 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587259c640 1 -- 172.21.15.31:0/3216720626 learned_addr learned my addr 172.21.15.31:0/3216720626 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f5869d9b640 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e8406947a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f5869d9b640 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 0x55e84068d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587259c640 1 -- 172.21.15.31:0/3216720626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 msgr2=0x55e84068d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587259c640 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 0x55e84068d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587259c640 1 -- 172.21.15.31:0/3216720626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 msgr2=0x55e8406947a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587259c640 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e8406947a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f587259c640 1 -- 172.21.15.31:0/3216720626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8402fa2d0 con 0x55e8402c1590 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.000+0000 7f5869d9b640 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 0x55e84068d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587259c640 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068d840 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f58640126e0 tx=0x7f5864012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c998708dc7ac50ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f5859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f5859ffb640 1 -- 172.21.15.31:0/3216720626 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5864004050 con 0x55e8402c1590 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f5859ffb640 1 -- 172.21.15.31:0/3216720626 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f58640041f0 con 0x55e8402c1590 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587fbfa740 1 -- 172.21.15.31:0/3216720626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f585c0118a0 con 0x55e8402c1590 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f5859ffb640 1 -- 172.21.15.31:0/3216720626 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f58640043f0 con 0x55e8402c1590 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587fbfa740 10 client.15303 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587fbfa740 1 -- 172.21.15.31:0/3216720626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e8403c46b0 con 0x55e8402c1590 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f5859ffb640 1 -- 172.21.15.31:0/3216720626 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f585c0118a0 con 0x55e8402c1590 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f5859ffb640 1 -- 172.21.15.31:0/3216720626 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f5864003bc0 con 0x55e8402c1590 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587fbfa740 10 client.15303 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587fbfa740 10 client.15303 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:01:04.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587fbfa740 20 client.15303 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55e8402a0e40 0x55e84069b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587fbfa740 4 client.15303 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.001+0000 7f587fbfa740 1 -- 172.21.15.31:0/3216720626 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e8405f6350 con 0x55e8402a0e40 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.002+0000 7f5869d9b640 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55e8402a0e40 0x55e84069b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.002+0000 7f5869d9b640 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55e8402a0e40 0x55e84069b100 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3ca1bb8cc9ba1f40 server_cookie=5379e542818ea8f2 in_seq=0 out_seq=0 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.002+0000 7f5859ffb640 10 client.15303 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f5859ffb640 1 -- 172.21.15.31:0/3216720626 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e8405f6350 con 0x55e8402a0e40 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f5859ffb640 10 client.15303 handle_command_reply: tid=0 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 client.15303 shutdown 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 -- 172.21.15.31:0/3216720626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 msgr2=0x55e84068d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068d840 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f58640126e0 tx=0x7f5864012ba0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 -- 172.21.15.31:0/3216720626 shutdown_connections 2024-04-25T16:01:04.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e8406947a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55e8402a0e40 0x55e84069b100 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402c1590 0x55e84068d840 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3216720626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402b93b0 0x55e84068d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 -- 172.21.15.31:0/3216720626 >> 172.21.15.31:0/3216720626 conn(0x55e8402d63c0 msgr2=0x55e84067dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f5859ffb640 0 client.15303 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:01:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 -- 172.21.15.31:0/3216720626 shutdown_connections 2024-04-25T16:01:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 1 -- 172.21.15.31:0/3216720626 wait complete. 2024-04-25T16:01:04.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.003+0000 7f587fbfa740 20 client.15303 trim_cache size 0 max 16384 2024-04-25T16:01:04.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f587fbfa740 1 Processor -- start 2024-04-25T16:01:04.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f587fbfa740 1 -- start start 2024-04-25T16:01:04.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84067e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402c1590 0x55e8406946c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e84067ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f587fbfa740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e840699440 con 0x55e8402b93b0 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f587fbfa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e840699b10 con 0x55e8402e2270 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f587fbfa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e84069a1e0 con 0x55e8402c1590 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84067e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84067e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56048/0 (socket says 172.21.15.31:56048) 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.022+0000 7f5869d9b640 1 -- 172.21.15.31:0/3717083145 learned_addr learned my addr 172.21.15.31:0/3717083145 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.023+0000 7f5869d9b640 1 -- 172.21.15.31:0/3717083145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402c1590 msgr2=0x55e8406946c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.023+0000 7f5869d9b640 1 --2- 172.21.15.31:0/3717083145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402c1590 0x55e8406946c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.023+0000 7f5869d9b640 1 -- 172.21.15.31:0/3717083145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 msgr2=0x55e84067ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.023+0000 7f5869d9b640 1 --2- 172.21.15.31:0/3717083145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e84067ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.023+0000 7f5869d9b640 1 -- 172.21.15.31:0/3717083145 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e840688120 con 0x55e8402b93b0 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.023+0000 7f5869d9b640 1 --2- 172.21.15.31:0/3717083145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84067e6b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f585c011660 tx=0x7f585c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92ac1329738e287d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.023+0000 7f585bfff640 1 -- 172.21.15.31:0/3717083145 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c0117f0 con 0x55e8402b93b0 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.023+0000 7f585bfff640 1 -- 172.21.15.31:0/3717083145 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f585c02f3b0 con 0x55e8402b93b0 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.023+0000 7f585bfff640 1 -- 172.21.15.31:0/3717083145 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c0035f0 con 0x55e8402b93b0 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 -- 172.21.15.31:0/3717083145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 msgr2=0x55e84067e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3717083145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84067e6b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f585c011660 tx=0x7f585c005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 -- 172.21.15.31:0/3717083145 shutdown_connections 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3717083145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402e2270 0x55e84067ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3717083145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402c1590 0x55e8406946c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 --2- 172.21.15.31:0/3717083145 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84067e6b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 -- 172.21.15.31:0/3717083145 >> 172.21.15.31:0/3717083145 conn(0x55e8402d63c0 msgr2=0x55e84068b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 -- 172.21.15.31:0/3717083145 shutdown_connections 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 -- 172.21.15.31:0/3717083145 wait complete. 2024-04-25T16:01:04.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 -- start start 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 Processor -- start 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84068f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402c1590 0x55e84068fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402e2270 0x55e840692d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e84069af80 con 0x55e8402b93b0 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e84069a8b0 con 0x55e8402c1590 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e84069b650 con 0x55e8402e2270 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587fbfa740 10 client.0 _ll_register_callbacks cb 0x7f5864009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84068f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f5869d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84068f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56054/0 (socket says 172.21.15.31:56054) 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f5869d9b640 1 -- 172.21.15.31:0/818258115 learned_addr learned my addr 172.21.15.31:0/818258115 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.024+0000 7f587259c640 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402c1590 0x55e84068fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5869d9b640 1 -- 172.21.15.31:0/818258115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402e2270 msgr2=0x55e840692d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5869d9b640 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402e2270 0x55e840692d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5869d9b640 1 -- 172.21.15.31:0/818258115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402c1590 msgr2=0x55e84068fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5869d9b640 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402c1590 0x55e84068fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5869d9b640 1 -- 172.21.15.31:0/818258115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e8406932f0 con 0x55e8402b93b0 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5869d9b640 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84068f8b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f585c00a010 tx=0x7f585c02fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3aabceeee46bb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5859ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5859ffb640 1 -- 172.21.15.31:0/818258115 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c005110 con 0x55e8402b93b0 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5859ffb640 1 -- 172.21.15.31:0/818258115 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f585c0052b0 con 0x55e8402b93b0 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f587fbfa740 1 -- 172.21.15.31:0/818258115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f585c02f3b0 con 0x55e8402b93b0 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5859ffb640 1 -- 172.21.15.31:0/818258115 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f585c02f560 con 0x55e8402b93b0 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f587fbfa740 10 client.15309 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f587fbfa740 1 -- 172.21.15.31:0/818258115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e840694570 con 0x55e8402b93b0 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.025+0000 7f5859ffb640 1 -- 172.21.15.31:0/818258115 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f585c02f3b0 con 0x55e8402b93b0 2024-04-25T16:01:04.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f5859ffb640 1 -- 172.21.15.31:0/818258115 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f585c01d020 con 0x55e8402b93b0 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f587fbfa740 10 client.15309 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f587fbfa740 10 client.15309 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f587fbfa740 20 client.15309 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f587fbfa740 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55e840276280 0x55e840706bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f587fbfa740 4 client.15309 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f587fbfa740 1 -- 172.21.15.31:0/818258115 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "status"}) -- 0x55e8405f01c0 con 0x55e840276280 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f587259c640 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55e840276280 0x55e840706bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f5859ffb640 10 client.15309 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.026+0000 7f587259c640 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55e840276280 0x55e840706bd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=541c7df1e4b09ad4 server_cookie=fb6811fbed4cfadc in_seq=0 out_seq=0 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f5859ffb640 1 -- 172.21.15.31:0/818258115 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55e8405f01c0 con 0x55e840276280 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f5859ffb640 10 client.15309 handle_command_reply: tid=0 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 client.15309 shutdown 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 -- 172.21.15.31:0/818258115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 msgr2=0x55e84068f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84068f8b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f585c00a010 tx=0x7f585c02fc70 comp rx=0 tx=0).stop 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 -- 172.21.15.31:0/818258115 shutdown_connections 2024-04-25T16:01:04.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e8402e2270 0x55e840692d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e8402c1590 0x55e84068fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55e840276280 0x55e840706bd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 --2- 172.21.15.31:0/818258115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e8402b93b0 0x55e84068f8b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f5859ffb640 0 client.15309 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 -- 172.21.15.31:0/818258115 >> 172.21.15.31:0/818258115 conn(0x55e8402d63c0 msgr2=0x55e84067e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 -- 172.21.15.31:0/818258115 shutdown_connections 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 1 -- 172.21.15.31:0/818258115 wait complete. 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f587fbfa740 20 client.15309 trim_cache size 0 max 16384 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:01:04.035 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 1, 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24553, 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 117.502501865, 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 134.928946271 2024-04-25T16:01:04.036 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f5878c67640 1 -- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f58600afaf0 msgr2=0x7f58600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f5878c67640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f58600afaf0 0x7f58600b2000 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f586c0c58a0 tx=0x7f586c08eb50 comp rx=0 tx=0).stop 2024-04-25T16:01:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f5878c67640 1 -- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 msgr2=0x7f58741a16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.027+0000 7f5878c67640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 0x7f58741a16c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f586c004af0 tx=0x7f586c031700 comp rx=0 tx=0).stop 2024-04-25T16:01:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.028+0000 7f5878c67640 1 -- 172.21.15.31:0/599209604 shutdown_connections 2024-04-25T16:01:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.028+0000 7f5878c67640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f58600afaf0 0x7f58600b2000 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.028+0000 7f5878c67640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5874109a30 0x7f58741a16c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.028+0000 7f5878c67640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5874104d30 0x7f587419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.028+0000 7f5878c67640 1 --2- 172.21.15.31:0/599209604 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5874103af0 0x7f587419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.028+0000 7f5878c67640 1 -- 172.21.15.31:0/599209604 >> 172.21.15.31:0/599209604 conn(0x7f58740fece0 msgr2=0x7f5874105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.028+0000 7f5878c67640 1 -- 172.21.15.31:0/599209604 shutdown_connections 2024-04-25T16:01:04.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.028+0000 7f5878c67640 1 -- 172.21.15.31:0/599209604 wait complete. 2024-04-25T16:01:04.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-25T16:01:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.152+0000 7f25fcee9640 1 -- 172.21.15.31:0/2846871146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f8103c30 msgr2=0x7f25f8104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.152+0000 7f25fcee9640 1 --2- 172.21.15.31:0/2846871146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f8103c30 0x7f25f8104040 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f25ec009900 tx=0x7f25ec02a410 comp rx=0 tx=0).stop 2024-04-25T16:01:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.152+0000 7f25fcee9640 1 -- 172.21.15.31:0/2846871146 shutdown_connections 2024-04-25T16:01:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.152+0000 7f25fcee9640 1 --2- 172.21.15.31:0/2846871146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f8108930 0x7f25f810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.152+0000 7f25fcee9640 1 --2- 172.21.15.31:0/2846871146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f8103c30 0x7f25f8104040 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.152+0000 7f25fcee9640 1 --2- 172.21.15.31:0/2846871146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25f81029f0 0x7f25f81036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.152+0000 7f25fcee9640 1 -- 172.21.15.31:0/2846871146 >> 172.21.15.31:0/2846871146 conn(0x7f25f80fdbc0 msgr2=0x7f25f8100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.152+0000 7f25fcee9640 1 -- 172.21.15.31:0/2846871146 shutdown_connections 2024-04-25T16:01:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.152+0000 7f25fcee9640 1 -- 172.21.15.31:0/2846871146 wait complete. 2024-04-25T16:01:04.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25fcee9640 1 Processor -- start 2024-04-25T16:01:04.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25fcee9640 1 -- start start 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25fcee9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f81029f0 0x7f25f819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25fcee9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25f8103c30 0x7f25f819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25fcee9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f8108930 0x7f25f81a16e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25fcee9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f25f810f610 con 0x7f25f81029f0 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25fcee9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f25f810f490 con 0x7f25f8103c30 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25fcee9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f25f810f790 con 0x7f25f8108930 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f81029f0 0x7f25f819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f81029f0 0x7f25f819a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56074/0 (socket says 172.21.15.31:56074) 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f659c640 1 -- 172.21.15.31:0/401565802 learned_addr learned my addr 172.21.15.31:0/401565802 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25f8103c30 0x7f25f819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f6d9d640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f8108930 0x7f25f81a16e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f659c640 1 -- 172.21.15.31:0/401565802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f8108930 msgr2=0x7f25f81a16e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f659c640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f8108930 0x7f25f81a16e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f659c640 1 -- 172.21.15.31:0/401565802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25f8103c30 msgr2=0x7f25f819a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f659c640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25f8103c30 0x7f25f819a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f659c640 1 -- 172.21.15.31:0/401565802 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f81a1d50 con 0x7f25f81029f0 2024-04-25T16:01:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f6d9d640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f8108930 0x7f25f81a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25f659c640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f81029f0 0x7f25f819a210 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f25e800e9f0 tx=0x7f25e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.153+0000 7f25e77fe640 1 -- 172.21.15.31:0/401565802 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e800ccc0 con 0x7f25f81029f0 2024-04-25T16:01:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.154+0000 7f25e77fe640 1 -- 172.21.15.31:0/401565802 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25e80035f0 con 0x7f25f81029f0 2024-04-25T16:01:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.154+0000 7f25e77fe640 1 -- 172.21.15.31:0/401565802 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e800ce60 con 0x7f25f81029f0 2024-04-25T16:01:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.154+0000 7f25fcee9640 1 -- 172.21.15.31:0/401565802 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f81a1fe0 con 0x7f25f81029f0 2024-04-25T16:01:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.154+0000 7f25fcee9640 1 -- 172.21.15.31:0/401565802 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f81a23f0 con 0x7f25f81029f0 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.156+0000 7f2603d95740 1 Processor -- start 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.156+0000 7f25e77fe640 1 -- 172.21.15.31:0/401565802 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f25e8003aa0 con 0x7f25f81029f0 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.156+0000 7f2603d95740 1 -- start start 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f2603d95740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cfa1c270 0x5571cfdbd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f2603d95740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 0x5571cfdbd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f2603d95740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfdc47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f2603d95740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5571cfd22740 con 0x5571cfa1c270 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f2603d95740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5571cfcbe3a0 con 0x5571cf9fb590 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f2603d95740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5571cfd1e700 con 0x5571cf9f33b0 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cfa1c270 0x5571cfdbd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cfa1c270 0x5571cfdbd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56088/0 (socket says 172.21.15.31:56088) 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f6d9d640 1 -- 172.21.15.31:0/3132815149 learned_addr learned my addr 172.21.15.31:0/3132815149 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfdc47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f659c640 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 0x5571cfdbd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f5d9b640 1 -- 172.21.15.31:0/3132815149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 msgr2=0x5571cfdbd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 0x5571cfdbd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f5d9b640 1 -- 172.21.15.31:0/3132815149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cfa1c270 msgr2=0x5571cfdbd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25e77fe640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25e00afaf0 0x7f25e00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cfa1c270 0x5571cfdbd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f5d9b640 1 -- 172.21.15.31:0/3132815149 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571cfcda750 con 0x5571cf9fb590 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f6d9d640 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cfa1c270 0x5571cfdbd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f6d9d640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25e00afaf0 0x7f25e00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25e77fe640 1 -- 172.21.15.31:0/401565802 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f25e80defa0 con 0x7f25f81029f0 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.157+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfdc47f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25f819b2b0 tx=0x7f25ec032b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca2edee26d9db4da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.158+0000 7f25e4ff9640 1 -- 172.21.15.31:0/3132815149 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec03f070 con 0x5571cf9fb590 2024-04-25T16:01:04.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.158+0000 7f25e4ff9640 1 -- 172.21.15.31:0/3132815149 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25ec004040 con 0x5571cf9fb590 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.158+0000 7f25e4ff9640 1 -- 172.21.15.31:0/3132815149 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec035070 con 0x5571cf9fb590 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 -- 172.21.15.31:0/3132815149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 msgr2=0x5571cfdc47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfdc47f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f25f819b2b0 tx=0x7f25ec032b20 comp rx=0 tx=0).stop 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f25f6d9d640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25e00afaf0 0x7f25e00b2000 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5571cfdbe540 tx=0x7f25f0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 -- 172.21.15.31:0/3132815149 shutdown_connections 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfdc47f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 0x5571cfdbd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 --2- 172.21.15.31:0/3132815149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cfa1c270 0x5571cfdbd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 -- 172.21.15.31:0/3132815149 >> 172.21.15.31:0/3132815149 conn(0x5571cfa103c0 msgr2=0x5571cfdb8d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 -- 172.21.15.31:0/3132815149 shutdown_connections 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 -- 172.21.15.31:0/3132815149 wait complete. 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 -- start start 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 Processor -- start 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9fb590 0x5571cfdc7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.159+0000 7f2603d95740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cfa1c270 0x5571cfdce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f2603d95740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5571cfdc51e0 con 0x5571cf9f33b0 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f2603d95740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5571cfca9fe0 con 0x5571cfa1c270 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f2603d95740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5571cfdc5360 con 0x5571cf9fb590 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f2603d95740 10 client.0 _ll_register_callbacks cb 0x5571cfcc0150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56094/0 (socket says 172.21.15.31:56094) 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f659c640 1 -- 172.21.15.31:0/238619366 learned_addr learned my addr 172.21.15.31:0/238619366 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f6d9d640 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cfa1c270 0x5571cfdce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9fb590 0x5571cfdc7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f659c640 1 -- 172.21.15.31:0/238619366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9fb590 msgr2=0x5571cfdc7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f659c640 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9fb590 0x5571cfdc7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f659c640 1 -- 172.21.15.31:0/238619366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cfa1c270 msgr2=0x5571cfdce6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f659c640 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cfa1c270 0x5571cfdce6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f659c640 1 -- 172.21.15.31:0/238619366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571cfa342d0 con 0x5571cf9f33b0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f6d9d640 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cfa1c270 0x5571cfdce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25f659c640 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc7230 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f25e80dcb70 tx=0x7f25e80a2dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d2f916b672bc3d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25ddffb640 1 -- 172.21.15.31:0/238619366 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e80a6ca0 con 0x5571cf9f33b0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25ddffb640 1 -- 172.21.15.31:0/238619366 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f25e80aff00 con 0x5571cf9f33b0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f2603d95740 1 -- 172.21.15.31:0/238619366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25ec004040 con 0x5571cf9f33b0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f25ddffb640 1 -- 172.21.15.31:0/238619366 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f25e80037a0 con 0x5571cf9f33b0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.160+0000 7f2603d95740 10 client.15321 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f2603d95740 1 -- 172.21.15.31:0/238619366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5571cfafe6b0 con 0x5571cf9f33b0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f25ddffb640 1 -- 172.21.15.31:0/238619366 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25ec004040 con 0x5571cf9f33b0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f25ddffb640 1 -- 172.21.15.31:0/238619366 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f25e80a2ea0 con 0x5571cf9f33b0 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f2603d95740 10 client.15321 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f2603d95740 10 client.15321 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f2603d95740 20 client.15321 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:04.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f2603d95740 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571cf9dae40 0x5571cfdd4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f2603d95740 4 client.15321 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f2603d95740 1 -- 172.21.15.31:0/238619366 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571cfd30350 con 0x5571cf9dae40 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.161+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571cf9dae40 0x5571cfdd4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.162+0000 7f25ddffb640 10 client.15321 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.162+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571cf9dae40 0x5571cfdd4f80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16215c6477ec7ae7 server_cookie=581c517175de0049 in_seq=0 out_seq=0 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.162+0000 7f25ddffb640 1 -- 172.21.15.31:0/238619366 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5571cfd30350 con 0x5571cf9dae40 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.162+0000 7f25ddffb640 10 client.15321 handle_command_reply: tid=0 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.162+0000 7f2603d95740 1 client.15321 shutdown 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 -- 172.21.15.31:0/238619366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 msgr2=0x5571cfdc7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc7230 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f25e80dcb70 tx=0x7f25e80a2dc0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 -- 172.21.15.31:0/238619366 shutdown_connections 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571cf9dae40 0x5571cfdd4f80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cfa1c270 0x5571cfdce6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9fb590 0x5571cfdc7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f25ddffb640 0 client.15321 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:04.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 --2- 172.21.15.31:0/238619366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc7230 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 -- 172.21.15.31:0/238619366 >> 172.21.15.31:0/238619366 conn(0x5571cfa103c0 msgr2=0x5571cfdb7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 -- 172.21.15.31:0/238619366 shutdown_connections 2024-04-25T16:01:04.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 1 -- 172.21.15.31:0/238619366 wait complete. 2024-04-25T16:01:04.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.163+0000 7f2603d95740 20 client.15321 trim_cache size 0 max 16384 2024-04-25T16:01:04.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f2603d95740 1 Processor -- start 2024-04-25T16:01:04.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f2603d95740 1 -- start start 2024-04-25T16:01:04.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f2603d95740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9dae40 0x5571cfdc5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f2603d95740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f2603d95740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfdce660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f2603d95740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5571cfdd32c0 con 0x5571cf9f33b0 2024-04-25T16:01:04.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f2603d95740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5571cfdd3990 con 0x5571cf9fb590 2024-04-25T16:01:04.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f2603d95740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5571cfdd4060 con 0x5571cf9dae40 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56098/0 (socket says 172.21.15.31:56098) 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f5d9b640 1 -- 172.21.15.31:0/895917112 learned_addr learned my addr 172.21.15.31:0/895917112 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f6d9d640 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfdce660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f659c640 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9dae40 0x5571cfdc5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f5d9b640 1 -- 172.21.15.31:0/895917112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9dae40 msgr2=0x5571cfdc5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9dae40 0x5571cfdc5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f5d9b640 1 -- 172.21.15.31:0/895917112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 msgr2=0x5571cfdce660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfdce660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f5d9b640 1 -- 172.21.15.31:0/895917112 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5571cfdc2130 con 0x5571cf9f33b0 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f659c640 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9dae40 0x5571cfdc5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.182+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc5d90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f25ec005f70 tx=0x7f25ec02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b83caf24516b44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f25dffff640 1 -- 172.21.15.31:0/895917112 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec056070 con 0x5571cf9f33b0 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f25dffff640 1 -- 172.21.15.31:0/895917112 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25ec005500 con 0x5571cf9f33b0 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f25dffff640 1 -- 172.21.15.31:0/895917112 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec035040 con 0x5571cf9f33b0 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f2603d95740 1 -- 172.21.15.31:0/895917112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 msgr2=0x5571cfdc5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f2603d95740 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc5d90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f25ec005f70 tx=0x7f25ec02a410 comp rx=0 tx=0).stop 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f2603d95740 1 -- 172.21.15.31:0/895917112 shutdown_connections 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f2603d95740 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfdce660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f2603d95740 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9f33b0 0x5571cfdc5d90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f2603d95740 1 --2- 172.21.15.31:0/895917112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9dae40 0x5571cfdc5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f2603d95740 1 -- 172.21.15.31:0/895917112 >> 172.21.15.31:0/895917112 conn(0x5571cfa103c0 msgr2=0x5571cfdc09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f2603d95740 1 -- 172.21.15.31:0/895917112 shutdown_connections 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.183+0000 7f2603d95740 1 -- 172.21.15.31:0/895917112 wait complete. 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f2603d95740 1 -- start start 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f2603d95740 1 Processor -- start 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f2603d95740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9dae40 0x5571cfdc9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f2603d95740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 0x5571cfdca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f2603d95740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfe3bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f2603d95740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5571cfdd4e00 con 0x5571cf9dae40 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f2603d95740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5571cfdd4730 con 0x5571cf9fb590 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f2603d95740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5571cfdd54d0 con 0x5571cf9f33b0 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f2603d95740 10 client.0 _ll_register_callbacks cb 0x7f25e80a33f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f25f659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9dae40 0x5571cfdc9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f25f659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9dae40 0x5571cfdc9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56100/0 (socket says 172.21.15.31:56100) 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f25f659c640 1 -- 172.21.15.31:0/4254505141 learned_addr learned my addr 172.21.15.31:0/4254505141 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 0x5571cfdca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f25f6d9d640 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfe3bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f25f659c640 1 -- 172.21.15.31:0/4254505141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 msgr2=0x5571cfdca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f25f659c640 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 0x5571cfdca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f25f659c640 1 -- 172.21.15.31:0/4254505141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 msgr2=0x5571cfe3bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:04.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.184+0000 7f25f659c640 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfe3bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f25f659c640 1 -- 172.21.15.31:0/4254505141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571cfe3c360 con 0x5571cf9dae40 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f25f659c640 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9dae40 0x5571cfdc9ed0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f25e80a3270 tx=0x7f25e8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d47e73d6255662 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f25ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f25ddffb640 1 -- 172.21.15.31:0/4254505141 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25e80a6ca0 con 0x5571cf9dae40 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f25ddffb640 1 -- 172.21.15.31:0/4254505141 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f25e80aff00 con 0x5571cf9dae40 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f2603d95740 1 -- 172.21.15.31:0/4254505141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25ec005500 con 0x5571cf9dae40 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f25ddffb640 1 -- 172.21.15.31:0/4254505141 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f25e80a7300 con 0x5571cf9dae40 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f2603d95740 10 client.15327 fetch_fsmap learned FSMap version 26 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f2603d95740 1 -- 172.21.15.31:0/4254505141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5571cfe3d5a0 con 0x5571cf9dae40 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f25ddffb640 1 -- 172.21.15.31:0/4254505141 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25ec005500 con 0x5571cf9dae40 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f25ddffb640 1 -- 172.21.15.31:0/4254505141 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7f25e8082b70 con 0x5571cf9dae40 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f2603d95740 10 client.15327 fetch_fsmap finished waiting for FSMap version 26 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f2603d95740 10 client.15327 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-25T16:01:04.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f2603d95740 20 client.15327 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f2603d95740 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571cfa1c270 0x5571cfe40e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f2603d95740 4 client.15327 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.185+0000 7f2603d95740 1 -- 172.21.15.31:0/4254505141 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "status"}) -- 0x5571cfd2a1c0 con 0x5571cfa1c270 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.186+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571cfa1c270 0x5571cfe40e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.186+0000 7f25f5d9b640 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571cfa1c270 0x5571cfe40e80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2ddd530183aeab8 server_cookie=f0a7b18e605907ae in_seq=0 out_seq=0 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.186+0000 7f25ddffb640 10 client.15327 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f25ddffb640 1 -- 172.21.15.31:0/4254505141 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x5571cfd2a1c0 con 0x5571cfa1c270 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f25ddffb640 10 client.15327 handle_command_reply: tid=0 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 client.15327 shutdown 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 -- 172.21.15.31:0/4254505141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9dae40 msgr2=0x5571cfdc9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9dae40 0x5571cfdc9ed0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f25e80a3270 tx=0x7f25e8004740 comp rx=0 tx=0).stop 2024-04-25T16:01:04.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 -- 172.21.15.31:0/4254505141 shutdown_connections 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571cfa1c270 0x5571cfe40e80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5571cf9fb590 0x5571cfe3bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5571cf9f33b0 0x5571cfdca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f25ddffb640 0 client.15327 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 --2- 172.21.15.31:0/4254505141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5571cf9dae40 0x5571cfdc9ed0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 -- 172.21.15.31:0/4254505141 >> 172.21.15.31:0/4254505141 conn(0x5571cfa103c0 msgr2=0x5571cfdc5700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 -- 172.21.15.31:0/4254505141 shutdown_connections 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 1 -- 172.21.15.31:0/4254505141 wait complete. 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f2603d95740 20 client.15327 trim_cache size 0 max 16384 2024-04-25T16:01:04.195 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "cluster_fsid": "74fd3742-031b-11ef-bc93-c7b262605968", 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "whoami": 0, 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "id": 24563, 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "want_state": "up:active", 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "state": "up:active", 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "fs_name": "cephfs", 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "rank_uptime": 120.691825279, 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "mdsmap_epoch": 26, 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch": 77, 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_epoch_barrier": 75, 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout: "uptime": 125.905358127 2024-04-25T16:01:04.196 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f25fcee9640 1 -- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25e00afaf0 msgr2=0x7f25e00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f25fcee9640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25e00afaf0 0x7f25e00b2000 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x5571cfdbe540 tx=0x7f25f0004a40 comp rx=0 tx=0).stop 2024-04-25T16:01:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f25fcee9640 1 -- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f81029f0 msgr2=0x7f25f819a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.187+0000 7f25fcee9640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f81029f0 0x7f25f819a210 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f25e800e9f0 tx=0x7f25e800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.188+0000 7f25fcee9640 1 -- 172.21.15.31:0/401565802 shutdown_connections 2024-04-25T16:01:04.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.188+0000 7f25fcee9640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f25e00afaf0 0x7f25e00b2000 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.188+0000 7f25fcee9640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f8108930 0x7f25f81a16e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.188+0000 7f25fcee9640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f25f8103c30 0x7f25f819a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.188+0000 7f25fcee9640 1 --2- 172.21.15.31:0/401565802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f25f81029f0 0x7f25f819a210 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.188+0000 7f25fcee9640 1 -- 172.21.15.31:0/401565802 >> 172.21.15.31:0/401565802 conn(0x7f25f80fdbc0 msgr2=0x7f25f8104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.188+0000 7f25fcee9640 1 -- 172.21.15.31:0/401565802 shutdown_connections 2024-04-25T16:01:04.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:04.188+0000 7f25fcee9640 1 -- 172.21.15.31:0/401565802 wait complete. 2024-04-25T16:01:04.201 INFO:teuthology.run_tasks:Running task sequential... 2024-04-25T16:01:04.213 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-25T16:01:04.213 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-25T16:01:04.213 INFO:teuthology.task.print:Enabling mgr modules 2024-04-25T16:01:04.213 INFO:teuthology.run_tasks:Running task kclient... 2024-04-25T16:01:04.280 INFO:tasks.kclient:Mounting kernel clients... 2024-04-25T16:01:04.280 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-25T16:01:04.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:04.280 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-25T16:01:04.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:04.296 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-04-25T16:01:04.363 INFO:teuthology.orchestra.run.smithi031.stderr:Cannot find device "ceph-brx" 2024-04-25T16:01:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:01:04.365 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-04-25T16:01:04.365 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-25T16:01:04.365 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T16:01:04.365 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-25T16:01:04.654 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:01:04.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:04 smithi031 ceph-mon[29140]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:01:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:04 smithi063 ceph-mon[32413]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:01:04.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:04 smithi186 ceph-mon[33258]: pgmap v344: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-25T16:01:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.426+0000 7f28b1c2c700 1 -- 172.21.15.31:0/1513233994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac104490 msgr2=0x7f28ac1048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.426+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/1513233994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac104490 0x7f28ac1048a0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f289c0099c0 tx=0x7f289c009cd0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.426+0000 7f28b1c2c700 1 -- 172.21.15.31:0/1513233994 shutdown_connections 2024-04-25T16:01:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.426+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/1513233994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac109710 0x7f28ac111070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.426+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/1513233994 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac104490 0x7f28ac1048a0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.426+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/1513233994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac108dc0 0x7f28ac1091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.426+0000 7f28b1c2c700 1 -- 172.21.15.31:0/1513233994 >> 172.21.15.31:0/1513233994 conn(0x7f28ac0ffda0 msgr2=0x7f28ac1021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:06.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.426+0000 7f28b1c2c700 1 -- 172.21.15.31:0/1513233994 shutdown_connections 2024-04-25T16:01:06.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.426+0000 7f28b1c2c700 1 -- 172.21.15.31:0/1513233994 wait complete. 2024-04-25T16:01:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28b1c2c700 1 Processor -- start 2024-04-25T16:01:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28b1c2c700 1 -- start start 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28b1c2c700 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac104490 0x7f28ac1a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28b1c2c700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac108dc0 0x7f28ac1a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28b1c2c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac109710 0x7f28ac19ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28b1c2c700 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac114340 con 0x7f28ac108dc0 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28b1c2c700 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac1141c0 con 0x7f28ac104490 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28b1c2c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28ac1144c0 con 0x7f28ac109710 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28aaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac108dc0 0x7f28ac1a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28aaffd700 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac108dc0 0x7f28ac1a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56114/0 (socket says 172.21.15.31:56114) 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28aaffd700 1 -- 172.21.15.31:0/2178348298 learned_addr learned my addr 172.21.15.31:0/2178348298 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.427+0000 7f28abfff700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac109710 0x7f28ac19ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28ab7fe700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac104490 0x7f28ac1a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28aaffd700 1 -- 172.21.15.31:0/2178348298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac109710 msgr2=0x7f28ac19ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28aaffd700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac109710 0x7f28ac19ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28aaffd700 1 -- 172.21.15.31:0/2178348298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac104490 msgr2=0x7f28ac1a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28aaffd700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac104490 0x7f28ac1a08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28aaffd700 1 -- 172.21.15.31:0/2178348298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28ac19b170 con 0x7f28ac108dc0 2024-04-25T16:01:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28ab7fe700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac104490 0x7f28ac1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28aaffd700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac108dc0 0x7f28ac1a0e20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c9e0 tx=0x7f28a000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28a8ff9700 1 -- 172.21.15.31:0/2178348298 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0007aa0 con 0x7f28ac108dc0 2024-04-25T16:01:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28a8ff9700 1 -- 172.21.15.31:0/2178348298 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28a0007c40 con 0x7f28ac108dc0 2024-04-25T16:01:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28a8ff9700 1 -- 172.21.15.31:0/2178348298 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28a0018660 con 0x7f28ac108dc0 2024-04-25T16:01:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28ac19b460 con 0x7f28ac108dc0 2024-04-25T16:01:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.428+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28ac1a7770 con 0x7f28ac108dc0 2024-04-25T16:01:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.429+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28ac051190 con 0x7f28ac108dc0 2024-04-25T16:01:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.431+0000 7f28a8ff9700 1 -- 172.21.15.31:0/2178348298 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f28a0018800 con 0x7f28ac108dc0 2024-04-25T16:01:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.431+0000 7f28a8ff9700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28940afb80 0x7f28940b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:06.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.432+0000 7f28ab7fe700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28940afb80 0x7f28940b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:06.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.432+0000 7f28a8ff9700 1 -- 172.21.15.31:0/2178348298 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f28a00dee60 con 0x7f28ac108dc0 2024-04-25T16:01:06.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.432+0000 7f28ab7fe700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28940afb80 0x7f28940b2090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f289c000c00 tx=0x7f289c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:06.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.435+0000 7f28a8ff9700 1 -- 172.21.15.31:0/2178348298 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28a0014030 con 0x7f28ac108dc0 2024-04-25T16:01:06.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.692+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f28ac19bff0 con 0x7f28ac108dc0 2024-04-25T16:01:06.694 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:06 smithi031 ceph-mon[29140]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:01:06.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.693+0000 7f28a8ff9700 1 -- 172.21.15.31:0/2178348298 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v26) ==== 53+0+73 (secure 0 0 0) 0x7f28a00a7ae0 con 0x7f28ac108dc0 2024-04-25T16:01:06.695 INFO:teuthology.orchestra.run.smithi031.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-25T16:01:06.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.696+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28940afb80 msgr2=0x7f28940b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.696+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28940afb80 0x7f28940b2090 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f289c000c00 tx=0x7f289c01a040 comp rx=0 tx=0).stop 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.696+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac108dc0 msgr2=0x7f28ac1a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.696+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac108dc0 0x7f28ac1a0e20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f28a000c9e0 tx=0x7f28a000ccf0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.696+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 shutdown_connections 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.696+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28940afb80 0x7f28940b2090 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.697+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28ac104490 0x7f28ac1a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.697+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac109710 0x7f28ac19ab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.697+0000 7f28b1c2c700 1 --2- 172.21.15.31:0/2178348298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28ac108dc0 0x7f28ac1a0e20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.697+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 >> 172.21.15.31:0/2178348298 conn(0x7f28ac0ffda0 msgr2=0x7f28ac10a670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.697+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 shutdown_connections 2024-04-25T16:01:06.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:06.697+0000 7f28b1c2c700 1 -- 172.21.15.31:0/2178348298 wait complete. 2024-04-25T16:01:06.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:06 smithi063 ceph-mon[32413]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:01:06.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:06 smithi186 ceph-mon[33258]: pgmap v345: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-25T16:01:07.400 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-25T16:01:07.401 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-25T16:01:07.401 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi031.front.sepia.ceph.com 2024-04-25T16:01:07.401 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-25T16:01:07.401 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-25T16:01:07.401 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-25T16:01:07.401 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-25T16:01:07.401 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-25T16:01:07.401 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-25T16:01:07.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:07.401 DEBUG:teuthology.orchestra.run.smithi031:> ip addr 2024-04-25T16:01:07.417 INFO:teuthology.orchestra.run.smithi031.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-25T16:01:07.417 INFO:teuthology.orchestra.run.smithi031.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-25T16:01:07.417 INFO:teuthology.orchestra.run.smithi031.stdout: inet 127.0.0.1/8 scope host lo 2024-04-25T16:01:07.417 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 ::1/128 scope host 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cc brd ff:ff:ff:ff:ff:ff 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f0 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether 0c:c4:7a:6b:f6:cd brd ff:ff:ff:ff:ff:ff 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp7s0f1 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:ba brd ff:ff:ff:ff:ff:ff 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f0 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: inet 172.21.15.31/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft 42095sec preferred_lft 42095sec 2024-04-25T16:01:07.418 INFO:teuthology.orchestra.run.smithi031.stdout: inet6 fe80::ae1f:6bff:fea5:caba/64 scope link noprefixroute 2024-04-25T16:01:07.419 INFO:teuthology.orchestra.run.smithi031.stdout: valid_lft forever preferred_lft forever 2024-04-25T16:01:07.419 INFO:teuthology.orchestra.run.smithi031.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-25T16:01:07.419 INFO:teuthology.orchestra.run.smithi031.stdout: link/ether ac:1f:6b:a5:ca:bb brd ff:ff:ff:ff:ff:ff 2024-04-25T16:01:07.419 INFO:teuthology.orchestra.run.smithi031.stdout: altname enp1s0f1 2024-04-25T16:01:07.419 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-25T16:01:07.419 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:01:07.419 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T16:01:07.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add name ceph-brx type bridge 2024-04-25T16:01:07.419 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr flush dev ceph-brx 2024-04-25T16:01:07.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set ceph-brx up 2024-04-25T16:01:07.420 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-25T16:01:07.420 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T16:01:07.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:01:07.586 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:07 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2178348298' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:01:07.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:01:07.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:07.616 DEBUG:teuthology.orchestra.run.smithi031:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-25T16:01:07.640 INFO:teuthology.orchestra.run.smithi031.stdout:1 2024-04-25T16:01:07.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:07.642 DEBUG:teuthology.orchestra.run.smithi031:> route 2024-04-25T16:01:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:Kernel IP routing table 2024-04-25T16:01:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-25T16:01:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-25T16:01:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-25T16:01:07.708 INFO:teuthology.orchestra.run.smithi031.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-25T16:01:07.708 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:01:07.709 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T16:01:07.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-25T16:01:07.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-25T16:01:07.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-25T16:01:07.709 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T16:01:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:01:07.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:07 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2178348298' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:01:07.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:07 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2178348298' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-25T16:01:07.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:01:07.985 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:07.985 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-25T16:01:08.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:08.001 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list-id 2024-04-25T16:01:08.057 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:01:08.057 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T16:01:08.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:01:08.057 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-25T16:01:08.057 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T16:01:08.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:01:08.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:01:08.176 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-25T16:01:08.176 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:01:08.176 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T16:01:08.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-25T16:01:08.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-25T16:01:08.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-25T16:01:08.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-25T16:01:08.176 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-25T16:01:08.176 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T16:01:08.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:01:08.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:01:08.303 DEBUG:teuthology.orchestra.run.smithi031:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-25T16:01:08.303 DEBUG:teuthology.orchestra.run.smithi031:> set -e 2024-04-25T16:01:08.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set brx.0 up 2024-04-25T16:01:08.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link set dev brx.0 master ceph-brx 2024-04-25T16:01:08.303 DEBUG:teuthology.orchestra.run.smithi031:> ') 2024-04-25T16:01:08.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:01:08.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:01:08.382 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T16:01:08.382 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-25T16:01:08.397 INFO:teuthology.orchestra.run.smithi031.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-25T16:01:08.397 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-25T16:01:08.397 DEBUG:teuthology.orchestra.run.smithi031:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-25T16:01:08.452 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-25T16:01:08.452 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-25T16:01:08.452 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/80cd7e96-3109-49c9-8578-0278f2cf668e 2024-04-25T16:01:08.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:08.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/80cd7e96-3109-49c9-8578-0278f2cf668e /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-04-25T16:01:08.554 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-04-25T16:01:08.554 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-04-25T16:01:08.676 INFO:teuthology.orchestra.run.smithi031.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=secure,nowsync,nofallback 2024-04-25T16:01:08.676 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync". 2024-04-25T16:01:08.676 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: resolved to: "172.21.15.31:3300,172.21.15.63:3300,172.21.15.186:3300" 2024-04-25T16:01:08.676 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: trying mount with new device syntax: 0@74fd3742-031b-11ef-bc93-c7b262605968.cephfs=/volumes/qa/sv_1/80cd7e96-3109-49c9-8578-0278f2cf668e 2024-04-25T16:01:08.676 INFO:teuthology.orchestra.run.smithi031.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=secure,nowsync,name=0,key=0,mon_addr=172.21.15.31:3300/172.21.15.63:3300/172.21.15.186:3300" will pass to kernel 2024-04-25T16:01:08.677 INFO:teuthology.orchestra.run.smithi031.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-25T16:01:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: You just mounted a file system that supports labels which does not 2024-04-25T16:01:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-25T16:01:08.677 INFO:teuthology.orchestra.run.smithi031.stdout: applications will generate AVC messages and not be allowed access to 2024-04-25T16:01:08.678 INFO:teuthology.orchestra.run.smithi031.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-25T16:01:08.679 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-25T16:01:08.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:08.679 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-25T16:01:08.708 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T16:01:08.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T16:01:08.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:01:08.807 INFO:teuthology.orchestra.run.smithi031.stdout:client25094 2024-04-25T16:01:08.807 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T16:01:08.808 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T16:01:08.824 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T16:01:08.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:08 smithi063 ceph-mon[32413]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:01:08.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:08 smithi186 ceph-mon[33258]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:01:08.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T16:01:08.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:01:08.923 INFO:teuthology.orchestra.run.smithi031.stdout:client25094 2024-04-25T16:01:08.923 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-25T16:01:08.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:08 smithi031 ceph-mon[29140]: pgmap v346: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-04-25T16:01:08.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-25T16:01:08.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:01:08.980 INFO:teuthology.orchestra.run.smithi031.stdout:client25094 2024-04-25T16:01:08.980 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/74fd3742-031b-11ef-bc93-c7b262605968.client25094/status') 2024-04-25T16:01:09.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-25T16:01:09.080 INFO:teuthology.orchestra.run.smithi031.stdout:instance: client.25094 (3)172.21.15.31:0/1644811692 2024-04-25T16:01:09.080 INFO:teuthology.orchestra.run.smithi031.stdout:blocklisted: false 2024-04-25T16:01:09.080 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records in 2024-04-25T16:01:09.080 INFO:teuthology.orchestra.run.smithi031.stderr:0+1 records out 2024-04-25T16:01:09.080 INFO:teuthology.orchestra.run.smithi031.stderr:71 bytes copied, 2.7239e-05 s, 2.6 MB/s 2024-04-25T16:01:09.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-25T16:01:09.088 INFO:teuthology.run_tasks:Running task exec... 2024-04-25T16:01:09.098 INFO:teuthology.task.exec:Executing custom commands... 2024-04-25T16:01:09.098 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T16:01:09.098 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-04-25T16:01:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 -- 172.21.15.31:0/391563970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 msgr2=0x7f387c105500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 --2- 172.21.15.31:0/391563970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 0x7f387c105500 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f386c009900 tx=0x7f386c02a410 comp rx=0 tx=0).stop 2024-04-25T16:01:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 -- 172.21.15.31:0/391563970 shutdown_connections 2024-04-25T16:01:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 --2- 172.21.15.31:0/391563970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f387c1022a0 0x7f387c10bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 --2- 172.21.15.31:0/391563970 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f387c1017a0 0x7f387c101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 --2- 172.21.15.31:0/391563970 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 0x7f387c105500 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 -- 172.21.15.31:0/391563970 >> 172.21.15.31:0/391563970 conn(0x7f387c0fd3c0 msgr2=0x7f387c0ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 -- 172.21.15.31:0/391563970 shutdown_connections 2024-04-25T16:01:09.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 -- 172.21.15.31:0/391563970 wait complete. 2024-04-25T16:01:09.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.210+0000 7f3880b4d640 1 Processor -- start 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f3880b4d640 1 -- start start 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f3880b4d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f387c1017a0 0x7f387c19a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f3880b4d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f387c1022a0 0x7f387c19a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f3880b4d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 0x7f387c1a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f3880b4d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f387c10f640 con 0x7f387c1050f0 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f3880b4d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f387c10f4c0 con 0x7f387c1022a0 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f3880b4d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f387c10f7c0 con 0x7f387c1017a0 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 0x7f387c1a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 0x7f387c1a1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56122/0 (socket says 172.21.15.31:56122) 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387ad9d640 1 -- 172.21.15.31:0/286349367 learned_addr learned my addr 172.21.15.31:0/286349367 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f3873fff640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f387c1022a0 0x7f387c19a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387a59c640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f387c1017a0 0x7f387c19a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387ad9d640 1 -- 172.21.15.31:0/286349367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f387c1017a0 msgr2=0x7f387c19a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387ad9d640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f387c1017a0 0x7f387c19a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387ad9d640 1 -- 172.21.15.31:0/286349367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f387c1022a0 msgr2=0x7f387c19a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387ad9d640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f387c1022a0 0x7f387c19a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387ad9d640 1 -- 172.21.15.31:0/286349367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f387c1a1d80 con 0x7f387c1050f0 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387a59c640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f387c1017a0 0x7f387c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.211+0000 7f387ad9d640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 0x7f387c1a1710 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f387400eaa0 tx=0x7f387400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.212+0000 7f38737fe640 1 -- 172.21.15.31:0/286349367 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f387400cdb0 con 0x7f387c1050f0 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.212+0000 7f38737fe640 1 -- 172.21.15.31:0/286349367 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3874005020 con 0x7f387c1050f0 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.212+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f387c1a2070 con 0x7f387c1050f0 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.212+0000 7f38737fe640 1 -- 172.21.15.31:0/286349367 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38740116d0 con 0x7f387c1050f0 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.212+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f387c1a2570 con 0x7f387c1050f0 2024-04-25T16:01:09.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.213+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f387c108a80 con 0x7f387c1050f0 2024-04-25T16:01:09.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.214+0000 7f38737fe640 1 -- 172.21.15.31:0/286349367 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f38740051d0 con 0x7f387c1050f0 2024-04-25T16:01:09.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.215+0000 7f38737fe640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38680afae0 0x7f38680b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:09.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.215+0000 7f387a59c640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38680afae0 0x7f38680b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:09.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.215+0000 7f38737fe640 1 -- 172.21.15.31:0/286349367 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f38740df1a0 con 0x7f387c1050f0 2024-04-25T16:01:09.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.216+0000 7f387a59c640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38680afae0 0x7f38680b1ff0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f386c005f70 tx=0x7f386c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:09.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.219+0000 7f38737fe640 1 -- 172.21.15.31:0/286349367 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3874016030 con 0x7f387c1050f0 2024-04-25T16:01:09.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.452+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f387c02db50 con 0x7f387c1050f0 2024-04-25T16:01:09.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.511+0000 7f38737fe640 1 -- 172.21.15.31:0/286349367 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v27) ==== 117+0+0 (secure 0 0 0) 0x7f38740a1e20 con 0x7f387c1050f0 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38680afae0 msgr2=0x7f38680b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38680afae0 0x7f38680b1ff0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f386c005f70 tx=0x7f386c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 msgr2=0x7f387c1a1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 0x7f387c1a1710 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f387400eaa0 tx=0x7f387400ef60 comp rx=0 tx=0).stop 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 shutdown_connections 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f38680afae0 0x7f38680b1ff0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f387c1050f0 0x7f387c1a1710 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f387c1022a0 0x7f387c19a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 --2- 172.21.15.31:0/286349367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f387c1017a0 0x7f387c19a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 >> 172.21.15.31:0/286349367 conn(0x7f387c0fd3c0 msgr2=0x7f387c105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 shutdown_connections 2024-04-25T16:01:09.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.514+0000 7f3880b4d640 1 -- 172.21.15.31:0/286349367 wait complete. 2024-04-25T16:01:09.528 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-25T16:01:09.540 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-25T16:01:09.575 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:01:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.685+0000 7f08a8449640 1 -- 172.21.15.31:0/3193201060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0108920 msgr2=0x7f08a010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.685+0000 7f08a8449640 1 --2- 172.21.15.31:0/3193201060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0108920 0x7f08a010ad40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0898009ca0 tx=0x7f089802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.685+0000 7f08a8449640 1 -- 172.21.15.31:0/3193201060 shutdown_connections 2024-04-25T16:01:09.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.685+0000 7f08a8449640 1 --2- 172.21.15.31:0/3193201060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0108920 0x7f08a010ad40 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.685+0000 7f08a8449640 1 --2- 172.21.15.31:0/3193201060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a0103c20 0x7f08a0104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.685+0000 7f08a8449640 1 --2- 172.21.15.31:0/3193201060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a01029e0 0x7f08a01036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.685+0000 7f08a8449640 1 -- 172.21.15.31:0/3193201060 >> 172.21.15.31:0/3193201060 conn(0x7f08a00fdbf0 msgr2=0x7f08a0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.685+0000 7f08a8449640 1 -- 172.21.15.31:0/3193201060 shutdown_connections 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.685+0000 7f08a8449640 1 -- 172.21.15.31:0/3193201060 wait complete. 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a8449640 1 Processor -- start 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a8449640 1 -- start start 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a8449640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a01029e0 0x7f08a0075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a8449640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0103c20 0x7f08a019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a8449640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a0108920 0x7f08a01a1530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a8449640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f08a010e560 con 0x7f08a0103c20 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a8449640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f08a010e3e0 con 0x7f08a01029e0 2024-04-25T16:01:09.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a8449640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f08a010e6e0 con 0x7f08a0108920 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a59e4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0103c20 0x7f08a019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a59e4640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0103c20 0x7f08a019efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56140/0 (socket says 172.21.15.31:56140) 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.686+0000 7f08a59e4640 1 -- 172.21.15.31:0/810397632 learned_addr learned my addr 172.21.15.31:0/810397632 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a69e6640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a0108920 0x7f08a01a1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a61e5640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a01029e0 0x7f08a0075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a59e4640 1 -- 172.21.15.31:0/810397632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a0108920 msgr2=0x7f08a01a1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a59e4640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a0108920 0x7f08a01a1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a59e4640 1 -- 172.21.15.31:0/810397632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a01029e0 msgr2=0x7f08a0075860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a59e4640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a01029e0 0x7f08a0075860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a59e4640 1 -- 172.21.15.31:0/810397632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a01a1d50 con 0x7f08a0103c20 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a69e6640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a0108920 0x7f08a01a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a59e4640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0103c20 0x7f08a019efc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f089000d950 tx=0x7f089000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08977fe640 1 -- 172.21.15.31:0/810397632 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0890017070 con 0x7f08a0103c20 2024-04-25T16:01:09.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08977fe640 1 -- 172.21.15.31:0/810397632 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0890009c30 con 0x7f08a0103c20 2024-04-25T16:01:09.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a01a1fe0 con 0x7f08a0103c20 2024-04-25T16:01:09.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08977fe640 1 -- 172.21.15.31:0/810397632 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089000f620 con 0x7f08a0103c20 2024-04-25T16:01:09.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.687+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a01a24d0 con 0x7f08a0103c20 2024-04-25T16:01:09.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.688+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a0107730 con 0x7f08a0103c20 2024-04-25T16:01:09.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.691+0000 7f08977fe640 1 -- 172.21.15.31:0/810397632 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0890010460 con 0x7f08a0103c20 2024-04-25T16:01:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.691+0000 7f08977fe640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f088c0afaf0 0x7f088c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.691+0000 7f08a61e5640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f088c0afaf0 0x7f088c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.691+0000 7f08977fe640 1 -- 172.21.15.31:0/810397632 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f08900de900 con 0x7f08a0103c20 2024-04-25T16:01:09.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.692+0000 7f08a61e5640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f088c0afaf0 0x7f088c0b2000 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f089c005fd0 tx=0x7f089c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:09.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.695+0000 7f08977fe640 1 -- 172.21.15.31:0/810397632 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f08900a7490 con 0x7f08a0103c20 2024-04-25T16:01:09.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:09 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/286349367' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T16:01:09.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.883+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08a002dda0 con 0x7f08a0103c20 2024-04-25T16:01:09.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.885+0000 7f08977fe640 1 -- 172.21.15.31:0/810397632 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f0890009de0 con 0x7f08a0103c20 2024-04-25T16:01:09.887 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:09.887 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f088c0afaf0 msgr2=0x7f088c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f088c0afaf0 0x7f088c0b2000 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f089c005fd0 tx=0x7f089c005850 comp rx=0 tx=0).stop 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0103c20 msgr2=0x7f08a019efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0103c20 0x7f08a019efc0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f089000d950 tx=0x7f089000de10 comp rx=0 tx=0).stop 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 shutdown_connections 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f088c0afaf0 0x7f088c0b2000 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f08a0108920 0x7f08a01a1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f08a0103c20 0x7f08a019efc0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 --2- 172.21.15.31:0/810397632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08a01029e0 0x7f08a0075860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:09.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 >> 172.21.15.31:0/810397632 conn(0x7f08a00fdbf0 msgr2=0x7f08a0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 shutdown_connections 2024-04-25T16:01:09.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:09.887+0000 7f08a8449640 1 -- 172.21.15.31:0/810397632 wait complete. 2024-04-25T16:01:09.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:09 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/286349367' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T16:01:09.899 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:09 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/286349367' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-04-25T16:01:09.901 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:59:00.978038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:59:01.829535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:01:09.901 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:01:10.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.014+0000 7f5078e92640 1 -- 172.21.15.31:0/4260345795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 msgr2=0x7f507410be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.014+0000 7f5078e92640 1 --2- 172.21.15.31:0/4260345795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 0x7f507410be50 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f506c009ca0 tx=0x7f506c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.014+0000 7f5078e92640 1 -- 172.21.15.31:0/4260345795 shutdown_connections 2024-04-25T16:01:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.014+0000 7f5078e92640 1 --2- 172.21.15.31:0/4260345795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 0x7f507410be50 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.014+0000 7f5078e92640 1 --2- 172.21.15.31:0/4260345795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5074104d30 0x7f5074105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.014+0000 7f5078e92640 1 --2- 172.21.15.31:0/4260345795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5074103af0 0x7f50741047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.014+0000 7f5078e92640 1 -- 172.21.15.31:0/4260345795 >> 172.21.15.31:0/4260345795 conn(0x7f50740fece0 msgr2=0x7f5074101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.014+0000 7f5078e92640 1 -- 172.21.15.31:0/4260345795 shutdown_connections 2024-04-25T16:01:10.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.014+0000 7f5078e92640 1 -- 172.21.15.31:0/4260345795 wait complete. 2024-04-25T16:01:10.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.015+0000 7f5078e92640 1 Processor -- start 2024-04-25T16:01:10.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.015+0000 7f5078e92640 1 -- start start 2024-04-25T16:01:10.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.015+0000 7f5078e92640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5074103af0 0x7f507419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.015+0000 7f5078e92640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5074104d30 0x7f507419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.015+0000 7f5078e92640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 0x7f50741a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.015+0000 7f5078e92640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f507410f600 con 0x7f5074109a30 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.015+0000 7f5078e92640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f507410f480 con 0x7f5074104d30 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.015+0000 7f5078e92640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f507410f780 con 0x7f5074103af0 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5072d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 0x7f50741a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5072d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 0x7f50741a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56162/0 (socket says 172.21.15.31:56162) 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5072d9d640 1 -- 172.21.15.31:0/1053164338 learned_addr learned my addr 172.21.15.31:0/1053164338 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f507259c640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5074103af0 0x7f507419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5069d9b640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5074104d30 0x7f507419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5072d9d640 1 -- 172.21.15.31:0/1053164338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5074103af0 msgr2=0x7f507419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5072d9d640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5074103af0 0x7f507419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5072d9d640 1 -- 172.21.15.31:0/1053164338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5074104d30 msgr2=0x7f507419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5072d9d640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5074104d30 0x7f507419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5072d9d640 1 -- 172.21.15.31:0/1053164338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50741a1d00 con 0x7f5074109a30 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5072d9d640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 0x7f50741a1690 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f506c004af0 tx=0x7f506c031700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f506bfff640 1 -- 172.21.15.31:0/1053164338 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c003b70 con 0x7f5074109a30 2024-04-25T16:01:10.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50741a1f90 con 0x7f5074109a30 2024-04-25T16:01:10.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f506bfff640 1 -- 172.21.15.31:0/1053164338 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f506c003d10 con 0x7f5074109a30 2024-04-25T16:01:10.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.016+0000 7f506bfff640 1 -- 172.21.15.31:0/1053164338 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506c02fb20 con 0x7f5074109a30 2024-04-25T16:01:10.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.017+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50741a23f0 con 0x7f5074109a30 2024-04-25T16:01:10.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.017+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f507406a160 con 0x7f5074109a30 2024-04-25T16:01:10.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.019+0000 7f506bfff640 1 -- 172.21.15.31:0/1053164338 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f506c004580 con 0x7f5074109a30 2024-04-25T16:01:10.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.020+0000 7f506bfff640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50600afaf0 0x7f50600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.020+0000 7f507259c640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50600afaf0 0x7f50600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.020+0000 7f506bfff640 1 -- 172.21.15.31:0/1053164338 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f506c100380 con 0x7f5074109a30 2024-04-25T16:01:10.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.020+0000 7f507259c640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50600afaf0 0x7f50600b2000 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f50741046f0 tx=0x7f5064005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:10.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.023+0000 7f506bfff640 1 -- 172.21.15.31:0/1053164338 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f506c0c8f10 con 0x7f5074109a30 2024-04-25T16:01:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.209+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f507402dbd0 con 0x7f5074109a30 2024-04-25T16:01:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.210+0000 7f506bfff640 1 -- 172.21.15.31:0/1053164338 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7f506c0c8810 con 0x7f5074109a30 2024-04-25T16:01:10.212 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50600afaf0 msgr2=0x7f50600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50600afaf0 0x7f50600b2000 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f50741046f0 tx=0x7f5064005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 msgr2=0x7f50741a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 0x7f50741a1690 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f506c004af0 tx=0x7f506c031700 comp rx=0 tx=0).stop 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 shutdown_connections 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50600afaf0 0x7f50600b2000 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5074109a30 0x7f50741a1690 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5074104d30 0x7f507419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 --2- 172.21.15.31:0/1053164338 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5074103af0 0x7f507419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 >> 172.21.15.31:0/1053164338 conn(0x7f50740fece0 msgr2=0x7f5074105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 shutdown_connections 2024-04-25T16:01:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.212+0000 7f5078e92640 1 -- 172.21.15.31:0/1053164338 wait complete. 2024-04-25T16:01:10.225 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:01:10.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.338+0000 7fd80de27640 1 -- 172.21.15.31:0/1513235918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8081029f0 msgr2=0x7fd8081036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.338+0000 7fd80de27640 1 --2- 172.21.15.31:0/1513235918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8081029f0 0x7fd8081036f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8009900 tx=0x7fd7f802a410 comp rx=0 tx=0).stop 2024-04-25T16:01:10.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.338+0000 7fd80de27640 1 -- 172.21.15.31:0/1513235918 shutdown_connections 2024-04-25T16:01:10.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.338+0000 7fd80de27640 1 --2- 172.21.15.31:0/1513235918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd808108930 0x7fd80810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.338+0000 7fd80de27640 1 --2- 172.21.15.31:0/1513235918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd808103c30 0x7fd808104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.338+0000 7fd80de27640 1 --2- 172.21.15.31:0/1513235918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8081029f0 0x7fd8081036f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.338+0000 7fd80de27640 1 -- 172.21.15.31:0/1513235918 >> 172.21.15.31:0/1513235918 conn(0x7fd8080fdbc0 msgr2=0x7fd808100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:10.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.339+0000 7fd80de27640 1 -- 172.21.15.31:0/1513235918 shutdown_connections 2024-04-25T16:01:10.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.339+0000 7fd80de27640 1 -- 172.21.15.31:0/1513235918 wait complete. 2024-04-25T16:01:10.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.339+0000 7fd80de27640 1 Processor -- start 2024-04-25T16:01:10.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.339+0000 7fd80de27640 1 -- start start 2024-04-25T16:01:10.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.339+0000 7fd80de27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8081029f0 0x7fd80819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd80de27640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd808103c30 0x7fd80819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd80de27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd808108930 0x7fd8081a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd80de27640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd80810e4d0 con 0x7fd808108930 2024-04-25T16:01:10.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd80de27640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd80810e350 con 0x7fd808103c30 2024-04-25T16:01:10.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd80de27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd80810e650 con 0x7fd8081029f0 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd807fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd808108930 0x7fd8081a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd807fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd808108930 0x7fd8081a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56184/0 (socket says 172.21.15.31:56184) 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd807fff640 1 -- 172.21.15.31:0/1117106736 learned_addr learned my addr 172.21.15.31:0/1117106736 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd806ffd640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd808103c30 0x7fd80819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd8077fe640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8081029f0 0x7fd80819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd807fff640 1 -- 172.21.15.31:0/1117106736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8081029f0 msgr2=0x7fd80819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd807fff640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8081029f0 0x7fd80819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd807fff640 1 -- 172.21.15.31:0/1117106736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd808103c30 msgr2=0x7fd80819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd807fff640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd808103c30 0x7fd80819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd807fff640 1 -- 172.21.15.31:0/1117106736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8081a1dc0 con 0x7fd808108930 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd8077fe640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8081029f0 0x7fd80819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd807fff640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd808108930 0x7fd8081a1750 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd800009fd0 tx=0x7fd80000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd804ff9640 1 -- 172.21.15.31:0/1117106736 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800017070 con 0x7fd808108930 2024-04-25T16:01:10.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8081a20b0 con 0x7fd808108930 2024-04-25T16:01:10.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd804ff9640 1 -- 172.21.15.31:0/1117106736 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd800003e20 con 0x7fd808108930 2024-04-25T16:01:10.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd804ff9640 1 -- 172.21.15.31:0/1117106736 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800010460 con 0x7fd808108930 2024-04-25T16:01:10.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.340+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8081a2600 con 0x7fd808108930 2024-04-25T16:01:10.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.341+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8081a2920 con 0x7fd808108930 2024-04-25T16:01:10.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.343+0000 7fd804ff9640 1 -- 172.21.15.31:0/1117106736 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd800003970 con 0x7fd808108930 2024-04-25T16:01:10.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.343+0000 7fd804ff9640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7f00afaf0 0x7fd7f00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.344+0000 7fd8077fe640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7f00afaf0 0x7fd7f00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.344+0000 7fd804ff9640 1 -- 172.21.15.31:0/1117106736 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd8000de9e0 con 0x7fd808108930 2024-04-25T16:01:10.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.344+0000 7fd8077fe640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7f00afaf0 0x7fd7f00b2000 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8005f70 tx=0x7fd7f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:10.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.347+0000 7fd804ff9640 1 -- 172.21.15.31:0/1117106736 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8000a7570 con 0x7fd808108930 2024-04-25T16:01:10.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.536+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd8081a2af0 con 0x7fd808108930 2024-04-25T16:01:10.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.537+0000 7fd804ff9640 1 -- 172.21.15.31:0/1117106736 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7fd8000a6e70 con 0x7fd808108930 2024-04-25T16:01:10.539 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T16:01:10.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7f00afaf0 msgr2=0x7fd7f00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7f00afaf0 0x7fd7f00b2000 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd7f8005f70 tx=0x7fd7f8000bc0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd808108930 msgr2=0x7fd8081a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd808108930 0x7fd8081a1750 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd800009fd0 tx=0x7fd80000b260 comp rx=0 tx=0).stop 2024-04-25T16:01:10.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 shutdown_connections 2024-04-25T16:01:10.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7f00afaf0 0x7fd7f00b2000 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd808108930 0x7fd8081a1750 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd808103c30 0x7fd80819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 --2- 172.21.15.31:0/1117106736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8081029f0 0x7fd80819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 >> 172.21.15.31:0/1117106736 conn(0x7fd8080fdbc0 msgr2=0x7fd808074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:10.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 shutdown_connections 2024-04-25T16:01:10.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.539+0000 7fd80de27640 1 -- 172.21.15.31:0/1117106736 wait complete. 2024-04-25T16:01:10.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:01:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.666+0000 7fbed9e27640 1 -- 172.21.15.31:0/1389739754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4109a30 msgr2=0x7fbed410be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.666+0000 7fbed9e27640 1 --2- 172.21.15.31:0/1389739754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4109a30 0x7fbed410be50 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009ca0 tx=0x7fbecc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.666+0000 7fbed9e27640 1 -- 172.21.15.31:0/1389739754 shutdown_connections 2024-04-25T16:01:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.666+0000 7fbed9e27640 1 --2- 172.21.15.31:0/1389739754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4109a30 0x7fbed410be50 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.666+0000 7fbed9e27640 1 --2- 172.21.15.31:0/1389739754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed4104d30 0x7fbed4105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.666+0000 7fbed9e27640 1 --2- 172.21.15.31:0/1389739754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4103af0 0x7fbed41047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.666+0000 7fbed9e27640 1 -- 172.21.15.31:0/1389739754 >> 172.21.15.31:0/1389739754 conn(0x7fbed40fece0 msgr2=0x7fbed4101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.666+0000 7fbed9e27640 1 -- 172.21.15.31:0/1389739754 shutdown_connections 2024-04-25T16:01:10.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.666+0000 7fbed9e27640 1 -- 172.21.15.31:0/1389739754 wait complete. 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed9e27640 1 Processor -- start 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed9e27640 1 -- start start 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed9e27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4103af0 0x7fbed4077960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed9e27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed4104d30 0x7fbed4076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed9e27640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4109a30 0x7fbed4076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed9e27640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbed410f620 con 0x7fbed4103af0 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed9e27640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbed410f4a0 con 0x7fbed4109a30 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed9e27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbed410f7a0 con 0x7fbed4104d30 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4103af0 0x7fbed4077960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4103af0 0x7fbed4077960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56206/0 (socket says 172.21.15.31:56206) 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.667+0000 7fbed37fe640 1 -- 172.21.15.31:0/3935931436 learned_addr learned my addr 172.21.15.31:0/3935931436 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed3fff640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4109a30 0x7fbed4076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed2ffd640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed4104d30 0x7fbed4076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed37fe640 1 -- 172.21.15.31:0/3935931436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed4104d30 msgr2=0x7fbed4076000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed37fe640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed4104d30 0x7fbed4076000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed37fe640 1 -- 172.21.15.31:0/3935931436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4109a30 msgr2=0x7fbed4076540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed37fe640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4109a30 0x7fbed4076540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed37fe640 1 -- 172.21.15.31:0/3935931436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed4076c70 con 0x7fbed4103af0 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed3fff640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4109a30 0x7fbed4076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed37fe640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4103af0 0x7fbed4077960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbec400e9f0 tx=0x7fbec400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed0ff9640 1 -- 172.21.15.31:0/3935931436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec400ccc0 con 0x7fbed4103af0 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed41a4290 con 0x7fbed4103af0 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed41a4780 con 0x7fbed4103af0 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed0ff9640 1 -- 172.21.15.31:0/3935931436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbec40035f0 con 0x7fbed4103af0 2024-04-25T16:01:10.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.668+0000 7fbed0ff9640 1 -- 172.21.15.31:0/3935931436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec400ce60 con 0x7fbed4103af0 2024-04-25T16:01:10.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.669+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed4108840 con 0x7fbed4103af0 2024-04-25T16:01:10.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.672+0000 7fbed0ff9640 1 -- 172.21.15.31:0/3935931436 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbec4003aa0 con 0x7fbed4103af0 2024-04-25T16:01:10.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.672+0000 7fbed0ff9640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbebc0afaf0 0x7fbebc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.672+0000 7fbed2ffd640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbebc0afaf0 0x7fbebc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.672+0000 7fbed0ff9640 1 -- 172.21.15.31:0/3935931436 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbec40df220 con 0x7fbed4103af0 2024-04-25T16:01:10.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.672+0000 7fbed2ffd640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbebc0afaf0 0x7fbebc0b2000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbec8005fd0 tx=0x7fbec8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:10.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.675+0000 7fbed0ff9640 1 -- 172.21.15.31:0/3935931436 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbec4053ff0 con 0x7fbed4103af0 2024-04-25T16:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.862+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbed402df10 con 0x7fbed4103af0 2024-04-25T16:01:10.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.863+0000 7fbed0ff9640 1 -- 172.21.15.31:0/3935931436 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7fbec40a7e10 con 0x7fbed4103af0 2024-04-25T16:01:10.865 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T16:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbebc0afaf0 msgr2=0x7fbebc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbebc0afaf0 0x7fbebc0b2000 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbec8005fd0 tx=0x7fbec8005850 comp rx=0 tx=0).stop 2024-04-25T16:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4103af0 msgr2=0x7fbed4077960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4103af0 0x7fbed4077960 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fbec400e9f0 tx=0x7fbec400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 shutdown_connections 2024-04-25T16:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbebc0afaf0 0x7fbebc0b2000 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbed4109a30 0x7fbed4076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbed4104d30 0x7fbed4076000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 --2- 172.21.15.31:0/3935931436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbed4103af0 0x7fbed4077960 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 >> 172.21.15.31:0/3935931436 conn(0x7fbed40fece0 msgr2=0x7fbed4105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:10.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 shutdown_connections 2024-04-25T16:01:10.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.865+0000 7fbed9e27640 1 -- 172.21.15.31:0/3935931436 wait complete. 2024-04-25T16:01:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:10 smithi063 ceph-mon[32413]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:01:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:10 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/286349367' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T16:01:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:10 smithi063 ceph-mon[32413]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:01:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:10 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/810397632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:10 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1053164338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:10.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:10 smithi031 ceph-mon[29140]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:01:10.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:10 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/286349367' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T16:01:10.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:10 smithi031 ceph-mon[29140]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:01:10.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:10 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/810397632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:10.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:10 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1053164338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:10.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:01:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:10 smithi186 ceph-mon[33258]: pgmap v347: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s 2024-04-25T16:01:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:10 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/286349367' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-04-25T16:01:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:10 smithi186 ceph-mon[33258]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:01:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:10 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/810397632' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:10.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:10 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1053164338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.988+0000 7f24a5bf3640 1 -- 172.21.15.31:0/2769498961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 msgr2=0x7f24a0105150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.988+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/2769498961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 0x7f24a0105150 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2494009900 tx=0x7f249402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.989+0000 7f24a5bf3640 1 -- 172.21.15.31:0/2769498961 shutdown_connections 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.989+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/2769498961 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24a0109a40 0x7f24a010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.989+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/2769498961 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 0x7f24a0105150 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.989+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/2769498961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24a0103b00 0x7f24a0104800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.989+0000 7f24a5bf3640 1 -- 172.21.15.31:0/2769498961 >> 172.21.15.31:0/2769498961 conn(0x7f24a00fed10 msgr2=0x7f24a01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.989+0000 7f24a5bf3640 1 -- 172.21.15.31:0/2769498961 shutdown_connections 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.989+0000 7f24a5bf3640 1 -- 172.21.15.31:0/2769498961 wait complete. 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.989+0000 7f24a5bf3640 1 Processor -- start 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.989+0000 7f24a5bf3640 1 -- start start 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f24a5bf3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24a0103b00 0x7f24a019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f24a5bf3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 0x7f24a019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f24a5bf3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24a0109a40 0x7f24a01a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f24a5bf3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f24a010f620 con 0x7f24a0104d40 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f24a5bf3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f24a010f4a0 con 0x7f24a0109a40 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f24a5bf3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f24a010f7a0 con 0x7f24a0103b00 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 0x7f24a019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 0x7f24a019a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56222/0 (socket says 172.21.15.31:56222) 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249effd640 1 -- 172.21.15.31:0/882905547 learned_addr learned my addr 172.21.15.31:0/882905547 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249ffff640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24a0109a40 0x7f24a01a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249f7fe640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24a0103b00 0x7f24a019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249effd640 1 -- 172.21.15.31:0/882905547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24a0103b00 msgr2=0x7f24a019a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249effd640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24a0103b00 0x7f24a019a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249effd640 1 -- 172.21.15.31:0/882905547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24a0109a40 msgr2=0x7f24a01a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249effd640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24a0109a40 0x7f24a01a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249effd640 1 -- 172.21.15.31:0/882905547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24a01a1da0 con 0x7f24a0104d40 2024-04-25T16:01:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249f7fe640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24a0103b00 0x7f24a019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249effd640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 0x7f24a019a7a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2494006010 tx=0x7f2494031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249cff9640 1 -- 172.21.15.31:0/882905547 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f249403f030 con 0x7f24a0104d40 2024-04-25T16:01:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f249cff9640 1 -- 172.21.15.31:0/882905547 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2494003aa0 con 0x7f24a0104d40 2024-04-25T16:01:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.990+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24a01a2030 con 0x7f24a0104d40 2024-04-25T16:01:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.991+0000 7f249cff9640 1 -- 172.21.15.31:0/882905547 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2494035070 con 0x7f24a0104d40 2024-04-25T16:01:10.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.991+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24a01a2520 con 0x7f24a0104d40 2024-04-25T16:01:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.992+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24a0108850 con 0x7f24a0104d40 2024-04-25T16:01:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.993+0000 7f249cff9640 1 -- 172.21.15.31:0/882905547 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2494042030 con 0x7f24a0104d40 2024-04-25T16:01:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.994+0000 7f249cff9640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24880afaf0 0x7f24880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.994+0000 7f249f7fe640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24880afaf0 0x7f24880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.994+0000 7f249cff9640 1 -- 172.21.15.31:0/882905547 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f24940ff3e0 con 0x7f24a0104d40 2024-04-25T16:01:10.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.994+0000 7f249f7fe640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24880afaf0 0x7f24880b2000 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f24a0104700 tx=0x7f2490005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:10.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:10.998+0000 7f249cff9640 1 -- 172.21.15.31:0/882905547 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f24940c7fd0 con 0x7f24a0104d40 2024-04-25T16:01:11.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.187+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24a00637a0 con 0x7f24a0104d40 2024-04-25T16:01:11.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.189+0000 7f249cff9640 1 -- 172.21.15.31:0/882905547 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f24940c78d0 con 0x7f24a0104d40 2024-04-25T16:01:11.191 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:11.191 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:01:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24880afaf0 msgr2=0x7f24880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24880afaf0 0x7f24880b2000 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f24a0104700 tx=0x7f2490005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 msgr2=0x7f24a019a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 0x7f24a019a7a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f2494006010 tx=0x7f2494031a50 comp rx=0 tx=0).stop 2024-04-25T16:01:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 shutdown_connections 2024-04-25T16:01:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f24880afaf0 0x7f24880b2000 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24a0109a40 0x7f24a01a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f24a0104d40 0x7f24a019a7a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 --2- 172.21.15.31:0/882905547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24a0103b00 0x7f24a019a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 >> 172.21.15.31:0/882905547 conn(0x7f24a00fed10 msgr2=0x7f24a0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 shutdown_connections 2024-04-25T16:01:11.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.191+0000 7f24a5bf3640 1 -- 172.21.15.31:0/882905547 wait complete. 2024-04-25T16:01:11.205 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:59:00.978038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:59:01.829535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:01:11.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:01:11.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 -- 172.21.15.31:0/2426674354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd58109890 msgr2=0x7fbd5810bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 --2- 172.21.15.31:0/2426674354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd58109890 0x7fbd5810bcb0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009900 tx=0x7fbd5402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 -- 172.21.15.31:0/2426674354 shutdown_connections 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 --2- 172.21.15.31:0/2426674354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd58109890 0x7fbd5810bcb0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 --2- 172.21.15.31:0/2426674354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd58106f30 0x7fbd58109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 --2- 172.21.15.31:0/2426674354 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd581045d0 0x7fbd581069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 -- 172.21.15.31:0/2426674354 >> 172.21.15.31:0/2426674354 conn(0x7fbd580fdbd0 msgr2=0x7fbd58100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 -- 172.21.15.31:0/2426674354 shutdown_connections 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 -- 172.21.15.31:0/2426674354 wait complete. 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 Processor -- start 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 -- start start 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd581045d0 0x7fbd5819a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd58106f30 0x7fbd5819aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd58109890 0x7fbd581a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5810f610 con 0x7fbd581045d0 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5810f490 con 0x7fbd58106f30 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd60496640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd5810f790 con 0x7fbd58109890 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5e232640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd581045d0 0x7fbd5819a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5e232640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd581045d0 0x7fbd5819a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56246/0 (socket says 172.21.15.31:56246) 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5e232640 1 -- 172.21.15.31:0/3399385418 learned_addr learned my addr 172.21.15.31:0/3399385418 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5da31640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd58106f30 0x7fbd5819aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5ea33640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd58109890 0x7fbd581a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5e232640 1 -- 172.21.15.31:0/3399385418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd58109890 msgr2=0x7fbd581a1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5e232640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd58109890 0x7fbd581a1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5e232640 1 -- 172.21.15.31:0/3399385418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd58106f30 msgr2=0x7fbd5819aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5e232640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd58106f30 0x7fbd5819aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5e232640 1 -- 172.21.15.31:0/3399385418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd581a2080 con 0x7fbd581045d0 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.319+0000 7fbd5e232640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd581045d0 0x7fbd5819a540 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000eb90 tx=0x7fbd5000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:11.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.320+0000 7fbd4f7fe640 1 -- 172.21.15.31:0/3399385418 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50010630 con 0x7fbd581045d0 2024-04-25T16:01:11.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.320+0000 7fbd4f7fe640 1 -- 172.21.15.31:0/3399385418 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd5000ee90 con 0x7fbd581045d0 2024-04-25T16:01:11.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.320+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd581a2370 con 0x7fbd581045d0 2024-04-25T16:01:11.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.320+0000 7fbd4f7fe640 1 -- 172.21.15.31:0/3399385418 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd50018ac0 con 0x7fbd581045d0 2024-04-25T16:01:11.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.320+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd581a28c0 con 0x7fbd581045d0 2024-04-25T16:01:11.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.321+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd5806c2a0 con 0x7fbd581045d0 2024-04-25T16:01:11.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.322+0000 7fbd4f7fe640 1 -- 172.21.15.31:0/3399385418 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbd50018c60 con 0x7fbd581045d0 2024-04-25T16:01:11.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.323+0000 7fbd4f7fe640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbd440afaf0 0x7fbd440b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.323+0000 7fbd5da31640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbd440afaf0 0x7fbd440b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.324+0000 7fbd4f7fe640 1 -- 172.21.15.31:0/3399385418 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbd500df650 con 0x7fbd581045d0 2024-04-25T16:01:11.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.324+0000 7fbd5da31640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbd440afaf0 0x7fbd440b2000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbd5819b5e0 tx=0x7fbd4800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:11.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.328+0000 7fbd4f7fe640 1 -- 172.21.15.31:0/3399385418 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbd500a7bf0 con 0x7fbd581045d0 2024-04-25T16:01:11.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.514+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbd581a2b20 con 0x7fbd581045d0 2024-04-25T16:01:11.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.515+0000 7fbd4f7fe640 1 -- 172.21.15.31:0/3399385418 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7fbd500b02c0 con 0x7fbd581045d0 2024-04-25T16:01:11.516 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbd440afaf0 msgr2=0x7fbd440b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbd440afaf0 0x7fbd440b2000 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fbd5819b5e0 tx=0x7fbd4800b390 comp rx=0 tx=0).stop 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd581045d0 msgr2=0x7fbd5819a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd581045d0 0x7fbd5819a540 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbd5000eb90 tx=0x7fbd5000c6a0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 shutdown_connections 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbd440afaf0 0x7fbd440b2000 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd58109890 0x7fbd581a1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd58106f30 0x7fbd5819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 --2- 172.21.15.31:0/3399385418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd581045d0 0x7fbd5819a540 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 >> 172.21.15.31:0/3399385418 conn(0x7fbd580fdbd0 msgr2=0x7fbd58105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 shutdown_connections 2024-04-25T16:01:11.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.517+0000 7fbd60496640 1 -- 172.21.15.31:0/3399385418 wait complete. 2024-04-25T16:01:11.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:01:11.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.644+0000 7fef834e0640 1 -- 172.21.15.31:0/1714124789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c108930 msgr2=0x7fef7c10ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.644+0000 7fef834e0640 1 --2- 172.21.15.31:0/1714124789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c108930 0x7fef7c10ad50 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fef74009ca0 tx=0x7fef7402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.644+0000 7fef834e0640 1 -- 172.21.15.31:0/1714124789 shutdown_connections 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.644+0000 7fef834e0640 1 --2- 172.21.15.31:0/1714124789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c108930 0x7fef7c10ad50 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.644+0000 7fef834e0640 1 --2- 172.21.15.31:0/1714124789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7c103c30 0x7fef7c104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.644+0000 7fef834e0640 1 --2- 172.21.15.31:0/1714124789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef7c1029f0 0x7fef7c1036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.644+0000 7fef834e0640 1 -- 172.21.15.31:0/1714124789 >> 172.21.15.31:0/1714124789 conn(0x7fef7c0fdbc0 msgr2=0x7fef7c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 -- 172.21.15.31:0/1714124789 shutdown_connections 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 -- 172.21.15.31:0/1714124789 wait complete. 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 Processor -- start 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 -- start start 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1029f0 0x7fef7c0ffa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7c103c30 0x7fef7c0fe0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef7c108930 0x7fef7c0fe630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef7c10e520 con 0x7fef7c1029f0 2024-04-25T16:01:11.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef7c10e3a0 con 0x7fef7c108930 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.645+0000 7fef834e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef7c10e6a0 con 0x7fef7c103c30 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef8127c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1029f0 0x7fef7c0ffa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef8127c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1029f0 0x7fef7c0ffa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56260/0 (socket says 172.21.15.31:56260) 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef8127c640 1 -- 172.21.15.31:0/188066102 learned_addr learned my addr 172.21.15.31:0/188066102 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef81a7d640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef7c108930 0x7fef7c0fe630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef80a7b640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7c103c30 0x7fef7c0fe0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef8127c640 1 -- 172.21.15.31:0/188066102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7c103c30 msgr2=0x7fef7c0fe0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef8127c640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7c103c30 0x7fef7c0fe0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef8127c640 1 -- 172.21.15.31:0/188066102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef7c108930 msgr2=0x7fef7c0fe630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef8127c640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef7c108930 0x7fef7c0fe630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef8127c640 1 -- 172.21.15.31:0/188066102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef7c0fed60 con 0x7fef7c1029f0 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef81a7d640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef7c108930 0x7fef7c0fe630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef80a7b640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7c103c30 0x7fef7c0fe0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef8127c640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1029f0 0x7fef7c0ffa50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fef78005fd0 tx=0x7fef7800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:11.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef727fc640 1 -- 172.21.15.31:0/188066102 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78017070 con 0x7fef7c1029f0 2024-04-25T16:01:11.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef727fc640 1 -- 172.21.15.31:0/188066102 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef78009c30 con 0x7fef7c1029f0 2024-04-25T16:01:11.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef7c1a42a0 con 0x7fef7c1029f0 2024-04-25T16:01:11.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef727fc640 1 -- 172.21.15.31:0/188066102 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef78003d10 con 0x7fef7c1029f0 2024-04-25T16:01:11.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.646+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef7c1a4760 con 0x7fef7c1029f0 2024-04-25T16:01:11.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.647+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef7c06a160 con 0x7fef7c1029f0 2024-04-25T16:01:11.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.650+0000 7fef727fc640 1 -- 172.21.15.31:0/188066102 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fef78003eb0 con 0x7fef7c1029f0 2024-04-25T16:01:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.650+0000 7fef727fc640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef680afaf0 0x7fef680b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.651+0000 7fef80a7b640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef680afaf0 0x7fef680b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.651+0000 7fef727fc640 1 -- 172.21.15.31:0/188066102 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fef780df110 con 0x7fef7c1029f0 2024-04-25T16:01:11.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.651+0000 7fef80a7b640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef680afaf0 0x7fef680b2000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0ff3f0 tx=0x7fef6c005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:11.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.654+0000 7fef727fc640 1 -- 172.21.15.31:0/188066102 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef78020030 con 0x7fef7c1029f0 2024-04-25T16:01:11.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.841+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fef7c1a4a80 con 0x7fef7c1029f0 2024-04-25T16:01:11.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.841+0000 7fef727fc640 1 -- 172.21.15.31:0/188066102 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7fef78009de0 con 0x7fef7c1029f0 2024-04-25T16:01:11.842 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef680afaf0 msgr2=0x7fef680b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef680afaf0 0x7fef680b2000 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fef7c0ff3f0 tx=0x7fef6c005e50 comp rx=0 tx=0).stop 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1029f0 msgr2=0x7fef7c0ffa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1029f0 0x7fef7c0ffa50 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fef78005fd0 tx=0x7fef7800dde0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 shutdown_connections 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef680afaf0 0x7fef680b2000 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef7c108930 0x7fef7c0fe630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7c103c30 0x7fef7c0fe0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 --2- 172.21.15.31:0/188066102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef7c1029f0 0x7fef7c0ffa50 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 >> 172.21.15.31:0/188066102 conn(0x7fef7c0fdbc0 msgr2=0x7fef7c109d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:11.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 shutdown_connections 2024-04-25T16:01:11.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.844+0000 7fef834e0640 1 -- 172.21.15.31:0/188066102 wait complete. 2024-04-25T16:01:11.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:11 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1117106736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:11.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:11 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3935931436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:11.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:11 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/882905547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:11.856 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:01:11.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:11 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1117106736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:11.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:11 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3935931436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:11.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:11 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/882905547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:11.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:11 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1117106736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:11.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:11 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3935931436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:11.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:11 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/882905547' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 -- 172.21.15.31:0/731423782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960109a30 msgr2=0x7f896010be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 --2- 172.21.15.31:0/731423782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960109a30 0x7f896010be50 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8958009ca0 tx=0x7f895802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 -- 172.21.15.31:0/731423782 shutdown_connections 2024-04-25T16:01:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 --2- 172.21.15.31:0/731423782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960109a30 0x7f896010be50 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 --2- 172.21.15.31:0/731423782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960104d30 0x7f8960105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 --2- 172.21.15.31:0/731423782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8960103af0 0x7f89601047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 -- 172.21.15.31:0/731423782 >> 172.21.15.31:0/731423782 conn(0x7f89600fece0 msgr2=0x7f8960101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 -- 172.21.15.31:0/731423782 shutdown_connections 2024-04-25T16:01:11.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 -- 172.21.15.31:0/731423782 wait complete. 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.971+0000 7f8964f27640 1 Processor -- start 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f8964f27640 1 -- start start 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f8964f27640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8960103af0 0x7f896019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f8964f27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960104d30 0x7f896019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f8964f27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960109a30 0x7f89601a1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f8964f27640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f896010f620 con 0x7f8960104d30 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f8964f27640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f896010f4a0 con 0x7f8960103af0 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f8964f27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f896010f7a0 con 0x7f8960109a30 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960104d30 0x7f896019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960104d30 0x7f896019a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56278/0 (socket says 172.21.15.31:56278) 2024-04-25T16:01:11.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895dd9b640 1 -- 172.21.15.31:0/43483836 learned_addr learned my addr 172.21.15.31:0/43483836 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895e59c640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8960103af0 0x7f896019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895ed9d640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960109a30 0x7f89601a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895dd9b640 1 -- 172.21.15.31:0/43483836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960109a30 msgr2=0x7f89601a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895dd9b640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960109a30 0x7f89601a1680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895dd9b640 1 -- 172.21.15.31:0/43483836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8960103af0 msgr2=0x7f896019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895dd9b640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8960103af0 0x7f896019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895dd9b640 1 -- 172.21.15.31:0/43483836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89601a1cf0 con 0x7f8960104d30 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895e59c640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8960103af0 0x7f896019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.972+0000 7f895dd9b640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960104d30 0x7f896019a6f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f895400e9f0 tx=0x7f895400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.973+0000 7f894f7fe640 1 -- 172.21.15.31:0/43483836 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895400cd80 con 0x7f8960104d30 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.973+0000 7f894f7fe640 1 -- 172.21.15.31:0/43483836 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89540054d0 con 0x7f8960104d30 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.973+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89601a1fe0 con 0x7f8960104d30 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.973+0000 7f894f7fe640 1 -- 172.21.15.31:0/43483836 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8954010690 con 0x7f8960104d30 2024-04-25T16:01:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.973+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89601a2440 con 0x7f8960104d30 2024-04-25T16:01:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.974+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8960108840 con 0x7f8960104d30 2024-04-25T16:01:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.975+0000 7f894f7fe640 1 -- 172.21.15.31:0/43483836 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8954004e30 con 0x7f8960104d30 2024-04-25T16:01:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.976+0000 7f894f7fe640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89480afaf0 0x7f89480b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.976+0000 7f895e59c640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89480afaf0 0x7f89480b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.977+0000 7f894f7fe640 1 -- 172.21.15.31:0/43483836 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f89540df370 con 0x7f8960104d30 2024-04-25T16:01:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.977+0000 7f895e59c640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89480afaf0 0x7f89480b2000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f89601046f0 tx=0x7f8950005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:11.980+0000 7f894f7fe640 1 -- 172.21.15.31:0/43483836 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8954053ff0 con 0x7f8960104d30 2024-04-25T16:01:12.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.168+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f896002dce0 con 0x7f8960104d30 2024-04-25T16:01:12.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.168+0000 7f894f7fe640 1 -- 172.21.15.31:0/43483836 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7f89540a7f60 con 0x7f8960104d30 2024-04-25T16:01:12.169 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T16:01:12.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89480afaf0 msgr2=0x7f89480b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89480afaf0 0x7f89480b2000 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f89601046f0 tx=0x7f8950005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:12.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960104d30 msgr2=0x7f896019a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960104d30 0x7f896019a6f0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f895400e9f0 tx=0x7f895400edf0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 shutdown_connections 2024-04-25T16:01:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89480afaf0 0x7f89480b2000 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960109a30 0x7f89601a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8960104d30 0x7f896019a6f0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 --2- 172.21.15.31:0/43483836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8960103af0 0x7f896019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 >> 172.21.15.31:0/43483836 conn(0x7f89600fece0 msgr2=0x7f8960105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 shutdown_connections 2024-04-25T16:01:12.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.170+0000 7f8964f27640 1 -- 172.21.15.31:0/43483836 wait complete. 2024-04-25T16:01:12.183 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-25T16:01:12.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:01:12.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.294+0000 7f2d8e796640 1 -- 172.21.15.31:0/557652468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88104450 msgr2=0x7f2d8810aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.294+0000 7f2d8e796640 1 --2- 172.21.15.31:0/557652468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88104450 0x7f2d8810aba0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009ca0 tx=0x7f2d8002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.294+0000 7f2d8e796640 1 -- 172.21.15.31:0/557652468 shutdown_connections 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.294+0000 7f2d8e796640 1 --2- 172.21.15.31:0/557652468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88104450 0x7f2d8810aba0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.294+0000 7f2d8e796640 1 --2- 172.21.15.31:0/557652468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d88103930 0x7f2d88103dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.294+0000 7f2d8e796640 1 --2- 172.21.15.31:0/557652468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d88102f50 0x7f2d88103360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.294+0000 7f2d8e796640 1 -- 172.21.15.31:0/557652468 >> 172.21.15.31:0/557652468 conn(0x7f2d880fd330 msgr2=0x7f2d880ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.294+0000 7f2d8e796640 1 -- 172.21.15.31:0/557652468 shutdown_connections 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.294+0000 7f2d8e796640 1 -- 172.21.15.31:0/557652468 wait complete. 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d8e796640 1 Processor -- start 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d8e796640 1 -- start start 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d8e796640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88102f50 0x7f2d8810d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d8e796640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d88103930 0x7f2d8810b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d8e796640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d88104450 0x7f2d8810bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:12.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d8e796640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810f620 con 0x7f2d88102f50 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d8e796640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810f4a0 con 0x7f2d88103930 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d8e796640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2d8810f7a0 con 0x7f2d88104450 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d87fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88102f50 0x7f2d8810d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d87fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88102f50 0x7f2d8810d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56300/0 (socket says 172.21.15.31:56300) 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.295+0000 7f2d87fff640 1 -- 172.21.15.31:0/2576856330 learned_addr learned my addr 172.21.15.31:0/2576856330 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d877fe640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d88103930 0x7f2d8810b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d8cd33640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d88104450 0x7f2d8810bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d87fff640 1 -- 172.21.15.31:0/2576856330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d88104450 msgr2=0x7f2d8810bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d87fff640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d88104450 0x7f2d8810bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d87fff640 1 -- 172.21.15.31:0/2576856330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d88103930 msgr2=0x7f2d8810b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d87fff640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d88103930 0x7f2d8810b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d87fff640 1 -- 172.21.15.31:0/2576856330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d8819dff0 con 0x7f2d88102f50 2024-04-25T16:01:12.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d877fe640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d88103930 0x7f2d8810b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:12.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d87fff640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88102f50 0x7f2d8810d2c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800d770 tx=0x7f2d7800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:12.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d857fa640 1 -- 172.21.15.31:0/2576856330 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d78017070 con 0x7f2d88102f50 2024-04-25T16:01:12.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d857fa640 1 -- 172.21.15.31:0/2576856330 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d78009c30 con 0x7f2d88102f50 2024-04-25T16:01:12.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d8810c500 con 0x7f2d88102f50 2024-04-25T16:01:12.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d857fa640 1 -- 172.21.15.31:0/2576856330 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d7800f620 con 0x7f2d88102f50 2024-04-25T16:01:12.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.296+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d881a8bf0 con 0x7f2d88102f50 2024-04-25T16:01:12.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.297+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d88109900 con 0x7f2d88102f50 2024-04-25T16:01:12.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.299+0000 7f2d857fa640 1 -- 172.21.15.31:0/2576856330 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2d7801a070 con 0x7f2d88102f50 2024-04-25T16:01:12.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.299+0000 7f2d857fa640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d700afb70 0x7f2d700b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:12.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.299+0000 7f2d877fe640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d700afb70 0x7f2d700b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:12.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.299+0000 7f2d857fa640 1 -- 172.21.15.31:0/2576856330 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2d780de7f0 con 0x7f2d88102f50 2024-04-25T16:01:12.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.300+0000 7f2d877fe640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d700afb70 0x7f2d700b2080 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c005fd0 tx=0x7f2d7c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:12.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.304+0000 7f2d857fa640 1 -- 172.21.15.31:0/2576856330 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d78009de0 con 0x7f2d88102f50 2024-04-25T16:01:12.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.531+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2d8802dd70 con 0x7f2d88102f50 2024-04-25T16:01:12.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.532+0000 7f2d857fa640 1 -- 172.21.15.31:0/2576856330 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11917 (secure 0 0 0) 0x7f2d780a7380 con 0x7f2d88102f50 2024-04-25T16:01:12.533 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:12.533 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:01:09.454389+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24535,24521,24563,14631,24553]},"id":1}]} 2024-04-25T16:01:12.533 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-25T16:01:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d700afb70 msgr2=0x7f2d700b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d700afb70 0x7f2d700b2080 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c005fd0 tx=0x7f2d7c005850 comp rx=0 tx=0).stop 2024-04-25T16:01:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88102f50 msgr2=0x7f2d8810d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88102f50 0x7f2d8810d2c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f2d7800d770 tx=0x7f2d7800dc30 comp rx=0 tx=0).stop 2024-04-25T16:01:12.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 shutdown_connections 2024-04-25T16:01:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2d700afb70 0x7f2d700b2080 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d88104450 0x7f2d8810bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d88103930 0x7f2d8810b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 --2- 172.21.15.31:0/2576856330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2d88102f50 0x7f2d8810d2c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 >> 172.21.15.31:0/2576856330 conn(0x7f2d880fd330 msgr2=0x7f2d88106ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 shutdown_connections 2024-04-25T16:01:12.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.534+0000 7f2d8e796640 1 -- 172.21.15.31:0/2576856330 wait complete. 2024-04-25T16:01:12.547 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-25T16:01:12.547 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-25T16:01:12.547 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:01:12.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.658+0000 7fe6fef84640 1 -- 172.21.15.31:0/2866590792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f8106f30 msgr2=0x7fe6f8109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.658+0000 7fe6fef84640 1 --2- 172.21.15.31:0/2866590792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f8106f30 0x7fe6f8109350 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4009900 tx=0x7fe6e402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:12.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.658+0000 7fe6fef84640 1 -- 172.21.15.31:0/2866590792 shutdown_connections 2024-04-25T16:01:12.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.658+0000 7fe6fef84640 1 --2- 172.21.15.31:0/2866590792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6f8109890 0x7fe6f810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.658+0000 7fe6fef84640 1 --2- 172.21.15.31:0/2866590792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f8106f30 0x7fe6f8109350 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.658+0000 7fe6fef84640 1 --2- 172.21.15.31:0/2866590792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6f81045d0 0x7fe6f81069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.658+0000 7fe6fef84640 1 -- 172.21.15.31:0/2866590792 >> 172.21.15.31:0/2866590792 conn(0x7fe6f80fdbd0 msgr2=0x7fe6f8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.658+0000 7fe6fef84640 1 -- 172.21.15.31:0/2866590792 shutdown_connections 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.658+0000 7fe6fef84640 1 -- 172.21.15.31:0/2866590792 wait complete. 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fef84640 1 Processor -- start 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fef84640 1 -- start start 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fef84640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f81045d0 0x7fe6f819c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fef84640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6f8106f30 0x7fe6f819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fef84640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6f8109890 0x7fe6f81a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fef84640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f810f610 con 0x7fe6f81045d0 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fef84640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f810f490 con 0x7fe6f8109890 2024-04-25T16:01:12.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fef84640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f810f790 con 0x7fe6f8106f30 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fcd20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f81045d0 0x7fe6f819c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fcd20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f81045d0 0x7fe6f819c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56316/0 (socket says 172.21.15.31:56316) 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.659+0000 7fe6fcd20640 1 -- 172.21.15.31:0/932914703 learned_addr learned my addr 172.21.15.31:0/932914703 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6f7fff640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6f8106f30 0x7fe6f819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fd521640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6f8109890 0x7fe6f81a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fcd20640 1 -- 172.21.15.31:0/932914703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6f8106f30 msgr2=0x7fe6f819ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fcd20640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6f8106f30 0x7fe6f819ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fcd20640 1 -- 172.21.15.31:0/932914703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6f8109890 msgr2=0x7fe6f81a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fcd20640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6f8109890 0x7fe6f81a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fcd20640 1 -- 172.21.15.31:0/932914703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f81a4020 con 0x7fe6f81045d0 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6f7fff640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6f8106f30 0x7fe6f819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fd521640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6f8109890 0x7fe6f81a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fcd20640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f81045d0 0x7fe6f819c4e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00d740 tx=0x7fe6ec00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:12.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6f5ffb640 1 -- 172.21.15.31:0/932914703 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec017070 con 0x7fe6f81045d0 2024-04-25T16:01:12.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6f5ffb640 1 -- 172.21.15.31:0/932914703 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6ec00de50 con 0x7fe6f81045d0 2024-04-25T16:01:12.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6f81a4310 con 0x7fe6f81045d0 2024-04-25T16:01:12.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6f5ffb640 1 -- 172.21.15.31:0/932914703 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec004ce0 con 0x7fe6f81045d0 2024-04-25T16:01:12.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.660+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6f81a4810 con 0x7fe6f81045d0 2024-04-25T16:01:12.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.661+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6f81a4a70 con 0x7fe6f81045d0 2024-04-25T16:01:12.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.663+0000 7fe6f5ffb640 1 -- 172.21.15.31:0/932914703 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe6ec004e80 con 0x7fe6f81045d0 2024-04-25T16:01:12.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.663+0000 7fe6f5ffb640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6e80afaf0 0x7fe6e80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:12.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.664+0000 7fe6f7fff640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6e80afaf0 0x7fe6e80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:12.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.664+0000 7fe6f5ffb640 1 -- 172.21.15.31:0/932914703 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe6ec0dea60 con 0x7fe6f81045d0 2024-04-25T16:01:12.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.664+0000 7fe6f7fff640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6e80afaf0 0x7fe6e80b2000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4000c00 tx=0x7fe6e4005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:12.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.667+0000 7fe6f5ffb640 1 -- 172.21.15.31:0/932914703 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe6ec0a7650 con 0x7fe6f81045d0 2024-04-25T16:01:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.849+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe6f81028b0 con 0x7fe6e80afaf0 2024-04-25T16:01:12.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.856+0000 7fe6f5ffb640 1 -- 172.21.15.31:0/932914703 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259069 (secure 0 0 0) 0x7fe6f81028b0 con 0x7fe6e80afaf0 2024-04-25T16:01:12.860 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:12.861 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6e80afaf0 msgr2=0x7fe6e80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6e80afaf0 0x7fe6e80b2000 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe6e4000c00 tx=0x7fe6e4005d50 comp rx=0 tx=0).stop 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f81045d0 msgr2=0x7fe6f819c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f81045d0 0x7fe6f819c4e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00d740 tx=0x7fe6ec00dc00 comp rx=0 tx=0).stop 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 shutdown_connections 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe6e80afaf0 0x7fe6e80b2000 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6f8109890 0x7fe6f81a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6f8106f30 0x7fe6f819ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 --2- 172.21.15.31:0/932914703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe6f81045d0 0x7fe6f819c4e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 >> 172.21.15.31:0/932914703 conn(0x7fe6f80fdbd0 msgr2=0x7fe6f81021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 shutdown_connections 2024-04-25T16:01:12.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:12.859+0000 7fe6fef84640 1 -- 172.21.15.31:0/932914703 wait complete. 2024-04-25T16:01:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:12 smithi063 ceph-mon[32413]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:01:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:12 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3399385418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:12 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/188066102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:12 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/43483836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:12.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:12 smithi031 ceph-mon[29140]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:01:12.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:12 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3399385418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:12.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:12 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/188066102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:12.871 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:12 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/43483836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:12.872 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-04-25T16:01:11.347617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654990,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1256,"num_read_kb":746,"num_write":367,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2953032,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":804,"ondisk_log_size":804,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355336,"kb_used_data":10344,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124586488,"statfs":{"total":1151940427776,"available":1151576563712,"internally_reserved":0,"allocated":10592256,"data_stored":6775626,"data_compressed":113085,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003064"},"pg_stats":[{"pgid":"3.3f","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354813+0000","last_change":"2024-04-25T15:59:02.997678+0000","last_active":"2024-04-25T16:00:19.354813+0000","last_peered":"2024-04-25T16:00:19.354813+0000","last_clean":"2024-04-25T16:00:19.354813+0000","last_became_active":"2024-04-25T15:59:02.997335+0000","last_became_peered":"2024-04-25T15:59:02.997335+0000","last_unstale":"2024-04-25T16:00:19.354813+0000","last_undegraded":"2024-04-25T16:00:19.354813+0000","last_fullsized":"2024-04-25T16:00:19.354813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:33.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"77'4","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351614+0000","last_change":"2024-04-25T15:59:03.001049+0000","last_active":"2024-04-25T16:00:19.351614+0000","last_peered":"2024-04-25T16:00:19.351614+0000","last_clean":"2024-04-25T16:00:19.351614+0000","last_became_active":"2024-04-25T15:59:03.000663+0000","last_became_peered":"2024-04-25T15:59:03.000663+0000","last_unstale":"2024-04-25T16:00:19.351614+0000","last_undegraded":"2024-04-25T16:00:19.351614+0000","last_fullsized":"2024-04-25T16:00:19.351614+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:04:36.670519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"77'7","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354238+0000","last_change":"2024-04-25T15:59:03.003861+0000","last_active":"2024-04-25T16:00:19.354238+0000","last_peered":"2024-04-25T16:00:19.354238+0000","last_clean":"2024-04-25T16:00:19.354238+0000","last_became_active":"2024-04-25T15:59:03.003508+0000","last_became_peered":"2024-04-25T15:59:03.003508+0000","last_unstale":"2024-04-25T16:00:19.354238+0000","last_undegraded":"2024-04-25T16:00:19.354238+0000","last_fullsized":"2024-04-25T16:00:19.354238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:25.679389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337369+0000","last_change":"2024-04-25T15:59:03.003330+0000","last_active":"2024-04-25T16:00:19.337369+0000","last_peered":"2024-04-25T16:00:19.337369+0000","last_clean":"2024-04-25T16:00:19.337369+0000","last_became_active":"2024-04-25T15:59:03.002872+0000","last_became_peered":"2024-04-25T15:59:03.002872+0000","last_unstale":"2024-04-25T16:00:19.337369+0000","last_undegraded":"2024-04-25T16:00:19.337369+0000","last_fullsized":"2024-04-25T16:00:19.337369+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:02.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.324906+0000","last_change":"2024-04-25T15:59:02.997684+0000","last_active":"2024-04-25T16:00:19.324906+0000","last_peered":"2024-04-25T16:00:19.324906+0000","last_clean":"2024-04-25T16:00:19.324906+0000","last_became_active":"2024-04-25T15:59:02.997370+0000","last_became_peered":"2024-04-25T15:59:02.997370+0000","last_unstale":"2024-04-25T16:00:19.324906+0000","last_undegraded":"2024-04-25T16:00:19.324906+0000","last_fullsized":"2024-04-25T16:00:19.324906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:24.349085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348756+0000","last_change":"2024-04-25T15:59:03.003638+0000","last_active":"2024-04-25T16:00:19.348756+0000","last_peered":"2024-04-25T16:00:19.348756+0000","last_clean":"2024-04-25T16:00:19.348756+0000","last_became_active":"2024-04-25T15:59:03.003299+0000","last_became_peered":"2024-04-25T15:59:03.003299+0000","last_unstale":"2024-04-25T16:00:19.348756+0000","last_undegraded":"2024-04-25T16:00:19.348756+0000","last_fullsized":"2024-04-25T16:00:19.348756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:46.842345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.338851+0000","last_change":"2024-04-25T15:59:02.998591+0000","last_active":"2024-04-25T16:00:19.338851+0000","last_peered":"2024-04-25T16:00:19.338851+0000","last_clean":"2024-04-25T16:00:19.338851+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.338851+0000","last_undegraded":"2024-04-25T16:00:19.338851+0000","last_fullsized":"2024-04-25T16:00:19.338851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.444499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343142+0000","last_change":"2024-04-25T15:59:03.006225+0000","last_active":"2024-04-25T16:00:19.343142+0000","last_peered":"2024-04-25T16:00:19.343142+0000","last_clean":"2024-04-25T16:00:19.343142+0000","last_became_active":"2024-04-25T15:59:03.005883+0000","last_became_peered":"2024-04-25T15:59:03.005883+0000","last_unstale":"2024-04-25T16:00:19.343142+0000","last_undegraded":"2024-04-25T16:00:19.343142+0000","last_fullsized":"2024-04-25T16:00:19.343142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:17.773196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359702+0000","last_change":"2024-04-25T15:59:03.003605+0000","last_active":"2024-04-25T16:00:19.359702+0000","last_peered":"2024-04-25T16:00:19.359702+0000","last_clean":"2024-04-25T16:00:19.359702+0000","last_became_active":"2024-04-25T15:59:03.003271+0000","last_became_peered":"2024-04-25T15:59:03.003271+0000","last_unstale":"2024-04-25T16:00:19.359702+0000","last_undegraded":"2024-04-25T16:00:19.359702+0000","last_fullsized":"2024-04-25T16:00:19.359702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:28.576407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'16","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348846+0000","last_change":"2024-04-25T15:59:50.679099+0000","last_active":"2024-04-25T16:00:19.348846+0000","last_peered":"2024-04-25T16:00:19.348846+0000","last_clean":"2024-04-25T16:00:19.348846+0000","last_became_active":"2024-04-25T15:59:50.678558+0000","last_became_peered":"2024-04-25T15:59:50.678558+0000","last_unstale":"2024-04-25T16:00:19.348846+0000","last_undegraded":"2024-04-25T16:00:19.348846+0000","last_fullsized":"2024-04-25T16:00:19.348846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:40.691050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359322+0000","last_change":"2024-04-25T15:59:02.999307+0000","last_active":"2024-04-25T16:00:19.359322+0000","last_peered":"2024-04-25T16:00:19.359322+0000","last_clean":"2024-04-25T16:00:19.359322+0000","last_became_active":"2024-04-25T15:59:02.998976+0000","last_became_peered":"2024-04-25T15:59:02.998976+0000","last_unstale":"2024-04-25T16:00:19.359322+0000","last_undegraded":"2024-04-25T16:00:19.359322+0000","last_fullsized":"2024-04-25T16:00:19.359322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:29.626440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347700+0000","last_change":"2024-04-25T15:59:03.005208+0000","last_active":"2024-04-25T16:00:19.347700+0000","last_peered":"2024-04-25T16:00:19.347700+0000","last_clean":"2024-04-25T16:00:19.347700+0000","last_became_active":"2024-04-25T15:59:03.004869+0000","last_became_peered":"2024-04-25T15:59:03.004869+0000","last_unstale":"2024-04-25T16:00:19.347700+0000","last_undegraded":"2024-04-25T16:00:19.347700+0000","last_fullsized":"2024-04-25T16:00:19.347700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:00:24.040763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"77'10","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349448+0000","last_change":"2024-04-25T15:59:50.673556+0000","last_active":"2024-04-25T16:00:19.349448+0000","last_peered":"2024-04-25T16:00:19.349448+0000","last_clean":"2024-04-25T16:00:19.349448+0000","last_became_active":"2024-04-25T15:59:50.673119+0000","last_became_peered":"2024-04-25T15:59:50.673119+0000","last_unstale":"2024-04-25T16:00:19.349448+0000","last_undegraded":"2024-04-25T16:00:19.349448+0000","last_fullsized":"2024-04-25T16:00:19.349448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:09:50.286968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350359+0000","last_change":"2024-04-25T15:59:03.002160+0000","last_active":"2024-04-25T16:00:19.350359+0000","last_peered":"2024-04-25T16:00:19.350359+0000","last_clean":"2024-04-25T16:00:19.350359+0000","last_became_active":"2024-04-25T15:59:03.001855+0000","last_became_peered":"2024-04-25T15:59:03.001855+0000","last_unstale":"2024-04-25T16:00:19.350359+0000","last_undegraded":"2024-04-25T16:00:19.350359+0000","last_fullsized":"2024-04-25T16:00:19.350359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:43.383851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350513+0000","last_change":"2024-04-25T15:59:02.995318+0000","last_active":"2024-04-25T16:00:19.350513+0000","last_peered":"2024-04-25T16:00:19.350513+0000","last_clean":"2024-04-25T16:00:19.350513+0000","last_became_active":"2024-04-25T15:59:02.994956+0000","last_became_peered":"2024-04-25T15:59:02.994956+0000","last_unstale":"2024-04-25T16:00:19.350513+0000","last_undegraded":"2024-04-25T16:00:19.350513+0000","last_fullsized":"2024-04-25T16:00:19.350513+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:59.659400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359091+0000","last_change":"2024-04-25T15:59:02.996485+0000","last_active":"2024-04-25T16:00:19.359091+0000","last_peered":"2024-04-25T16:00:19.359091+0000","last_clean":"2024-04-25T16:00:19.359091+0000","last_became_active":"2024-04-25T15:59:02.996011+0000","last_became_peered":"2024-04-25T15:59:02.996011+0000","last_unstale":"2024-04-25T16:00:19.359091+0000","last_undegraded":"2024-04-25T16:00:19.359091+0000","last_fullsized":"2024-04-25T16:00:19.359091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:56.025254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.355086+0000","last_change":"2024-04-25T15:59:02.998455+0000","last_active":"2024-04-25T16:00:19.355086+0000","last_peered":"2024-04-25T16:00:19.355086+0000","last_clean":"2024-04-25T16:00:19.355086+0000","last_became_active":"2024-04-25T15:59:02.998126+0000","last_became_peered":"2024-04-25T15:59:02.998126+0000","last_unstale":"2024-04-25T16:00:19.355086+0000","last_undegraded":"2024-04-25T16:00:19.355086+0000","last_fullsized":"2024-04-25T16:00:19.355086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:20.342413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350929+0000","last_change":"2024-04-25T15:59:02.998766+0000","last_active":"2024-04-25T16:00:19.350929+0000","last_peered":"2024-04-25T16:00:19.350929+0000","last_clean":"2024-04-25T16:00:19.350929+0000","last_became_active":"2024-04-25T15:59:02.998415+0000","last_became_peered":"2024-04-25T15:59:02.998415+0000","last_unstale":"2024-04-25T16:00:19.350929+0000","last_undegraded":"2024-04-25T16:00:19.350929+0000","last_fullsized":"2024-04-25T16:00:19.350929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:55:05.717847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340892+0000","last_change":"2024-04-25T15:59:02.994982+0000","last_active":"2024-04-25T16:00:19.340892+0000","last_peered":"2024-04-25T16:00:19.340892+0000","last_clean":"2024-04-25T16:00:19.340892+0000","last_became_active":"2024-04-25T15:59:02.993928+0000","last_became_peered":"2024-04-25T15:59:02.993928+0000","last_unstale":"2024-04-25T16:00:19.340892+0000","last_undegraded":"2024-04-25T16:00:19.340892+0000","last_fullsized":"2024-04-25T16:00:19.340892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:22:25.346541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'13","reported_seq":153,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:08.445335+0000","last_change":"2024-04-25T15:59:02.512828+0000","last_active":"2024-04-25T16:01:08.445335+0000","last_peered":"2024-04-25T16:01:08.445335+0000","last_clean":"2024-04-25T16:01:08.445335+0000","last_became_active":"2024-04-25T15:59:02.512291+0000","last_became_peered":"2024-04-25T15:59:02.512291+0000","last_unstale":"2024-04-25T16:01:08.445335+0000","last_undegraded":"2024-04-25T16:01:08.445335+0000","last_fullsized":"2024-04-25T16:01:08.445335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:02.372140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"77'5","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340094+0000","last_change":"2024-04-25T15:59:03.008472+0000","last_active":"2024-04-25T16:00:19.340094+0000","last_peered":"2024-04-25T16:00:19.340094+0000","last_clean":"2024-04-25T16:00:19.340094+0000","last_became_active":"2024-04-25T15:59:03.008108+0000","last_became_peered":"2024-04-25T15:59:03.008108+0000","last_unstale":"2024-04-25T16:00:19.340094+0000","last_undegraded":"2024-04-25T16:00:19.340094+0000","last_fullsized":"2024-04-25T16:00:19.340094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:40.087592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'2","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807946+0000","last_change":"2024-04-25T15:59:02.517846+0000","last_active":"2024-04-25T15:59:50.807946+0000","last_peered":"2024-04-25T15:59:50.807946+0000","last_clean":"2024-04-25T15:59:50.807946+0000","last_became_active":"2024-04-25T15:59:02.517366+0000","last_became_peered":"2024-04-25T15:59:02.517366+0000","last_unstale":"2024-04-25T15:59:50.807946+0000","last_undegraded":"2024-04-25T15:59:50.807946+0000","last_fullsized":"2024-04-25T15:59:50.807946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:29.620827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'6","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354259+0000","last_change":"2024-04-25T15:59:03.005202+0000","last_active":"2024-04-25T16:00:19.354259+0000","last_peered":"2024-04-25T16:00:19.354259+0000","last_clean":"2024-04-25T16:00:19.354259+0000","last_became_active":"2024-04-25T15:59:03.003447+0000","last_became_peered":"2024-04-25T15:59:03.003447+0000","last_unstale":"2024-04-25T16:00:19.354259+0000","last_undegraded":"2024-04-25T16:00:19.354259+0000","last_fullsized":"2024-04-25T16:00:19.354259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.740016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'12","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.703214+0000","last_change":"2024-04-25T15:59:50.702764+0000","last_active":"2024-04-25T15:59:50.703214+0000","last_peered":"2024-04-25T15:59:50.703214+0000","last_clean":"2024-04-25T15:59:50.703214+0000","last_became_active":"2024-04-25T15:59:50.679231+0000","last_became_peered":"2024-04-25T15:59:50.679231+0000","last_unstale":"2024-04-25T15:59:50.703214+0000","last_undegraded":"2024-04-25T15:59:50.703214+0000","last_fullsized":"2024-04-25T15:59:50.703214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:37.561992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360223+0000","last_change":"2024-04-25T15:59:03.004210+0000","last_active":"2024-04-25T16:00:19.360223+0000","last_peered":"2024-04-25T16:00:19.360223+0000","last_clean":"2024-04-25T16:00:19.360223+0000","last_became_active":"2024-04-25T15:59:03.002769+0000","last_became_peered":"2024-04-25T15:59:03.002769+0000","last_unstale":"2024-04-25T16:00:19.360223+0000","last_undegraded":"2024-04-25T16:00:19.360223+0000","last_fullsized":"2024-04-25T16:00:19.360223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:50.608327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669960+0000","last_change":"2024-04-25T15:59:02.506199+0000","last_active":"2024-04-25T15:59:50.669960+0000","last_peered":"2024-04-25T15:59:50.669960+0000","last_clean":"2024-04-25T15:59:50.669960+0000","last_became_active":"2024-04-25T15:59:02.505847+0000","last_became_peered":"2024-04-25T15:59:02.505847+0000","last_unstale":"2024-04-25T15:59:50.669960+0000","last_undegraded":"2024-04-25T15:59:50.669960+0000","last_fullsized":"2024-04-25T15:59:50.669960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:35.227112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"77'15","reported_seq":38,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359258+0000","last_change":"2024-04-25T15:59:50.679040+0000","last_active":"2024-04-25T16:00:19.359258+0000","last_peered":"2024-04-25T16:00:19.359258+0000","last_clean":"2024-04-25T16:00:19.359258+0000","last_became_active":"2024-04-25T15:59:50.678094+0000","last_became_peered":"2024-04-25T15:59:50.678094+0000","last_unstale":"2024-04-25T16:00:19.359258+0000","last_undegraded":"2024-04-25T16:00:19.359258+0000","last_fullsized":"2024-04-25T16:00:19.359258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:11.957966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'5","reported_seq":145,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.443810+0000","last_change":"2024-04-25T15:59:02.507763+0000","last_active":"2024-04-25T16:01:06.443810+0000","last_peered":"2024-04-25T16:01:06.443810+0000","last_clean":"2024-04-25T16:01:06.443810+0000","last_became_active":"2024-04-25T15:59:02.507338+0000","last_became_peered":"2024-04-25T15:59:02.507338+0000","last_unstale":"2024-04-25T16:01:06.443810+0000","last_undegraded":"2024-04-25T16:01:06.443810+0000","last_fullsized":"2024-04-25T16:01:06.443810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:31.710709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.326104+0000","last_change":"2024-04-25T15:59:03.002590+0000","last_active":"2024-04-25T16:00:19.326104+0000","last_peered":"2024-04-25T16:00:19.326104+0000","last_clean":"2024-04-25T16:00:19.326104+0000","last_became_active":"2024-04-25T15:59:03.002262+0000","last_became_peered":"2024-04-25T15:59:03.002262+0000","last_unstale":"2024-04-25T16:00:19.326104+0000","last_undegraded":"2024-04-25T16:00:19.326104+0000","last_fullsized":"2024-04-25T16:00:19.326104+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:48.251484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671899+0000","last_change":"2024-04-25T15:59:02.518853+0000","last_active":"2024-04-25T15:59:50.671899+0000","last_peered":"2024-04-25T15:59:50.671899+0000","last_clean":"2024-04-25T15:59:50.671899+0000","last_became_active":"2024-04-25T15:59:02.518505+0000","last_became_peered":"2024-04-25T15:59:02.518505+0000","last_unstale":"2024-04-25T15:59:50.671899+0000","last_undegraded":"2024-04-25T15:59:50.671899+0000","last_fullsized":"2024-04-25T15:59:50.671899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:01:56.282245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345600+0000","last_change":"2024-04-25T15:59:03.009733+0000","last_active":"2024-04-25T16:00:19.345600+0000","last_peered":"2024-04-25T16:00:19.345600+0000","last_clean":"2024-04-25T16:00:19.345600+0000","last_became_active":"2024-04-25T15:59:03.009251+0000","last_became_peered":"2024-04-25T15:59:03.009251+0000","last_unstale":"2024-04-25T16:00:19.345600+0000","last_undegraded":"2024-04-25T16:00:19.345600+0000","last_fullsized":"2024-04-25T16:00:19.345600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:22.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"75'3","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670602+0000","last_change":"2024-04-25T15:59:02.512493+0000","last_active":"2024-04-25T15:59:50.670602+0000","last_peered":"2024-04-25T15:59:50.670602+0000","last_clean":"2024-04-25T15:59:50.670602+0000","last_became_active":"2024-04-25T15:59:02.512103+0000","last_became_peered":"2024-04-25T15:59:02.512103+0000","last_unstale":"2024-04-25T15:59:50.670602+0000","last_undegraded":"2024-04-25T15:59:50.670602+0000","last_fullsized":"2024-04-25T15:59:50.670602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:29.457755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342733+0000","last_change":"2024-04-25T15:59:03.001406+0000","last_active":"2024-04-25T16:00:19.342733+0000","last_peered":"2024-04-25T16:00:19.342733+0000","last_clean":"2024-04-25T16:00:19.342733+0000","last_became_active":"2024-04-25T15:59:02.999038+0000","last_became_peered":"2024-04-25T15:59:02.999038+0000","last_unstale":"2024-04-25T16:00:19.342733+0000","last_undegraded":"2024-04-25T16:00:19.342733+0000","last_fullsized":"2024-04-25T16:00:19.342733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:56.354821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671531+0000","last_change":"2024-04-25T15:59:02.509619+0000","last_active":"2024-04-25T15:59:50.671531+0000","last_peered":"2024-04-25T15:59:50.671531+0000","last_clean":"2024-04-25T15:59:50.671531+0000","last_became_active":"2024-04-25T15:59:02.509254+0000","last_became_peered":"2024-04-25T15:59:02.509254+0000","last_unstale":"2024-04-25T15:59:50.671531+0000","last_undegraded":"2024-04-25T15:59:50.671531+0000","last_fullsized":"2024-04-25T15:59:50.671531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:00.382581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347276+0000","last_change":"2024-04-25T15:59:03.000009+0000","last_active":"2024-04-25T16:00:19.347276+0000","last_peered":"2024-04-25T16:00:19.347276+0000","last_clean":"2024-04-25T16:00:19.347276+0000","last_became_active":"2024-04-25T15:59:02.999662+0000","last_became_peered":"2024-04-25T15:59:02.999662+0000","last_unstale":"2024-04-25T16:00:19.347276+0000","last_undegraded":"2024-04-25T16:00:19.347276+0000","last_fullsized":"2024-04-25T16:00:19.347276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:05:38.004819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"75'4","reported_seq":259,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.443147+0000","last_change":"2024-04-25T15:59:02.514429+0000","last_active":"2024-04-25T16:01:06.443147+0000","last_peered":"2024-04-25T16:01:06.443147+0000","last_clean":"2024-04-25T16:01:06.443147+0000","last_became_active":"2024-04-25T15:59:02.514144+0000","last_became_peered":"2024-04-25T15:59:02.514144+0000","last_unstale":"2024-04-25T16:01:06.443147+0000","last_undegraded":"2024-04-25T16:01:06.443147+0000","last_fullsized":"2024-04-25T16:01:06.443147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:45.144630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":236,"num_read_kb":236,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360393+0000","last_change":"2024-04-25T15:59:03.008460+0000","last_active":"2024-04-25T16:00:19.360393+0000","last_peered":"2024-04-25T16:00:19.360393+0000","last_clean":"2024-04-25T16:00:19.360393+0000","last_became_active":"2024-04-25T15:59:03.008024+0000","last_became_peered":"2024-04-25T15:59:03.008024+0000","last_unstale":"2024-04-25T16:00:19.360393+0000","last_undegraded":"2024-04-25T16:00:19.360393+0000","last_fullsized":"2024-04-25T16:00:19.360393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:10.288760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"75'4","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676828+0000","last_change":"2024-04-25T15:59:02.512494+0000","last_active":"2024-04-25T15:59:50.676828+0000","last_peered":"2024-04-25T15:59:50.676828+0000","last_clean":"2024-04-25T15:59:50.676828+0000","last_became_active":"2024-04-25T15:59:02.511182+0000","last_became_peered":"2024-04-25T15:59:02.511182+0000","last_unstale":"2024-04-25T15:59:50.676828+0000","last_undegraded":"2024-04-25T15:59:50.676828+0000","last_fullsized":"2024-04-25T15:59:50.676828+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:54.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350559+0000","last_change":"2024-04-25T15:59:03.003848+0000","last_active":"2024-04-25T16:00:19.350559+0000","last_peered":"2024-04-25T16:00:19.350559+0000","last_clean":"2024-04-25T16:00:19.350559+0000","last_became_active":"2024-04-25T15:59:03.003512+0000","last_became_peered":"2024-04-25T15:59:03.003512+0000","last_unstale":"2024-04-25T16:00:19.350559+0000","last_undegraded":"2024-04-25T16:00:19.350559+0000","last_fullsized":"2024-04-25T16:00:19.350559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:54.667054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672694+0000","last_change":"2024-04-25T15:59:02.509343+0000","last_active":"2024-04-25T15:59:50.672694+0000","last_peered":"2024-04-25T15:59:50.672694+0000","last_clean":"2024-04-25T15:59:50.672694+0000","last_became_active":"2024-04-25T15:59:02.508980+0000","last_became_peered":"2024-04-25T15:59:02.508980+0000","last_unstale":"2024-04-25T15:59:50.672694+0000","last_undegraded":"2024-04-25T15:59:50.672694+0000","last_fullsized":"2024-04-25T15:59:50.672694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:54.344717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359525+0000","last_change":"2024-04-25T15:59:02.999862+0000","last_active":"2024-04-25T16:00:19.359525+0000","last_peered":"2024-04-25T16:00:19.359525+0000","last_clean":"2024-04-25T16:00:19.359525+0000","last_became_active":"2024-04-25T15:59:02.999472+0000","last_became_peered":"2024-04-25T15:59:02.999472+0000","last_unstale":"2024-04-25T16:00:19.359525+0000","last_undegraded":"2024-04-25T16:00:19.359525+0000","last_fullsized":"2024-04-25T16:00:19.359525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:46.165045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"77'32","reported_seq":295,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.443183+0000","last_change":"2024-04-25T15:59:02.508949+0000","last_active":"2024-04-25T16:01:06.443183+0000","last_peered":"2024-04-25T16:01:06.443183+0000","last_clean":"2024-04-25T16:01:06.443183+0000","last_became_active":"2024-04-25T15:59:02.508564+0000","last_became_peered":"2024-04-25T15:59:02.508564+0000","last_unstale":"2024-04-25T16:01:06.443183+0000","last_undegraded":"2024-04-25T16:01:06.443183+0000","last_fullsized":"2024-04-25T16:01:06.443183+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:12.590193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57434,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":244,"num_read_kb":179,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344171+0000","last_change":"2024-04-25T15:59:02.998575+0000","last_active":"2024-04-25T16:00:19.344171+0000","last_peered":"2024-04-25T16:00:19.344171+0000","last_clean":"2024-04-25T16:00:19.344171+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.344171+0000","last_undegraded":"2024-04-25T16:00:19.344171+0000","last_fullsized":"2024-04-25T16:00:19.344171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:40.640468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'9","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677368+0000","last_change":"2024-04-25T15:59:02.519110+0000","last_active":"2024-04-25T15:59:50.677368+0000","last_peered":"2024-04-25T15:59:50.677368+0000","last_clean":"2024-04-25T15:59:50.677368+0000","last_became_active":"2024-04-25T15:59:02.518373+0000","last_became_peered":"2024-04-25T15:59:02.518373+0000","last_unstale":"2024-04-25T15:59:50.677368+0000","last_undegraded":"2024-04-25T15:59:50.677368+0000","last_fullsized":"2024-04-25T15:59:50.677368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:27.840296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344110+0000","last_change":"2024-04-25T15:59:02.991002+0000","last_active":"2024-04-25T16:00:19.344110+0000","last_peered":"2024-04-25T16:00:19.344110+0000","last_clean":"2024-04-25T16:00:19.344110+0000","last_became_active":"2024-04-25T15:59:02.990605+0000","last_became_peered":"2024-04-25T15:59:02.990605+0000","last_unstale":"2024-04-25T16:00:19.344110+0000","last_undegraded":"2024-04-25T16:00:19.344110+0000","last_fullsized":"2024-04-25T16:00:19.344110+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:22:00.723255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671723+0000","last_change":"2024-04-25T15:59:02.509809+0000","last_active":"2024-04-25T15:59:50.671723+0000","last_peered":"2024-04-25T15:59:50.671723+0000","last_clean":"2024-04-25T15:59:50.671723+0000","last_became_active":"2024-04-25T15:59:02.509370+0000","last_became_peered":"2024-04-25T15:59:02.509370+0000","last_unstale":"2024-04-25T15:59:50.671723+0000","last_undegraded":"2024-04-25T15:59:50.671723+0000","last_fullsized":"2024-04-25T15:59:50.671723+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:24.416072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343672+0000","last_change":"2024-04-25T15:59:02.999836+0000","last_active":"2024-04-25T16:00:19.343672+0000","last_peered":"2024-04-25T16:00:19.343672+0000","last_clean":"2024-04-25T16:00:19.343672+0000","last_became_active":"2024-04-25T15:59:02.997770+0000","last_became_peered":"2024-04-25T15:59:02.997770+0000","last_unstale":"2024-04-25T16:00:19.343672+0000","last_undegraded":"2024-04-25T16:00:19.343672+0000","last_fullsized":"2024-04-25T16:00:19.343672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:30.953339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"77'7","reported_seq":148,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.462147+0000","last_change":"2024-04-25T15:59:02.510465+0000","last_active":"2024-04-25T16:01:06.462147+0000","last_peered":"2024-04-25T16:01:06.462147+0000","last_clean":"2024-04-25T16:01:06.462147+0000","last_became_active":"2024-04-25T15:59:02.510067+0000","last_became_peered":"2024-04-25T15:59:02.510067+0000","last_unstale":"2024-04-25T16:01:06.462147+0000","last_undegraded":"2024-04-25T16:01:06.462147+0000","last_fullsized":"2024-04-25T16:01:06.462147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:40.739358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346234+0000","last_change":"2024-04-25T15:59:03.000028+0000","last_active":"2024-04-25T16:00:19.346234+0000","last_peered":"2024-04-25T16:00:19.346234+0000","last_clean":"2024-04-25T16:00:19.346234+0000","last_became_active":"2024-04-25T15:59:02.999704+0000","last_became_peered":"2024-04-25T15:59:02.999704+0000","last_unstale":"2024-04-25T16:00:19.346234+0000","last_undegraded":"2024-04-25T16:00:19.346234+0000","last_fullsized":"2024-04-25T16:00:19.346234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:55.015085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'17","reported_seq":37,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669662+0000","last_change":"2024-04-25T15:59:02.514442+0000","last_active":"2024-04-25T15:59:50.669662+0000","last_peered":"2024-04-25T15:59:50.669662+0000","last_clean":"2024-04-25T15:59:50.669662+0000","last_became_active":"2024-04-25T15:59:02.513951+0000","last_became_peered":"2024-04-25T15:59:02.513951+0000","last_unstale":"2024-04-25T15:59:50.669662+0000","last_undegraded":"2024-04-25T15:59:50.669662+0000","last_fullsized":"2024-04-25T15:59:50.669662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:56.443999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"77'6","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.358196+0000","last_change":"2024-04-25T15:59:50.677155+0000","last_active":"2024-04-25T16:00:19.358196+0000","last_peered":"2024-04-25T16:00:19.358196+0000","last_clean":"2024-04-25T16:00:19.358196+0000","last_became_active":"2024-04-25T15:59:50.676721+0000","last_became_peered":"2024-04-25T15:59:50.676721+0000","last_unstale":"2024-04-25T16:00:19.358196+0000","last_undegraded":"2024-04-25T16:00:19.358196+0000","last_fullsized":"2024-04-25T16:00:19.358196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:04.138269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.683753+0000","last_change":"2024-04-25T15:59:50.683228+0000","last_active":"2024-04-25T15:59:50.683753+0000","last_peered":"2024-04-25T15:59:50.683753+0000","last_clean":"2024-04-25T15:59:50.683753+0000","last_became_active":"2024-04-25T15:59:50.675414+0000","last_became_peered":"2024-04-25T15:59:50.675414+0000","last_unstale":"2024-04-25T15:59:50.683753+0000","last_undegraded":"2024-04-25T15:59:50.683753+0000","last_fullsized":"2024-04-25T15:59:50.683753+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:46.577883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341606+0000","last_change":"2024-04-25T15:59:03.004639+0000","last_active":"2024-04-25T16:00:19.341606+0000","last_peered":"2024-04-25T16:00:19.341606+0000","last_clean":"2024-04-25T16:00:19.341606+0000","last_became_active":"2024-04-25T15:59:03.004260+0000","last_became_peered":"2024-04-25T15:59:03.004260+0000","last_unstale":"2024-04-25T16:00:19.341606+0000","last_undegraded":"2024-04-25T16:00:19.341606+0000","last_fullsized":"2024-04-25T16:00:19.341606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:00.450758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666396+0000","last_change":"2024-04-25T15:59:02.509716+0000","last_active":"2024-04-25T15:59:50.666396+0000","last_peered":"2024-04-25T15:59:50.666396+0000","last_clean":"2024-04-25T15:59:50.666396+0000","last_became_active":"2024-04-25T15:59:02.509303+0000","last_became_peered":"2024-04-25T15:59:02.509303+0000","last_unstale":"2024-04-25T15:59:50.666396+0000","last_undegraded":"2024-04-25T15:59:50.666396+0000","last_fullsized":"2024-04-25T15:59:50.666396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:34.037590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349639+0000","last_change":"2024-04-25T15:59:03.005141+0000","last_active":"2024-04-25T16:00:19.349639+0000","last_peered":"2024-04-25T16:00:19.349639+0000","last_clean":"2024-04-25T16:00:19.349639+0000","last_became_active":"2024-04-25T15:59:03.004760+0000","last_became_peered":"2024-04-25T15:59:03.004760+0000","last_unstale":"2024-04-25T16:00:19.349639+0000","last_undegraded":"2024-04-25T16:00:19.349639+0000","last_fullsized":"2024-04-25T16:00:19.349639+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:14.923126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"75'5","reported_seq":143,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.442998+0000","last_change":"2024-04-25T15:59:02.512779+0000","last_active":"2024-04-25T16:01:06.442998+0000","last_peered":"2024-04-25T16:01:06.442998+0000","last_clean":"2024-04-25T16:01:06.442998+0000","last_became_active":"2024-04-25T15:59:02.511240+0000","last_became_peered":"2024-04-25T15:59:02.511240+0000","last_unstale":"2024-04-25T16:01:06.442998+0000","last_undegraded":"2024-04-25T16:01:06.442998+0000","last_fullsized":"2024-04-25T16:01:06.442998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:51.703751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351726+0000","last_change":"2024-04-25T15:59:03.004841+0000","last_active":"2024-04-25T16:00:19.351726+0000","last_peered":"2024-04-25T16:00:19.351726+0000","last_clean":"2024-04-25T16:00:19.351726+0000","last_became_active":"2024-04-25T15:59:03.004488+0000","last_became_peered":"2024-04-25T15:59:03.004488+0000","last_unstale":"2024-04-25T16:00:19.351726+0000","last_undegraded":"2024-04-25T16:00:19.351726+0000","last_fullsized":"2024-04-25T16:00:19.351726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:07.672312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345959+0000","last_change":"2024-04-25T15:59:03.001296+0000","last_active":"2024-04-25T16:00:19.345959+0000","last_peered":"2024-04-25T16:00:19.345959+0000","last_clean":"2024-04-25T16:00:19.345959+0000","last_became_active":"2024-04-25T15:59:03.000568+0000","last_became_peered":"2024-04-25T15:59:03.000568+0000","last_unstale":"2024-04-25T16:00:19.345959+0000","last_undegraded":"2024-04-25T16:00:19.345959+0000","last_fullsized":"2024-04-25T16:00:19.345959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:51.335849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"77'6","reported_seq":147,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:08.465129+0000","last_change":"2024-04-25T15:59:02.513943+0000","last_active":"2024-04-25T16:01:08.465129+0000","last_peered":"2024-04-25T16:01:08.465129+0000","last_clean":"2024-04-25T16:01:08.465129+0000","last_became_active":"2024-04-25T15:59:02.513595+0000","last_became_peered":"2024-04-25T15:59:02.513595+0000","last_unstale":"2024-04-25T16:01:08.465129+0000","last_undegraded":"2024-04-25T16:01:08.465129+0000","last_fullsized":"2024-04-25T16:01:08.465129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:50.720333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.352962+0000","last_change":"2024-04-25T15:59:03.008561+0000","last_active":"2024-04-25T16:00:19.352962+0000","last_peered":"2024-04-25T16:00:19.352962+0000","last_clean":"2024-04-25T16:00:19.352962+0000","last_became_active":"2024-04-25T15:59:03.008134+0000","last_became_peered":"2024-04-25T15:59:03.008134+0000","last_unstale":"2024-04-25T16:00:19.352962+0000","last_undegraded":"2024-04-25T16:00:19.352962+0000","last_fullsized":"2024-04-25T16:00:19.352962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:43.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"75'6","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669630+0000","last_change":"2024-04-25T15:59:02.510731+0000","last_active":"2024-04-25T15:59:50.669630+0000","last_peered":"2024-04-25T15:59:50.669630+0000","last_clean":"2024-04-25T15:59:50.669630+0000","last_became_active":"2024-04-25T15:59:02.510356+0000","last_became_peered":"2024-04-25T15:59:02.510356+0000","last_unstale":"2024-04-25T15:59:50.669630+0000","last_undegraded":"2024-04-25T15:59:50.669630+0000","last_fullsized":"2024-04-25T15:59:50.669630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:33.066858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":95,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677008+0000","last_change":"2024-04-25T15:57:47.988899+0000","last_active":"2024-04-25T15:59:50.677008+0000","last_peered":"2024-04-25T15:59:50.677008+0000","last_clean":"2024-04-25T15:59:50.677008+0000","last_became_active":"2024-04-25T15:57:47.982613+0000","last_became_peered":"2024-04-25T15:57:47.982613+0000","last_unstale":"2024-04-25T15:59:50.677008+0000","last_undegraded":"2024-04-25T15:59:50.677008+0000","last_fullsized":"2024-04-25T15:59:50.677008+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_clean_scrub_stamp":"2024-04-25T15:54:18.013489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:39.323767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"77'5","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341689+0000","last_change":"2024-04-25T15:59:02.998360+0000","last_active":"2024-04-25T16:00:19.341689+0000","last_peered":"2024-04-25T16:00:19.341689+0000","last_clean":"2024-04-25T16:00:19.341689+0000","last_became_active":"2024-04-25T15:59:02.998048+0000","last_became_peered":"2024-04-25T15:59:02.998048+0000","last_unstale":"2024-04-25T16:00:19.341689+0000","last_undegraded":"2024-04-25T16:00:19.341689+0000","last_fullsized":"2024-04-25T16:00:19.341689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:01.524608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677349+0000","last_change":"2024-04-25T15:59:02.515769+0000","last_active":"2024-04-25T15:59:50.677349+0000","last_peered":"2024-04-25T15:59:50.677349+0000","last_clean":"2024-04-25T15:59:50.677349+0000","last_became_active":"2024-04-25T15:59:02.515467+0000","last_became_peered":"2024-04-25T15:59:02.515467+0000","last_unstale":"2024-04-25T15:59:50.677349+0000","last_undegraded":"2024-04-25T15:59:50.677349+0000","last_fullsized":"2024-04-25T15:59:50.677349+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:59:50.107475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"75'4","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670140+0000","last_change":"2024-04-25T15:59:02.512701+0000","last_active":"2024-04-25T15:59:50.670140+0000","last_peered":"2024-04-25T15:59:50.670140+0000","last_clean":"2024-04-25T15:59:50.670140+0000","last_became_active":"2024-04-25T15:59:02.512352+0000","last_became_peered":"2024-04-25T15:59:02.512352+0000","last_unstale":"2024-04-25T15:59:50.670140+0000","last_undegraded":"2024-04-25T15:59:50.670140+0000","last_fullsized":"2024-04-25T15:59:50.670140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:31.772382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"77'9","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.330017+0000","last_change":"2024-04-25T15:59:02.989698+0000","last_active":"2024-04-25T16:00:19.330017+0000","last_peered":"2024-04-25T16:00:19.330017+0000","last_clean":"2024-04-25T16:00:19.330017+0000","last_became_active":"2024-04-25T15:59:02.989284+0000","last_became_peered":"2024-04-25T15:59:02.989284+0000","last_unstale":"2024-04-25T16:00:19.330017+0000","last_undegraded":"2024-04-25T16:00:19.330017+0000","last_fullsized":"2024-04-25T16:00:19.330017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:12.501738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"77'13","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666592+0000","last_change":"2024-04-25T15:59:02.517272+0000","last_active":"2024-04-25T15:59:50.666592+0000","last_peered":"2024-04-25T15:59:50.666592+0000","last_clean":"2024-04-25T15:59:50.666592+0000","last_became_active":"2024-04-25T15:59:02.516370+0000","last_became_peered":"2024-04-25T15:59:02.516370+0000","last_unstale":"2024-04-25T15:59:50.666592+0000","last_undegraded":"2024-04-25T15:59:50.666592+0000","last_fullsized":"2024-04-25T15:59:50.666592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:31.139689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345612+0000","last_change":"2024-04-25T15:59:03.003111+0000","last_active":"2024-04-25T16:00:19.345612+0000","last_peered":"2024-04-25T16:00:19.345612+0000","last_clean":"2024-04-25T16:00:19.345612+0000","last_became_active":"2024-04-25T15:59:03.002766+0000","last_became_peered":"2024-04-25T15:59:03.002766+0000","last_unstale":"2024-04-25T16:00:19.345612+0000","last_undegraded":"2024-04-25T16:00:19.345612+0000","last_fullsized":"2024-04-25T16:00:19.345612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:39.866119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'7","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670151+0000","last_change":"2024-04-25T15:59:02.521970+0000","last_active":"2024-04-25T15:59:50.670151+0000","last_peered":"2024-04-25T15:59:50.670151+0000","last_clean":"2024-04-25T15:59:50.670151+0000","last_became_active":"2024-04-25T15:59:02.521650+0000","last_became_peered":"2024-04-25T15:59:02.521650+0000","last_unstale":"2024-04-25T15:59:50.670151+0000","last_undegraded":"2024-04-25T15:59:50.670151+0000","last_fullsized":"2024-04-25T15:59:50.670151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:32.996868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351310+0000","last_change":"2024-04-25T15:59:02.996358+0000","last_active":"2024-04-25T16:00:19.351310+0000","last_peered":"2024-04-25T16:00:19.351310+0000","last_clean":"2024-04-25T16:00:19.351310+0000","last_became_active":"2024-04-25T15:59:02.995988+0000","last_became_peered":"2024-04-25T15:59:02.995988+0000","last_unstale":"2024-04-25T16:00:19.351310+0000","last_undegraded":"2024-04-25T16:00:19.351310+0000","last_fullsized":"2024-04-25T16:00:19.351310+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:51.021841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"75'2","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.674464+0000","last_change":"2024-04-25T15:59:02.518330+0000","last_active":"2024-04-25T15:59:50.674464+0000","last_peered":"2024-04-25T15:59:50.674464+0000","last_clean":"2024-04-25T15:59:50.674464+0000","last_became_active":"2024-04-25T15:59:02.517978+0000","last_became_peered":"2024-04-25T15:59:02.517978+0000","last_unstale":"2024-04-25T15:59:50.674464+0000","last_undegraded":"2024-04-25T15:59:50.674464+0000","last_fullsized":"2024-04-25T15:59:50.674464+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:25.443963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.302105+0000","last_change":"2024-04-25T15:59:03.000198+0000","last_active":"2024-04-25T16:00:19.302105+0000","last_peered":"2024-04-25T16:00:19.302105+0000","last_clean":"2024-04-25T16:00:19.302105+0000","last_became_active":"2024-04-25T15:59:02.999878+0000","last_became_peered":"2024-04-25T15:59:02.999878+0000","last_unstale":"2024-04-25T16:00:19.302105+0000","last_undegraded":"2024-04-25T16:00:19.302105+0000","last_fullsized":"2024-04-25T16:00:19.302105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:37.966595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807467+0000","last_change":"2024-04-25T15:59:02.511298+0000","last_active":"2024-04-25T15:59:50.807467+0000","last_peered":"2024-04-25T15:59:50.807467+0000","last_clean":"2024-04-25T15:59:50.807467+0000","last_became_active":"2024-04-25T15:59:02.510521+0000","last_became_peered":"2024-04-25T15:59:02.510521+0000","last_unstale":"2024-04-25T15:59:50.807467+0000","last_undegraded":"2024-04-25T15:59:50.807467+0000","last_fullsized":"2024-04-25T15:59:50.807467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:00.807919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347593+0000","last_change":"2024-04-25T15:59:02.999212+0000","last_active":"2024-04-25T16:00:19.347593+0000","last_peered":"2024-04-25T16:00:19.347593+0000","last_clean":"2024-04-25T16:00:19.347593+0000","last_became_active":"2024-04-25T15:59:02.998881+0000","last_became_peered":"2024-04-25T15:59:02.998881+0000","last_unstale":"2024-04-25T16:00:19.347593+0000","last_undegraded":"2024-04-25T16:00:19.347593+0000","last_fullsized":"2024-04-25T16:00:19.347593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:09.092196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"75'3","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676880+0000","last_change":"2024-04-25T15:59:02.517412+0000","last_active":"2024-04-25T15:59:50.676880+0000","last_peered":"2024-04-25T15:59:50.676880+0000","last_clean":"2024-04-25T15:59:50.676880+0000","last_became_active":"2024-04-25T15:59:02.517077+0000","last_became_peered":"2024-04-25T15:59:02.517077+0000","last_unstale":"2024-04-25T15:59:50.676880+0000","last_undegraded":"2024-04-25T15:59:50.676880+0000","last_fullsized":"2024-04-25T15:59:50.676880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:00.161199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"77'8","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342365+0000","last_change":"2024-04-25T15:59:50.678844+0000","last_active":"2024-04-25T16:00:19.342365+0000","last_peered":"2024-04-25T16:00:19.342365+0000","last_clean":"2024-04-25T16:00:19.342365+0000","last_became_active":"2024-04-25T15:59:50.678411+0000","last_became_peered":"2024-04-25T15:59:50.678411+0000","last_unstale":"2024-04-25T16:00:19.342365+0000","last_undegraded":"2024-04-25T16:00:19.342365+0000","last_fullsized":"2024-04-25T16:00:19.342365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:56:48.018678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669652+0000","last_change":"2024-04-25T15:59:02.515912+0000","last_active":"2024-04-25T15:59:50.669652+0000","last_peered":"2024-04-25T15:59:50.669652+0000","last_clean":"2024-04-25T15:59:50.669652+0000","last_became_active":"2024-04-25T15:59:02.514949+0000","last_became_peered":"2024-04-25T15:59:02.514949+0000","last_unstale":"2024-04-25T15:59:50.669652+0000","last_undegraded":"2024-04-25T15:59:50.669652+0000","last_fullsized":"2024-04-25T15:59:50.669652+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:33.467223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360414+0000","last_change":"2024-04-25T15:59:02.988896+0000","last_active":"2024-04-25T16:00:19.360414+0000","last_peered":"2024-04-25T16:00:19.360414+0000","last_clean":"2024-04-25T16:00:19.360414+0000","last_became_active":"2024-04-25T15:59:02.988478+0000","last_became_peered":"2024-04-25T15:59:02.988478+0000","last_unstale":"2024-04-25T16:00:19.360414+0000","last_undegraded":"2024-04-25T16:00:19.360414+0000","last_fullsized":"2024-04-25T16:00:19.360414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:47.014678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"75'5","reported_seq":97,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.781861+0000","last_change":"2024-04-25T15:59:50.687632+0000","last_active":"2024-04-25T16:01:06.781861+0000","last_peered":"2024-04-25T16:01:06.781861+0000","last_clean":"2024-04-25T16:01:06.781861+0000","last_became_active":"2024-04-25T15:59:50.680402+0000","last_became_peered":"2024-04-25T15:59:50.680402+0000","last_unstale":"2024-04-25T16:01:06.781861+0000","last_undegraded":"2024-04-25T16:01:06.781861+0000","last_fullsized":"2024-04-25T16:01:06.781861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:12.544489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359317+0000","last_change":"2024-04-25T15:59:02.990507+0000","last_active":"2024-04-25T16:00:19.359317+0000","last_peered":"2024-04-25T16:00:19.359317+0000","last_clean":"2024-04-25T16:00:19.359317+0000","last_became_active":"2024-04-25T15:59:02.990160+0000","last_became_peered":"2024-04-25T15:59:02.990160+0000","last_unstale":"2024-04-25T16:00:19.359317+0000","last_undegraded":"2024-04-25T16:00:19.359317+0000","last_fullsized":"2024-04-25T16:00:19.359317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:18:21.526155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672520+0000","last_change":"2024-04-25T15:59:02.510956+0000","last_active":"2024-04-25T15:59:50.672520+0000","last_peered":"2024-04-25T15:59:50.672520+0000","last_clean":"2024-04-25T15:59:50.672520+0000","last_became_active":"2024-04-25T15:59:02.510578+0000","last_became_peered":"2024-04-25T15:59:02.510578+0000","last_unstale":"2024-04-25T15:59:50.672520+0000","last_undegraded":"2024-04-25T15:59:50.672520+0000","last_fullsized":"2024-04-25T15:59:50.672520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:11.512365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337102+0000","last_change":"2024-04-25T15:59:03.001122+0000","last_active":"2024-04-25T16:00:19.337102+0000","last_peered":"2024-04-25T16:00:19.337102+0000","last_clean":"2024-04-25T16:00:19.337102+0000","last_became_active":"2024-04-25T15:59:03.000672+0000","last_became_peered":"2024-04-25T15:59:03.000672+0000","last_unstale":"2024-04-25T16:00:19.337102+0000","last_undegraded":"2024-04-25T16:00:19.337102+0000","last_fullsized":"2024-04-25T16:00:19.337102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:33.216434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'5","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670676+0000","last_change":"2024-04-25T15:59:02.520645+0000","last_active":"2024-04-25T15:59:50.670676+0000","last_peered":"2024-04-25T15:59:50.670676+0000","last_clean":"2024-04-25T15:59:50.670676+0000","last_became_active":"2024-04-25T15:59:02.520197+0000","last_became_peered":"2024-04-25T15:59:02.520197+0000","last_unstale":"2024-04-25T15:59:50.670676+0000","last_undegraded":"2024-04-25T15:59:50.670676+0000","last_fullsized":"2024-04-25T15:59:50.670676+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:20.120554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347691+0000","last_change":"2024-04-25T15:59:03.001668+0000","last_active":"2024-04-25T16:00:19.347691+0000","last_peered":"2024-04-25T16:00:19.347691+0000","last_clean":"2024-04-25T16:00:19.347691+0000","last_became_active":"2024-04-25T15:59:03.001333+0000","last_became_peered":"2024-04-25T15:59:03.001333+0000","last_unstale":"2024-04-25T16:00:19.347691+0000","last_undegraded":"2024-04-25T16:00:19.347691+0000","last_fullsized":"2024-04-25T16:00:19.347691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:24.992476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346081+0000","last_change":"2024-04-25T15:59:03.005533+0000","last_active":"2024-04-25T16:00:19.346081+0000","last_peered":"2024-04-25T16:00:19.346081+0000","last_clean":"2024-04-25T16:00:19.346081+0000","last_became_active":"2024-04-25T15:59:03.004124+0000","last_became_peered":"2024-04-25T15:59:03.004124+0000","last_unstale":"2024-04-25T16:00:19.346081+0000","last_undegraded":"2024-04-25T16:00:19.346081+0000","last_fullsized":"2024-04-25T16:00:19.346081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:27:28.507065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"77'12","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.331274+0000","last_change":"2024-04-25T15:59:02.993416+0000","last_active":"2024-04-25T16:00:19.331274+0000","last_peered":"2024-04-25T16:00:19.331274+0000","last_clean":"2024-04-25T16:00:19.331274+0000","last_became_active":"2024-04-25T15:59:02.993098+0000","last_became_peered":"2024-04-25T15:59:02.993098+0000","last_unstale":"2024-04-25T16:00:19.331274+0000","last_undegraded":"2024-04-25T16:00:19.331274+0000","last_fullsized":"2024-04-25T16:00:19.331274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:46.859932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350142+0000","last_change":"2024-04-25T15:59:02.996858+0000","last_active":"2024-04-25T16:00:19.350142+0000","last_peered":"2024-04-25T16:00:19.350142+0000","last_clean":"2024-04-25T16:00:19.350142+0000","last_became_active":"2024-04-25T15:59:02.996530+0000","last_became_peered":"2024-04-25T15:59:02.996530+0000","last_unstale":"2024-04-25T16:00:19.350142+0000","last_undegraded":"2024-04-25T16:00:19.350142+0000","last_fullsized":"2024-04-25T16:00:19.350142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:36.723948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"77'7","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350676+0000","last_change":"2024-04-25T15:59:02.995942+0000","last_active":"2024-04-25T16:00:19.350676+0000","last_peered":"2024-04-25T16:00:19.350676+0000","last_clean":"2024-04-25T16:00:19.350676+0000","last_became_active":"2024-04-25T15:59:02.995567+0000","last_became_peered":"2024-04-25T15:59:02.995567+0000","last_unstale":"2024-04-25T16:00:19.350676+0000","last_undegraded":"2024-04-25T16:00:19.350676+0000","last_fullsized":"2024-04-25T16:00:19.350676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:32.692392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350261+0000","last_change":"2024-04-25T15:59:03.000326+0000","last_active":"2024-04-25T16:00:19.350261+0000","last_peered":"2024-04-25T16:00:19.350261+0000","last_clean":"2024-04-25T16:00:19.350261+0000","last_became_active":"2024-04-25T15:59:02.999961+0000","last_became_peered":"2024-04-25T15:59:02.999961+0000","last_unstale":"2024-04-25T16:00:19.350261+0000","last_undegraded":"2024-04-25T16:00:19.350261+0000","last_fullsized":"2024-04-25T16:00:19.350261+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:10.631460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341696+0000","last_change":"2024-04-25T15:59:03.003625+0000","last_active":"2024-04-25T16:00:19.341696+0000","last_peered":"2024-04-25T16:00:19.341696+0000","last_clean":"2024-04-25T16:00:19.341696+0000","last_became_active":"2024-04-25T15:59:03.003280+0000","last_became_peered":"2024-04-25T15:59:03.003280+0000","last_unstale":"2024-04-25T16:00:19.341696+0000","last_undegraded":"2024-04-25T16:00:19.341696+0000","last_fullsized":"2024-04-25T16:00:19.341696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:21:54.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346578+0000","last_change":"2024-04-25T15:59:02.994645+0000","last_active":"2024-04-25T16:00:19.346578+0000","last_peered":"2024-04-25T16:00:19.346578+0000","last_clean":"2024-04-25T16:00:19.346578+0000","last_became_active":"2024-04-25T15:59:02.993924+0000","last_became_peered":"2024-04-25T15:59:02.993924+0000","last_unstale":"2024-04-25T16:00:19.346578+0000","last_undegraded":"2024-04-25T16:00:19.346578+0000","last_fullsized":"2024-04-25T16:00:19.346578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.644708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340331+0000","last_change":"2024-04-25T15:59:03.006156+0000","last_active":"2024-04-25T16:00:19.340331+0000","last_peered":"2024-04-25T16:00:19.340331+0000","last_clean":"2024-04-25T16:00:19.340331+0000","last_became_active":"2024-04-25T15:59:03.005795+0000","last_became_peered":"2024-04-25T15:59:03.005795+0000","last_unstale":"2024-04-25T16:00:19.340331+0000","last_undegraded":"2024-04-25T16:00:19.340331+0000","last_fullsized":"2024-04-25T16:00:19.340331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:23.523868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337613+0000","last_change":"2024-04-25T15:59:03.010404+0000","last_active":"2024-04-25T16:00:19.337613+0000","last_peered":"2024-04-25T16:00:19.337613+0000","last_clean":"2024-04-25T16:00:19.337613+0000","last_became_active":"2024-04-25T15:59:03.010100+0000","last_became_peered":"2024-04-25T15:59:03.010100+0000","last_unstale":"2024-04-25T16:00:19.337613+0000","last_undegraded":"2024-04-25T16:00:19.337613+0000","last_fullsized":"2024-04-25T16:00:19.337613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:22.718842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.335498+0000","last_change":"2024-04-25T15:59:03.009240+0000","last_active":"2024-04-25T16:00:19.335498+0000","last_peered":"2024-04-25T16:00:19.335498+0000","last_clean":"2024-04-25T16:00:19.335498+0000","last_became_active":"2024-04-25T15:59:03.008723+0000","last_became_peered":"2024-04-25T15:59:03.008723+0000","last_unstale":"2024-04-25T16:00:19.335498+0000","last_undegraded":"2024-04-25T16:00:19.335498+0000","last_fullsized":"2024-04-25T16:00:19.335498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:14.049934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359019+0000","last_change":"2024-04-25T15:59:02.994359+0000","last_active":"2024-04-25T16:00:19.359019+0000","last_peered":"2024-04-25T16:00:19.359019+0000","last_clean":"2024-04-25T16:00:19.359019+0000","last_became_active":"2024-04-25T15:59:02.993990+0000","last_became_peered":"2024-04-25T15:59:02.993990+0000","last_unstale":"2024-04-25T16:00:19.359019+0000","last_undegraded":"2024-04-25T16:00:19.359019+0000","last_fullsized":"2024-04-25T16:00:19.359019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:55.477414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350058+0000","last_change":"2024-04-25T15:59:02.995510+0000","last_active":"2024-04-25T16:00:19.350058+0000","last_peered":"2024-04-25T16:00:19.350058+0000","last_clean":"2024-04-25T16:00:19.350058+0000","last_became_active":"2024-04-25T15:59:02.995008+0000","last_became_peered":"2024-04-25T15:59:02.995008+0000","last_unstale":"2024-04-25T16:00:19.350058+0000","last_undegraded":"2024-04-25T16:00:19.350058+0000","last_fullsized":"2024-04-25T16:00:19.350058+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:40.439615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341453+0000","last_change":"2024-04-25T15:59:03.005846+0000","last_active":"2024-04-25T16:00:19.341453+0000","last_peered":"2024-04-25T16:00:19.341453+0000","last_clean":"2024-04-25T16:00:19.341453+0000","last_became_active":"2024-04-25T15:59:03.003779+0000","last_became_peered":"2024-04-25T15:59:03.003779+0000","last_unstale":"2024-04-25T16:00:19.341453+0000","last_undegraded":"2024-04-25T16:00:19.341453+0000","last_fullsized":"2024-04-25T16:00:19.341453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.698405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247542,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64388,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1180,"num_read_kb":682,"num_write":242,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1192,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":193472,"data_compressed":5607,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710757,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":839680,"data_stored":402102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906988,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":1051075,"data_compressed":37695,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070513,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":402010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266743,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":851968,"data_stored":402366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462972,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":995047,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626497,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":401727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":461244,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":403066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":459124,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":401552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986252,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":402120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822726,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":994193,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60356,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58236,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":59702,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:01:12.876 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:01:12.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:12 smithi186 ceph-mon[33258]: pgmap v348: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-25T16:01:12.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:12 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3399385418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:12.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:12 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/188066102' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:12.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:12 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/43483836' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:13.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.023+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2020179862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 msgr2=0x7f3fa4104810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.023+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2020179862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 0x7f3fa4104810 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f3f94009900 tx=0x7f3f9402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:13.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.023+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2020179862 shutdown_connections 2024-04-25T16:01:13.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.023+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2020179862 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa4109a50 0x7f3fa410be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.023+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2020179862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fa4104d50 0x7f3fa4105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.023+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2020179862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 0x7f3fa4104810 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.023+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2020179862 >> 172.21.15.31:0/2020179862 conn(0x7f3fa40fed40 msgr2=0x7f3fa41011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:13.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.023+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2020179862 shutdown_connections 2024-04-25T16:01:13.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.024+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2020179862 wait complete. 2024-04-25T16:01:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.024+0000 7f3faa2b6640 1 Processor -- start 2024-04-25T16:01:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.024+0000 7f3faa2b6640 1 -- start start 2024-04-25T16:01:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.024+0000 7f3faa2b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 0x7f3fa4077910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3faa2b6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa4104d50 0x7f3fa4076000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3faa2b6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fa4109a50 0x7f3fa4076540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3faa2b6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa410f610 con 0x7f3fa4103b10 2024-04-25T16:01:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3faa2b6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa410f490 con 0x7f3fa4104d50 2024-04-25T16:01:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3faa2b6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa410f790 con 0x7f3fa4109a50 2024-04-25T16:01:13.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 0x7f3fa4077910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 0x7f3fa4077910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56324/0 (socket says 172.21.15.31:56324) 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa3fff640 1 -- 172.21.15.31:0/2591517176 learned_addr learned my addr 172.21.15.31:0/2591517176 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa37fe640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa4104d50 0x7f3fa4076000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa3fff640 1 -- 172.21.15.31:0/2591517176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fa4109a50 msgr2=0x7f3fa4076540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa3fff640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fa4109a50 0x7f3fa4076540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa3fff640 1 -- 172.21.15.31:0/2591517176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa4104d50 msgr2=0x7f3fa4076000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa3fff640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa4104d50 0x7f3fa4076000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa3fff640 1 -- 172.21.15.31:0/2591517176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa4076c20 con 0x7f3fa4103b10 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa3fff640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 0x7f3fa4077910 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3f94006010 tx=0x7f3f94031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa17fa640 1 -- 172.21.15.31:0/2591517176 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9403f030 con 0x7f3fa4103b10 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa41a4390 con 0x7f3fa4103b10 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa17fa640 1 -- 172.21.15.31:0/2591517176 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f94003aa0 con 0x7f3fa4103b10 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3fa17fa640 1 -- 172.21.15.31:0/2591517176 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f94035070 con 0x7f3fa4103b10 2024-04-25T16:01:13.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.025+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa41a4740 con 0x7f3fa4103b10 2024-04-25T16:01:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.027+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa4108860 con 0x7f3fa4103b10 2024-04-25T16:01:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.028+0000 7f3fa17fa640 1 -- 172.21.15.31:0/2591517176 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3f94042030 con 0x7f3fa4103b10 2024-04-25T16:01:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.029+0000 7f3fa17fa640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3f8c0afaf0 0x7f3f8c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.029+0000 7f3fa37fe640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3f8c0afaf0 0x7f3f8c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.029+0000 7f3fa17fa640 1 -- 172.21.15.31:0/2591517176 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3f940ff690 con 0x7f3fa4103b10 2024-04-25T16:01:13.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.029+0000 7f3fa37fe640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3f8c0afaf0 0x7f3f8c0b2000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3f98005fd0 tx=0x7f3f98005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:13.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.032+0000 7f3fa17fa640 1 -- 172.21.15.31:0/2591517176 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f940c8280 con 0x7f3fa4103b10 2024-04-25T16:01:13.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.215+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3fa4062fe0 con 0x7f3f8c0afaf0 2024-04-25T16:01:13.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.222+0000 7f3fa17fa640 1 -- 172.21.15.31:0/2591517176 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259069 (secure 0 0 0) 0x7f3fa4062fe0 con 0x7f3f8c0afaf0 2024-04-25T16:01:13.225 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:13.227 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T16:01:13.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3f8c0afaf0 msgr2=0x7f3f8c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3f8c0afaf0 0x7f3f8c0b2000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3f98005fd0 tx=0x7f3f98005850 comp rx=0 tx=0).stop 2024-04-25T16:01:13.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 msgr2=0x7f3fa4077910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 0x7f3fa4077910 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f3f94006010 tx=0x7f3f94031a50 comp rx=0 tx=0).stop 2024-04-25T16:01:13.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 shutdown_connections 2024-04-25T16:01:13.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3f8c0afaf0 0x7f3f8c0b2000 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fa4109a50 0x7f3fa4076540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fa4104d50 0x7f3fa4076000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 --2- 172.21.15.31:0/2591517176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fa4103b10 0x7f3fa4077910 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 >> 172.21.15.31:0/2591517176 conn(0x7f3fa40fed40 msgr2=0x7f3fa4105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:13.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 shutdown_connections 2024-04-25T16:01:13.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.226+0000 7f3faa2b6640 1 -- 172.21.15.31:0/2591517176 wait complete. 2024-04-25T16:01:13.238 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-04-25T16:01:11.347617+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654990,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1256,"num_read_kb":746,"num_write":367,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2953032,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":804,"ondisk_log_size":804,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355336,"kb_used_data":10344,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124586488,"statfs":{"total":1151940427776,"available":1151576563712,"internally_reserved":0,"allocated":10592256,"data_stored":6775626,"data_compressed":113085,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":100,"num_read_kb":50,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003064"},"pg_stats":[{"pgid":"3.3f","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354813+0000","last_change":"2024-04-25T15:59:02.997678+0000","last_active":"2024-04-25T16:00:19.354813+0000","last_peered":"2024-04-25T16:00:19.354813+0000","last_clean":"2024-04-25T16:00:19.354813+0000","last_became_active":"2024-04-25T15:59:02.997335+0000","last_became_peered":"2024-04-25T15:59:02.997335+0000","last_unstale":"2024-04-25T16:00:19.354813+0000","last_undegraded":"2024-04-25T16:00:19.354813+0000","last_fullsized":"2024-04-25T16:00:19.354813+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:33.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"77'4","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351614+0000","last_change":"2024-04-25T15:59:03.001049+0000","last_active":"2024-04-25T16:00:19.351614+0000","last_peered":"2024-04-25T16:00:19.351614+0000","last_clean":"2024-04-25T16:00:19.351614+0000","last_became_active":"2024-04-25T15:59:03.000663+0000","last_became_peered":"2024-04-25T15:59:03.000663+0000","last_unstale":"2024-04-25T16:00:19.351614+0000","last_undegraded":"2024-04-25T16:00:19.351614+0000","last_fullsized":"2024-04-25T16:00:19.351614+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:04:36.670519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"77'7","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354238+0000","last_change":"2024-04-25T15:59:03.003861+0000","last_active":"2024-04-25T16:00:19.354238+0000","last_peered":"2024-04-25T16:00:19.354238+0000","last_clean":"2024-04-25T16:00:19.354238+0000","last_became_active":"2024-04-25T15:59:03.003508+0000","last_became_peered":"2024-04-25T15:59:03.003508+0000","last_unstale":"2024-04-25T16:00:19.354238+0000","last_undegraded":"2024-04-25T16:00:19.354238+0000","last_fullsized":"2024-04-25T16:00:19.354238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:25.679389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337369+0000","last_change":"2024-04-25T15:59:03.003330+0000","last_active":"2024-04-25T16:00:19.337369+0000","last_peered":"2024-04-25T16:00:19.337369+0000","last_clean":"2024-04-25T16:00:19.337369+0000","last_became_active":"2024-04-25T15:59:03.002872+0000","last_became_peered":"2024-04-25T15:59:03.002872+0000","last_unstale":"2024-04-25T16:00:19.337369+0000","last_undegraded":"2024-04-25T16:00:19.337369+0000","last_fullsized":"2024-04-25T16:00:19.337369+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:02.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.324906+0000","last_change":"2024-04-25T15:59:02.997684+0000","last_active":"2024-04-25T16:00:19.324906+0000","last_peered":"2024-04-25T16:00:19.324906+0000","last_clean":"2024-04-25T16:00:19.324906+0000","last_became_active":"2024-04-25T15:59:02.997370+0000","last_became_peered":"2024-04-25T15:59:02.997370+0000","last_unstale":"2024-04-25T16:00:19.324906+0000","last_undegraded":"2024-04-25T16:00:19.324906+0000","last_fullsized":"2024-04-25T16:00:19.324906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:24.349085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348756+0000","last_change":"2024-04-25T15:59:03.003638+0000","last_active":"2024-04-25T16:00:19.348756+0000","last_peered":"2024-04-25T16:00:19.348756+0000","last_clean":"2024-04-25T16:00:19.348756+0000","last_became_active":"2024-04-25T15:59:03.003299+0000","last_became_peered":"2024-04-25T15:59:03.003299+0000","last_unstale":"2024-04-25T16:00:19.348756+0000","last_undegraded":"2024-04-25T16:00:19.348756+0000","last_fullsized":"2024-04-25T16:00:19.348756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:46.842345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.338851+0000","last_change":"2024-04-25T15:59:02.998591+0000","last_active":"2024-04-25T16:00:19.338851+0000","last_peered":"2024-04-25T16:00:19.338851+0000","last_clean":"2024-04-25T16:00:19.338851+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.338851+0000","last_undegraded":"2024-04-25T16:00:19.338851+0000","last_fullsized":"2024-04-25T16:00:19.338851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.444499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343142+0000","last_change":"2024-04-25T15:59:03.006225+0000","last_active":"2024-04-25T16:00:19.343142+0000","last_peered":"2024-04-25T16:00:19.343142+0000","last_clean":"2024-04-25T16:00:19.343142+0000","last_became_active":"2024-04-25T15:59:03.005883+0000","last_became_peered":"2024-04-25T15:59:03.005883+0000","last_unstale":"2024-04-25T16:00:19.343142+0000","last_undegraded":"2024-04-25T16:00:19.343142+0000","last_fullsized":"2024-04-25T16:00:19.343142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:17.773196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359702+0000","last_change":"2024-04-25T15:59:03.003605+0000","last_active":"2024-04-25T16:00:19.359702+0000","last_peered":"2024-04-25T16:00:19.359702+0000","last_clean":"2024-04-25T16:00:19.359702+0000","last_became_active":"2024-04-25T15:59:03.003271+0000","last_became_peered":"2024-04-25T15:59:03.003271+0000","last_unstale":"2024-04-25T16:00:19.359702+0000","last_undegraded":"2024-04-25T16:00:19.359702+0000","last_fullsized":"2024-04-25T16:00:19.359702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:28.576407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'16","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348846+0000","last_change":"2024-04-25T15:59:50.679099+0000","last_active":"2024-04-25T16:00:19.348846+0000","last_peered":"2024-04-25T16:00:19.348846+0000","last_clean":"2024-04-25T16:00:19.348846+0000","last_became_active":"2024-04-25T15:59:50.678558+0000","last_became_peered":"2024-04-25T15:59:50.678558+0000","last_unstale":"2024-04-25T16:00:19.348846+0000","last_undegraded":"2024-04-25T16:00:19.348846+0000","last_fullsized":"2024-04-25T16:00:19.348846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:40.691050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359322+0000","last_change":"2024-04-25T15:59:02.999307+0000","last_active":"2024-04-25T16:00:19.359322+0000","last_peered":"2024-04-25T16:00:19.359322+0000","last_clean":"2024-04-25T16:00:19.359322+0000","last_became_active":"2024-04-25T15:59:02.998976+0000","last_became_peered":"2024-04-25T15:59:02.998976+0000","last_unstale":"2024-04-25T16:00:19.359322+0000","last_undegraded":"2024-04-25T16:00:19.359322+0000","last_fullsized":"2024-04-25T16:00:19.359322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:29.626440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347700+0000","last_change":"2024-04-25T15:59:03.005208+0000","last_active":"2024-04-25T16:00:19.347700+0000","last_peered":"2024-04-25T16:00:19.347700+0000","last_clean":"2024-04-25T16:00:19.347700+0000","last_became_active":"2024-04-25T15:59:03.004869+0000","last_became_peered":"2024-04-25T15:59:03.004869+0000","last_unstale":"2024-04-25T16:00:19.347700+0000","last_undegraded":"2024-04-25T16:00:19.347700+0000","last_fullsized":"2024-04-25T16:00:19.347700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:00:24.040763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"77'10","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349448+0000","last_change":"2024-04-25T15:59:50.673556+0000","last_active":"2024-04-25T16:00:19.349448+0000","last_peered":"2024-04-25T16:00:19.349448+0000","last_clean":"2024-04-25T16:00:19.349448+0000","last_became_active":"2024-04-25T15:59:50.673119+0000","last_became_peered":"2024-04-25T15:59:50.673119+0000","last_unstale":"2024-04-25T16:00:19.349448+0000","last_undegraded":"2024-04-25T16:00:19.349448+0000","last_fullsized":"2024-04-25T16:00:19.349448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:09:50.286968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350359+0000","last_change":"2024-04-25T15:59:03.002160+0000","last_active":"2024-04-25T16:00:19.350359+0000","last_peered":"2024-04-25T16:00:19.350359+0000","last_clean":"2024-04-25T16:00:19.350359+0000","last_became_active":"2024-04-25T15:59:03.001855+0000","last_became_peered":"2024-04-25T15:59:03.001855+0000","last_unstale":"2024-04-25T16:00:19.350359+0000","last_undegraded":"2024-04-25T16:00:19.350359+0000","last_fullsized":"2024-04-25T16:00:19.350359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:43.383851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350513+0000","last_change":"2024-04-25T15:59:02.995318+0000","last_active":"2024-04-25T16:00:19.350513+0000","last_peered":"2024-04-25T16:00:19.350513+0000","last_clean":"2024-04-25T16:00:19.350513+0000","last_became_active":"2024-04-25T15:59:02.994956+0000","last_became_peered":"2024-04-25T15:59:02.994956+0000","last_unstale":"2024-04-25T16:00:19.350513+0000","last_undegraded":"2024-04-25T16:00:19.350513+0000","last_fullsized":"2024-04-25T16:00:19.350513+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:59.659400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359091+0000","last_change":"2024-04-25T15:59:02.996485+0000","last_active":"2024-04-25T16:00:19.359091+0000","last_peered":"2024-04-25T16:00:19.359091+0000","last_clean":"2024-04-25T16:00:19.359091+0000","last_became_active":"2024-04-25T15:59:02.996011+0000","last_became_peered":"2024-04-25T15:59:02.996011+0000","last_unstale":"2024-04-25T16:00:19.359091+0000","last_undegraded":"2024-04-25T16:00:19.359091+0000","last_fullsized":"2024-04-25T16:00:19.359091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:56.025254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.355086+0000","last_change":"2024-04-25T15:59:02.998455+0000","last_active":"2024-04-25T16:00:19.355086+0000","last_peered":"2024-04-25T16:00:19.355086+0000","last_clean":"2024-04-25T16:00:19.355086+0000","last_became_active":"2024-04-25T15:59:02.998126+0000","last_became_peered":"2024-04-25T15:59:02.998126+0000","last_unstale":"2024-04-25T16:00:19.355086+0000","last_undegraded":"2024-04-25T16:00:19.355086+0000","last_fullsized":"2024-04-25T16:00:19.355086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:20.342413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350929+0000","last_change":"2024-04-25T15:59:02.998766+0000","last_active":"2024-04-25T16:00:19.350929+0000","last_peered":"2024-04-25T16:00:19.350929+0000","last_clean":"2024-04-25T16:00:19.350929+0000","last_became_active":"2024-04-25T15:59:02.998415+0000","last_became_peered":"2024-04-25T15:59:02.998415+0000","last_unstale":"2024-04-25T16:00:19.350929+0000","last_undegraded":"2024-04-25T16:00:19.350929+0000","last_fullsized":"2024-04-25T16:00:19.350929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:55:05.717847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340892+0000","last_change":"2024-04-25T15:59:02.994982+0000","last_active":"2024-04-25T16:00:19.340892+0000","last_peered":"2024-04-25T16:00:19.340892+0000","last_clean":"2024-04-25T16:00:19.340892+0000","last_became_active":"2024-04-25T15:59:02.993928+0000","last_became_peered":"2024-04-25T15:59:02.993928+0000","last_unstale":"2024-04-25T16:00:19.340892+0000","last_undegraded":"2024-04-25T16:00:19.340892+0000","last_fullsized":"2024-04-25T16:00:19.340892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:22:25.346541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'13","reported_seq":153,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:08.445335+0000","last_change":"2024-04-25T15:59:02.512828+0000","last_active":"2024-04-25T16:01:08.445335+0000","last_peered":"2024-04-25T16:01:08.445335+0000","last_clean":"2024-04-25T16:01:08.445335+0000","last_became_active":"2024-04-25T15:59:02.512291+0000","last_became_peered":"2024-04-25T15:59:02.512291+0000","last_unstale":"2024-04-25T16:01:08.445335+0000","last_undegraded":"2024-04-25T16:01:08.445335+0000","last_fullsized":"2024-04-25T16:01:08.445335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:02.372140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"77'5","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340094+0000","last_change":"2024-04-25T15:59:03.008472+0000","last_active":"2024-04-25T16:00:19.340094+0000","last_peered":"2024-04-25T16:00:19.340094+0000","last_clean":"2024-04-25T16:00:19.340094+0000","last_became_active":"2024-04-25T15:59:03.008108+0000","last_became_peered":"2024-04-25T15:59:03.008108+0000","last_unstale":"2024-04-25T16:00:19.340094+0000","last_undegraded":"2024-04-25T16:00:19.340094+0000","last_fullsized":"2024-04-25T16:00:19.340094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:40.087592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'2","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807946+0000","last_change":"2024-04-25T15:59:02.517846+0000","last_active":"2024-04-25T15:59:50.807946+0000","last_peered":"2024-04-25T15:59:50.807946+0000","last_clean":"2024-04-25T15:59:50.807946+0000","last_became_active":"2024-04-25T15:59:02.517366+0000","last_became_peered":"2024-04-25T15:59:02.517366+0000","last_unstale":"2024-04-25T15:59:50.807946+0000","last_undegraded":"2024-04-25T15:59:50.807946+0000","last_fullsized":"2024-04-25T15:59:50.807946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:29.620827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'6","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354259+0000","last_change":"2024-04-25T15:59:03.005202+0000","last_active":"2024-04-25T16:00:19.354259+0000","last_peered":"2024-04-25T16:00:19.354259+0000","last_clean":"2024-04-25T16:00:19.354259+0000","last_became_active":"2024-04-25T15:59:03.003447+0000","last_became_peered":"2024-04-25T15:59:03.003447+0000","last_unstale":"2024-04-25T16:00:19.354259+0000","last_undegraded":"2024-04-25T16:00:19.354259+0000","last_fullsized":"2024-04-25T16:00:19.354259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.740016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'12","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.703214+0000","last_change":"2024-04-25T15:59:50.702764+0000","last_active":"2024-04-25T15:59:50.703214+0000","last_peered":"2024-04-25T15:59:50.703214+0000","last_clean":"2024-04-25T15:59:50.703214+0000","last_became_active":"2024-04-25T15:59:50.679231+0000","last_became_peered":"2024-04-25T15:59:50.679231+0000","last_unstale":"2024-04-25T15:59:50.703214+0000","last_undegraded":"2024-04-25T15:59:50.703214+0000","last_fullsized":"2024-04-25T15:59:50.703214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:37.561992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360223+0000","last_change":"2024-04-25T15:59:03.004210+0000","last_active":"2024-04-25T16:00:19.360223+0000","last_peered":"2024-04-25T16:00:19.360223+0000","last_clean":"2024-04-25T16:00:19.360223+0000","last_became_active":"2024-04-25T15:59:03.002769+0000","last_became_peered":"2024-04-25T15:59:03.002769+0000","last_unstale":"2024-04-25T16:00:19.360223+0000","last_undegraded":"2024-04-25T16:00:19.360223+0000","last_fullsized":"2024-04-25T16:00:19.360223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:50.608327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669960+0000","last_change":"2024-04-25T15:59:02.506199+0000","last_active":"2024-04-25T15:59:50.669960+0000","last_peered":"2024-04-25T15:59:50.669960+0000","last_clean":"2024-04-25T15:59:50.669960+0000","last_became_active":"2024-04-25T15:59:02.505847+0000","last_became_peered":"2024-04-25T15:59:02.505847+0000","last_unstale":"2024-04-25T15:59:50.669960+0000","last_undegraded":"2024-04-25T15:59:50.669960+0000","last_fullsized":"2024-04-25T15:59:50.669960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:35.227112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"77'15","reported_seq":38,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359258+0000","last_change":"2024-04-25T15:59:50.679040+0000","last_active":"2024-04-25T16:00:19.359258+0000","last_peered":"2024-04-25T16:00:19.359258+0000","last_clean":"2024-04-25T16:00:19.359258+0000","last_became_active":"2024-04-25T15:59:50.678094+0000","last_became_peered":"2024-04-25T15:59:50.678094+0000","last_unstale":"2024-04-25T16:00:19.359258+0000","last_undegraded":"2024-04-25T16:00:19.359258+0000","last_fullsized":"2024-04-25T16:00:19.359258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:11.957966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'5","reported_seq":145,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.443810+0000","last_change":"2024-04-25T15:59:02.507763+0000","last_active":"2024-04-25T16:01:06.443810+0000","last_peered":"2024-04-25T16:01:06.443810+0000","last_clean":"2024-04-25T16:01:06.443810+0000","last_became_active":"2024-04-25T15:59:02.507338+0000","last_became_peered":"2024-04-25T15:59:02.507338+0000","last_unstale":"2024-04-25T16:01:06.443810+0000","last_undegraded":"2024-04-25T16:01:06.443810+0000","last_fullsized":"2024-04-25T16:01:06.443810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:31.710709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":121,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.326104+0000","last_change":"2024-04-25T15:59:03.002590+0000","last_active":"2024-04-25T16:00:19.326104+0000","last_peered":"2024-04-25T16:00:19.326104+0000","last_clean":"2024-04-25T16:00:19.326104+0000","last_became_active":"2024-04-25T15:59:03.002262+0000","last_became_peered":"2024-04-25T15:59:03.002262+0000","last_unstale":"2024-04-25T16:00:19.326104+0000","last_undegraded":"2024-04-25T16:00:19.326104+0000","last_fullsized":"2024-04-25T16:00:19.326104+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:48.251484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671899+0000","last_change":"2024-04-25T15:59:02.518853+0000","last_active":"2024-04-25T15:59:50.671899+0000","last_peered":"2024-04-25T15:59:50.671899+0000","last_clean":"2024-04-25T15:59:50.671899+0000","last_became_active":"2024-04-25T15:59:02.518505+0000","last_became_peered":"2024-04-25T15:59:02.518505+0000","last_unstale":"2024-04-25T15:59:50.671899+0000","last_undegraded":"2024-04-25T15:59:50.671899+0000","last_fullsized":"2024-04-25T15:59:50.671899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:01:56.282245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345600+0000","last_change":"2024-04-25T15:59:03.009733+0000","last_active":"2024-04-25T16:00:19.345600+0000","last_peered":"2024-04-25T16:00:19.345600+0000","last_clean":"2024-04-25T16:00:19.345600+0000","last_became_active":"2024-04-25T15:59:03.009251+0000","last_became_peered":"2024-04-25T15:59:03.009251+0000","last_unstale":"2024-04-25T16:00:19.345600+0000","last_undegraded":"2024-04-25T16:00:19.345600+0000","last_fullsized":"2024-04-25T16:00:19.345600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:22.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"75'3","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670602+0000","last_change":"2024-04-25T15:59:02.512493+0000","last_active":"2024-04-25T15:59:50.670602+0000","last_peered":"2024-04-25T15:59:50.670602+0000","last_clean":"2024-04-25T15:59:50.670602+0000","last_became_active":"2024-04-25T15:59:02.512103+0000","last_became_peered":"2024-04-25T15:59:02.512103+0000","last_unstale":"2024-04-25T15:59:50.670602+0000","last_undegraded":"2024-04-25T15:59:50.670602+0000","last_fullsized":"2024-04-25T15:59:50.670602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:29.457755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342733+0000","last_change":"2024-04-25T15:59:03.001406+0000","last_active":"2024-04-25T16:00:19.342733+0000","last_peered":"2024-04-25T16:00:19.342733+0000","last_clean":"2024-04-25T16:00:19.342733+0000","last_became_active":"2024-04-25T15:59:02.999038+0000","last_became_peered":"2024-04-25T15:59:02.999038+0000","last_unstale":"2024-04-25T16:00:19.342733+0000","last_undegraded":"2024-04-25T16:00:19.342733+0000","last_fullsized":"2024-04-25T16:00:19.342733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:56.354821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671531+0000","last_change":"2024-04-25T15:59:02.509619+0000","last_active":"2024-04-25T15:59:50.671531+0000","last_peered":"2024-04-25T15:59:50.671531+0000","last_clean":"2024-04-25T15:59:50.671531+0000","last_became_active":"2024-04-25T15:59:02.509254+0000","last_became_peered":"2024-04-25T15:59:02.509254+0000","last_unstale":"2024-04-25T15:59:50.671531+0000","last_undegraded":"2024-04-25T15:59:50.671531+0000","last_fullsized":"2024-04-25T15:59:50.671531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:00.382581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347276+0000","last_change":"2024-04-25T15:59:03.000009+0000","last_active":"2024-04-25T16:00:19.347276+0000","last_peered":"2024-04-25T16:00:19.347276+0000","last_clean":"2024-04-25T16:00:19.347276+0000","last_became_active":"2024-04-25T15:59:02.999662+0000","last_became_peered":"2024-04-25T15:59:02.999662+0000","last_unstale":"2024-04-25T16:00:19.347276+0000","last_undegraded":"2024-04-25T16:00:19.347276+0000","last_fullsized":"2024-04-25T16:00:19.347276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:05:38.004819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"75'4","reported_seq":259,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.443147+0000","last_change":"2024-04-25T15:59:02.514429+0000","last_active":"2024-04-25T16:01:06.443147+0000","last_peered":"2024-04-25T16:01:06.443147+0000","last_clean":"2024-04-25T16:01:06.443147+0000","last_became_active":"2024-04-25T15:59:02.514144+0000","last_became_peered":"2024-04-25T15:59:02.514144+0000","last_unstale":"2024-04-25T16:01:06.443147+0000","last_undegraded":"2024-04-25T16:01:06.443147+0000","last_fullsized":"2024-04-25T16:01:06.443147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:45.144630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":236,"num_read_kb":236,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360393+0000","last_change":"2024-04-25T15:59:03.008460+0000","last_active":"2024-04-25T16:00:19.360393+0000","last_peered":"2024-04-25T16:00:19.360393+0000","last_clean":"2024-04-25T16:00:19.360393+0000","last_became_active":"2024-04-25T15:59:03.008024+0000","last_became_peered":"2024-04-25T15:59:03.008024+0000","last_unstale":"2024-04-25T16:00:19.360393+0000","last_undegraded":"2024-04-25T16:00:19.360393+0000","last_fullsized":"2024-04-25T16:00:19.360393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:10.288760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"75'4","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676828+0000","last_change":"2024-04-25T15:59:02.512494+0000","last_active":"2024-04-25T15:59:50.676828+0000","last_peered":"2024-04-25T15:59:50.676828+0000","last_clean":"2024-04-25T15:59:50.676828+0000","last_became_active":"2024-04-25T15:59:02.511182+0000","last_became_peered":"2024-04-25T15:59:02.511182+0000","last_unstale":"2024-04-25T15:59:50.676828+0000","last_undegraded":"2024-04-25T15:59:50.676828+0000","last_fullsized":"2024-04-25T15:59:50.676828+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:54.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350559+0000","last_change":"2024-04-25T15:59:03.003848+0000","last_active":"2024-04-25T16:00:19.350559+0000","last_peered":"2024-04-25T16:00:19.350559+0000","last_clean":"2024-04-25T16:00:19.350559+0000","last_became_active":"2024-04-25T15:59:03.003512+0000","last_became_peered":"2024-04-25T15:59:03.003512+0000","last_unstale":"2024-04-25T16:00:19.350559+0000","last_undegraded":"2024-04-25T16:00:19.350559+0000","last_fullsized":"2024-04-25T16:00:19.350559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:54.667054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672694+0000","last_change":"2024-04-25T15:59:02.509343+0000","last_active":"2024-04-25T15:59:50.672694+0000","last_peered":"2024-04-25T15:59:50.672694+0000","last_clean":"2024-04-25T15:59:50.672694+0000","last_became_active":"2024-04-25T15:59:02.508980+0000","last_became_peered":"2024-04-25T15:59:02.508980+0000","last_unstale":"2024-04-25T15:59:50.672694+0000","last_undegraded":"2024-04-25T15:59:50.672694+0000","last_fullsized":"2024-04-25T15:59:50.672694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:54.344717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359525+0000","last_change":"2024-04-25T15:59:02.999862+0000","last_active":"2024-04-25T16:00:19.359525+0000","last_peered":"2024-04-25T16:00:19.359525+0000","last_clean":"2024-04-25T16:00:19.359525+0000","last_became_active":"2024-04-25T15:59:02.999472+0000","last_became_peered":"2024-04-25T15:59:02.999472+0000","last_unstale":"2024-04-25T16:00:19.359525+0000","last_undegraded":"2024-04-25T16:00:19.359525+0000","last_fullsized":"2024-04-25T16:00:19.359525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:46.165045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"77'32","reported_seq":295,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.443183+0000","last_change":"2024-04-25T15:59:02.508949+0000","last_active":"2024-04-25T16:01:06.443183+0000","last_peered":"2024-04-25T16:01:06.443183+0000","last_clean":"2024-04-25T16:01:06.443183+0000","last_became_active":"2024-04-25T15:59:02.508564+0000","last_became_peered":"2024-04-25T15:59:02.508564+0000","last_unstale":"2024-04-25T16:01:06.443183+0000","last_undegraded":"2024-04-25T16:01:06.443183+0000","last_fullsized":"2024-04-25T16:01:06.443183+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:12.590193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57434,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":244,"num_read_kb":179,"num_write":34,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344171+0000","last_change":"2024-04-25T15:59:02.998575+0000","last_active":"2024-04-25T16:00:19.344171+0000","last_peered":"2024-04-25T16:00:19.344171+0000","last_clean":"2024-04-25T16:00:19.344171+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.344171+0000","last_undegraded":"2024-04-25T16:00:19.344171+0000","last_fullsized":"2024-04-25T16:00:19.344171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:40.640468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'9","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677368+0000","last_change":"2024-04-25T15:59:02.519110+0000","last_active":"2024-04-25T15:59:50.677368+0000","last_peered":"2024-04-25T15:59:50.677368+0000","last_clean":"2024-04-25T15:59:50.677368+0000","last_became_active":"2024-04-25T15:59:02.518373+0000","last_became_peered":"2024-04-25T15:59:02.518373+0000","last_unstale":"2024-04-25T15:59:50.677368+0000","last_undegraded":"2024-04-25T15:59:50.677368+0000","last_fullsized":"2024-04-25T15:59:50.677368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:27.840296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344110+0000","last_change":"2024-04-25T15:59:02.991002+0000","last_active":"2024-04-25T16:00:19.344110+0000","last_peered":"2024-04-25T16:00:19.344110+0000","last_clean":"2024-04-25T16:00:19.344110+0000","last_became_active":"2024-04-25T15:59:02.990605+0000","last_became_peered":"2024-04-25T15:59:02.990605+0000","last_unstale":"2024-04-25T16:00:19.344110+0000","last_undegraded":"2024-04-25T16:00:19.344110+0000","last_fullsized":"2024-04-25T16:00:19.344110+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:22:00.723255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671723+0000","last_change":"2024-04-25T15:59:02.509809+0000","last_active":"2024-04-25T15:59:50.671723+0000","last_peered":"2024-04-25T15:59:50.671723+0000","last_clean":"2024-04-25T15:59:50.671723+0000","last_became_active":"2024-04-25T15:59:02.509370+0000","last_became_peered":"2024-04-25T15:59:02.509370+0000","last_unstale":"2024-04-25T15:59:50.671723+0000","last_undegraded":"2024-04-25T15:59:50.671723+0000","last_fullsized":"2024-04-25T15:59:50.671723+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:24.416072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343672+0000","last_change":"2024-04-25T15:59:02.999836+0000","last_active":"2024-04-25T16:00:19.343672+0000","last_peered":"2024-04-25T16:00:19.343672+0000","last_clean":"2024-04-25T16:00:19.343672+0000","last_became_active":"2024-04-25T15:59:02.997770+0000","last_became_peered":"2024-04-25T15:59:02.997770+0000","last_unstale":"2024-04-25T16:00:19.343672+0000","last_undegraded":"2024-04-25T16:00:19.343672+0000","last_fullsized":"2024-04-25T16:00:19.343672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:30.953339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"77'7","reported_seq":148,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.462147+0000","last_change":"2024-04-25T15:59:02.510465+0000","last_active":"2024-04-25T16:01:06.462147+0000","last_peered":"2024-04-25T16:01:06.462147+0000","last_clean":"2024-04-25T16:01:06.462147+0000","last_became_active":"2024-04-25T15:59:02.510067+0000","last_became_peered":"2024-04-25T15:59:02.510067+0000","last_unstale":"2024-04-25T16:01:06.462147+0000","last_undegraded":"2024-04-25T16:01:06.462147+0000","last_fullsized":"2024-04-25T16:01:06.462147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:40.739358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346234+0000","last_change":"2024-04-25T15:59:03.000028+0000","last_active":"2024-04-25T16:00:19.346234+0000","last_peered":"2024-04-25T16:00:19.346234+0000","last_clean":"2024-04-25T16:00:19.346234+0000","last_became_active":"2024-04-25T15:59:02.999704+0000","last_became_peered":"2024-04-25T15:59:02.999704+0000","last_unstale":"2024-04-25T16:00:19.346234+0000","last_undegraded":"2024-04-25T16:00:19.346234+0000","last_fullsized":"2024-04-25T16:00:19.346234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:55.015085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'17","reported_seq":37,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669662+0000","last_change":"2024-04-25T15:59:02.514442+0000","last_active":"2024-04-25T15:59:50.669662+0000","last_peered":"2024-04-25T15:59:50.669662+0000","last_clean":"2024-04-25T15:59:50.669662+0000","last_became_active":"2024-04-25T15:59:02.513951+0000","last_became_peered":"2024-04-25T15:59:02.513951+0000","last_unstale":"2024-04-25T15:59:50.669662+0000","last_undegraded":"2024-04-25T15:59:50.669662+0000","last_fullsized":"2024-04-25T15:59:50.669662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:56.443999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"77'6","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.358196+0000","last_change":"2024-04-25T15:59:50.677155+0000","last_active":"2024-04-25T16:00:19.358196+0000","last_peered":"2024-04-25T16:00:19.358196+0000","last_clean":"2024-04-25T16:00:19.358196+0000","last_became_active":"2024-04-25T15:59:50.676721+0000","last_became_peered":"2024-04-25T15:59:50.676721+0000","last_unstale":"2024-04-25T16:00:19.358196+0000","last_undegraded":"2024-04-25T16:00:19.358196+0000","last_fullsized":"2024-04-25T16:00:19.358196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:04.138269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.683753+0000","last_change":"2024-04-25T15:59:50.683228+0000","last_active":"2024-04-25T15:59:50.683753+0000","last_peered":"2024-04-25T15:59:50.683753+0000","last_clean":"2024-04-25T15:59:50.683753+0000","last_became_active":"2024-04-25T15:59:50.675414+0000","last_became_peered":"2024-04-25T15:59:50.675414+0000","last_unstale":"2024-04-25T15:59:50.683753+0000","last_undegraded":"2024-04-25T15:59:50.683753+0000","last_fullsized":"2024-04-25T15:59:50.683753+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:46.577883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341606+0000","last_change":"2024-04-25T15:59:03.004639+0000","last_active":"2024-04-25T16:00:19.341606+0000","last_peered":"2024-04-25T16:00:19.341606+0000","last_clean":"2024-04-25T16:00:19.341606+0000","last_became_active":"2024-04-25T15:59:03.004260+0000","last_became_peered":"2024-04-25T15:59:03.004260+0000","last_unstale":"2024-04-25T16:00:19.341606+0000","last_undegraded":"2024-04-25T16:00:19.341606+0000","last_fullsized":"2024-04-25T16:00:19.341606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:00.450758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666396+0000","last_change":"2024-04-25T15:59:02.509716+0000","last_active":"2024-04-25T15:59:50.666396+0000","last_peered":"2024-04-25T15:59:50.666396+0000","last_clean":"2024-04-25T15:59:50.666396+0000","last_became_active":"2024-04-25T15:59:02.509303+0000","last_became_peered":"2024-04-25T15:59:02.509303+0000","last_unstale":"2024-04-25T15:59:50.666396+0000","last_undegraded":"2024-04-25T15:59:50.666396+0000","last_fullsized":"2024-04-25T15:59:50.666396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:34.037590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349639+0000","last_change":"2024-04-25T15:59:03.005141+0000","last_active":"2024-04-25T16:00:19.349639+0000","last_peered":"2024-04-25T16:00:19.349639+0000","last_clean":"2024-04-25T16:00:19.349639+0000","last_became_active":"2024-04-25T15:59:03.004760+0000","last_became_peered":"2024-04-25T15:59:03.004760+0000","last_unstale":"2024-04-25T16:00:19.349639+0000","last_undegraded":"2024-04-25T16:00:19.349639+0000","last_fullsized":"2024-04-25T16:00:19.349639+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:14.923126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"75'5","reported_seq":143,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.442998+0000","last_change":"2024-04-25T15:59:02.512779+0000","last_active":"2024-04-25T16:01:06.442998+0000","last_peered":"2024-04-25T16:01:06.442998+0000","last_clean":"2024-04-25T16:01:06.442998+0000","last_became_active":"2024-04-25T15:59:02.511240+0000","last_became_peered":"2024-04-25T15:59:02.511240+0000","last_unstale":"2024-04-25T16:01:06.442998+0000","last_undegraded":"2024-04-25T16:01:06.442998+0000","last_fullsized":"2024-04-25T16:01:06.442998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:51.703751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":118,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351726+0000","last_change":"2024-04-25T15:59:03.004841+0000","last_active":"2024-04-25T16:00:19.351726+0000","last_peered":"2024-04-25T16:00:19.351726+0000","last_clean":"2024-04-25T16:00:19.351726+0000","last_became_active":"2024-04-25T15:59:03.004488+0000","last_became_peered":"2024-04-25T15:59:03.004488+0000","last_unstale":"2024-04-25T16:00:19.351726+0000","last_undegraded":"2024-04-25T16:00:19.351726+0000","last_fullsized":"2024-04-25T16:00:19.351726+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:07.672312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345959+0000","last_change":"2024-04-25T15:59:03.001296+0000","last_active":"2024-04-25T16:00:19.345959+0000","last_peered":"2024-04-25T16:00:19.345959+0000","last_clean":"2024-04-25T16:00:19.345959+0000","last_became_active":"2024-04-25T15:59:03.000568+0000","last_became_peered":"2024-04-25T15:59:03.000568+0000","last_unstale":"2024-04-25T16:00:19.345959+0000","last_undegraded":"2024-04-25T16:00:19.345959+0000","last_fullsized":"2024-04-25T16:00:19.345959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:51.335849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"77'6","reported_seq":147,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:08.465129+0000","last_change":"2024-04-25T15:59:02.513943+0000","last_active":"2024-04-25T16:01:08.465129+0000","last_peered":"2024-04-25T16:01:08.465129+0000","last_clean":"2024-04-25T16:01:08.465129+0000","last_became_active":"2024-04-25T15:59:02.513595+0000","last_became_peered":"2024-04-25T15:59:02.513595+0000","last_unstale":"2024-04-25T16:01:08.465129+0000","last_undegraded":"2024-04-25T16:01:08.465129+0000","last_fullsized":"2024-04-25T16:01:08.465129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:50.720333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.352962+0000","last_change":"2024-04-25T15:59:03.008561+0000","last_active":"2024-04-25T16:00:19.352962+0000","last_peered":"2024-04-25T16:00:19.352962+0000","last_clean":"2024-04-25T16:00:19.352962+0000","last_became_active":"2024-04-25T15:59:03.008134+0000","last_became_peered":"2024-04-25T15:59:03.008134+0000","last_unstale":"2024-04-25T16:00:19.352962+0000","last_undegraded":"2024-04-25T16:00:19.352962+0000","last_fullsized":"2024-04-25T16:00:19.352962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:43.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"75'6","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669630+0000","last_change":"2024-04-25T15:59:02.510731+0000","last_active":"2024-04-25T15:59:50.669630+0000","last_peered":"2024-04-25T15:59:50.669630+0000","last_clean":"2024-04-25T15:59:50.669630+0000","last_became_active":"2024-04-25T15:59:02.510356+0000","last_became_peered":"2024-04-25T15:59:02.510356+0000","last_unstale":"2024-04-25T15:59:50.669630+0000","last_undegraded":"2024-04-25T15:59:50.669630+0000","last_fullsized":"2024-04-25T15:59:50.669630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:33.066858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":95,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677008+0000","last_change":"2024-04-25T15:57:47.988899+0000","last_active":"2024-04-25T15:59:50.677008+0000","last_peered":"2024-04-25T15:59:50.677008+0000","last_clean":"2024-04-25T15:59:50.677008+0000","last_became_active":"2024-04-25T15:57:47.982613+0000","last_became_peered":"2024-04-25T15:57:47.982613+0000","last_unstale":"2024-04-25T15:59:50.677008+0000","last_undegraded":"2024-04-25T15:59:50.677008+0000","last_fullsized":"2024-04-25T15:59:50.677008+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_clean_scrub_stamp":"2024-04-25T15:54:18.013489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:39.323767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"77'5","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341689+0000","last_change":"2024-04-25T15:59:02.998360+0000","last_active":"2024-04-25T16:00:19.341689+0000","last_peered":"2024-04-25T16:00:19.341689+0000","last_clean":"2024-04-25T16:00:19.341689+0000","last_became_active":"2024-04-25T15:59:02.998048+0000","last_became_peered":"2024-04-25T15:59:02.998048+0000","last_unstale":"2024-04-25T16:00:19.341689+0000","last_undegraded":"2024-04-25T16:00:19.341689+0000","last_fullsized":"2024-04-25T16:00:19.341689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:01.524608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677349+0000","last_change":"2024-04-25T15:59:02.515769+0000","last_active":"2024-04-25T15:59:50.677349+0000","last_peered":"2024-04-25T15:59:50.677349+0000","last_clean":"2024-04-25T15:59:50.677349+0000","last_became_active":"2024-04-25T15:59:02.515467+0000","last_became_peered":"2024-04-25T15:59:02.515467+0000","last_unstale":"2024-04-25T15:59:50.677349+0000","last_undegraded":"2024-04-25T15:59:50.677349+0000","last_fullsized":"2024-04-25T15:59:50.677349+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:59:50.107475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"75'4","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670140+0000","last_change":"2024-04-25T15:59:02.512701+0000","last_active":"2024-04-25T15:59:50.670140+0000","last_peered":"2024-04-25T15:59:50.670140+0000","last_clean":"2024-04-25T15:59:50.670140+0000","last_became_active":"2024-04-25T15:59:02.512352+0000","last_became_peered":"2024-04-25T15:59:02.512352+0000","last_unstale":"2024-04-25T15:59:50.670140+0000","last_undegraded":"2024-04-25T15:59:50.670140+0000","last_fullsized":"2024-04-25T15:59:50.670140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:31.772382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"77'9","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.330017+0000","last_change":"2024-04-25T15:59:02.989698+0000","last_active":"2024-04-25T16:00:19.330017+0000","last_peered":"2024-04-25T16:00:19.330017+0000","last_clean":"2024-04-25T16:00:19.330017+0000","last_became_active":"2024-04-25T15:59:02.989284+0000","last_became_peered":"2024-04-25T15:59:02.989284+0000","last_unstale":"2024-04-25T16:00:19.330017+0000","last_undegraded":"2024-04-25T16:00:19.330017+0000","last_fullsized":"2024-04-25T16:00:19.330017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:12.501738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"77'13","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666592+0000","last_change":"2024-04-25T15:59:02.517272+0000","last_active":"2024-04-25T15:59:50.666592+0000","last_peered":"2024-04-25T15:59:50.666592+0000","last_clean":"2024-04-25T15:59:50.666592+0000","last_became_active":"2024-04-25T15:59:02.516370+0000","last_became_peered":"2024-04-25T15:59:02.516370+0000","last_unstale":"2024-04-25T15:59:50.666592+0000","last_undegraded":"2024-04-25T15:59:50.666592+0000","last_fullsized":"2024-04-25T15:59:50.666592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:31.139689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345612+0000","last_change":"2024-04-25T15:59:03.003111+0000","last_active":"2024-04-25T16:00:19.345612+0000","last_peered":"2024-04-25T16:00:19.345612+0000","last_clean":"2024-04-25T16:00:19.345612+0000","last_became_active":"2024-04-25T15:59:03.002766+0000","last_became_peered":"2024-04-25T15:59:03.002766+0000","last_unstale":"2024-04-25T16:00:19.345612+0000","last_undegraded":"2024-04-25T16:00:19.345612+0000","last_fullsized":"2024-04-25T16:00:19.345612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:39.866119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'7","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670151+0000","last_change":"2024-04-25T15:59:02.521970+0000","last_active":"2024-04-25T15:59:50.670151+0000","last_peered":"2024-04-25T15:59:50.670151+0000","last_clean":"2024-04-25T15:59:50.670151+0000","last_became_active":"2024-04-25T15:59:02.521650+0000","last_became_peered":"2024-04-25T15:59:02.521650+0000","last_unstale":"2024-04-25T15:59:50.670151+0000","last_undegraded":"2024-04-25T15:59:50.670151+0000","last_fullsized":"2024-04-25T15:59:50.670151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:32.996868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351310+0000","last_change":"2024-04-25T15:59:02.996358+0000","last_active":"2024-04-25T16:00:19.351310+0000","last_peered":"2024-04-25T16:00:19.351310+0000","last_clean":"2024-04-25T16:00:19.351310+0000","last_became_active":"2024-04-25T15:59:02.995988+0000","last_became_peered":"2024-04-25T15:59:02.995988+0000","last_unstale":"2024-04-25T16:00:19.351310+0000","last_undegraded":"2024-04-25T16:00:19.351310+0000","last_fullsized":"2024-04-25T16:00:19.351310+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:51.021841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"75'2","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.674464+0000","last_change":"2024-04-25T15:59:02.518330+0000","last_active":"2024-04-25T15:59:50.674464+0000","last_peered":"2024-04-25T15:59:50.674464+0000","last_clean":"2024-04-25T15:59:50.674464+0000","last_became_active":"2024-04-25T15:59:02.517978+0000","last_became_peered":"2024-04-25T15:59:02.517978+0000","last_unstale":"2024-04-25T15:59:50.674464+0000","last_undegraded":"2024-04-25T15:59:50.674464+0000","last_fullsized":"2024-04-25T15:59:50.674464+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:25.443963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.302105+0000","last_change":"2024-04-25T15:59:03.000198+0000","last_active":"2024-04-25T16:00:19.302105+0000","last_peered":"2024-04-25T16:00:19.302105+0000","last_clean":"2024-04-25T16:00:19.302105+0000","last_became_active":"2024-04-25T15:59:02.999878+0000","last_became_peered":"2024-04-25T15:59:02.999878+0000","last_unstale":"2024-04-25T16:00:19.302105+0000","last_undegraded":"2024-04-25T16:00:19.302105+0000","last_fullsized":"2024-04-25T16:00:19.302105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:37.966595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807467+0000","last_change":"2024-04-25T15:59:02.511298+0000","last_active":"2024-04-25T15:59:50.807467+0000","last_peered":"2024-04-25T15:59:50.807467+0000","last_clean":"2024-04-25T15:59:50.807467+0000","last_became_active":"2024-04-25T15:59:02.510521+0000","last_became_peered":"2024-04-25T15:59:02.510521+0000","last_unstale":"2024-04-25T15:59:50.807467+0000","last_undegraded":"2024-04-25T15:59:50.807467+0000","last_fullsized":"2024-04-25T15:59:50.807467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:00.807919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347593+0000","last_change":"2024-04-25T15:59:02.999212+0000","last_active":"2024-04-25T16:00:19.347593+0000","last_peered":"2024-04-25T16:00:19.347593+0000","last_clean":"2024-04-25T16:00:19.347593+0000","last_became_active":"2024-04-25T15:59:02.998881+0000","last_became_peered":"2024-04-25T15:59:02.998881+0000","last_unstale":"2024-04-25T16:00:19.347593+0000","last_undegraded":"2024-04-25T16:00:19.347593+0000","last_fullsized":"2024-04-25T16:00:19.347593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:09.092196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"75'3","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676880+0000","last_change":"2024-04-25T15:59:02.517412+0000","last_active":"2024-04-25T15:59:50.676880+0000","last_peered":"2024-04-25T15:59:50.676880+0000","last_clean":"2024-04-25T15:59:50.676880+0000","last_became_active":"2024-04-25T15:59:02.517077+0000","last_became_peered":"2024-04-25T15:59:02.517077+0000","last_unstale":"2024-04-25T15:59:50.676880+0000","last_undegraded":"2024-04-25T15:59:50.676880+0000","last_fullsized":"2024-04-25T15:59:50.676880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:00.161199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"77'8","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342365+0000","last_change":"2024-04-25T15:59:50.678844+0000","last_active":"2024-04-25T16:00:19.342365+0000","last_peered":"2024-04-25T16:00:19.342365+0000","last_clean":"2024-04-25T16:00:19.342365+0000","last_became_active":"2024-04-25T15:59:50.678411+0000","last_became_peered":"2024-04-25T15:59:50.678411+0000","last_unstale":"2024-04-25T16:00:19.342365+0000","last_undegraded":"2024-04-25T16:00:19.342365+0000","last_fullsized":"2024-04-25T16:00:19.342365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:56:48.018678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669652+0000","last_change":"2024-04-25T15:59:02.515912+0000","last_active":"2024-04-25T15:59:50.669652+0000","last_peered":"2024-04-25T15:59:50.669652+0000","last_clean":"2024-04-25T15:59:50.669652+0000","last_became_active":"2024-04-25T15:59:02.514949+0000","last_became_peered":"2024-04-25T15:59:02.514949+0000","last_unstale":"2024-04-25T15:59:50.669652+0000","last_undegraded":"2024-04-25T15:59:50.669652+0000","last_fullsized":"2024-04-25T15:59:50.669652+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:33.467223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360414+0000","last_change":"2024-04-25T15:59:02.988896+0000","last_active":"2024-04-25T16:00:19.360414+0000","last_peered":"2024-04-25T16:00:19.360414+0000","last_clean":"2024-04-25T16:00:19.360414+0000","last_became_active":"2024-04-25T15:59:02.988478+0000","last_became_peered":"2024-04-25T15:59:02.988478+0000","last_unstale":"2024-04-25T16:00:19.360414+0000","last_undegraded":"2024-04-25T16:00:19.360414+0000","last_fullsized":"2024-04-25T16:00:19.360414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:47.014678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"75'5","reported_seq":97,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:06.781861+0000","last_change":"2024-04-25T15:59:50.687632+0000","last_active":"2024-04-25T16:01:06.781861+0000","last_peered":"2024-04-25T16:01:06.781861+0000","last_clean":"2024-04-25T16:01:06.781861+0000","last_became_active":"2024-04-25T15:59:50.680402+0000","last_became_peered":"2024-04-25T15:59:50.680402+0000","last_unstale":"2024-04-25T16:01:06.781861+0000","last_undegraded":"2024-04-25T16:01:06.781861+0000","last_fullsized":"2024-04-25T16:01:06.781861+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:12.544489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359317+0000","last_change":"2024-04-25T15:59:02.990507+0000","last_active":"2024-04-25T16:00:19.359317+0000","last_peered":"2024-04-25T16:00:19.359317+0000","last_clean":"2024-04-25T16:00:19.359317+0000","last_became_active":"2024-04-25T15:59:02.990160+0000","last_became_peered":"2024-04-25T15:59:02.990160+0000","last_unstale":"2024-04-25T16:00:19.359317+0000","last_undegraded":"2024-04-25T16:00:19.359317+0000","last_fullsized":"2024-04-25T16:00:19.359317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:18:21.526155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672520+0000","last_change":"2024-04-25T15:59:02.510956+0000","last_active":"2024-04-25T15:59:50.672520+0000","last_peered":"2024-04-25T15:59:50.672520+0000","last_clean":"2024-04-25T15:59:50.672520+0000","last_became_active":"2024-04-25T15:59:02.510578+0000","last_became_peered":"2024-04-25T15:59:02.510578+0000","last_unstale":"2024-04-25T15:59:50.672520+0000","last_undegraded":"2024-04-25T15:59:50.672520+0000","last_fullsized":"2024-04-25T15:59:50.672520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:11.512365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337102+0000","last_change":"2024-04-25T15:59:03.001122+0000","last_active":"2024-04-25T16:00:19.337102+0000","last_peered":"2024-04-25T16:00:19.337102+0000","last_clean":"2024-04-25T16:00:19.337102+0000","last_became_active":"2024-04-25T15:59:03.000672+0000","last_became_peered":"2024-04-25T15:59:03.000672+0000","last_unstale":"2024-04-25T16:00:19.337102+0000","last_undegraded":"2024-04-25T16:00:19.337102+0000","last_fullsized":"2024-04-25T16:00:19.337102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:33.216434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'5","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670676+0000","last_change":"2024-04-25T15:59:02.520645+0000","last_active":"2024-04-25T15:59:50.670676+0000","last_peered":"2024-04-25T15:59:50.670676+0000","last_clean":"2024-04-25T15:59:50.670676+0000","last_became_active":"2024-04-25T15:59:02.520197+0000","last_became_peered":"2024-04-25T15:59:02.520197+0000","last_unstale":"2024-04-25T15:59:50.670676+0000","last_undegraded":"2024-04-25T15:59:50.670676+0000","last_fullsized":"2024-04-25T15:59:50.670676+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:20.120554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347691+0000","last_change":"2024-04-25T15:59:03.001668+0000","last_active":"2024-04-25T16:00:19.347691+0000","last_peered":"2024-04-25T16:00:19.347691+0000","last_clean":"2024-04-25T16:00:19.347691+0000","last_became_active":"2024-04-25T15:59:03.001333+0000","last_became_peered":"2024-04-25T15:59:03.001333+0000","last_unstale":"2024-04-25T16:00:19.347691+0000","last_undegraded":"2024-04-25T16:00:19.347691+0000","last_fullsized":"2024-04-25T16:00:19.347691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:24.992476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346081+0000","last_change":"2024-04-25T15:59:03.005533+0000","last_active":"2024-04-25T16:00:19.346081+0000","last_peered":"2024-04-25T16:00:19.346081+0000","last_clean":"2024-04-25T16:00:19.346081+0000","last_became_active":"2024-04-25T15:59:03.004124+0000","last_became_peered":"2024-04-25T15:59:03.004124+0000","last_unstale":"2024-04-25T16:00:19.346081+0000","last_undegraded":"2024-04-25T16:00:19.346081+0000","last_fullsized":"2024-04-25T16:00:19.346081+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:27:28.507065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"77'12","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.331274+0000","last_change":"2024-04-25T15:59:02.993416+0000","last_active":"2024-04-25T16:00:19.331274+0000","last_peered":"2024-04-25T16:00:19.331274+0000","last_clean":"2024-04-25T16:00:19.331274+0000","last_became_active":"2024-04-25T15:59:02.993098+0000","last_became_peered":"2024-04-25T15:59:02.993098+0000","last_unstale":"2024-04-25T16:00:19.331274+0000","last_undegraded":"2024-04-25T16:00:19.331274+0000","last_fullsized":"2024-04-25T16:00:19.331274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:46.859932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350142+0000","last_change":"2024-04-25T15:59:02.996858+0000","last_active":"2024-04-25T16:00:19.350142+0000","last_peered":"2024-04-25T16:00:19.350142+0000","last_clean":"2024-04-25T16:00:19.350142+0000","last_became_active":"2024-04-25T15:59:02.996530+0000","last_became_peered":"2024-04-25T15:59:02.996530+0000","last_unstale":"2024-04-25T16:00:19.350142+0000","last_undegraded":"2024-04-25T16:00:19.350142+0000","last_fullsized":"2024-04-25T16:00:19.350142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:36.723948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"77'7","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350676+0000","last_change":"2024-04-25T15:59:02.995942+0000","last_active":"2024-04-25T16:00:19.350676+0000","last_peered":"2024-04-25T16:00:19.350676+0000","last_clean":"2024-04-25T16:00:19.350676+0000","last_became_active":"2024-04-25T15:59:02.995567+0000","last_became_peered":"2024-04-25T15:59:02.995567+0000","last_unstale":"2024-04-25T16:00:19.350676+0000","last_undegraded":"2024-04-25T16:00:19.350676+0000","last_fullsized":"2024-04-25T16:00:19.350676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:32.692392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350261+0000","last_change":"2024-04-25T15:59:03.000326+0000","last_active":"2024-04-25T16:00:19.350261+0000","last_peered":"2024-04-25T16:00:19.350261+0000","last_clean":"2024-04-25T16:00:19.350261+0000","last_became_active":"2024-04-25T15:59:02.999961+0000","last_became_peered":"2024-04-25T15:59:02.999961+0000","last_unstale":"2024-04-25T16:00:19.350261+0000","last_undegraded":"2024-04-25T16:00:19.350261+0000","last_fullsized":"2024-04-25T16:00:19.350261+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:10.631460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341696+0000","last_change":"2024-04-25T15:59:03.003625+0000","last_active":"2024-04-25T16:00:19.341696+0000","last_peered":"2024-04-25T16:00:19.341696+0000","last_clean":"2024-04-25T16:00:19.341696+0000","last_became_active":"2024-04-25T15:59:03.003280+0000","last_became_peered":"2024-04-25T15:59:03.003280+0000","last_unstale":"2024-04-25T16:00:19.341696+0000","last_undegraded":"2024-04-25T16:00:19.341696+0000","last_fullsized":"2024-04-25T16:00:19.341696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:21:54.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346578+0000","last_change":"2024-04-25T15:59:02.994645+0000","last_active":"2024-04-25T16:00:19.346578+0000","last_peered":"2024-04-25T16:00:19.346578+0000","last_clean":"2024-04-25T16:00:19.346578+0000","last_became_active":"2024-04-25T15:59:02.993924+0000","last_became_peered":"2024-04-25T15:59:02.993924+0000","last_unstale":"2024-04-25T16:00:19.346578+0000","last_undegraded":"2024-04-25T16:00:19.346578+0000","last_fullsized":"2024-04-25T16:00:19.346578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.644708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340331+0000","last_change":"2024-04-25T15:59:03.006156+0000","last_active":"2024-04-25T16:00:19.340331+0000","last_peered":"2024-04-25T16:00:19.340331+0000","last_clean":"2024-04-25T16:00:19.340331+0000","last_became_active":"2024-04-25T15:59:03.005795+0000","last_became_peered":"2024-04-25T15:59:03.005795+0000","last_unstale":"2024-04-25T16:00:19.340331+0000","last_undegraded":"2024-04-25T16:00:19.340331+0000","last_fullsized":"2024-04-25T16:00:19.340331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:23.523868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337613+0000","last_change":"2024-04-25T15:59:03.010404+0000","last_active":"2024-04-25T16:00:19.337613+0000","last_peered":"2024-04-25T16:00:19.337613+0000","last_clean":"2024-04-25T16:00:19.337613+0000","last_became_active":"2024-04-25T15:59:03.010100+0000","last_became_peered":"2024-04-25T15:59:03.010100+0000","last_unstale":"2024-04-25T16:00:19.337613+0000","last_undegraded":"2024-04-25T16:00:19.337613+0000","last_fullsized":"2024-04-25T16:00:19.337613+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:22.718842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.335498+0000","last_change":"2024-04-25T15:59:03.009240+0000","last_active":"2024-04-25T16:00:19.335498+0000","last_peered":"2024-04-25T16:00:19.335498+0000","last_clean":"2024-04-25T16:00:19.335498+0000","last_became_active":"2024-04-25T15:59:03.008723+0000","last_became_peered":"2024-04-25T15:59:03.008723+0000","last_unstale":"2024-04-25T16:00:19.335498+0000","last_undegraded":"2024-04-25T16:00:19.335498+0000","last_fullsized":"2024-04-25T16:00:19.335498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:14.049934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359019+0000","last_change":"2024-04-25T15:59:02.994359+0000","last_active":"2024-04-25T16:00:19.359019+0000","last_peered":"2024-04-25T16:00:19.359019+0000","last_clean":"2024-04-25T16:00:19.359019+0000","last_became_active":"2024-04-25T15:59:02.993990+0000","last_became_peered":"2024-04-25T15:59:02.993990+0000","last_unstale":"2024-04-25T16:00:19.359019+0000","last_undegraded":"2024-04-25T16:00:19.359019+0000","last_fullsized":"2024-04-25T16:00:19.359019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:55.477414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350058+0000","last_change":"2024-04-25T15:59:02.995510+0000","last_active":"2024-04-25T16:00:19.350058+0000","last_peered":"2024-04-25T16:00:19.350058+0000","last_clean":"2024-04-25T16:00:19.350058+0000","last_became_active":"2024-04-25T15:59:02.995008+0000","last_became_peered":"2024-04-25T15:59:02.995008+0000","last_unstale":"2024-04-25T16:00:19.350058+0000","last_undegraded":"2024-04-25T16:00:19.350058+0000","last_fullsized":"2024-04-25T16:00:19.350058+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:40.439615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341453+0000","last_change":"2024-04-25T15:59:03.005846+0000","last_active":"2024-04-25T16:00:19.341453+0000","last_peered":"2024-04-25T16:00:19.341453+0000","last_clean":"2024-04-25T16:00:19.341453+0000","last_became_active":"2024-04-25T15:59:03.003779+0000","last_became_peered":"2024-04-25T15:59:03.003779+0000","last_unstale":"2024-04-25T16:00:19.341453+0000","last_undegraded":"2024-04-25T16:00:19.341453+0000","last_fullsized":"2024-04-25T16:00:19.341453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.698405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247542,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64388,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1180,"num_read_kb":682,"num_write":242,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1192,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":193472,"data_compressed":5607,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710757,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":839680,"data_stored":402102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906988,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":1051075,"data_compressed":37695,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070513,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":402010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266743,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":851968,"data_stored":402366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462972,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":995047,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626497,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":401727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574939,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":461244,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":403066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411414,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":459124,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":401552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986252,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":402120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822726,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":994193,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60356,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58236,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":59702,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:01:13.242 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-25T16:01:13.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-25T16:01:13.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.390+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/3532026522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 msgr2=0x7ff9e0104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.390+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/3532026522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 0x7ff9e0104800 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc009900 tx=0x7ff9dc02a410 comp rx=0 tx=0).stop 2024-04-25T16:01:13.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.390+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/3532026522 shutdown_connections 2024-04-25T16:01:13.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.390+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/3532026522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0109a40 0x7ff9e010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.390+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/3532026522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e0104d40 0x7ff9e0105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.390+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/3532026522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 0x7ff9e0104800 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.390+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/3532026522 >> 172.21.15.31:0/3532026522 conn(0x7ff9e00fed10 msgr2=0x7ff9e01011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:13.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.390+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/3532026522 shutdown_connections 2024-04-25T16:01:13.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.390+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/3532026522 wait complete. 2024-04-25T16:01:13.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e7ed8640 1 Processor -- start 2024-04-25T16:01:13.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e7ed8640 1 -- start start 2024-04-25T16:01:13.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e7ed8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 0x7ff9e019a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e7ed8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0104d40 0x7ff9e019a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e7ed8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e0109a40 0x7ff9e01a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e7ed8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e010f600 con 0x7ff9e0103b00 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e7ed8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e010f480 con 0x7ff9e0104d40 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e7ed8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff9e010f780 con 0x7ff9e0109a40 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e5c74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 0x7ff9e019a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e5c74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 0x7ff9e019a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56342/0 (socket says 172.21.15.31:56342) 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e5c74640 1 -- 172.21.15.31:0/2292497765 learned_addr learned my addr 172.21.15.31:0/2292497765 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.391+0000 7ff9e6475640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e0109a40 0x7ff9e01a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e5473640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0104d40 0x7ff9e019a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e5c74640 1 -- 172.21.15.31:0/2292497765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e0109a40 msgr2=0x7ff9e01a1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e5c74640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e0109a40 0x7ff9e01a1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e5c74640 1 -- 172.21.15.31:0/2292497765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0104d40 msgr2=0x7ff9e019a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e5c74640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0104d40 0x7ff9e019a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e5c74640 1 -- 172.21.15.31:0/2292497765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9e01a1da0 con 0x7ff9e0103b00 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e6475640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e0109a40 0x7ff9e01a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e5473640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0104d40 0x7ff9e019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e5c74640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 0x7ff9e019a260 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc006010 tx=0x7ff9dc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9d6ffd640 1 -- 172.21.15.31:0/2292497765 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc03f030 con 0x7ff9e0103b00 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9d6ffd640 1 -- 172.21.15.31:0/2292497765 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff9dc003aa0 con 0x7ff9e0103b00 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9e01a2030 con 0x7ff9e0103b00 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9d6ffd640 1 -- 172.21.15.31:0/2292497765 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9dc035070 con 0x7ff9e0103b00 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.392+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9e01a2520 con 0x7ff9e0103b00 2024-04-25T16:01:13.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.393+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9e0108850 con 0x7ff9e0103b00 2024-04-25T16:01:13.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.394+0000 7ff9d6ffd640 1 -- 172.21.15.31:0/2292497765 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff9dc042030 con 0x7ff9e0103b00 2024-04-25T16:01:13.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.395+0000 7ff9d6ffd640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff9cc0afaf0 0x7ff9cc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.395+0000 7ff9e5473640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff9cc0afaf0 0x7ff9cc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.395+0000 7ff9d6ffd640 1 -- 172.21.15.31:0/2292497765 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff9dc0ff770 con 0x7ff9e0103b00 2024-04-25T16:01:13.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.396+0000 7ff9e5473640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff9cc0afaf0 0x7ff9cc0b2000 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff9e019b300 tx=0x7ff9d0005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:13.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.399+0000 7ff9d6ffd640 1 -- 172.21.15.31:0/2292497765 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff9dc0c8360 con 0x7ff9e0103b00 2024-04-25T16:01:13.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.584+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 --> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff9e0062fe0 con 0x7ff9cc0afaf0 2024-04-25T16:01:13.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.591+0000 7ff9d6ffd640 1 -- 172.21.15.31:0/2292497765 <== mgr.14152 v2:172.21.15.31:6800/3984865329 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259072 (secure 0 0 0) 0x7ff9e0062fe0 con 0x7ff9cc0afaf0 2024-04-25T16:01:13.595 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:13.596 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2024-04-25T16:01:13.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff9cc0afaf0 msgr2=0x7ff9cc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff9cc0afaf0 0x7ff9cc0b2000 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff9e019b300 tx=0x7ff9d0005e70 comp rx=0 tx=0).stop 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 msgr2=0x7ff9e019a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 0x7ff9e019a260 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc006010 tx=0x7ff9dc004270 comp rx=0 tx=0).stop 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 shutdown_connections 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff9cc0afaf0 0x7ff9cc0b2000 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9e0109a40 0x7ff9e01a1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff9e0104d40 0x7ff9e019a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 --2- 172.21.15.31:0/2292497765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff9e0103b00 0x7ff9e019a260 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 >> 172.21.15.31:0/2292497765 conn(0x7ff9e00fed10 msgr2=0x7ff9e0105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 shutdown_connections 2024-04-25T16:01:13.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.594+0000 7ff9e7ed8640 1 -- 172.21.15.31:0/2292497765 wait complete. 2024-04-25T16:01:13.607 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-04-25T16:01:13.348058+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":655606,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1298,"num_read_kb":763,"num_write":373,"num_write_kb":2291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2953032,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":809,"ondisk_log_size":809,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355336,"kb_used_data":10344,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124586488,"statfs":{"total":1151940427776,"available":1151576563712,"internally_reserved":0,"allocated":10592256,"data_stored":6777166,"data_compressed":113085,"data_compressed_allocated":147456,"data_compressed_original":1794048,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":142,"num_read_kb":67,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"3.3f","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354985+0000","last_change":"2024-04-25T15:59:02.997678+0000","last_active":"2024-04-25T16:00:19.354985+0000","last_peered":"2024-04-25T16:00:19.354985+0000","last_clean":"2024-04-25T16:00:19.354985+0000","last_became_active":"2024-04-25T15:59:02.997335+0000","last_became_peered":"2024-04-25T15:59:02.997335+0000","last_unstale":"2024-04-25T16:00:19.354985+0000","last_undegraded":"2024-04-25T16:00:19.354985+0000","last_fullsized":"2024-04-25T16:00:19.354985+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:36:33.221819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"77'4","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351614+0000","last_change":"2024-04-25T15:59:03.001049+0000","last_active":"2024-04-25T16:00:19.351614+0000","last_peered":"2024-04-25T16:00:19.351614+0000","last_clean":"2024-04-25T16:00:19.351614+0000","last_became_active":"2024-04-25T15:59:03.000663+0000","last_became_peered":"2024-04-25T15:59:03.000663+0000","last_unstale":"2024-04-25T16:00:19.351614+0000","last_undegraded":"2024-04-25T16:00:19.351614+0000","last_fullsized":"2024-04-25T16:00:19.351614+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:04:36.670519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"77'7","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354238+0000","last_change":"2024-04-25T15:59:03.003861+0000","last_active":"2024-04-25T16:00:19.354238+0000","last_peered":"2024-04-25T16:00:19.354238+0000","last_clean":"2024-04-25T16:00:19.354238+0000","last_became_active":"2024-04-25T15:59:03.003508+0000","last_became_peered":"2024-04-25T15:59:03.003508+0000","last_unstale":"2024-04-25T16:00:19.354238+0000","last_undegraded":"2024-04-25T16:00:19.354238+0000","last_fullsized":"2024-04-25T16:00:19.354238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:48:25.679389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337369+0000","last_change":"2024-04-25T15:59:03.003330+0000","last_active":"2024-04-25T16:00:19.337369+0000","last_peered":"2024-04-25T16:00:19.337369+0000","last_clean":"2024-04-25T16:00:19.337369+0000","last_became_active":"2024-04-25T15:59:03.002872+0000","last_became_peered":"2024-04-25T15:59:03.002872+0000","last_unstale":"2024-04-25T16:00:19.337369+0000","last_undegraded":"2024-04-25T16:00:19.337369+0000","last_fullsized":"2024-04-25T16:00:19.337369+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:19:02.359046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.324906+0000","last_change":"2024-04-25T15:59:02.997684+0000","last_active":"2024-04-25T16:00:19.324906+0000","last_peered":"2024-04-25T16:00:19.324906+0000","last_clean":"2024-04-25T16:00:19.324906+0000","last_became_active":"2024-04-25T15:59:02.997370+0000","last_became_peered":"2024-04-25T15:59:02.997370+0000","last_unstale":"2024-04-25T16:00:19.324906+0000","last_undegraded":"2024-04-25T16:00:19.324906+0000","last_fullsized":"2024-04-25T16:00:19.324906+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:24.349085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348756+0000","last_change":"2024-04-25T15:59:03.003638+0000","last_active":"2024-04-25T16:00:19.348756+0000","last_peered":"2024-04-25T16:00:19.348756+0000","last_clean":"2024-04-25T16:00:19.348756+0000","last_became_active":"2024-04-25T15:59:03.003299+0000","last_became_peered":"2024-04-25T15:59:03.003299+0000","last_unstale":"2024-04-25T16:00:19.348756+0000","last_undegraded":"2024-04-25T16:00:19.348756+0000","last_fullsized":"2024-04-25T16:00:19.348756+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:16:46.842345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.338851+0000","last_change":"2024-04-25T15:59:02.998591+0000","last_active":"2024-04-25T16:00:19.338851+0000","last_peered":"2024-04-25T16:00:19.338851+0000","last_clean":"2024-04-25T16:00:19.338851+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.338851+0000","last_undegraded":"2024-04-25T16:00:19.338851+0000","last_fullsized":"2024-04-25T16:00:19.338851+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:51:48.444499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343142+0000","last_change":"2024-04-25T15:59:03.006225+0000","last_active":"2024-04-25T16:00:19.343142+0000","last_peered":"2024-04-25T16:00:19.343142+0000","last_clean":"2024-04-25T16:00:19.343142+0000","last_became_active":"2024-04-25T15:59:03.005883+0000","last_became_peered":"2024-04-25T15:59:03.005883+0000","last_unstale":"2024-04-25T16:00:19.343142+0000","last_undegraded":"2024-04-25T16:00:19.343142+0000","last_fullsized":"2024-04-25T16:00:19.343142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:37:17.773196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359702+0000","last_change":"2024-04-25T15:59:03.003605+0000","last_active":"2024-04-25T16:00:19.359702+0000","last_peered":"2024-04-25T16:00:19.359702+0000","last_clean":"2024-04-25T16:00:19.359702+0000","last_became_active":"2024-04-25T15:59:03.003271+0000","last_became_peered":"2024-04-25T15:59:03.003271+0000","last_unstale":"2024-04-25T16:00:19.359702+0000","last_undegraded":"2024-04-25T16:00:19.359702+0000","last_fullsized":"2024-04-25T16:00:19.359702+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:28.576407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'16","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.348846+0000","last_change":"2024-04-25T15:59:50.679099+0000","last_active":"2024-04-25T16:00:19.348846+0000","last_peered":"2024-04-25T16:00:19.348846+0000","last_clean":"2024-04-25T16:00:19.348846+0000","last_became_active":"2024-04-25T15:59:50.678558+0000","last_became_peered":"2024-04-25T15:59:50.678558+0000","last_unstale":"2024-04-25T16:00:19.348846+0000","last_undegraded":"2024-04-25T16:00:19.348846+0000","last_fullsized":"2024-04-25T16:00:19.348846+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:17:40.691050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359322+0000","last_change":"2024-04-25T15:59:02.999307+0000","last_active":"2024-04-25T16:00:19.359322+0000","last_peered":"2024-04-25T16:00:19.359322+0000","last_clean":"2024-04-25T16:00:19.359322+0000","last_became_active":"2024-04-25T15:59:02.998976+0000","last_became_peered":"2024-04-25T15:59:02.998976+0000","last_unstale":"2024-04-25T16:00:19.359322+0000","last_undegraded":"2024-04-25T16:00:19.359322+0000","last_fullsized":"2024-04-25T16:00:19.359322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:36:29.626440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347700+0000","last_change":"2024-04-25T15:59:03.005208+0000","last_active":"2024-04-25T16:00:19.347700+0000","last_peered":"2024-04-25T16:00:19.347700+0000","last_clean":"2024-04-25T16:00:19.347700+0000","last_became_active":"2024-04-25T15:59:03.004869+0000","last_became_peered":"2024-04-25T15:59:03.004869+0000","last_unstale":"2024-04-25T16:00:19.347700+0000","last_undegraded":"2024-04-25T16:00:19.347700+0000","last_fullsized":"2024-04-25T16:00:19.347700+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:00:24.040763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"77'10","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349448+0000","last_change":"2024-04-25T15:59:50.673556+0000","last_active":"2024-04-25T16:00:19.349448+0000","last_peered":"2024-04-25T16:00:19.349448+0000","last_clean":"2024-04-25T16:00:19.349448+0000","last_became_active":"2024-04-25T15:59:50.673119+0000","last_became_peered":"2024-04-25T15:59:50.673119+0000","last_unstale":"2024-04-25T16:00:19.349448+0000","last_undegraded":"2024-04-25T16:00:19.349448+0000","last_fullsized":"2024-04-25T16:00:19.349448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:09:50.286968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350359+0000","last_change":"2024-04-25T15:59:03.002160+0000","last_active":"2024-04-25T16:00:19.350359+0000","last_peered":"2024-04-25T16:00:19.350359+0000","last_clean":"2024-04-25T16:00:19.350359+0000","last_became_active":"2024-04-25T15:59:03.001855+0000","last_became_peered":"2024-04-25T15:59:03.001855+0000","last_unstale":"2024-04-25T16:00:19.350359+0000","last_undegraded":"2024-04-25T16:00:19.350359+0000","last_fullsized":"2024-04-25T16:00:19.350359+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:14:43.383851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350513+0000","last_change":"2024-04-25T15:59:02.995318+0000","last_active":"2024-04-25T16:00:19.350513+0000","last_peered":"2024-04-25T16:00:19.350513+0000","last_clean":"2024-04-25T16:00:19.350513+0000","last_became_active":"2024-04-25T15:59:02.994956+0000","last_became_peered":"2024-04-25T15:59:02.994956+0000","last_unstale":"2024-04-25T16:00:19.350513+0000","last_undegraded":"2024-04-25T16:00:19.350513+0000","last_fullsized":"2024-04-25T16:00:19.350513+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:05:59.659400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359091+0000","last_change":"2024-04-25T15:59:02.996485+0000","last_active":"2024-04-25T16:00:19.359091+0000","last_peered":"2024-04-25T16:00:19.359091+0000","last_clean":"2024-04-25T16:00:19.359091+0000","last_became_active":"2024-04-25T15:59:02.996011+0000","last_became_peered":"2024-04-25T15:59:02.996011+0000","last_unstale":"2024-04-25T16:00:19.359091+0000","last_undegraded":"2024-04-25T16:00:19.359091+0000","last_fullsized":"2024-04-25T16:00:19.359091+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:51:56.025254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.355086+0000","last_change":"2024-04-25T15:59:02.998455+0000","last_active":"2024-04-25T16:00:19.355086+0000","last_peered":"2024-04-25T16:00:19.355086+0000","last_clean":"2024-04-25T16:00:19.355086+0000","last_became_active":"2024-04-25T15:59:02.998126+0000","last_became_peered":"2024-04-25T15:59:02.998126+0000","last_unstale":"2024-04-25T16:00:19.355086+0000","last_undegraded":"2024-04-25T16:00:19.355086+0000","last_fullsized":"2024-04-25T16:00:19.355086+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:18:20.342413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350929+0000","last_change":"2024-04-25T15:59:02.998766+0000","last_active":"2024-04-25T16:00:19.350929+0000","last_peered":"2024-04-25T16:00:19.350929+0000","last_clean":"2024-04-25T16:00:19.350929+0000","last_became_active":"2024-04-25T15:59:02.998415+0000","last_became_peered":"2024-04-25T15:59:02.998415+0000","last_unstale":"2024-04-25T16:00:19.350929+0000","last_undegraded":"2024-04-25T16:00:19.350929+0000","last_fullsized":"2024-04-25T16:00:19.350929+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:55:05.717847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340892+0000","last_change":"2024-04-25T15:59:02.994982+0000","last_active":"2024-04-25T16:00:19.340892+0000","last_peered":"2024-04-25T16:00:19.340892+0000","last_clean":"2024-04-25T16:00:19.340892+0000","last_became_active":"2024-04-25T15:59:02.993928+0000","last_became_peered":"2024-04-25T15:59:02.993928+0000","last_unstale":"2024-04-25T16:00:19.340892+0000","last_undegraded":"2024-04-25T16:00:19.340892+0000","last_fullsized":"2024-04-25T16:00:19.340892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:22:25.346541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"75'13","reported_seq":153,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:08.445335+0000","last_change":"2024-04-25T15:59:02.512828+0000","last_active":"2024-04-25T16:01:08.445335+0000","last_peered":"2024-04-25T16:01:08.445335+0000","last_clean":"2024-04-25T16:01:08.445335+0000","last_became_active":"2024-04-25T15:59:02.512291+0000","last_became_peered":"2024-04-25T15:59:02.512291+0000","last_unstale":"2024-04-25T16:01:08.445335+0000","last_undegraded":"2024-04-25T16:01:08.445335+0000","last_fullsized":"2024-04-25T16:01:08.445335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:01:02.372140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":120,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"77'5","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340094+0000","last_change":"2024-04-25T15:59:03.008472+0000","last_active":"2024-04-25T16:00:19.340094+0000","last_peered":"2024-04-25T16:00:19.340094+0000","last_clean":"2024-04-25T16:00:19.340094+0000","last_became_active":"2024-04-25T15:59:03.008108+0000","last_became_peered":"2024-04-25T15:59:03.008108+0000","last_unstale":"2024-04-25T16:00:19.340094+0000","last_undegraded":"2024-04-25T16:00:19.340094+0000","last_fullsized":"2024-04-25T16:00:19.340094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:57:40.087592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"75'2","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807946+0000","last_change":"2024-04-25T15:59:02.517846+0000","last_active":"2024-04-25T15:59:50.807946+0000","last_peered":"2024-04-25T15:59:50.807946+0000","last_clean":"2024-04-25T15:59:50.807946+0000","last_became_active":"2024-04-25T15:59:02.517366+0000","last_became_peered":"2024-04-25T15:59:02.517366+0000","last_unstale":"2024-04-25T15:59:50.807946+0000","last_undegraded":"2024-04-25T15:59:50.807946+0000","last_fullsized":"2024-04-25T15:59:50.807946+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:04:29.620827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"77'6","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.354259+0000","last_change":"2024-04-25T15:59:03.005202+0000","last_active":"2024-04-25T16:00:19.354259+0000","last_peered":"2024-04-25T16:00:19.354259+0000","last_clean":"2024-04-25T16:00:19.354259+0000","last_became_active":"2024-04-25T15:59:03.003447+0000","last_became_peered":"2024-04-25T15:59:03.003447+0000","last_unstale":"2024-04-25T16:00:19.354259+0000","last_undegraded":"2024-04-25T16:00:19.354259+0000","last_fullsized":"2024-04-25T16:00:19.354259+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:16:28.740016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"75'12","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.703214+0000","last_change":"2024-04-25T15:59:50.702764+0000","last_active":"2024-04-25T15:59:50.703214+0000","last_peered":"2024-04-25T15:59:50.703214+0000","last_clean":"2024-04-25T15:59:50.703214+0000","last_became_active":"2024-04-25T15:59:50.679231+0000","last_became_peered":"2024-04-25T15:59:50.679231+0000","last_unstale":"2024-04-25T15:59:50.703214+0000","last_undegraded":"2024-04-25T15:59:50.703214+0000","last_fullsized":"2024-04-25T15:59:50.703214+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:24:37.561992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,7],"acting":[11,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360223+0000","last_change":"2024-04-25T15:59:03.004210+0000","last_active":"2024-04-25T16:00:19.360223+0000","last_peered":"2024-04-25T16:00:19.360223+0000","last_clean":"2024-04-25T16:00:19.360223+0000","last_became_active":"2024-04-25T15:59:03.002769+0000","last_became_peered":"2024-04-25T15:59:03.002769+0000","last_unstale":"2024-04-25T16:00:19.360223+0000","last_undegraded":"2024-04-25T16:00:19.360223+0000","last_fullsized":"2024-04-25T16:00:19.360223+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:44:50.608327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669960+0000","last_change":"2024-04-25T15:59:02.506199+0000","last_active":"2024-04-25T15:59:50.669960+0000","last_peered":"2024-04-25T15:59:50.669960+0000","last_clean":"2024-04-25T15:59:50.669960+0000","last_became_active":"2024-04-25T15:59:02.505847+0000","last_became_peered":"2024-04-25T15:59:02.505847+0000","last_unstale":"2024-04-25T15:59:50.669960+0000","last_undegraded":"2024-04-25T15:59:50.669960+0000","last_fullsized":"2024-04-25T15:59:50.669960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:33:35.227112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"77'15","reported_seq":38,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359258+0000","last_change":"2024-04-25T15:59:50.679040+0000","last_active":"2024-04-25T16:00:19.359258+0000","last_peered":"2024-04-25T16:00:19.359258+0000","last_clean":"2024-04-25T16:00:19.359258+0000","last_became_active":"2024-04-25T15:59:50.678094+0000","last_became_peered":"2024-04-25T15:59:50.678094+0000","last_unstale":"2024-04-25T16:00:19.359258+0000","last_undegraded":"2024-04-25T16:00:19.359258+0000","last_fullsized":"2024-04-25T16:00:19.359258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:10:11.957966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"75'5","reported_seq":150,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:11.451194+0000","last_change":"2024-04-25T15:59:02.507763+0000","last_active":"2024-04-25T16:01:11.451194+0000","last_peered":"2024-04-25T16:01:11.451194+0000","last_clean":"2024-04-25T16:01:11.451194+0000","last_became_active":"2024-04-25T15:59:02.507338+0000","last_became_peered":"2024-04-25T15:59:02.507338+0000","last_unstale":"2024-04-25T16:01:11.451194+0000","last_undegraded":"2024-04-25T16:01:11.451194+0000","last_fullsized":"2024-04-25T16:01:11.451194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:43:31.710709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.326104+0000","last_change":"2024-04-25T15:59:03.002590+0000","last_active":"2024-04-25T16:00:19.326104+0000","last_peered":"2024-04-25T16:00:19.326104+0000","last_clean":"2024-04-25T16:00:19.326104+0000","last_became_active":"2024-04-25T15:59:03.002262+0000","last_became_peered":"2024-04-25T15:59:03.002262+0000","last_unstale":"2024-04-25T16:00:19.326104+0000","last_undegraded":"2024-04-25T16:00:19.326104+0000","last_fullsized":"2024-04-25T16:00:19.326104+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:49:48.251484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671899+0000","last_change":"2024-04-25T15:59:02.518853+0000","last_active":"2024-04-25T15:59:50.671899+0000","last_peered":"2024-04-25T15:59:50.671899+0000","last_clean":"2024-04-25T15:59:50.671899+0000","last_became_active":"2024-04-25T15:59:02.518505+0000","last_became_peered":"2024-04-25T15:59:02.518505+0000","last_unstale":"2024-04-25T15:59:50.671899+0000","last_undegraded":"2024-04-25T15:59:50.671899+0000","last_fullsized":"2024-04-25T15:59:50.671899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:01:56.282245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345600+0000","last_change":"2024-04-25T15:59:03.009733+0000","last_active":"2024-04-25T16:00:19.345600+0000","last_peered":"2024-04-25T16:00:19.345600+0000","last_clean":"2024-04-25T16:00:19.345600+0000","last_became_active":"2024-04-25T15:59:03.009251+0000","last_became_peered":"2024-04-25T15:59:03.009251+0000","last_unstale":"2024-04-25T16:00:19.345600+0000","last_undegraded":"2024-04-25T16:00:19.345600+0000","last_fullsized":"2024-04-25T16:00:19.345600+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:35:22.117921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"75'3","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670602+0000","last_change":"2024-04-25T15:59:02.512493+0000","last_active":"2024-04-25T15:59:50.670602+0000","last_peered":"2024-04-25T15:59:50.670602+0000","last_clean":"2024-04-25T15:59:50.670602+0000","last_became_active":"2024-04-25T15:59:02.512103+0000","last_became_peered":"2024-04-25T15:59:02.512103+0000","last_unstale":"2024-04-25T15:59:50.670602+0000","last_undegraded":"2024-04-25T15:59:50.670602+0000","last_fullsized":"2024-04-25T15:59:50.670602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:46:29.457755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342733+0000","last_change":"2024-04-25T15:59:03.001406+0000","last_active":"2024-04-25T16:00:19.342733+0000","last_peered":"2024-04-25T16:00:19.342733+0000","last_clean":"2024-04-25T16:00:19.342733+0000","last_became_active":"2024-04-25T15:59:02.999038+0000","last_became_peered":"2024-04-25T15:59:02.999038+0000","last_unstale":"2024-04-25T16:00:19.342733+0000","last_undegraded":"2024-04-25T16:00:19.342733+0000","last_fullsized":"2024-04-25T16:00:19.342733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:41:56.354821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671531+0000","last_change":"2024-04-25T15:59:02.509619+0000","last_active":"2024-04-25T15:59:50.671531+0000","last_peered":"2024-04-25T15:59:50.671531+0000","last_clean":"2024-04-25T15:59:50.671531+0000","last_became_active":"2024-04-25T15:59:02.509254+0000","last_became_peered":"2024-04-25T15:59:02.509254+0000","last_unstale":"2024-04-25T15:59:50.671531+0000","last_undegraded":"2024-04-25T15:59:50.671531+0000","last_fullsized":"2024-04-25T15:59:50.671531+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:45:00.382581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347276+0000","last_change":"2024-04-25T15:59:03.000009+0000","last_active":"2024-04-25T16:00:19.347276+0000","last_peered":"2024-04-25T16:00:19.347276+0000","last_clean":"2024-04-25T16:00:19.347276+0000","last_became_active":"2024-04-25T15:59:02.999662+0000","last_became_peered":"2024-04-25T15:59:02.999662+0000","last_unstale":"2024-04-25T16:00:19.347276+0000","last_undegraded":"2024-04-25T16:00:19.347276+0000","last_fullsized":"2024-04-25T16:00:19.347276+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:05:38.004819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"77'5","reported_seq":270,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:11.450417+0000","last_change":"2024-04-25T15:59:02.514429+0000","last_active":"2024-04-25T16:01:11.450417+0000","last_peered":"2024-04-25T16:01:11.450417+0000","last_clean":"2024-04-25T16:01:11.450417+0000","last_became_active":"2024-04-25T15:59:02.514144+0000","last_became_peered":"2024-04-25T15:59:02.514144+0000","last_unstale":"2024-04-25T16:01:11.450417+0000","last_undegraded":"2024-04-25T16:01:11.450417+0000","last_fullsized":"2024-04-25T16:01:11.450417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:16:45.144630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":246,"num_read_kb":246,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360393+0000","last_change":"2024-04-25T15:59:03.008460+0000","last_active":"2024-04-25T16:00:19.360393+0000","last_peered":"2024-04-25T16:00:19.360393+0000","last_clean":"2024-04-25T16:00:19.360393+0000","last_became_active":"2024-04-25T15:59:03.008024+0000","last_became_peered":"2024-04-25T15:59:03.008024+0000","last_unstale":"2024-04-25T16:00:19.360393+0000","last_undegraded":"2024-04-25T16:00:19.360393+0000","last_fullsized":"2024-04-25T16:00:19.360393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:38:10.288760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"75'4","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.676828+0000","last_change":"2024-04-25T15:59:02.512494+0000","last_active":"2024-04-25T15:59:50.676828+0000","last_peered":"2024-04-25T15:59:50.676828+0000","last_clean":"2024-04-25T15:59:50.676828+0000","last_became_active":"2024-04-25T15:59:02.511182+0000","last_became_peered":"2024-04-25T15:59:02.511182+0000","last_unstale":"2024-04-25T15:59:50.676828+0000","last_undegraded":"2024-04-25T15:59:50.676828+0000","last_fullsized":"2024-04-25T15:59:50.676828+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:02:54.572097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350559+0000","last_change":"2024-04-25T15:59:03.003848+0000","last_active":"2024-04-25T16:00:19.350559+0000","last_peered":"2024-04-25T16:00:19.350559+0000","last_clean":"2024-04-25T16:00:19.350559+0000","last_became_active":"2024-04-25T15:59:03.003512+0000","last_became_peered":"2024-04-25T15:59:03.003512+0000","last_unstale":"2024-04-25T16:00:19.350559+0000","last_undegraded":"2024-04-25T16:00:19.350559+0000","last_fullsized":"2024-04-25T16:00:19.350559+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:09:54.667054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672694+0000","last_change":"2024-04-25T15:59:02.509343+0000","last_active":"2024-04-25T15:59:50.672694+0000","last_peered":"2024-04-25T15:59:50.672694+0000","last_clean":"2024-04-25T15:59:50.672694+0000","last_became_active":"2024-04-25T15:59:02.508980+0000","last_became_peered":"2024-04-25T15:59:02.508980+0000","last_unstale":"2024-04-25T15:59:50.672694+0000","last_undegraded":"2024-04-25T15:59:50.672694+0000","last_fullsized":"2024-04-25T15:59:50.672694+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:30:54.344717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359525+0000","last_change":"2024-04-25T15:59:02.999862+0000","last_active":"2024-04-25T16:00:19.359525+0000","last_peered":"2024-04-25T16:00:19.359525+0000","last_clean":"2024-04-25T16:00:19.359525+0000","last_became_active":"2024-04-25T15:59:02.999472+0000","last_became_peered":"2024-04-25T15:59:02.999472+0000","last_unstale":"2024-04-25T16:00:19.359525+0000","last_undegraded":"2024-04-25T16:00:19.359525+0000","last_fullsized":"2024-04-25T16:00:19.359525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:27:46.165045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"77'34","reported_seq":308,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:11.450897+0000","last_change":"2024-04-25T15:59:02.508949+0000","last_active":"2024-04-25T16:01:11.450897+0000","last_peered":"2024-04-25T16:01:11.450897+0000","last_clean":"2024-04-25T16:01:11.450897+0000","last_became_active":"2024-04-25T15:59:02.508564+0000","last_became_peered":"2024-04-25T15:59:02.508564+0000","last_unstale":"2024-04-25T16:01:11.450897+0000","last_undegraded":"2024-04-25T16:01:11.450897+0000","last_fullsized":"2024-04-25T16:01:11.450897+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:44:12.590193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":57742,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":255,"num_read_kb":185,"num_write":36,"num_write_kb":71,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344171+0000","last_change":"2024-04-25T15:59:02.998575+0000","last_active":"2024-04-25T16:00:19.344171+0000","last_peered":"2024-04-25T16:00:19.344171+0000","last_clean":"2024-04-25T16:00:19.344171+0000","last_became_active":"2024-04-25T15:59:02.998241+0000","last_became_peered":"2024-04-25T15:59:02.998241+0000","last_unstale":"2024-04-25T16:00:19.344171+0000","last_undegraded":"2024-04-25T16:00:19.344171+0000","last_fullsized":"2024-04-25T16:00:19.344171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:49:40.640468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"75'9","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677368+0000","last_change":"2024-04-25T15:59:02.519110+0000","last_active":"2024-04-25T15:59:50.677368+0000","last_peered":"2024-04-25T15:59:50.677368+0000","last_clean":"2024-04-25T15:59:50.677368+0000","last_became_active":"2024-04-25T15:59:02.518373+0000","last_became_peered":"2024-04-25T15:59:02.518373+0000","last_unstale":"2024-04-25T15:59:50.677368+0000","last_undegraded":"2024-04-25T15:59:50.677368+0000","last_fullsized":"2024-04-25T15:59:50.677368+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:27:27.840296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.344110+0000","last_change":"2024-04-25T15:59:02.991002+0000","last_active":"2024-04-25T16:00:19.344110+0000","last_peered":"2024-04-25T16:00:19.344110+0000","last_clean":"2024-04-25T16:00:19.344110+0000","last_became_active":"2024-04-25T15:59:02.990605+0000","last_became_peered":"2024-04-25T15:59:02.990605+0000","last_unstale":"2024-04-25T16:00:19.344110+0000","last_undegraded":"2024-04-25T16:00:19.344110+0000","last_fullsized":"2024-04-25T16:00:19.344110+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:22:00.723255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"75'5","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.671723+0000","last_change":"2024-04-25T15:59:02.509809+0000","last_active":"2024-04-25T15:59:50.671723+0000","last_peered":"2024-04-25T15:59:50.671723+0000","last_clean":"2024-04-25T15:59:50.671723+0000","last_became_active":"2024-04-25T15:59:02.509370+0000","last_became_peered":"2024-04-25T15:59:02.509370+0000","last_unstale":"2024-04-25T15:59:50.671723+0000","last_undegraded":"2024-04-25T15:59:50.671723+0000","last_fullsized":"2024-04-25T15:59:50.671723+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:39:24.416072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.343672+0000","last_change":"2024-04-25T15:59:02.999836+0000","last_active":"2024-04-25T16:00:19.343672+0000","last_peered":"2024-04-25T16:00:19.343672+0000","last_clean":"2024-04-25T16:00:19.343672+0000","last_became_active":"2024-04-25T15:59:02.997770+0000","last_became_peered":"2024-04-25T15:59:02.997770+0000","last_unstale":"2024-04-25T16:00:19.343672+0000","last_undegraded":"2024-04-25T16:00:19.343672+0000","last_fullsized":"2024-04-25T16:00:19.343672+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:10:30.953339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"77'7","reported_seq":153,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:11.470959+0000","last_change":"2024-04-25T15:59:02.510465+0000","last_active":"2024-04-25T16:01:11.470959+0000","last_peered":"2024-04-25T16:01:11.470959+0000","last_clean":"2024-04-25T16:01:11.470959+0000","last_became_active":"2024-04-25T15:59:02.510067+0000","last_became_peered":"2024-04-25T15:59:02.510067+0000","last_unstale":"2024-04-25T16:01:11.470959+0000","last_undegraded":"2024-04-25T16:01:11.470959+0000","last_fullsized":"2024-04-25T16:01:11.470959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:35:40.739358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":128,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346234+0000","last_change":"2024-04-25T15:59:03.000028+0000","last_active":"2024-04-25T16:00:19.346234+0000","last_peered":"2024-04-25T16:00:19.346234+0000","last_clean":"2024-04-25T16:00:19.346234+0000","last_became_active":"2024-04-25T15:59:02.999704+0000","last_became_peered":"2024-04-25T15:59:02.999704+0000","last_unstale":"2024-04-25T16:00:19.346234+0000","last_undegraded":"2024-04-25T16:00:19.346234+0000","last_fullsized":"2024-04-25T16:00:19.346234+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:54:55.015085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"75'17","reported_seq":37,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669662+0000","last_change":"2024-04-25T15:59:02.514442+0000","last_active":"2024-04-25T15:59:50.669662+0000","last_peered":"2024-04-25T15:59:50.669662+0000","last_clean":"2024-04-25T15:59:50.669662+0000","last_became_active":"2024-04-25T15:59:02.513951+0000","last_became_peered":"2024-04-25T15:59:02.513951+0000","last_unstale":"2024-04-25T15:59:50.669662+0000","last_undegraded":"2024-04-25T15:59:50.669662+0000","last_fullsized":"2024-04-25T15:59:50.669662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:07:56.443999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"77'6","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.358196+0000","last_change":"2024-04-25T15:59:50.677155+0000","last_active":"2024-04-25T16:00:19.358196+0000","last_peered":"2024-04-25T16:00:19.358196+0000","last_clean":"2024-04-25T16:00:19.358196+0000","last_became_active":"2024-04-25T15:59:50.676721+0000","last_became_peered":"2024-04-25T15:59:50.676721+0000","last_unstale":"2024-04-25T16:00:19.358196+0000","last_undegraded":"2024-04-25T16:00:19.358196+0000","last_fullsized":"2024-04-25T16:00:19.358196+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:37:04.138269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"75'2","reported_seq":17,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.683753+0000","last_change":"2024-04-25T15:59:50.683228+0000","last_active":"2024-04-25T15:59:50.683753+0000","last_peered":"2024-04-25T15:59:50.683753+0000","last_clean":"2024-04-25T15:59:50.683753+0000","last_became_active":"2024-04-25T15:59:50.675414+0000","last_became_peered":"2024-04-25T15:59:50.675414+0000","last_unstale":"2024-04-25T15:59:50.683753+0000","last_undegraded":"2024-04-25T15:59:50.683753+0000","last_fullsized":"2024-04-25T15:59:50.683753+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:31:46.577883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341606+0000","last_change":"2024-04-25T15:59:03.004639+0000","last_active":"2024-04-25T16:00:19.341606+0000","last_peered":"2024-04-25T16:00:19.341606+0000","last_clean":"2024-04-25T16:00:19.341606+0000","last_became_active":"2024-04-25T15:59:03.004260+0000","last_became_peered":"2024-04-25T15:59:03.004260+0000","last_unstale":"2024-04-25T16:00:19.341606+0000","last_undegraded":"2024-04-25T16:00:19.341606+0000","last_fullsized":"2024-04-25T16:00:19.341606+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:22:00.450758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"75'1","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666396+0000","last_change":"2024-04-25T15:59:02.509716+0000","last_active":"2024-04-25T15:59:50.666396+0000","last_peered":"2024-04-25T15:59:50.666396+0000","last_clean":"2024-04-25T15:59:50.666396+0000","last_became_active":"2024-04-25T15:59:02.509303+0000","last_became_peered":"2024-04-25T15:59:02.509303+0000","last_unstale":"2024-04-25T15:59:50.666396+0000","last_undegraded":"2024-04-25T15:59:50.666396+0000","last_fullsized":"2024-04-25T15:59:50.666396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:55:34.037590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"77'5","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.349983+0000","last_change":"2024-04-25T15:59:03.005141+0000","last_active":"2024-04-25T16:00:19.349983+0000","last_peered":"2024-04-25T16:00:19.349983+0000","last_clean":"2024-04-25T16:00:19.349983+0000","last_became_active":"2024-04-25T15:59:03.004760+0000","last_became_peered":"2024-04-25T15:59:03.004760+0000","last_unstale":"2024-04-25T16:00:19.349983+0000","last_undegraded":"2024-04-25T16:00:19.349983+0000","last_fullsized":"2024-04-25T16:00:19.349983+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:24:14.923126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"75'5","reported_seq":148,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:10.448709+0000","last_change":"2024-04-25T15:59:02.512779+0000","last_active":"2024-04-25T16:01:10.448709+0000","last_peered":"2024-04-25T16:01:10.448709+0000","last_clean":"2024-04-25T16:01:10.448709+0000","last_became_active":"2024-04-25T15:59:02.511240+0000","last_became_peered":"2024-04-25T15:59:02.511240+0000","last_unstale":"2024-04-25T16:01:10.448709+0000","last_undegraded":"2024-04-25T16:01:10.448709+0000","last_fullsized":"2024-04-25T16:01:10.448709+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:15:51.703751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":123,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"77'7","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351921+0000","last_change":"2024-04-25T15:59:03.004841+0000","last_active":"2024-04-25T16:00:19.351921+0000","last_peered":"2024-04-25T16:00:19.351921+0000","last_clean":"2024-04-25T16:00:19.351921+0000","last_became_active":"2024-04-25T15:59:03.004488+0000","last_became_peered":"2024-04-25T15:59:03.004488+0000","last_unstale":"2024-04-25T16:00:19.351921+0000","last_undegraded":"2024-04-25T16:00:19.351921+0000","last_fullsized":"2024-04-25T16:00:19.351921+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:33:07.672312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345959+0000","last_change":"2024-04-25T15:59:03.001296+0000","last_active":"2024-04-25T16:00:19.345959+0000","last_peered":"2024-04-25T16:00:19.345959+0000","last_clean":"2024-04-25T16:00:19.345959+0000","last_became_active":"2024-04-25T15:59:03.000568+0000","last_became_peered":"2024-04-25T15:59:03.000568+0000","last_unstale":"2024-04-25T16:00:19.345959+0000","last_undegraded":"2024-04-25T16:00:19.345959+0000","last_fullsized":"2024-04-25T16:00:19.345959+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:51.335849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"77'6","reported_seq":147,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:08.465129+0000","last_change":"2024-04-25T15:59:02.513943+0000","last_active":"2024-04-25T16:01:08.465129+0000","last_peered":"2024-04-25T16:01:08.465129+0000","last_clean":"2024-04-25T16:01:08.465129+0000","last_became_active":"2024-04-25T15:59:02.513595+0000","last_became_peered":"2024-04-25T15:59:02.513595+0000","last_unstale":"2024-04-25T16:01:08.465129+0000","last_undegraded":"2024-04-25T16:01:08.465129+0000","last_fullsized":"2024-04-25T16:01:08.465129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:21:50.720333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":121,"num_read_kb":121,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.352962+0000","last_change":"2024-04-25T15:59:03.008561+0000","last_active":"2024-04-25T16:00:19.352962+0000","last_peered":"2024-04-25T16:00:19.352962+0000","last_clean":"2024-04-25T16:00:19.352962+0000","last_became_active":"2024-04-25T15:59:03.008134+0000","last_became_peered":"2024-04-25T15:59:03.008134+0000","last_unstale":"2024-04-25T16:00:19.352962+0000","last_undegraded":"2024-04-25T16:00:19.352962+0000","last_fullsized":"2024-04-25T16:00:19.352962+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:56:43.740721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"75'6","reported_seq":29,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669630+0000","last_change":"2024-04-25T15:59:02.510731+0000","last_active":"2024-04-25T15:59:50.669630+0000","last_peered":"2024-04-25T15:59:50.669630+0000","last_clean":"2024-04-25T15:59:50.669630+0000","last_became_active":"2024-04-25T15:59:02.510356+0000","last_became_peered":"2024-04-25T15:59:02.510356+0000","last_unstale":"2024-04-25T15:59:50.669630+0000","last_undegraded":"2024-04-25T15:59:50.669630+0000","last_fullsized":"2024-04-25T15:59:50.669630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T19:56:33.066858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":95,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677008+0000","last_change":"2024-04-25T15:57:47.988899+0000","last_active":"2024-04-25T15:59:50.677008+0000","last_peered":"2024-04-25T15:59:50.677008+0000","last_clean":"2024-04-25T15:59:50.677008+0000","last_became_active":"2024-04-25T15:57:47.982613+0000","last_became_peered":"2024-04-25T15:57:47.982613+0000","last_unstale":"2024-04-25T15:59:50.677008+0000","last_undegraded":"2024-04-25T15:59:50.677008+0000","last_fullsized":"2024-04-25T15:59:50.677008+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:54:18.013489+0000","last_clean_scrub_stamp":"2024-04-25T15:54:18.013489+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:38:39.323767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"77'5","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341689+0000","last_change":"2024-04-25T15:59:02.998360+0000","last_active":"2024-04-25T16:00:19.341689+0000","last_peered":"2024-04-25T16:00:19.341689+0000","last_clean":"2024-04-25T16:00:19.341689+0000","last_became_active":"2024-04-25T15:59:02.998048+0000","last_became_peered":"2024-04-25T15:59:02.998048+0000","last_unstale":"2024-04-25T16:00:19.341689+0000","last_undegraded":"2024-04-25T16:00:19.341689+0000","last_fullsized":"2024-04-25T16:00:19.341689+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:50:01.524608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.677349+0000","last_change":"2024-04-25T15:59:02.515769+0000","last_active":"2024-04-25T15:59:50.677349+0000","last_peered":"2024-04-25T15:59:50.677349+0000","last_clean":"2024-04-25T15:59:50.677349+0000","last_became_active":"2024-04-25T15:59:02.515467+0000","last_became_peered":"2024-04-25T15:59:02.515467+0000","last_unstale":"2024-04-25T15:59:50.677349+0000","last_undegraded":"2024-04-25T15:59:50.677349+0000","last_fullsized":"2024-04-25T15:59:50.677349+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:59:50.107475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"75'4","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670140+0000","last_change":"2024-04-25T15:59:02.512701+0000","last_active":"2024-04-25T15:59:50.670140+0000","last_peered":"2024-04-25T15:59:50.670140+0000","last_clean":"2024-04-25T15:59:50.670140+0000","last_became_active":"2024-04-25T15:59:02.512352+0000","last_became_peered":"2024-04-25T15:59:02.512352+0000","last_unstale":"2024-04-25T15:59:50.670140+0000","last_undegraded":"2024-04-25T15:59:50.670140+0000","last_fullsized":"2024-04-25T15:59:50.670140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:39:31.772382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"77'9","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.330017+0000","last_change":"2024-04-25T15:59:02.989698+0000","last_active":"2024-04-25T16:00:19.330017+0000","last_peered":"2024-04-25T16:00:19.330017+0000","last_clean":"2024-04-25T16:00:19.330017+0000","last_became_active":"2024-04-25T15:59:02.989284+0000","last_became_peered":"2024-04-25T15:59:02.989284+0000","last_unstale":"2024-04-25T16:00:19.330017+0000","last_undegraded":"2024-04-25T16:00:19.330017+0000","last_fullsized":"2024-04-25T16:00:19.330017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:08:12.501738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"77'13","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.666592+0000","last_change":"2024-04-25T15:59:02.517272+0000","last_active":"2024-04-25T15:59:50.666592+0000","last_peered":"2024-04-25T15:59:50.666592+0000","last_clean":"2024-04-25T15:59:50.666592+0000","last_became_active":"2024-04-25T15:59:02.516370+0000","last_became_peered":"2024-04-25T15:59:02.516370+0000","last_unstale":"2024-04-25T15:59:50.666592+0000","last_undegraded":"2024-04-25T15:59:50.666592+0000","last_fullsized":"2024-04-25T15:59:50.666592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:42:31.139689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.345612+0000","last_change":"2024-04-25T15:59:03.003111+0000","last_active":"2024-04-25T16:00:19.345612+0000","last_peered":"2024-04-25T16:00:19.345612+0000","last_clean":"2024-04-25T16:00:19.345612+0000","last_became_active":"2024-04-25T15:59:03.002766+0000","last_became_peered":"2024-04-25T15:59:03.002766+0000","last_unstale":"2024-04-25T16:00:19.345612+0000","last_undegraded":"2024-04-25T16:00:19.345612+0000","last_fullsized":"2024-04-25T16:00:19.345612+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:50:39.866119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"75'7","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670214+0000","last_change":"2024-04-25T15:59:02.521970+0000","last_active":"2024-04-25T15:59:50.670214+0000","last_peered":"2024-04-25T15:59:50.670214+0000","last_clean":"2024-04-25T15:59:50.670214+0000","last_became_active":"2024-04-25T15:59:02.521650+0000","last_became_peered":"2024-04-25T15:59:02.521650+0000","last_unstale":"2024-04-25T15:59:50.670214+0000","last_undegraded":"2024-04-25T15:59:50.670214+0000","last_fullsized":"2024-04-25T15:59:50.670214+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:10:32.996868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.351310+0000","last_change":"2024-04-25T15:59:02.996358+0000","last_active":"2024-04-25T16:00:19.351310+0000","last_peered":"2024-04-25T16:00:19.351310+0000","last_clean":"2024-04-25T16:00:19.351310+0000","last_became_active":"2024-04-25T15:59:02.995988+0000","last_became_peered":"2024-04-25T15:59:02.995988+0000","last_unstale":"2024-04-25T16:00:19.351310+0000","last_undegraded":"2024-04-25T16:00:19.351310+0000","last_fullsized":"2024-04-25T16:00:19.351310+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:36:51.021841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"75'2","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.674559+0000","last_change":"2024-04-25T15:59:02.518330+0000","last_active":"2024-04-25T15:59:50.674559+0000","last_peered":"2024-04-25T15:59:50.674559+0000","last_clean":"2024-04-25T15:59:50.674559+0000","last_became_active":"2024-04-25T15:59:02.517978+0000","last_became_peered":"2024-04-25T15:59:02.517978+0000","last_unstale":"2024-04-25T15:59:50.674559+0000","last_undegraded":"2024-04-25T15:59:50.674559+0000","last_fullsized":"2024-04-25T15:59:50.674559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:16:25.443963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.302105+0000","last_change":"2024-04-25T15:59:03.000198+0000","last_active":"2024-04-25T16:00:19.302105+0000","last_peered":"2024-04-25T16:00:19.302105+0000","last_clean":"2024-04-25T16:00:19.302105+0000","last_became_active":"2024-04-25T15:59:02.999878+0000","last_became_peered":"2024-04-25T15:59:02.999878+0000","last_unstale":"2024-04-25T16:00:19.302105+0000","last_undegraded":"2024-04-25T16:00:19.302105+0000","last_fullsized":"2024-04-25T16:00:19.302105+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:26:37.966595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"75'7","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.807467+0000","last_change":"2024-04-25T15:59:02.511298+0000","last_active":"2024-04-25T15:59:50.807467+0000","last_peered":"2024-04-25T15:59:50.807467+0000","last_clean":"2024-04-25T15:59:50.807467+0000","last_became_active":"2024-04-25T15:59:02.510521+0000","last_became_peered":"2024-04-25T15:59:02.510521+0000","last_unstale":"2024-04-25T15:59:50.807467+0000","last_undegraded":"2024-04-25T15:59:50.807467+0000","last_fullsized":"2024-04-25T15:59:50.807467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T03:10:00.807919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347593+0000","last_change":"2024-04-25T15:59:02.999212+0000","last_active":"2024-04-25T16:00:19.347593+0000","last_peered":"2024-04-25T16:00:19.347593+0000","last_clean":"2024-04-25T16:00:19.347593+0000","last_became_active":"2024-04-25T15:59:02.998881+0000","last_became_peered":"2024-04-25T15:59:02.998881+0000","last_unstale":"2024-04-25T16:00:19.347593+0000","last_undegraded":"2024-04-25T16:00:19.347593+0000","last_fullsized":"2024-04-25T16:00:19.347593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:35:09.092196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"77'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:09.573190+0000","last_change":"2024-04-25T15:59:02.517412+0000","last_active":"2024-04-25T16:01:09.573190+0000","last_peered":"2024-04-25T16:01:09.573190+0000","last_clean":"2024-04-25T16:01:09.573190+0000","last_became_active":"2024-04-25T15:59:02.517077+0000","last_became_peered":"2024-04-25T15:59:02.517077+0000","last_unstale":"2024-04-25T16:01:09.573190+0000","last_undegraded":"2024-04-25T16:01:09.573190+0000","last_fullsized":"2024-04-25T16:01:09.573190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:52:00.161199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"77'8","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.342365+0000","last_change":"2024-04-25T15:59:50.678844+0000","last_active":"2024-04-25T16:00:19.342365+0000","last_peered":"2024-04-25T16:00:19.342365+0000","last_clean":"2024-04-25T16:00:19.342365+0000","last_became_active":"2024-04-25T15:59:50.678411+0000","last_became_peered":"2024-04-25T15:59:50.678411+0000","last_unstale":"2024-04-25T16:00:19.342365+0000","last_undegraded":"2024-04-25T16:00:19.342365+0000","last_fullsized":"2024-04-25T16:00:19.342365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T22:56:48.018678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,10],"acting":[9,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.b","version":"75'4","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.669652+0000","last_change":"2024-04-25T15:59:02.515912+0000","last_active":"2024-04-25T15:59:50.669652+0000","last_peered":"2024-04-25T15:59:50.669652+0000","last_clean":"2024-04-25T15:59:50.669652+0000","last_became_active":"2024-04-25T15:59:02.514949+0000","last_became_peered":"2024-04-25T15:59:02.514949+0000","last_unstale":"2024-04-25T15:59:50.669652+0000","last_undegraded":"2024-04-25T15:59:50.669652+0000","last_fullsized":"2024-04-25T15:59:50.669652+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:31:33.467223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"77'5","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.360414+0000","last_change":"2024-04-25T15:59:02.988896+0000","last_active":"2024-04-25T16:00:19.360414+0000","last_peered":"2024-04-25T16:00:19.360414+0000","last_clean":"2024-04-25T16:00:19.360414+0000","last_became_active":"2024-04-25T15:59:02.988478+0000","last_became_peered":"2024-04-25T15:59:02.988478+0000","last_unstale":"2024-04-25T16:00:19.360414+0000","last_undegraded":"2024-04-25T16:00:19.360414+0000","last_fullsized":"2024-04-25T16:00:19.360414+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:35:47.014678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"77'6","reported_seq":104,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:01:11.789876+0000","last_change":"2024-04-25T15:59:50.687632+0000","last_active":"2024-04-25T16:01:11.789876+0000","last_peered":"2024-04-25T16:01:11.789876+0000","last_clean":"2024-04-25T16:01:11.789876+0000","last_became_active":"2024-04-25T15:59:50.680402+0000","last_became_peered":"2024-04-25T15:59:50.680402+0000","last_unstale":"2024-04-25T16:01:11.789876+0000","last_undegraded":"2024-04-25T16:01:11.789876+0000","last_fullsized":"2024-04-25T16:01:11.789876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:34:12.544489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1478,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":84,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1170,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,11,0],"acting":[1,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359317+0000","last_change":"2024-04-25T15:59:02.990507+0000","last_active":"2024-04-25T16:00:19.359317+0000","last_peered":"2024-04-25T16:00:19.359317+0000","last_clean":"2024-04-25T16:00:19.359317+0000","last_became_active":"2024-04-25T15:59:02.990160+0000","last_became_peered":"2024-04-25T15:59:02.990160+0000","last_unstale":"2024-04-25T16:00:19.359317+0000","last_undegraded":"2024-04-25T16:00:19.359317+0000","last_fullsized":"2024-04-25T16:00:19.359317+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:18:21.526155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"75'3","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.672520+0000","last_change":"2024-04-25T15:59:02.510956+0000","last_active":"2024-04-25T15:59:50.672520+0000","last_peered":"2024-04-25T15:59:50.672520+0000","last_clean":"2024-04-25T15:59:50.672520+0000","last_became_active":"2024-04-25T15:59:02.510578+0000","last_became_peered":"2024-04-25T15:59:02.510578+0000","last_unstale":"2024-04-25T15:59:50.672520+0000","last_undegraded":"2024-04-25T15:59:50.672520+0000","last_fullsized":"2024-04-25T15:59:50.672520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:13:11.512365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'7","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.337102+0000","last_change":"2024-04-25T15:59:03.001122+0000","last_active":"2024-04-25T16:00:19.337102+0000","last_peered":"2024-04-25T16:00:19.337102+0000","last_clean":"2024-04-25T16:00:19.337102+0000","last_became_active":"2024-04-25T15:59:03.000672+0000","last_became_peered":"2024-04-25T15:59:03.000672+0000","last_unstale":"2024-04-25T16:00:19.337102+0000","last_undegraded":"2024-04-25T16:00:19.337102+0000","last_fullsized":"2024-04-25T16:00:19.337102+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:11:33.216434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"75'5","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T15:59:50.670676+0000","last_change":"2024-04-25T15:59:02.520645+0000","last_active":"2024-04-25T15:59:50.670676+0000","last_peered":"2024-04-25T15:59:50.670676+0000","last_clean":"2024-04-25T15:59:50.670676+0000","last_became_active":"2024-04-25T15:59:02.520197+0000","last_became_peered":"2024-04-25T15:59:02.520197+0000","last_unstale":"2024-04-25T15:59:50.670676+0000","last_undegraded":"2024-04-25T15:59:50.670676+0000","last_fullsized":"2024-04-25T15:59:50.670676+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:01.482234+0000","last_clean_scrub_stamp":"2024-04-25T15:59:01.482234+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T20:09:20.120554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"77'9","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.347691+0000","last_change":"2024-04-25T15:59:03.001668+0000","last_active":"2024-04-25T16:00:19.347691+0000","last_peered":"2024-04-25T16:00:19.347691+0000","last_clean":"2024-04-25T16:00:19.347691+0000","last_became_active":"2024-04-25T15:59:03.001333+0000","last_became_peered":"2024-04-25T15:59:03.001333+0000","last_unstale":"2024-04-25T16:00:19.347691+0000","last_undegraded":"2024-04-25T16:00:19.347691+0000","last_fullsized":"2024-04-25T16:00:19.347691+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:45:24.992476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"77'9","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346453+0000","last_change":"2024-04-25T15:59:03.005533+0000","last_active":"2024-04-25T16:00:19.346453+0000","last_peered":"2024-04-25T16:00:19.346453+0000","last_clean":"2024-04-25T16:00:19.346453+0000","last_became_active":"2024-04-25T15:59:03.004124+0000","last_became_peered":"2024-04-25T15:59:03.004124+0000","last_unstale":"2024-04-25T16:00:19.346453+0000","last_undegraded":"2024-04-25T16:00:19.346453+0000","last_fullsized":"2024-04-25T16:00:19.346453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:27:28.507065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"77'12","reported_seq":30,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.331417+0000","last_change":"2024-04-25T15:59:02.993416+0000","last_active":"2024-04-25T16:00:19.331417+0000","last_peered":"2024-04-25T16:00:19.331417+0000","last_clean":"2024-04-25T16:00:19.331417+0000","last_became_active":"2024-04-25T15:59:02.993098+0000","last_became_peered":"2024-04-25T15:59:02.993098+0000","last_unstale":"2024-04-25T16:00:19.331417+0000","last_undegraded":"2024-04-25T16:00:19.331417+0000","last_fullsized":"2024-04-25T16:00:19.331417+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:22:46.859932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"77'8","reported_seq":26,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350241+0000","last_change":"2024-04-25T15:59:02.996858+0000","last_active":"2024-04-25T16:00:19.350241+0000","last_peered":"2024-04-25T16:00:19.350241+0000","last_clean":"2024-04-25T16:00:19.350241+0000","last_became_active":"2024-04-25T15:59:02.996530+0000","last_became_peered":"2024-04-25T15:59:02.996530+0000","last_unstale":"2024-04-25T16:00:19.350241+0000","last_undegraded":"2024-04-25T16:00:19.350241+0000","last_fullsized":"2024-04-25T16:00:19.350241+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T23:42:36.723948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"77'7","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350676+0000","last_change":"2024-04-25T15:59:02.995942+0000","last_active":"2024-04-25T16:00:19.350676+0000","last_peered":"2024-04-25T16:00:19.350676+0000","last_clean":"2024-04-25T16:00:19.350676+0000","last_became_active":"2024-04-25T15:59:02.995567+0000","last_became_peered":"2024-04-25T15:59:02.995567+0000","last_unstale":"2024-04-25T16:00:19.350676+0000","last_undegraded":"2024-04-25T16:00:19.350676+0000","last_fullsized":"2024-04-25T16:00:19.350676+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:02:32.692392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"77'4","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350261+0000","last_change":"2024-04-25T15:59:03.000326+0000","last_active":"2024-04-25T16:00:19.350261+0000","last_peered":"2024-04-25T16:00:19.350261+0000","last_clean":"2024-04-25T16:00:19.350261+0000","last_became_active":"2024-04-25T15:59:02.999961+0000","last_became_peered":"2024-04-25T15:59:02.999961+0000","last_unstale":"2024-04-25T16:00:19.350261+0000","last_undegraded":"2024-04-25T16:00:19.350261+0000","last_fullsized":"2024-04-25T16:00:19.350261+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T00:25:10.631460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"77'6","reported_seq":23,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341696+0000","last_change":"2024-04-25T15:59:03.003625+0000","last_active":"2024-04-25T16:00:19.341696+0000","last_peered":"2024-04-25T16:00:19.341696+0000","last_clean":"2024-04-25T16:00:19.341696+0000","last_became_active":"2024-04-25T15:59:03.003280+0000","last_became_peered":"2024-04-25T15:59:03.003280+0000","last_unstale":"2024-04-25T16:00:19.341696+0000","last_undegraded":"2024-04-25T16:00:19.341696+0000","last_fullsized":"2024-04-25T16:00:19.341696+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T18:21:54.065689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"77'15","reported_seq":32,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.346578+0000","last_change":"2024-04-25T15:59:02.994645+0000","last_active":"2024-04-25T16:00:19.346578+0000","last_peered":"2024-04-25T16:00:19.346578+0000","last_clean":"2024-04-25T16:00:19.346578+0000","last_became_active":"2024-04-25T15:59:02.993924+0000","last_became_peered":"2024-04-25T15:59:02.993924+0000","last_unstale":"2024-04-25T16:00:19.346578+0000","last_undegraded":"2024-04-25T16:00:19.346578+0000","last_fullsized":"2024-04-25T16:00:19.346578+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:26:40.644708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"77'2","reported_seq":19,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.340331+0000","last_change":"2024-04-25T15:59:03.006156+0000","last_active":"2024-04-25T16:00:19.340331+0000","last_peered":"2024-04-25T16:00:19.340331+0000","last_clean":"2024-04-25T16:00:19.340331+0000","last_became_active":"2024-04-25T15:59:03.005795+0000","last_became_peered":"2024-04-25T15:59:03.005795+0000","last_unstale":"2024-04-25T16:00:19.340331+0000","last_undegraded":"2024-04-25T16:00:19.340331+0000","last_fullsized":"2024-04-25T16:00:19.340331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T21:29:23.523868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"77'6","reported_seq":24,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.339195+0000","last_change":"2024-04-25T15:59:03.010404+0000","last_active":"2024-04-25T16:00:19.339195+0000","last_peered":"2024-04-25T16:00:19.339195+0000","last_clean":"2024-04-25T16:00:19.339195+0000","last_became_active":"2024-04-25T15:59:03.010100+0000","last_became_peered":"2024-04-25T15:59:03.010100+0000","last_unstale":"2024-04-25T16:00:19.339195+0000","last_undegraded":"2024-04-25T16:00:19.339195+0000","last_fullsized":"2024-04-25T16:00:19.339195+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T17:14:22.718842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.335498+0000","last_change":"2024-04-25T15:59:03.009240+0000","last_active":"2024-04-25T16:00:19.335498+0000","last_peered":"2024-04-25T16:00:19.335498+0000","last_clean":"2024-04-25T16:00:19.335498+0000","last_became_active":"2024-04-25T15:59:03.008723+0000","last_became_peered":"2024-04-25T15:59:03.008723+0000","last_unstale":"2024-04-25T16:00:19.335498+0000","last_undegraded":"2024-04-25T16:00:19.335498+0000","last_fullsized":"2024-04-25T16:00:19.335498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:00:14.049934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"77'10","reported_seq":27,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.359019+0000","last_change":"2024-04-25T15:59:02.994359+0000","last_active":"2024-04-25T16:00:19.359019+0000","last_peered":"2024-04-25T16:00:19.359019+0000","last_clean":"2024-04-25T16:00:19.359019+0000","last_became_active":"2024-04-25T15:59:02.993990+0000","last_became_peered":"2024-04-25T15:59:02.993990+0000","last_unstale":"2024-04-25T16:00:19.359019+0000","last_undegraded":"2024-04-25T16:00:19.359019+0000","last_fullsized":"2024-04-25T16:00:19.359019+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T16:32:55.477414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"77'11","reported_seq":28,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.350058+0000","last_change":"2024-04-25T15:59:02.995510+0000","last_active":"2024-04-25T16:00:19.350058+0000","last_peered":"2024-04-25T16:00:19.350058+0000","last_clean":"2024-04-25T16:00:19.350058+0000","last_became_active":"2024-04-25T15:59:02.995008+0000","last_became_peered":"2024-04-25T15:59:02.995008+0000","last_unstale":"2024-04-25T16:00:19.350058+0000","last_undegraded":"2024-04-25T16:00:19.350058+0000","last_fullsized":"2024-04-25T16:00:19.350058+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T01:59:40.439615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"77'8","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-04-25T16:00:19.341453+0000","last_change":"2024-04-25T15:59:03.005846+0000","last_active":"2024-04-25T16:00:19.341453+0000","last_peered":"2024-04-25T16:00:19.341453+0000","last_clean":"2024-04-25T16:00:19.341453+0000","last_became_active":"2024-04-25T15:59:03.003779+0000","last_became_peered":"2024-04-25T15:59:03.003779+0000","last_unstale":"2024-04-25T16:00:19.341453+0000","last_undegraded":"2024-04-25T16:00:19.341453+0000","last_fullsized":"2024-04-25T16:00:19.341453+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-25T15:59:02.484195+0000","last_clean_scrub_stamp":"2024-04-25T15:59:02.484195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-27T02:59:47.698405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":4132576,"data_compressed":247542,"data_compressed_allocated":315392,"data_compressed_original":4096000,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65004,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1222,"num_read_kb":699,"num_write":248,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1192,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":195012,"data_compressed":5607,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":70,"seq":300647710758,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":839680,"data_stored":402410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":64,"seq":274877906989,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":1051383,"data_compressed":37695,"data_compressed_allocated":49152,"data_compressed_original":598016,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":59,"seq":253403070513,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":402010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":53,"seq":227633266744,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32008,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713144,"statfs":{"total":95995035648,"available":95962259456,"internally_reserved":0,"allocated":851968,"data_stored":402366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":47,"seq":201863462973,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":995047,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626497,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":401727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":461860,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":403066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":909312,"data_stored":459432,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182480,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":401552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986253,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27840,"kb_used_data":824,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717312,"statfs":{"total":95995035648,"available":95966527488,"internally_reserved":0,"allocated":843776,"data_stored":402120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822727,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":921600,"data_stored":994193,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34206,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35826,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":60972,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58544,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":2820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":60010,"data_compressed":1869,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-25T16:01:13.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:01:13.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.759+0000 7f407cad9640 1 -- 172.21.15.31:0/4009901895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4078109890 msgr2=0x7f407810bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.759+0000 7f407cad9640 1 --2- 172.21.15.31:0/4009901895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4078109890 0x7f407810bcb0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4070009ca0 tx=0x7f407002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.759+0000 7f407cad9640 1 -- 172.21.15.31:0/4009901895 shutdown_connections 2024-04-25T16:01:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.759+0000 7f407cad9640 1 --2- 172.21.15.31:0/4009901895 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4078109890 0x7f407810bcb0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.759+0000 7f407cad9640 1 --2- 172.21.15.31:0/4009901895 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4078106f30 0x7f4078109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.759+0000 7f407cad9640 1 --2- 172.21.15.31:0/4009901895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40781045d0 0x7f40781069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.759+0000 7f407cad9640 1 -- 172.21.15.31:0/4009901895 >> 172.21.15.31:0/4009901895 conn(0x7f40780fdbd0 msgr2=0x7f4078100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.759+0000 7f407cad9640 1 -- 172.21.15.31:0/4009901895 shutdown_connections 2024-04-25T16:01:13.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.759+0000 7f407cad9640 1 -- 172.21.15.31:0/4009901895 wait complete. 2024-04-25T16:01:13.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.760+0000 7f407cad9640 1 Processor -- start 2024-04-25T16:01:13.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.760+0000 7f407cad9640 1 -- start start 2024-04-25T16:01:13.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407cad9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40781045d0 0x7f407819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407cad9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4078106f30 0x7f407819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407cad9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4078109890 0x7f40781a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407cad9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f407810f610 con 0x7f40781045d0 2024-04-25T16:01:13.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407cad9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f407810f490 con 0x7f4078109890 2024-04-25T16:01:13.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407cad9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f407810f790 con 0x7f4078106f30 2024-04-25T16:01:13.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40781045d0 0x7f407819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40781045d0 0x7f407819a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46692/0 (socket says 172.21.15.31:46692) 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407659c640 1 -- 172.21.15.31:0/1639549633 learned_addr learned my addr 172.21.15.31:0/1639549633 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f4076d9d640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4078109890 0x7f40781a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f4075d9b640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4078106f30 0x7f407819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407659c640 1 -- 172.21.15.31:0/1639549633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4078106f30 msgr2=0x7f407819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407659c640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4078106f30 0x7f407819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407659c640 1 -- 172.21.15.31:0/1639549633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4078109890 msgr2=0x7f40781a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407659c640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4078109890 0x7f40781a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407659c640 1 -- 172.21.15.31:0/1639549633 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40781a1d80 con 0x7f40781045d0 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f4075d9b640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4078106f30 0x7f407819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407659c640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40781045d0 0x7f407819a240 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f406c00d740 tx=0x7f406c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f40677fe640 1 -- 172.21.15.31:0/1639549633 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c017070 con 0x7f40781045d0 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f40677fe640 1 -- 172.21.15.31:0/1639549633 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f406c00de50 con 0x7f40781045d0 2024-04-25T16:01:13.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40781a2070 con 0x7f40781045d0 2024-04-25T16:01:13.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.761+0000 7f40677fe640 1 -- 172.21.15.31:0/1639549633 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c004ce0 con 0x7f40781045d0 2024-04-25T16:01:13.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.762+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40781a2570 con 0x7f40781045d0 2024-04-25T16:01:13.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.763+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40780ff6f0 con 0x7f40781045d0 2024-04-25T16:01:13.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.764+0000 7f40677fe640 1 -- 172.21.15.31:0/1639549633 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f406c004e80 con 0x7f40781045d0 2024-04-25T16:01:13.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.765+0000 7f40677fe640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f40600afaf0 0x7f40600b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:13.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.765+0000 7f4075d9b640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f40600afaf0 0x7f40600b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:13.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.765+0000 7f40677fe640 1 -- 172.21.15.31:0/1639549633 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f406c0de9c0 con 0x7f40781045d0 2024-04-25T16:01:13.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.766+0000 7f4075d9b640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f40600afaf0 0x7f40600b2000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4068005d00 tx=0x7f4068005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:13.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.769+0000 7f40677fe640 1 -- 172.21.15.31:0/1639549633 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f406c0a75b0 con 0x7f40781045d0 2024-04-25T16:01:13.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:13 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2576856330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:13.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:13 smithi063 ceph-mon[32413]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:01:13.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:13 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2576856330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:13.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:13 smithi186 ceph-mon[33258]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:01:13.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:13 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2576856330' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:13.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:13 smithi031 ceph-mon[29140]: from='client.15399 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:01:13.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.958+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40780637a0 con 0x7f40781045d0 2024-04-25T16:01:13.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.959+0000 7f40677fe640 1 -- 172.21.15.31:0/1639549633 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f406c0a6eb0 con 0x7f40781045d0 2024-04-25T16:01:13.961 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:13.961 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.961+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f40600afaf0 msgr2=0x7f40600b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.961+0000 7f407cad9640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f40600afaf0 0x7f40600b2000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4068005d00 tx=0x7f4068005ed0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.961+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40781045d0 msgr2=0x7f407819a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.961+0000 7f407cad9640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40781045d0 0x7f407819a240 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f406c00d740 tx=0x7f406c00dc00 comp rx=0 tx=0).stop 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.962+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 shutdown_connections 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.962+0000 7f407cad9640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f40600afaf0 0x7f40600b2000 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.962+0000 7f407cad9640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4078109890 0x7f40781a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.962+0000 7f407cad9640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4078106f30 0x7f407819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.962+0000 7f407cad9640 1 --2- 172.21.15.31:0/1639549633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f40781045d0 0x7f407819a240 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.962+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 >> 172.21.15.31:0/1639549633 conn(0x7f40780fdbd0 msgr2=0x7f40781052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.962+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 shutdown_connections 2024-04-25T16:01:13.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:13.962+0000 7f407cad9640 1 -- 172.21.15.31:0/1639549633 wait complete. 2024-04-25T16:01:13.974 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:59:00.978038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:59:01.829535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:01:13.975 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.089+0000 7fd6f9070640 1 -- 172.21.15.31:0/3323742782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4108930 msgr2=0x7fd6f410ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.089+0000 7fd6f9070640 1 --2- 172.21.15.31:0/3323742782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4108930 0x7fd6f410ad50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec009ca0 tx=0x7fd6ec02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.089+0000 7fd6f9070640 1 -- 172.21.15.31:0/3323742782 shutdown_connections 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.089+0000 7fd6f9070640 1 --2- 172.21.15.31:0/3323742782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4108930 0x7fd6f410ad50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.089+0000 7fd6f9070640 1 --2- 172.21.15.31:0/3323742782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6f4103c30 0x7fd6f4104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.089+0000 7fd6f9070640 1 --2- 172.21.15.31:0/3323742782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f41029f0 0x7fd6f41036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.089+0000 7fd6f9070640 1 -- 172.21.15.31:0/3323742782 >> 172.21.15.31:0/3323742782 conn(0x7fd6f40fdbc0 msgr2=0x7fd6f4100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.089+0000 7fd6f9070640 1 -- 172.21.15.31:0/3323742782 shutdown_connections 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.089+0000 7fd6f9070640 1 -- 172.21.15.31:0/3323742782 wait complete. 2024-04-25T16:01:14.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f9070640 1 Processor -- start 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f9070640 1 -- start start 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f9070640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f41029f0 0x7fd6f40ffc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f9070640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4103c30 0x7fd6f40fe2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f9070640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6f4108930 0x7fd6f40fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f9070640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f410e500 con 0x7fd6f4103c30 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f9070640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f410e380 con 0x7fd6f4108930 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f9070640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6f410e680 con 0x7fd6f41029f0 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4103c30 0x7fd6f40fe2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4103c30 0x7fd6f40fe2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46718/0 (socket says 172.21.15.31:46718) 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f259c640 1 -- 172.21.15.31:0/232038998 learned_addr learned my addr 172.21.15.31:0/232038998 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f2d9d640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f41029f0 0x7fd6f40ffc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f359e640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6f4108930 0x7fd6f40fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f259c640 1 -- 172.21.15.31:0/232038998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f41029f0 msgr2=0x7fd6f40ffc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f259c640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f41029f0 0x7fd6f40ffc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f259c640 1 -- 172.21.15.31:0/232038998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6f4108930 msgr2=0x7fd6f40fe7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.090+0000 7fd6f259c640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6f4108930 0x7fd6f40fe7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.091+0000 7fd6f259c640 1 -- 172.21.15.31:0/232038998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6f40fee90 con 0x7fd6f4103c30 2024-04-25T16:01:14.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.091+0000 7fd6f259c640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4103c30 0x7fd6f40fe2b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800d950 tx=0x7fd6e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:14.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.091+0000 7fd6dbfff640 1 -- 172.21.15.31:0/232038998 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e8017030 con 0x7fd6f4103c30 2024-04-25T16:01:14.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.091+0000 7fd6dbfff640 1 -- 172.21.15.31:0/232038998 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd6e8004020 con 0x7fd6f4103c30 2024-04-25T16:01:14.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.091+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6f41a4290 con 0x7fd6f4103c30 2024-04-25T16:01:14.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.091+0000 7fd6dbfff640 1 -- 172.21.15.31:0/232038998 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e80047a0 con 0x7fd6f4103c30 2024-04-25T16:01:14.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.091+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6f41a46a0 con 0x7fd6f4103c30 2024-04-25T16:01:14.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.092+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6f41a4960 con 0x7fd6f4103c30 2024-04-25T16:01:14.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.093+0000 7fd6dbfff640 1 -- 172.21.15.31:0/232038998 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd6e8009a30 con 0x7fd6f4103c30 2024-04-25T16:01:14.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.094+0000 7fd6dbfff640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd6dc0afaf0 0x7fd6dc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.094+0000 7fd6dbfff640 1 -- 172.21.15.31:0/232038998 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd6e80de5c0 con 0x7fd6f4103c30 2024-04-25T16:01:14.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.094+0000 7fd6f2d9d640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd6dc0afaf0 0x7fd6dc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.095+0000 7fd6f2d9d640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd6dc0afaf0 0x7fd6dc0b2000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6f41035f0 tx=0x7fd6e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:14.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.099+0000 7fd6dbfff640 1 -- 172.21.15.31:0/232038998 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd6e80a71b0 con 0x7fd6f4103c30 2024-04-25T16:01:14.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.288+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd6f41a4b30 con 0x7fd6f4103c30 2024-04-25T16:01:14.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.288+0000 7fd6dbfff640 1 -- 172.21.15.31:0/232038998 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7fd6e80a6ab0 con 0x7fd6f4103c30 2024-04-25T16:01:14.289 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T16:01:14.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd6dc0afaf0 msgr2=0x7fd6dc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd6dc0afaf0 0x7fd6dc0b2000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd6f41035f0 tx=0x7fd6e4005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:14.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4103c30 msgr2=0x7fd6f40fe2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4103c30 0x7fd6f40fe2b0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd6e800d950 tx=0x7fd6e800de10 comp rx=0 tx=0).stop 2024-04-25T16:01:14.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 shutdown_connections 2024-04-25T16:01:14.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd6dc0afaf0 0x7fd6dc0b2000 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd6f4108930 0x7fd6f40fe7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd6f4103c30 0x7fd6f40fe2b0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 --2- 172.21.15.31:0/232038998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6f41029f0 0x7fd6f40ffc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 >> 172.21.15.31:0/232038998 conn(0x7fd6f40fdbc0 msgr2=0x7fd6f4074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:14.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 shutdown_connections 2024-04-25T16:01:14.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.290+0000 7fd6f9070640 1 -- 172.21.15.31:0/232038998 wait complete. 2024-04-25T16:01:14.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:01:14.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.419+0000 7f1d9aad8640 1 -- 172.21.15.31:0/3232164070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 msgr2=0x7f1d94104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.419+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/3232164070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 0x7f1d94104030 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1d84009900 tx=0x7f1d8402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:14.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.419+0000 7f1d9aad8640 1 -- 172.21.15.31:0/3232164070 shutdown_connections 2024-04-25T16:01:14.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.419+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/3232164070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d94108920 0x7f1d9410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.419+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/3232164070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 0x7f1d94104030 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.419+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/3232164070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d941029e0 0x7f1d941036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.419+0000 7f1d9aad8640 1 -- 172.21.15.31:0/3232164070 >> 172.21.15.31:0/3232164070 conn(0x7f1d940fdbd0 msgr2=0x7f1d94100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:14.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 -- 172.21.15.31:0/3232164070 shutdown_connections 2024-04-25T16:01:14.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 -- 172.21.15.31:0/3232164070 wait complete. 2024-04-25T16:01:14.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 Processor -- start 2024-04-25T16:01:14.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 -- start start 2024-04-25T16:01:14.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d941029e0 0x7f1d9406a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 0x7f1d9419f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d94108920 0x7f1d941a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9410e500 con 0x7f1d94103c20 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9410e380 con 0x7f1d94108920 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.420+0000 7f1d9aad8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9410e680 con 0x7f1d941029e0 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d93fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 0x7f1d9419f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d93fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 0x7f1d9419f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46732/0 (socket says 172.21.15.31:46732) 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d93fff640 1 -- 172.21.15.31:0/403515161 learned_addr learned my addr 172.21.15.31:0/403515161 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d99075640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d94108920 0x7f1d941a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d98874640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d941029e0 0x7f1d9406a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d93fff640 1 -- 172.21.15.31:0/403515161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d941029e0 msgr2=0x7f1d9406a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d93fff640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d941029e0 0x7f1d9406a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d93fff640 1 -- 172.21.15.31:0/403515161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d94108920 msgr2=0x7f1d941a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d93fff640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d94108920 0x7f1d941a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d93fff640 1 -- 172.21.15.31:0/403515161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d941a2080 con 0x7f1d94103c20 2024-04-25T16:01:14.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d98874640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d941029e0 0x7f1d9406a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:14.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d93fff640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 0x7f1d9419f310 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006010 tx=0x7f1d840042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:14.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d91ffb640 1 -- 172.21.15.31:0/403515161 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d8403f030 con 0x7f1d94103c20 2024-04-25T16:01:14.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d941a2310 con 0x7f1d94103c20 2024-04-25T16:01:14.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d91ffb640 1 -- 172.21.15.31:0/403515161 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d84031910 con 0x7f1d94103c20 2024-04-25T16:01:14.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.421+0000 7f1d91ffb640 1 -- 172.21.15.31:0/403515161 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d84035070 con 0x7f1d94103c20 2024-04-25T16:01:14.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.422+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d941a2850 con 0x7f1d94103c20 2024-04-25T16:01:14.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.423+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d94107730 con 0x7f1d94103c20 2024-04-25T16:01:14.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.425+0000 7f1d91ffb640 1 -- 172.21.15.31:0/403515161 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1d84042030 con 0x7f1d94103c20 2024-04-25T16:01:14.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.425+0000 7f1d91ffb640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d800afaf0 0x7f1d800b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.425+0000 7f1d91ffb640 1 -- 172.21.15.31:0/403515161 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1d840ff880 con 0x7f1d94103c20 2024-04-25T16:01:14.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.425+0000 7f1d98874640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d800afaf0 0x7f1d800b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.426+0000 7f1d98874640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d800afaf0 0x7f1d800b2000 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1d941035e0 tx=0x7f1d7c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:14.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.429+0000 7f1d91ffb640 1 -- 172.21.15.31:0/403515161 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1d840c8470 con 0x7f1d94103c20 2024-04-25T16:01:14.588 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:14 smithi031 ceph-mon[29140]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:01:14.588 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:14 smithi031 ceph-mon[29140]: pgmap v349: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-25T16:01:14.588 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:14 smithi031 ceph-mon[29140]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:01:14.588 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:14 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1639549633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:14.588 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:14 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/232038998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:14.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.616+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1d9406c2a0 con 0x7f1d94103c20 2024-04-25T16:01:14.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.616+0000 7f1d91ffb640 1 -- 172.21.15.31:0/403515161 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7f1d840c7d70 con 0x7f1d94103c20 2024-04-25T16:01:14.617 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T16:01:14.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d800afaf0 msgr2=0x7f1d800b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d800afaf0 0x7f1d800b2000 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1d941035e0 tx=0x7f1d7c005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:14.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 msgr2=0x7f1d9419f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 0x7f1d9419f310 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1d84006010 tx=0x7f1d840042f0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 shutdown_connections 2024-04-25T16:01:14.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d800afaf0 0x7f1d800b2000 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1d94108920 0x7f1d941a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1d94103c20 0x7f1d9419f310 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 --2- 172.21.15.31:0/403515161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d941029e0 0x7f1d9406a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 >> 172.21.15.31:0/403515161 conn(0x7f1d940fdbd0 msgr2=0x7f1d94104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 shutdown_connections 2024-04-25T16:01:14.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.618+0000 7f1d9aad8640 1 -- 172.21.15.31:0/403515161 wait complete. 2024-04-25T16:01:14.631 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:01:14.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.750+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/1005086960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc48106f30 msgr2=0x7fdc48109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.750+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/1005086960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc48106f30 0x7fdc48109350 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c009900 tx=0x7fdc3c02a410 comp rx=0 tx=0).stop 2024-04-25T16:01:14.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.750+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/1005086960 shutdown_connections 2024-04-25T16:01:14.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.750+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/1005086960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc48109890 0x7fdc4810bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.750+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/1005086960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc48106f30 0x7fdc48109350 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.750+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/1005086960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc481045d0 0x7fdc481069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.750+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/1005086960 >> 172.21.15.31:0/1005086960 conn(0x7fdc480fdbd0 msgr2=0x7fdc48100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:14.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.750+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/1005086960 shutdown_connections 2024-04-25T16:01:14.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.750+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/1005086960 wait complete. 2024-04-25T16:01:14.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.751+0000 7fdc4f8f9640 1 Processor -- start 2024-04-25T16:01:14.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.751+0000 7fdc4f8f9640 1 -- start start 2024-04-25T16:01:14.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4f8f9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc481045d0 0x7fdc4819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4f8f9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc48106f30 0x7fdc4819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4f8f9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc48109890 0x7fdc481a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4f8f9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4810f5f0 con 0x7fdc48109890 2024-04-25T16:01:14.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4f8f9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4810f470 con 0x7fdc481045d0 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4f8f9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdc4810f770 con 0x7fdc48106f30 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4de96640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc48109890 0x7fdc481a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4de96640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc48109890 0x7fdc481a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46756/0 (socket says 172.21.15.31:46756) 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4de96640 1 -- 172.21.15.31:0/3626612956 learned_addr learned my addr 172.21.15.31:0/3626612956 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4d695640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc481045d0 0x7fdc4819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4ce94640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc48106f30 0x7fdc4819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4de96640 1 -- 172.21.15.31:0/3626612956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc48106f30 msgr2=0x7fdc4819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4de96640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc48106f30 0x7fdc4819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4de96640 1 -- 172.21.15.31:0/3626612956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc481045d0 msgr2=0x7fdc4819a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4de96640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc481045d0 0x7fdc4819a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4de96640 1 -- 172.21.15.31:0/3626612956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc481a20b0 con 0x7fdc48109890 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4ce94640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc48106f30 0x7fdc4819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.752+0000 7fdc4de96640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc48109890 0x7fdc481a1a40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800eb40 tx=0x7fdc3800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.753+0000 7fdc467fc640 1 -- 172.21.15.31:0/3626612956 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38004b20 con 0x7fdc48109890 2024-04-25T16:01:14.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.753+0000 7fdc467fc640 1 -- 172.21.15.31:0/3626612956 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdc38004cc0 con 0x7fdc48109890 2024-04-25T16:01:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.753+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc481a23a0 con 0x7fdc48109890 2024-04-25T16:01:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.753+0000 7fdc467fc640 1 -- 172.21.15.31:0/3626612956 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc38010720 con 0x7fdc48109890 2024-04-25T16:01:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.753+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc481a28a0 con 0x7fdc48109890 2024-04-25T16:01:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.754+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc4806c2a0 con 0x7fdc48109890 2024-04-25T16:01:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.755+0000 7fdc467fc640 1 -- 172.21.15.31:0/3626612956 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdc38005560 con 0x7fdc48109890 2024-04-25T16:01:14.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.756+0000 7fdc467fc640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc300afaf0 0x7fdc300b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.756+0000 7fdc4d695640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc300afaf0 0x7fdc300b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.756+0000 7fdc467fc640 1 -- 172.21.15.31:0/3626612956 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdc380df3d0 con 0x7fdc48109890 2024-04-25T16:01:14.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.756+0000 7fdc4d695640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc300afaf0 0x7fdc300b2000 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdc4806a8b0 tx=0x7fdc40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:14.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.759+0000 7fdc467fc640 1 -- 172.21.15.31:0/3626612956 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc380a7fc0 con 0x7fdc48109890 2024-04-25T16:01:14.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:14 smithi063 ceph-mon[32413]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:01:14.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:14 smithi063 ceph-mon[32413]: pgmap v349: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-25T16:01:14.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:14 smithi063 ceph-mon[32413]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:01:14.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:14 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1639549633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:14.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:14 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/232038998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:14.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:14 smithi186 ceph-mon[33258]: from='client.15405 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:01:14.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:14 smithi186 ceph-mon[33258]: pgmap v349: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 426 B/s wr, 12 op/s 2024-04-25T16:01:14.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:14 smithi186 ceph-mon[33258]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-25T16:01:14.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:14 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1639549633' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:14.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:14 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/232038998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:01:14.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.947+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdc481a2b60 con 0x7fdc48109890 2024-04-25T16:01:14.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.948+0000 7fdc467fc640 1 -- 172.21.15.31:0/3626612956 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7fdc380a78c0 con 0x7fdc48109890 2024-04-25T16:01:14.948 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T16:01:14.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc300afaf0 msgr2=0x7fdc300b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc300afaf0 0x7fdc300b2000 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdc4806a8b0 tx=0x7fdc40005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:14.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc48109890 msgr2=0x7fdc481a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:14.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc48109890 0x7fdc481a1a40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdc3800eb40 tx=0x7fdc3800c2d0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 shutdown_connections 2024-04-25T16:01:14.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdc300afaf0 0x7fdc300b2000 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdc48109890 0x7fdc481a1a40 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc48106f30 0x7fdc4819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 --2- 172.21.15.31:0/3626612956 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc481045d0 0x7fdc4819a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:14.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 >> 172.21.15.31:0/3626612956 conn(0x7fdc480fdbd0 msgr2=0x7fdc48105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:14.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 shutdown_connections 2024-04-25T16:01:14.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:14.950+0000 7fdc4f8f9640 1 -- 172.21.15.31:0/3626612956 wait complete. 2024-04-25T16:01:14.963 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:01:15.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.075+0000 7f0487892640 1 -- 172.21.15.31:0/3154134978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480109750 msgr2=0x7f048010bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.075+0000 7f0487892640 1 --2- 172.21.15.31:0/3154134978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480109750 0x7f048010bc60 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0478009ca0 tx=0x7f047802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.075+0000 7f0487892640 1 -- 172.21.15.31:0/3154134978 shutdown_connections 2024-04-25T16:01:15.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.075+0000 7f0487892640 1 --2- 172.21.15.31:0/3154134978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480109750 0x7f048010bc60 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.075+0000 7f0487892640 1 --2- 172.21.15.31:0/3154134978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0480106be0 0x7f04801090c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.075+0000 7f0487892640 1 --2- 172.21.15.31:0/3154134978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04801041f0 0x7f0480106610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.075+0000 7f0487892640 1 -- 172.21.15.31:0/3154134978 >> 172.21.15.31:0/3154134978 conn(0x7f04800fdbd0 msgr2=0x7f0480100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.076+0000 7f0487892640 1 -- 172.21.15.31:0/3154134978 shutdown_connections 2024-04-25T16:01:15.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.076+0000 7f0487892640 1 -- 172.21.15.31:0/3154134978 wait complete. 2024-04-25T16:01:15.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.076+0000 7f0487892640 1 Processor -- start 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.076+0000 7f0487892640 1 -- start start 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.076+0000 7f0487892640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04801041f0 0x7f048019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0487892640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480106be0 0x7f048019a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0487892640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0480109750 0x7f04801a1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0487892640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f048010f5c0 con 0x7f0480106be0 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0487892640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f048010f440 con 0x7f0480109750 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0487892640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f048010f740 con 0x7f04801041f0 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0484e2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480106be0 0x7f048019a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0484e2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480106be0 0x7f048019a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46772/0 (socket says 172.21.15.31:46772) 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0484e2d640 1 -- 172.21.15.31:0/3570502057 learned_addr learned my addr 172.21.15.31:0/3570502057 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0485e2f640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0480109750 0x7f04801a1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f048562e640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04801041f0 0x7f048019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0484e2d640 1 -- 172.21.15.31:0/3570502057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04801041f0 msgr2=0x7f048019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0484e2d640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04801041f0 0x7f048019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0484e2d640 1 -- 172.21.15.31:0/3570502057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0480109750 msgr2=0x7f04801a1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0484e2d640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0480109750 0x7f04801a1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0484e2d640 1 -- 172.21.15.31:0/3570502057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04801a1ca0 con 0x7f0480106be0 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f048562e640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04801041f0 0x7f048019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f0484e2d640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480106be0 0x7f048019a6a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f047000e9f0 tx=0x7f047000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.077+0000 7f04767fc640 1 -- 172.21.15.31:0/3570502057 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f047000cd80 con 0x7f0480106be0 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.078+0000 7f04767fc640 1 -- 172.21.15.31:0/3570502057 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f04700054d0 con 0x7f0480106be0 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.078+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04801a1f90 con 0x7f0480106be0 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.078+0000 7f04767fc640 1 -- 172.21.15.31:0/3570502057 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0470010690 con 0x7f0480106be0 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.078+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04801a2400 con 0x7f0480106be0 2024-04-25T16:01:15.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.079+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04800ff6f0 con 0x7f0480106be0 2024-04-25T16:01:15.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.080+0000 7f04767fc640 1 -- 172.21.15.31:0/3570502057 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0470004e30 con 0x7f0480106be0 2024-04-25T16:01:15.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.081+0000 7f04767fc640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f046c0afaf0 0x7f046c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.081+0000 7f04767fc640 1 -- 172.21.15.31:0/3570502057 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f04700df2b0 con 0x7f0480106be0 2024-04-25T16:01:15.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.081+0000 7f048562e640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f046c0afaf0 0x7f046c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.081+0000 7f048562e640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f046c0afaf0 0x7f046c0b2000 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f048006a850 tx=0x7f047c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.084+0000 7f04767fc640 1 -- 172.21.15.31:0/3570502057 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0470053ff0 con 0x7f0480106be0 2024-04-25T16:01:15.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.311+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f048002dda0 con 0x7f0480106be0 2024-04-25T16:01:15.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.311+0000 7f04767fc640 1 -- 172.21.15.31:0/3570502057 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11917 (secure 0 0 0) 0x7f04700a7ea0 con 0x7f0480106be0 2024-04-25T16:01:15.312 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:15.313 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:01:09.454389+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24535,24521,24563,14631,24553]},"id":1}]} 2024-04-25T16:01:15.313 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-25T16:01:15.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f046c0afaf0 msgr2=0x7f046c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f046c0afaf0 0x7f046c0b2000 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f048006a850 tx=0x7f047c005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:15.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480106be0 msgr2=0x7f048019a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480106be0 0x7f048019a6a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f047000e9f0 tx=0x7f047000edf0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 shutdown_connections 2024-04-25T16:01:15.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f046c0afaf0 0x7f046c0b2000 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0480109750 0x7f04801a1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0480106be0 0x7f048019a6a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 --2- 172.21.15.31:0/3570502057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04801041f0 0x7f048019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 >> 172.21.15.31:0/3570502057 conn(0x7f04800fdbd0 msgr2=0x7f0480104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 shutdown_connections 2024-04-25T16:01:15.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.314+0000 7f0487892640 1 -- 172.21.15.31:0/3570502057 wait complete. 2024-04-25T16:01:15.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:01:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.439+0000 7f37d1534640 1 -- 172.21.15.31:0/2354432599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 msgr2=0x7f37cc10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.439+0000 7f37d1534640 1 --2- 172.21.15.31:0/2354432599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 0x7f37cc10ad40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f37c4009ca0 tx=0x7f37c402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.439+0000 7f37d1534640 1 -- 172.21.15.31:0/2354432599 shutdown_connections 2024-04-25T16:01:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.439+0000 7f37d1534640 1 --2- 172.21.15.31:0/2354432599 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 0x7f37cc10ad40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.439+0000 7f37d1534640 1 --2- 172.21.15.31:0/2354432599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37cc103c20 0x7f37cc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.439+0000 7f37d1534640 1 --2- 172.21.15.31:0/2354432599 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37cc1029e0 0x7f37cc1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.439+0000 7f37d1534640 1 -- 172.21.15.31:0/2354432599 >> 172.21.15.31:0/2354432599 conn(0x7f37cc0fdbd0 msgr2=0x7f37cc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.439+0000 7f37d1534640 1 -- 172.21.15.31:0/2354432599 shutdown_connections 2024-04-25T16:01:15.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.439+0000 7f37d1534640 1 -- 172.21.15.31:0/2354432599 wait complete. 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.440+0000 7f37d1534640 1 Processor -- start 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.440+0000 7f37d1534640 1 -- start start 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.440+0000 7f37d1534640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37cc1029e0 0x7f37cc075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.440+0000 7f37d1534640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37cc103c20 0x7f37cc19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.440+0000 7f37d1534640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 0x7f37cc1a14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.440+0000 7f37d1534640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc10e510 con 0x7f37cc108920 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.440+0000 7f37d1534640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc10e390 con 0x7f37cc103c20 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.440+0000 7f37d1534640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37cc10e690 con 0x7f37cc1029e0 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 0x7f37cc1a14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 0x7f37cc1a14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46798/0 (socket says 172.21.15.31:46798) 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37cb7fe640 1 -- 172.21.15.31:0/1249715236 learned_addr learned my addr 172.21.15.31:0/1249715236 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37caffd640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37cc1029e0 0x7f37cc075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37ca7fc640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37cc103c20 0x7f37cc19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37cb7fe640 1 -- 172.21.15.31:0/1249715236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37cc1029e0 msgr2=0x7f37cc075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37cb7fe640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37cc1029e0 0x7f37cc075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37cb7fe640 1 -- 172.21.15.31:0/1249715236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37cc103c20 msgr2=0x7f37cc19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37cb7fe640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37cc103c20 0x7f37cc19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37cb7fe640 1 -- 172.21.15.31:0/1249715236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37cc1a1cd0 con 0x7f37cc108920 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37cb7fe640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 0x7f37cc1a14b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f37c40097b0 tx=0x7f37c4003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37b3fff640 1 -- 172.21.15.31:0/1249715236 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c4004240 con 0x7f37cc108920 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37b3fff640 1 -- 172.21.15.31:0/1249715236 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37c402f380 con 0x7f37cc108920 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37cc1a1f60 con 0x7f37cc108920 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37b3fff640 1 -- 172.21.15.31:0/1249715236 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37c402fc30 con 0x7f37cc108920 2024-04-25T16:01:15.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.441+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37cc1a23c0 con 0x7f37cc108920 2024-04-25T16:01:15.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.443+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37cc06a190 con 0x7f37cc108920 2024-04-25T16:01:15.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.444+0000 7f37b3fff640 1 -- 172.21.15.31:0/1249715236 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f37c40043e0 con 0x7f37cc108920 2024-04-25T16:01:15.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.444+0000 7f37b3fff640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37b40afaf0 0x7f37b40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.445+0000 7f37caffd640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37b40afaf0 0x7f37b40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.445+0000 7f37b3fff640 1 -- 172.21.15.31:0/1249715236 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f37c4100340 con 0x7f37cc108920 2024-04-25T16:01:15.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.445+0000 7f37caffd640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37b40afaf0 0x7f37b40b2000 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f37cc1035e0 tx=0x7f37c000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.449+0000 7f37b3fff640 1 -- 172.21.15.31:0/1249715236 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37c40c8ed0 con 0x7f37cc108920 2024-04-25T16:01:15.605 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:15 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/403515161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:15.606 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:15 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3626612956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:15.606 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:15 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3570502057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:15.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.667+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f37cc063860 con 0x7f37cc108920 2024-04-25T16:01:15.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.669+0000 7f37b3fff640 1 -- 172.21.15.31:0/1249715236 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f37c40098b0 con 0x7f37cc108920 2024-04-25T16:01:15.670 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:15.670 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:01:15.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37b40afaf0 msgr2=0x7f37b40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37b40afaf0 0x7f37b40b2000 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f37cc1035e0 tx=0x7f37c000b390 comp rx=0 tx=0).stop 2024-04-25T16:01:15.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 msgr2=0x7f37cc1a14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 0x7f37cc1a14b0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f37c40097b0 tx=0x7f37c4003c20 comp rx=0 tx=0).stop 2024-04-25T16:01:15.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 shutdown_connections 2024-04-25T16:01:15.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37b40afaf0 0x7f37b40b2000 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37cc108920 0x7f37cc1a14b0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37cc103c20 0x7f37cc19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 --2- 172.21.15.31:0/1249715236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37cc1029e0 0x7f37cc075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 >> 172.21.15.31:0/1249715236 conn(0x7f37cc0fdbd0 msgr2=0x7f37cc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 shutdown_connections 2024-04-25T16:01:15.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.671+0000 7f37d1534640 1 -- 172.21.15.31:0/1249715236 wait complete. 2024-04-25T16:01:15.684 DEBUG:tasks.fwd_scrub:Yielding 2024-04-25T16:01:15.684 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-25T16:01:15.695 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T16:01:15.695 DEBUG:teuthology.orchestra.run.smithi031:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-25T16:01:15.695 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-25T16:01:15.695 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:01:15.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:15.696 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:01:15.698 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T16:01:15.746 INFO:tasks.workunit:Pulling workunits from ref 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:01:15.746 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T16:01:15.746 INFO:tasks.workunit:timeout=3h 2024-04-25T16:01:15.747 INFO:tasks.workunit:cleanup=True 2024-04-25T16:01:15.747 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T16:01:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-25T16:01:15.762 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-25T16:01:15.762 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 73h/115d Inode: 1099511628283 Links: 2 2024-04-25T16:01:15.762 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-25T16:01:15.762 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-25T16:01:15.763 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2024-04-25 15:59:10.374766689 +0000 2024-04-25T16:01:15.763 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2024-04-25 15:59:10.374766689 +0000 2024-04-25T16:01:15.763 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2024-04-25 16:01:08.700854893 +0000 2024-04-25T16:01:15.763 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: 2024-04-25 15:59:10.374766689 +0000 2024-04-25T16:01:15.763 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-25T16:01:15.763 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-25T16:01:15.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.812+0000 7f4b7e38f640 1 -- 172.21.15.31:0/3608104730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 msgr2=0x7f4b7810bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.812+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/3608104730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 0x7f4b7810bcc0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4b70009ca0 tx=0x7f4b7002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.812+0000 7f4b7e38f640 1 -- 172.21.15.31:0/3608104730 shutdown_connections 2024-04-25T16:01:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.812+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/3608104730 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 0x7f4b7810bcc0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.812+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/3608104730 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b78106f40 0x7f4b78109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.812+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/3608104730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b781045e0 0x7f4b78106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.812+0000 7f4b7e38f640 1 -- 172.21.15.31:0/3608104730 >> 172.21.15.31:0/3608104730 conn(0x7f4b780fdbc0 msgr2=0x7f4b78100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.812+0000 7f4b7e38f640 1 -- 172.21.15.31:0/3608104730 shutdown_connections 2024-04-25T16:01:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.812+0000 7f4b7e38f640 1 -- 172.21.15.31:0/3608104730 wait complete. 2024-04-25T16:01:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.813+0000 7f4b7e38f640 1 Processor -- start 2024-04-25T16:01:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.813+0000 7f4b7e38f640 1 -- start start 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.813+0000 7f4b7e38f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b781045e0 0x7f4b78103a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.813+0000 7f4b7e38f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b78106f40 0x7f4b781020d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.813+0000 7f4b7e38f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 0x7f4b78102610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.813+0000 7f4b7e38f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7810f620 con 0x7f4b781098a0 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.813+0000 7f4b7e38f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7810f4a0 con 0x7f4b781045e0 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.813+0000 7f4b7e38f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7810f7a0 con 0x7f4b78106f40 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7c92c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 0x7f4b78102610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7c92c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 0x7f4b78102610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46816/0 (socket says 172.21.15.31:46816) 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7c92c640 1 -- 172.21.15.31:0/935102679 learned_addr learned my addr 172.21.15.31:0/935102679 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:15.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b77fff640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b781045e0 0x7f4b78103a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b777fe640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b78106f40 0x7f4b781020d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7c92c640 1 -- 172.21.15.31:0/935102679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b78106f40 msgr2=0x7f4b781020d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7c92c640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b78106f40 0x7f4b781020d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7c92c640 1 -- 172.21.15.31:0/935102679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b781045e0 msgr2=0x7f4b78103a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7c92c640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b781045e0 0x7f4b78103a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7c92c640 1 -- 172.21.15.31:0/935102679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b78102d40 con 0x7f4b781098a0 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7c92c640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 0x7f4b78102610 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4b70009430 tx=0x7f4b70003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b757fa640 1 -- 172.21.15.31:0/935102679 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7002f380 con 0x7f4b781098a0 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b7e38f640 1 -- 172.21.15.31:0/935102679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b781a42b0 con 0x7f4b781098a0 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b757fa640 1 -- 172.21.15.31:0/935102679 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b700317c0 con 0x7f4b781098a0 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.814+0000 7f4b757fa640 1 -- 172.21.15.31:0/935102679 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7002fcd0 con 0x7f4b781098a0 2024-04-25T16:01:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.815+0000 7f4b7e38f640 1 -- 172.21.15.31:0/935102679 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b781a4710 con 0x7f4b781098a0 2024-04-25T16:01:15.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.817+0000 7f4b85240740 1 Processor -- start 2024-04-25T16:01:15.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.817+0000 7f4b757fa640 1 -- 172.21.15.31:0/935102679 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4b700305c0 con 0x7f4b781098a0 2024-04-25T16:01:15.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.817+0000 7f4b85240740 1 -- start start 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b85240740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752e7410 0x561d756884a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b85240740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75688a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b85240740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752c6730 0x561d7568f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b85240740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d755edb00 con 0x561d752be550 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b85240740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561d75589920 con 0x561d752c6730 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b85240740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561d755e9ac0 con 0x561d752e7410 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b77fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75688a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b77fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75688a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46822/0 (socket says 172.21.15.31:46822) 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b77fff640 1 -- 172.21.15.31:0/3822485996 learned_addr learned my addr 172.21.15.31:0/3822485996 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b777fe640 1 --2- 172.21.15.31:0/3822485996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752c6730 0x561d7568f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b777fe640 1 --2- 172.21.15.31:0/3822485996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752e7410 0x561d756884a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b757fa640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b600afc90 0x7f4b600b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b77fff640 1 -- 172.21.15.31:0/3822485996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752e7410 msgr2=0x561d756884a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:15.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b77fff640 1 --2- 172.21.15.31:0/3822485996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752e7410 0x561d756884a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b77fff640 1 -- 172.21.15.31:0/3822485996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752c6730 msgr2=0x561d7568f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b77fff640 1 --2- 172.21.15.31:0/3822485996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752c6730 0x561d7568f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b77fff640 1 -- 172.21.15.31:0/3822485996 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d753c77e0 con 0x561d752be550 2024-04-25T16:01:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b7c92c640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b600afc90 0x7f4b600b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b77fff640 1 --2- 172.21.15.31:0/3822485996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75688a20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800b740 tx=0x7f4b6800bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48286eea0c731e41 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.818+0000 7f4b757fa640 1 -- 172.21.15.31:0/935102679 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4b70100360 con 0x7f4b781098a0 2024-04-25T16:01:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.819+0000 7f4b5e7fc640 1 -- 172.21.15.31:0/3822485996 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68003d50 con 0x561d752be550 2024-04-25T16:01:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.819+0000 7f4b5e7fc640 1 -- 172.21.15.31:0/3822485996 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b68015070 con 0x561d752be550 2024-04-25T16:01:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.819+0000 7f4b5e7fc640 1 -- 172.21.15.31:0/3822485996 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68004cb0 con 0x561d752be550 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 -- 172.21.15.31:0/3822485996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 msgr2=0x561d75688a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 --2- 172.21.15.31:0/3822485996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75688a20 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800b740 tx=0x7f4b6800bc00 comp rx=0 tx=0).stop 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b7c92c640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b600afc90 0x7f4b600b21a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4b700c87e0 tx=0x7f4b700099f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 -- 172.21.15.31:0/3822485996 shutdown_connections 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 --2- 172.21.15.31:0/3822485996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752c6730 0x561d7568f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 --2- 172.21.15.31:0/3822485996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75688a20 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 --2- 172.21.15.31:0/3822485996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752e7410 0x561d756884a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 -- 172.21.15.31:0/3822485996 >> 172.21.15.31:0/3822485996 conn(0x561d752db560 msgr2=0x561d75683e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 -- 172.21.15.31:0/3822485996 shutdown_connections 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 -- 172.21.15.31:0/3822485996 wait complete. 2024-04-25T16:01:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 -- start start 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.820+0000 7f4b85240740 1 Processor -- start 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b85240740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75692320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b85240740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 0x561d75692860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b85240740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752e7410 0x561d756997c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b85240740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561d75690370 con 0x561d752be550 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b85240740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561d755875d0 con 0x561d752e7410 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b85240740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561d756904f0 con 0x561d752c6730 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b85240740 10 client.0 _ll_register_callbacks cb 0x561d75575080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75692320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75692320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46836/0 (socket says 172.21.15.31:46836) 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 -- 172.21.15.31:0/2738437898 learned_addr learned my addr 172.21.15.31:0/2738437898 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752e7410 0x561d756997c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b77fff640 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 0x561d75692860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 -- 172.21.15.31:0/2738437898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 msgr2=0x561d75692860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:15.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 0x561d75692860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 -- 172.21.15.31:0/2738437898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752e7410 msgr2=0x561d756997c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752e7410 0x561d756997c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 -- 172.21.15.31:0/2738437898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d75427f70 con 0x561d752be550 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b77fff640 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 0x561d75692860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b777fe640 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75692320 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c014d70 tx=0x7f4b6c016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e033cb595125e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b577fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b577fe640 1 -- 172.21.15.31:0/2738437898 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6c004020 con 0x561d752be550 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b577fe640 1 -- 172.21.15.31:0/2738437898 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4b6c0041c0 con 0x561d752be550 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.821+0000 7f4b85240740 1 -- 172.21.15.31:0/2738437898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b68015070 con 0x561d752be550 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b577fe640 1 -- 172.21.15.31:0/2738437898 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4b6c0043c0 con 0x561d752be550 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b85240740 10 client.15459 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b85240740 1 -- 172.21.15.31:0/2738437898 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561d752bf170 con 0x561d752be550 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b577fe640 1 -- 172.21.15.31:0/2738437898 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b68015070 con 0x561d752be550 2024-04-25T16:01:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b577fe640 1 -- 172.21.15.31:0/2738437898 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4b6c00f4b0 con 0x561d752be550 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b85240740 10 client.15459 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b85240740 10 client.15459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b85240740 20 client.15459 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b85240740 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561d752a61f0 0x561d7569fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b85240740 4 client.15459 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b85240740 1 -- 172.21.15.31:0/2738437898 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d755fb710 con 0x561d752a61f0 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.822+0000 7f4b77fff640 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561d752a61f0 0x561d7569fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.823+0000 7f4b577fe640 10 client.15459 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.823+0000 7f4b77fff640 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561d752a61f0 0x561d7569fd50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96067477ab79781f server_cookie=9e6455a5a0d72200 in_seq=0 out_seq=0 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b577fe640 1 -- 172.21.15.31:0/2738437898 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561d755fb710 con 0x561d752a61f0 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b577fe640 10 client.15459 handle_command_reply: tid=0 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 client.15459 shutdown 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 -- 172.21.15.31:0/2738437898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 msgr2=0x561d75692320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75692320 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c014d70 tx=0x7f4b6c016040 comp rx=0 tx=0).stop 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 -- 172.21.15.31:0/2738437898 shutdown_connections 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561d752a61f0 0x561d7569fd50 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752e7410 0x561d756997c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b577fe640 0 client.15459 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 0x561d75692860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 --2- 172.21.15.31:0/2738437898 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752be550 0x561d75692320 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 -- 172.21.15.31:0/2738437898 >> 172.21.15.31:0/2738437898 conn(0x561d752db560 msgr2=0x561d75682ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 -- 172.21.15.31:0/2738437898 shutdown_connections 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 1 -- 172.21.15.31:0/2738437898 wait complete. 2024-04-25T16:01:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.824+0000 7f4b85240740 20 client.15459 trim_cache size 0 max 16384 2024-04-25T16:01:15.838 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 994e2f20102a20fa9a27966c14b625cc01884df8 2024-04-25T16:01:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b85240740 1 Processor -- start 2024-04-25T16:01:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b85240740 1 -- start start 2024-04-25T16:01:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b85240740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752db560 0x561d756918d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b85240740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 0x561d75699750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b85240740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 0x561d756a4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b85240740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781ad3e0 con 0x561d752db560 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b85240740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781aef20 con 0x561d752be550 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b85240740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781adab0 con 0x561d752c6730 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b777fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752db560 0x561d756918d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b777fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752db560 0x561d756918d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46852/0 (socket says 172.21.15.31:46852) 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b777fe640 1 -- 172.21.15.31:0/2175136762 learned_addr learned my addr 172.21.15.31:0/2175136762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.843+0000 7f4b77fff640 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 0x561d75699750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b777fe640 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 0x561d756a4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b77fff640 1 -- 172.21.15.31:0/2175136762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 msgr2=0x561d756a4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b77fff640 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 0x561d756a4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b77fff640 1 -- 172.21.15.31:0/2175136762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752db560 msgr2=0x561d756918d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b77fff640 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752db560 0x561d756918d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b77fff640 1 -- 172.21.15.31:0/2175136762 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d7568fb30 con 0x561d752be550 2024-04-25T16:01:15.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b777fe640 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752db560 0x561d756918d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:15.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b77fff640 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 0x561d75699750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x561d75693390 tx=0x7f4b68004290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9d2338a27253f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b5dffb640 1 -- 172.21.15.31:0/2175136762 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68005020 con 0x561d752be550 2024-04-25T16:01:15.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b5dffb640 1 -- 172.21.15.31:0/2175136762 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b68015040 con 0x561d752be550 2024-04-25T16:01:15.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.844+0000 7f4b5dffb640 1 -- 172.21.15.31:0/2175136762 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68011440 con 0x561d752be550 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.845+0000 7f4b85240740 1 -- 172.21.15.31:0/2175136762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 msgr2=0x561d75699750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.845+0000 7f4b85240740 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 0x561d75699750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x561d75693390 tx=0x7f4b68004290 comp rx=0 tx=0).stop 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.845+0000 7f4b85240740 1 -- 172.21.15.31:0/2175136762 shutdown_connections 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.845+0000 7f4b85240740 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752c6730 0x561d756a4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.845+0000 7f4b85240740 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 0x561d75699750 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.845+0000 7f4b85240740 1 --2- 172.21.15.31:0/2175136762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752db560 0x561d756918d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.845+0000 7f4b85240740 1 -- 172.21.15.31:0/2175136762 >> 172.21.15.31:0/2175136762 conn(0x561d75692a30 msgr2=0x561d75687660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.845+0000 7f4b85240740 1 -- 172.21.15.31:0/2175136762 shutdown_connections 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.845+0000 7f4b85240740 1 -- 172.21.15.31:0/2175136762 wait complete. 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b85240740 1 -- start start 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b85240740 1 Processor -- start 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b85240740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 0x561d75698a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b85240740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752c6730 0x561d75698f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b85240740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752db560 0x561d75706300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b85240740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781ae850 con 0x561d752c6730 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b85240740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781ae180 con 0x561d752be550 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b85240740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781ab3c0 con 0x561d752db560 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b85240740 10 client.0 _ll_register_callbacks cb 0x7f4b6c0149b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b77fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752c6730 0x561d75698f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b77fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752c6730 0x561d75698f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46868/0 (socket says 172.21.15.31:46868) 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.846+0000 7f4b77fff640 1 -- 172.21.15.31:0/3296798600 learned_addr learned my addr 172.21.15.31:0/3296798600 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b777fe640 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 0x561d75698a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b777fe640 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752db560 0x561d75706300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b77fff640 1 -- 172.21.15.31:0/3296798600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752db560 msgr2=0x561d75706300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b77fff640 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752db560 0x561d75706300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b77fff640 1 -- 172.21.15.31:0/3296798600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 msgr2=0x561d75698a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b77fff640 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 0x561d75698a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b77fff640 1 -- 172.21.15.31:0/3296798600 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d757068c0 con 0x561d752c6730 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b777fe640 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752db560 0x561d75706300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b77fff640 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752c6730 0x561d75698f70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006010 tx=0x7f4b68003a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea2464ec4e5e2a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b577fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b577fe640 1 -- 172.21.15.31:0/3296798600 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68036060 con 0x561d752c6730 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b577fe640 1 -- 172.21.15.31:0/3296798600 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4b680048e0 con 0x561d752c6730 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b85240740 1 -- 172.21.15.31:0/3296798600 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b68015040 con 0x561d752c6730 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b577fe640 1 -- 172.21.15.31:0/3296798600 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4b68004f20 con 0x561d752c6730 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b85240740 10 client.15465 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.847+0000 7f4b85240740 1 -- 172.21.15.31:0/3296798600 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561d75707c20 con 0x561d752c6730 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.848+0000 7f4b577fe640 1 -- 172.21.15.31:0/3296798600 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b68015040 con 0x561d752c6730 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.848+0000 7f4b577fe640 1 -- 172.21.15.31:0/3296798600 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4b680274d0 con 0x561d752c6730 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.848+0000 7f4b85240740 10 client.15465 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.848+0000 7f4b85240740 10 client.15465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:15.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.848+0000 7f4b85240740 20 client.15465 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.848+0000 7f4b85240740 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561d752e7410 0x561d7570a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.848+0000 7f4b85240740 4 client.15465 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.848+0000 7f4b85240740 1 -- 172.21.15.31:0/3296798600 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561d755f5580 con 0x561d752e7410 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.848+0000 7f4b777fe640 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561d752e7410 0x561d7570a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.849+0000 7f4b777fe640 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561d752e7410 0x561d7570a500 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a504325758fbc89a server_cookie=b073883e201fde1b in_seq=0 out_seq=0 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.849+0000 7f4b577fe640 10 client.15465 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b577fe640 1 -- 172.21.15.31:0/3296798600 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561d755f5580 con 0x561d752e7410 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b577fe640 10 client.15465 handle_command_reply: tid=0 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 client.15465 shutdown 2024-04-25T16:01:15.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 -- 172.21.15.31:0/3296798600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752c6730 msgr2=0x561d75698f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752c6730 0x561d75698f70 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4b68006010 tx=0x7f4b68003a40 comp rx=0 tx=0).stop 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 -- 172.21.15.31:0/3296798600 shutdown_connections 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561d752e7410 0x561d7570a500 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561d752db560 0x561d75706300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561d752c6730 0x561d75698f70 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 --2- 172.21.15.31:0/3296798600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561d752be550 0x561d75698a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b577fe640 0 client.15465 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 -- 172.21.15.31:0/3296798600 >> 172.21.15.31:0/3296798600 conn(0x561d75692a30 msgr2=0x561d756873c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 -- 172.21.15.31:0/3296798600 shutdown_connections 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 1 -- 172.21.15.31:0/3296798600 wait complete. 2024-04-25T16:01:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.850+0000 7f4b85240740 20 client.15465 trim_cache size 0 max 16384 2024-04-25T16:01:15.858 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:15.858 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:01:15.858 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "250a16bf-0c0a-4aad-a879-a6f040330707", 2024-04-25T16:01:15.858 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:01:15.858 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 -- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b600afc90 msgr2=0x7f4b600b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b600afc90 0x7f4b600b21a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4b700c87e0 tx=0x7f4b700099f0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 -- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 msgr2=0x7f4b78102610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 0x7f4b78102610 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4b70009430 tx=0x7f4b70003ba0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 -- 172.21.15.31:0/935102679 shutdown_connections 2024-04-25T16:01:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4b600afc90 0x7f4b600b21a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4b781098a0 0x7f4b78102610 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b78106f40 0x7f4b781020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 --2- 172.21.15.31:0/935102679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4b781045e0 0x7f4b78103a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:15.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 -- 172.21.15.31:0/935102679 >> 172.21.15.31:0/935102679 conn(0x7f4b780fdbc0 msgr2=0x7f4b781051b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:15.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 -- 172.21.15.31:0/935102679 shutdown_connections 2024-04-25T16:01:15.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:15.851+0000 7f4b7e38f640 1 -- 172.21.15.31:0/935102679 wait complete. 2024-04-25T16:01:15.860 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T16:01:15.864 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:01:15.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:01:15.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:15 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/403515161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:15.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:15 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3626612956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:15.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:15 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3570502057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:15.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:15 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/403515161' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:01:15.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:15 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3626612956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:01:15.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:15 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3570502057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:16.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.030+0000 7fc6a45ee640 1 -- 172.21.15.31:0/3469171027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c103c30 msgr2=0x7fc69c104040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:16.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.030+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/3469171027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c103c30 0x7fc69c104040 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc690009900 tx=0x7fc69002a410 comp rx=0 tx=0).stop 2024-04-25T16:01:16.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.030+0000 7fc6a45ee640 1 -- 172.21.15.31:0/3469171027 shutdown_connections 2024-04-25T16:01:16.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.030+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/3469171027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc69c108930 0x7fc69c10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.030+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/3469171027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c103c30 0x7fc69c104040 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.030+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/3469171027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc69c1029f0 0x7fc69c1036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.030+0000 7fc6a45ee640 1 -- 172.21.15.31:0/3469171027 >> 172.21.15.31:0/3469171027 conn(0x7fc69c0fdbc0 msgr2=0x7fc69c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:16.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.030+0000 7fc6a45ee640 1 -- 172.21.15.31:0/3469171027 shutdown_connections 2024-04-25T16:01:16.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.030+0000 7fc6a45ee640 1 -- 172.21.15.31:0/3469171027 wait complete. 2024-04-25T16:01:16.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a45ee640 1 Processor -- start 2024-04-25T16:01:16.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a45ee640 1 -- start start 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a45ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c1029f0 0x7fc69c199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a45ee640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc69c103c30 0x7fc69c199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a45ee640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc69c108930 0x7fc69c1a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a45ee640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c10e4f0 con 0x7fc69c1029f0 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a45ee640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c10e370 con 0x7fc69c103c30 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a45ee640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c10e670 con 0x7fc69c108930 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a238a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c1029f0 0x7fc69c199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a238a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c1029f0 0x7fc69c199100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46888/0 (socket says 172.21.15.31:46888) 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a238a640 1 -- 172.21.15.31:0/59130547 learned_addr learned my addr 172.21.15.31:0/59130547 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:16.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc69c103c30 0x7fc69c199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc69c108930 0x7fc69c1a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a238a640 1 -- 172.21.15.31:0/59130547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc69c108930 msgr2=0x7fc69c1a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a238a640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc69c108930 0x7fc69c1a05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a238a640 1 -- 172.21.15.31:0/59130547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc69c103c30 msgr2=0x7fc69c199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a238a640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc69c103c30 0x7fc69c199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a238a640 1 -- 172.21.15.31:0/59130547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc69c1a0c40 con 0x7fc69c1029f0 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.031+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc69c108930 0x7fc69c1a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.032+0000 7fc6a238a640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c1029f0 0x7fc69c199100 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc698005fd0 tx=0x7fc69800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.032+0000 7fc68f7fe640 1 -- 172.21.15.31:0/59130547 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698017070 con 0x7fc69c1029f0 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.032+0000 7fc68f7fe640 1 -- 172.21.15.31:0/59130547 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc698009c30 con 0x7fc69c1029f0 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.032+0000 7fc68f7fe640 1 -- 172.21.15.31:0/59130547 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc698003d10 con 0x7fc69c1029f0 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.032+0000 7fc6a45ee640 1 -- 172.21.15.31:0/59130547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc69c1a0f30 con 0x7fc69c1029f0 2024-04-25T16:01:16.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.032+0000 7fc6a45ee640 1 -- 172.21.15.31:0/59130547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc69c1a1340 con 0x7fc69c1029f0 2024-04-25T16:01:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.034+0000 7fc6ab576740 1 Processor -- start 2024-04-25T16:01:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6ab576740 1 -- start start 2024-04-25T16:01:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc68f7fe640 1 -- 172.21.15.31:0/59130547 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc698003eb0 con 0x7fc69c1029f0 2024-04-25T16:01:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344922c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e3449231f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 0x55e344923e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6ab576740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e344885d80 con 0x55e34457f390 2024-04-25T16:01:16.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6ab576740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e34483f8b0 con 0x55e34455e6b0 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6ab576740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e344881d40 con 0x55e3445564d0 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6a2b8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344922c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6a2b8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344922c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46896/0 (socket says 172.21.15.31:46896) 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/3579134115 learned_addr learned my addr 172.21.15.31:0/3579134115 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 0x55e344923e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6a238a640 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e3449231f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.035+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/3579134115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 msgr2=0x55e3449231f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e3449231f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/3579134115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 msgr2=0x55e344923e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 0x55e344923e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/3579134115 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e344810a40 con 0x55e34457f390 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 0x55e344923e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc68f7fe640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6880afaf0 0x7fc6880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344922c70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc694009ea0 tx=0x7fc694009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8fdfd70521e4ce8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc6a238a640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6880afaf0 0x7fc6880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc68cbfe640 1 -- 172.21.15.31:0/3579134115 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694016070 con 0x55e34457f390 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc68f7fe640 1 -- 172.21.15.31:0/59130547 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc6980de910 con 0x7fc69c1029f0 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc68cbfe640 1 -- 172.21.15.31:0/3579134115 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc69400ee80 con 0x55e34457f390 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.036+0000 7fc68cbfe640 1 -- 172.21.15.31:0/3579134115 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6940116d0 con 0x55e34457f390 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6a238a640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6880afaf0 0x7fc6880b2000 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55e344922990 tx=0x7fc6980e4550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6ab576740 1 -- 172.21.15.31:0/3579134115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 msgr2=0x55e344922c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6ab576740 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344922c70 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc694009ea0 tx=0x7fc694009f90 comp rx=0 tx=0).stop 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6ab576740 1 -- 172.21.15.31:0/3579134115 shutdown_connections 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6ab576740 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 0x55e344923e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6ab576740 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e3449231f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6ab576740 1 --2- 172.21.15.31:0/3579134115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344922c70 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6ab576740 1 -- 172.21.15.31:0/3579134115 >> 172.21.15.31:0/3579134115 conn(0x55e3445734e0 msgr2=0x55e34491c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6ab576740 1 -- 172.21.15.31:0/3579134115 shutdown_connections 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.037+0000 7fc6ab576740 1 -- 172.21.15.31:0/3579134115 wait complete. 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6ab576740 1 -- start start 2024-04-25T16:01:16.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6ab576740 1 Processor -- start 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e34492ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 0x55e34492cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6ab576740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e344924990 con 0x55e34457f390 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6ab576740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e34480cd10 con 0x55e34455e6b0 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6ab576740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e344924b10 con 0x55e3445564d0 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6ab576740 10 client.0 _ll_register_callbacks cb 0x55e344839210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a2b8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a2b8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46910/0 (socket says 172.21.15.31:46910) 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/2248047124 learned_addr learned my addr 172.21.15.31:0/2248047124 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e34492ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 0x55e34492cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/2248047124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 msgr2=0x55e34492ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:16.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e34492ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/2248047124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 msgr2=0x55e34492cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 0x55e34492cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.038+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/2248047124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3446617d0 con 0x55e34457f390 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933f20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc694005f40 tx=0x7fc694000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fb66cf80494546 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc685ffb640 1 -- 172.21.15.31:0/2248047124 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694016040 con 0x55e34457f390 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6ab576740 1 -- 172.21.15.31:0/2248047124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e344893990 con 0x55e34457f390 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc685ffb640 1 -- 172.21.15.31:0/2248047124 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc694003970 con 0x55e34457f390 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc685ffb640 1 -- 172.21.15.31:0/2248047124 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc694003b70 con 0x55e34457f390 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6ab576740 10 client.15477 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6ab576740 1 -- 172.21.15.31:0/2248047124 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e3445973f0 con 0x55e34457f390 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc685ffb640 1 -- 172.21.15.31:0/2248047124 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e344893990 con 0x55e34457f390 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc685ffb640 1 -- 172.21.15.31:0/2248047124 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc694026030 con 0x55e34457f390 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6ab576740 10 client.15477 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:16.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6ab576740 10 client.15477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6ab576740 20 client.15477 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6ab576740 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e34453df60 0x55e34493a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6ab576740 4 client.15477 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.039+0000 7fc6ab576740 1 -- 172.21.15.31:0/2248047124 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e34488d800 con 0x55e34453df60 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.040+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e34453df60 0x55e34493a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.040+0000 7fc685ffb640 10 client.15477 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.040+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e34453df60 0x55e34493a0d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c289f8ded642795 server_cookie=ddfcb190f9ebfb1b in_seq=0 out_seq=0 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc685ffb640 1 -- 172.21.15.31:0/2248047124 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e34488d800 con 0x55e34453df60 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc685ffb640 10 client.15477 handle_command_reply: tid=0 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 client.15477 shutdown 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 -- 172.21.15.31:0/2248047124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 msgr2=0x55e344933f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933f20 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc694005f40 tx=0x7fc694000f40 comp rx=0 tx=0).stop 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 -- 172.21.15.31:0/2248047124 shutdown_connections 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e34453df60 0x55e34493a0d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933f20 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc685ffb640 0 client.15477 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34455e6b0 0x55e34492cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 --2- 172.21.15.31:0/2248047124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e34492ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 -- 172.21.15.31:0/2248047124 >> 172.21.15.31:0/2248047124 conn(0x55e3445734e0 msgr2=0x55e34491f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 -- 172.21.15.31:0/2248047124 shutdown_connections 2024-04-25T16:01:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 1 -- 172.21.15.31:0/2248047124 wait complete. 2024-04-25T16:01:16.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.041+0000 7fc6ab576740 20 client.15477 trim_cache size 0 max 16384 2024-04-25T16:01:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.059+0000 7fc6ab576740 1 Processor -- start 2024-04-25T16:01:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.059+0000 7fc6ab576740 1 -- start start 2024-04-25T16:01:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e3445564d0 0x55e34491c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e34455e6b0 0x55e34492ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6ab576740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1a9100 con 0x55e34457f390 2024-04-25T16:01:16.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6ab576740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1aac40 con 0x55e3445564d0 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6ab576740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1a97d0 con 0x55e34455e6b0 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46924/0 (socket says 172.21.15.31:46924) 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/3996817768 learned_addr learned my addr 172.21.15.31:0/3996817768 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/3996817768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e3445564d0 0x55e34491c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/3996817768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e34455e6b0 0x55e34492ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/3996817768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e34455e6b0 msgr2=0x55e34492ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/3996817768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e34455e6b0 0x55e34492ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/3996817768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e3445564d0 msgr2=0x55e34491c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/3996817768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e3445564d0 0x55e34491c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 -- 172.21.15.31:0/3996817768 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e344923310 con 0x55e34457f390 2024-04-25T16:01:16.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.060+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/3996817768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933e40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc694003b40 tx=0x7fc694009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9779a67109585695 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:16.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc687fff640 1 -- 172.21.15.31:0/3996817768 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc694016040 con 0x55e34457f390 2024-04-25T16:01:16.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc687fff640 1 -- 172.21.15.31:0/3996817768 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc694038070 con 0x55e34457f390 2024-04-25T16:01:16.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc687fff640 1 -- 172.21.15.31:0/3996817768 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6940273f0 con 0x55e34457f390 2024-04-25T16:01:16.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc6ab576740 1 -- 172.21.15.31:0/3996817768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 msgr2=0x55e344933e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:16.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc6ab576740 1 --2- 172.21.15.31:0/3996817768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933e40 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fc694003b40 tx=0x7fc694009660 comp rx=0 tx=0).stop 2024-04-25T16:01:16.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc6ab576740 1 -- 172.21.15.31:0/3996817768 shutdown_connections 2024-04-25T16:01:16.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc6ab576740 1 --2- 172.21.15.31:0/3996817768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34457f390 0x55e344933e40 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc6ab576740 1 --2- 172.21.15.31:0/3996817768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e34455e6b0 0x55e34492ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc6ab576740 1 --2- 172.21.15.31:0/3996817768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e3445564d0 0x55e34491c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.061+0000 7fc6ab576740 1 -- 172.21.15.31:0/3996817768 >> 172.21.15.31:0/3996817768 conn(0x55e3445734e0 msgr2=0x55e34491bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 -- 172.21.15.31:0/3996817768 shutdown_connections 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 -- 172.21.15.31:0/3996817768 wait complete. 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 -- start start 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 Processor -- start 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e344931970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34455e6b0 0x55e344931eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34457f390 0x55e344932de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1aa570 con 0x55e34455e6b0 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1a9ea0 con 0x55e34457f390 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc69c1a70e0 con 0x55e3445564d0 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6ab576740 10 client.0 _ll_register_callbacks cb 0x7fc690009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6a1b89640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34455e6b0 0x55e344931eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6a1b89640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34455e6b0 0x55e344931eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46930/0 (socket says 172.21.15.31:46930) 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6a1b89640 1 -- 172.21.15.31:0/70469724 learned_addr learned my addr 172.21.15.31:0/70469724 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e344931970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.062+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34457f390 0x55e344932de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6a1b89640 1 -- 172.21.15.31:0/70469724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 msgr2=0x55e344931970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e344931970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6a1b89640 1 -- 172.21.15.31:0/70469724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34457f390 msgr2=0x55e344932de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34457f390 0x55e344932de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6a1b89640 1 -- 172.21.15.31:0/70469724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e3445570f0 con 0x55e34455e6b0 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34457f390 0x55e344932de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6a1b89640 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34455e6b0 0x55e344931eb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc690005940 tx=0x7fc690004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=311d65423d510267 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc685ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc685ffb640 1 -- 172.21.15.31:0/70469724 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc690035040 con 0x55e34455e6b0 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc685ffb640 1 -- 172.21.15.31:0/70469724 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc690005040 con 0x55e34455e6b0 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6ab576740 1 -- 172.21.15.31:0/70469724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc694038070 con 0x55e34455e6b0 2024-04-25T16:01:16.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc685ffb640 1 -- 172.21.15.31:0/70469724 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc690005240 con 0x55e34455e6b0 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6ab576740 10 client.15483 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc6ab576740 1 -- 172.21.15.31:0/70469724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e3448464b0 con 0x55e34455e6b0 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.063+0000 7fc685ffb640 1 -- 172.21.15.31:0/70469724 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc694038070 con 0x55e34455e6b0 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc685ffb640 1 -- 172.21.15.31:0/70469724 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc6900047e0 con 0x55e34455e6b0 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc6ab576740 10 client.15483 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc6ab576740 10 client.15483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc6ab576740 20 client.15483 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc6ab576740 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e3445133a0 0x55e3449a4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc6ab576740 4 client.15483 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc6ab576740 1 -- 172.21.15.31:0/70469724 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e3448084a0 con 0x55e3445133a0 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e3445133a0 0x55e3449a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:16.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc685ffb640 10 client.15483 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.064+0000 7fc6a2b8b640 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e3445133a0 0x55e3449a4620 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34aef907c403cda7 server_cookie=cc79605b3767cd88 in_seq=0 out_seq=0 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc685ffb640 1 -- 172.21.15.31:0/70469724 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e3448084a0 con 0x55e3445133a0 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc685ffb640 10 client.15483 handle_command_reply: tid=0 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 client.15483 shutdown 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 -- 172.21.15.31:0/70469724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34455e6b0 msgr2=0x55e344931eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34455e6b0 0x55e344931eb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc690005940 tx=0x7fc690004040 comp rx=0 tx=0).stop 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 -- 172.21.15.31:0/70469724 shutdown_connections 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e3445133a0 0x55e3449a4620 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e34457f390 0x55e344932de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e34455e6b0 0x55e344931eb0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 --2- 172.21.15.31:0/70469724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e3445564d0 0x55e344931970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc685ffb640 0 client.15483 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 -- 172.21.15.31:0/70469724 >> 172.21.15.31:0/70469724 conn(0x55e3445734e0 msgr2=0x55e34491a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:16.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 -- 172.21.15.31:0/70469724 shutdown_connections 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 1 -- 172.21.15.31:0/70469724 wait complete. 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6ab576740 20 client.15483 trim_cache size 0 max 16384 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stdout: "250a16bf-0c0a-4aad-a879-a6f040330707": { 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "250a16bf-0c0a-4aad-a879-a6f040330707", 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:01:16.072 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:01:16.073 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6a45ee640 1 -- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6880afaf0 msgr2=0x7fc6880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.065+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6880afaf0 0x7fc6880b2000 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55e344922990 tx=0x7fc6980e4550 comp rx=0 tx=0).stop 2024-04-25T16:01:16.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 -- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c1029f0 msgr2=0x7fc69c199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c1029f0 0x7fc69c199100 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc698005fd0 tx=0x7fc69800dde0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 -- 172.21.15.31:0/59130547 shutdown_connections 2024-04-25T16:01:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6880afaf0 0x7fc6880b2000 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc69c108930 0x7fc69c1a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc69c103c30 0x7fc69c199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 --2- 172.21.15.31:0/59130547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc69c1029f0 0x7fc69c199100 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 -- 172.21.15.31:0/59130547 >> 172.21.15.31:0/59130547 conn(0x7fc69c0fdbc0 msgr2=0x7fc69c104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 -- 172.21.15.31:0/59130547 shutdown_connections 2024-04-25T16:01:16.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:16.066+0000 7fc6a45ee640 1 -- 172.21.15.31:0/59130547 wait complete. 2024-04-25T16:01:16.079 INFO:tasks.cephfs.filesystem:scrub status for tag:250a16bf-0c0a-4aad-a879-a6f040330707 - {'path': '/', 'tag': '250a16bf-0c0a-4aad-a879-a6f040330707', 'options': 'recursive,force'} 2024-04-25T16:01:16.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:16 smithi063 ceph-mon[32413]: pgmap v350: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-25T16:01:16.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:16 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1249715236' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:16.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:16 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:01:16.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:16 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:16.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:16 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:01:16.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:16 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:16.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:16 smithi186 ceph-mon[33258]: pgmap v350: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-25T16:01:16.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:16 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1249715236' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:16.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:16 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:01:16.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:16 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:16.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:16 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:01:16.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:16 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:16.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:16 smithi031 ceph-mon[29140]: pgmap v350: 97 pgs: 97 active+clean; 640 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-25T16:01:16.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:16 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1249715236' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:01:16.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:16 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:01:16.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:16 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:16.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:16 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:01:16.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:16 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:18.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:18 smithi063 ceph-mon[32413]: pgmap v351: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-25T16:01:18.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:18 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:01:18.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:18 smithi186 ceph-mon[33258]: pgmap v351: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-25T16:01:18.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:18 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:01:18.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:18 smithi031 ceph-mon[29140]: pgmap v351: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-25T16:01:18.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:18 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:01:20.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:20 smithi063 ceph-mon[32413]: pgmap v352: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-25T16:01:20.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:20 smithi186 ceph-mon[33258]: pgmap v352: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-25T16:01:20.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:20 smithi031 ceph-mon[29140]: pgmap v352: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-25T16:01:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:22 smithi063 ceph-mon[32413]: pgmap v353: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T16:01:22.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:22 smithi186 ceph-mon[33258]: pgmap v353: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T16:01:22.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:22 smithi031 ceph-mon[29140]: pgmap v353: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.7 KiB/s wr, 10 op/s 2024-04-25T16:01:24.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:24 smithi063 ceph-mon[32413]: pgmap v354: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:01:24.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:24 smithi186 ceph-mon[33258]: pgmap v354: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:01:24.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:24 smithi031 ceph-mon[29140]: pgmap v354: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:01:26.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:26 smithi063 ceph-mon[32413]: pgmap v355: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T16:01:26.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:26 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:01:26.880 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:26 smithi186 ceph-mon[33258]: pgmap v355: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T16:01:26.880 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:26 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:01:26.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:26 smithi031 ceph-mon[29140]: pgmap v355: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.3 KiB/s wr, 10 op/s 2024-04-25T16:01:26.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:26 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:01:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.839+0000 7fd94ac05640 1 -- 172.21.15.31:0/1339577131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9441046c0 msgr2=0x7fd94410ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.839+0000 7fd94ac05640 1 --2- 172.21.15.31:0/1339577131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9441046c0 0x7fd94410ae10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009ca0 tx=0x7fd93c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.839+0000 7fd94ac05640 1 -- 172.21.15.31:0/1339577131 shutdown_connections 2024-04-25T16:01:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.839+0000 7fd94ac05640 1 --2- 172.21.15.31:0/1339577131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd9441046c0 0x7fd94410ae10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.839+0000 7fd94ac05640 1 --2- 172.21.15.31:0/1339577131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd944103b60 0x7fd944104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.839+0000 7fd94ac05640 1 --2- 172.21.15.31:0/1339577131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd944103000 0x7fd944103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.839+0000 7fd94ac05640 1 -- 172.21.15.31:0/1339577131 >> 172.21.15.31:0/1339577131 conn(0x7fd9440fd3a0 msgr2=0x7fd9440ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.839+0000 7fd94ac05640 1 -- 172.21.15.31:0/1339577131 shutdown_connections 2024-04-25T16:01:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.839+0000 7fd94ac05640 1 -- 172.21.15.31:0/1339577131 wait complete. 2024-04-25T16:01:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd94ac05640 1 Processor -- start 2024-04-25T16:01:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd94ac05640 1 -- start start 2024-04-25T16:01:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd94ac05640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd944103000 0x7fd944199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd94ac05640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd944103b60 0x7fd94419a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd94ac05640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94419f470 0x7fd9441a1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd94ac05640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd94410f890 con 0x7fd944103b60 2024-04-25T16:01:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd94ac05640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd94410f710 con 0x7fd944103000 2024-04-25T16:01:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd94ac05640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd94410fa10 con 0x7fd94419f470 2024-04-25T16:01:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd943fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd944103b60 0x7fd94419a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd943fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd944103b60 0x7fd94419a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56160/0 (socket says 172.21.15.31:56160) 2024-04-25T16:01:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd943fff640 1 -- 172.21.15.31:0/730731563 learned_addr learned my addr 172.21.15.31:0/730731563 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd9491a2640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94419f470 0x7fd9441a1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.840+0000 7fd9489a1640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd944103000 0x7fd944199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd943fff640 1 -- 172.21.15.31:0/730731563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94419f470 msgr2=0x7fd9441a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:28.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd943fff640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94419f470 0x7fd9441a1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd943fff640 1 -- 172.21.15.31:0/730731563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd944103000 msgr2=0x7fd944199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd943fff640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd944103000 0x7fd944199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd943fff640 1 -- 172.21.15.31:0/730731563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9441a1f00 con 0x7fd944103b60 2024-04-25T16:01:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd943fff640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd944103b60 0x7fd94419a4f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd93000e9f0 tx=0x7fd93000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd941ffb640 1 -- 172.21.15.31:0/730731563 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93000cd80 con 0x7fd944103b60 2024-04-25T16:01:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd941ffb640 1 -- 172.21.15.31:0/730731563 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9300108e0 con 0x7fd944103b60 2024-04-25T16:01:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd94ac05640 1 -- 172.21.15.31:0/730731563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9441a21f0 con 0x7fd944103b60 2024-04-25T16:01:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd941ffb640 1 -- 172.21.15.31:0/730731563 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd930018980 con 0x7fd944103b60 2024-04-25T16:01:28.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.841+0000 7fd94ac05640 1 -- 172.21.15.31:0/730731563 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9441a2740 con 0x7fd944103b60 2024-04-25T16:01:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.843+0000 7fd951b92740 1 Processor -- start 2024-04-25T16:01:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd951b92740 1 -- start start 2024-04-25T16:01:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd941ffb640 1 -- 172.21.15.31:0/730731563 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd930010430 con 0x7fd944103b60 2024-04-25T16:01:28.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd951b92740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce8e1b0 0x55ecdd232d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd951b92740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce652f0 0x55ecdd233300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd951b92740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce6d4d0 0x55ecdd23a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd951b92740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ecdd194460 con 0x55ecdce652f0 2024-04-25T16:01:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd951b92740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ecdd14e590 con 0x55ecdce6d4d0 2024-04-25T16:01:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd951b92740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ecdd190420 con 0x55ecdce8e1b0 2024-04-25T16:01:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd9489a1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce652f0 0x55ecdd233300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd9489a1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce652f0 0x55ecdd233300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56174/0 (socket says 172.21.15.31:56174) 2024-04-25T16:01:28.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd9489a1640 1 -- 172.21.15.31:0/2212282194 learned_addr learned my addr 172.21.15.31:0/2212282194 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd9491a2640 1 --2- 172.21.15.31:0/2212282194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce8e1b0 0x55ecdd232d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd943fff640 1 --2- 172.21.15.31:0/2212282194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce6d4d0 0x55ecdd23a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd9489a1640 1 -- 172.21.15.31:0/2212282194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce8e1b0 msgr2=0x55ecdd232d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd9489a1640 1 --2- 172.21.15.31:0/2212282194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce8e1b0 0x55ecdd232d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd9489a1640 1 -- 172.21.15.31:0/2212282194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce6d4d0 msgr2=0x55ecdd23a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd9489a1640 1 --2- 172.21.15.31:0/2212282194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce6d4d0 0x55ecdd23a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd9489a1640 1 -- 172.21.15.31:0/2212282194 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecdd15b0d0 con 0x55ecdce652f0 2024-04-25T16:01:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.844+0000 7fd941ffb640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd9340afae0 0x7fd9340b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.845+0000 7fd9489a1640 1 --2- 172.21.15.31:0/2212282194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce652f0 0x55ecdd233300 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd93800bc90 tx=0x7fd93800f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e2373069a69c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.845+0000 7fd9491a2640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd9340afae0 0x7fd9340b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.845+0000 7fd941ffb640 1 -- 172.21.15.31:0/730731563 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd9300def90 con 0x7fd944103b60 2024-04-25T16:01:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.845+0000 7fd92effd640 1 -- 172.21.15.31:0/2212282194 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938015070 con 0x55ecdce652f0 2024-04-25T16:01:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.845+0000 7fd92effd640 1 -- 172.21.15.31:0/2212282194 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd93801a910 con 0x55ecdce652f0 2024-04-25T16:01:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.845+0000 7fd92effd640 1 -- 172.21.15.31:0/2212282194 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938011860 con 0x55ecdce652f0 2024-04-25T16:01:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 -- 172.21.15.31:0/2212282194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce652f0 msgr2=0x55ecdd233300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd9491a2640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd9340afae0 0x7fd9340b1ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55ecdd233f50 tx=0x7fd93c009380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 --2- 172.21.15.31:0/2212282194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce652f0 0x55ecdd233300 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fd93800bc90 tx=0x7fd93800f730 comp rx=0 tx=0).stop 2024-04-25T16:01:28.859 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 -- 172.21.15.31:0/2212282194 shutdown_connections 2024-04-25T16:01:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 --2- 172.21.15.31:0/2212282194 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce6d4d0 0x55ecdd23a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 --2- 172.21.15.31:0/2212282194 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce652f0 0x55ecdd233300 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 --2- 172.21.15.31:0/2212282194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce8e1b0 0x55ecdd232d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 -- 172.21.15.31:0/2212282194 >> 172.21.15.31:0/2212282194 conn(0x55ecdce82300 msgr2=0x55ecdd22a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 -- 172.21.15.31:0/2212282194 shutdown_connections 2024-04-25T16:01:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 -- 172.21.15.31:0/2212282194 wait complete. 2024-04-25T16:01:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 -- start start 2024-04-25T16:01:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.846+0000 7fd951b92740 1 Processor -- start 2024-04-25T16:01:28.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd951b92740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd23cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd951b92740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd23d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd951b92740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd243f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd951b92740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ecdd23ac50 con 0x55ecdce8e1b0 2024-04-25T16:01:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd951b92740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ecdd11bb60 con 0x55ecdce652f0 2024-04-25T16:01:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd951b92740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ecdd23add0 con 0x55ecdce6d4d0 2024-04-25T16:01:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd951b92740 10 client.0 _ll_register_callbacks cb 0x55ecdd1402b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9491a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd243f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9491a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd243f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56188/0 (socket says 172.21.15.31:56188) 2024-04-25T16:01:28.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9489a1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd23cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9491a2640 1 -- 172.21.15.31:0/3038198208 learned_addr learned my addr 172.21.15.31:0/3038198208 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd943fff640 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd23d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9491a2640 1 -- 172.21.15.31:0/3038198208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 msgr2=0x55ecdd23d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9491a2640 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd23d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9491a2640 1 -- 172.21.15.31:0/3038198208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 msgr2=0x55ecdd23cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9491a2640 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd23cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9491a2640 1 -- 172.21.15.31:0/3038198208 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecdcea6210 con 0x55ecdce8e1b0 2024-04-25T16:01:28.862 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd9491a2640 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd243f10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd93c035000 tx=0x7fd93c02a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c0070a051ccccf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd923fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.847+0000 7fd923fff640 1 -- 172.21.15.31:0/3038198208 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c004310 con 0x55ecdce8e1b0 2024-04-25T16:01:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd923fff640 1 -- 172.21.15.31:0/3038198208 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd93c0044b0 con 0x55ecdce8e1b0 2024-04-25T16:01:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd951b92740 1 -- 172.21.15.31:0/3038198208 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd93801a910 con 0x55ecdce8e1b0 2024-04-25T16:01:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd923fff640 1 -- 172.21.15.31:0/3038198208 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd93c0046b0 con 0x55ecdce8e1b0 2024-04-25T16:01:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd951b92740 10 client.15495 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd951b92740 1 -- 172.21.15.31:0/3038198208 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ecdcf72620 con 0x55ecdce8e1b0 2024-04-25T16:01:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd923fff640 1 -- 172.21.15.31:0/3038198208 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd93801a910 con 0x55ecdce8e1b0 2024-04-25T16:01:28.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd923fff640 1 -- 172.21.15.31:0/3038198208 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd93c051070 con 0x55ecdce8e1b0 2024-04-25T16:01:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd951b92740 10 client.15495 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd951b92740 10 client.15495 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd951b92740 20 client.15495 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd951b92740 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ecdce4cd80 0x55ecdd24a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd951b92740 4 client.15495 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd951b92740 1 -- 172.21.15.31:0/3038198208 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ecdd1a2070 con 0x55ecdce4cd80 2024-04-25T16:01:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.848+0000 7fd9489a1640 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ecdce4cd80 0x55ecdd24a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.849+0000 7fd923fff640 10 client.15495 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:28.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.849+0000 7fd9489a1640 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ecdce4cd80 0x55ecdd24a850 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ed5796066d23b25 server_cookie=dec3a348011e7fcd in_seq=0 out_seq=0 2024-04-25T16:01:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd923fff640 1 -- 172.21.15.31:0/3038198208 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ecdd1a2070 con 0x55ecdce4cd80 2024-04-25T16:01:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd923fff640 10 client.15495 handle_command_reply: tid=0 2024-04-25T16:01:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 client.15495 shutdown 2024-04-25T16:01:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 -- 172.21.15.31:0/3038198208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 msgr2=0x55ecdd243f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd243f10 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fd93c035000 tx=0x7fd93c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 -- 172.21.15.31:0/3038198208 shutdown_connections 2024-04-25T16:01:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ecdce4cd80 0x55ecdd24a850 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd243f10 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd23d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd923fff640 0 client.15495 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 --2- 172.21.15.31:0/3038198208 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd23cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 -- 172.21.15.31:0/3038198208 >> 172.21.15.31:0/3038198208 conn(0x55ecdce82300 msgr2=0x55ecdd22d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 -- 172.21.15.31:0/3038198208 shutdown_connections 2024-04-25T16:01:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 1 -- 172.21.15.31:0/3038198208 wait complete. 2024-04-25T16:01:28.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.850+0000 7fd951b92740 20 client.15495 trim_cache size 0 max 16384 2024-04-25T16:01:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.865+0000 7fd951b92740 1 Processor -- start 2024-04-25T16:01:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.865+0000 7fd951b92740 1 -- start start 2024-04-25T16:01:28.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.865+0000 7fd951b92740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd236f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.865+0000 7fd951b92740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd237470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.865+0000 7fd951b92740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd229c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.865+0000 7fd951b92740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441a9000 con 0x55ecdce8e1b0 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.865+0000 7fd951b92740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441aab40 con 0x55ecdce652f0 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.865+0000 7fd951b92740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441a96d0 con 0x55ecdce6d4d0 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9491a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd229c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9491a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd229c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56198/0 (socket says 172.21.15.31:56198) 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9491a2640 1 -- 172.21.15.31:0/3776408897 learned_addr learned my addr 172.21.15.31:0/3776408897 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd943fff640 1 --2- 172.21.15.31:0/3776408897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd237470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9489a1640 1 --2- 172.21.15.31:0/3776408897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd236f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9491a2640 1 -- 172.21.15.31:0/3776408897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 msgr2=0x55ecdd237470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9491a2640 1 --2- 172.21.15.31:0/3776408897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd237470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.872 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9491a2640 1 -- 172.21.15.31:0/3776408897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 msgr2=0x55ecdd236f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9491a2640 1 --2- 172.21.15.31:0/3776408897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd236f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9491a2640 1 -- 172.21.15.31:0/3776408897 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ecdd23d6e0 con 0x55ecdce8e1b0 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd9491a2640 1 --2- 172.21.15.31:0/3776408897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd229c50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd93c02ade0 tx=0x7fd93c0317f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9286c8cfd3baddf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd92e7fc640 1 -- 172.21.15.31:0/3776408897 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c030e10 con 0x55ecdce8e1b0 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd92e7fc640 1 -- 172.21.15.31:0/3776408897 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd93c058030 con 0x55ecdce8e1b0 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.866+0000 7fd92e7fc640 1 -- 172.21.15.31:0/3776408897 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c049420 con 0x55ecdce8e1b0 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 -- 172.21.15.31:0/3776408897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 msgr2=0x55ecdd229c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 --2- 172.21.15.31:0/3776408897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd229c50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fd93c02ade0 tx=0x7fd93c0317f0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 -- 172.21.15.31:0/3776408897 shutdown_connections 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 --2- 172.21.15.31:0/3776408897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd229c50 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 --2- 172.21.15.31:0/3776408897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd237470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 --2- 172.21.15.31:0/3776408897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd236f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 -- 172.21.15.31:0/3776408897 >> 172.21.15.31:0/3776408897 conn(0x55ecdce82300 msgr2=0x55ecdd23b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:28.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 -- 172.21.15.31:0/3776408897 shutdown_connections 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 -- 172.21.15.31:0/3776408897 wait complete. 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 -- start start 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.867+0000 7fd951b92740 1 Processor -- start 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd951b92740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd240580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd951b92740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd240ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd951b92740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd2b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd951b92740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441aa470 con 0x55ecdce8e1b0 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd951b92740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441a9da0 con 0x55ecdce652f0 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd951b92740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441a7000 con 0x55ecdce6d4d0 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd951b92740 10 client.0 _ll_register_callbacks cb 0x7fd93c0318c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9491a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd2b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9491a2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd2b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56210/0 (socket says 172.21.15.31:56210) 2024-04-25T16:01:28.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9491a2640 1 -- 172.21.15.31:0/2198583493 learned_addr learned my addr 172.21.15.31:0/2198583493 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd943fff640 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd240ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9489a1640 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd240580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9491a2640 1 -- 172.21.15.31:0/2198583493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 msgr2=0x55ecdd240ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9491a2640 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd240ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9491a2640 1 -- 172.21.15.31:0/2198583493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 msgr2=0x55ecdd240580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9491a2640 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd240580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9491a2640 1 -- 172.21.15.31:0/2198583493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ecdd2b1120 con 0x55ecdce8e1b0 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd9491a2640 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd2b0bb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0099a0 tx=0x7fd93c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75b54e6bcdc8fa3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.868+0000 7fd923fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd923fff640 1 -- 172.21.15.31:0/2198583493 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c061040 con 0x55ecdce8e1b0 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd923fff640 1 -- 172.21.15.31:0/2198583493 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd93c030ae0 con 0x55ecdce8e1b0 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd951b92740 1 -- 172.21.15.31:0/2198583493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd93c058030 con 0x55ecdce8e1b0 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd923fff640 1 -- 172.21.15.31:0/2198583493 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd93c02fc60 con 0x55ecdce8e1b0 2024-04-25T16:01:28.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd951b92740 10 client.15501 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd951b92740 1 -- 172.21.15.31:0/2198583493 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ecdd2b2350 con 0x55ecdce8e1b0 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd923fff640 1 -- 172.21.15.31:0/2198583493 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd93c058030 con 0x55ecdce8e1b0 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd923fff640 1 -- 172.21.15.31:0/2198583493 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd93c051040 con 0x55ecdce8e1b0 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd951b92740 10 client.15501 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd951b92740 10 client.15501 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd951b92740 20 client.15501 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd951b92740 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ecdce221c0 0x55ecdd2b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd951b92740 4 client.15501 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.869+0000 7fd951b92740 1 -- 172.21.15.31:0/2198583493 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ecdd19bee0 con 0x55ecdce221c0 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.870+0000 7fd9489a1640 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ecdce221c0 0x55ecdd2b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.870+0000 7fd923fff640 10 client.15501 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:28.876 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.870+0000 7fd9489a1640 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ecdce221c0 0x55ecdd2b4c60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=302eeb1f6fe1ca54 server_cookie=25806d1d0ef60b26 in_seq=0 out_seq=0 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: pgmap v356: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:01:28.877 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd923fff640 1 -- 172.21.15.31:0/2198583493 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ecdd19bee0 con 0x55ecdce221c0 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd923fff640 10 client.15501 handle_command_reply: tid=0 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd951b92740 1 client.15501 shutdown 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd951b92740 1 -- 172.21.15.31:0/2198583493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 msgr2=0x55ecdd2b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd951b92740 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd2b0bb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fd93c0099a0 tx=0x7fd93c0095c0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd951b92740 1 -- 172.21.15.31:0/2198583493 shutdown_connections 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd951b92740 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ecdce221c0 0x55ecdd2b4c60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd951b92740 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ecdce8e1b0 0x55ecdd2b0bb0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd923fff640 0 client.15501 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd951b92740 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ecdce6d4d0 0x55ecdd240ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.880+0000 7fd951b92740 1 --2- 172.21.15.31:0/2198583493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ecdce652f0 0x55ecdd240580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd951b92740 1 -- 172.21.15.31:0/2198583493 >> 172.21.15.31:0/2198583493 conn(0x55ecdce82300 msgr2=0x55ecdd23b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd951b92740 1 -- 172.21.15.31:0/2198583493 shutdown_connections 2024-04-25T16:01:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd951b92740 1 -- 172.21.15.31:0/2198583493 wait complete. 2024-04-25T16:01:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd951b92740 20 client.15501 trim_cache size 0 max 16384 2024-04-25T16:01:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:01:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:01:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:01:28.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:01:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 -- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd9340afae0 msgr2=0x7fd9340b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd9340afae0 0x7fd9340b1ff0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x55ecdd233f50 tx=0x7fd93c009380 comp rx=0 tx=0).stop 2024-04-25T16:01:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 -- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd944103b60 msgr2=0x7fd94419a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd944103b60 0x7fd94419a4f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fd93000e9f0 tx=0x7fd93000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 -- 172.21.15.31:0/730731563 shutdown_connections 2024-04-25T16:01:28.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd9340afae0 0x7fd9340b1ff0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94419f470 0x7fd9441a1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd944103b60 0x7fd94419a4f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 --2- 172.21.15.31:0/730731563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd944103000 0x7fd944199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.881+0000 7fd94ac05640 1 -- 172.21.15.31:0/730731563 >> 172.21.15.31:0/730731563 conn(0x7fd9440fd3a0 msgr2=0x7fd9440fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.882+0000 7fd94ac05640 1 -- 172.21.15.31:0/730731563 shutdown_connections 2024-04-25T16:01:28.885 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:28.882+0000 7fd94ac05640 1 -- 172.21.15.31:0/730731563 wait complete. 2024-04-25T16:01:28.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: pgmap v356: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-25T16:01:28.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:01:28.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:01:28.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: pgmap v356: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-04-25T16:01:28.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:01:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:01:28.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:01:30.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:30 smithi063 ceph-mon[32413]: pgmap v357: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:01:30.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:30 smithi186 ceph-mon[33258]: pgmap v357: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:01:30.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:30 smithi031 ceph-mon[29140]: pgmap v357: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:01:32.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:32 smithi063 ceph-mon[32413]: pgmap v358: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:01:32.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:32 smithi186 ceph-mon[33258]: pgmap v358: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:01:32.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:32 smithi031 ceph-mon[29140]: pgmap v358: 97 pgs: 97 active+clean; 650 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:01:34.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:34 smithi063 ceph-mon[32413]: pgmap v359: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 16 op/s 2024-04-25T16:01:34.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:34 smithi186 ceph-mon[33258]: pgmap v359: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 16 op/s 2024-04-25T16:01:34.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:34 smithi031 ceph-mon[29140]: pgmap v359: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 16 op/s 2024-04-25T16:01:36.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:36 smithi063 ceph-mon[32413]: pgmap v360: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:01:36.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:36 smithi186 ceph-mon[33258]: pgmap v360: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:01:36.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:36 smithi031 ceph-mon[29140]: pgmap v360: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:01:38.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:38 smithi063 ceph-mon[32413]: pgmap v361: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-04-25T16:01:38.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:38 smithi186 ceph-mon[33258]: pgmap v361: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-04-25T16:01:38.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:38 smithi031 ceph-mon[29140]: pgmap v361: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-04-25T16:01:40.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:40 smithi063 ceph-mon[32413]: pgmap v362: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-04-25T16:01:40.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:40 smithi186 ceph-mon[33258]: pgmap v362: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-04-25T16:01:40.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:40 smithi031 ceph-mon[29140]: pgmap v362: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-04-25T16:01:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.993+0000 7fb8d1c96640 1 -- 172.21.15.31:0/2323845659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc108960 msgr2=0x7fb8cc10ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.993+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/2323845659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc108960 0x7fb8cc10ad80 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4009ca0 tx=0x7fb8c402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.993+0000 7fb8d1c96640 1 -- 172.21.15.31:0/2323845659 shutdown_connections 2024-04-25T16:01:41.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.993+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/2323845659 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc108960 0x7fb8cc10ad80 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.993+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/2323845659 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8cc103c60 0x7fb8cc104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.993+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/2323845659 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8cc102a20 0x7fb8cc103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.993+0000 7fb8d1c96640 1 -- 172.21.15.31:0/2323845659 >> 172.21.15.31:0/2323845659 conn(0x7fb8cc0fdc10 msgr2=0x7fb8cc1000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.993+0000 7fb8d1c96640 1 -- 172.21.15.31:0/2323845659 shutdown_connections 2024-04-25T16:01:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.993+0000 7fb8d1c96640 1 -- 172.21.15.31:0/2323845659 wait complete. 2024-04-25T16:01:41.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8d1c96640 1 Processor -- start 2024-04-25T16:01:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8d1c96640 1 -- start start 2024-04-25T16:01:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8d1c96640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8cc102a20 0x7fb8cc199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8d1c96640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc103c60 0x7fb8cc1995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8d1c96640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8cc108960 0x7fb8cc1a0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8d1c96640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc10e580 con 0x7fb8cc103c60 2024-04-25T16:01:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8d1c96640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc10e400 con 0x7fb8cc102a20 2024-04-25T16:01:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8d1c96640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc10e700 con 0x7fb8cc108960 2024-04-25T16:01:41.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc103c60 0x7fb8cc1995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc103c60 0x7fb8cc1995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48878/0 (socket says 172.21.15.31:48878) 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8caffd640 1 -- 172.21.15.31:0/3977477657 learned_addr learned my addr 172.21.15.31:0/3977477657 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8cbfff640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8cc108960 0x7fb8cc1a0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.994+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8cc102a20 0x7fb8cc199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8caffd640 1 -- 172.21.15.31:0/3977477657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8cc108960 msgr2=0x7fb8cc1a0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8caffd640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8cc108960 0x7fb8cc1a0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8caffd640 1 -- 172.21.15.31:0/3977477657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8cc102a20 msgr2=0x7fb8cc199090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8caffd640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8cc102a20 0x7fb8cc199090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8caffd640 1 -- 172.21.15.31:0/3977477657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8cc1a0bd0 con 0x7fb8cc103c60 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8caffd640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc103c60 0x7fb8cc1995d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009900 tx=0x7fb8c0009dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8c8ff9640 1 -- 172.21.15.31:0/3977477657 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c000d030 con 0x7fb8cc103c60 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8c8ff9640 1 -- 172.21.15.31:0/3977477657 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8c000d1d0 con 0x7fb8cc103c60 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8d1c96640 1 -- 172.21.15.31:0/3977477657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8cc06a420 con 0x7fb8cc103c60 2024-04-25T16:01:41.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8c8ff9640 1 -- 172.21.15.31:0/3977477657 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0005280 con 0x7fb8cc103c60 2024-04-25T16:01:41.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.995+0000 7fb8d1c96640 1 -- 172.21.15.31:0/3977477657 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8cc06a970 con 0x7fb8cc103c60 2024-04-25T16:01:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.998+0000 7fb8d8c27740 1 Processor -- start 2024-04-25T16:01:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.998+0000 7fb8d8c27740 1 -- start start 2024-04-25T16:01:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc38970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc38ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc3fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8c8ff9640 1 -- 172.21.15.31:0/3977477657 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb8c0005420 con 0x7fb8cc103c60 2024-04-25T16:01:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ea2db9e460 con 0x55ea2d8774d0 2024-04-25T16:01:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ea2db58590 con 0x55ea2d8981b0 2024-04-25T16:01:42.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ea2db9a420 con 0x55ea2d86f2f0 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc3fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8cbfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc38970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc3fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48884/0 (socket says 172.21.15.31:48884) 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8caffd640 1 -- 172.21.15.31:0/1864435685 learned_addr learned my addr 172.21.15.31:0/1864435685 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc38ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8cbfff640 1 -- 172.21.15.31:0/1864435685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 msgr2=0x55ea2dc38ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8cbfff640 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc38ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8cbfff640 1 -- 172.21.15.31:0/1864435685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 msgr2=0x55ea2dc3fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8cbfff640 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc3fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8cbfff640 1 -- 172.21.15.31:0/1864435685 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea2db650d0 con 0x55ea2d8981b0 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc38ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8caffd640 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc3fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:41.999+0000 7fb8c8ff9640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8b40afae0 0x7fb8b40b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.000+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8b40afae0 0x7fb8b40b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.000+0000 7fb8cbfff640 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc38970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4006010 tx=0x7fb8c40098b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b6ac89e5188e028 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.000+0000 7fb8c8ff9640 1 -- 172.21.15.31:0/3977477657 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb8c00dfed0 con 0x7fb8cc103c60 2024-04-25T16:01:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.000+0000 7fb8b1ffb640 1 -- 172.21.15.31:0/1864435685 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4031790 con 0x55ea2d8981b0 2024-04-25T16:01:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.000+0000 7fb8b1ffb640 1 -- 172.21.15.31:0/1864435685 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8c4036030 con 0x55ea2d8981b0 2024-04-25T16:01:42.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.000+0000 7fb8b1ffb640 1 -- 172.21.15.31:0/1864435685 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c4004840 con 0x55ea2d8981b0 2024-04-25T16:01:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 -- 172.21.15.31:0/1864435685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 msgr2=0x55ea2dc38970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:42.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc38970 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb8c4006010 tx=0x7fb8c40098b0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8b40afae0 0x7fb8b40b1ff0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x55ea2dc39d50 tx=0x7fb8bc00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 -- 172.21.15.31:0/1864435685 shutdown_connections 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc3fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc38ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/1864435685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc38970 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 -- 172.21.15.31:0/1864435685 >> 172.21.15.31:0/1864435685 conn(0x55ea2d88c300 msgr2=0x55ea2dc34300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 -- 172.21.15.31:0/1864435685 shutdown_connections 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 -- 172.21.15.31:0/1864435685 wait complete. 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.001+0000 7fb8d8c27740 1 -- start start 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8d8c27740 1 Processor -- start 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc42520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 0x55ea2dc42a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc499c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ea2dc40660 con 0x55ea2d86f2f0 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ea2db25b60 con 0x55ea2d8981b0 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ea2dc407e0 con 0x55ea2d8774d0 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8d8c27740 10 client.0 _ll_register_callbacks cb 0x55ea2db4a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc42520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc42520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48894/0 (socket says 172.21.15.31:48894) 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8cb7fe640 1 -- 172.21.15.31:0/2072604842 learned_addr learned my addr 172.21.15.31:0/2072604842 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 0x55ea2dc42a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8cbfff640 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8cb7fe640 1 -- 172.21.15.31:0/2072604842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 msgr2=0x55ea2dc42a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 0x55ea2dc42a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8cb7fe640 1 -- 172.21.15.31:0/2072604842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 msgr2=0x55ea2dc499c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc499c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.002+0000 7fb8cb7fe640 1 -- 172.21.15.31:0/2072604842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea2d8b0210 con 0x55ea2d86f2f0 2024-04-25T16:01:42.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.003+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc42520 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc010e30 tx=0x7fb8bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8edd815c253bd6ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.003+0000 7fb8aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.003+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2072604842 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc010e60 con 0x55ea2d86f2f0 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.003+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2072604842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8c4036030 con 0x55ea2d86f2f0 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.003+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2072604842 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb8bc027400 con 0x55ea2d86f2f0 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.003+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2072604842 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb8bc00fe90 con 0x55ea2d86f2f0 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.003+0000 7fb8d8c27740 10 client.15513 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.003+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2072604842 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ea2d97c620 con 0x55ea2d86f2f0 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.003+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2072604842 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8c4036030 con 0x55ea2d86f2f0 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2072604842 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb8bc032600 con 0x55ea2d86f2f0 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8d8c27740 10 client.15513 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8d8c27740 10 client.15513 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8d8c27740 20 client.15513 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ea2d856d80 0x55ea2dc50280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8d8c27740 4 client.15513 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2072604842 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ea2dbac070 con 0x55ea2d856d80 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ea2d856d80 0x55ea2dc50280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ea2d856d80 0x55ea2dc50280 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c6d126107f15f09 server_cookie=2ff4c8839f5df500 in_seq=0 out_seq=0 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.004+0000 7fb8aaffd640 10 client.15513 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.005+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2072604842 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ea2dbac070 con 0x55ea2d856d80 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.005+0000 7fb8aaffd640 10 client.15513 handle_command_reply: tid=0 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.005+0000 7fb8d8c27740 1 client.15513 shutdown 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.005+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2072604842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 msgr2=0x55ea2dc42520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.005+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc42520 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc010e30 tx=0x7fb8bc0046c0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.005+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2072604842 shutdown_connections 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.006+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ea2d856d80 0x55ea2dc50280 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.006+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc499c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.006+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 0x55ea2dc42a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.006+0000 7fb8aaffd640 0 client.15513 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.006+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2072604842 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc42520 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.006+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2072604842 >> 172.21.15.31:0/2072604842 conn(0x55ea2d88c300 msgr2=0x55ea2dc37210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.006+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2072604842 shutdown_connections 2024-04-25T16:01:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.006+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2072604842 wait complete. 2024-04-25T16:01:42.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.006+0000 7fb8d8c27740 20 client.15513 trim_cache size 0 max 16384 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.021+0000 7fb8d8c27740 1 Processor -- start 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.021+0000 7fb8d8c27740 1 -- start start 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.021+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc37100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.021+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc49950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.021+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc37640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.021+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc1a8fe0 con 0x55ea2d8774d0 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.021+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc1aab20 con 0x55ea2d8981b0 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.021+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc1a96b0 con 0x55ea2d86f2f0 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.021+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc49950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8caffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc49950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48904/0 (socket says 172.21.15.31:48904) 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8caffd640 1 -- 172.21.15.31:0/2385290398 learned_addr learned my addr 172.21.15.31:0/2385290398 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8cbfff640 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc37640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc37100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8caffd640 1 -- 172.21.15.31:0/2385290398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 msgr2=0x55ea2dc37100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc37100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8caffd640 1 -- 172.21.15.31:0/2385290398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 msgr2=0x55ea2dc37640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc37640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8caffd640 1 -- 172.21.15.31:0/2385290398 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ea2dc3fee0 con 0x55ea2d8774d0 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8cbfff640 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc37640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc49950 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb8c000d4a0 tx=0x7fb8c0091040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532ff25394de2290 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8b17fa640 1 -- 172.21.15.31:0/2385290398 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c001e040 con 0x55ea2d8774d0 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8b17fa640 1 -- 172.21.15.31:0/2385290398 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8c00a7f80 con 0x55ea2d8774d0 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.022+0000 7fb8b17fa640 1 -- 172.21.15.31:0/2385290398 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8c0007860 con 0x55ea2d8774d0 2024-04-25T16:01:42.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2385290398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 msgr2=0x55ea2dc49950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc49950 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fb8c000d4a0 tx=0x7fb8c0091040 comp rx=0 tx=0).stop 2024-04-25T16:01:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2385290398 shutdown_connections 2024-04-25T16:01:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc37640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d8774d0 0x55ea2dc49950 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2385290398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc37100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2385290398 >> 172.21.15.31:0/2385290398 conn(0x55ea2d88c300 msgr2=0x55ea2dc3cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2385290398 shutdown_connections 2024-04-25T16:01:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2385290398 wait complete. 2024-04-25T16:01:42.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.023+0000 7fb8d8c27740 1 -- start start 2024-04-25T16:01:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8d8c27740 1 Processor -- start 2024-04-25T16:01:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc47470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 0x55ea2dc479b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8d8c27740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc488e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc1aa450 con 0x55ea2d86f2f0 2024-04-25T16:01:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc1a9d80 con 0x55ea2d8981b0 2024-04-25T16:01:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8d8c27740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb8cc1a6fc0 con 0x55ea2d8774d0 2024-04-25T16:01:42.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8d8c27740 10 client.0 _ll_register_callbacks cb 0x7fb8bc005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc47470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc47470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48920/0 (socket says 172.21.15.31:48920) 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8cb7fe640 1 -- 172.21.15.31:0/2953711745 learned_addr learned my addr 172.21.15.31:0/2953711745 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 0x55ea2dc479b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8cbfff640 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8cb7fe640 1 -- 172.21.15.31:0/2953711745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 msgr2=0x55ea2dc479b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 0x55ea2dc479b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8cb7fe640 1 -- 172.21.15.31:0/2953711745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 msgr2=0x55ea2dc488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc488e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8cb7fe640 1 -- 172.21.15.31:0/2953711745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ea2dc48ea0 con 0x55ea2d86f2f0 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.024+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 0x55ea2dc479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8cb7fe640 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc47470 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc032aa0 tx=0x7fb8bc033f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56bd80790b866d8c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:42.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2953711745 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8bc037040 con 0x55ea2d86f2f0 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2953711745 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb8bc0325f0 con 0x55ea2d86f2f0 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2953711745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb8c00a7f80 con 0x55ea2d86f2f0 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2953711745 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb8bc010e60 con 0x55ea2d86f2f0 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8d8c27740 10 client.15519 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2953711745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ea2dcb8920 con 0x55ea2d86f2f0 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2953711745 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8c00a7f80 con 0x55ea2d86f2f0 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2953711745 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb8bc004aa0 con 0x55ea2d86f2f0 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8d8c27740 10 client.15519 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8d8c27740 10 client.15519 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8d8c27740 20 client.15519 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ea2d82c1c0 0x55ea2dcbb340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:42.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.025+0000 7fb8d8c27740 4 client.15519 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:01:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.026+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2953711745 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ea2dba5ee0 con 0x55ea2d82c1c0 2024-04-25T16:01:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.026+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ea2d82c1c0 0x55ea2dcbb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.026+0000 7fb8caffd640 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ea2d82c1c0 0x55ea2dcbb340 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=846fb1176fea8aaf server_cookie=82ae56ccfcd20e4 in_seq=0 out_seq=0 2024-04-25T16:01:42.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.026+0000 7fb8aaffd640 10 client.15519 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.033+0000 7fb8aaffd640 1 -- 172.21.15.31:0/2953711745 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ea2dba5ee0 con 0x55ea2d82c1c0 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.033+0000 7fb8aaffd640 10 client.15519 handle_command_reply: tid=0 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.033+0000 7fb8d8c27740 1 client.15519 shutdown 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2953711745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 msgr2=0x55ea2dc47470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc47470 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb8bc032aa0 tx=0x7fb8bc033f50 comp rx=0 tx=0).stop 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2953711745 shutdown_connections 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ea2d82c1c0 0x55ea2dcbb340 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ea2d8981b0 0x55ea2dc488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ea2d8774d0 0x55ea2dc479b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8aaffd640 0 client.15519 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:42.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 --2- 172.21.15.31:0/2953711745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ea2d86f2f0 0x55ea2dc47470 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2953711745 >> 172.21.15.31:0/2953711745 conn(0x55ea2d88c300 msgr2=0x55ea2dc42060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2953711745 shutdown_connections 2024-04-25T16:01:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 1 -- 172.21.15.31:0/2953711745 wait complete. 2024-04-25T16:01:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d8c27740 20 client.15519 trim_cache size 0 max 16384 2024-04-25T16:01:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:01:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:01:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:01:42.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:01:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 -- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8b40afae0 msgr2=0x7fb8b40b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8b40afae0 0x7fb8b40b1ff0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x55ea2dc39d50 tx=0x7fb8bc00a2d0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 -- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc103c60 msgr2=0x7fb8cc1995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc103c60 0x7fb8cc1995d0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb8c0009900 tx=0x7fb8c0009dc0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 -- 172.21.15.31:0/3977477657 shutdown_connections 2024-04-25T16:01:42.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8b40afae0 0x7fb8b40b1ff0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8cc108960 0x7fb8cc1a0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb8cc103c60 0x7fb8cc1995d0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 --2- 172.21.15.31:0/3977477657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb8cc102a20 0x7fb8cc199090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.034+0000 7fb8d1c96640 1 -- 172.21.15.31:0/3977477657 >> 172.21.15.31:0/3977477657 conn(0x7fb8cc0fdc10 msgr2=0x7fb8cc104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.035+0000 7fb8d1c96640 1 -- 172.21.15.31:0/3977477657 shutdown_connections 2024-04-25T16:01:42.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:42.035+0000 7fb8d1c96640 1 -- 172.21.15.31:0/3977477657 wait complete. 2024-04-25T16:01:42.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:42 smithi063 ceph-mon[32413]: pgmap v363: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-25T16:01:42.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:42 smithi186 ceph-mon[33258]: pgmap v363: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-25T16:01:42.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:42 smithi031 ceph-mon[29140]: pgmap v363: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-25T16:01:44.767 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:44 smithi031 ceph-mon[29140]: pgmap v364: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-04-25T16:01:44.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:44 smithi063 ceph-mon[32413]: pgmap v364: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-04-25T16:01:44.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:44 smithi186 ceph-mon[33258]: pgmap v364: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 14 op/s 2024-04-25T16:01:46.080 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:01:46.080 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:01:46.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.197+0000 7f27d2306640 1 -- 172.21.15.31:0/3743533809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc108920 msgr2=0x7f27cc10ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.197+0000 7f27d2306640 1 --2- 172.21.15.31:0/3743533809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc108920 0x7f27cc10ad40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009ca0 tx=0x7f27c402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.197+0000 7f27d2306640 1 -- 172.21.15.31:0/3743533809 shutdown_connections 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.197+0000 7f27d2306640 1 --2- 172.21.15.31:0/3743533809 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc108920 0x7f27cc10ad40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.197+0000 7f27d2306640 1 --2- 172.21.15.31:0/3743533809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27cc103c20 0x7f27cc104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.197+0000 7f27d2306640 1 --2- 172.21.15.31:0/3743533809 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27cc1029e0 0x7f27cc1036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.197+0000 7f27d2306640 1 -- 172.21.15.31:0/3743533809 >> 172.21.15.31:0/3743533809 conn(0x7f27cc0fdbd0 msgr2=0x7f27cc100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.197+0000 7f27d2306640 1 -- 172.21.15.31:0/3743533809 shutdown_connections 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.197+0000 7f27d2306640 1 -- 172.21.15.31:0/3743533809 wait complete. 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27d2306640 1 Processor -- start 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27d2306640 1 -- start start 2024-04-25T16:01:46.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27d2306640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc1029e0 0x7f27cc199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27d2306640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27cc103c20 0x7f27cc199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27d2306640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27cc108920 0x7f27cc1a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27d2306640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc10e4f0 con 0x7f27cc1029e0 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27d2306640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc10e370 con 0x7f27cc108920 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27d2306640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc10e670 con 0x7f27cc103c20 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27cbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc1029e0 0x7f27cc199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27cbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc1029e0 0x7f27cc199330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35328/0 (socket says 172.21.15.31:35328) 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27cbfff640 1 -- 172.21.15.31:0/507384193 learned_addr learned my addr 172.21.15.31:0/507384193 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27cc103c20 0x7f27cc199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.198+0000 7f27d08a3640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27cc108920 0x7f27cc1a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27cbfff640 1 -- 172.21.15.31:0/507384193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27cc103c20 msgr2=0x7f27cc199870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27cbfff640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27cc103c20 0x7f27cc199870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27cbfff640 1 -- 172.21.15.31:0/507384193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27cc108920 msgr2=0x7f27cc1a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27cbfff640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27cc108920 0x7f27cc1a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27cbfff640 1 -- 172.21.15.31:0/507384193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27cc1a0dc0 con 0x7f27cc1029e0 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27d08a3640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27cc108920 0x7f27cc1a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27cbfff640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc1029e0 0x7f27cc199330 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f27bc007b50 tx=0x7f27bc00e420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27c97fa640 1 -- 172.21.15.31:0/507384193 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc007c40 con 0x7f27cc1029e0 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27c97fa640 1 -- 172.21.15.31:0/507384193 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27bc007de0 con 0x7f27cc1029e0 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27c97fa640 1 -- 172.21.15.31:0/507384193 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00aca0 con 0x7f27cc1029e0 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27d2306640 1 -- 172.21.15.31:0/507384193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27cc069ef0 con 0x7f27cc1029e0 2024-04-25T16:01:46.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.199+0000 7f27d2306640 1 -- 172.21.15.31:0/507384193 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27cc06a440 con 0x7f27cc1029e0 2024-04-25T16:01:46.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.201+0000 7f27d91ba740 1 Processor -- start 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.201+0000 7f27d91ba740 1 -- start start 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.201+0000 7f27c97fa640 1 -- 172.21.15.31:0/507384193 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f27bc00eb30 con 0x7f27cc1029e0 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 0x55e9cfd57a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd57fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd58c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27d91ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e9cfcbcd80 con 0x55e9cf9956b0 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27d91ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e9cfc768b0 con 0x55e9cf98d4d0 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27d91ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e9cfcb8d40 con 0x55e9cf9b6390 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd58c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd58c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35344/0 (socket says 172.21.15.31:35344) 2024-04-25T16:01:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27cb7fe640 1 -- 172.21.15.31:0/1225296628 learned_addr learned my addr 172.21.15.31:0/1225296628 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27cbfff640 1 --2- 172.21.15.31:0/1225296628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd57fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27d08a3640 1 --2- 172.21.15.31:0/1225296628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 0x55e9cfd57a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27c97fa640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f27b40b3f00 0x7f27b40b6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27cb7fe640 1 -- 172.21.15.31:0/1225296628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 msgr2=0x55e9cfd57a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/1225296628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 0x55e9cfd57a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27cb7fe640 1 -- 172.21.15.31:0/1225296628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 msgr2=0x55e9cfd57fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/1225296628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd57fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27cb7fe640 1 -- 172.21.15.31:0/1225296628 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9cfc47a40 con 0x55e9cf9956b0 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27d08a3640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f27b40b3f00 0x7f27b40b6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.202+0000 7f27c97fa640 1 -- 172.21.15.31:0/507384193 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f27bc008230 con 0x7f27cc1029e0 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.203+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/1225296628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd58c70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f27c000cc20 tx=0x7f27c000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fff7e6a5d8d6989 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.203+0000 7f27b27fc640 1 -- 172.21.15.31:0/1225296628 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0014070 con 0x55e9cf9956b0 2024-04-25T16:01:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.203+0000 7f27b27fc640 1 -- 172.21.15.31:0/1225296628 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c00047a0 con 0x55e9cf9956b0 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.203+0000 7f27b27fc640 1 -- 172.21.15.31:0/1225296628 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0010650 con 0x55e9cf9956b0 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 -- 172.21.15.31:0/1225296628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 msgr2=0x55e9cfd58c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1225296628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd58c70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f27c000cc20 tx=0x7f27c000e720 comp rx=0 tx=0).stop 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d08a3640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f27b40b3f00 0x7f27b40b6410 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55e9cfd58ab0 tx=0x7f27c40095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 -- 172.21.15.31:0/1225296628 shutdown_connections 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1225296628 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd58c70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1225296628 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd57fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1225296628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 0x55e9cfd57a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 -- 172.21.15.31:0/1225296628 >> 172.21.15.31:0/1225296628 conn(0x55e9cf9aa4e0 msgr2=0x55e9cfd53270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 -- 172.21.15.31:0/1225296628 shutdown_connections 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 -- 172.21.15.31:0/1225296628 wait complete. 2024-04-25T16:01:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 -- start start 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.204+0000 7f27d91ba740 1 Processor -- start 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd61940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd61e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 0x55e9cfd68de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d91ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e9cfd59780 con 0x55e9cf9956b0 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d91ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e9cfc43d10 con 0x55e9cf98d4d0 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d91ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e9cfd59900 con 0x55e9cf9b6390 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d91ba740 10 client.0 _ll_register_callbacks cb 0x55e9cfc70210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd61e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd61e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35352/0 (socket says 172.21.15.31:35352) 2024-04-25T16:01:46.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cb7fe640 1 -- 172.21.15.31:0/1730771450 learned_addr learned my addr 172.21.15.31:0/1730771450 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cbfff640 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd61940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d08a3640 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 0x55e9cfd68de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cb7fe640 1 -- 172.21.15.31:0/1730771450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 msgr2=0x55e9cfd68de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 0x55e9cfd68de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cb7fe640 1 -- 172.21.15.31:0/1730771450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 msgr2=0x55e9cfd61940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd61940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cb7fe640 1 -- 172.21.15.31:0/1730771450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9cfa987d0 con 0x55e9cf9956b0 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d08a3640 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 0x55e9cfd68de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cbfff640 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd61940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd61e80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f27c0005250 tx=0x7f27c0005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=918a26c3c3c8233 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:46.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27ab7fe640 1 -- 172.21.15.31:0/1730771450 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c0014070 con 0x55e9cf9956b0 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27d91ba740 1 -- 172.21.15.31:0/1730771450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c00047a0 con 0x55e9cf9956b0 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.205+0000 7f27ab7fe640 1 -- 172.21.15.31:0/1730771450 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f27c0004020 con 0x55e9cf9956b0 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27ab7fe640 1 -- 172.21.15.31:0/1730771450 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f27c0004260 con 0x55e9cf9956b0 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27d91ba740 10 client.15531 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27d91ba740 1 -- 172.21.15.31:0/1730771450 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e9cf9ce3f0 con 0x55e9cf9956b0 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27ab7fe640 1 -- 172.21.15.31:0/1730771450 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c00047a0 con 0x55e9cf9956b0 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27ab7fe640 1 -- 172.21.15.31:0/1730771450 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f27c0010430 con 0x55e9cf9956b0 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27d91ba740 10 client.15531 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27d91ba740 10 client.15531 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27d91ba740 20 client.15531 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e9cf974f60 0x55e9cfd6f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27d91ba740 4 client.15531 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.206+0000 7f27d91ba740 1 -- 172.21.15.31:0/1730771450 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9cfcca990 con 0x55e9cf974f60 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.207+0000 7f27cbfff640 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e9cf974f60 0x55e9cfd6f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.207+0000 7f27cbfff640 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e9cf974f60 0x55e9cfd6f000 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee2790c296872625 server_cookie=259899d0f4d31347 in_seq=0 out_seq=0 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.207+0000 7f27ab7fe640 10 client.15531 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27ab7fe640 1 -- 172.21.15.31:0/1730771450 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e9cfcca990 con 0x55e9cf974f60 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27ab7fe640 10 client.15531 handle_command_reply: tid=0 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 client.15531 shutdown 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 -- 172.21.15.31:0/1730771450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 msgr2=0x55e9cfd61e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd61e80 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f27c0005250 tx=0x7f27c0005280 comp rx=0 tx=0).stop 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 -- 172.21.15.31:0/1730771450 shutdown_connections 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e9cf974f60 0x55e9cfd6f000 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9b6390 0x55e9cfd68de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd61e80 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27ab7fe640 0 client.15531 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:46.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1730771450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd61940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 -- 172.21.15.31:0/1730771450 >> 172.21.15.31:0/1730771450 conn(0x55e9cf9aa4e0 msgr2=0x55e9cfd56180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 -- 172.21.15.31:0/1730771450 shutdown_connections 2024-04-25T16:01:46.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 1 -- 172.21.15.31:0/1730771450 wait complete. 2024-04-25T16:01:46.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.208+0000 7f27d91ba740 20 client.15531 trim_cache size 0 max 16384 2024-04-25T16:01:46.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d91ba740 1 Processor -- start 2024-04-25T16:01:46.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d91ba740 1 -- start start 2024-04-25T16:01:46.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd5cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd68d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd60680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d91ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc1a8e70 con 0x55e9cf9956b0 2024-04-25T16:01:46.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d91ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc1aa9b0 con 0x55e9cf9aa4e0 2024-04-25T16:01:46.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d91ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc1a9540 con 0x55e9cf98d4d0 2024-04-25T16:01:46.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d08a3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd60680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.226+0000 7f27d08a3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd60680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35368/0 (socket says 172.21.15.31:35368) 2024-04-25T16:01:46.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27d08a3640 1 -- 172.21.15.31:0/1314269011 learned_addr learned my addr 172.21.15.31:0/1314269011 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd68d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27cbfff640 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd5cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27d08a3640 1 -- 172.21.15.31:0/1314269011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf98d4d0 msgr2=0x55e9cfd68d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27d08a3640 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd68d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27d08a3640 1 -- 172.21.15.31:0/1314269011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9aa4e0 msgr2=0x55e9cfd5cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27d08a3640 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd5cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27d08a3640 1 -- 172.21.15.31:0/1314269011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9cf98e0f0 con 0x55e9cf9956b0 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27cbfff640 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd5cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27d08a3640 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd60680 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f27c4031870 tx=0x7f27c4040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d5a32a4e41a8d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27b1ffb640 1 -- 172.21.15.31:0/1314269011 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c4051070 con 0x55e9cf9956b0 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27b1ffb640 1 -- 172.21.15.31:0/1314269011 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c4036040 con 0x55e9cf9956b0 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.227+0000 7f27b1ffb640 1 -- 172.21.15.31:0/1314269011 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27c402fdf0 con 0x55e9cf9956b0 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 -- 172.21.15.31:0/1314269011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 msgr2=0x55e9cfd60680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd60680 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f27c4031870 tx=0x7f27c4040000 comp rx=0 tx=0).stop 2024-04-25T16:01:46.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 -- 172.21.15.31:0/1314269011 shutdown_connections 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf9956b0 0x55e9cfd60680 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd68d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 --2- 172.21.15.31:0/1314269011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd5cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 -- 172.21.15.31:0/1314269011 >> 172.21.15.31:0/1314269011 conn(0x55e9cfd4e4d0 msgr2=0x55e9cfd5ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 -- 172.21.15.31:0/1314269011 shutdown_connections 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 -- 172.21.15.31:0/1314269011 wait complete. 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 -- start start 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.228+0000 7f27d91ba740 1 Processor -- start 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd661a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9956b0 0x55e9cfd666e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27d91ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd67610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27d91ba740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc1aa2e0 con 0x55e9cf98d4d0 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27d91ba740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc1a9c10 con 0x55e9cf9956b0 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27d91ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f27cc1a6e50 con 0x55e9cf9aa4e0 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27d91ba740 10 client.0 _ll_register_callbacks cb 0x7f27bc009bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd661a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd661a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35376/0 (socket says 172.21.15.31:35376) 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cbfff640 1 -- 172.21.15.31:0/2091586238 learned_addr learned my addr 172.21.15.31:0/2091586238 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9956b0 0x55e9cfd666e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27d08a3640 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd67610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cbfff640 1 -- 172.21.15.31:0/2091586238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9aa4e0 msgr2=0x55e9cfd67610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cbfff640 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd67610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cbfff640 1 -- 172.21.15.31:0/2091586238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9956b0 msgr2=0x55e9cfd666e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cbfff640 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9956b0 0x55e9cfd666e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cbfff640 1 -- 172.21.15.31:0/2091586238 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9cfc7d4b0 con 0x55e9cf98d4d0 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27d08a3640 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd67610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27cbfff640 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd661a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0dde70 tx=0x7f27bc009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7908d4d301774263 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.229+0000 7f27ab7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27ab7fe640 1 -- 172.21.15.31:0/2091586238 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27bc0a81a0 con 0x55e9cf98d4d0 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27ab7fe640 1 -- 172.21.15.31:0/2091586238 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f27bc0a4ae0 con 0x55e9cf98d4d0 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27d91ba740 1 -- 172.21.15.31:0/2091586238 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27c4036040 con 0x55e9cf98d4d0 2024-04-25T16:01:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27ab7fe640 1 -- 172.21.15.31:0/2091586238 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f27bc0a7fc0 con 0x55e9cf98d4d0 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27d91ba740 10 client.15537 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27d91ba740 1 -- 172.21.15.31:0/2091586238 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55e9cfd68d60 con 0x55e9cf98d4d0 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27ab7fe640 1 -- 172.21.15.31:0/2091586238 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f27c4036040 con 0x55e9cf98d4d0 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27ab7fe640 1 -- 172.21.15.31:0/2091586238 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f27bc014070 con 0x55e9cf98d4d0 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27d91ba740 10 client.15537 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27d91ba740 10 client.15537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27d91ba740 20 client.15537 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27d91ba740 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e9cf9b6390 0x55e9cfdd9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27d91ba740 4 client.15537 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.230+0000 7f27d91ba740 1 -- 172.21.15.31:0/2091586238 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e9cfcc4800 con 0x55e9cf9b6390 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.231+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e9cf9b6390 0x55e9cfdd9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.231+0000 7f27ab7fe640 10 client.15537 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.231+0000 7f27cb7fe640 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e9cf9b6390 0x55e9cfdd9700 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8460e5a9598a312e server_cookie=f739e149150749d4 in_seq=0 out_seq=0 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.231+0000 7f27ab7fe640 1 -- 172.21.15.31:0/2091586238 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e9cfcc4800 con 0x55e9cf9b6390 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.231+0000 7f27ab7fe640 10 client.15537 handle_command_reply: tid=0 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.231+0000 7f27d91ba740 1 client.15537 shutdown 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 -- 172.21.15.31:0/2091586238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf98d4d0 msgr2=0x55e9cfd661a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd661a0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f27bc0dde70 tx=0x7f27bc009ce0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 -- 172.21.15.31:0/2091586238 shutdown_connections 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e9cf9b6390 0x55e9cfdd9700 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e9cf9aa4e0 0x55e9cfd67610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e9cf9956b0 0x55e9cfd666e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27ab7fe640 0 client.15537 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 --2- 172.21.15.31:0/2091586238 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e9cf98d4d0 0x55e9cfd661a0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 -- 172.21.15.31:0/2091586238 >> 172.21.15.31:0/2091586238 conn(0x55e9cfd4e4d0 msgr2=0x55e9cfd519c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 -- 172.21.15.31:0/2091586238 shutdown_connections 2024-04-25T16:01:46.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 1 -- 172.21.15.31:0/2091586238 wait complete. 2024-04-25T16:01:46.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d91ba740 20 client.15537 trim_cache size 0 max 16384 2024-04-25T16:01:46.241 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:46.241 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:01:46.241 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:01:46.241 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:46.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 -- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f27b40b3f00 msgr2=0x7f27b40b6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f27b40b3f00 0x7f27b40b6410 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55e9cfd58ab0 tx=0x7f27c40095c0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 -- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc1029e0 msgr2=0x7f27cc199330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc1029e0 0x7f27cc199330 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f27bc007b50 tx=0x7f27bc00e420 comp rx=0 tx=0).stop 2024-04-25T16:01:46.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 -- 172.21.15.31:0/507384193 shutdown_connections 2024-04-25T16:01:46.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f27b40b3f00 0x7f27b40b6410 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f27cc108920 0x7f27cc1a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27cc103c20 0x7f27cc199870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 --2- 172.21.15.31:0/507384193 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f27cc1029e0 0x7f27cc199330 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 -- 172.21.15.31:0/507384193 >> 172.21.15.31:0/507384193 conn(0x7f27cc0fdbd0 msgr2=0x7f27cc104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 -- 172.21.15.31:0/507384193 shutdown_connections 2024-04-25T16:01:46.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.232+0000 7f27d2306640 1 -- 172.21.15.31:0/507384193 wait complete. 2024-04-25T16:01:46.244 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:01:46.245 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:01:46.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.356+0000 7fad84682640 1 -- 172.21.15.31:0/3207445539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c1029c0 msgr2=0x7fad7c1036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.356+0000 7fad84682640 1 --2- 172.21.15.31:0/3207445539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c1029c0 0x7fad7c1036c0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fad78009900 tx=0x7fad7802a410 comp rx=0 tx=0).stop 2024-04-25T16:01:46.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.356+0000 7fad84682640 1 -- 172.21.15.31:0/3207445539 shutdown_connections 2024-04-25T16:01:46.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.356+0000 7fad84682640 1 --2- 172.21.15.31:0/3207445539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c108900 0x7fad7c10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.356+0000 7fad84682640 1 --2- 172.21.15.31:0/3207445539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7c103c00 0x7fad7c104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.356+0000 7fad84682640 1 --2- 172.21.15.31:0/3207445539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c1029c0 0x7fad7c1036c0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.356+0000 7fad84682640 1 -- 172.21.15.31:0/3207445539 >> 172.21.15.31:0/3207445539 conn(0x7fad7c0fdbd0 msgr2=0x7fad7c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.356+0000 7fad84682640 1 -- 172.21.15.31:0/3207445539 shutdown_connections 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.356+0000 7fad84682640 1 -- 172.21.15.31:0/3207445539 wait complete. 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad84682640 1 Processor -- start 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad84682640 1 -- start start 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad84682640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1029c0 0x7fad7c19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad84682640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c103c00 0x7fad7c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad84682640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7c108900 0x7fad7c1a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad84682640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c10f5e0 con 0x7fad7c103c00 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad84682640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c10f460 con 0x7fad7c1029c0 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad84682640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fad7c10f760 con 0x7fad7c108900 2024-04-25T16:01:46.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad81c1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c103c00 0x7fad7c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad81c1d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c103c00 0x7fad7c19a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35388/0 (socket says 172.21.15.31:35388) 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad81c1d640 1 -- 172.21.15.31:0/827822910 learned_addr learned my addr 172.21.15.31:0/827822910 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad8241e640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1029c0 0x7fad7c19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad82c1f640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7c108900 0x7fad7c1a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad81c1d640 1 -- 172.21.15.31:0/827822910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7c108900 msgr2=0x7fad7c1a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad81c1d640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7c108900 0x7fad7c1a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad81c1d640 1 -- 172.21.15.31:0/827822910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1029c0 msgr2=0x7fad7c19a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad81c1d640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1029c0 0x7fad7c19a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.357+0000 7fad81c1d640 1 -- 172.21.15.31:0/827822910 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad7c1a1d70 con 0x7fad7c103c00 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.358+0000 7fad81c1d640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c103c00 0x7fad7c19a770 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00d950 tx=0x7fad6c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.358+0000 7fad737fe640 1 -- 172.21.15.31:0/827822910 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c017030 con 0x7fad7c103c00 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.358+0000 7fad737fe640 1 -- 172.21.15.31:0/827822910 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad6c004020 con 0x7fad7c103c00 2024-04-25T16:01:46.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.358+0000 7fad737fe640 1 -- 172.21.15.31:0/827822910 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad6c0047a0 con 0x7fad7c103c00 2024-04-25T16:01:46.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.358+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad7c1a2060 con 0x7fad7c103c00 2024-04-25T16:01:46.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.358+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad7c1a2600 con 0x7fad7c103c00 2024-04-25T16:01:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.359+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad7c107710 con 0x7fad7c103c00 2024-04-25T16:01:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.360+0000 7fad737fe640 1 -- 172.21.15.31:0/827822910 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fad6c009a30 con 0x7fad7c103c00 2024-04-25T16:01:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.361+0000 7fad737fe640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fad680afb70 0x7fad680b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.361+0000 7fad8241e640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fad680afb70 0x7fad680b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.361+0000 7fad737fe640 1 -- 172.21.15.31:0/827822910 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fad6c0de710 con 0x7fad7c103c00 2024-04-25T16:01:46.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.362+0000 7fad8241e640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fad680afb70 0x7fad680b2080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fad78009230 tx=0x7fad78031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.366+0000 7fad737fe640 1 -- 172.21.15.31:0/827822910 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad6c0a7300 con 0x7fad7c103c00 2024-04-25T16:01:46.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.595+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad7c02db50 con 0x7fad7c103c00 2024-04-25T16:01:46.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.595+0000 7fad737fe640 1 -- 172.21.15.31:0/827822910 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11917 (secure 0 0 0) 0x7fad6c0a6c00 con 0x7fad7c103c00 2024-04-25T16:01:46.596 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:01:46.596 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:01:09.454389+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24535,24521,24563,14631,24553]},"id":1}]} 2024-04-25T16:01:46.597 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-25T16:01:46.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fad680afb70 msgr2=0x7fad680b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fad680afb70 0x7fad680b2080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fad78009230 tx=0x7fad78031400 comp rx=0 tx=0).stop 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c103c00 msgr2=0x7fad7c19a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c103c00 0x7fad7c19a770 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00d950 tx=0x7fad6c00de10 comp rx=0 tx=0).stop 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 shutdown_connections 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fad680afb70 0x7fad680b2080 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fad7c108900 0x7fad7c1a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fad7c103c00 0x7fad7c19a770 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 --2- 172.21.15.31:0/827822910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fad7c1029c0 0x7fad7c19a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.597+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 >> 172.21.15.31:0/827822910 conn(0x7fad7c0fdbd0 msgr2=0x7fad7c1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.598+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 shutdown_connections 2024-04-25T16:01:46.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.598+0000 7fad84682640 1 -- 172.21.15.31:0/827822910 wait complete. 2024-04-25T16:01:46.608 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:46 smithi031 ceph-mon[29140]: pgmap v365: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T16:01:46.610 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:01:46.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 -- 172.21.15.31:0/743942625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4180106f30 msgr2=0x7f4180109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 --2- 172.21.15.31:0/743942625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4180106f30 0x7f4180109350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f4174009900 tx=0x7f417402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:46.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 -- 172.21.15.31:0/743942625 shutdown_connections 2024-04-25T16:01:46.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 --2- 172.21.15.31:0/743942625 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4180109890 0x7f418010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 --2- 172.21.15.31:0/743942625 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4180106f30 0x7f4180109350 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 --2- 172.21.15.31:0/743942625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41801045d0 0x7f41801069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 -- 172.21.15.31:0/743942625 >> 172.21.15.31:0/743942625 conn(0x7f41800fdbd0 msgr2=0x7f4180100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 -- 172.21.15.31:0/743942625 shutdown_connections 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 -- 172.21.15.31:0/743942625 wait complete. 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.724+0000 7f41861a8640 1 Processor -- start 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f41861a8640 1 -- start start 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f41861a8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41801045d0 0x7f418019a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f41861a8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4180106f30 0x7f418019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f41861a8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4180109890 0x7f41801a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f41861a8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f418010f640 con 0x7f41801045d0 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f41861a8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f418010f4c0 con 0x7f4180109890 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f41861a8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f418010f7c0 con 0x7f4180106f30 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41801045d0 0x7f418019a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41801045d0 0x7f418019a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35412/0 (socket says 172.21.15.31:35412) 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417f7fe640 1 -- 172.21.15.31:0/219340671 learned_addr learned my addr 172.21.15.31:0/219340671 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417ffff640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4180109890 0x7f41801a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417effd640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4180106f30 0x7f418019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417f7fe640 1 -- 172.21.15.31:0/219340671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4180106f30 msgr2=0x7f418019aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417f7fe640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4180106f30 0x7f418019aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417f7fe640 1 -- 172.21.15.31:0/219340671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4180109890 msgr2=0x7f41801a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417f7fe640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4180109890 0x7f41801a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417f7fe640 1 -- 172.21.15.31:0/219340671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41801a20e0 con 0x7f41801045d0 2024-04-25T16:01:46.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417ffff640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4180109890 0x7f41801a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:46.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.725+0000 7f417f7fe640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41801045d0 0x7f418019a5a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4170005fd0 tx=0x7f417000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.726+0000 7f417cff9640 1 -- 172.21.15.31:0/219340671 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170017070 con 0x7f41801045d0 2024-04-25T16:01:46.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.726+0000 7f417cff9640 1 -- 172.21.15.31:0/219340671 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4170009c30 con 0x7f41801045d0 2024-04-25T16:01:46.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.726+0000 7f41861a8640 1 -- 172.21.15.31:0/219340671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41801a23d0 con 0x7f41801045d0 2024-04-25T16:01:46.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.726+0000 7f417cff9640 1 -- 172.21.15.31:0/219340671 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4170003d30 con 0x7f41801045d0 2024-04-25T16:01:46.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.726+0000 7f41861a8640 1 -- 172.21.15.31:0/219340671 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41801a2970 con 0x7f41801045d0 2024-04-25T16:01:46.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.728+0000 7f417cff9640 1 -- 172.21.15.31:0/219340671 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4170003ed0 con 0x7f41801045d0 2024-04-25T16:01:46.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.728+0000 7f418d056740 1 Processor -- start 2024-04-25T16:01:46.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.728+0000 7f418d056740 1 -- start start 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.728+0000 7f418d056740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf95390 0x55d0fd337810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.728+0000 7f418d056740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd337d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.728+0000 7f418d056740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd338a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.728+0000 7f418d056740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd29bd80 con 0x55d0fcf746b0 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.728+0000 7f418d056740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd255a20 con 0x55d0fcf6c4d0 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f418d056740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd297d40 con 0x55d0fcf95390 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd338a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd338a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35422/0 (socket says 172.21.15.31:35422) 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417effd640 1 -- 172.21.15.31:0/1463846121 learned_addr learned my addr 172.21.15.31:0/1463846121 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417cff9640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41680afb70 0x7f41680b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417f7fe640 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd337d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417ffff640 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf95390 0x55d0fd337810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417ffff640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41680afb70 0x7f41680b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417cff9640 1 -- 172.21.15.31:0/219340671 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f41700de960 con 0x7f41801045d0 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417effd640 1 -- 172.21.15.31:0/1463846121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf95390 msgr2=0x55d0fd337810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417effd640 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf95390 0x55d0fd337810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417effd640 1 -- 172.21.15.31:0/1463846121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf6c4d0 msgr2=0x55d0fd337d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417effd640 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd337d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417effd640 1 -- 172.21.15.31:0/1463846121 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0fd260c30 con 0x55d0fcf746b0 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417f7fe640 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd337d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417effd640 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd338a20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4174005ab0 tx=0x7f4174004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457d70d9c703393f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.729+0000 7f417ffff640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41680afb70 0x7f41680b2080 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55d0fd338860 tx=0x7f4178009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f4165ffb640 1 -- 172.21.15.31:0/1463846121 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417403f070 con 0x55d0fcf746b0 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f4165ffb640 1 -- 172.21.15.31:0/1463846121 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4174004430 con 0x55d0fcf746b0 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f4165ffb640 1 -- 172.21.15.31:0/1463846121 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4174035030 con 0x55d0fcf746b0 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f418d056740 1 -- 172.21.15.31:0/1463846121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 msgr2=0x55d0fd338a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f418d056740 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd338a20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f4174005ab0 tx=0x7f4174004040 comp rx=0 tx=0).stop 2024-04-25T16:01:46.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f418d056740 1 -- 172.21.15.31:0/1463846121 shutdown_connections 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f418d056740 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd338a20 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f418d056740 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd337d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f418d056740 1 --2- 172.21.15.31:0/1463846121 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf95390 0x55d0fd337810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.730+0000 7f418d056740 1 -- 172.21.15.31:0/1463846121 >> 172.21.15.31:0/1463846121 conn(0x55d0fcf894e0 msgr2=0x55d0fd333020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 -- 172.21.15.31:0/1463846121 shutdown_connections 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 -- 172.21.15.31:0/1463846121 wait complete. 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 -- start start 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 Processor -- start 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd341560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd341aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf95390 0x55d0fd348a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd3394d0 con 0x55d0fcf746b0 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd222dc0 con 0x55d0fcf95390 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d0fd339650 con 0x55d0fcf6c4d0 2024-04-25T16:01:46.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f418d056740 10 client.0 _ll_register_callbacks cb 0x55d0fd23e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.731+0000 7f417effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd341aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd341aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35434/0 (socket says 172.21.15.31:35434) 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417effd640 1 -- 172.21.15.31:0/2474586927 learned_addr learned my addr 172.21.15.31:0/2474586927 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417f7fe640 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd341560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417ffff640 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf95390 0x55d0fd348a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417effd640 1 -- 172.21.15.31:0/2474586927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 msgr2=0x55d0fd341560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417effd640 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd341560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417effd640 1 -- 172.21.15.31:0/2474586927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf95390 msgr2=0x55d0fd348a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417effd640 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf95390 0x55d0fd348a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417effd640 1 -- 172.21.15.31:0/2474586927 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0fcfad3f0 con 0x55d0fcf746b0 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417ffff640 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf95390 0x55d0fd348a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f417effd640 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd341aa0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4174005f70 tx=0x7f4174004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12773fd2dc21c096 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f415effd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:46.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f415effd640 1 -- 172.21.15.31:0/2474586927 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417403f030 con 0x55d0fcf746b0 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f418d056740 1 -- 172.21.15.31:0/2474586927 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4174004430 con 0x55d0fcf746b0 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f415effd640 1 -- 172.21.15.31:0/2474586927 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4174003a00 con 0x55d0fcf746b0 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f415effd640 1 -- 172.21.15.31:0/2474586927 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4174035040 con 0x55d0fcf746b0 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f418d056740 10 client.15555 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.732+0000 7f418d056740 1 -- 172.21.15.31:0/2474586927 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0fd0777d0 con 0x55d0fcf746b0 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f415effd640 1 -- 172.21.15.31:0/2474586927 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4174004430 con 0x55d0fcf746b0 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f415effd640 1 -- 172.21.15.31:0/2474586927 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f41740315a0 con 0x55d0fcf746b0 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f418d056740 10 client.15555 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f418d056740 10 client.15555 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f418d056740 20 client.15555 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f418d056740 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d0fcf53f60 0x55d0fd34ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f418d056740 4 client.15555 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:46.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f418d056740 1 -- 172.21.15.31:0/2474586927 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0fd2a9990 con 0x55d0fcf53f60 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f417f7fe640 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d0fcf53f60 0x55d0fd34ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f415effd640 10 client.15555 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.733+0000 7f417f7fe640 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d0fcf53f60 0x55d0fd34ee50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=50f172d673904f21 server_cookie=d3f7d1c315fcd548 in_seq=0 out_seq=0 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f415effd640 1 -- 172.21.15.31:0/2474586927 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d0fd2a9990 con 0x55d0fcf53f60 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f415effd640 10 client.15555 handle_command_reply: tid=0 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 client.15555 shutdown 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 -- 172.21.15.31:0/2474586927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 msgr2=0x55d0fd341aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd341aa0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f4174005f70 tx=0x7f4174004970 comp rx=0 tx=0).stop 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 -- 172.21.15.31:0/2474586927 shutdown_connections 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d0fcf53f60 0x55d0fd34ee50 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf95390 0x55d0fd348a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf746b0 0x55d0fd341aa0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f415effd640 0 client.15555 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 --2- 172.21.15.31:0/2474586927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd341560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 -- 172.21.15.31:0/2474586927 >> 172.21.15.31:0/2474586927 conn(0x55d0fcf894e0 msgr2=0x55d0fd332040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.734+0000 7f418d056740 1 -- 172.21.15.31:0/2474586927 shutdown_connections 2024-04-25T16:01:46.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.735+0000 7f418d056740 1 -- 172.21.15.31:0/2474586927 wait complete. 2024-04-25T16:01:46.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.735+0000 7f418d056740 20 client.15555 trim_cache size 0 max 16384 2024-04-25T16:01:46.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.748+0000 7f418d056740 1 Processor -- start 2024-04-25T16:01:46.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.748+0000 7f418d056740 1 -- start start 2024-04-25T16:01:46.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.748+0000 7f418d056740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd3403c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.748+0000 7f418d056740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 0x55d0fd348920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.748+0000 7f418d056740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd331610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.748+0000 7f418d056740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41801ab260 con 0x55d0fcf95390 2024-04-25T16:01:46.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.748+0000 7f418d056740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f41801acda0 con 0x55d0fcf746b0 2024-04-25T16:01:46.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.748+0000 7f418d056740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41801ab930 con 0x55d0fcf6c4d0 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd331610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd331610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35446/0 (socket says 172.21.15.31:35446) 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417ffff640 1 -- 172.21.15.31:0/1544897487 learned_addr learned my addr 172.21.15.31:0/1544897487 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417f7fe640 1 --2- 172.21.15.31:0/1544897487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd3403c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417effd640 1 --2- 172.21.15.31:0/1544897487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 0x55d0fd348920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417ffff640 1 -- 172.21.15.31:0/1544897487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 msgr2=0x55d0fd3403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417ffff640 1 --2- 172.21.15.31:0/1544897487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd3403c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417ffff640 1 -- 172.21.15.31:0/1544897487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 msgr2=0x55d0fd348920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417ffff640 1 --2- 172.21.15.31:0/1544897487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 0x55d0fd348920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417ffff640 1 -- 172.21.15.31:0/1544897487 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0fd348e60 con 0x55d0fcf95390 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f417ffff640 1 --2- 172.21.15.31:0/1544897487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd331610 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4178005bd0 tx=0x7f4178009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b899840beaf51a03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f41657fa640 1 -- 172.21.15.31:0/1544897487 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178004050 con 0x55d0fcf95390 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f41657fa640 1 -- 172.21.15.31:0/1544897487 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41780046a0 con 0x55d0fcf95390 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.749+0000 7f41657fa640 1 -- 172.21.15.31:0/1544897487 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417802bc40 con 0x55d0fcf95390 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 -- 172.21.15.31:0/1544897487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 msgr2=0x55d0fd331610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 --2- 172.21.15.31:0/1544897487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd331610 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f4178005bd0 tx=0x7f4178009920 comp rx=0 tx=0).stop 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 -- 172.21.15.31:0/1544897487 shutdown_connections 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 --2- 172.21.15.31:0/1544897487 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd331610 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 --2- 172.21.15.31:0/1544897487 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 0x55d0fd348920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 --2- 172.21.15.31:0/1544897487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd3403c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 -- 172.21.15.31:0/1544897487 >> 172.21.15.31:0/1544897487 conn(0x55d0fcf894e0 msgr2=0x55d0fd33fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 -- 172.21.15.31:0/1544897487 shutdown_connections 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 -- 172.21.15.31:0/1544897487 wait complete. 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.750+0000 7f418d056740 1 -- start start 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f418d056740 1 Processor -- start 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f418d056740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd346030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f418d056740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 0x55d0fd346570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f418d056740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd3474a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f418d056740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41801ac6d0 con 0x55d0fcf95390 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f418d056740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f41801ac000 con 0x55d0fcf746b0 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f418d056740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41801a9240 con 0x55d0fcf6c4d0 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f418d056740 10 client.0 _ll_register_callbacks cb 0x7f41700a6bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:46.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd3474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd3474a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35454/0 (socket says 172.21.15.31:35454) 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417ffff640 1 -- 172.21.15.31:0/1813458988 learned_addr learned my addr 172.21.15.31:0/1813458988 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417f7fe640 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd346030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417effd640 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 0x55d0fd346570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417ffff640 1 -- 172.21.15.31:0/1813458988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 msgr2=0x55d0fd346030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417ffff640 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd346030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417ffff640 1 -- 172.21.15.31:0/1813458988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 msgr2=0x55d0fd346570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417ffff640 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 0x55d0fd346570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.751+0000 7f417ffff640 1 -- 172.21.15.31:0/1813458988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0fd347a60 con 0x55d0fcf95390 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f417ffff640 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd3474a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4178009cb0 tx=0x7f4178005e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33044167ecdf6ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f415effd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f415effd640 1 -- 172.21.15.31:0/1813458988 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4178039060 con 0x55d0fcf95390 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f415effd640 1 -- 172.21.15.31:0/1813458988 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f417802b980 con 0x55d0fcf95390 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f418d056740 1 -- 172.21.15.31:0/1813458988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41780046a0 con 0x55d0fcf95390 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f415effd640 1 -- 172.21.15.31:0/1813458988 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4178004850 con 0x55d0fcf95390 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f418d056740 10 client.15561 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f418d056740 1 -- 172.21.15.31:0/1813458988 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d0fd3b70d0 con 0x55d0fcf95390 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f415effd640 1 -- 172.21.15.31:0/1813458988 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41780046a0 con 0x55d0fcf95390 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f415effd640 1 -- 172.21.15.31:0/1813458988 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f417801d040 con 0x55d0fcf95390 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f418d056740 10 client.15561 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f418d056740 10 client.15561 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f418d056740 20 client.15561 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f418d056740 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d0fcf293a0 0x55d0fd3b9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f418d056740 4 client.15561 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.752+0000 7f418d056740 1 -- 172.21.15.31:0/1813458988 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d0fd2a3800 con 0x55d0fcf293a0 2024-04-25T16:01:46.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.753+0000 7f417f7fe640 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d0fcf293a0 0x55d0fd3b9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.753+0000 7f415effd640 10 client.15561 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.753+0000 7f417f7fe640 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d0fcf293a0 0x55d0fd3b9980 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9715e4cee06c775b server_cookie=c4253a633fa6e34d in_seq=0 out_seq=0 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.753+0000 7f415effd640 1 -- 172.21.15.31:0/1813458988 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d0fd2a3800 con 0x55d0fcf293a0 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.753+0000 7f415effd640 10 client.15561 handle_command_reply: tid=0 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.753+0000 7f418d056740 1 client.15561 shutdown 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.753+0000 7f418d056740 1 -- 172.21.15.31:0/1813458988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 msgr2=0x55d0fd3474a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.753+0000 7f418d056740 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd3474a0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f4178009cb0 tx=0x7f4178005e10 comp rx=0 tx=0).stop 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f418d056740 1 -- 172.21.15.31:0/1813458988 shutdown_connections 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f418d056740 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d0fcf293a0 0x55d0fd3b9980 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f418d056740 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0fcf95390 0x55d0fd3474a0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f418d056740 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0fcf746b0 0x55d0fd346570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f415effd640 0 client.15561 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:01:46.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f418d056740 1 --2- 172.21.15.31:0/1813458988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0fcf6c4d0 0x55d0fd346030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f418d056740 1 -- 172.21.15.31:0/1813458988 >> 172.21.15.31:0/1813458988 conn(0x55d0fcf894e0 msgr2=0x55d0fd335e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f418d056740 1 -- 172.21.15.31:0/1813458988 shutdown_connections 2024-04-25T16:01:46.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f418d056740 1 -- 172.21.15.31:0/1813458988 wait complete. 2024-04-25T16:01:46.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f418d056740 20 client.15561 trim_cache size 0 max 16384 2024-04-25T16:01:46.763 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:01:46.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 -- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41680afb70 msgr2=0x7f41680b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41680afb70 0x7f41680b2080 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x55d0fd338860 tx=0x7f4178009500 comp rx=0 tx=0).stop 2024-04-25T16:01:46.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 -- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41801045d0 msgr2=0x7f418019a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41801045d0 0x7f418019a5a0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f4170005fd0 tx=0x7f417000dde0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 -- 172.21.15.31:0/219340671 shutdown_connections 2024-04-25T16:01:46.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41680afb70 0x7f41680b2080 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4180109890 0x7f41801a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4180106f30 0x7f418019aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 --2- 172.21.15.31:0/219340671 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41801045d0 0x7f418019a5a0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 -- 172.21.15.31:0/219340671 >> 172.21.15.31:0/219340671 conn(0x7f41800fdbd0 msgr2=0x7f4180105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 -- 172.21.15.31:0/219340671 shutdown_connections 2024-04-25T16:01:46.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.754+0000 7f41861a8640 1 -- 172.21.15.31:0/219340671 wait complete. 2024-04-25T16:01:46.767 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:01:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:46 smithi063 ceph-mon[32413]: pgmap v365: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T16:01:46.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 -- 172.21.15.31:0/182466352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d20106f30 msgr2=0x7f6d20109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/182466352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d20106f30 0x7f6d20109350 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6d14009900 tx=0x7f6d1402a3f0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 -- 172.21.15.31:0/182466352 shutdown_connections 2024-04-25T16:01:46.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/182466352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d20109890 0x7f6d2010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/182466352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d20106f30 0x7f6d20109350 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/182466352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d201045d0 0x7f6d201069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 -- 172.21.15.31:0/182466352 >> 172.21.15.31:0/182466352 conn(0x7f6d200fdbd0 msgr2=0x7f6d20100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 -- 172.21.15.31:0/182466352 shutdown_connections 2024-04-25T16:01:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 -- 172.21.15.31:0/182466352 wait complete. 2024-04-25T16:01:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.879+0000 7f6d26ad1640 1 Processor -- start 2024-04-25T16:01:46.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d26ad1640 1 -- start start 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d26ad1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d201045d0 0x7f6d2019a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d26ad1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d20106f30 0x7f6d2019ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d26ad1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d20109890 0x7f6d201a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d26ad1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2010f610 con 0x7f6d201045d0 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d26ad1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2010f490 con 0x7f6d20109890 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d26ad1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2010f790 con 0x7f6d20106f30 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2486d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d201045d0 0x7f6d2019a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2486d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d201045d0 0x7f6d2019a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35484/0 (socket says 172.21.15.31:35484) 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2486d640 1 -- 172.21.15.31:0/2392807544 learned_addr learned my addr 172.21.15.31:0/2392807544 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2506e640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d20109890 0x7f6d201a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d20106f30 0x7f6d2019ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2486d640 1 -- 172.21.15.31:0/2392807544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d20106f30 msgr2=0x7f6d2019ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2486d640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d20106f30 0x7f6d2019ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2486d640 1 -- 172.21.15.31:0/2392807544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d20109890 msgr2=0x7f6d201a1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2486d640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d20109890 0x7f6d201a1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2486d640 1 -- 172.21.15.31:0/2392807544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d201a2110 con 0x7f6d201045d0 2024-04-25T16:01:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.880+0000 7f6d2486d640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d201045d0 0x7f6d2019a5d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800ea50 tx=0x7f6d1800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.881+0000 7f6d1dffb640 1 -- 172.21.15.31:0/2392807544 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1800cdc0 con 0x7f6d201045d0 2024-04-25T16:01:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.881+0000 7f6d26ad1640 1 -- 172.21.15.31:0/2392807544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d201a2400 con 0x7f6d201045d0 2024-04-25T16:01:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.881+0000 7f6d1dffb640 1 -- 172.21.15.31:0/2392807544 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d18005020 con 0x7f6d201045d0 2024-04-25T16:01:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.881+0000 7f6d1dffb640 1 -- 172.21.15.31:0/2392807544 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d180116b0 con 0x7f6d201045d0 2024-04-25T16:01:46.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.881+0000 7f6d26ad1640 1 -- 172.21.15.31:0/2392807544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d201a2900 con 0x7f6d201045d0 2024-04-25T16:01:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.883+0000 7f6d1dffb640 1 -- 172.21.15.31:0/2392807544 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6d180051d0 con 0x7f6d201045d0 2024-04-25T16:01:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.883+0000 7f6d2d97d740 1 Processor -- start 2024-04-25T16:01:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.883+0000 7f6d2d97d740 1 -- start start 2024-04-25T16:01:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 0x5643cae9c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad14d0 0x5643cae9cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643cae9da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5643cae00d80 con 0x5643caad96b0 2024-04-25T16:01:46.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643cadbaa20 con 0x5643caad14d0 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5643cadfcd40 con 0x5643caafa390 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643cae9da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643cae9da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35494/0 (socket says 172.21.15.31:35494) 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1ffff640 1 -- 172.21.15.31:0/4254426357 learned_addr learned my addr 172.21.15.31:0/4254426357 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2486d640 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad14d0 0x5643cae9cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1dffb640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d080afaf0 0x7f6d080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2506e640 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 0x5643cae9c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1ffff640 1 -- 172.21.15.31:0/4254426357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 msgr2=0x5643cae9c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2506e640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d080afaf0 0x7f6d080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 0x5643cae9c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1ffff640 1 -- 172.21.15.31:0/4254426357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad14d0 msgr2=0x5643cae9cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1dffb640 1 -- 172.21.15.31:0/2392807544 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6d180deec0 con 0x7f6d201045d0 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad14d0 0x5643cae9cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1ffff640 1 -- 172.21.15.31:0/4254426357 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643cadc5c30 con 0x5643caad96b0 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d2486d640 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad14d0 0x5643cae9cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.884+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643cae9da20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006010 tx=0x7f6d14032bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f340497ab632fa35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.885+0000 7f6d2506e640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d080afaf0 0x7f6d080b2000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x5643cae9d860 tx=0x7f6d10004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.885+0000 7f6d06ffd640 1 -- 172.21.15.31:0/4254426357 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1403f030 con 0x5643caad96b0 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.885+0000 7f6d06ffd640 1 -- 172.21.15.31:0/4254426357 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d14032e20 con 0x5643caad96b0 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.885+0000 7f6d06ffd640 1 -- 172.21.15.31:0/4254426357 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d14035070 con 0x5643caad96b0 2024-04-25T16:01:46.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.885+0000 7f6d2d97d740 1 -- 172.21.15.31:0/4254426357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 msgr2=0x5643cae9da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.885+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643cae9da20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6d14006010 tx=0x7f6d14032bf0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.886+0000 7f6d2d97d740 1 -- 172.21.15.31:0/4254426357 shutdown_connections 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.886+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643cae9da20 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.886+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad14d0 0x5643cae9cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.886+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/4254426357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 0x5643cae9c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.886+0000 7f6d2d97d740 1 -- 172.21.15.31:0/4254426357 >> 172.21.15.31:0/4254426357 conn(0x5643caaee4e0 msgr2=0x5643cae98020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.886+0000 7f6d2d97d740 1 -- 172.21.15.31:0/4254426357 shutdown_connections 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.886+0000 7f6d2d97d740 1 -- 172.21.15.31:0/4254426357 wait complete. 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.886+0000 7f6d2d97d740 1 -- start start 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.886+0000 7f6d2d97d740 1 Processor -- start 2024-04-25T16:01:46.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 0x5643caea6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 0x5643caea6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caafa390 0x5643caead9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5643cae9e4d0 con 0x5643caafa390 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5643cad87dc0 con 0x5643caad96b0 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5643cae9e650 con 0x5643caad14d0 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2d97d740 10 client.0 _ll_register_callbacks cb 0x5643cada3600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2506e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caafa390 0x5643caead9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2506e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caafa390 0x5643caead9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35498/0 (socket says 172.21.15.31:35498) 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2506e640 1 -- 172.21.15.31:0/2461749204 learned_addr learned my addr 172.21.15.31:0/2461749204 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 0x5643caea6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2486d640 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 0x5643caea6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2506e640 1 -- 172.21.15.31:0/2461749204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 msgr2=0x5643caea6500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2506e640 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 0x5643caea6500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2506e640 1 -- 172.21.15.31:0/2461749204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 msgr2=0x5643caea6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2506e640 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 0x5643caea6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2506e640 1 -- 172.21.15.31:0/2461749204 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643cab123f0 con 0x5643caafa390 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2506e640 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caafa390 0x5643caead9a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6d10003fb0 tx=0x7f6d100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d9adaad2cb720b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6cfffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6cfffff640 1 -- 172.21.15.31:0/2461749204 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d10024070 con 0x5643caafa390 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6cfffff640 1 -- 172.21.15.31:0/2461749204 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6d10012870 con 0x5643caafa390 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.887+0000 7f6d2d97d740 1 -- 172.21.15.31:0/2461749204 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d14032e20 con 0x5643caafa390 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6cfffff640 1 -- 172.21.15.31:0/2461749204 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d10030050 con 0x5643caafa390 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6d2d97d740 10 client.15573 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:46.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6d2d97d740 1 -- 172.21.15.31:0/2461749204 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643cabdc7d0 con 0x5643caafa390 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6cfffff640 1 -- 172.21.15.31:0/2461749204 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d14032e20 con 0x5643caafa390 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6cfffff640 1 -- 172.21.15.31:0/2461749204 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6d1001c070 con 0x5643caafa390 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6d2d97d740 10 client.15573 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6d2d97d740 10 client.15573 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6d2d97d740 20 client.15573 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5643caab8f60 0x5643caeb42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6d2d97d740 4 client.15573 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6d2d97d740 1 -- 172.21.15.31:0/2461749204 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5643cae0e990 con 0x5643caab8f60 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.888+0000 7f6d2486d640 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5643caab8f60 0x5643caeb42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.889+0000 7f6cfffff640 10 client.15573 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.889+0000 7f6d2486d640 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5643caab8f60 0x5643caeb42e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4fa275762712ae64 server_cookie=f1ed6ea6ee453b91 in_seq=0 out_seq=0 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.889+0000 7f6cfffff640 1 -- 172.21.15.31:0/2461749204 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5643cae0e990 con 0x5643caab8f60 2024-04-25T16:01:46.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.889+0000 7f6cfffff640 10 client.15573 handle_command_reply: tid=0 2024-04-25T16:01:46.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.889+0000 7f6d2d97d740 1 client.15573 shutdown 2024-04-25T16:01:46.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 -- 172.21.15.31:0/2461749204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caafa390 msgr2=0x5643caead9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caafa390 0x5643caead9a0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6d10003fb0 tx=0x7f6d100046c0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 -- 172.21.15.31:0/2461749204 shutdown_connections 2024-04-25T16:01:46.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caafa390 0x5643caead9a0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 0x5643caea6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5643caab8f60 0x5643caeb42e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/2461749204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 0x5643caea6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6cfffff640 0 client.15573 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:01:46.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 -- 172.21.15.31:0/2461749204 >> 172.21.15.31:0/2461749204 conn(0x5643caaee4e0 msgr2=0x5643cae97040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 -- 172.21.15.31:0/2461749204 shutdown_connections 2024-04-25T16:01:46.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 1 -- 172.21.15.31:0/2461749204 wait complete. 2024-04-25T16:01:46.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.890+0000 7f6d2d97d740 20 client.15573 trim_cache size 0 max 16384 2024-04-25T16:01:46.900 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:46 smithi186 ceph-mon[33258]: pgmap v365: 97 pgs: 97 active+clean; 653 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-25T16:01:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.903+0000 7f6d2d97d740 1 Processor -- start 2024-04-25T16:01:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.903+0000 7f6d2d97d740 1 -- start start 2024-04-25T16:01:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 0x5643caea1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643caea17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caafa390 0x5643caead8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201ab2c0 con 0x5643caad96b0 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201ace00 con 0x5643caafa390 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201ab990 con 0x5643caad14d0 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d1ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643caea17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d1ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643caea17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35500/0 (socket says 172.21.15.31:35500) 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d1ffff640 1 -- 172.21.15.31:0/1971722100 learned_addr learned my addr 172.21.15.31:0/1971722100 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d2486d640 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 0x5643caea1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d2506e640 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caafa390 0x5643caead8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d1ffff640 1 -- 172.21.15.31:0/1971722100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 msgr2=0x5643caea1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 0x5643caea1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d1ffff640 1 -- 172.21.15.31:0/1971722100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caafa390 msgr2=0x5643caead8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caafa390 0x5643caead8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d1ffff640 1 -- 172.21.15.31:0/1971722100 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5643caeade00 con 0x5643caad96b0 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d2506e640 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caafa390 0x5643caead8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:46.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643caea17d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6d14005f70 tx=0x7f6d1402aa70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc88ab5b0f347b98 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d067fc640 1 -- 172.21.15.31:0/1971722100 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d14035040 con 0x5643caad96b0 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d067fc640 1 -- 172.21.15.31:0/1971722100 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d14048030 con 0x5643caad96b0 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.904+0000 7f6d067fc640 1 -- 172.21.15.31:0/1971722100 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1403f040 con 0x5643caad96b0 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 -- 172.21.15.31:0/1971722100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 msgr2=0x5643caea17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643caea17d0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6d14005f70 tx=0x7f6d1402aa70 comp rx=0 tx=0).stop 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 -- 172.21.15.31:0/1971722100 shutdown_connections 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caafa390 0x5643caead8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad96b0 0x5643caea17d0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/1971722100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caad14d0 0x5643caea1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 -- 172.21.15.31:0/1971722100 >> 172.21.15.31:0/1971722100 conn(0x5643caaee4e0 msgr2=0x5643cae9c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 -- 172.21.15.31:0/1971722100 shutdown_connections 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 -- 172.21.15.31:0/1971722100 wait complete. 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.905+0000 7f6d2d97d740 1 -- start start 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2d97d740 1 Processor -- start 2024-04-25T16:01:46.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad14d0 0x5643caeaa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 0x5643caeaaf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2d97d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 0x5643caeabe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201ac730 con 0x5643caad14d0 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201ac060 con 0x5643caad96b0 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2d97d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6d201a92a0 con 0x5643caafa390 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2d97d740 10 client.0 _ll_register_callbacks cb 0x7f6d100042a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2486d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad14d0 0x5643caeaa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2486d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad14d0 0x5643caeaa9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35512/0 (socket says 172.21.15.31:35512) 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2486d640 1 -- 172.21.15.31:0/725654023 learned_addr learned my addr 172.21.15.31:0/725654023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 0x5643caeaaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2506e640 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 0x5643caeabe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2486d640 1 -- 172.21.15.31:0/725654023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 msgr2=0x5643caeabe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2486d640 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 0x5643caeabe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2486d640 1 -- 172.21.15.31:0/725654023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 msgr2=0x5643caeaaf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2486d640 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 0x5643caeaaf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2486d640 1 -- 172.21.15.31:0/725654023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5643caeac3f0 con 0x5643caad14d0 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d2506e640 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 0x5643caeabe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.906+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 0x5643caeaaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2486d640 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad14d0 0x5643caeaa9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6d180dc9d0 tx=0x7f6d180df0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4133c32ad5d67343 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6cfffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6cfffff640 1 -- 172.21.15.31:0/725654023 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d180c1060 con 0x5643caad14d0 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6cfffff640 1 -- 172.21.15.31:0/725654023 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6d180a1b40 con 0x5643caad14d0 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2d97d740 1 -- 172.21.15.31:0/725654023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d14048030 con 0x5643caad14d0 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6cfffff640 1 -- 172.21.15.31:0/725654023 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6d180c8050 con 0x5643caad14d0 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2d97d740 10 client.15579 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2d97d740 1 -- 172.21.15.31:0/725654023 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5643caead6b0 con 0x5643caad14d0 2024-04-25T16:01:46.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6cfffff640 1 -- 172.21.15.31:0/725654023 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6d14048030 con 0x5643caad14d0 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6cfffff640 1 -- 172.21.15.31:0/725654023 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6d1800f040 con 0x5643caad14d0 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2d97d740 10 client.15579 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2d97d740 10 client.15579 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2d97d740 20 client.15579 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5643caa8e3a0 0x5643caf1e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2d97d740 4 client.15579 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.907+0000 7f6d2d97d740 1 -- 172.21.15.31:0/725654023 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5643cae08800 con 0x5643caa8e3a0 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5643caa8e3a0 0x5643caf1e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d1ffff640 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5643caa8e3a0 0x5643caf1e850 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8f29c47c41a09e3c server_cookie=56981d9e5f220abd in_seq=0 out_seq=0 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6cfffff640 10 client.15579 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6cfffff640 1 -- 172.21.15.31:0/725654023 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5643cae08800 con 0x5643caa8e3a0 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6cfffff640 10 client.15579 handle_command_reply: tid=0 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d2d97d740 1 client.15579 shutdown 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d2d97d740 1 -- 172.21.15.31:0/725654023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad14d0 msgr2=0x5643caeaa9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad14d0 0x5643caeaa9c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6d180dc9d0 tx=0x7f6d180df0c0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d2d97d740 1 -- 172.21.15.31:0/725654023 shutdown_connections 2024-04-25T16:01:46.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5643caa8e3a0 0x5643caf1e850 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5643caafa390 0x5643caeabe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5643caad96b0 0x5643caeaaf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d2d97d740 1 --2- 172.21.15.31:0/725654023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5643caad14d0 0x5643caeaa9c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6cfffff640 0 client.15579 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:01:46.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.908+0000 7f6d2d97d740 1 -- 172.21.15.31:0/725654023 >> 172.21.15.31:0/725654023 conn(0x5643caaee4e0 msgr2=0x5643cae9c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d2d97d740 1 -- 172.21.15.31:0/725654023 shutdown_connections 2024-04-25T16:01:46.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d2d97d740 1 -- 172.21.15.31:0/725654023 wait complete. 2024-04-25T16:01:46.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d2d97d740 20 client.15579 trim_cache size 0 max 16384 2024-04-25T16:01:46.917 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:01:46.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 -- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d080afaf0 msgr2=0x7f6d080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d080afaf0 0x7f6d080b2000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x5643cae9d860 tx=0x7f6d10004730 comp rx=0 tx=0).stop 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 -- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d201045d0 msgr2=0x7f6d2019a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d201045d0 0x7f6d2019a5d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6d1800ea50 tx=0x7f6d1800ef10 comp rx=0 tx=0).stop 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 -- 172.21.15.31:0/2392807544 shutdown_connections 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6d080afaf0 0x7f6d080b2000 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6d20109890 0x7f6d201a1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6d20106f30 0x7f6d2019ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 --2- 172.21.15.31:0/2392807544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6d201045d0 0x7f6d2019a5d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 -- 172.21.15.31:0/2392807544 >> 172.21.15.31:0/2392807544 conn(0x7f6d200fdbd0 msgr2=0x7f6d20104b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 -- 172.21.15.31:0/2392807544 shutdown_connections 2024-04-25T16:01:46.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:46.909+0000 7f6d26ad1640 1 -- 172.21.15.31:0/2392807544 wait complete. 2024-04-25T16:01:46.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:01:47.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.035+0000 7f41a9b9f640 1 -- 172.21.15.31:0/133521478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4109850 msgr2=0x7f41a410bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.035+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/133521478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4109850 0x7f41a410bc70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f419c009ca0 tx=0x7f419c02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.035+0000 7f41a9b9f640 1 -- 172.21.15.31:0/133521478 shutdown_connections 2024-04-25T16:01:47.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.035+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/133521478 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4109850 0x7f41a410bc70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.035+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/133521478 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a4104b50 0x7f41a4104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.035+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/133521478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41a4103910 0x7f41a4104610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.035+0000 7f41a9b9f640 1 -- 172.21.15.31:0/133521478 >> 172.21.15.31:0/133521478 conn(0x7f41a40feb20 msgr2=0x7f41a4100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.035+0000 7f41a9b9f640 1 -- 172.21.15.31:0/133521478 shutdown_connections 2024-04-25T16:01:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.035+0000 7f41a9b9f640 1 -- 172.21.15.31:0/133521478 wait complete. 2024-04-25T16:01:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a9b9f640 1 Processor -- start 2024-04-25T16:01:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a9b9f640 1 -- start start 2024-04-25T16:01:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a9b9f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4103910 0x7f41a419a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a9b9f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41a4104b50 0x7f41a419a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a9b9f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a4109850 0x7f41a41a14e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a9b9f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41a410f440 con 0x7f41a4103910 2024-04-25T16:01:47.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a9b9f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f41a410f2c0 con 0x7f41a4109850 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a9b9f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41a410f5c0 con 0x7f41a4104b50 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a8b9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4103910 0x7f41a419a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a8b9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4103910 0x7f41a419a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35536/0 (socket says 172.21.15.31:35536) 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a8b9d640 1 -- 172.21.15.31:0/1344583399 learned_addr learned my addr 172.21.15.31:0/1344583399 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.036+0000 7f41a939e640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a4109850 0x7f41a41a14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a3fff640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41a4104b50 0x7f41a419a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a8b9d640 1 -- 172.21.15.31:0/1344583399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41a4104b50 msgr2=0x7f41a419a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41a4104b50 0x7f41a419a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a8b9d640 1 -- 172.21.15.31:0/1344583399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a4109850 msgr2=0x7f41a41a14e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a4109850 0x7f41a41a14e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a8b9d640 1 -- 172.21.15.31:0/1344583399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a41a1b50 con 0x7f41a4103910 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a939e640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a4109850 0x7f41a41a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4103910 0x7f41a419a010 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4198005d00 tx=0x7f419800dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a1ffb640 1 -- 172.21.15.31:0/1344583399 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4198017070 con 0x7f41a4103910 2024-04-25T16:01:47.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a9b9f640 1 -- 172.21.15.31:0/1344583399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a41a1e40 con 0x7f41a4103910 2024-04-25T16:01:47.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a1ffb640 1 -- 172.21.15.31:0/1344583399 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4198009c30 con 0x7f41a4103910 2024-04-25T16:01:47.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a1ffb640 1 -- 172.21.15.31:0/1344583399 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419800f620 con 0x7f41a4103910 2024-04-25T16:01:47.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.037+0000 7f41a9b9f640 1 -- 172.21.15.31:0/1344583399 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a41a22a0 con 0x7f41a4103910 2024-04-25T16:01:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41b1d8b740 1 Processor -- start 2024-04-25T16:01:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41b1d8b740 1 -- start start 2024-04-25T16:01:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41a1ffb640 1 -- 172.21.15.31:0/1344583399 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f419801a070 con 0x7f41a4103910 2024-04-25T16:01:47.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 0x56102ee4e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea834d0 0x56102ee4ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea8b6b0 0x56102ee4fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56102edb2d80 con 0x56102ea834d0 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56102ed6ca20 con 0x56102ea8b6b0 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56102edaed40 con 0x56102eaac390 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41a8b9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea834d0 0x56102ee4ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41a8b9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea834d0 0x56102ee4ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35544/0 (socket says 172.21.15.31:35544) 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41a8b9d640 1 -- 172.21.15.31:0/731789457 learned_addr learned my addr 172.21.15.31:0/731789457 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41a939e640 1 --2- 172.21.15.31:0/731789457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 0x56102ee4e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.040+0000 7f41a3fff640 1 --2- 172.21.15.31:0/731789457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea8b6b0 0x56102ee4fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f41a8b9d640 1 -- 172.21.15.31:0/731789457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 msgr2=0x56102ee4e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/731789457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 0x56102ee4e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f41a8b9d640 1 -- 172.21.15.31:0/731789457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea8b6b0 msgr2=0x56102ee4fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/731789457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea8b6b0 0x56102ee4fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f41a8b9d640 1 -- 172.21.15.31:0/731789457 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56102ed77c30 con 0x56102ea834d0 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f41a1ffb640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41940afae0 0x7f41940b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/731789457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea834d0 0x56102ee4ef60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f41980e4c10 tx=0x7f41980e6690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7871cfa710c332 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f41a939e640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41940afae0 0x7f41940b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f418effd640 1 -- 172.21.15.31:0/731789457 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41980a2ad0 con 0x56102ea834d0 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f41a1ffb640 1 -- 172.21.15.31:0/1344583399 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f41980de720 con 0x7f41a4103910 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f418effd640 1 -- 172.21.15.31:0/731789457 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41980a6720 con 0x56102ea834d0 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.041+0000 7f418effd640 1 -- 172.21.15.31:0/731789457 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41980a6380 con 0x56102ea834d0 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 -- 172.21.15.31:0/731789457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea834d0 msgr2=0x56102ee4ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41a939e640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41940afae0 0x7f41940b1ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f419c0094e0 tx=0x7f419c031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/731789457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea834d0 0x56102ee4ef60 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f41980e4c10 tx=0x7f41980e6690 comp rx=0 tx=0).stop 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 -- 172.21.15.31:0/731789457 shutdown_connections 2024-04-25T16:01:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/731789457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea8b6b0 0x56102ee4fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/731789457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea834d0 0x56102ee4ef60 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/731789457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 0x56102ee4e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 -- 172.21.15.31:0/731789457 >> 172.21.15.31:0/731789457 conn(0x56102eaa04e0 msgr2=0x56102ee4a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 -- 172.21.15.31:0/731789457 shutdown_connections 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 -- 172.21.15.31:0/731789457 wait complete. 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.042+0000 7f41b1d8b740 1 -- start start 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41b1d8b740 1 Processor -- start 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee58790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee58cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 0x56102ee5fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56102ee50700 con 0x56102ea8b6b0 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56102ed39dc0 con 0x56102ea834d0 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56102ee50880 con 0x56102eaac390 2024-04-25T16:01:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41b1d8b740 10 client.0 _ll_register_callbacks cb 0x56102ed55600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee58cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee58cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35552/0 (socket says 172.21.15.31:35552) 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a3fff640 1 -- 172.21.15.31:0/3511061707 learned_addr learned my addr 172.21.15.31:0/3511061707 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a939e640 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 0x56102ee5fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee58790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a3fff640 1 -- 172.21.15.31:0/3511061707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 msgr2=0x56102ee5fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a3fff640 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 0x56102ee5fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a3fff640 1 -- 172.21.15.31:0/3511061707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 msgr2=0x56102ee58790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a3fff640 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee58790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a3fff640 1 -- 172.21.15.31:0/3511061707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56102eac43f0 con 0x56102ea8b6b0 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.043+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee58790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41a3fff640 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee58cd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f419000be90 tx=0x7f4190012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d89d154b02d17b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f4183fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:47.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f4183fff640 1 -- 172.21.15.31:0/3511061707 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190004230 con 0x56102ea8b6b0 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f4183fff640 1 -- 172.21.15.31:0/3511061707 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f41900043d0 con 0x56102ea8b6b0 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41b1d8b740 1 -- 172.21.15.31:0/3511061707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41980a6720 con 0x56102ea8b6b0 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f4183fff640 1 -- 172.21.15.31:0/3511061707 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4190005020 con 0x56102ea8b6b0 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41b1d8b740 10 client.15591 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41b1d8b740 1 -- 172.21.15.31:0/3511061707 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56102eb8e7d0 con 0x56102ea8b6b0 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f4183fff640 1 -- 172.21.15.31:0/3511061707 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41980a6720 con 0x56102ea8b6b0 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f4183fff640 1 -- 172.21.15.31:0/3511061707 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f41900106a0 con 0x56102ea8b6b0 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41b1d8b740 10 client.15591 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41b1d8b740 10 client.15591 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41b1d8b740 20 client.15591 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56102ea6af60 0x56102ee66590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41b1d8b740 4 client.15591 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.044+0000 7f41b1d8b740 1 -- 172.21.15.31:0/3511061707 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56102edc0990 con 0x56102ea6af60 2024-04-25T16:01:47.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.045+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56102ea6af60 0x56102ee66590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.045+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56102ea6af60 0x56102ee66590 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dbe8a4369d4b0ef6 server_cookie=857bb066610783f2 in_seq=0 out_seq=0 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.045+0000 7f4183fff640 10 client.15591 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f4183fff640 1 -- 172.21.15.31:0/3511061707 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56102edc0990 con 0x56102ea6af60 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f4183fff640 10 client.15591 handle_command_reply: tid=0 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 client.15591 shutdown 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 -- 172.21.15.31:0/3511061707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 msgr2=0x56102ee58cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee58cd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f419000be90 tx=0x7f4190012820 comp rx=0 tx=0).stop 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 -- 172.21.15.31:0/3511061707 shutdown_connections 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102eaac390 0x56102ee5fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee58cd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56102ea6af60 0x56102ee66590 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/3511061707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee58790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f4183fff640 0 client.15591 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:01:47.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 -- 172.21.15.31:0/3511061707 >> 172.21.15.31:0/3511061707 conn(0x56102eaa04e0 msgr2=0x56102ee493f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 -- 172.21.15.31:0/3511061707 shutdown_connections 2024-04-25T16:01:47.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.046+0000 7f41b1d8b740 1 -- 172.21.15.31:0/3511061707 wait complete. 2024-04-25T16:01:47.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.047+0000 7f41b1d8b740 20 client.15591 trim_cache size 0 max 16384 2024-04-25T16:01:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41b1d8b740 1 Processor -- start 2024-04-25T16:01:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41b1d8b740 1 -- start start 2024-04-25T16:01:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee56ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea8b6b0 0x56102ee57430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102eaac390 0x56102ee6ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41a41ab040 con 0x56102eaac390 2024-04-25T16:01:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f41a41acb80 con 0x56102ea834d0 2024-04-25T16:01:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41a41ab710 con 0x56102ea8b6b0 2024-04-25T16:01:47.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41a939e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102eaac390 0x56102ee6ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41a939e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102eaac390 0x56102ee6ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35562/0 (socket says 172.21.15.31:35562) 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.060+0000 7f41a939e640 1 -- 172.21.15.31:0/209513627 learned_addr learned my addr 172.21.15.31:0/209513627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee56ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f41a3fff640 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea8b6b0 0x56102ee57430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f41a939e640 1 -- 172.21.15.31:0/209513627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea8b6b0 msgr2=0x56102ee57430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f41a939e640 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea8b6b0 0x56102ee57430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f41a939e640 1 -- 172.21.15.31:0/209513627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 msgr2=0x56102ee56ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f41a939e640 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee56ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f41a939e640 1 -- 172.21.15.31:0/209513627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56102ee3bba0 con 0x56102eaac390 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee56ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f41a939e640 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102eaac390 0x56102ee6ad80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f419c035000 tx=0x7f419c044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4fbf13a7d79939 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f418e7fc640 1 -- 172.21.15.31:0/209513627 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c004040 con 0x56102eaac390 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f418e7fc640 1 -- 172.21.15.31:0/209513627 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f419c036040 con 0x56102eaac390 2024-04-25T16:01:47.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.061+0000 7f418e7fc640 1 -- 172.21.15.31:0/209513627 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c02feb0 con 0x56102eaac390 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 -- 172.21.15.31:0/209513627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102eaac390 msgr2=0x56102ee6ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102eaac390 0x56102ee6ad80 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f419c035000 tx=0x7f419c044000 comp rx=0 tx=0).stop 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 -- 172.21.15.31:0/209513627 shutdown_connections 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102eaac390 0x56102ee6ad80 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea8b6b0 0x56102ee57430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/209513627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102ea834d0 0x56102ee56ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 -- 172.21.15.31:0/209513627 >> 172.21.15.31:0/209513627 conn(0x56102eaa04e0 msgr2=0x56102ee534e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 -- 172.21.15.31:0/209513627 shutdown_connections 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 -- 172.21.15.31:0/209513627 wait complete. 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 -- start start 2024-04-25T16:01:47.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.062+0000 7f41b1d8b740 1 Processor -- start 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea834d0 0x56102ee5c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee5cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41b1d8b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102eaac390 0x56102ee5db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f41a41ac4b0 con 0x56102ea8b6b0 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f41a41abde0 con 0x56102eaac390 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41b1d8b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f41a41a9020 con 0x56102ea834d0 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41b1d8b740 10 client.0 _ll_register_callbacks cb 0x7f419000b820 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee5cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee5cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35570/0 (socket says 172.21.15.31:35570) 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a3fff640 1 -- 172.21.15.31:0/1837297692 learned_addr learned my addr 172.21.15.31:0/1837297692 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea834d0 0x56102ee5c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a939e640 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102eaac390 0x56102ee5db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a3fff640 1 -- 172.21.15.31:0/1837297692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea834d0 msgr2=0x56102ee5c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a3fff640 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea834d0 0x56102ee5c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a3fff640 1 -- 172.21.15.31:0/1837297692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102eaac390 msgr2=0x56102ee5db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a3fff640 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102eaac390 0x56102ee5db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a3fff640 1 -- 172.21.15.31:0/1837297692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56102ee5e0e0 con 0x56102ea8b6b0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41a3fff640 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee5cbf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f41900059f0 tx=0x7f419000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f492057d6b0b2c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f4183fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f4183fff640 1 -- 172.21.15.31:0/1837297692 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419002a040 con 0x56102ea8b6b0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f4183fff640 1 -- 172.21.15.31:0/1837297692 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f41900053c0 con 0x56102ea8b6b0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.063+0000 7f41b1d8b740 1 -- 172.21.15.31:0/1837297692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f419c036040 con 0x56102ea8b6b0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f4183fff640 1 -- 172.21.15.31:0/1837297692 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4190012d20 con 0x56102ea8b6b0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f41b1d8b740 10 client.15597 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f41b1d8b740 1 -- 172.21.15.31:0/1837297692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56102ee5f300 con 0x56102ea8b6b0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f4183fff640 1 -- 172.21.15.31:0/1837297692 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f419c036040 con 0x56102ea8b6b0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f4183fff640 1 -- 172.21.15.31:0/1837297692 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4190004ac0 con 0x56102ea8b6b0 2024-04-25T16:01:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f41b1d8b740 10 client.15597 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f41b1d8b740 10 client.15597 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f41b1d8b740 20 client.15597 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56102ea403a0 0x56102eed0ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f41b1d8b740 4 client.15597 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f41b1d8b740 1 -- 172.21.15.31:0/1837297692 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56102edba800 con 0x56102ea403a0 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.064+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56102ea403a0 0x56102eed0ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f4183fff640 10 client.15597 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41a8b9d640 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56102ea403a0 0x56102eed0ee0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=839991961fe629e4 server_cookie=50a63ad160278e62 in_seq=0 out_seq=0 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f4183fff640 1 -- 172.21.15.31:0/1837297692 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56102edba800 con 0x56102ea403a0 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f4183fff640 10 client.15597 handle_command_reply: tid=0 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 client.15597 shutdown 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 -- 172.21.15.31:0/1837297692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 msgr2=0x56102ee5cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee5cbf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f41900059f0 tx=0x7f419000ba10 comp rx=0 tx=0).stop 2024-04-25T16:01:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 -- 172.21.15.31:0/1837297692 shutdown_connections 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56102ea403a0 0x56102eed0ee0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56102eaac390 0x56102ee5db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56102ea8b6b0 0x56102ee5cbf0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 --2- 172.21.15.31:0/1837297692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56102ea834d0 0x56102ee5c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f4183fff640 0 client.15597 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 -- 172.21.15.31:0/1837297692 >> 172.21.15.31:0/1837297692 conn(0x56102eaa04e0 msgr2=0x56102ee56ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 -- 172.21.15.31:0/1837297692 shutdown_connections 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 1 -- 172.21.15.31:0/1837297692 wait complete. 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.065+0000 7f41b1d8b740 20 client.15597 trim_cache size 0 max 16384 2024-04-25T16:01:47.072 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:01:47.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 -- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41940afae0 msgr2=0x7f41940b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41940afae0 0x7f41940b1ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f419c0094e0 tx=0x7f419c031af0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 -- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4103910 msgr2=0x7f41a419a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4103910 0x7f41a419a010 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4198005d00 tx=0x7f419800dde0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 -- 172.21.15.31:0/1344583399 shutdown_connections 2024-04-25T16:01:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f41940afae0 0x7f41940b1ff0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f41a4109850 0x7f41a41a14e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41a4104b50 0x7f41a419a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 --2- 172.21.15.31:0/1344583399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f41a4103910 0x7f41a419a010 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 -- 172.21.15.31:0/1344583399 >> 172.21.15.31:0/1344583399 conn(0x7f41a40feb20 msgr2=0x7f41a4105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 -- 172.21.15.31:0/1344583399 shutdown_connections 2024-04-25T16:01:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.066+0000 7f41a9b9f640 1 -- 172.21.15.31:0/1344583399 wait complete. 2024-04-25T16:01:47.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 -- 172.21.15.31:0/1423874038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90108930 msgr2=0x7f4a9010ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 --2- 172.21.15.31:0/1423874038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90108930 0x7f4a9010ad50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009ca0 tx=0x7f4a8002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 -- 172.21.15.31:0/1423874038 shutdown_connections 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 --2- 172.21.15.31:0/1423874038 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90108930 0x7f4a9010ad50 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 --2- 172.21.15.31:0/1423874038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a90103c30 0x7f4a90104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 --2- 172.21.15.31:0/1423874038 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a901029f0 0x7f4a901036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 -- 172.21.15.31:0/1423874038 >> 172.21.15.31:0/1423874038 conn(0x7f4a900fdbc0 msgr2=0x7f4a90100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 -- 172.21.15.31:0/1423874038 shutdown_connections 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 -- 172.21.15.31:0/1423874038 wait complete. 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 Processor -- start 2024-04-25T16:01:47.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.191+0000 7f4a97b00640 1 -- start start 2024-04-25T16:01:47.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a97b00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a901029f0 0x7f4a9019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a97b00640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90103c30 0x7f4a9019a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a97b00640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a90108930 0x7f4a901a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a97b00640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9010e520 con 0x7f4a90103c30 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a97b00640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9010e3a0 con 0x7f4a90108930 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a97b00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9010e6a0 con 0x7f4a901029f0 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9509b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90103c30 0x7f4a9019a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9509b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90103c30 0x7f4a9019a720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35590/0 (socket says 172.21.15.31:35590) 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9509b640 1 -- 172.21.15.31:0/961396409 learned_addr learned my addr 172.21.15.31:0/961396409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9589c640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a901029f0 0x7f4a9019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9609d640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a90108930 0x7f4a901a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9509b640 1 -- 172.21.15.31:0/961396409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a901029f0 msgr2=0x7f4a9019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9509b640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a901029f0 0x7f4a9019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9509b640 1 -- 172.21.15.31:0/961396409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a90108930 msgr2=0x7f4a901a16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9509b640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a90108930 0x7f4a901a16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9509b640 1 -- 172.21.15.31:0/961396409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a901a1d20 con 0x7f4a90103c30 2024-04-25T16:01:47.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a9509b640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90103c30 0x7f4a9019a720 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4a8400e9f0 tx=0x7f4a8400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a8effd640 1 -- 172.21.15.31:0/961396409 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8400cd80 con 0x7f4a90103c30 2024-04-25T16:01:47.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a97b00640 1 -- 172.21.15.31:0/961396409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a901a1fb0 con 0x7f4a90103c30 2024-04-25T16:01:47.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a8effd640 1 -- 172.21.15.31:0/961396409 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a840054d0 con 0x7f4a90103c30 2024-04-25T16:01:47.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.192+0000 7f4a8effd640 1 -- 172.21.15.31:0/961396409 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a84010690 con 0x7f4a90103c30 2024-04-25T16:01:47.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.193+0000 7f4a97b00640 1 -- 172.21.15.31:0/961396409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a901a24a0 con 0x7f4a90103c30 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.195+0000 7f4a9e9b3740 1 Processor -- start 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.195+0000 7f4a9e9b3740 1 -- start start 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.195+0000 7f4a8effd640 1 -- 172.21.15.31:0/961396409 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4a84004e90 con 0x7f4a90103c30 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a2a390 0x55d3e1dcc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 0x55d3e1dccd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a096b0 0x55d3e1dcda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d3e1d30d80 con 0x55d3e1a096b0 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d3e1ceaa20 con 0x55d3e1a2a390 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d3e1d2cd40 con 0x55d3e1a014d0 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9509b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a096b0 0x55d3e1dcda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9609d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a2a390 0x55d3e1dcc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9509b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a096b0 0x55d3e1dcda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35598/0 (socket says 172.21.15.31:35598) 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9509b640 1 -- 172.21.15.31:0/1845357146 learned_addr learned my addr 172.21.15.31:0/1845357146 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9589c640 1 --2- 172.21.15.31:0/1845357146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 0x55d3e1dccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9509b640 1 -- 172.21.15.31:0/1845357146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 msgr2=0x55d3e1dccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9509b640 1 --2- 172.21.15.31:0/1845357146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 0x55d3e1dccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9509b640 1 -- 172.21.15.31:0/1845357146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a2a390 msgr2=0x55d3e1dcc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9509b640 1 --2- 172.21.15.31:0/1845357146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a2a390 0x55d3e1dcc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9509b640 1 -- 172.21.15.31:0/1845357146 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3e1cf5c30 con 0x55d3e1a096b0 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a9609d640 1 --2- 172.21.15.31:0/1845357146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a2a390 0x55d3e1dcc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.196+0000 7f4a8effd640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a780afaf0 0x7f4a780b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.197+0000 7f4a9609d640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a780afaf0 0x7f4a780b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.197+0000 7f4a9509b640 1 --2- 172.21.15.31:0/1845357146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a096b0 0x55d3e1dcda20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4a840e2fd0 tx=0x7f4a840e7220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1df41d49938db5c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.197+0000 7f4a8effd640 1 -- 172.21.15.31:0/961396409 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4a840df0d0 con 0x7f4a90103c30 2024-04-25T16:01:47.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.197+0000 7f4a77fff640 1 -- 172.21.15.31:0/1845357146 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a840a3480 con 0x55d3e1a096b0 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.197+0000 7f4a77fff640 1 -- 172.21.15.31:0/1845357146 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a840a70d0 con 0x55d3e1a096b0 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.197+0000 7f4a77fff640 1 -- 172.21.15.31:0/1845357146 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a84005180 con 0x55d3e1a096b0 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/1845357146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a096b0 msgr2=0x55d3e1dcda20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9609d640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a780afaf0 0x7f4a780b2000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55d3e1dcd860 tx=0x7f4a80031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/1845357146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a096b0 0x55d3e1dcda20 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4a840e2fd0 tx=0x7f4a840e7220 comp rx=0 tx=0).stop 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/1845357146 shutdown_connections 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/1845357146 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a096b0 0x55d3e1dcda20 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/1845357146 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 0x55d3e1dccd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/1845357146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a2a390 0x55d3e1dcc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/1845357146 >> 172.21.15.31:0/1845357146 conn(0x55d3e1a1e4e0 msgr2=0x55d3e1dc8020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/1845357146 shutdown_connections 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/1845357146 wait complete. 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 -- start start 2024-04-25T16:01:47.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.198+0000 7f4a9e9b3740 1 Processor -- start 2024-04-25T16:01:47.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a014d0 0x55d3e1dd6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a096b0 0x55d3e1dd6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d3e1dce530 con 0x55d3e1a2a390 2024-04-25T16:01:47.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d3e1cb7dc0 con 0x55d3e1a014d0 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d3e1dce6b0 con 0x55d3e1a096b0 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9e9b3740 10 client.0 _ll_register_callbacks cb 0x55d3e1cd3600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9609d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9609d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35600/0 (socket says 172.21.15.31:35600) 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9609d640 1 -- 172.21.15.31:0/2163876562 learned_addr learned my addr 172.21.15.31:0/2163876562 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9509b640 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a096b0 0x55d3e1dd6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9589c640 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a014d0 0x55d3e1dd6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9609d640 1 -- 172.21.15.31:0/2163876562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a096b0 msgr2=0x55d3e1dd6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9609d640 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a096b0 0x55d3e1dd6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9609d640 1 -- 172.21.15.31:0/2163876562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a014d0 msgr2=0x55d3e1dd6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9609d640 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a014d0 0x55d3e1dd6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.199+0000 7f4a9609d640 1 -- 172.21.15.31:0/2163876562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3e1a423f0 con 0x55d3e1a2a390 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a9509b640 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a096b0 0x55d3e1dd6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:01:47.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a9609d640 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddda00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4a80031980 tx=0x7f4a8002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07d879a80b0a882 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a757fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a757fa640 1 -- 172.21.15.31:0/2163876562 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80009360 con 0x55d3e1a2a390 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a757fa640 1 -- 172.21.15.31:0/2163876562 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4a80004250 con 0x55d3e1a2a390 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2163876562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a840a70d0 con 0x55d3e1a2a390 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a757fa640 1 -- 172.21.15.31:0/2163876562 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a80004450 con 0x55d3e1a2a390 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a9e9b3740 10 client.15609 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2163876562 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d3e1b0c7d0 con 0x55d3e1a2a390 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a757fa640 1 -- 172.21.15.31:0/2163876562 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a840a70d0 con 0x55d3e1a2a390 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a757fa640 1 -- 172.21.15.31:0/2163876562 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4a80051070 con 0x55d3e1a2a390 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a9e9b3740 10 client.15609 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.200+0000 7f4a9e9b3740 10 client.15609 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.201+0000 7f4a9e9b3740 20 client.15609 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.201+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3e19e8f60 0x55d3e1de4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.201+0000 7f4a9e9b3740 4 client.15609 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.201+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2163876562 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3e1d3e990 con 0x55d3e19e8f60 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.201+0000 7f4a9589c640 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3e19e8f60 0x55d3e1de4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.201+0000 7f4a757fa640 10 client.15609 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.201+0000 7f4a9589c640 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3e19e8f60 0x55d3e1de4220 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f8a3fbdfc955afc server_cookie=1c3ab3e6b3079462 in_seq=0 out_seq=0 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a757fa640 1 -- 172.21.15.31:0/2163876562 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d3e1d3e990 con 0x55d3e19e8f60 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a757fa640 10 client.15609 handle_command_reply: tid=0 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a9e9b3740 1 client.15609 shutdown 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2163876562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 msgr2=0x55d3e1ddda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddda00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4a80031980 tx=0x7f4a8002a8c0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2163876562 shutdown_connections 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3e19e8f60 0x55d3e1de4220 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddda00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a096b0 0x55d3e1dd6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2163876562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a014d0 0x55d3e1dd6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a757fa640 0 client.15609 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:01:47.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.202+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2163876562 >> 172.21.15.31:0/2163876562 conn(0x55d3e1a1e4e0 msgr2=0x55d3e1dcaf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.203+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2163876562 shutdown_connections 2024-04-25T16:01:47.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.203+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2163876562 wait complete. 2024-04-25T16:01:47.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.203+0000 7f4a9e9b3740 20 client.15609 trim_cache size 0 max 16384 2024-04-25T16:01:47.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.216+0000 7f4a9e9b3740 1 Processor -- start 2024-04-25T16:01:47.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.216+0000 7f4a9e9b3740 1 -- start start 2024-04-25T16:01:47.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.216+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a014d0 0x55d3e1dc8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.216+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 0x55d3e1ddd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.216+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a2a390 0x55d3e1dd4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a901ab230 con 0x55d3e1a014d0 2024-04-25T16:01:47.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4a901acd70 con 0x55d3e1a096b0 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a901ab900 con 0x55d3e1a2a390 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9589c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a014d0 0x55d3e1dc8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9589c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a014d0 0x55d3e1dc8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35614/0 (socket says 172.21.15.31:35614) 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9589c640 1 -- 172.21.15.31:0/2811228223 learned_addr learned my addr 172.21.15.31:0/2811228223 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9509b640 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 0x55d3e1ddd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9609d640 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a2a390 0x55d3e1dd4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9589c640 1 -- 172.21.15.31:0/2811228223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a2a390 msgr2=0x55d3e1dd4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9589c640 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a2a390 0x55d3e1dd4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9589c640 1 -- 172.21.15.31:0/2811228223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 msgr2=0x55d3e1ddd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9589c640 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 0x55d3e1ddd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9589c640 1 -- 172.21.15.31:0/2811228223 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3e1dd1cc0 con 0x55d3e1a014d0 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9609d640 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a2a390 0x55d3e1dd4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a9589c640 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a014d0 0x55d3e1dc8160 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4a88000c00 tx=0x7f4a88005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2d3ce5e13d999a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a777fe640 1 -- 172.21.15.31:0/2811228223 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8800f810 con 0x55d3e1a014d0 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a777fe640 1 -- 172.21.15.31:0/2811228223 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a8800fe60 con 0x55d3e1a014d0 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.217+0000 7f4a777fe640 1 -- 172.21.15.31:0/2811228223 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8801edb0 con 0x55d3e1a014d0 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2811228223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a014d0 msgr2=0x55d3e1dc8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a014d0 0x55d3e1dc8160 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f4a88000c00 tx=0x7f4a88005f90 comp rx=0 tx=0).stop 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2811228223 shutdown_connections 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a2a390 0x55d3e1dd4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 0x55d3e1ddd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/2811228223 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a014d0 0x55d3e1dc8160 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2811228223 >> 172.21.15.31:0/2811228223 conn(0x55d3e1a1e4e0 msgr2=0x55d3e1dc7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2811228223 shutdown_connections 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/2811228223 wait complete. 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 -- start start 2024-04-25T16:01:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.218+0000 7f4a9e9b3740 1 Processor -- start 2024-04-25T16:01:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 0x55d3e1ddab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 0x55d3e1ddb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9e9b3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a901ac6a0 con 0x55d3e1a2a390 2024-04-25T16:01:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4a901abfd0 con 0x55d3e1a096b0 2024-04-25T16:01:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9e9b3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a901a9210 con 0x55d3e1a014d0 2024-04-25T16:01:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9e9b3740 10 client.0 _ll_register_callbacks cb 0x7f4a84005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9609d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9609d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35620/0 (socket says 172.21.15.31:35620) 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9609d640 1 -- 172.21.15.31:0/520532826 learned_addr learned my addr 172.21.15.31:0/520532826 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9589c640 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 0x55d3e1ddab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9609d640 1 -- 172.21.15.31:0/520532826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 msgr2=0x55d3e1ddab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9609d640 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 0x55d3e1ddab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9509b640 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 0x55d3e1ddb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9609d640 1 -- 172.21.15.31:0/520532826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 msgr2=0x55d3e1ddb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9609d640 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 0x55d3e1ddb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9609d640 1 -- 172.21.15.31:0/520532826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3e1ddde60 con 0x55d3e1a2a390 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9589c640 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 0x55d3e1ddab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a9609d640 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddc000 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009600 tx=0x7f4a80009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f367809c5510f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.219+0000 7f4a757fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a757fa640 1 -- 172.21.15.31:0/520532826 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a80003d40 con 0x55d3e1a2a390 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a757fa640 1 -- 172.21.15.31:0/520532826 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4a80004250 con 0x55d3e1a2a390 2024-04-25T16:01:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/520532826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a8800fe60 con 0x55d3e1a2a390 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a757fa640 1 -- 172.21.15.31:0/520532826 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4a80004b40 con 0x55d3e1a2a390 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a9e9b3740 10 client.15615 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/520532826 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d3e1ddd650 con 0x55d3e1a2a390 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a757fa640 1 -- 172.21.15.31:0/520532826 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a8800fe60 con 0x55d3e1a2a390 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a757fa640 1 -- 172.21.15.31:0/520532826 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f4a80051040 con 0x55d3e1a2a390 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a9e9b3740 10 client.15615 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a9e9b3740 10 client.15615 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a9e9b3740 20 client.15615 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3e19be3a0 0x55d3e1e4e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a9e9b3740 4 client.15615 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.220+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/520532826 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3e1d38800 con 0x55d3e19be3a0 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.221+0000 7f4a9589c640 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3e19be3a0 0x55d3e1e4e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.221+0000 7f4a9589c640 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3e19be3a0 0x55d3e1e4e840 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5f7af4d104d36199 server_cookie=48c37e9112481a26 in_seq=0 out_seq=0 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.221+0000 7f4a757fa640 10 client.15615 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.221+0000 7f4a757fa640 1 -- 172.21.15.31:0/520532826 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3e1d38800 con 0x55d3e19be3a0 2024-04-25T16:01:47.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.221+0000 7f4a757fa640 10 client.15615 handle_command_reply: tid=0 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.221+0000 7f4a9e9b3740 1 client.15615 shutdown 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.221+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/520532826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 msgr2=0x55d3e1ddc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.221+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddc000 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009600 tx=0x7f4a80009360 comp rx=0 tx=0).stop 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/520532826 shutdown_connections 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3e19be3a0 0x55d3e1e4e840 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3e1a2a390 0x55d3e1ddc000 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3e1a096b0 0x55d3e1ddb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a757fa640 0 client.15615 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a9e9b3740 1 --2- 172.21.15.31:0/520532826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3e1a014d0 0x55d3e1ddab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/520532826 >> 172.21.15.31:0/520532826 conn(0x55d3e1a1e4e0 msgr2=0x55d3e1dcae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/520532826 shutdown_connections 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a9e9b3740 1 -- 172.21.15.31:0/520532826 wait complete. 2024-04-25T16:01:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a9e9b3740 20 client.15615 trim_cache size 0 max 16384 2024-04-25T16:01:47.229 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:01:47.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 -- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a780afaf0 msgr2=0x7f4a780b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a780afaf0 0x7f4a780b2000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55d3e1dcd860 tx=0x7f4a80031af0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 -- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90103c30 msgr2=0x7f4a9019a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90103c30 0x7f4a9019a720 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4a8400e9f0 tx=0x7f4a8400edf0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 -- 172.21.15.31:0/961396409 shutdown_connections 2024-04-25T16:01:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a780afaf0 0x7f4a780b2000 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a90108930 0x7f4a901a16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a90103c30 0x7f4a9019a720 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 --2- 172.21.15.31:0/961396409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a901029f0 0x7f4a9019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 -- 172.21.15.31:0/961396409 >> 172.21.15.31:0/961396409 conn(0x7f4a900fdbc0 msgr2=0x7f4a90109d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 -- 172.21.15.31:0/961396409 shutdown_connections 2024-04-25T16:01:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.222+0000 7f4a97b00640 1 -- 172.21.15.31:0/961396409 wait complete. 2024-04-25T16:01:47.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.347+0000 7f55adc96640 1 -- 172.21.15.31:0/130986420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 msgr2=0x7f55a81047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.347+0000 7f55adc96640 1 --2- 172.21.15.31:0/130986420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 0x7f55a81047f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f5598009900 tx=0x7f559802a410 comp rx=0 tx=0).stop 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.347+0000 7f55adc96640 1 -- 172.21.15.31:0/130986420 shutdown_connections 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.347+0000 7f55adc96640 1 --2- 172.21.15.31:0/130986420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55a8109a30 0x7f55a810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.347+0000 7f55adc96640 1 --2- 172.21.15.31:0/130986420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a8104d30 0x7f55a8105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.347+0000 7f55adc96640 1 --2- 172.21.15.31:0/130986420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 0x7f55a81047f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.347+0000 7f55adc96640 1 -- 172.21.15.31:0/130986420 >> 172.21.15.31:0/130986420 conn(0x7f55a80fece0 msgr2=0x7f55a8101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.347+0000 7f55adc96640 1 -- 172.21.15.31:0/130986420 shutdown_connections 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.347+0000 7f55adc96640 1 -- 172.21.15.31:0/130986420 wait complete. 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.348+0000 7f55adc96640 1 Processor -- start 2024-04-25T16:01:47.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.348+0000 7f55adc96640 1 -- start start 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55adc96640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 0x7f55a819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55adc96640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55a8104d30 0x7f55a819a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55adc96640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a8109a30 0x7f55a81a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55adc96640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810f5f0 con 0x7f55a8103af0 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55adc96640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810f470 con 0x7f55a8104d30 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55adc96640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810f770 con 0x7f55a8109a30 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 0x7f55a819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 0x7f55a819a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35632/0 (socket says 172.21.15.31:35632) 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a77fe640 1 -- 172.21.15.31:0/1265174691 learned_addr learned my addr 172.21.15.31:0/1265174691 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55a8104d30 0x7f55a819a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a7fff640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a8109a30 0x7f55a81a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a77fe640 1 -- 172.21.15.31:0/1265174691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a8109a30 msgr2=0x7f55a81a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a77fe640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a8109a30 0x7f55a81a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a77fe640 1 -- 172.21.15.31:0/1265174691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55a8104d30 msgr2=0x7f55a819a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a77fe640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55a8104d30 0x7f55a819a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a77fe640 1 -- 172.21.15.31:0/1265174691 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a81a1dc0 con 0x7f55a8103af0 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55a8104d30 0x7f55a819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a77fe640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 0x7f55a819a280 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5598006010 tx=0x7f5598004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.349+0000 7f55a4ff9640 1 -- 172.21.15.31:0/1265174691 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559803f030 con 0x7f55a8103af0 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.350+0000 7f55adc96640 1 -- 172.21.15.31:0/1265174691 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a81a2050 con 0x7f55a8103af0 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.350+0000 7f55a4ff9640 1 -- 172.21.15.31:0/1265174691 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5598003aa0 con 0x7f55a8103af0 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.350+0000 7f55a4ff9640 1 -- 172.21.15.31:0/1265174691 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5598035070 con 0x7f55a8103af0 2024-04-25T16:01:47.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.350+0000 7f55adc96640 1 -- 172.21.15.31:0/1265174691 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a81a2590 con 0x7f55a8103af0 2024-04-25T16:01:47.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.352+0000 7f55b4c29740 1 Processor -- start 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.352+0000 7f55b4c29740 1 -- start start 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.352+0000 7f55a4ff9640 1 -- 172.21.15.31:0/1265174691 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5598042030 con 0x7f55a8103af0 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.352+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f671002390 0x55f6713a4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.352+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 0x55f6713a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55b4c29740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f671308d80 con 0x55f671002390 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55b4c29740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f6712c2a20 con 0x55f670fd94d0 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55b4c29740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f671304d40 con 0x55f670fe16b0 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f671002390 0x55f6713a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f671002390 0x55f6713a4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35634/0 (socket says 172.21.15.31:35634) 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a7fff640 1 -- 172.21.15.31:0/893455567 learned_addr learned my addr 172.21.15.31:0/893455567 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a77fe640 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 0x55f6713a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a7fff640 1 -- 172.21.15.31:0/893455567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 msgr2=0x55f6713a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a7fff640 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 0x55f6713a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a7fff640 1 -- 172.21.15.31:0/893455567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 msgr2=0x55f6713a4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a7fff640 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713a4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a7fff640 1 -- 172.21.15.31:0/893455567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f6712cdc30 con 0x55f671002390 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a77fe640 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713a4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a4ff9640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55900afaf0 0x7f55900b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a7fff640 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f671002390 0x55f6713a4810 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f55a0009fa0 tx=0x7f55a000f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5afbcd0128780f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f55a77fe640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55900afaf0 0x7f55900b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f558dffb640 1 -- 172.21.15.31:0/893455567 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0015070 con 0x55f671002390 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f558dffb640 1 -- 172.21.15.31:0/893455567 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55a001ab10 con 0x55f671002390 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.353+0000 7f558dffb640 1 -- 172.21.15.31:0/893455567 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0011d70 con 0x55f671002390 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55a4ff9640 1 -- 172.21.15.31:0/1265174691 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f55980ff2f0 con 0x7f55a8103af0 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55b4c29740 1 -- 172.21.15.31:0/893455567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f671002390 msgr2=0x55f6713a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55a77fe640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55900afaf0 0x7f55900b2000 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55f6713a4530 tx=0x7f5598031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55b4c29740 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f671002390 0x55f6713a4810 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f55a0009fa0 tx=0x7f55a000f7f0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55b4c29740 1 -- 172.21.15.31:0/893455567 shutdown_connections 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55b4c29740 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 0x55f6713a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55b4c29740 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713a4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55b4c29740 1 --2- 172.21.15.31:0/893455567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f671002390 0x55f6713a4810 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55b4c29740 1 -- 172.21.15.31:0/893455567 >> 172.21.15.31:0/893455567 conn(0x55f670ff64e0 msgr2=0x55f6713a0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55b4c29740 1 -- 172.21.15.31:0/893455567 shutdown_connections 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.354+0000 7f55b4c29740 1 -- 172.21.15.31:0/893455567 wait complete. 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55b4c29740 1 -- start start 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55b4c29740 1 Processor -- start 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 0x55f6713aead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f671002390 0x55f6713b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55b4c29740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f6713a6530 con 0x55f670fd94d0 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55b4c29740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f67128fdc0 con 0x55f671002390 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55b4c29740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f6713a66b0 con 0x55f670fe16b0 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55a7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55b4c29740 10 client.0 _ll_register_callbacks cb 0x55f6712ab600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55a7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35648/0 (socket says 172.21.15.31:35648) 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.355+0000 7f55a7fff640 1 -- 172.21.15.31:0/126208591 learned_addr learned my addr 172.21.15.31:0/126208591 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 0x55f6713aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55a7fff640 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f671002390 0x55f6713b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55a7fff640 1 -- 172.21.15.31:0/126208591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 msgr2=0x55f6713aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55a7fff640 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 0x55f6713aead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55a7fff640 1 -- 172.21.15.31:0/126208591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f671002390 msgr2=0x55f6713b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55a7fff640 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f671002390 0x55f6713b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55a7fff640 1 -- 172.21.15.31:0/126208591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f67101a3f0 con 0x55f670fd94d0 2024-04-25T16:01:47.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55a7fff640 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae590 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f55a0006010 tx=0x7f55a0009bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df294905a250a322 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f5586ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f5586ffd640 1 -- 172.21.15.31:0/126208591 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a002d060 con 0x55f670fd94d0 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f5586ffd640 1 -- 172.21.15.31:0/126208591 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f55a0012d80 con 0x55f670fd94d0 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55b4c29740 1 -- 172.21.15.31:0/126208591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55a001ab10 con 0x55f670fd94d0 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f5586ffd640 1 -- 172.21.15.31:0/126208591 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f55a0022040 con 0x55f670fd94d0 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55b4c29740 10 client.15627 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f55b4c29740 1 -- 172.21.15.31:0/126208591 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f6710e47d0 con 0x55f670fd94d0 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.356+0000 7f5586ffd640 1 -- 172.21.15.31:0/126208591 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55a001ab10 con 0x55f670fd94d0 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f5586ffd640 1 -- 172.21.15.31:0/126208591 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f55a0005040 con 0x55f670fd94d0 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f55b4c29740 10 client.15627 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f55b4c29740 10 client.15627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f55b4c29740 20 client.15627 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f55b4c29740 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f670fc0f60 0x55f6713bbee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f55b4c29740 4 client.15627 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:47.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f55b4c29740 1 -- 172.21.15.31:0/126208591 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f671316990 con 0x55f670fc0f60 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f670fc0f60 0x55f6713bbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f670fc0f60 0x55f6713bbee0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a649caa2d408723b server_cookie=6930cda461585332 in_seq=0 out_seq=0 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.357+0000 7f5586ffd640 10 client.15627 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f5586ffd640 1 -- 172.21.15.31:0/126208591 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f671316990 con 0x55f670fc0f60 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f5586ffd640 10 client.15627 handle_command_reply: tid=0 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f55b4c29740 1 client.15627 shutdown 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f55b4c29740 1 -- 172.21.15.31:0/126208591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 msgr2=0x55f6713ae590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f55b4c29740 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae590 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f55a0006010 tx=0x7f55a0009bc0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f55b4c29740 1 -- 172.21.15.31:0/126208591 shutdown_connections 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f55b4c29740 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f670fc0f60 0x55f6713bbee0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f55b4c29740 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f671002390 0x55f6713b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f55b4c29740 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f670fe16b0 0x55f6713aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f55b4c29740 1 --2- 172.21.15.31:0/126208591 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae590 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f5586ffd640 0 client.15627 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:47.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.358+0000 7f55b4c29740 1 -- 172.21.15.31:0/126208591 >> 172.21.15.31:0/126208591 conn(0x55f670ff64e0 msgr2=0x55f67139f040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.359+0000 7f55b4c29740 1 -- 172.21.15.31:0/126208591 shutdown_connections 2024-04-25T16:01:47.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.359+0000 7f55b4c29740 1 -- 172.21.15.31:0/126208591 wait complete. 2024-04-25T16:01:47.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.359+0000 7f55b4c29740 20 client.15627 trim_cache size 0 max 16384 2024-04-25T16:01:47.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.372+0000 7f55b4c29740 1 Processor -- start 2024-04-25T16:01:47.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.372+0000 7f55b4c29740 1 -- start start 2024-04-25T16:01:47.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.372+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.372+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fe16b0 0x55f6713ae6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.372+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f671002390 0x55f6713c0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55b4c29740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f55a81ab2c0 con 0x55f670fd94d0 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55b4c29740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f55a81ace00 con 0x55f670fe16b0 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55b4c29740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f55a81ab990 con 0x55f671002390 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35656/0 (socket says 172.21.15.31:35656) 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a7fff640 1 -- 172.21.15.31:0/2508580759 learned_addr learned my addr 172.21.15.31:0/2508580759 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/2508580759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fe16b0 0x55f6713ae6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a7fff640 1 -- 172.21.15.31:0/2508580759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f671002390 msgr2=0x55f6713c0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a7fff640 1 --2- 172.21.15.31:0/2508580759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f671002390 0x55f6713c0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a7fff640 1 -- 172.21.15.31:0/2508580759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fe16b0 msgr2=0x55f6713ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a7fff640 1 --2- 172.21.15.31:0/2508580759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fe16b0 0x55f6713ae6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a7fff640 1 -- 172.21.15.31:0/2508580759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f67139b7f0 con 0x55f670fd94d0 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/2508580759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fe16b0 0x55f6713ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f55a7fff640 1 --2- 172.21.15.31:0/2508580759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae160 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f55a0005600 tx=0x7f55a0005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feddea3f3b080c6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f558d7fa640 1 -- 172.21.15.31:0/2508580759 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a0038070 con 0x55f670fd94d0 2024-04-25T16:01:47.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f558d7fa640 1 -- 172.21.15.31:0/2508580759 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55a0003e20 con 0x55f670fd94d0 2024-04-25T16:01:47.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.373+0000 7f558d7fa640 1 -- 172.21.15.31:0/2508580759 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55a002d040 con 0x55f670fd94d0 2024-04-25T16:01:47.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.374+0000 7f55b4c29740 1 -- 172.21.15.31:0/2508580759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 msgr2=0x55f6713ae160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.374+0000 7f55b4c29740 1 --2- 172.21.15.31:0/2508580759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae160 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f55a0005600 tx=0x7f55a0005d30 comp rx=0 tx=0).stop 2024-04-25T16:01:47.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.374+0000 7f55b4c29740 1 -- 172.21.15.31:0/2508580759 shutdown_connections 2024-04-25T16:01:47.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.374+0000 7f55b4c29740 1 --2- 172.21.15.31:0/2508580759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f671002390 0x55f6713c0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.374+0000 7f55b4c29740 1 --2- 172.21.15.31:0/2508580759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fe16b0 0x55f6713ae6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.374+0000 7f55b4c29740 1 --2- 172.21.15.31:0/2508580759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fd94d0 0x55f6713ae160 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.374+0000 7f55b4c29740 1 -- 172.21.15.31:0/2508580759 >> 172.21.15.31:0/2508580759 conn(0x55f670ff64e0 msgr2=0x55f6713a92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.374+0000 7f55b4c29740 1 -- 172.21.15.31:0/2508580759 shutdown_connections 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.374+0000 7f55b4c29740 1 -- 172.21.15.31:0/2508580759 wait complete. 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55b4c29740 1 -- start start 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55b4c29740 1 Processor -- start 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713b2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fe16b0 0x55f6713b2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55b4c29740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f671002390 0x55f6713b3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55b4c29740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f55a81ac730 con 0x55f670fe16b0 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55b4c29740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f55a81ac060 con 0x55f670fd94d0 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55b4c29740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f55a81a92a0 con 0x55f671002390 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55b4c29740 10 client.0 _ll_register_callbacks cb 0x7f55a0009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fe16b0 0x55f6713b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fe16b0 0x55f6713b2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35660/0 (socket says 172.21.15.31:35660) 2024-04-25T16:01:47.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a6ffd640 1 -- 172.21.15.31:0/692297550 learned_addr learned my addr 172.21.15.31:0/692297550 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a7fff640 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f671002390 0x55f6713b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a7fff640 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a6ffd640 1 -- 172.21.15.31:0/692297550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f671002390 msgr2=0x55f6713b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f671002390 0x55f6713b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a6ffd640 1 -- 172.21.15.31:0/692297550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 msgr2=0x55f6713b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a6ffd640 1 -- 172.21.15.31:0/692297550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f6713b3d30 con 0x55f670fe16b0 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.375+0000 7f55a7fff640 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f55a6ffd640 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fe16b0 0x55f6713b2840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f559c004700 tx=0x7f559c012900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a882714d75c6e88 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f5586ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f5586ffd640 1 -- 172.21.15.31:0/692297550 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c01e940 con 0x55f670fe16b0 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f5586ffd640 1 -- 172.21.15.31:0/692297550 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f559c005020 con 0x55f670fe16b0 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f55b4c29740 1 -- 172.21.15.31:0/692297550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55a0003e20 con 0x55f670fe16b0 2024-04-25T16:01:47.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f5586ffd640 1 -- 172.21.15.31:0/692297550 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f559c02e050 con 0x55f670fe16b0 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f55b4c29740 10 client.15633 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f55b4c29740 1 -- 172.21.15.31:0/692297550 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f6713b50b0 con 0x55f670fe16b0 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f5586ffd640 1 -- 172.21.15.31:0/692297550 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f55a0003e20 con 0x55f670fe16b0 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f5586ffd640 1 -- 172.21.15.31:0/692297550 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f559c011440 con 0x55f670fe16b0 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f55b4c29740 10 client.15633 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f55b4c29740 10 client.15633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.376+0000 7f55b4c29740 20 client.15633 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.377+0000 7f55b4c29740 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f670f963a0 0x55f6714265c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.377+0000 7f55b4c29740 4 client.15633 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.377+0000 7f55b4c29740 1 -- 172.21.15.31:0/692297550 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f671310800 con 0x55f670f963a0 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.377+0000 7f55a7fff640 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f670f963a0 0x55f6714265c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.377+0000 7f55a7fff640 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f670f963a0 0x55f6714265c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=982903c378ca88a2 server_cookie=c67601d1410dd579 in_seq=0 out_seq=0 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.377+0000 7f5586ffd640 10 client.15633 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:47.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.377+0000 7f5586ffd640 1 -- 172.21.15.31:0/692297550 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f671310800 con 0x55f670f963a0 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.377+0000 7f5586ffd640 10 client.15633 handle_command_reply: tid=0 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.377+0000 7f55b4c29740 1 client.15633 shutdown 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 -- 172.21.15.31:0/692297550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fe16b0 msgr2=0x55f6713b2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fe16b0 0x55f6713b2840 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f559c004700 tx=0x7f559c012900 comp rx=0 tx=0).stop 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 -- 172.21.15.31:0/692297550 shutdown_connections 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f670f963a0 0x55f6714265c0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f671002390 0x55f6713b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f670fe16b0 0x55f6713b2840 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f5586ffd640 0 client.15633 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 --2- 172.21.15.31:0/692297550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f670fd94d0 0x55f6713b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 -- 172.21.15.31:0/692297550 >> 172.21.15.31:0/692297550 conn(0x55f670ff64e0 msgr2=0x55f6713ae160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 -- 172.21.15.31:0/692297550 shutdown_connections 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 1 -- 172.21.15.31:0/692297550 wait complete. 2024-04-25T16:01:47.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55b4c29740 20 client.15633 trim_cache size 0 max 16384 2024-04-25T16:01:47.385 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:01:47.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 -- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55900afaf0 msgr2=0x7f55900b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55900afaf0 0x7f55900b2000 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x55f6713a4530 tx=0x7f5598031400 comp rx=0 tx=0).stop 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 -- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 msgr2=0x7f55a819a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 0x7f55a819a280 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f5598006010 tx=0x7f5598004270 comp rx=0 tx=0).stop 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 -- 172.21.15.31:0/1265174691 shutdown_connections 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55900afaf0 0x7f55900b2000 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55a8109a30 0x7f55a81a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f55a8104d30 0x7f55a819a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 --2- 172.21.15.31:0/1265174691 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f55a8103af0 0x7f55a819a280 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.378+0000 7f55adc96640 1 -- 172.21.15.31:0/1265174691 >> 172.21.15.31:0/1265174691 conn(0x7f55a80fece0 msgr2=0x7f55a8105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.379+0000 7f55adc96640 1 -- 172.21.15.31:0/1265174691 shutdown_connections 2024-04-25T16:01:47.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:47.379+0000 7f55adc96640 1 -- 172.21.15.31:0/1265174691 wait complete. 2024-04-25T16:01:47.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:47 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/827822910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:47.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:47 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/827822910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:47.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:47 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/827822910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:01:48.392 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:01:48.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:01:48.392 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:01:48.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.523+0000 7ff5d7c34640 1 -- 172.21.15.31:0/3424946734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d0103c20 msgr2=0x7ff5d0104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.523+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/3424946734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d0103c20 0x7ff5d0104030 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009900 tx=0x7ff5cc02a410 comp rx=0 tx=0).stop 2024-04-25T16:01:48.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.523+0000 7ff5d7c34640 1 -- 172.21.15.31:0/3424946734 shutdown_connections 2024-04-25T16:01:48.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.523+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/3424946734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d0108920 0x7ff5d010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.523+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/3424946734 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d0103c20 0x7ff5d0104030 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.523+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/3424946734 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5d01029e0 0x7ff5d01036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.523+0000 7ff5d7c34640 1 -- 172.21.15.31:0/3424946734 >> 172.21.15.31:0/3424946734 conn(0x7ff5d00fdbd0 msgr2=0x7ff5d0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.523+0000 7ff5d7c34640 1 -- 172.21.15.31:0/3424946734 shutdown_connections 2024-04-25T16:01:48.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.523+0000 7ff5d7c34640 1 -- 172.21.15.31:0/3424946734 wait complete. 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d7c34640 1 Processor -- start 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d7c34640 1 -- start start 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d7c34640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d01029e0 0x7ff5d019a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d7c34640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d0103c20 0x7ff5d019aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d7c34640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5d0108920 0x7ff5d01a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d7c34640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d010f650 con 0x7ff5d01029e0 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d7c34640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d010f4d0 con 0x7ff5d0108920 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d7c34640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d010f7d0 con 0x7ff5d0103c20 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d59d0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d01029e0 0x7ff5d019a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d59d0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d01029e0 0x7ff5d019a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35672/0 (socket says 172.21.15.31:35672) 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d59d0640 1 -- 172.21.15.31:0/2958444131 learned_addr learned my addr 172.21.15.31:0/2958444131 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.524+0000 7ff5d51cf640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d0103c20 0x7ff5d019aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5d0108920 0x7ff5d01a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d59d0640 1 -- 172.21.15.31:0/2958444131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d0103c20 msgr2=0x7ff5d019aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d0103c20 0x7ff5d019aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d59d0640 1 -- 172.21.15.31:0/2958444131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5d0108920 msgr2=0x7ff5d01a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5d0108920 0x7ff5d01a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d59d0640 1 -- 172.21.15.31:0/2958444131 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d01a2090 con 0x7ff5d01029e0 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5d0108920 0x7ff5d01a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d01029e0 0x7ff5d019a550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000e9f0 tx=0x7ff5c000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5c6ffd640 1 -- 172.21.15.31:0/2958444131 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c000ccc0 con 0x7ff5d01029e0 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5c6ffd640 1 -- 172.21.15.31:0/2958444131 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c00035f0 con 0x7ff5d01029e0 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d7c34640 1 -- 172.21.15.31:0/2958444131 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d01a2380 con 0x7ff5d01029e0 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5c6ffd640 1 -- 172.21.15.31:0/2958444131 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c000ce60 con 0x7ff5d01029e0 2024-04-25T16:01:48.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.525+0000 7ff5d7c34640 1 -- 172.21.15.31:0/2958444131 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d01a28a0 con 0x7ff5d01029e0 2024-04-25T16:01:48.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.528+0000 7ff5debc0740 1 Processor -- start 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.528+0000 7ff5c6ffd640 1 -- 172.21.15.31:0/2958444131 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff5c0010690 con 0x7ff5d01029e0 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.528+0000 7ff5debc0740 1 -- start start 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.528+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f338b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f33e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 0x561045f3ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5debc0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561045e94b00 con 0x561045b8e410 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5debc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561045e30920 con 0x561045b65550 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5debc0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561045e90ac0 con 0x561045b6d730 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f338b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f338b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35682/0 (socket says 172.21.15.31:35682) 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 -- 172.21.15.31:0/1277420329 learned_addr learned my addr 172.21.15.31:0/1277420329 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f33e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d51cf640 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 0x561045f3ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 -- 172.21.15.31:0/1277420329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 msgr2=0x561045f3ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5c6ffd640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5bc0afaf0 0x7ff5bc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 0x561045f3ad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 -- 172.21.15.31:0/1277420329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 msgr2=0x561045f33e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f33e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 -- 172.21.15.31:0/1277420329 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561045c6e7e0 con 0x561045b8e410 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f33e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5c6ffd640 1 -- 172.21.15.31:0/2958444131 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff5c00defc0 con 0x7ff5d01029e0 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5bc0afaf0 0x7ff5bc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.529+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f338b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800f410 tx=0x7ff5c800f8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eaaed1d54bf0423 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.530+0000 7ff5bbfff640 1 -- 172.21.15.31:0/1277420329 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8015070 con 0x561045b8e410 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.530+0000 7ff5bbfff640 1 -- 172.21.15.31:0/1277420329 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c8011be0 con 0x561045b8e410 2024-04-25T16:01:48.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.530+0000 7ff5bbfff640 1 -- 172.21.15.31:0/1277420329 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c801a790 con 0x561045b8e410 2024-04-25T16:01:48.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 -- 172.21.15.31:0/1277420329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 msgr2=0x561045f338b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5bc0afaf0 0x7ff5bc0b2000 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff5c801b000 tx=0x7ff5c801adc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f338b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff5c800f410 tx=0x7ff5c800f8d0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 -- 172.21.15.31:0/1277420329 shutdown_connections 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 0x561045f3ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f33e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1277420329 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f338b0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 -- 172.21.15.31:0/1277420329 >> 172.21.15.31:0/1277420329 conn(0x561045b82560 msgr2=0x561045f2ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 -- 172.21.15.31:0/1277420329 shutdown_connections 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 -- 172.21.15.31:0/1277420329 wait complete. 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 -- start start 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.531+0000 7ff5debc0740 1 Processor -- start 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f3d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b6d730 0x561045f3db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b8e410 0x561045f44ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5debc0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561045f3b780 con 0x561045b6d730 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5debc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561045e2e5d0 con 0x561045b65550 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5debc0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561045f3b900 con 0x561045b8e410 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5debc0740 10 client.0 _ll_register_callbacks cb 0x561045e1c080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d51cf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b6d730 0x561045f3db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d51cf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b6d730 0x561045f3db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35698/0 (socket says 172.21.15.31:35698) 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d51cf640 1 -- 172.21.15.31:0/1061938693 learned_addr learned my addr 172.21.15.31:0/1061938693 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f3d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b8e410 0x561045f44ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d51cf640 1 -- 172.21.15.31:0/1061938693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b8e410 msgr2=0x561045f44ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d51cf640 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b8e410 0x561045f44ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d51cf640 1 -- 172.21.15.31:0/1061938693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 msgr2=0x561045f3d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d51cf640 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f3d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d51cf640 1 -- 172.21.15.31:0/1061938693 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561045ccef70 con 0x561045b6d730 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5d51cf640 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b6d730 0x561045f3db80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc000c00 tx=0x7ff5cc0044d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0e1f8596397cd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.532+0000 7ff5b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1061938693 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc035070 con 0x561045b6d730 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1061938693 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff5cc003b40 con 0x561045b6d730 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5debc0740 1 -- 172.21.15.31:0/1061938693 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5c8011be0 con 0x561045b6d730 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1061938693 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff5cc003d40 con 0x561045b6d730 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5debc0740 10 client.15645 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5debc0740 1 -- 172.21.15.31:0/1061938693 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561045b66170 con 0x561045b6d730 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1061938693 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c8011be0 con 0x561045b6d730 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1061938693 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff5cc03b700 con 0x561045b6d730 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5debc0740 10 client.15645 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5debc0740 10 client.15645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5debc0740 20 client.15645 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561045b4d1f0 0x561045f4b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5debc0740 4 client.15645 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:48.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.533+0000 7ff5debc0740 1 -- 172.21.15.31:0/1061938693 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561045ea2710 con 0x561045b4d1f0 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.534+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561045b4d1f0 0x561045f4b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.534+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561045b4d1f0 0x561045f4b040 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11997d9721a2e65a server_cookie=6932a07a50c2856a in_seq=0 out_seq=0 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.534+0000 7ff5b97fa640 10 client.15645 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1061938693 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561045ea2710 con 0x561045b4d1f0 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5b97fa640 10 client.15645 handle_command_reply: tid=0 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 client.15645 shutdown 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 -- 172.21.15.31:0/1061938693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b6d730 msgr2=0x561045f3db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b6d730 0x561045f3db80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc000c00 tx=0x7ff5cc0044d0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 -- 172.21.15.31:0/1061938693 shutdown_connections 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561045b4d1f0 0x561045f4b040 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b8e410 0x561045f44ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b6d730 0x561045f3db80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1061938693 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f3d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5b97fa640 0 client.15645 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 -- 172.21.15.31:0/1061938693 >> 172.21.15.31:0/1061938693 conn(0x561045b82560 msgr2=0x561045f29ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 -- 172.21.15.31:0/1061938693 shutdown_connections 2024-04-25T16:01:48.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 1 -- 172.21.15.31:0/1061938693 wait complete. 2024-04-25T16:01:48.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.535+0000 7ff5debc0740 20 client.15645 trim_cache size 0 max 16384 2024-04-25T16:01:48.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5debc0740 1 Processor -- start 2024-04-25T16:01:48.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5debc0740 1 -- start start 2024-04-25T16:01:48.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b65550 0x561045f3c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b6d730 0x561045f371d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f4f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5debc0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d01a9070 con 0x561045b8e410 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5debc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d01aabb0 con 0x561045b6d730 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5debc0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d01a9740 con 0x561045b65550 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d61d1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f4f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d61d1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f4f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35702/0 (socket says 172.21.15.31:35702) 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d61d1640 1 -- 172.21.15.31:0/3740928427 learned_addr learned my addr 172.21.15.31:0/3740928427 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d51cf640 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b6d730 0x561045f371d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b65550 0x561045f3c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d61d1640 1 -- 172.21.15.31:0/3740928427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b65550 msgr2=0x561045f3c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b65550 0x561045f3c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d61d1640 1 -- 172.21.15.31:0/3740928427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b6d730 msgr2=0x561045f371d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b6d730 0x561045f371d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d61d1640 1 -- 172.21.15.31:0/3740928427 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561045f384f0 con 0x561045b8e410 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.554+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b65550 0x561045f3c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.555+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f4f7f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x561045f34a80 tx=0x7ff5c8026000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c9d7280d71d887b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.555+0000 7ff5bb7fe640 1 -- 172.21.15.31:0/3740928427 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c802c040 con 0x561045b8e410 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.555+0000 7ff5bb7fe640 1 -- 172.21.15.31:0/3740928427 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c8036070 con 0x561045b8e410 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.555+0000 7ff5bb7fe640 1 -- 172.21.15.31:0/3740928427 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c8010e30 con 0x561045b8e410 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 -- 172.21.15.31:0/3740928427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 msgr2=0x561045f4f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f4f7f0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x561045f34a80 tx=0x7ff5c8026000 comp rx=0 tx=0).stop 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 -- 172.21.15.31:0/3740928427 shutdown_connections 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f4f7f0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b6d730 0x561045f371d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 --2- 172.21.15.31:0/3740928427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b65550 0x561045f3c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 -- 172.21.15.31:0/3740928427 >> 172.21.15.31:0/3740928427 conn(0x561045b82560 msgr2=0x561045f36ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 -- 172.21.15.31:0/3740928427 shutdown_connections 2024-04-25T16:01:48.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 -- 172.21.15.31:0/3740928427 wait complete. 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 -- start start 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 Processor -- start 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f41720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 0x561045f41c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f42b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d01aa4e0 con 0x561045b8e410 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d01a9e10 con 0x561045b65550 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.556+0000 7ff5debc0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d01a7050 con 0x561045b6d730 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5debc0740 10 client.0 _ll_register_callbacks cb 0x7ff5cc0040d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d61d1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f42b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d61d1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f42b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35712/0 (socket says 172.21.15.31:35712) 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d61d1640 1 -- 172.21.15.31:0/1794954069 learned_addr learned my addr 172.21.15.31:0/1794954069 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f41720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d51cf640 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 0x561045f41c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d61d1640 1 -- 172.21.15.31:0/1794954069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 msgr2=0x561045f41c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 0x561045f41c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d61d1640 1 -- 172.21.15.31:0/1794954069 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 msgr2=0x561045f41720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f41720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d61d1640 1 -- 172.21.15.31:0/1794954069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561045f43150 con 0x561045b8e410 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5d61d1640 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f42b90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff5c801afa0 tx=0x7ff5c800f360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b16cecee06d0875 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1794954069 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c801f030 con 0x561045b8e410 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1794954069 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff5c801a3e0 con 0x561045b8e410 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5debc0740 1 -- 172.21.15.31:0/1794954069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5c8036070 con 0x561045b8e410 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.557+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1794954069 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff5c802c040 con 0x561045b8e410 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5debc0740 10 client.15651 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5debc0740 1 -- 172.21.15.31:0/1794954069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x561045f44410 con 0x561045b8e410 2024-04-25T16:01:48.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1794954069 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5c8036070 con 0x561045b8e410 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1794954069 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7ff5c8011b00 con 0x561045b8e410 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5debc0740 10 client.15651 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5debc0740 10 client.15651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5debc0740 20 client.15651 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561045b22630 0x561045fb5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5debc0740 4 client.15651 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5debc0740 1 -- 172.21.15.31:0/1794954069 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561045e9c580 con 0x561045b22630 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.558+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561045b22630 0x561045fb5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.559+0000 7ff5b97fa640 10 client.15651 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:48.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.559+0000 7ff5d59d0640 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561045b22630 0x561045fb5990 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e322648dc1693485 server_cookie=2a1de8f07ad91585 in_seq=0 out_seq=0 2024-04-25T16:01:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5b97fa640 1 -- 172.21.15.31:0/1794954069 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561045e9c580 con 0x561045b22630 2024-04-25T16:01:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5b97fa640 10 client.15651 handle_command_reply: tid=0 2024-04-25T16:01:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 client.15651 shutdown 2024-04-25T16:01:48.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 -- 172.21.15.31:0/1794954069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 msgr2=0x561045f42b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f42b90 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff5c801afa0 tx=0x7ff5c800f360 comp rx=0 tx=0).stop 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 -- 172.21.15.31:0/1794954069 shutdown_connections 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561045b22630 0x561045fb5990 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561045b8e410 0x561045f42b90 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561045b6d730 0x561045f41c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5b97fa640 0 client.15651 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 --2- 172.21.15.31:0/1794954069 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561045b65550 0x561045f41720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 -- 172.21.15.31:0/1794954069 >> 172.21.15.31:0/1794954069 conn(0x561045b82560 msgr2=0x561045f2dc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.560+0000 7ff5debc0740 1 -- 172.21.15.31:0/1794954069 shutdown_connections 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5debc0740 1 -- 172.21.15.31:0/1794954069 wait complete. 2024-04-25T16:01:48.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5debc0740 20 client.15651 trim_cache size 0 max 16384 2024-04-25T16:01:48.567 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:48.567 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:01:48.567 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "d5387d0a-cb2a-453c-a57c-600bd815ae30", 2024-04-25T16:01:48.567 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:01:48.567 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 -- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5bc0afaf0 msgr2=0x7ff5bc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5bc0afaf0 0x7ff5bc0b2000 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff5c801b000 tx=0x7ff5c801adc0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 -- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d01029e0 msgr2=0x7ff5d019a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d01029e0 0x7ff5d019a550 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff5c000e9f0 tx=0x7ff5c000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 -- 172.21.15.31:0/2958444131 shutdown_connections 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5bc0afaf0 0x7ff5bc0b2000 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff5d0108920 0x7ff5d01a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5d0103c20 0x7ff5d019aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 --2- 172.21.15.31:0/2958444131 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff5d01029e0 0x7ff5d019a550 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 -- 172.21.15.31:0/2958444131 >> 172.21.15.31:0/2958444131 conn(0x7ff5d00fdbd0 msgr2=0x7ff5d0104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 -- 172.21.15.31:0/2958444131 shutdown_connections 2024-04-25T16:01:48.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.561+0000 7ff5d7c34640 1 -- 172.21.15.31:0/2958444131 wait complete. 2024-04-25T16:01:48.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:01:48.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:01:48.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.686+0000 7f955a772640 1 -- 172.21.15.31:0/30902778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95541029f0 msgr2=0x7f95541036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.686+0000 7f955a772640 1 --2- 172.21.15.31:0/30902778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95541029f0 0x7f95541036f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f9544009900 tx=0x7f954402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:48.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.686+0000 7f955a772640 1 -- 172.21.15.31:0/30902778 shutdown_connections 2024-04-25T16:01:48.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.686+0000 7f955a772640 1 --2- 172.21.15.31:0/30902778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9554108930 0x7f955410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.686+0000 7f955a772640 1 --2- 172.21.15.31:0/30902778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554103c30 0x7f9554104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.686+0000 7f955a772640 1 --2- 172.21.15.31:0/30902778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95541029f0 0x7f95541036f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.686+0000 7f955a772640 1 -- 172.21.15.31:0/30902778 >> 172.21.15.31:0/30902778 conn(0x7f95540fdbc0 msgr2=0x7f9554100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.686+0000 7f955a772640 1 -- 172.21.15.31:0/30902778 shutdown_connections 2024-04-25T16:01:48.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.686+0000 7f955a772640 1 -- 172.21.15.31:0/30902778 wait complete. 2024-04-25T16:01:48.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.687+0000 7f955a772640 1 Processor -- start 2024-04-25T16:01:48.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.687+0000 7f955a772640 1 -- start start 2024-04-25T16:01:48.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.687+0000 7f955a772640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95541029f0 0x7f955419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.687+0000 7f955a772640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9554103c30 0x7f955419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.687+0000 7f955a772640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554108930 0x7f95541a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.687+0000 7f955a772640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f955410e4f0 con 0x7f9554103c30 2024-04-25T16:01:48.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.687+0000 7f955a772640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f955410e370 con 0x7f9554108930 2024-04-25T16:01:48.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.687+0000 7f955a772640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f955410e670 con 0x7f95541029f0 2024-04-25T16:01:48.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95537fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9554103c30 0x7f955419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95537fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9554103c30 0x7f955419a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35732/0 (socket says 172.21.15.31:35732) 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95537fe640 1 -- 172.21.15.31:0/2728790063 learned_addr learned my addr 172.21.15.31:0/2728790063 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f9558d0f640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554108930 0x7f95541a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f9553fff640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95541029f0 0x7f955419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95537fe640 1 -- 172.21.15.31:0/2728790063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95541029f0 msgr2=0x7f955419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95537fe640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95541029f0 0x7f955419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95537fe640 1 -- 172.21.15.31:0/2728790063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554108930 msgr2=0x7f95541a1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95537fe640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554108930 0x7f95541a1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95537fe640 1 -- 172.21.15.31:0/2728790063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95541a1dc0 con 0x7f9554103c30 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f9558d0f640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554108930 0x7f95541a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f9553fff640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95541029f0 0x7f955419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95537fe640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9554103c30 0x7f955419a7c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f954800e9f0 tx=0x7f954800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95517fa640 1 -- 172.21.15.31:0/2728790063 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954800cd80 con 0x7f9554103c30 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95517fa640 1 -- 172.21.15.31:0/2728790063 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95480108e0 con 0x7f9554103c30 2024-04-25T16:01:48.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f95517fa640 1 -- 172.21.15.31:0/2728790063 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9548018980 con 0x7f9554103c30 2024-04-25T16:01:48.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f955a772640 1 -- 172.21.15.31:0/2728790063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95541a20b0 con 0x7f9554103c30 2024-04-25T16:01:48.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.688+0000 7f955a772640 1 -- 172.21.15.31:0/2728790063 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95541a2600 con 0x7f9554103c30 2024-04-25T16:01:48.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9561624740 1 Processor -- start 2024-04-25T16:01:48.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9561624740 1 -- start start 2024-04-25T16:01:48.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f95517fa640 1 -- 172.21.15.31:0/2728790063 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9548010430 con 0x7f9554103c30 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9561624740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e1ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9561624740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e1bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9561624740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e1cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9561624740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5639a2d80d80 con 0x5639a2a514d0 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9561624740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5639a2d3a8b0 con 0x5639a2a7a390 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9561624740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5639a2d7cd40 con 0x5639a2a596b0 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9553fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e1bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9553fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e1bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35742/0 (socket says 172.21.15.31:35742) 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.691+0000 7f9553fff640 1 -- 172.21.15.31:0/4037245250 learned_addr learned my addr 172.21.15.31:0/4037245250 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f9558d0f640 1 --2- 172.21.15.31:0/4037245250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e1ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f95537fe640 1 --2- 172.21.15.31:0/4037245250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e1cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f9553fff640 1 -- 172.21.15.31:0/4037245250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 msgr2=0x5639a2e1cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f9553fff640 1 --2- 172.21.15.31:0/4037245250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e1cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f9553fff640 1 -- 172.21.15.31:0/4037245250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 msgr2=0x5639a2e1ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f9553fff640 1 --2- 172.21.15.31:0/4037245250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e1ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f9553fff640 1 -- 172.21.15.31:0/4037245250 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a2d0ba40 con 0x5639a2a514d0 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f95517fa640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f953c0afaf0 0x7f953c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f9558d0f640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f953c0afaf0 0x7f953c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f9553fff640 1 --2- 172.21.15.31:0/4037245250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e1bfe0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9544031440 tx=0x7f9544032ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3608483fdb5e5d39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f95517fa640 1 -- 172.21.15.31:0/2728790063 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f95480defa0 con 0x7f9554103c30 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f953a7fc640 1 -- 172.21.15.31:0/4037245250 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954403f070 con 0x5639a2a514d0 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f953a7fc640 1 -- 172.21.15.31:0/4037245250 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9544004040 con 0x5639a2a514d0 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.692+0000 7f953a7fc640 1 -- 172.21.15.31:0/4037245250 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544035070 con 0x5639a2a514d0 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9561624740 1 -- 172.21.15.31:0/4037245250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 msgr2=0x5639a2e1bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9561624740 1 --2- 172.21.15.31:0/4037245250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e1bfe0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f9544031440 tx=0x7f9544032ba0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9558d0f640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f953c0afaf0 0x7f953c0b2000 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x5639a2e1cab0 tx=0x7f954c009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9561624740 1 -- 172.21.15.31:0/4037245250 shutdown_connections 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9561624740 1 --2- 172.21.15.31:0/4037245250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e1cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9561624740 1 --2- 172.21.15.31:0/4037245250 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e1bfe0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9561624740 1 --2- 172.21.15.31:0/4037245250 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e1ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9561624740 1 -- 172.21.15.31:0/4037245250 >> 172.21.15.31:0/4037245250 conn(0x5639a2a6e4e0 msgr2=0x5639a2e17270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9561624740 1 -- 172.21.15.31:0/4037245250 shutdown_connections 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.693+0000 7f9561624740 1 -- 172.21.15.31:0/4037245250 wait complete. 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9561624740 1 -- start start 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9561624740 1 Processor -- start 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9561624740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e25920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9561624740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e25e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9561624740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e2cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9561624740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5639a2e1d720 con 0x5639a2a514d0 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9561624740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5639a2d07d10 con 0x5639a2a7a390 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9561624740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5639a2e1d8a0 con 0x5639a2a596b0 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9561624740 10 client.0 _ll_register_callbacks cb 0x5639a2d34210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9553fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e25920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9553fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e25920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35754/0 (socket says 172.21.15.31:35754) 2024-04-25T16:01:48.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9553fff640 1 -- 172.21.15.31:0/3949981792 learned_addr learned my addr 172.21.15.31:0/3949981792 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f95537fe640 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e25e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9558d0f640 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e2cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9553fff640 1 -- 172.21.15.31:0/3949981792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 msgr2=0x5639a2e25e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9553fff640 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e25e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9553fff640 1 -- 172.21.15.31:0/3949981792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 msgr2=0x5639a2e2cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9553fff640 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e2cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.694+0000 7f9553fff640 1 -- 172.21.15.31:0/3949981792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639a2b5c7d0 con 0x5639a2a514d0 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f95537fe640 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e25e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f9553fff640 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e25920 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9544004480 tx=0x7f95440044b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8395390ee5573c34 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f95337fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f95337fe640 1 -- 172.21.15.31:0/3949981792 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954403f070 con 0x5639a2a514d0 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f95337fe640 1 -- 172.21.15.31:0/3949981792 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f95440035f0 con 0x5639a2a514d0 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f9561624740 1 -- 172.21.15.31:0/3949981792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9544004040 con 0x5639a2a514d0 2024-04-25T16:01:48.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f95337fe640 1 -- 172.21.15.31:0/3949981792 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f9544035040 con 0x5639a2a514d0 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f9561624740 10 client.15663 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f9561624740 1 -- 172.21.15.31:0/3949981792 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5639a2a923f0 con 0x5639a2a514d0 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f95337fe640 1 -- 172.21.15.31:0/3949981792 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9544004040 con 0x5639a2a514d0 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f95337fe640 1 -- 172.21.15.31:0/3949981792 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f9544004aa0 con 0x5639a2a514d0 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f9561624740 10 client.15663 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f9561624740 10 client.15663 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f9561624740 20 client.15663 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f9561624740 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5639a2a38f60 0x5639a2e32e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.695+0000 7f9561624740 4 client.15663 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.696+0000 7f9561624740 1 -- 172.21.15.31:0/3949981792 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639a2d8e990 con 0x5639a2a38f60 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.696+0000 7f95537fe640 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5639a2a38f60 0x5639a2e32e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.696+0000 7f95537fe640 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5639a2a38f60 0x5639a2e32e10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=211ca48e3a7ce04f server_cookie=52150d187ffebd in_seq=0 out_seq=0 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.696+0000 7f95337fe640 10 client.15663 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f95337fe640 1 -- 172.21.15.31:0/3949981792 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5639a2d8e990 con 0x5639a2a38f60 2024-04-25T16:01:48.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f95337fe640 10 client.15663 handle_command_reply: tid=0 2024-04-25T16:01:48.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 client.15663 shutdown 2024-04-25T16:01:48.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 -- 172.21.15.31:0/3949981792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 msgr2=0x5639a2e25920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e25920 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f9544004480 tx=0x7f95440044b0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 -- 172.21.15.31:0/3949981792 shutdown_connections 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5639a2a38f60 0x5639a2e32e10 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e2cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e25e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f95337fe640 0 client.15663 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 --2- 172.21.15.31:0/3949981792 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e25920 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 -- 172.21.15.31:0/3949981792 >> 172.21.15.31:0/3949981792 conn(0x5639a2a6e4e0 msgr2=0x5639a2e1a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 -- 172.21.15.31:0/3949981792 shutdown_connections 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 1 -- 172.21.15.31:0/3949981792 wait complete. 2024-04-25T16:01:48.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.697+0000 7f9561624740 20 client.15663 trim_cache size 0 max 16384 2024-04-25T16:01:48.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:48 smithi031 ceph-mon[29140]: pgmap v366: 97 pgs: 97 active+clean; 654 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:01:48.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.715+0000 7f9561624740 1 Processor -- start 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.715+0000 7f9561624740 1 -- start start 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9561624740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e163f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9561624740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e2cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9561624740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e2d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9561624740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95541ab1c0 con 0x5639a2a514d0 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9561624740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95541acd00 con 0x5639a2a7a390 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9561624740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95541ab890 con 0x5639a2a596b0 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9553fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e163f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9553fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e163f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35758/0 (socket says 172.21.15.31:35758) 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9553fff640 1 -- 172.21.15.31:0/2833470381 learned_addr learned my addr 172.21.15.31:0/2833470381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f95537fe640 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e2cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9558d0f640 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e2d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9553fff640 1 -- 172.21.15.31:0/2833470381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 msgr2=0x5639a2e2cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9553fff640 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e2cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9553fff640 1 -- 172.21.15.31:0/2833470381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 msgr2=0x5639a2e2d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9553fff640 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e2d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9553fff640 1 -- 172.21.15.31:0/2833470381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639a2a520f0 con 0x5639a2a514d0 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9558d0f640 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e2d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f95537fe640 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e2cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9553fff640 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e163f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9544004e40 tx=0x7f9544005380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7df76cd0091e1f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.716+0000 7f9539ffb640 1 -- 172.21.15.31:0/2833470381 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954403f070 con 0x5639a2a514d0 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9539ffb640 1 -- 172.21.15.31:0/2833470381 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f954404d070 con 0x5639a2a514d0 2024-04-25T16:01:48.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9539ffb640 1 -- 172.21.15.31:0/2833470381 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544031cb0 con 0x5639a2a514d0 2024-04-25T16:01:48.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9561624740 1 -- 172.21.15.31:0/2833470381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 msgr2=0x5639a2e163f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9561624740 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e163f0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f9544004e40 tx=0x7f9544005380 comp rx=0 tx=0).stop 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9561624740 1 -- 172.21.15.31:0/2833470381 shutdown_connections 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9561624740 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a7a390 0x5639a2e2d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9561624740 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a596b0 0x5639a2e2cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9561624740 1 --2- 172.21.15.31:0/2833470381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a514d0 0x5639a2e163f0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9561624740 1 -- 172.21.15.31:0/2833470381 >> 172.21.15.31:0/2833470381 conn(0x5639a2a6e4e0 msgr2=0x5639a2e25b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9561624740 1 -- 172.21.15.31:0/2833470381 shutdown_connections 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.717+0000 7f9561624740 1 -- 172.21.15.31:0/2833470381 wait complete. 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9561624740 1 -- start start 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9561624740 1 Processor -- start 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9561624740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a514d0 0x5639a2e2d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9561624740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a596b0 0x5639a2e2a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9561624740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a7a390 0x5639a2e2a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9561624740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95541ac630 con 0x5639a2a7a390 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9561624740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95541abf60 con 0x5639a2a596b0 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9561624740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95541a91a0 con 0x5639a2a514d0 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9561624740 10 client.0 _ll_register_callbacks cb 0x7f954c0096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9558d0f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a7a390 0x5639a2e2a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9558d0f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a7a390 0x5639a2e2a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35770/0 (socket says 172.21.15.31:35770) 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9558d0f640 1 -- 172.21.15.31:0/3552749524 learned_addr learned my addr 172.21.15.31:0/3552749524 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f95537fe640 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a596b0 0x5639a2e2a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.718+0000 7f9553fff640 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a514d0 0x5639a2e2d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f9558d0f640 1 -- 172.21.15.31:0/3552749524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a514d0 msgr2=0x5639a2e2d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f9558d0f640 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a514d0 0x5639a2e2d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f9558d0f640 1 -- 172.21.15.31:0/3552749524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a596b0 msgr2=0x5639a2e2a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f9558d0f640 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a596b0 0x5639a2e2a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f9558d0f640 1 -- 172.21.15.31:0/3552749524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639a2d414b0 con 0x5639a2a7a390 2024-04-25T16:01:48.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f9558d0f640 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a7a390 0x5639a2e2a690 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f954c011e10 tx=0x7f954c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f64c6f064f05df0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f95337fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f95337fe640 1 -- 172.21.15.31:0/3552749524 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954c01f610 con 0x5639a2a7a390 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f95337fe640 1 -- 172.21.15.31:0/3552749524 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f954c01f7b0 con 0x5639a2a7a390 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f9561624740 1 -- 172.21.15.31:0/3552749524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f954404d070 con 0x5639a2a7a390 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f95337fe640 1 -- 172.21.15.31:0/3552749524 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f954c01f9b0 con 0x5639a2a7a390 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f9561624740 10 client.15669 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f9561624740 1 -- 172.21.15.31:0/3552749524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5639a2e9a270 con 0x5639a2a7a390 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.719+0000 7f95337fe640 1 -- 172.21.15.31:0/3552749524 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f954404d070 con 0x5639a2a7a390 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f95337fe640 1 -- 172.21.15.31:0/3552749524 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f954c029070 con 0x5639a2a7a390 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f9561624740 10 client.15669 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f9561624740 10 client.15669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f9561624740 20 client.15669 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f9561624740 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5639a2a0e3a0 0x5639a2e9cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:48.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f9561624740 4 client.15669 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f9561624740 1 -- 172.21.15.31:0/3552749524 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5639a2d88800 con 0x5639a2a0e3a0 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f9553fff640 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5639a2a0e3a0 0x5639a2e9cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f95337fe640 10 client.15669 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.720+0000 7f9553fff640 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5639a2a0e3a0 0x5639a2e9cc90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71b3599b24b3f4ad server_cookie=489d70020dd4b7a9 in_seq=0 out_seq=0 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f95337fe640 1 -- 172.21.15.31:0/3552749524 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5639a2d88800 con 0x5639a2a0e3a0 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f95337fe640 10 client.15669 handle_command_reply: tid=0 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 client.15669 shutdown 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 -- 172.21.15.31:0/3552749524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a7a390 msgr2=0x5639a2e2a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a7a390 0x5639a2e2a690 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f954c011e10 tx=0x7f954c004a40 comp rx=0 tx=0).stop 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 -- 172.21.15.31:0/3552749524 shutdown_connections 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5639a2a0e3a0 0x5639a2e9cc90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5639a2a7a390 0x5639a2e2a690 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5639a2a596b0 0x5639a2e2a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 --2- 172.21.15.31:0/3552749524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5639a2a514d0 0x5639a2e2d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 -- 172.21.15.31:0/3552749524 >> 172.21.15.31:0/3552749524 conn(0x5639a2a6e4e0 msgr2=0x5639a2e1a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f95337fe640 0 client.15669 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 -- 172.21.15.31:0/3552749524 shutdown_connections 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 1 -- 172.21.15.31:0/3552749524 wait complete. 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.721+0000 7f9561624740 20 client.15669 trim_cache size 0 max 16384 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stdout: "d5387d0a-cb2a-453c-a57c-600bd815ae30": { 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:01:48.728 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "d5387d0a-cb2a-453c-a57c-600bd815ae30", 2024-04-25T16:01:48.729 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:01:48.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:01:48.729 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:01:48.729 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:01:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 -- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f953c0afaf0 msgr2=0x7f953c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f953c0afaf0 0x7f953c0b2000 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x5639a2e1cab0 tx=0x7f954c009800 comp rx=0 tx=0).stop 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 -- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9554103c30 msgr2=0x7f955419a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9554103c30 0x7f955419a7c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f954800e9f0 tx=0x7f954800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 -- 172.21.15.31:0/2728790063 shutdown_connections 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f953c0afaf0 0x7f953c0b2000 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554108930 0x7f95541a1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9554103c30 0x7f955419a7c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 --2- 172.21.15.31:0/2728790063 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95541029f0 0x7f955419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 -- 172.21.15.31:0/2728790063 >> 172.21.15.31:0/2728790063 conn(0x7f95540fdbc0 msgr2=0x7f9554109ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 -- 172.21.15.31:0/2728790063 shutdown_connections 2024-04-25T16:01:48.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:01:48.722+0000 7f955a772640 1 -- 172.21.15.31:0/2728790063 wait complete. 2024-04-25T16:01:48.733 INFO:tasks.cephfs.filesystem:scrub status for tag:d5387d0a-cb2a-453c-a57c-600bd815ae30 - {'path': '/', 'tag': 'd5387d0a-cb2a-453c-a57c-600bd815ae30', 'options': 'recursive,force'} 2024-04-25T16:01:48.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:48 smithi063 ceph-mon[32413]: pgmap v366: 97 pgs: 97 active+clean; 654 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:01:48.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:48 smithi186 ceph-mon[33258]: pgmap v366: 97 pgs: 97 active+clean; 654 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-25T16:01:49.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:49 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:01:49.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:49 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:49.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:49 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:01:49.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:49 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:49.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:49 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:01:49.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:49 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:49.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:49 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:01:49.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:49 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:49.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:49 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:01:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:49 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:49 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:01:49.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:49 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:01:50.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:50 smithi063 ceph-mon[32413]: pgmap v367: 97 pgs: 97 active+clean; 654 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:01:50.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:50 smithi186 ceph-mon[33258]: pgmap v367: 97 pgs: 97 active+clean; 654 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:01:50.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:50 smithi031 ceph-mon[29140]: pgmap v367: 97 pgs: 97 active+clean; 654 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-04-25T16:01:52.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:52 smithi063 ceph-mon[32413]: pgmap v368: 97 pgs: 97 active+clean; 654 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-25T16:01:52.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:52 smithi186 ceph-mon[33258]: pgmap v368: 97 pgs: 97 active+clean; 654 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-25T16:01:52.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:52 smithi031 ceph-mon[29140]: pgmap v368: 97 pgs: 97 active+clean; 654 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 9 op/s 2024-04-25T16:01:53.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:53 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:01:53.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:53 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:01:53.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:53 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:01:54.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:54 smithi063 ceph-mon[32413]: pgmap v369: 97 pgs: 97 active+clean; 663 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-04-25T16:01:54.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:54 smithi186 ceph-mon[33258]: pgmap v369: 97 pgs: 97 active+clean; 663 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-04-25T16:01:54.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:54 smithi031 ceph-mon[29140]: pgmap v369: 97 pgs: 97 active+clean; 663 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.3 KiB/s wr, 13 op/s 2024-04-25T16:01:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.147+0000 7fca4015e640 1 -- 172.21.15.31:0/368108251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 msgr2=0x7fca38103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.147+0000 7fca4015e640 1 --2- 172.21.15.31:0/368108251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 0x7fca38103710 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fca34009900 tx=0x7fca3402a410 comp rx=0 tx=0).stop 2024-04-25T16:01:55.150 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.147+0000 7fca4015e640 1 -- 172.21.15.31:0/368108251 shutdown_connections 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.147+0000 7fca4015e640 1 --2- 172.21.15.31:0/368108251 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca38108950 0x7fca3810ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.147+0000 7fca4015e640 1 --2- 172.21.15.31:0/368108251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca38103c50 0x7fca38104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.147+0000 7fca4015e640 1 --2- 172.21.15.31:0/368108251 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 0x7fca38103710 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.147+0000 7fca4015e640 1 -- 172.21.15.31:0/368108251 >> 172.21.15.31:0/368108251 conn(0x7fca380fdc00 msgr2=0x7fca381000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.147+0000 7fca4015e640 1 -- 172.21.15.31:0/368108251 shutdown_connections 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.147+0000 7fca4015e640 1 -- 172.21.15.31:0/368108251 wait complete. 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca4015e640 1 Processor -- start 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca4015e640 1 -- start start 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca4015e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 0x7fca3819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca4015e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca38103c50 0x7fca3819a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca4015e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca38108950 0x7fca381a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca4015e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fca3810e4f0 con 0x7fca38102a10 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca4015e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca3810e370 con 0x7fca38108950 2024-04-25T16:01:55.151 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca4015e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fca3810e670 con 0x7fca38103c50 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca3defa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 0x7fca3819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca3defa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 0x7fca3819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46038/0 (socket says 172.21.15.31:46038) 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.148+0000 7fca3defa640 1 -- 172.21.15.31:0/3815296703 learned_addr learned my addr 172.21.15.31:0/3815296703 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca38108950 0x7fca381a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca3d6f9640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca38103c50 0x7fca3819a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca3defa640 1 -- 172.21.15.31:0/3815296703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca38103c50 msgr2=0x7fca3819a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca3defa640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca38103c50 0x7fca3819a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca3defa640 1 -- 172.21.15.31:0/3815296703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca38108950 msgr2=0x7fca381a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca3defa640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca38108950 0x7fca381a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca3defa640 1 -- 172.21.15.31:0/3815296703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca381a1de0 con 0x7fca38102a10 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca3defa640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 0x7fca3819a2a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fca34006010 tx=0x7fca34031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca2effd640 1 -- 172.21.15.31:0/3815296703 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3403f030 con 0x7fca38102a10 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca4015e640 1 -- 172.21.15.31:0/3815296703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca381a2070 con 0x7fca38102a10 2024-04-25T16:01:55.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca2effd640 1 -- 172.21.15.31:0/3815296703 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca34003aa0 con 0x7fca38102a10 2024-04-25T16:01:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca4015e640 1 -- 172.21.15.31:0/3815296703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca381a25b0 con 0x7fca38102a10 2024-04-25T16:01:55.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.149+0000 7fca2effd640 1 -- 172.21.15.31:0/3815296703 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34035070 con 0x7fca38102a10 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.152+0000 7fca4700f740 1 Processor -- start 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.152+0000 7fca4700f740 1 -- start start 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca2effd640 1 -- 172.21.15.31:0/3815296703 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fca34042030 con 0x7fca38102a10 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca4700f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 0x5597cd801d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca4700f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd802300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca4700f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd43c4d0 0x5597cd809200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca4700f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5597cd763460 con 0x5597cd43c4d0 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca4700f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5597cd71d590 con 0x5597cd4342f0 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca4700f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5597cd75f420 con 0x5597cd45d1b0 2024-04-25T16:01:55.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3d6f9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd43c4d0 0x5597cd809200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3defa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd802300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3d6f9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd43c4d0 0x5597cd809200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46046/0 (socket says 172.21.15.31:46046) 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3d6f9640 1 -- 172.21.15.31:0/2956472050 learned_addr learned my addr 172.21.15.31:0/2956472050 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/2956472050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 0x5597cd801d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3d6f9640 1 -- 172.21.15.31:0/2956472050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 msgr2=0x5597cd801d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3d6f9640 1 --2- 172.21.15.31:0/2956472050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 0x5597cd801d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3d6f9640 1 -- 172.21.15.31:0/2956472050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 msgr2=0x5597cd802300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3d6f9640 1 --2- 172.21.15.31:0/2956472050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd802300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3d6f9640 1 -- 172.21.15.31:0/2956472050 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597cd72a0d0 con 0x5597cd43c4d0 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca2effd640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca240afae0 0x7fca240b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.153+0000 7fca3d6f9640 1 --2- 172.21.15.31:0/2956472050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd43c4d0 0x5597cd809200 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fca2800bd10 tx=0x7fca280107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e29aa52e233e03df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.154+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca240afae0 0x7fca240b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.154+0000 7fca2effd640 1 -- 172.21.15.31:0/3815296703 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fca340ff0f0 con 0x7fca38102a10 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.154+0000 7fca23fff640 1 -- 172.21.15.31:0/2956472050 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28005220 con 0x5597cd43c4d0 2024-04-25T16:01:55.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.154+0000 7fca23fff640 1 -- 172.21.15.31:0/2956472050 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca28004c50 con 0x5597cd43c4d0 2024-04-25T16:01:55.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.154+0000 7fca23fff640 1 -- 172.21.15.31:0/2956472050 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca28011680 con 0x5597cd43c4d0 2024-04-25T16:01:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 -- 172.21.15.31:0/2956472050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd43c4d0 msgr2=0x5597cd809200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca240afae0 0x7fca240b1ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x5597cd802f50 tx=0x7fca300096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 --2- 172.21.15.31:0/2956472050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd43c4d0 0x5597cd809200 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fca2800bd10 tx=0x7fca280107a0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 -- 172.21.15.31:0/2956472050 shutdown_connections 2024-04-25T16:01:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 --2- 172.21.15.31:0/2956472050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd43c4d0 0x5597cd809200 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 --2- 172.21.15.31:0/2956472050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd802300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 --2- 172.21.15.31:0/2956472050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 0x5597cd801d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 -- 172.21.15.31:0/2956472050 >> 172.21.15.31:0/2956472050 conn(0x5597cd451300 msgr2=0x5597cd7f9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 -- 172.21.15.31:0/2956472050 shutdown_connections 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 -- 172.21.15.31:0/2956472050 wait complete. 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 -- start start 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.155+0000 7fca4700f740 1 Processor -- start 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca4700f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd80bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca4700f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 0x5597cd80bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca4700f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd812f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca4700f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5597cd809c50 con 0x5597cd45d1b0 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca4700f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5597cd6eab60 con 0x5597cd4342f0 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca4700f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5597cd809dd0 con 0x5597cd43c4d0 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca4700f740 10 client.0 _ll_register_callbacks cb 0x5597cd70f2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3e6fb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd812f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3e6fb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd812f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46056/0 (socket says 172.21.15.31:46056) 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3e6fb640 1 -- 172.21.15.31:0/1265301331 learned_addr learned my addr 172.21.15.31:0/1265301331 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:55.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3d6f9640 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 0x5597cd80bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3defa640 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd80bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3e6fb640 1 -- 172.21.15.31:0/1265301331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 msgr2=0x5597cd80bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 0x5597cd80bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3e6fb640 1 -- 172.21.15.31:0/1265301331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 msgr2=0x5597cd80bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd80bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.156+0000 7fca3e6fb640 1 -- 172.21.15.31:0/1265301331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597cd475210 con 0x5597cd45d1b0 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd812f50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fca30010790 tx=0x7fca30004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70f5dd06f72e3523 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca217fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca217fa640 1 -- 172.21.15.31:0/1265301331 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca30004b20 con 0x5597cd45d1b0 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca217fa640 1 -- 172.21.15.31:0/1265301331 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fca30004cc0 con 0x5597cd45d1b0 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca4700f740 1 -- 172.21.15.31:0/1265301331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca28004c50 con 0x5597cd45d1b0 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca217fa640 1 -- 172.21.15.31:0/1265301331 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fca3001f8b0 con 0x5597cd45d1b0 2024-04-25T16:01:55.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca4700f740 10 client.15681 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca4700f740 1 -- 172.21.15.31:0/1265301331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5597cd541620 con 0x5597cd45d1b0 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca217fa640 1 -- 172.21.15.31:0/1265301331 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca28004c50 con 0x5597cd45d1b0 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca217fa640 1 -- 172.21.15.31:0/1265301331 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fca30026070 con 0x5597cd45d1b0 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca4700f740 10 client.15681 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca4700f740 10 client.15681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.157+0000 7fca4700f740 20 client.15681 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.158+0000 7fca4700f740 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5597cd41bd80 0x5597cd8198a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.158+0000 7fca4700f740 4 client.15681 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.158+0000 7fca4700f740 1 -- 172.21.15.31:0/1265301331 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597cd771070 con 0x5597cd41bd80 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.158+0000 7fca3defa640 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5597cd41bd80 0x5597cd8198a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.158+0000 7fca3defa640 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5597cd41bd80 0x5597cd8198a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e88a6cab0da2ab18 server_cookie=127995e98edc2699 in_seq=0 out_seq=0 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.158+0000 7fca217fa640 10 client.15681 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:55.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca217fa640 1 -- 172.21.15.31:0/1265301331 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5597cd771070 con 0x5597cd41bd80 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca217fa640 10 client.15681 handle_command_reply: tid=0 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca4700f740 1 client.15681 shutdown 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca4700f740 1 -- 172.21.15.31:0/1265301331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 msgr2=0x5597cd812f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca4700f740 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd812f50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fca30010790 tx=0x7fca30004ab0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca4700f740 1 -- 172.21.15.31:0/1265301331 shutdown_connections 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca4700f740 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5597cd41bd80 0x5597cd8198a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca4700f740 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd812f50 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca4700f740 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 0x5597cd80bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca217fa640 0 client.15681 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.159+0000 7fca4700f740 1 --2- 172.21.15.31:0/1265301331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd80bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.160+0000 7fca4700f740 1 -- 172.21.15.31:0/1265301331 >> 172.21.15.31:0/1265301331 conn(0x5597cd451300 msgr2=0x5597cd7f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.160+0000 7fca4700f740 1 -- 172.21.15.31:0/1265301331 shutdown_connections 2024-04-25T16:01:55.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.160+0000 7fca4700f740 1 -- 172.21.15.31:0/1265301331 wait complete. 2024-04-25T16:01:55.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.160+0000 7fca4700f740 20 client.15681 trim_cache size 0 max 16384 2024-04-25T16:01:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca4700f740 1 Processor -- start 2024-04-25T16:01:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca4700f740 1 -- start start 2024-04-25T16:01:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca4700f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd4342f0 0x5597cd806590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca4700f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd43c4d0 0x5597cd7f92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca4700f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 0x5597cd80a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca4700f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fca381a90e0 con 0x5597cd4342f0 2024-04-25T16:01:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca4700f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca381aac20 con 0x5597cd43c4d0 2024-04-25T16:01:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca4700f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fca381a97b0 con 0x5597cd45d1b0 2024-04-25T16:01:55.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca3defa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd4342f0 0x5597cd806590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca3defa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd4342f0 0x5597cd806590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46060/0 (socket says 172.21.15.31:46060) 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.175+0000 7fca3defa640 1 -- 172.21.15.31:0/131404946 learned_addr learned my addr 172.21.15.31:0/131404946 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 0x5597cd80a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca3defa640 1 -- 172.21.15.31:0/131404946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 msgr2=0x5597cd80a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca3d6f9640 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd43c4d0 0x5597cd7f92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca3defa640 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 0x5597cd80a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca3defa640 1 -- 172.21.15.31:0/131404946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd43c4d0 msgr2=0x5597cd7f92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca3defa640 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd43c4d0 0x5597cd7f92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca3defa640 1 -- 172.21.15.31:0/131404946 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597cd802b70 con 0x5597cd4342f0 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca3d6f9640 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd43c4d0 0x5597cd7f92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca3defa640 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd4342f0 0x5597cd806590 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fca34031440 tx=0x7fca340fcad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e9a672cdb7b4f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca237fe640 1 -- 172.21.15.31:0/131404946 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34049040 con 0x5597cd4342f0 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca237fe640 1 -- 172.21.15.31:0/131404946 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca340c71a0 con 0x5597cd4342f0 2024-04-25T16:01:55.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.176+0000 7fca237fe640 1 -- 172.21.15.31:0/131404946 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca34102d80 con 0x5597cd4342f0 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 -- 172.21.15.31:0/131404946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd4342f0 msgr2=0x5597cd806590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd4342f0 0x5597cd806590 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fca34031440 tx=0x7fca340fcad0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 -- 172.21.15.31:0/131404946 shutdown_connections 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd45d1b0 0x5597cd80a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd43c4d0 0x5597cd7f92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 --2- 172.21.15.31:0/131404946 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd4342f0 0x5597cd806590 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 -- 172.21.15.31:0/131404946 >> 172.21.15.31:0/131404946 conn(0x5597cd451300 msgr2=0x5597cd7f8ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 -- 172.21.15.31:0/131404946 shutdown_connections 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 -- 172.21.15.31:0/131404946 wait complete. 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 -- start start 2024-04-25T16:01:55.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.177+0000 7fca4700f740 1 Processor -- start 2024-04-25T16:01:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca4700f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd80f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca4700f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 0x5597cd80fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca4700f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd810c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca4700f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fca381aa550 con 0x5597cd45d1b0 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca4700f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca381a9e80 con 0x5597cd4342f0 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca4700f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fca381a70c0 con 0x5597cd43c4d0 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca4700f740 10 client.0 _ll_register_callbacks cb 0x7fca30009d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3e6fb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd810c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3e6fb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd810c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46066/0 (socket says 172.21.15.31:46066) 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3e6fb640 1 -- 172.21.15.31:0/4288913114 learned_addr learned my addr 172.21.15.31:0/4288913114 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3defa640 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd80f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3d6f9640 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 0x5597cd80fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3e6fb640 1 -- 172.21.15.31:0/4288913114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 msgr2=0x5597cd80fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 0x5597cd80fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3e6fb640 1 -- 172.21.15.31:0/4288913114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 msgr2=0x5597cd80f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd80f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3e6fb640 1 -- 172.21.15.31:0/4288913114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597cd811250 con 0x5597cd45d1b0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3d6f9640 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 0x5597cd80fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca3e6fb640 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd810c90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fca30006010 tx=0x7fca30005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f92f428136c11b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.178+0000 7fca217fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca217fa640 1 -- 172.21.15.31:0/4288913114 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca3001e420 con 0x5597cd45d1b0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca217fa640 1 -- 172.21.15.31:0/4288913114 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fca3001e5c0 con 0x5597cd45d1b0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca4700f740 1 -- 172.21.15.31:0/4288913114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fca340c71a0 con 0x5597cd45d1b0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca217fa640 1 -- 172.21.15.31:0/4288913114 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fca30003970 con 0x5597cd45d1b0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca4700f740 10 client.15687 fetch_fsmap learned FSMap version 27 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca4700f740 1 -- 172.21.15.31:0/4288913114 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5597cd812450 con 0x5597cd45d1b0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca217fa640 1 -- 172.21.15.31:0/4288913114 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca340c71a0 con 0x5597cd45d1b0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca217fa640 1 -- 172.21.15.31:0/4288913114 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fca30026040 con 0x5597cd45d1b0 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca4700f740 10 client.15687 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca4700f740 10 client.15687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca4700f740 20 client.15687 populate_metadata read hostname 'smithi031' 2024-04-25T16:01:55.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca4700f740 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5597cd3f11c0 0x5597cd883fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:01:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca4700f740 4 client.15687 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:01:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.179+0000 7fca4700f740 1 -- 172.21.15.31:0/4288913114 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5597cd76aee0 con 0x5597cd3f11c0 2024-04-25T16:01:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.180+0000 7fca3defa640 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5597cd3f11c0 0x5597cd883fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:01:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.180+0000 7fca217fa640 10 client.15687 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:55.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.180+0000 7fca3defa640 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5597cd3f11c0 0x5597cd883fd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1c1d770134a607e server_cookie=9204fe8bc3b599cb in_seq=0 out_seq=0 2024-04-25T16:01:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca217fa640 1 -- 172.21.15.31:0/4288913114 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5597cd76aee0 con 0x5597cd3f11c0 2024-04-25T16:01:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca217fa640 10 client.15687 handle_command_reply: tid=0 2024-04-25T16:01:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 client.15687 shutdown 2024-04-25T16:01:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 -- 172.21.15.31:0/4288913114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 msgr2=0x5597cd810c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd810c90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fca30006010 tx=0x7fca30005cb0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 -- 172.21.15.31:0/4288913114 shutdown_connections 2024-04-25T16:01:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5597cd3f11c0 0x5597cd883fd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597cd45d1b0 0x5597cd810c90 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597cd43c4d0 0x5597cd80fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 --2- 172.21.15.31:0/4288913114 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597cd4342f0 0x5597cd80f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca217fa640 0 client.15687 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 -- 172.21.15.31:0/4288913114 >> 172.21.15.31:0/4288913114 conn(0x5597cd451300 msgr2=0x5597cd80a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 -- 172.21.15.31:0/4288913114 shutdown_connections 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 1 -- 172.21.15.31:0/4288913114 wait complete. 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4700f740 20 client.15687 trim_cache size 0 max 16384 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:01:55.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:01:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4015e640 1 -- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca240afae0 msgr2=0x7fca240b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.188+0000 7fca4015e640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca240afae0 0x7fca240b1ff0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x5597cd802f50 tx=0x7fca300096a0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 -- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 msgr2=0x7fca3819a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:01:55.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 0x7fca3819a2a0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fca34006010 tx=0x7fca34031a50 comp rx=0 tx=0).stop 2024-04-25T16:01:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 -- 172.21.15.31:0/3815296703 shutdown_connections 2024-04-25T16:01:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca240afae0 0x7fca240b1ff0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca38108950 0x7fca381a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca38103c50 0x7fca3819a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 --2- 172.21.15.31:0/3815296703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca38102a10 0x7fca3819a2a0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:01:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 -- 172.21.15.31:0/3815296703 >> 172.21.15.31:0/3815296703 conn(0x7fca380fdc00 msgr2=0x7fca38109f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:01:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 -- 172.21.15.31:0/3815296703 shutdown_connections 2024-04-25T16:01:55.192 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:01:55.189+0000 7fca4015e640 1 -- 172.21.15.31:0/3815296703 wait complete. 2024-04-25T16:01:56.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:56 smithi063 ceph-mon[32413]: pgmap v370: 97 pgs: 97 active+clean; 663 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-25T16:01:56.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:56 smithi186 ceph-mon[33258]: pgmap v370: 97 pgs: 97 active+clean; 663 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-25T16:01:56.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:56 smithi031 ceph-mon[29140]: pgmap v370: 97 pgs: 97 active+clean; 663 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.3 KiB/s wr, 11 op/s 2024-04-25T16:01:58.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:01:58 smithi063 ceph-mon[32413]: pgmap v371: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:01:58.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:01:58 smithi186 ceph-mon[33258]: pgmap v371: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:01:58.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:01:58 smithi031 ceph-mon[29140]: pgmap v371: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:02:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:00 smithi063 ceph-mon[32413]: pgmap v372: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:02:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:00 smithi186 ceph-mon[33258]: pgmap v372: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:02:00.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:00 smithi031 ceph-mon[29140]: pgmap v372: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-25T16:02:02.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:02 smithi063 ceph-mon[32413]: pgmap v373: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-25T16:02:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:02 smithi186 ceph-mon[33258]: pgmap v373: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-25T16:02:02.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:02 smithi031 ceph-mon[29140]: pgmap v373: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-04-25T16:02:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:04 smithi063 ceph-mon[32413]: pgmap v374: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-04-25T16:02:04.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:04 smithi186 ceph-mon[33258]: pgmap v374: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-04-25T16:02:04.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:04 smithi031 ceph-mon[29140]: pgmap v374: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.6 KiB/s wr, 14 op/s 2024-04-25T16:02:06.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:06 smithi063 ceph-mon[32413]: pgmap v375: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-25T16:02:06.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:06 smithi186 ceph-mon[33258]: pgmap v375: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-25T16:02:06.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:06 smithi031 ceph-mon[29140]: pgmap v375: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.302+0000 7f2989d1f640 1 -- 172.21.15.31:0/3792250412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 msgr2=0x7f2984104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.302+0000 7f2989d1f640 1 --2- 172.21.15.31:0/3792250412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 0x7f2984104070 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2978009900 tx=0x7f297802a410 comp rx=0 tx=0).stop 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.302+0000 7f2989d1f640 1 -- 172.21.15.31:0/3792250412 shutdown_connections 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.302+0000 7f2989d1f640 1 --2- 172.21.15.31:0/3792250412 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984108960 0x7f298410ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.302+0000 7f2989d1f640 1 --2- 172.21.15.31:0/3792250412 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 0x7f2984104070 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.302+0000 7f2989d1f640 1 --2- 172.21.15.31:0/3792250412 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2984102a20 0x7f2984103720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.302+0000 7f2989d1f640 1 -- 172.21.15.31:0/3792250412 >> 172.21.15.31:0/3792250412 conn(0x7f29840fdc10 msgr2=0x7f29841000c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.302+0000 7f2989d1f640 1 -- 172.21.15.31:0/3792250412 shutdown_connections 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.302+0000 7f2989d1f640 1 -- 172.21.15.31:0/3792250412 wait complete. 2024-04-25T16:02:08.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.303+0000 7f2989d1f640 1 Processor -- start 2024-04-25T16:02:08.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.303+0000 7f2989d1f640 1 -- start start 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2989d1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2984102a20 0x7f298406ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2989d1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 0x7f298419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2989d1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984108960 0x7f29841a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2989d1f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f298410e520 con 0x7f2984103c60 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2989d1f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f298410e3a0 con 0x7f2984102a20 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2989d1f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f298410e6a0 con 0x7f2984108960 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2982ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 0x7f298419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2982ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 0x7f298419efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48038/0 (socket says 172.21.15.31:48038) 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2982ffd640 1 -- 172.21.15.31:0/673928364 learned_addr learned my addr 172.21.15.31:0/673928364 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f29837fe640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2984102a20 0x7f298406ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2983fff640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984108960 0x7f29841a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2982ffd640 1 -- 172.21.15.31:0/673928364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984108960 msgr2=0x7f29841a1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:08.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2982ffd640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984108960 0x7f29841a1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2982ffd640 1 -- 172.21.15.31:0/673928364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2984102a20 msgr2=0x7f298406ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2982ffd640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2984102a20 0x7f298406ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f2982ffd640 1 -- 172.21.15.31:0/673928364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29841a1d40 con 0x7f2984103c60 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.304+0000 7f29837fe640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2984102a20 0x7f298406ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.305+0000 7f2982ffd640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 0x7f298419efd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2978006010 tx=0x7f29780042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.305+0000 7f2980ff9640 1 -- 172.21.15.31:0/673928364 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f297803f030 con 0x7f2984103c60 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.305+0000 7f2989d1f640 1 -- 172.21.15.31:0/673928364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29841a1fd0 con 0x7f2984103c60 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.305+0000 7f2980ff9640 1 -- 172.21.15.31:0/673928364 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2978031910 con 0x7f2984103c60 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.305+0000 7f2980ff9640 1 -- 172.21.15.31:0/673928364 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978035070 con 0x7f2984103c60 2024-04-25T16:02:08.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.305+0000 7f2989d1f640 1 -- 172.21.15.31:0/673928364 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29841a2430 con 0x7f2984103c60 2024-04-25T16:02:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2990bcf740 1 Processor -- start 2024-04-25T16:02:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2980ff9640 1 -- 172.21.15.31:0/673928364 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2978042030 con 0x7f2984103c60 2024-04-25T16:02:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2990bcf740 1 -- start start 2024-04-25T16:02:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 0x55c0461a3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 0x55c0461a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045de24d0 0x55c0461aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2990bcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c046109460 con 0x55c045de24d0 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2990bcf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c0460c3590 con 0x55c045e031b0 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2990bcf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c046105420 con 0x55c045dda2f0 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2982ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045de24d0 0x55c0461aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2983fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 0x55c0461a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2982ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045de24d0 0x55c0461aadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48054/0 (socket says 172.21.15.31:48054) 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2982ffd640 1 -- 172.21.15.31:0/337297928 learned_addr learned my addr 172.21.15.31:0/337297928 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f29837fe640 1 --2- 172.21.15.31:0/337297928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 0x55c0461a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2980ff9640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f296c0afae0 0x7f296c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2982ffd640 1 -- 172.21.15.31:0/337297928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 msgr2=0x55c0461a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2982ffd640 1 --2- 172.21.15.31:0/337297928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 0x55c0461a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2982ffd640 1 -- 172.21.15.31:0/337297928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 msgr2=0x55c0461a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2982ffd640 1 --2- 172.21.15.31:0/337297928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 0x55c0461a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.308+0000 7f2982ffd640 1 -- 172.21.15.31:0/337297928 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0460d00d0 con 0x55c045de24d0 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.309+0000 7f2983fff640 1 --2- 172.21.15.31:0/337297928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 0x55c0461a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.309+0000 7f2983fff640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f296c0afae0 0x7f296c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.309+0000 7f2980ff9640 1 -- 172.21.15.31:0/673928364 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f29780ff0d0 con 0x7f2984103c60 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.309+0000 7f2982ffd640 1 --2- 172.21.15.31:0/337297928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045de24d0 0x55c0461aadf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f29780dec90 tx=0x7f29780c3530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3ae807fdc35bc90 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.309+0000 7f2969ffb640 1 -- 172.21.15.31:0/337297928 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2978049070 con 0x55c045de24d0 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.309+0000 7f2969ffb640 1 -- 172.21.15.31:0/337297928 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29780dd310 con 0x55c045de24d0 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.309+0000 7f2969ffb640 1 -- 172.21.15.31:0/337297928 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29780c6fc0 con 0x55c045de24d0 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2983fff640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f296c0afae0 0x7f296c0b1ff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55c0461a4b40 tx=0x7f297c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2990bcf740 1 -- 172.21.15.31:0/337297928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045de24d0 msgr2=0x55c0461aadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2990bcf740 1 --2- 172.21.15.31:0/337297928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045de24d0 0x55c0461aadf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f29780dec90 tx=0x7f29780c3530 comp rx=0 tx=0).stop 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2990bcf740 1 -- 172.21.15.31:0/337297928 shutdown_connections 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2990bcf740 1 --2- 172.21.15.31:0/337297928 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045de24d0 0x55c0461aadf0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2990bcf740 1 --2- 172.21.15.31:0/337297928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 0x55c0461a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2990bcf740 1 --2- 172.21.15.31:0/337297928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 0x55c0461a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2990bcf740 1 -- 172.21.15.31:0/337297928 >> 172.21.15.31:0/337297928 conn(0x55c045df7300 msgr2=0x55c04619f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2990bcf740 1 -- 172.21.15.31:0/337297928 shutdown_connections 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.310+0000 7f2990bcf740 1 -- 172.21.15.31:0/337297928 wait complete. 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f2990bcf740 1 -- start start 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f2990bcf740 1 Processor -- start 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dda2f0 0x55c0461ad390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 0x55c0461ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 0x55c0461b4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f2990bcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c0461ab600 con 0x55c045dda2f0 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f2990bcf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c046090b60 con 0x55c045e031b0 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f2990bcf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c0461ab780 con 0x55c045de24d0 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f2990bcf740 10 client.0 _ll_register_callbacks cb 0x55c0460b52b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f29837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dda2f0 0x55c0461ad390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f29837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dda2f0 0x55c0461ad390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48058/0 (socket says 172.21.15.31:48058) 2024-04-25T16:02:08.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.311+0000 7f29837fe640 1 -- 172.21.15.31:0/13108033 learned_addr learned my addr 172.21.15.31:0/13108033 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f2982ffd640 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 0x55c0461ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f2983fff640 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 0x55c0461b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f29837fe640 1 -- 172.21.15.31:0/13108033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 msgr2=0x55c0461ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f29837fe640 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 0x55c0461ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f29837fe640 1 -- 172.21.15.31:0/13108033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 msgr2=0x55c0461b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f29837fe640 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 0x55c0461b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f29837fe640 1 -- 172.21.15.31:0/13108033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c045e1b210 con 0x55c045dda2f0 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f29837fe640 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dda2f0 0x55c0461ad390 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2974012a10 tx=0x7f2974012ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=482def15dea06d59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f2962ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f2962ffd640 1 -- 172.21.15.31:0/13108033 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974004050 con 0x55c045dda2f0 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f2962ffd640 1 -- 172.21.15.31:0/13108033 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f29740041f0 con 0x55c045dda2f0 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f2990bcf740 1 -- 172.21.15.31:0/13108033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c046117070 con 0x55c045dda2f0 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f2962ffd640 1 -- 172.21.15.31:0/13108033 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f29740043f0 con 0x55c045dda2f0 2024-04-25T16:02:08.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f2990bcf740 10 client.15699 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.312+0000 7f2990bcf740 1 -- 172.21.15.31:0/13108033 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c045ee7620 con 0x55c045dda2f0 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.313+0000 7f2962ffd640 1 -- 172.21.15.31:0/13108033 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c046117070 con 0x55c045dda2f0 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.313+0000 7f2962ffd640 1 -- 172.21.15.31:0/13108033 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f2974003c70 con 0x55c045dda2f0 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.313+0000 7f2990bcf740 10 client.15699 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.313+0000 7f2990bcf740 10 client.15699 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.313+0000 7f2990bcf740 20 client.15699 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.313+0000 7f2990bcf740 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c045dc1d80 0x55c0461bb0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.313+0000 7f2990bcf740 4 client.15699 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.313+0000 7f2990bcf740 1 -- 172.21.15.31:0/13108033 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c046110ee0 con 0x55c045dc1d80 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.313+0000 7f2982ffd640 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c045dc1d80 0x55c0461bb0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.314+0000 7f2982ffd640 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c045dc1d80 0x55c0461bb0e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fd7226b0acec0f4 server_cookie=d82df8db83a5b02f in_seq=0 out_seq=0 2024-04-25T16:02:08.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.314+0000 7f2962ffd640 10 client.15699 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:08.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.314+0000 7f2962ffd640 1 -- 172.21.15.31:0/13108033 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c046110ee0 con 0x55c045dc1d80 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.314+0000 7f2962ffd640 10 client.15699 handle_command_reply: tid=0 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.314+0000 7f2990bcf740 1 client.15699 shutdown 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 -- 172.21.15.31:0/13108033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dda2f0 msgr2=0x55c0461ad390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dda2f0 0x55c0461ad390 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2974012a10 tx=0x7f2974012ed0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 -- 172.21.15.31:0/13108033 shutdown_connections 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c045dc1d80 0x55c0461bb0e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045e031b0 0x55c0461b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 0x55c0461ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 --2- 172.21.15.31:0/13108033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dda2f0 0x55c0461ad390 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2962ffd640 0 client.15699 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 -- 172.21.15.31:0/13108033 >> 172.21.15.31:0/13108033 conn(0x55c045df7300 msgr2=0x55c0461a2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 -- 172.21.15.31:0/13108033 shutdown_connections 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 1 -- 172.21.15.31:0/13108033 wait complete. 2024-04-25T16:02:08.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.315+0000 7f2990bcf740 20 client.15699 trim_cache size 0 max 16384 2024-04-25T16:02:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.330+0000 7f2990bcf740 1 Processor -- start 2024-04-25T16:02:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.330+0000 7f2990bcf740 1 -- start start 2024-04-25T16:02:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.330+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461a2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.330+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045dda2f0 0x55c0461b4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.330+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 0x55c04619ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.330+0000 7f2990bcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29841ab2c0 con 0x55c045dc1d80 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.330+0000 7f2990bcf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29841ace00 con 0x55c045dda2f0 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.330+0000 7f2990bcf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29841ab990 con 0x55c045de24d0 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.330+0000 7f29837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461a2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48060/0 (socket says 172.21.15.31:48060) 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29837fe640 1 -- 172.21.15.31:0/1295670220 learned_addr learned my addr 172.21.15.31:0/1295670220 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f2983fff640 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 0x55c04619ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f2982ffd640 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045dda2f0 0x55c0461b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29837fe640 1 -- 172.21.15.31:0/1295670220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 msgr2=0x55c04619ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29837fe640 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 0x55c04619ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29837fe640 1 -- 172.21.15.31:0/1295670220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045dda2f0 msgr2=0x55c0461b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29837fe640 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045dda2f0 0x55c0461b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29837fe640 1 -- 172.21.15.31:0/1295670220 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0461ab810 con 0x55c045dc1d80 2024-04-25T16:02:08.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f2983fff640 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 0x55c04619ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f2982ffd640 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045dda2f0 0x55c0461b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29837fe640 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461a2810 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2974000c00 tx=0x7f2974010a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f29a299103e50f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29697fa640 1 -- 172.21.15.31:0/1295670220 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974010aa0 con 0x55c045dc1d80 2024-04-25T16:02:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29697fa640 1 -- 172.21.15.31:0/1295670220 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f297400fe70 con 0x55c045dc1d80 2024-04-25T16:02:08.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.331+0000 7f29697fa640 1 -- 172.21.15.31:0/1295670220 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974004590 con 0x55c045dc1d80 2024-04-25T16:02:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.332+0000 7f2990bcf740 1 -- 172.21.15.31:0/1295670220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 msgr2=0x55c0461a2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.332+0000 7f2990bcf740 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461a2810 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f2974000c00 tx=0x7f2974010a30 comp rx=0 tx=0).stop 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.332+0000 7f2990bcf740 1 -- 172.21.15.31:0/1295670220 shutdown_connections 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.332+0000 7f2990bcf740 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045de24d0 0x55c04619ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.332+0000 7f2990bcf740 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045dda2f0 0x55c0461b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.332+0000 7f2990bcf740 1 --2- 172.21.15.31:0/1295670220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461a2810 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.332+0000 7f2990bcf740 1 -- 172.21.15.31:0/1295670220 >> 172.21.15.31:0/1295670220 conn(0x55c045df7300 msgr2=0x55c0461a2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.332+0000 7f2990bcf740 1 -- 172.21.15.31:0/1295670220 shutdown_connections 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.332+0000 7f2990bcf740 1 -- 172.21.15.31:0/1295670220 wait complete. 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2990bcf740 1 -- start start 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2990bcf740 1 Processor -- start 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461b2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 0x55c0461b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2990bcf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045de24d0 0x55c0461b3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2990bcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f29841ac730 con 0x55c045dc1d80 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2990bcf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f29841ac060 con 0x55c045de24d0 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2990bcf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29841a92a0 con 0x55c045dda2f0 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2990bcf740 10 client.0 _ll_register_callbacks cb 0x7f2974009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f29837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f29837fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:48076/0 (socket says 172.21.15.31:48076) 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f29837fe640 1 -- 172.21.15.31:0/614189482 learned_addr learned my addr 172.21.15.31:0/614189482 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.333+0000 7f2983fff640 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045de24d0 0x55c0461b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2982ffd640 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 0x55c0461b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f29837fe640 1 -- 172.21.15.31:0/614189482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 msgr2=0x55c0461b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f29837fe640 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 0x55c0461b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f29837fe640 1 -- 172.21.15.31:0/614189482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045de24d0 msgr2=0x55c0461b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f29837fe640 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045de24d0 0x55c0461b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f29837fe640 1 -- 172.21.15.31:0/614189482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0461b4c90 con 0x55c045dc1d80 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2982ffd640 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 0x55c0461b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2983fff640 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045de24d0 0x55c0461b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f29837fe640 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461b2190 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2974009730 tx=0x7f2974010670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9072172aa2d7d17e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2962ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2962ffd640 1 -- 172.21.15.31:0/614189482 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2974010ca0 con 0x55c045dc1d80 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2962ffd640 1 -- 172.21.15.31:0/614189482 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2974027450 con 0x55c045dc1d80 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2990bcf740 1 -- 172.21.15.31:0/614189482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f297400fe70 con 0x55c045dc1d80 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2962ffd640 1 -- 172.21.15.31:0/614189482 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2974010e40 con 0x55c045dc1d80 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2990bcf740 10 client.15705 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2990bcf740 1 -- 172.21.15.31:0/614189482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55c046223490 con 0x55c045dc1d80 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.334+0000 7f2962ffd640 1 -- 172.21.15.31:0/614189482 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f297400fe70 con 0x55c045dc1d80 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2962ffd640 1 -- 172.21.15.31:0/614189482 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f297402e020 con 0x55c045dc1d80 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2990bcf740 10 client.15705 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2990bcf740 10 client.15705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2990bcf740 20 client.15705 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2990bcf740 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c045e031b0 0x55c046225e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2990bcf740 4 client.15705 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:02:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2990bcf740 1 -- 172.21.15.31:0/614189482 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c04608c5f0 con 0x55c045e031b0 2024-04-25T16:02:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2982ffd640 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c045e031b0 0x55c046225e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2982ffd640 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c045e031b0 0x55c046225e50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65b8e34303beaaf4 server_cookie=9e02fc973420f7eb in_seq=0 out_seq=0 2024-04-25T16:02:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.335+0000 7f2962ffd640 10 client.15705 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2962ffd640 1 -- 172.21.15.31:0/614189482 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c04608c5f0 con 0x55c045e031b0 2024-04-25T16:02:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2962ffd640 10 client.15705 handle_command_reply: tid=0 2024-04-25T16:02:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 client.15705 shutdown 2024-04-25T16:02:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 -- 172.21.15.31:0/614189482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 msgr2=0x55c0461b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461b2190 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2974009730 tx=0x7f2974010670 comp rx=0 tx=0).stop 2024-04-25T16:02:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 -- 172.21.15.31:0/614189482 shutdown_connections 2024-04-25T16:02:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c045e031b0 0x55c046225e50 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c045de24d0 0x55c0461b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c045dda2f0 0x55c0461b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2962ffd640 0 client.15705 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 --2- 172.21.15.31:0/614189482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c045dc1d80 0x55c0461b2190 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 -- 172.21.15.31:0/614189482 >> 172.21.15.31:0/614189482 conn(0x55c045df7300 msgr2=0x55c0461aced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 -- 172.21.15.31:0/614189482 shutdown_connections 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 1 -- 172.21.15.31:0/614189482 wait complete. 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.340+0000 7f2990bcf740 20 client.15705 trim_cache size 0 max 16384 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:02:08.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:02:08.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 -- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f296c0afae0 msgr2=0x7f296c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:08.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f296c0afae0 0x7f296c0b1ff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x55c0461a4b40 tx=0x7f297c004a40 comp rx=0 tx=0).stop 2024-04-25T16:02:08.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 -- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 msgr2=0x7f298419efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:08.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 0x7f298419efd0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f2978006010 tx=0x7f29780042f0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 -- 172.21.15.31:0/673928364 shutdown_connections 2024-04-25T16:02:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f296c0afae0 0x7f296c0b1ff0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2984108960 0x7f29841a1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2984103c60 0x7f298419efd0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 --2- 172.21.15.31:0/673928364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2984102a20 0x7f298406ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 -- 172.21.15.31:0/673928364 >> 172.21.15.31:0/673928364 conn(0x7f29840fdc10 msgr2=0x7f2984104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 -- 172.21.15.31:0/673928364 shutdown_connections 2024-04-25T16:02:08.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:08.341+0000 7f2989d1f640 1 -- 172.21.15.31:0/673928364 wait complete. 2024-04-25T16:02:08.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:08 smithi063 ceph-mon[32413]: pgmap v376: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 12 op/s 2024-04-25T16:02:08.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:08 smithi186 ceph-mon[33258]: pgmap v376: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 12 op/s 2024-04-25T16:02:08.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:08 smithi031 ceph-mon[29140]: pgmap v376: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 12 op/s 2024-04-25T16:02:10.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:10 smithi063 ceph-mon[32413]: pgmap v377: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T16:02:10.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:10 smithi186 ceph-mon[33258]: pgmap v377: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T16:02:10.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:10 smithi031 ceph-mon[29140]: pgmap v377: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T16:02:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:12 smithi063 ceph-mon[32413]: pgmap v378: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-25T16:02:12.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:12 smithi186 ceph-mon[33258]: pgmap v378: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-25T16:02:12.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:12 smithi031 ceph-mon[29140]: pgmap v378: 97 pgs: 97 active+clean; 664 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-25T16:02:14.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:14 smithi063 ceph-mon[32413]: pgmap v379: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T16:02:14.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:14 smithi186 ceph-mon[33258]: pgmap v379: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T16:02:14.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:14 smithi031 ceph-mon[29140]: pgmap v379: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T16:02:16.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:16 smithi063 ceph-mon[32413]: pgmap v380: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:02:16.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:16 smithi186 ceph-mon[33258]: pgmap v380: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:02:16.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:16 smithi031 ceph-mon[29140]: pgmap v380: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:02:18.734 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:02:18.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:02:18.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:18 smithi063 ceph-mon[32413]: pgmap v381: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.884+0000 7f38ba251640 1 -- 172.21.15.31:0/4020093874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4106c10 msgr2=0x7f38b41090f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.884+0000 7f38ba251640 1 --2- 172.21.15.31:0/4020093874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4106c10 0x7f38b41090f0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f38a8009900 tx=0x7f38a802a410 comp rx=0 tx=0).stop 2024-04-25T16:02:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.884+0000 7f38ba251640 1 -- 172.21.15.31:0/4020093874 shutdown_connections 2024-04-25T16:02:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.884+0000 7f38ba251640 1 --2- 172.21.15.31:0/4020093874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38b4109780 0x7f38b410bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.884+0000 7f38ba251640 1 --2- 172.21.15.31:0/4020093874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4106c10 0x7f38b41090f0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.884+0000 7f38ba251640 1 --2- 172.21.15.31:0/4020093874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38b4101e10 0x7f38b4102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.884+0000 7f38ba251640 1 -- 172.21.15.31:0/4020093874 >> 172.21.15.31:0/4020093874 conn(0x7f38b40fd340 msgr2=0x7f38b40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.884+0000 7f38ba251640 1 -- 172.21.15.31:0/4020093874 shutdown_connections 2024-04-25T16:02:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.884+0000 7f38ba251640 1 -- 172.21.15.31:0/4020093874 wait complete. 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38ba251640 1 Processor -- start 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38ba251640 1 -- start start 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38ba251640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4101e10 0x7f38b419a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38ba251640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38b4106c10 0x7f38b419a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38ba251640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38b4109780 0x7f38b41a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38ba251640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38b410f610 con 0x7f38b4101e10 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38ba251640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f38b410f490 con 0x7f38b4106c10 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38ba251640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f38b410f790 con 0x7f38b4109780 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4101e10 0x7f38b419a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38b2ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38b4106c10 0x7f38b419a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4101e10 0x7f38b419a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37616/0 (socket says 172.21.15.31:37616) 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38b2ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38b4106c10 0x7f38b419a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49092/0 (socket says 172.21.15.31:49092) 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38b37fe640 1 -- 172.21.15.31:0/873679390 learned_addr learned my addr 172.21.15.31:0/873679390 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.885+0000 7f38b3fff640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38b4109780 0x7f38b41a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b3fff640 1 -- 172.21.15.31:0/873679390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38b4106c10 msgr2=0x7f38b419a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b3fff640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38b4106c10 0x7f38b419a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b3fff640 1 -- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4101e10 msgr2=0x7f38b419a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b3fff640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4101e10 0x7f38b419a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b3fff640 1 -- 172.21.15.31:0/873679390 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b41a1c80 con 0x7f38b4109780 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38b4106c10 0x7f38b419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b37fe640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4101e10 0x7f38b419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b3fff640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38b4109780 0x7f38b41a16c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009fd0 tx=0x7f38ac00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b0ff9640 1 -- 172.21.15.31:0/873679390 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac00ce20 con 0x7f38b4109780 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b0ff9640 1 -- 172.21.15.31:0/873679390 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38ac005020 con 0x7f38b4109780 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38b0ff9640 1 -- 172.21.15.31:0/873679390 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac010690 con 0x7f38b4109780 2024-04-25T16:02:18.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38ba251640 1 -- 172.21.15.31:0/873679390 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b41a1f70 con 0x7f38b4109780 2024-04-25T16:02:18.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.886+0000 7f38ba251640 1 -- 172.21.15.31:0/873679390 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b41a2320 con 0x7f38b4109780 2024-04-25T16:02:18.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.889+0000 7f38b0ff9640 1 -- 172.21.15.31:0/873679390 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f38ac0054d0 con 0x7f38b4109780 2024-04-25T16:02:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.890+0000 7f38c11e3740 1 Processor -- start 2024-04-25T16:02:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.890+0000 7f38c11e3740 1 -- start start 2024-04-25T16:02:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.890+0000 7f38b0ff9640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f389c0afb70 0x7f389c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.890+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a6ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.890+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 0x55d027a6cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.890+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a6dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.890+0000 7f38c11e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d0279d1d80 con 0x55d0276aa6b0 2024-04-25T16:02:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.890+0000 7f38c11e3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d02798b8b0 con 0x55d0276cb390 2024-04-25T16:02:18.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.890+0000 7f38c11e3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d0279cdd40 con 0x55d0276a24d0 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f389c0afb70 0x7f389c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b0ff9640 1 -- 172.21.15.31:0/873679390 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f38ac0df320 con 0x7f38b4109780 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a6ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a6ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49100/0 (socket says 172.21.15.31:49100) 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b37fe640 1 -- 172.21.15.31:0/1645027021 learned_addr learned my addr 172.21.15.31:0/1645027021 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b3fff640 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a6dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 0x55d027a6cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b37fe640 1 -- 172.21.15.31:0/1645027021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 msgr2=0x55d027a6cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b37fe640 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 0x55d027a6cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b37fe640 1 -- 172.21.15.31:0/1645027021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 msgr2=0x55d027a6dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b37fe640 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a6dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b37fe640 1 -- 172.21.15.31:0/1645027021 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d02795ca40 con 0x55d0276cb390 2024-04-25T16:02:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b3fff640 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a6dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:02:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f389c0afb70 0x7f389c0b2080 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f38a8009230 tx=0x7f38a8031890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.891+0000 7f38b37fe640 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a6ca60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f38a400f4c0 tx=0x7f38a400f8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e695497d0bceed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.892+0000 7f3899ffb640 1 -- 172.21.15.31:0/1645027021 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4005220 con 0x55d0276cb390 2024-04-25T16:02:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.892+0000 7f3899ffb640 1 -- 172.21.15.31:0/1645027021 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38a40047a0 con 0x55d0276cb390 2024-04-25T16:02:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.892+0000 7f3899ffb640 1 -- 172.21.15.31:0/1645027021 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4010ab0 con 0x55d0276cb390 2024-04-25T16:02:18.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 -- 172.21.15.31:0/1645027021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 msgr2=0x55d027a6ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:18.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a6ca60 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f38a400f4c0 tx=0x7f38a400f8c0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 -- 172.21.15.31:0/1645027021 shutdown_connections 2024-04-25T16:02:18.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a6dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 0x55d027a6cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1645027021 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a6ca60 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 -- 172.21.15.31:0/1645027021 >> 172.21.15.31:0/1645027021 conn(0x55d0276bf4e0 msgr2=0x55d027a68270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 -- 172.21.15.31:0/1645027021 shutdown_connections 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 -- 172.21.15.31:0/1645027021 wait complete. 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.893+0000 7f38c11e3740 1 -- start start 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38c11e3740 1 Processor -- start 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 0x55d027a724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a729f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a738d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38c11e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d027a6e780 con 0x55d0276aa6b0 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38c11e3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d027958d10 con 0x55d0276cb390 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38c11e3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d027a6e900 con 0x55d0276a24d0 2024-04-25T16:02:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38b2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38b2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a729f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37640/0 (socket says 172.21.15.31:37640) 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38b2ffd640 1 -- 172.21.15.31:0/2894622495 learned_addr learned my addr 172.21.15.31:0/2894622495 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.894+0000 7f38b37fe640 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 0x55d027a724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38b3fff640 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a738d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38b2ffd640 1 -- 172.21.15.31:0/2894622495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 msgr2=0x55d027a724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 0x55d027a724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38b2ffd640 1 -- 172.21.15.31:0/2894622495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 msgr2=0x55d027a738d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a738d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38b2ffd640 1 -- 172.21.15.31:0/2894622495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0277ad7d0 con 0x55d0276aa6b0 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38b3fff640 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a729f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f38a8031500 tx=0x7f38a802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb81993424fa750 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38c11e3740 10 client.0 _ll_register_callbacks cb 0x55d027985210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f3892ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f38c11e3740 1 -- 172.21.15.31:0/2894622495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38a40047a0 con 0x55d0276aa6b0 2024-04-25T16:02:18.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f3892ffd640 1 -- 172.21.15.31:0/2894622495 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a8035050 con 0x55d0276aa6b0 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f3892ffd640 1 -- 172.21.15.31:0/2894622495 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f38a8003b80 con 0x55d0276aa6b0 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.895+0000 7f3892ffd640 1 -- 172.21.15.31:0/2894622495 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f38a8003d80 con 0x55d0276aa6b0 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f38c11e3740 10 client.15714 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f38c11e3740 1 -- 172.21.15.31:0/2894622495 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55d0276e33f0 con 0x55d0276aa6b0 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f3892ffd640 1 -- 172.21.15.31:0/2894622495 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f38a804a3f0 con 0x55d0276aa6b0 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f38c11e3740 10 client.15714 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f38c11e3740 10 client.15714 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f38c11e3740 20 client.15714 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f38c11e3740 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d027689f60 0x55d027a8c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f38c11e3740 4 client.15714 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f38c11e3740 1 -- 172.21.15.31:0/2894622495 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0279df990 con 0x55d027689f60 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.896+0000 7f38b37fe640 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d027689f60 0x55d027a8c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.897+0000 7f3892ffd640 10 client.15714 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:18.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.897+0000 7f38b37fe640 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d027689f60 0x55d027a8c750 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98901a62572a7002 server_cookie=9c723e782f00245a in_seq=0 out_seq=0 2024-04-25T16:02:18.901 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:18 smithi186 ceph-mon[33258]: pgmap v381: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:18.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.897+0000 7f3892ffd640 1 -- 172.21.15.31:0/2894622495 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d0279df990 con 0x55d027689f60 2024-04-25T16:02:18.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.897+0000 7f3892ffd640 10 client.15714 handle_command_reply: tid=0 2024-04-25T16:02:18.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 client.15714 shutdown 2024-04-25T16:02:18.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 -- 172.21.15.31:0/2894622495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 msgr2=0x55d027a729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:18.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a729f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f38a8031500 tx=0x7f38a802a410 comp rx=0 tx=0).stop 2024-04-25T16:02:18.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 -- 172.21.15.31:0/2894622495 shutdown_connections 2024-04-25T16:02:18.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d027689f60 0x55d027a8c750 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276cb390 0x55d027a738d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276aa6b0 0x55d027a729f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 --2- 172.21.15.31:0/2894622495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276a24d0 0x55d027a724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 -- 172.21.15.31:0/2894622495 >> 172.21.15.31:0/2894622495 conn(0x55d0276bf4e0 msgr2=0x55d027a6b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:18.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f3892ffd640 0 client.15714 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:18.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 -- 172.21.15.31:0/2894622495 shutdown_connections 2024-04-25T16:02:18.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 1 -- 172.21.15.31:0/2894622495 wait complete. 2024-04-25T16:02:18.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.898+0000 7f38c11e3740 20 client.15714 trim_cache size 0 max 16384 2024-04-25T16:02:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.933+0000 7f38c11e3740 1 Processor -- start 2024-04-25T16:02:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.933+0000 7f38c11e3740 1 -- start start 2024-04-25T16:02:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.933+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276a24d0 0x55d027a77e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.933+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 0x55d027a6c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d02765f3a0 0x55d027a6b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38c11e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38b41a9020 con 0x55d0276a24d0 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38c11e3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f38b41aab60 con 0x55d02765f3a0 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38c11e3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f38b41a96f0 con 0x55d0276cb390 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276a24d0 0x55d027a77e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276a24d0 0x55d027a77e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37650/0 (socket says 172.21.15.31:37650) 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b37fe640 1 -- 172.21.15.31:0/1885019927 learned_addr learned my addr 172.21.15.31:0/1885019927 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b3fff640 1 --2- 172.21.15.31:0/1885019927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d02765f3a0 0x55d027a6b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/1885019927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 0x55d027a6c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b37fe640 1 -- 172.21.15.31:0/1885019927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 msgr2=0x55d027a6c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b37fe640 1 --2- 172.21.15.31:0/1885019927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 0x55d027a6c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b37fe640 1 -- 172.21.15.31:0/1885019927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d02765f3a0 msgr2=0x55d027a6b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b37fe640 1 --2- 172.21.15.31:0/1885019927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d02765f3a0 0x55d027a6b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b37fe640 1 -- 172.21.15.31:0/1885019927 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0276a30f0 con 0x55d0276a24d0 2024-04-25T16:02:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.934+0000 7f38b37fe640 1 --2- 172.21.15.31:0/1885019927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276a24d0 0x55d027a77e20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f38a4010010 tx=0x7f38a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cedb2cf4f35fc27e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.935+0000 7f38997fa640 1 -- 172.21.15.31:0/1885019927 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4004950 con 0x55d0276a24d0 2024-04-25T16:02:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.935+0000 7f38997fa640 1 -- 172.21.15.31:0/1885019927 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38a4010700 con 0x55d0276a24d0 2024-04-25T16:02:18.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.935+0000 7f38997fa640 1 -- 172.21.15.31:0/1885019927 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a40035f0 con 0x55d0276a24d0 2024-04-25T16:02:18.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.935+0000 7f38c11e3740 1 -- 172.21.15.31:0/1885019927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276a24d0 msgr2=0x55d027a77e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:18.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1885019927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276a24d0 0x55d027a77e20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f38a4010010 tx=0x7f38a4004730 comp rx=0 tx=0).stop 2024-04-25T16:02:18.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 -- 172.21.15.31:0/1885019927 shutdown_connections 2024-04-25T16:02:18.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1885019927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d02765f3a0 0x55d027a6b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1885019927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 0x55d027a6c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1885019927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d0276a24d0 0x55d027a77e20 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 -- 172.21.15.31:0/1885019927 >> 172.21.15.31:0/1885019927 conn(0x55d0276bf4e0 msgr2=0x55d027a6b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 -- 172.21.15.31:0/1885019927 shutdown_connections 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 -- 172.21.15.31:0/1885019927 wait complete. 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 -- start start 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.936+0000 7f38c11e3740 1 Processor -- start 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d02765f3a0 0x55d027a842e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276a24d0 0x55d027a84820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38c11e3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 0x55d027af2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38c11e3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f38b41aa490 con 0x55d02765f3a0 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38c11e3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f38b41a9dc0 con 0x55d0276a24d0 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38c11e3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f38b41a7000 con 0x55d0276cb390 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d02765f3a0 0x55d027a842e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d02765f3a0 0x55d027a842e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37652/0 (socket says 172.21.15.31:37652) 2024-04-25T16:02:18.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b37fe640 1 -- 172.21.15.31:0/1565171544 learned_addr learned my addr 172.21.15.31:0/1565171544 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38c11e3740 10 client.0 _ll_register_callbacks cb 0x7f38a8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b3fff640 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 0x55d027af2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b37fe640 1 -- 172.21.15.31:0/1565171544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 msgr2=0x55d027af2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b37fe640 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 0x55d027af2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b37fe640 1 -- 172.21.15.31:0/1565171544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276a24d0 msgr2=0x55d027a84820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b37fe640 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276a24d0 0x55d027a84820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b37fe640 1 -- 172.21.15.31:0/1565171544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0279924b0 con 0x55d02765f3a0 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f38b37fe640 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d02765f3a0 0x55d027a842e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f38a4012010 tx=0x7f38a4004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61ad083dc769a24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.937+0000 7f3892ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f3892ffd640 1 -- 172.21.15.31:0/1565171544 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a403d060 con 0x55d02765f3a0 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f3892ffd640 1 -- 172.21.15.31:0/1565171544 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f38a40042b0 con 0x55d02765f3a0 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f38c11e3740 1 -- 172.21.15.31:0/1565171544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38a4010700 con 0x55d02765f3a0 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f3892ffd640 1 -- 172.21.15.31:0/1565171544 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f38a40108b0 con 0x55d02765f3a0 2024-04-25T16:02:18.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f38c11e3740 10 client.15720 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f38c11e3740 1 -- 172.21.15.31:0/1565171544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d027af3c60 con 0x55d02765f3a0 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f3892ffd640 1 -- 172.21.15.31:0/1565171544 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38a4010700 con 0x55d02765f3a0 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f3892ffd640 1 -- 172.21.15.31:0/1565171544 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f38a4003ce0 con 0x55d02765f3a0 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f38c11e3740 10 client.15720 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f38c11e3740 10 client.15720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f38c11e3740 20 client.15720 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.938+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d027af4410 0x55d027af6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.939+0000 7f38c11e3740 4 client.15720 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.939+0000 7f38c11e3740 1 -- 172.21.15.31:0/1565171544 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d0279d9800 con 0x55d027af4410 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.939+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d027af4410 0x55d027af6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.939+0000 7f3892ffd640 10 client.15720 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.939+0000 7f38b2ffd640 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d027af4410 0x55d027af6830 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d270ac8ffce2f9f9 server_cookie=55c3d42e00303dde in_seq=0 out_seq=0 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f3892ffd640 1 -- 172.21.15.31:0/1565171544 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d0279d9800 con 0x55d027af4410 2024-04-25T16:02:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f3892ffd640 10 client.15720 handle_command_reply: tid=0 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 client.15720 shutdown 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 -- 172.21.15.31:0/1565171544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d02765f3a0 msgr2=0x55d027a842e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d02765f3a0 0x55d027a842e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f38a4012010 tx=0x7f38a4004950 comp rx=0 tx=0).stop 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 -- 172.21.15.31:0/1565171544 shutdown_connections 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d027af4410 0x55d027af6830 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d0276cb390 0x55d027af2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d0276a24d0 0x55d027a84820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f3892ffd640 0 client.15720 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 --2- 172.21.15.31:0/1565171544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d02765f3a0 0x55d027a842e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 -- 172.21.15.31:0/1565171544 >> 172.21.15.31:0/1565171544 conn(0x55d0276bf4e0 msgr2=0x55d027a70ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 -- 172.21.15.31:0/1565171544 shutdown_connections 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 1 -- 172.21.15.31:0/1565171544 wait complete. 2024-04-25T16:02:18.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.940+0000 7f38c11e3740 20 client.15720 trim_cache size 0 max 16384 2024-04-25T16:02:18.946 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:02:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:02:18.946 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:02:18.946 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 -- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f389c0afb70 msgr2=0x7f389c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f389c0afb70 0x7f389c0b2080 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f38a8009230 tx=0x7f38a8031890 comp rx=0 tx=0).stop 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 -- 172.21.15.31:0/873679390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38b4109780 msgr2=0x7f38b41a16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38b4109780 0x7f38b41a16c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009fd0 tx=0x7f38ac00ef10 comp rx=0 tx=0).stop 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 -- 172.21.15.31:0/873679390 shutdown_connections 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f389c0afb70 0x7f389c0b2080 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38b4109780 0x7f38b41a16c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f38b4106c10 0x7f38b419a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 --2- 172.21.15.31:0/873679390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f38b4101e10 0x7f38b419a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 -- 172.21.15.31:0/873679390 >> 172.21.15.31:0/873679390 conn(0x7f38b40fd340 msgr2=0x7f38b4107980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 -- 172.21.15.31:0/873679390 shutdown_connections 2024-04-25T16:02:18.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:18.941+0000 7f38ba251640 1 -- 172.21.15.31:0/873679390 wait complete. 2024-04-25T16:02:18.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:18 smithi031 ceph-mon[29140]: pgmap v381: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:18.953 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:02:18.953 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:02:19.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.078+0000 7f4d2836c640 1 -- 172.21.15.31:0/1902393798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d20108910 msgr2=0x7f4d2010ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.078+0000 7f4d2836c640 1 --2- 172.21.15.31:0/1902393798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d20108910 0x7f4d2010ad30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009900 tx=0x7f4d1802a410 comp rx=0 tx=0).stop 2024-04-25T16:02:19.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.078+0000 7f4d2836c640 1 -- 172.21.15.31:0/1902393798 shutdown_connections 2024-04-25T16:02:19.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.078+0000 7f4d2836c640 1 --2- 172.21.15.31:0/1902393798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d20108910 0x7f4d2010ad30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.078+0000 7f4d2836c640 1 --2- 172.21.15.31:0/1902393798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d20103c10 0x7f4d20104020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.078+0000 7f4d2836c640 1 --2- 172.21.15.31:0/1902393798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d201029d0 0x7f4d201036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.078+0000 7f4d2836c640 1 -- 172.21.15.31:0/1902393798 >> 172.21.15.31:0/1902393798 conn(0x7f4d200fdba0 msgr2=0x7f4d20100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.078+0000 7f4d2836c640 1 -- 172.21.15.31:0/1902393798 shutdown_connections 2024-04-25T16:02:19.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.079+0000 7f4d2836c640 1 -- 172.21.15.31:0/1902393798 wait complete. 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.079+0000 7f4d2836c640 1 Processor -- start 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.079+0000 7f4d2836c640 1 -- start start 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d2836c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d201029d0 0x7f4d20199080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d2836c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d20103c10 0x7f4d201995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d2836c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d20108910 0x7f4d201a0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d2836c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2010e4e0 con 0x7f4d20103c10 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d2836c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2010e360 con 0x7f4d201029d0 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d2836c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2010e660 con 0x7f4d20108910 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d25907640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d20103c10 0x7f4d201995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d25907640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d20103c10 0x7f4d201995c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37662/0 (socket says 172.21.15.31:37662) 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d25907640 1 -- 172.21.15.31:0/217214638 learned_addr learned my addr 172.21.15.31:0/217214638 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d26108640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d201029d0 0x7f4d20199080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d26909640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d20108910 0x7f4d201a0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d25907640 1 -- 172.21.15.31:0/217214638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d20108910 msgr2=0x7f4d201a0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d25907640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d20108910 0x7f4d201a0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d25907640 1 -- 172.21.15.31:0/217214638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d201029d0 msgr2=0x7f4d20199080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d25907640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d201029d0 0x7f4d20199080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d25907640 1 -- 172.21.15.31:0/217214638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d201a0bc0 con 0x7f4d20103c10 2024-04-25T16:02:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d26909640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d20108910 0x7f4d201a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:02:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.080+0000 7f4d26108640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d201029d0 0x7f4d20199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.081+0000 7f4d25907640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d20103c10 0x7f4d201995c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000e9f0 tx=0x7f4d1000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.081+0000 7f4d177fe640 1 -- 172.21.15.31:0/217214638 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1000cd80 con 0x7f4d20103c10 2024-04-25T16:02:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.081+0000 7f4d177fe640 1 -- 172.21.15.31:0/217214638 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d100108e0 con 0x7f4d20103c10 2024-04-25T16:02:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.081+0000 7f4d177fe640 1 -- 172.21.15.31:0/217214638 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d10018980 con 0x7f4d20103c10 2024-04-25T16:02:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.081+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d201a0eb0 con 0x7f4d20103c10 2024-04-25T16:02:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.081+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d201a12b0 con 0x7f4d20103c10 2024-04-25T16:02:19.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.083+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d20107720 con 0x7f4d20103c10 2024-04-25T16:02:19.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.085+0000 7f4d177fe640 1 -- 172.21.15.31:0/217214638 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4d10010430 con 0x7f4d20103c10 2024-04-25T16:02:19.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.086+0000 7f4d177fe640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d0c0afbf0 0x7f4d0c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.086+0000 7f4d177fe640 1 -- 172.21.15.31:0/217214638 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4d100df3f0 con 0x7f4d20103c10 2024-04-25T16:02:19.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.086+0000 7f4d26108640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d0c0afbf0 0x7f4d0c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.087+0000 7f4d26108640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d0c0afbf0 0x7f4d0c0b2100 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d201035d0 tx=0x7f4d1c009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.093+0000 7f4d177fe640 1 -- 172.21.15.31:0/217214638 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d100a7f80 con 0x7f4d20103c10 2024-04-25T16:02:19.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.371+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4d2002dda0 con 0x7f4d20103c10 2024-04-25T16:02:19.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.371+0000 7f4d177fe640 1 -- 172.21.15.31:0/217214638 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11917 (secure 0 0 0) 0x7f4d100a7880 con 0x7f4d20103c10 2024-04-25T16:02:19.372 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:02:19.372 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:01:09.454389+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24535,24521,24563,14631,24553]},"id":1}]} 2024-04-25T16:02:19.373 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-25T16:02:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d0c0afbf0 msgr2=0x7f4d0c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d0c0afbf0 0x7f4d0c0b2100 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d201035d0 tx=0x7f4d1c009720 comp rx=0 tx=0).stop 2024-04-25T16:02:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d20103c10 msgr2=0x7f4d201995c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d20103c10 0x7f4d201995c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f4d1000e9f0 tx=0x7f4d1000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 shutdown_connections 2024-04-25T16:02:19.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d0c0afbf0 0x7f4d0c0b2100 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d20108910 0x7f4d201a0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d20103c10 0x7f4d201995c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 --2- 172.21.15.31:0/217214638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d201029d0 0x7f4d20199080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.374+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 >> 172.21.15.31:0/217214638 conn(0x7f4d200fdba0 msgr2=0x7f4d201048f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.375+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 shutdown_connections 2024-04-25T16:02:19.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.375+0000 7f4d2836c640 1 -- 172.21.15.31:0/217214638 wait complete. 2024-04-25T16:02:19.387 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:02:19.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.502+0000 7f7d5f2af640 1 -- 172.21.15.31:0/1995425154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d58103c20 msgr2=0x7f7d58104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.502+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/1995425154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d58103c20 0x7f7d58104030 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009900 tx=0x7f7d4802a410 comp rx=0 tx=0).stop 2024-04-25T16:02:19.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.502+0000 7f7d5f2af640 1 -- 172.21.15.31:0/1995425154 shutdown_connections 2024-04-25T16:02:19.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.502+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/1995425154 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d58108920 0x7f7d5810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.502+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/1995425154 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d58103c20 0x7f7d58104030 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.502+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/1995425154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d581029e0 0x7f7d581036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.502+0000 7f7d5f2af640 1 -- 172.21.15.31:0/1995425154 >> 172.21.15.31:0/1995425154 conn(0x7f7d580fdbd0 msgr2=0x7f7d58100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.502+0000 7f7d5f2af640 1 -- 172.21.15.31:0/1995425154 shutdown_connections 2024-04-25T16:02:19.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.502+0000 7f7d5f2af640 1 -- 172.21.15.31:0/1995425154 wait complete. 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5f2af640 1 Processor -- start 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5f2af640 1 -- start start 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5f2af640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d581029e0 0x7f7d5819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5f2af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d58103c20 0x7f7d5819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5f2af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d58108920 0x7f7d581a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5f2af640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810e500 con 0x7f7d58108920 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5f2af640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810e380 con 0x7f7d581029e0 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5f2af640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d5810e680 con 0x7f7d58103c20 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5d04b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d581029e0 0x7f7d5819a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5d84c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d58108920 0x7f7d581a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5d04b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d581029e0 0x7f7d5819a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49176/0 (socket says 172.21.15.31:49176) 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5d04b640 1 -- 172.21.15.31:0/2211825843 learned_addr learned my addr 172.21.15.31:0/2211825843 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.503+0000 7f7d5c84a640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d58103c20 0x7f7d5819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d5d04b640 1 -- 172.21.15.31:0/2211825843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d58103c20 msgr2=0x7f7d5819aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d58103c20 0x7f7d5819aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d5d04b640 1 -- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d58108920 msgr2=0x7f7d581a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d58108920 0x7f7d581a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d5d04b640 1 -- 172.21.15.31:0/2211825843 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d581a2090 con 0x7f7d581029e0 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d58108920 0x7f7d581a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d581029e0 0x7f7d5819a550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400e9f0 tx=0x7f7d5400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d4e7fc640 1 -- 172.21.15.31:0/2211825843 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5400ccc0 con 0x7f7d581029e0 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d4e7fc640 1 -- 172.21.15.31:0/2211825843 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d540035f0 con 0x7f7d581029e0 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d5f2af640 1 -- 172.21.15.31:0/2211825843 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d58108d90 con 0x7f7d581029e0 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.504+0000 7f7d4e7fc640 1 -- 172.21.15.31:0/2211825843 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5400ce60 con 0x7f7d581029e0 2024-04-25T16:02:19.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.505+0000 7f7d5f2af640 1 -- 172.21.15.31:0/2211825843 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d58109290 con 0x7f7d581029e0 2024-04-25T16:02:19.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.507+0000 7f7d4e7fc640 1 -- 172.21.15.31:0/2211825843 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7d54003aa0 con 0x7f7d581029e0 2024-04-25T16:02:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.508+0000 7f7d66163740 1 Processor -- start 2024-04-25T16:02:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.508+0000 7f7d66163740 1 -- start start 2024-04-25T16:02:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.508+0000 7f7d4e7fc640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7d440afaf0 0x7f7d440b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.508+0000 7f7d66163740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecedb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.508+0000 7f7d66163740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb104d0 0x5583ecedbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.508+0000 7f7d66163740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb186b0 0x5583ecedca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.508+0000 7f7d66163740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5583ece3fd80 con 0x5583ecb39390 2024-04-25T16:02:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.508+0000 7f7d66163740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5583ecdf9a20 con 0x5583ecb186b0 2024-04-25T16:02:19.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.508+0000 7f7d66163740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5583ece3bd40 con 0x5583ecb104d0 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7d440afaf0 0x7f7d440b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d4e7fc640 1 -- 172.21.15.31:0/2211825843 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7d540def40 con 0x7f7d581029e0 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d04b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecedb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5c84a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb104d0 0x5583ecedbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d04b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecedb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37698/0 (socket says 172.21.15.31:37698) 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5c84a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb104d0 0x5583ecedbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48438/0 (socket says 172.21.15.31:48438) 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d04b640 1 -- 172.21.15.31:0/553411850 learned_addr learned my addr 172.21.15.31:0/553411850 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb186b0 0x5583ecedca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d04b640 1 -- 172.21.15.31:0/553411850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb104d0 msgr2=0x5583ecedbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb104d0 0x5583ecedbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d04b640 1 -- 172.21.15.31:0/553411850 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb186b0 msgr2=0x5583ecedca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb186b0 0x5583ecedca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d04b640 1 -- 172.21.15.31:0/553411850 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583ece04c30 con 0x5583ecb39390 2024-04-25T16:02:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb186b0 0x5583ecedca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.509+0000 7f7d5c84a640 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb104d0 0x5583ecedbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.510+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecedb810 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7d540a73c0 tx=0x7f7d540a6f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33baa4357638da5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.510+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7d440afaf0 0x7f7d440b2000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009fd0 tx=0x7f7d5000b690 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.510+0000 7f7d437fe640 1 -- 172.21.15.31:0/553411850 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d540a7430 con 0x5583ecb39390 2024-04-25T16:02:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.510+0000 7f7d437fe640 1 -- 172.21.15.31:0/553411850 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d540ab7e0 con 0x5583ecb39390 2024-04-25T16:02:19.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.510+0000 7f7d437fe640 1 -- 172.21.15.31:0/553411850 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d540a1bc0 con 0x5583ecb39390 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.511+0000 7f7d66163740 1 -- 172.21.15.31:0/553411850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 msgr2=0x5583ecedb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.511+0000 7f7d66163740 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecedb810 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f7d540a73c0 tx=0x7f7d540a6f40 comp rx=0 tx=0).stop 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.511+0000 7f7d66163740 1 -- 172.21.15.31:0/553411850 shutdown_connections 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.511+0000 7f7d66163740 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb186b0 0x5583ecedca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.511+0000 7f7d66163740 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb104d0 0x5583ecedbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.511+0000 7f7d66163740 1 --2- 172.21.15.31:0/553411850 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecedb810 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.511+0000 7f7d66163740 1 -- 172.21.15.31:0/553411850 >> 172.21.15.31:0/553411850 conn(0x5583ecb2d4e0 msgr2=0x5583eced7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.511+0000 7f7d66163740 1 -- 172.21.15.31:0/553411850 shutdown_connections 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.511+0000 7f7d66163740 1 -- 172.21.15.31:0/553411850 wait complete. 2024-04-25T16:02:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.512+0000 7f7d66163740 1 -- start start 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.512+0000 7f7d66163740 1 Processor -- start 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.512+0000 7f7d66163740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.512+0000 7f7d66163740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 0x5583ecee5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.512+0000 7f7d66163740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583eceec9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d66163740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5583ecedd530 con 0x5583ecb39390 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d66163740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5583ecdc6dc0 con 0x5583ecb104d0 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d66163740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5583ecedd6b0 con 0x5583ecb186b0 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d66163740 10 client.0 _ll_register_callbacks cb 0x5583ecde2600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d84c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583eceec9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d84c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583eceec9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37712/0 (socket says 172.21.15.31:37712) 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d84c640 1 -- 172.21.15.31:0/1390119626 learned_addr learned my addr 172.21.15.31:0/1390119626 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5c84a640 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 0x5583ecee5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d84c640 1 -- 172.21.15.31:0/1390119626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 msgr2=0x5583ecee5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 0x5583ecee5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d84c640 1 -- 172.21.15.31:0/1390119626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 msgr2=0x5583ecee5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d84c640 1 -- 172.21.15.31:0/1390119626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583ecb513f0 con 0x5583ecb39390 2024-04-25T16:02:19.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583eceec9a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7d50003970 tx=0x7f7d50004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3920d01c4e2e007 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.513+0000 7f7d40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d40ff9640 1 -- 172.21.15.31:0/1390119626 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5002a410 con 0x5583ecb39390 2024-04-25T16:02:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d66163740 1 -- 172.21.15.31:0/1390119626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d540ab7e0 con 0x5583ecb39390 2024-04-25T16:02:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d40ff9640 1 -- 172.21.15.31:0/1390119626 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7d5002a5b0 con 0x5583ecb39390 2024-04-25T16:02:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d40ff9640 1 -- 172.21.15.31:0/1390119626 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d5002a7b0 con 0x5583ecb39390 2024-04-25T16:02:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d66163740 10 client.15738 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d66163740 1 -- 172.21.15.31:0/1390119626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5583ecc1b7d0 con 0x5583ecb39390 2024-04-25T16:02:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d40ff9640 1 -- 172.21.15.31:0/1390119626 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d540ab7e0 con 0x5583ecb39390 2024-04-25T16:02:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d40ff9640 1 -- 172.21.15.31:0/1390119626 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7d5001c050 con 0x5583ecb39390 2024-04-25T16:02:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d66163740 10 client.15738 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d66163740 10 client.15738 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d66163740 20 client.15738 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.514+0000 7f7d66163740 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5583ecaf7f60 0x5583ecef3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.515+0000 7f7d66163740 4 client.15738 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.515+0000 7f7d66163740 1 -- 172.21.15.31:0/1390119626 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583ece4d990 con 0x5583ecaf7f60 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.515+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5583ecaf7f60 0x5583ecef3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.515+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5583ecaf7f60 0x5583ecef3230 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c49746cbbcf7f377 server_cookie=1b59c40e57467566 in_seq=0 out_seq=0 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.515+0000 7f7d40ff9640 10 client.15738 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d40ff9640 1 -- 172.21.15.31:0/1390119626 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5583ece4d990 con 0x5583ecaf7f60 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d40ff9640 10 client.15738 handle_command_reply: tid=0 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d66163740 1 client.15738 shutdown 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d66163740 1 -- 172.21.15.31:0/1390119626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 msgr2=0x5583eceec9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d66163740 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583eceec9a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f7d50003970 tx=0x7f7d50004ab0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d66163740 1 -- 172.21.15.31:0/1390119626 shutdown_connections 2024-04-25T16:02:19.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d66163740 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5583ecaf7f60 0x5583ecef3230 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d66163740 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583eceec9a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d66163740 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 0x5583ecee5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d66163740 1 --2- 172.21.15.31:0/1390119626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d66163740 1 -- 172.21.15.31:0/1390119626 >> 172.21.15.31:0/1390119626 conn(0x5583ecb2d4e0 msgr2=0x5583eced6040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.516+0000 7f7d40ff9640 0 client.15738 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:02:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.517+0000 7f7d66163740 1 -- 172.21.15.31:0/1390119626 shutdown_connections 2024-04-25T16:02:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.517+0000 7f7d66163740 1 -- 172.21.15.31:0/1390119626 wait complete. 2024-04-25T16:02:19.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.517+0000 7f7d66163740 20 client.15738 trim_cache size 0 max 16384 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d66163740 1 Processor -- start 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d66163740 1 -- start start 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d66163740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d66163740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb186b0 0x5583ecee07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d66163740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb39390 0x5583eceec8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d66163740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d581a9080 con 0x5583ecb186b0 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d66163740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7d581aabc0 con 0x5583ecb104d0 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d66163740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d581a9750 con 0x5583ecb39390 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d5c84a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb186b0 0x5583ecee07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d5c84a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb186b0 0x5583ecee07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37716/0 (socket says 172.21.15.31:37716) 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d5c84a640 1 -- 172.21.15.31:0/2823714611 learned_addr learned my addr 172.21.15.31:0/2823714611 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.544+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb39390 0x5583eceec8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d5c84a640 1 -- 172.21.15.31:0/2823714611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb39390 msgr2=0x5583eceec8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d5c84a640 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb39390 0x5583eceec8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d5c84a640 1 -- 172.21.15.31:0/2823714611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 msgr2=0x5583ecee0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d5c84a640 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d5c84a640 1 -- 172.21.15.31:0/2823714611 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583eceece00 con 0x5583ecb186b0 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d5c84a640 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb186b0 0x5583ecee07d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7d4802a930 tx=0x7f7d48009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a9030e836b6e2ac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d42ffd640 1 -- 172.21.15.31:0/2823714611 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48035070 con 0x5583ecb186b0 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d42ffd640 1 -- 172.21.15.31:0/2823714611 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d48031c50 con 0x5583ecb186b0 2024-04-25T16:02:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.545+0000 7f7d42ffd640 1 -- 172.21.15.31:0/2823714611 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d48003850 con 0x5583ecb186b0 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.546+0000 7f7d66163740 1 -- 172.21.15.31:0/2823714611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb186b0 msgr2=0x5583ecee07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.546+0000 7f7d66163740 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb186b0 0x5583ecee07d0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f7d4802a930 tx=0x7f7d48009640 comp rx=0 tx=0).stop 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.546+0000 7f7d66163740 1 -- 172.21.15.31:0/2823714611 shutdown_connections 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.546+0000 7f7d66163740 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb39390 0x5583eceec8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.546+0000 7f7d66163740 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb186b0 0x5583ecee07d0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.546+0000 7f7d66163740 1 --2- 172.21.15.31:0/2823714611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.546+0000 7f7d66163740 1 -- 172.21.15.31:0/2823714611 >> 172.21.15.31:0/2823714611 conn(0x5583ecb2d4e0 msgr2=0x5583ecedb7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.546+0000 7f7d66163740 1 -- 172.21.15.31:0/2823714611 shutdown_connections 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.546+0000 7f7d66163740 1 -- 172.21.15.31:0/2823714611 wait complete. 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d66163740 1 -- start start 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d66163740 1 Processor -- start 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d66163740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d66163740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 0x5583ecee9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d66163740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecf595d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d66163740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7d581aa4f0 con 0x5583ecb39390 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d66163740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7d581a9e20 con 0x5583ecb104d0 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d5d84c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecf595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d66163740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d581a7080 con 0x5583ecb186b0 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d5d84c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecf595d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37718/0 (socket says 172.21.15.31:37718) 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d5d84c640 1 -- 172.21.15.31:0/2452943141 learned_addr learned my addr 172.21.15.31:0/2452943141 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.547+0000 7f7d66163740 10 client.0 _ll_register_callbacks cb 0x7f7d5000bf30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d5c84a640 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 0x5583ecee9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d5d84c640 1 -- 172.21.15.31:0/2452943141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 msgr2=0x5583ecee9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 0x5583ecee9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d5d84c640 1 -- 172.21.15.31:0/2452943141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 msgr2=0x5583ecee97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d5d84c640 1 -- 172.21.15.31:0/2452943141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583ecf59b10 con 0x5583ecb39390 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d5c84a640 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 0x5583ecee9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d5d84c640 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecf595d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7d50006000 tx=0x7f7d50003ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a2ca6f1ca5a3cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d40ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d40ff9640 1 -- 172.21.15.31:0/2452943141 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d50042020 con 0x5583ecb39390 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d40ff9640 1 -- 172.21.15.31:0/2452943141 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7d5002a950 con 0x5583ecb39390 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.548+0000 7f7d66163740 1 -- 172.21.15.31:0/2452943141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d48031c50 con 0x5583ecb39390 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.549+0000 7f7d40ff9640 1 -- 172.21.15.31:0/2452943141 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7d50039050 con 0x5583ecb39390 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.549+0000 7f7d66163740 10 client.15744 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.549+0000 7f7d66163740 1 -- 172.21.15.31:0/2452943141 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5583ecf5ac70 con 0x5583ecb39390 2024-04-25T16:02:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.549+0000 7f7d40ff9640 1 -- 172.21.15.31:0/2452943141 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d5003a040 con 0x5583ecb39390 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.549+0000 7f7d40ff9640 1 -- 172.21.15.31:0/2452943141 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f7d5001c040 con 0x5583ecb39390 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.549+0000 7f7d66163740 10 client.15744 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.549+0000 7f7d66163740 10 client.15744 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.549+0000 7f7d66163740 20 client.15744 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.549+0000 7f7d66163740 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5583ecacd3a0 0x5583ecf5d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.550+0000 7f7d66163740 4 client.15744 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.550+0000 7f7d66163740 1 -- 172.21.15.31:0/2452943141 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f7d48031c50 con 0x5583ecacd3a0 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.550+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5583ecacd3a0 0x5583ecf5d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.550+0000 7f7d40ff9640 10 client.15744 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:02:19.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.550+0000 7f7d5d04b640 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5583ecacd3a0 0x5583ecf5d690 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4a6b7e7b84cd1522 server_cookie=da34d7009d4f199b in_seq=0 out_seq=0 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.550+0000 7f7d40ff9640 1 -- 172.21.15.31:0/2452943141 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f7d48031c50 con 0x5583ecacd3a0 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.550+0000 7f7d40ff9640 10 client.15744 handle_command_reply: tid=0 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 client.15744 shutdown 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 -- 172.21.15.31:0/2452943141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 msgr2=0x5583ecf595d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecf595d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f7d50006000 tx=0x7f7d50003ef0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 -- 172.21.15.31:0/2452943141 shutdown_connections 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5583ecacd3a0 0x5583ecf5d690 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5583ecb39390 0x5583ecf595d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5583ecb186b0 0x5583ecee9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 --2- 172.21.15.31:0/2452943141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5583ecb104d0 0x5583ecee97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 -- 172.21.15.31:0/2452943141 >> 172.21.15.31:0/2452943141 conn(0x5583ecb2d4e0 msgr2=0x5583ecedb790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d40ff9640 0 client.15744 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 -- 172.21.15.31:0/2452943141 shutdown_connections 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 1 -- 172.21.15.31:0/2452943141 wait complete. 2024-04-25T16:02:19.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.551+0000 7f7d66163740 20 client.15744 trim_cache size 0 max 16384 2024-04-25T16:02:19.556 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:19.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 -- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7d440afaf0 msgr2=0x7f7d440b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7d440afaf0 0x7f7d440b2000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7d50009fd0 tx=0x7f7d5000b690 comp rx=0 tx=0).stop 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 -- 172.21.15.31:0/2211825843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d581029e0 msgr2=0x7f7d5819a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d581029e0 0x7f7d5819a550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7d5400e9f0 tx=0x7f7d5400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 -- 172.21.15.31:0/2211825843 shutdown_connections 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7d440afaf0 0x7f7d440b2000 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7d58108920 0x7f7d581a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d58103c20 0x7f7d5819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 --2- 172.21.15.31:0/2211825843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7d581029e0 0x7f7d5819a550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 -- 172.21.15.31:0/2211825843 >> 172.21.15.31:0/2211825843 conn(0x7f7d580fdbd0 msgr2=0x7f7d580ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 -- 172.21.15.31:0/2211825843 shutdown_connections 2024-04-25T16:02:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.552+0000 7f7d5f2af640 1 -- 172.21.15.31:0/2211825843 wait complete. 2024-04-25T16:02:19.564 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:02:19.699 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:19 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/217214638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:02:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.710+0000 7f175c4a1640 1 -- 172.21.15.31:0/1117893200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17541029f0 msgr2=0x7f17541036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.710+0000 7f175c4a1640 1 --2- 172.21.15.31:0/1117893200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17541029f0 0x7f17541036f0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f1750009900 tx=0x7f175002a410 comp rx=0 tx=0).stop 2024-04-25T16:02:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.710+0000 7f175c4a1640 1 -- 172.21.15.31:0/1117893200 shutdown_connections 2024-04-25T16:02:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.710+0000 7f175c4a1640 1 --2- 172.21.15.31:0/1117893200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754108930 0x7f175410ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.710+0000 7f175c4a1640 1 --2- 172.21.15.31:0/1117893200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1754103c30 0x7f1754104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.710+0000 7f175c4a1640 1 --2- 172.21.15.31:0/1117893200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f17541029f0 0x7f17541036f0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.710+0000 7f175c4a1640 1 -- 172.21.15.31:0/1117893200 >> 172.21.15.31:0/1117893200 conn(0x7f17540fdc00 msgr2=0x7f1754100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.710+0000 7f175c4a1640 1 -- 172.21.15.31:0/1117893200 shutdown_connections 2024-04-25T16:02:19.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.710+0000 7f175c4a1640 1 -- 172.21.15.31:0/1117893200 wait complete. 2024-04-25T16:02:19.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175c4a1640 1 Processor -- start 2024-04-25T16:02:19.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175c4a1640 1 -- start start 2024-04-25T16:02:19.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175c4a1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17541029f0 0x7f175419c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175c4a1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1754103c30 0x7f175419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175c4a1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1754108930 0x7f17541a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175c4a1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f175410f610 con 0x7f1754108930 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175c4a1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f175410f490 con 0x7f1754103c30 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175c4a1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f175410f790 con 0x7f17541029f0 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175aa3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1754108930 0x7f17541a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f1759a3c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1754103c30 0x7f175419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.711+0000 7f175aa3e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1754108930 0x7f17541a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37734/0 (socket says 172.21.15.31:37734) 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f175aa3e640 1 -- 172.21.15.31:0/2264362762 learned_addr learned my addr 172.21.15.31:0/2264362762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f175aa3e640 1 -- 172.21.15.31:0/2264362762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17541029f0 msgr2=0x7f175419c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f175aa3e640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17541029f0 0x7f175419c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f175aa3e640 1 -- 172.21.15.31:0/2264362762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1754103c30 msgr2=0x7f175419c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f175aa3e640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1754103c30 0x7f175419c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f175aa3e640 1 -- 172.21.15.31:0/2264362762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17541a3fa0 con 0x7f1754108930 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f1759a3c640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1754103c30 0x7f175419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f175aa3e640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1754108930 0x7f17541a3930 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f174c00eaa0 tx=0x7f174c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f174b7fe640 1 -- 172.21.15.31:0/2264362762 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c00cdb0 con 0x7f1754108930 2024-04-25T16:02:19.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f175c4a1640 1 -- 172.21.15.31:0/2264362762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17541a4290 con 0x7f1754108930 2024-04-25T16:02:19.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f174b7fe640 1 -- 172.21.15.31:0/2264362762 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f174c005020 con 0x7f1754108930 2024-04-25T16:02:19.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f174b7fe640 1 -- 172.21.15.31:0/2264362762 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c0116d0 con 0x7f1754108930 2024-04-25T16:02:19.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.712+0000 7f175c4a1640 1 -- 172.21.15.31:0/2264362762 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17541a4830 con 0x7f1754108930 2024-04-25T16:02:19.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.715+0000 7f1763432740 1 Processor -- start 2024-04-25T16:02:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.716+0000 7f174b7fe640 1 -- 172.21.15.31:0/2264362762 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f174c0054a0 con 0x7f1754108930 2024-04-25T16:02:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1763432740 1 -- start start 2024-04-25T16:02:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1763432740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90e9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1763432740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90e9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1763432740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90eaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1763432740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562af904dd80 con 0x562af8d1e4d0 2024-04-25T16:02:19.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1763432740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562af9007a20 con 0x562af8d266b0 2024-04-25T16:02:19.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1763432740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562af9049d40 con 0x562af8d47390 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1759a3c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90e9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f175a23d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90e9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f174b7fe640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17400afb70 0x7f17400b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1759a3c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90e9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48492/0 (socket says 172.21.15.31:48492) 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1759a3c640 1 -- 172.21.15.31:0/2522605305 learned_addr learned my addr 172.21.15.31:0/2522605305 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f1759a3c640 1 --2- 172.21.15.31:0/2522605305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90eaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.717+0000 7f174b7fe640 1 -- 172.21.15.31:0/2264362762 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f174c0ded00 con 0x7f1754108930 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f175aa3e640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17400afb70 0x7f17400b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f1759a3c640 1 -- 172.21.15.31:0/2522605305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 msgr2=0x562af90eaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f1759a3c640 1 --2- 172.21.15.31:0/2522605305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90eaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f1759a3c640 1 -- 172.21.15.31:0/2522605305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 msgr2=0x562af90e9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f1759a3c640 1 --2- 172.21.15.31:0/2522605305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90e9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f1759a3c640 1 -- 172.21.15.31:0/2522605305 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562af9012c30 con 0x562af8d47390 2024-04-25T16:02:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f175a23d640 1 --2- 172.21.15.31:0/2522605305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90e9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f1759a3c640 1 --2- 172.21.15.31:0/2522605305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90e9810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1744011c00 tx=0x7f1744009ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3d25a997f2e78b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f175aa3e640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17400afb70 0x7f17400b2080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f174c0057f0 tx=0x7f174c005840 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f1748bfe640 1 -- 172.21.15.31:0/2522605305 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174400fc50 con 0x562af8d47390 2024-04-25T16:02:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f1748bfe640 1 -- 172.21.15.31:0/2522605305 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f174400fdf0 con 0x562af8d47390 2024-04-25T16:02:19.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.718+0000 7f1748bfe640 1 -- 172.21.15.31:0/2522605305 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174400e610 con 0x562af8d47390 2024-04-25T16:02:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.719+0000 7f1763432740 1 -- 172.21.15.31:0/2522605305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 msgr2=0x562af90e9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.719+0000 7f1763432740 1 --2- 172.21.15.31:0/2522605305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90e9810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1744011c00 tx=0x7f1744009ba0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.719+0000 7f1763432740 1 -- 172.21.15.31:0/2522605305 shutdown_connections 2024-04-25T16:02:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.719+0000 7f1763432740 1 --2- 172.21.15.31:0/2522605305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90eaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.719+0000 7f1763432740 1 --2- 172.21.15.31:0/2522605305 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90e9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.719+0000 7f1763432740 1 --2- 172.21.15.31:0/2522605305 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90e9810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.719+0000 7f1763432740 1 -- 172.21.15.31:0/2522605305 >> 172.21.15.31:0/2522605305 conn(0x562af8d3b4e0 msgr2=0x562af90e5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.719+0000 7f1763432740 1 -- 172.21.15.31:0/2522605305 shutdown_connections 2024-04-25T16:02:19.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.719+0000 7f1763432740 1 -- 172.21.15.31:0/2522605305 wait complete. 2024-04-25T16:02:19.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1763432740 1 -- start start 2024-04-25T16:02:19.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1763432740 1 Processor -- start 2024-04-25T16:02:19.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1763432740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1763432740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90f3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1763432740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90faa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1763432740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562af90eb530 con 0x562af8d1e4d0 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1763432740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562af8fd4dc0 con 0x562af8d266b0 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1763432740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562af90eb6b0 con 0x562af8d47390 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1763432740 10 client.0 _ll_register_callbacks cb 0x562af8ff0600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1759a3c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1759a3c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90f35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37748/0 (socket says 172.21.15.31:37748) 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1759a3c640 1 -- 172.21.15.31:0/4270505531 learned_addr learned my addr 172.21.15.31:0/4270505531 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1759a3c640 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1759a3c640 1 -- 172.21.15.31:0/4270505531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 msgr2=0x562af90faa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1759a3c640 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90faa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1759a3c640 1 -- 172.21.15.31:0/4270505531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 msgr2=0x562af90f3b00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1759a3c640 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90f3b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.720+0000 7f1759a3c640 1 -- 172.21.15.31:0/4270505531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562af8d5f3f0 con 0x562af8d1e4d0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.721+0000 7f1759a3c640 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90f35c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f174400fd20 tx=0x7f1744005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=931e694916ad2c8d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.721+0000 7f173dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.721+0000 7f173dffb640 1 -- 172.21.15.31:0/4270505531 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744004490 con 0x562af8d1e4d0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.721+0000 7f173dffb640 1 -- 172.21.15.31:0/4270505531 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f174402c9c0 con 0x562af8d1e4d0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.721+0000 7f1763432740 1 -- 172.21.15.31:0/4270505531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f174400fdf0 con 0x562af8d1e4d0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.721+0000 7f173dffb640 1 -- 172.21.15.31:0/4270505531 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f174402cbc0 con 0x562af8d1e4d0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.721+0000 7f1763432740 10 client.15756 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.721+0000 7f1763432740 1 -- 172.21.15.31:0/4270505531 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562af8e297d0 con 0x562af8d1e4d0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.722+0000 7f173dffb640 1 -- 172.21.15.31:0/4270505531 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f174400fdf0 con 0x562af8d1e4d0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.722+0000 7f173dffb640 1 -- 172.21.15.31:0/4270505531 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f174401d020 con 0x562af8d1e4d0 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.722+0000 7f1763432740 10 client.15756 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.722+0000 7f1763432740 10 client.15756 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.722+0000 7f1763432740 20 client.15756 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.722+0000 7f1763432740 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562af8d05f60 0x562af9100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.722+0000 7f1763432740 4 client.15756 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:19.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.722+0000 7f1763432740 1 -- 172.21.15.31:0/4270505531 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562af905b990 con 0x562af8d05f60 2024-04-25T16:02:19.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.722+0000 7f175a23d640 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562af8d05f60 0x562af9100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.723+0000 7f173dffb640 10 client.15756 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:02:19.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.723+0000 7f175a23d640 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562af8d05f60 0x562af9100fc0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=26b459fd7a2f66c7 server_cookie=7a02cbab33096779 in_seq=0 out_seq=0 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f173dffb640 1 -- 172.21.15.31:0/4270505531 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562af905b990 con 0x562af8d05f60 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f173dffb640 10 client.15756 handle_command_reply: tid=0 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 client.15756 shutdown 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 -- 172.21.15.31:0/4270505531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 msgr2=0x562af90f35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90f35c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f174400fd20 tx=0x7f1744005850 comp rx=0 tx=0).stop 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 -- 172.21.15.31:0/4270505531 shutdown_connections 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562af8d05f60 0x562af9100fc0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d47390 0x562af90faa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90f3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 --2- 172.21.15.31:0/4270505531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d1e4d0 0x562af90f35c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 -- 172.21.15.31:0/4270505531 >> 172.21.15.31:0/4270505531 conn(0x562af8d3b4e0 msgr2=0x562af90e4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f173dffb640 0 client.15756 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:02:19.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 -- 172.21.15.31:0/4270505531 shutdown_connections 2024-04-25T16:02:19.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 1 -- 172.21.15.31:0/4270505531 wait complete. 2024-04-25T16:02:19.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.724+0000 7f1763432740 20 client.15756 trim_cache size 0 max 16384 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.751+0000 7f1763432740 1 Processor -- start 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.751+0000 7f1763432740 1 -- start start 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.751+0000 7f1763432740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 0x562af90ee980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.751+0000 7f1763432740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90fa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.751+0000 7f1763432740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 0x562af90f2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.751+0000 7f1763432740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17541a9100 con 0x562af8d47390 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f1763432740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f17541aac40 con 0x562af8d266b0 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f1763432740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17541a97d0 con 0x562af8d1e4d0 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f175a23d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90fa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f1759a3c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 0x562af90f2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f175a23d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90fa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49254/0 (socket says 172.21.15.31:49254) 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f175a23d640 1 -- 172.21.15.31:0/757732362 learned_addr learned my addr 172.21.15.31:0/757732362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f1759a3c640 1 --2- 172.21.15.31:0/757732362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 0x562af90ee980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f175a23d640 1 -- 172.21.15.31:0/757732362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 msgr2=0x562af90ee980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f175a23d640 1 --2- 172.21.15.31:0/757732362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 0x562af90ee980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f175a23d640 1 -- 172.21.15.31:0/757732362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 msgr2=0x562af90f2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f175a23d640 1 --2- 172.21.15.31:0/757732362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 0x562af90f2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f175a23d640 1 -- 172.21.15.31:0/757732362 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562af90f16d0 con 0x562af8d266b0 2024-04-25T16:02:19.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.752+0000 7f175a23d640 1 --2- 172.21.15.31:0/757732362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90fa9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1750009230 tx=0x7f1750000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d80a2b21a7064bec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.753+0000 7f173ffff640 1 -- 172.21.15.31:0/757732362 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1750035040 con 0x562af8d266b0 2024-04-25T16:02:19.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.753+0000 7f173ffff640 1 -- 172.21.15.31:0/757732362 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1750046020 con 0x562af8d266b0 2024-04-25T16:02:19.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.753+0000 7f173ffff640 1 -- 172.21.15.31:0/757732362 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1750005360 con 0x562af8d266b0 2024-04-25T16:02:19.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 -- 172.21.15.31:0/757732362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 msgr2=0x562af90fa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 --2- 172.21.15.31:0/757732362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90fa9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1750009230 tx=0x7f1750000bc0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 -- 172.21.15.31:0/757732362 shutdown_connections 2024-04-25T16:02:19.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 --2- 172.21.15.31:0/757732362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 0x562af90f2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 --2- 172.21.15.31:0/757732362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90fa9f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 --2- 172.21.15.31:0/757732362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 0x562af90ee980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 -- 172.21.15.31:0/757732362 >> 172.21.15.31:0/757732362 conn(0x562af8d3b4e0 msgr2=0x562af90e8850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 -- 172.21.15.31:0/757732362 shutdown_connections 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 -- 172.21.15.31:0/757732362 wait complete. 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.754+0000 7f1763432740 1 -- start start 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1763432740 1 Processor -- start 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1763432740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 0x562af90f7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1763432740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90f82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1763432740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 0x562af90f9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1763432740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f17541aa570 con 0x562af8d47390 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1759a3c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 0x562af90f7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1759a3c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 0x562af90f7d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48524/0 (socket says 172.21.15.31:48524) 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1759a3c640 1 -- 172.21.15.31:0/2132411803 learned_addr learned my addr 172.21.15.31:0/2132411803 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f175a23d640 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90f82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f17541a9ea0 con 0x562af8d266b0 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17541a7120 con 0x562af8d1e4d0 2024-04-25T16:02:19.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.755+0000 7f1759a3c640 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 0x562af90f9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f1759a3c640 1 -- 172.21.15.31:0/2132411803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 msgr2=0x562af90f7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f1759a3c640 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 0x562af90f7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f1759a3c640 1 -- 172.21.15.31:0/2132411803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 msgr2=0x562af90f82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f1759a3c640 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90f82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f1759a3c640 1 -- 172.21.15.31:0/2132411803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562af90f97c0 con 0x562af8d47390 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f1759a3c640 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 0x562af90f9200 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f17440115a0 tx=0x7f1744021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1d9a688c9250239 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f173dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f1763432740 10 client.0 _ll_register_callbacks cb 0x7f1744005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1750046020 con 0x562af8d47390 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f173dffb640 1 -- 172.21.15.31:0/2132411803 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1744003d80 con 0x562af8d47390 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.756+0000 7f173dffb640 1 -- 172.21.15.31:0/2132411803 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1744003f20 con 0x562af8d47390 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.757+0000 7f173dffb640 1 -- 172.21.15.31:0/2132411803 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1744005040 con 0x562af8d47390 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.757+0000 7f1763432740 10 client.15762 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.757+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=27}) -- 0x562af90fa830 con 0x562af8d47390 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.757+0000 7f173dffb640 1 -- 172.21.15.31:0/2132411803 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f174401d040 con 0x562af8d47390 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.757+0000 7f1763432740 10 client.15762 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.757+0000 7f1763432740 10 client.15762 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.757+0000 7f1763432740 20 client.15762 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:19.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.758+0000 7f1763432740 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562af8cdb3a0 0x562af916b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.758+0000 7f1763432740 4 client.15762 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.758+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562af9055800 con 0x562af8cdb3a0 2024-04-25T16:02:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.758+0000 7f175a23d640 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562af8cdb3a0 0x562af916b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.758+0000 7f173dffb640 10 client.15762 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:02:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.758+0000 7f175a23d640 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562af8cdb3a0 0x562af916b440 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=26e8b7a4d53eee42 server_cookie=c2cabbfc8ccfe4c9 in_seq=0 out_seq=0 2024-04-25T16:02:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.759+0000 7f173dffb640 1 -- 172.21.15.31:0/2132411803 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562af9055800 con 0x562af8cdb3a0 2024-04-25T16:02:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.759+0000 7f173dffb640 10 client.15762 handle_command_reply: tid=0 2024-04-25T16:02:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.759+0000 7f1763432740 1 client.15762 shutdown 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 msgr2=0x562af90f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 0x562af90f9200 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f17440115a0 tx=0x7f1744021040 comp rx=0 tx=0).stop 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 shutdown_connections 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562af8d47390 0x562af90f9200 secure :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f17440115a0 tx=0x7f1744021040 comp rx=0 tx=0).stop 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562af8d266b0 0x562af90f82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x562af8cdb3a0 0x562af916b440 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 --2- 172.21.15.31:0/2132411803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562af8d1e4d0 0x562af90f7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f173dffb640 0 client.15762 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 >> 172.21.15.31:0/2132411803 conn(0x562af8d3b4e0 msgr2=0x562af90e85b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 shutdown_connections 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 1 -- 172.21.15.31:0/2132411803 wait complete. 2024-04-25T16:02:19.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.760+0000 7f1763432740 20 client.15762 trim_cache size 0 max 16384 2024-04-25T16:02:19.763 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 -- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17400afb70 msgr2=0x7f17400b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17400afb70 0x7f17400b2080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f174c0057f0 tx=0x7f174c005840 comp rx=0 tx=0).stop 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 -- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1754108930 msgr2=0x7f17541a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1754108930 0x7f17541a3930 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f174c00eaa0 tx=0x7f174c00ef60 comp rx=0 tx=0).stop 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 -- 172.21.15.31:0/2264362762 shutdown_connections 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17400afb70 0x7f17400b2080 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1754108930 0x7f17541a3930 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1754103c30 0x7f175419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 --2- 172.21.15.31:0/2264362762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17541029f0 0x7f175419c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 -- 172.21.15.31:0/2264362762 >> 172.21.15.31:0/2264362762 conn(0x7f17540fdc00 msgr2=0x7f17540fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 -- 172.21.15.31:0/2264362762 shutdown_connections 2024-04-25T16:02:19.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.761+0000 7f175c4a1640 1 -- 172.21.15.31:0/2264362762 wait complete. 2024-04-25T16:02:19.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:02:19.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:19 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/217214638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:02:19.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:19 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/217214638' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:02:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.924+0000 7f5decd0d640 1 -- 172.21.15.31:0/2168462127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8103c20 msgr2=0x7f5de8104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.924+0000 7f5decd0d640 1 --2- 172.21.15.31:0/2168462127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8103c20 0x7f5de8104030 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009900 tx=0x7f5ddc02a410 comp rx=0 tx=0).stop 2024-04-25T16:02:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.925+0000 7f5decd0d640 1 -- 172.21.15.31:0/2168462127 shutdown_connections 2024-04-25T16:02:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.925+0000 7f5decd0d640 1 --2- 172.21.15.31:0/2168462127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de8108920 0x7f5de810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.925+0000 7f5decd0d640 1 --2- 172.21.15.31:0/2168462127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8103c20 0x7f5de8104030 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.925+0000 7f5decd0d640 1 --2- 172.21.15.31:0/2168462127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5de81029e0 0x7f5de81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.925+0000 7f5decd0d640 1 -- 172.21.15.31:0/2168462127 >> 172.21.15.31:0/2168462127 conn(0x7f5de80fdbd0 msgr2=0x7f5de8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.925+0000 7f5decd0d640 1 -- 172.21.15.31:0/2168462127 shutdown_connections 2024-04-25T16:02:19.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.925+0000 7f5decd0d640 1 -- 172.21.15.31:0/2168462127 wait complete. 2024-04-25T16:02:19.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5decd0d640 1 Processor -- start 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5decd0d640 1 -- start start 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5decd0d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de81029e0 0x7f5de806a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5decd0d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5de8103c20 0x7f5de819f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5decd0d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8108920 0x7f5de81a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5decd0d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5de810e510 con 0x7f5de8108920 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5decd0d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5de810e390 con 0x7f5de8103c20 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5decd0d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5de810e690 con 0x7f5de81029e0 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5de6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8108920 0x7f5de81a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5de6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8108920 0x7f5de81a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37792/0 (socket says 172.21.15.31:37792) 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5de6d9d640 1 -- 172.21.15.31:0/807378953 learned_addr learned my addr 172.21.15.31:0/807378953 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.926+0000 7f5de659c640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de81029e0 0x7f5de806a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.927+0000 7f5de6d9d640 1 -- 172.21.15.31:0/807378953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de81029e0 msgr2=0x7f5de806a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.927+0000 7f5de6d9d640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de81029e0 0x7f5de806a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.927+0000 7f5de6d9d640 1 -- 172.21.15.31:0/807378953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5de8103c20 msgr2=0x7f5de819f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.927+0000 7f5de6d9d640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5de8103c20 0x7f5de819f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.927+0000 7f5de6d9d640 1 -- 172.21.15.31:0/807378953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5de81a2060 con 0x7f5de8108920 2024-04-25T16:02:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.927+0000 7f5de6d9d640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8108920 0x7f5de81a1840 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5de000eaa0 tx=0x7f5de000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.927+0000 7f5dd77fe640 1 -- 172.21.15.31:0/807378953 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de000cdb0 con 0x7f5de8108920 2024-04-25T16:02:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.927+0000 7f5dd77fe640 1 -- 172.21.15.31:0/807378953 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de0005020 con 0x7f5de8108920 2024-04-25T16:02:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.927+0000 7f5decd0d640 1 -- 172.21.15.31:0/807378953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5de81a2350 con 0x7f5de8108920 2024-04-25T16:02:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.928+0000 7f5dd77fe640 1 -- 172.21.15.31:0/807378953 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de00116d0 con 0x7f5de8108920 2024-04-25T16:02:19.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.928+0000 7f5decd0d640 1 -- 172.21.15.31:0/807378953 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5de81a28a0 con 0x7f5de8108920 2024-04-25T16:02:19.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.931+0000 7f5dd77fe640 1 -- 172.21.15.31:0/807378953 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5de00051d0 con 0x7f5de8108920 2024-04-25T16:02:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.931+0000 7f5df3bc0740 1 Processor -- start 2024-04-25T16:02:19.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.932+0000 7f5df3bc0740 1 -- start start 2024-04-25T16:02:19.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.932+0000 7f5dd77fe640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5dd00afc40 0x7f5dd00b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.932+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 0x562322808810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.932+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 0x562322808d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.932+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5623224456b0 0x562322809a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.932+0000 7f5df3bc0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56232276cd80 con 0x56232243d4d0 2024-04-25T16:02:19.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.932+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5dd00afc40 0x7f5dd00b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.932+0000 7f5df3bc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562322726a20 con 0x5623224456b0 2024-04-25T16:02:19.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.932+0000 7f5df3bc0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562322768d40 con 0x562322466390 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de6d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5623224456b0 0x562322809a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de6d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5623224456b0 0x562322809a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49286/0 (socket says 172.21.15.31:49286) 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de6d9d640 1 -- 172.21.15.31:0/2537792169 learned_addr learned my addr 172.21.15.31:0/2537792169 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de659c640 1 --2- 172.21.15.31:0/2537792169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 0x562322808810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de6d9d640 1 -- 172.21.15.31:0/2537792169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 msgr2=0x562322808810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de6d9d640 1 --2- 172.21.15.31:0/2537792169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 0x562322808810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de6d9d640 1 -- 172.21.15.31:0/2537792169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 msgr2=0x562322808d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de6d9d640 1 --2- 172.21.15.31:0/2537792169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 0x562322808d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de6d9d640 1 -- 172.21.15.31:0/2537792169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562322731c30 con 0x5623224456b0 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de6d9d640 1 --2- 172.21.15.31:0/2537792169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5623224456b0 0x562322809a20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5de00a4400 tx=0x7f5de006d730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee6afdeb2ae06d9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5dd77fe640 1 -- 172.21.15.31:0/807378953 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5de00deca0 con 0x7f5de8108920 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.933+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5dd00afc40 0x7f5dd00b2150 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc031500 tx=0x7f5ddc031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.934+0000 7f5dd4ff9640 1 -- 172.21.15.31:0/2537792169 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de00a6ca0 con 0x5623224456b0 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.934+0000 7f5dd4ff9640 1 -- 172.21.15.31:0/2537792169 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de00a7890 con 0x5623224456b0 2024-04-25T16:02:19.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.934+0000 7f5dd4ff9640 1 -- 172.21.15.31:0/2537792169 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0071840 con 0x5623224456b0 2024-04-25T16:02:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 -- 172.21.15.31:0/2537792169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5623224456b0 msgr2=0x562322809a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/2537792169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5623224456b0 0x562322809a20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f5de00a4400 tx=0x7f5de006d730 comp rx=0 tx=0).stop 2024-04-25T16:02:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 -- 172.21.15.31:0/2537792169 shutdown_connections 2024-04-25T16:02:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/2537792169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5623224456b0 0x562322809a20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/2537792169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 0x562322808d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/2537792169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 0x562322808810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 -- 172.21.15.31:0/2537792169 >> 172.21.15.31:0/2537792169 conn(0x56232245a4e0 msgr2=0x562322804020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 -- 172.21.15.31:0/2537792169 shutdown_connections 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 -- 172.21.15.31:0/2537792169 wait complete. 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.935+0000 7f5df3bc0740 1 -- start start 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5df3bc0740 1 Processor -- start 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56232243d4d0 0x562322812500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x562322812a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 0x5623228199a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x562322812a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x562322812a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37800/0 (socket says 172.21.15.31:37800) 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5de5d9b640 1 -- 172.21.15.31:0/546451378 learned_addr learned my addr 172.21.15.31:0/546451378 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56232280a470 con 0x5623224456b0 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5623226f3dc0 con 0x56232243d4d0 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56232280a5f0 con 0x562322466390 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5de659c640 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56232243d4d0 0x562322812500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.936+0000 7f5de6d9d640 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 0x5623228199a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5df3bc0740 10 client.0 _ll_register_callbacks cb 0x56232270f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5de659c640 1 -- 172.21.15.31:0/546451378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 msgr2=0x5623228199a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5de659c640 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 0x5623228199a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5de659c640 1 -- 172.21.15.31:0/546451378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 msgr2=0x562322812a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5de659c640 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x562322812a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5de659c640 1 -- 172.21.15.31:0/546451378 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56232247e3f0 con 0x56232243d4d0 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x562322812a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5de659c640 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56232243d4d0 0x562322812500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x562322809860 tx=0x7f5dd800bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=840560d46a1ccc3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5dcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5dcdffb640 1 -- 172.21.15.31:0/546451378 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8003d90 con 0x56232243d4d0 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.937+0000 7f5dcdffb640 1 -- 172.21.15.31:0/546451378 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5dd8011440 con 0x56232243d4d0 2024-04-25T16:02:19.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.938+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5de00a7890 con 0x56232243d4d0 2024-04-25T16:02:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.938+0000 7f5dcdffb640 1 -- 172.21.15.31:0/546451378 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5dd8011640 con 0x56232243d4d0 2024-04-25T16:02:19.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.938+0000 7f5df3bc0740 10 client.25490 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.938+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5623225487d0 con 0x56232243d4d0 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.938+0000 7f5dcdffb640 1 -- 172.21.15.31:0/546451378 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5de00a7890 con 0x56232243d4d0 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.939+0000 7f5dcdffb640 1 -- 172.21.15.31:0/546451378 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5dd8026980 con 0x56232243d4d0 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.939+0000 7f5df3bc0740 10 client.25490 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.939+0000 7f5df3bc0740 10 client.25490 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.939+0000 7f5df3bc0740 20 client.25490 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.939+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x562322424f60 0x562322820300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.939+0000 7f5df3bc0740 4 client.25490 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.939+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56232277a990 con 0x562322424f60 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.939+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x562322424f60 0x562322820300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.940+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x562322424f60 0x562322820300 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1a761b609626a830 server_cookie=268eddf5c0e17450 in_seq=0 out_seq=0 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.940+0000 7f5dcdffb640 10 client.25490 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5dcdffb640 1 -- 172.21.15.31:0/546451378 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56232277a990 con 0x562322424f60 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5dcdffb640 10 client.25490 handle_command_reply: tid=0 2024-04-25T16:02:19.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 client.25490 shutdown 2024-04-25T16:02:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56232243d4d0 msgr2=0x562322812500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56232243d4d0 0x562322812500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x562322809860 tx=0x7f5dd800bf30 comp rx=0 tx=0).stop 2024-04-25T16:02:19.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 shutdown_connections 2024-04-25T16:02:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562322466390 0x5623228199a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x562322812a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x562322424f60 0x562322820300 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/546451378 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56232243d4d0 0x562322812500 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 >> 172.21.15.31:0/546451378 conn(0x56232245a4e0 msgr2=0x562322803040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5dcdffb640 0 client.25490 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:02:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.941+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 shutdown_connections 2024-04-25T16:02:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.942+0000 7f5df3bc0740 1 -- 172.21.15.31:0/546451378 wait complete. 2024-04-25T16:02:19.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.942+0000 7f5df3bc0740 20 client.25490 trim_cache size 0 max 16384 2024-04-25T16:02:19.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.965+0000 7f5df3bc0740 1 Processor -- start 2024-04-25T16:02:19.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.965+0000 7f5df3bc0740 1 -- start start 2024-04-25T16:02:19.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.965+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 0x5623227ff190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.965+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5623224456b0 0x56232280d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.965+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 0x56232280d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.965+0000 7f5df3bc0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5de81ad460 con 0x56232243d4d0 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.965+0000 7f5df3bc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5de81aefa0 con 0x562322466390 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.965+0000 7f5df3bc0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5de81adb30 con 0x5623224456b0 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 0x5623227ff190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 0x5623227ff190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37804/0 (socket says 172.21.15.31:37804) 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5623224456b0 0x56232280d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de659c640 1 -- 172.21.15.31:0/3730526986 learned_addr learned my addr 172.21.15.31:0/3730526986 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de6d9d640 1 --2- 172.21.15.31:0/3730526986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 0x56232280d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de5d9b640 1 -- 172.21.15.31:0/3730526986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 msgr2=0x56232280d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/3730526986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 0x56232280d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de5d9b640 1 -- 172.21.15.31:0/3730526986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 msgr2=0x5623227ff190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/3730526986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 0x5623227ff190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de5d9b640 1 -- 172.21.15.31:0/3730526986 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623228083d0 con 0x5623224456b0 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de659c640 1 --2- 172.21.15.31:0/3730526986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 0x5623227ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.966+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/3730526986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5623224456b0 0x56232280d290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x562322820a30 tx=0x7f5ddc03a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5a73e82b7583474 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5dcffff640 1 -- 172.21.15.31:0/3730526986 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc058070 con 0x5623224456b0 2024-04-25T16:02:19.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5dcffff640 1 -- 172.21.15.31:0/3730526986 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ddc0054f0 con 0x5623224456b0 2024-04-25T16:02:19.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5dcffff640 1 -- 172.21.15.31:0/3730526986 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ddc035040 con 0x5623224456b0 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5df3bc0740 1 -- 172.21.15.31:0/3730526986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5623224456b0 msgr2=0x56232280d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/3730526986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5623224456b0 0x56232280d290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x562322820a30 tx=0x7f5ddc03a000 comp rx=0 tx=0).stop 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5df3bc0740 1 -- 172.21.15.31:0/3730526986 shutdown_connections 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/3730526986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 0x56232280d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/3730526986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5623224456b0 0x56232280d290 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/3730526986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56232243d4d0 0x5623227ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.967+0000 7f5df3bc0740 1 -- 172.21.15.31:0/3730526986 >> 172.21.15.31:0/3730526986 conn(0x56232245a4e0 msgr2=0x562322808790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 -- 172.21.15.31:0/3730526986 shutdown_connections 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 -- 172.21.15.31:0/3730526986 wait complete. 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 -- start start 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 Processor -- start 2024-04-25T16:02:19.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56232243d4d0 0x562322818c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x5623228157e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 0x562322815d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5de81ae8d0 con 0x5623224456b0 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5de81ae200 con 0x562322466390 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5de81ab440 con 0x56232243d4d0 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.968+0000 7f5df3bc0740 10 client.0 _ll_register_callbacks cb 0x7f5dd8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x5623228157e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56232243d4d0 0x562322818c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x5623228157e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37808/0 (socket says 172.21.15.31:37808) 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de5d9b640 1 -- 172.21.15.31:0/1929493201 learned_addr learned my addr 172.21.15.31:0/1929493201 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de6d9d640 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 0x562322815d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de659c640 1 -- 172.21.15.31:0/1929493201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 msgr2=0x562322815d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de659c640 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 0x562322815d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de659c640 1 -- 172.21.15.31:0/1929493201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 msgr2=0x5623228157e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de659c640 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x5623228157e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de659c640 1 -- 172.21.15.31:0/1929493201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562322816260 con 0x56232243d4d0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x5623228157e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5de659c640 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56232243d4d0 0x562322818c10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8026e40 tx=0x7f5dd801bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6abe10139d2c55df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.969+0000 7f5dcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.970+0000 7f5dcdffb640 1 -- 172.21.15.31:0/1929493201 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd802b040 con 0x56232243d4d0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.970+0000 7f5df3bc0740 1 -- 172.21.15.31:0/1929493201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ddc0054f0 con 0x56232243d4d0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.970+0000 7f5dcdffb640 1 -- 172.21.15.31:0/1929493201 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5dd8011440 con 0x56232243d4d0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.970+0000 7f5dcdffb640 1 -- 172.21.15.31:0/1929493201 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5dd800b390 con 0x56232243d4d0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.970+0000 7f5df3bc0740 10 client.25456 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.970+0000 7f5df3bc0740 1 -- 172.21.15.31:0/1929493201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56232288fda0 con 0x56232243d4d0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.970+0000 7f5dcdffb640 1 -- 172.21.15.31:0/1929493201 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ddc0054f0 con 0x56232243d4d0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.970+0000 7f5dcdffb640 1 -- 172.21.15.31:0/1929493201 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f5dd80117e0 con 0x56232243d4d0 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.970+0000 7f5df3bc0740 10 client.25456 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:19.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.971+0000 7f5df3bc0740 10 client.25456 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:02:19.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.971+0000 7f5df3bc0740 20 client.25456 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:19.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.971+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5623223fa3a0 0x562322892790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:19.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.971+0000 7f5df3bc0740 4 client.25456 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:19.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.971+0000 7f5df3bc0740 1 -- 172.21.15.31:0/1929493201 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562322774800 con 0x5623223fa3a0 2024-04-25T16:02:19.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.971+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5623223fa3a0 0x562322892790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:19.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.971+0000 7f5dcdffb640 10 client.25456 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:02:19.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.971+0000 7f5de5d9b640 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5623223fa3a0 0x562322892790 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a5c6836480a0de3 server_cookie=a493beb92d50827b in_seq=0 out_seq=0 2024-04-25T16:02:19.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5dcdffb640 1 -- 172.21.15.31:0/1929493201 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562322774800 con 0x5623223fa3a0 2024-04-25T16:02:19.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5dcdffb640 10 client.25456 handle_command_reply: tid=0 2024-04-25T16:02:19.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 client.25456 shutdown 2024-04-25T16:02:19.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 -- 172.21.15.31:0/1929493201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56232243d4d0 msgr2=0x562322818c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56232243d4d0 0x562322818c10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8026e40 tx=0x7f5dd801bf40 comp rx=0 tx=0).stop 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 -- 172.21.15.31:0/1929493201 shutdown_connections 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5623223fa3a0 0x562322892790 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562322466390 0x562322815d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5623224456b0 0x5623228157e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5dcdffb640 0 client.25456 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 --2- 172.21.15.31:0/1929493201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56232243d4d0 0x562322818c10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 -- 172.21.15.31:0/1929493201 >> 172.21.15.31:0/1929493201 conn(0x56232245a4e0 msgr2=0x562322808790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 -- 172.21.15.31:0/1929493201 shutdown_connections 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 1 -- 172.21.15.31:0/1929493201 wait complete. 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.972+0000 7f5df3bc0740 20 client.25456 trim_cache size 0 max 16384 2024-04-25T16:02:19.977 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:19.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 -- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5dd00afc40 msgr2=0x7f5dd00b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5dd00afc40 0x7f5dd00b2150 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc031500 tx=0x7f5ddc031400 comp rx=0 tx=0).stop 2024-04-25T16:02:19.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 -- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8108920 msgr2=0x7f5de81a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:19.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8108920 0x7f5de81a1840 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f5de000eaa0 tx=0x7f5de000ef60 comp rx=0 tx=0).stop 2024-04-25T16:02:19.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 -- 172.21.15.31:0/807378953 shutdown_connections 2024-04-25T16:02:19.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5dd00afc40 0x7f5dd00b2150 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5de8108920 0x7f5de81a1840 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5de8103c20 0x7f5de819f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 --2- 172.21.15.31:0/807378953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5de81029e0 0x7f5de806a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:19.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 -- 172.21.15.31:0/807378953 >> 172.21.15.31:0/807378953 conn(0x7f5de80fdbd0 msgr2=0x7f5de8104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:19.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 -- 172.21.15.31:0/807378953 shutdown_connections 2024-04-25T16:02:19.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:19.973+0000 7f5decd0d640 1 -- 172.21.15.31:0/807378953 wait complete. 2024-04-25T16:02:19.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:02:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.148+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1307005650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f50103c20 msgr2=0x7f6f50104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.148+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1307005650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f50103c20 0x7f6f50104030 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009900 tx=0x7f6f4402a410 comp rx=0 tx=0).stop 2024-04-25T16:02:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.149+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1307005650 shutdown_connections 2024-04-25T16:02:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.149+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1307005650 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f50108920 0x7f6f5010ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.149+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1307005650 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f50103c20 0x7f6f50104030 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.149+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1307005650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f501029e0 0x7f6f501036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.149+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1307005650 >> 172.21.15.31:0/1307005650 conn(0x7f6f500fdbd0 msgr2=0x7f6f50100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.149+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1307005650 shutdown_connections 2024-04-25T16:02:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.149+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1307005650 wait complete. 2024-04-25T16:02:20.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f55b5e640 1 Processor -- start 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f55b5e640 1 -- start start 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f55b5e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f501029e0 0x7f6f5006a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f55b5e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f50103c20 0x7f6f5019f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f55b5e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f50108920 0x7f6f501a1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f55b5e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5010e550 con 0x7f6f50108920 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f55b5e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5010e3d0 con 0x7f6f501029e0 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f55b5e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f5010e6d0 con 0x7f6f50103c20 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f4effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f50103c20 0x7f6f5019f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f4effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f50103c20 0x7f6f5019f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48604/0 (socket says 172.21.15.31:48604) 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f4ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f50108920 0x7f6f501a1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f501029e0 0x7f6f5006a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.150+0000 7f6f4effd640 1 -- 172.21.15.31:0/1552470932 learned_addr learned my addr 172.21.15.31:0/1552470932 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.151+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/1552470932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f50103c20 msgr2=0x7f6f5019f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.151+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f50103c20 0x7f6f5019f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.151+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f50108920 msgr2=0x7f6f501a1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.151+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f50108920 0x7f6f501a1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.151+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/1552470932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f501a2080 con 0x7f6f501029e0 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.151+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f50108920 0x7f6f501a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.151+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f501029e0 0x7f6f5006a8f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6f501022b0 tx=0x7f6f4000ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.152+0000 7f6f4cff9640 1 -- 172.21.15.31:0/1552470932 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4000cc40 con 0x7f6f501029e0 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.152+0000 7f6f4cff9640 1 -- 172.21.15.31:0/1552470932 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f4000cde0 con 0x7f6f501029e0 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.152+0000 7f6f4cff9640 1 -- 172.21.15.31:0/1552470932 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f40010430 con 0x7f6f501029e0 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.152+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1552470932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f501a2370 con 0x7f6f501029e0 2024-04-25T16:02:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.152+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1552470932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f501a2810 con 0x7f6f501029e0 2024-04-25T16:02:20.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.155+0000 7f6f5ca0f740 1 Processor -- start 2024-04-25T16:02:20.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f5ca0f740 1 -- start start 2024-04-25T16:02:20.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f4cff9640 1 -- 172.21.15.31:0/1552470932 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6f40003680 con 0x7f6f501029e0 2024-04-25T16:02:20.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f98667390 0x564f98a146e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f9863e4d0 0x564f98a16cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f986466b0 0x564f98a19440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f5ca0f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564f9896dd80 con 0x564f986466b0 2024-04-25T16:02:20.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f5ca0f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564f98927a20 con 0x564f98667390 2024-04-25T16:02:20.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f5ca0f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564f98969d40 con 0x564f9863e4d0 2024-04-25T16:02:20.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f4f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f9863e4d0 0x564f98a16cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f4effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f986466b0 0x564f98a19440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f4ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f98667390 0x564f98a146e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f4effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f986466b0 0x564f98a19440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37840/0 (socket says 172.21.15.31:37840) 2024-04-25T16:02:20.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.156+0000 7f6f4effd640 1 -- 172.21.15.31:0/482665938 learned_addr learned my addr 172.21.15.31:0/482665938 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/482665938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f98667390 msgr2=0x564f98a146e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/482665938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f98667390 0x564f98a146e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/482665938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f986466b0 msgr2=0x564f98a19440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/482665938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f986466b0 0x564f98a19440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/482665938 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f98932c30 con 0x564f9863e4d0 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f4effd640 1 --2- 172.21.15.31:0/482665938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f986466b0 0x564f98a19440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/482665938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f9863e4d0 0x564f98a16cb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f400dc9f0 tx=0x7f6f400e2f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d425d95bb12e52b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f35ffb640 1 -- 172.21.15.31:0/482665938 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f400f2070 con 0x564f9863e4d0 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f4cff9640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6f380afaf0 0x7f6f380b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f35ffb640 1 -- 172.21.15.31:0/482665938 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f400e7c80 con 0x564f9863e4d0 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.157+0000 7f6f35ffb640 1 -- 172.21.15.31:0/482665938 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f400df620 con 0x564f9863e4d0 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.158+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6f380afaf0 0x7f6f380b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.158+0000 7f6f4cff9640 1 -- 172.21.15.31:0/1552470932 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6f400def10 con 0x7f6f501029e0 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.158+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/482665938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f9863e4d0 msgr2=0x564f98a16cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.158+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6f380afaf0 0x7f6f380b2000 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6f5006af70 tx=0x7f6f48009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.158+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/482665938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f9863e4d0 0x564f98a16cb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6f400dc9f0 tx=0x7f6f400e2f50 comp rx=0 tx=0).stop 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.159+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/482665938 shutdown_connections 2024-04-25T16:02:20.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.159+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/482665938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f986466b0 0x564f98a19440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.159+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/482665938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f9863e4d0 0x564f98a16cb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.159+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/482665938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f98667390 0x564f98a146e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.159+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/482665938 >> 172.21.15.31:0/482665938 conn(0x564f9865b4e0 msgr2=0x564f98a04fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.159+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/482665938 shutdown_connections 2024-04-25T16:02:20.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.159+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/482665938 wait complete. 2024-04-25T16:02:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.159+0000 7f6f5ca0f740 1 -- start start 2024-04-25T16:02:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.159+0000 7f6f5ca0f740 1 Processor -- start 2024-04-25T16:02:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.160+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 0x564f98a2ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.160+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a2d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.160+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98a342b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.160+0000 7f6f5ca0f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564f98a2af70 con 0x564f98667390 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.160+0000 7f6f4effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a2d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.160+0000 7f6f4f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 0x564f98a2ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.160+0000 7f6f4effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a2d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48616/0 (socket says 172.21.15.31:48616) 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.160+0000 7f6f4effd640 1 -- 172.21.15.31:0/1658262708 learned_addr learned my addr 172.21.15.31:0/1658262708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.160+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98a342b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564f988f4dc0 con 0x564f9863e4d0 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564f98a2b0f0 con 0x564f986466b0 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f5ca0f740 10 client.0 _ll_register_callbacks cb 0x564f98910600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f4effd640 1 -- 172.21.15.31:0/1658262708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 msgr2=0x564f98a2ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f4effd640 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 0x564f98a2ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f4effd640 1 -- 172.21.15.31:0/1658262708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 msgr2=0x564f98a342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f4effd640 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98a342b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f4effd640 1 -- 172.21.15.31:0/1658262708 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f9867f3f0 con 0x564f986466b0 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98a342b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:20.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f4effd640 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a2d350 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000c00 tx=0x7f6f440043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=303eb9a48b218bd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.161+0000 7f6f2effd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:02:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.162+0000 7f6f2effd640 1 -- 172.21.15.31:0/1658262708 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44035040 con 0x564f986466b0 2024-04-25T16:02:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.162+0000 7f6f2effd640 1 -- 172.21.15.31:0/1658262708 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6f44003ac0 con 0x564f986466b0 2024-04-25T16:02:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.162+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f400e7c80 con 0x564f986466b0 2024-04-25T16:02:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.162+0000 7f6f2effd640 1 -- 172.21.15.31:0/1658262708 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f44003cc0 con 0x564f986466b0 2024-04-25T16:02:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.162+0000 7f6f5ca0f740 10 client.25468 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.162+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564f987497d0 con 0x564f986466b0 2024-04-25T16:02:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.163+0000 7f6f2effd640 1 -- 172.21.15.31:0/1658262708 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f400e7c80 con 0x564f986466b0 2024-04-25T16:02:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.164+0000 7f6f2effd640 1 -- 172.21.15.31:0/1658262708 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6f4403c4d0 con 0x564f986466b0 2024-04-25T16:02:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.164+0000 7f6f5ca0f740 10 client.25468 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.164+0000 7f6f5ca0f740 10 client.25468 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:02:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.164+0000 7f6f5ca0f740 20 client.25468 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.164+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x564f98625f60 0x564f98a3abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.164+0000 7f6f5ca0f740 4 client.25468 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.164+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f9897b990 con 0x564f98625f60 2024-04-25T16:02:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.164+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x564f98625f60 0x564f98a3abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.165+0000 7f6f2effd640 10 client.25468 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.165+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x564f98625f60 0x564f98a3abd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9b6811b2ad2621b2 server_cookie=58d11747ab5bf484 in_seq=0 out_seq=0 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f2effd640 1 -- 172.21.15.31:0/1658262708 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564f9897b990 con 0x564f98625f60 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f2effd640 10 client.25468 handle_command_reply: tid=0 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 client.25468 shutdown 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 msgr2=0x564f98a2d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a2d350 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000c00 tx=0x7f6f440043b0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 shutdown_connections 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x564f98625f60 0x564f98a3abd0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98a342b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a2d350 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/1658262708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 0x564f98a2ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 >> 172.21.15.31:0/1658262708 conn(0x564f9865b4e0 msgr2=0x564f98a03610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f2effd640 0 client.25468 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:02:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 shutdown_connections 2024-04-25T16:02:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/1658262708 wait complete. 2024-04-25T16:02:20.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.166+0000 7f6f5ca0f740 20 client.25468 trim_cache size 0 max 16384 2024-04-25T16:02:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.193+0000 7f6f5ca0f740 1 Processor -- start 2024-04-25T16:02:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.193+0000 7f6f5ca0f740 1 -- start start 2024-04-25T16:02:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.193+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f9863e4d0 0x564f98a2bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.193+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f986466b0 0x564f98a341d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.193+0000 7f6f4f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f9863e4d0 0x564f98a2bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.193+0000 7f6f4f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f9863e4d0 0x564f98a2bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37858/0 (socket says 172.21.15.31:37858) 2024-04-25T16:02:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.193+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/2590498967 learned_addr learned my addr 172.21.15.31:0/2590498967 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f98667390 0x564f98a17640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2590498967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6f500728b0 con 0x564f9863e4d0 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2590498967 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6f500743f0 con 0x564f986466b0 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2590498967 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50072f80 con 0x564f98667390 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f4effd640 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f986466b0 0x564f98a341d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/2590498967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f98667390 msgr2=0x564f98a17640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f98667390 0x564f98a17640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f98667390 0x564f98a17640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/2590498967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f986466b0 msgr2=0x564f98a341d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f986466b0 0x564f98a341d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f4f7fe640 1 -- 172.21.15.31:0/2590498967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f98a34710 con 0x564f9863e4d0 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f98667390 0x564f98a17640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f9863e4d0 0x564f98a2bba0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6f400dca20 tx=0x7f6f40003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab155f4563f42cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.194+0000 7f6f357fa640 1 -- 172.21.15.31:0/2590498967 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f400a32c0 con 0x564f9863e4d0 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.195+0000 7f6f357fa640 1 -- 172.21.15.31:0/2590498967 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6f400a6f10 con 0x564f9863e4d0 2024-04-25T16:02:20.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.195+0000 7f6f357fa640 1 -- 172.21.15.31:0/2590498967 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f400a6b70 con 0x564f9863e4d0 2024-04-25T16:02:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2590498967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f9863e4d0 msgr2=0x564f98a2bba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f9863e4d0 0x564f98a2bba0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6f400dca20 tx=0x7f6f40003a20 comp rx=0 tx=0).stop 2024-04-25T16:02:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2590498967 shutdown_connections 2024-04-25T16:02:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f98667390 0x564f98a17640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f986466b0 0x564f98a341d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2590498967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f9863e4d0 0x564f98a2bba0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2590498967 >> 172.21.15.31:0/2590498967 conn(0x564f9865b4e0 msgr2=0x564f98a2b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2590498967 shutdown_connections 2024-04-25T16:02:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2590498967 wait complete. 2024-04-25T16:02:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.196+0000 7f6f5ca0f740 1 -- start start 2024-04-25T16:02:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.197+0000 7f6f5ca0f740 1 Processor -- start 2024-04-25T16:02:20.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.197+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 0x564f98a31030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.197+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a31570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.197+0000 7f6f5ca0f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98aa0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a31570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a31570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48638/0 (socket says 172.21.15.31:48638) 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 0x564f98a31030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 0x564f98a31030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49406/0 (socket says 172.21.15.31:49406) 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98aa0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98aa0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37866/0 (socket says 172.21.15.31:37866) 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4effd640 1 -- 172.21.15.31:0/2085345567 learned_addr learned my addr 172.21.15.31:0/2085345567 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50073d20 con 0x564f98667390 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50073650 con 0x564f9863e4d0 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6f50070940 con 0x564f986466b0 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4effd640 1 -- 172.21.15.31:0/2085345567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 msgr2=0x564f98a31030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4effd640 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 0x564f98a31030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.198+0000 7f6f4effd640 1 -- 172.21.15.31:0/2085345567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 msgr2=0x564f98aa0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.199+0000 7f6f4effd640 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98aa0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.199+0000 7f6f4effd640 1 -- 172.21.15.31:0/2085345567 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f98a31be0 con 0x564f986466b0 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.199+0000 7f6f5ca0f740 10 client.0 _ll_register_callbacks cb 0x7f6f44009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:20.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.199+0000 7f6f4ffff640 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98aa0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:20.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.199+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f400a6f10 con 0x564f986466b0 2024-04-25T16:02:20.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.199+0000 7f6f4effd640 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a31570 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000c00 tx=0x7f6f44004420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=965e466bb54c0501 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.199+0000 7f6f2effd640 10 client.25471 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:02:20.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.199+0000 7f6f2effd640 1 -- 172.21.15.31:0/2085345567 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f44035070 con 0x564f986466b0 2024-04-25T16:02:20.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.199+0000 7f6f2effd640 1 -- 172.21.15.31:0/2085345567 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6f4403c640 con 0x564f986466b0 2024-04-25T16:02:20.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f2effd640 1 -- 172.21.15.31:0/2085345567 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6f440038c0 con 0x564f986466b0 2024-04-25T16:02:20.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f5ca0f740 10 client.25471 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:20.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=27}) -- 0x564f98aa2350 con 0x564f986466b0 2024-04-25T16:02:20.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f2effd640 1 -- 172.21.15.31:0/2085345567 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f6f44059030 con 0x564f986466b0 2024-04-25T16:02:20.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f5ca0f740 10 client.25471 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:20.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f5ca0f740 10 client.25471 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:02:20.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f5ca0f740 20 client.25471 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:20.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x564f985fb3a0 0x564f98aa4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f5ca0f740 4 client.25471 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:20.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.200+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564f98975800 con 0x564f985fb3a0 2024-04-25T16:02:20.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.201+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x564f985fb3a0 0x564f98aa4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.201+0000 7f6f4f7fe640 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x564f985fb3a0 0x564f98aa4c70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2fd326cadfe3cc09 server_cookie=ade4ba0b4f546963 in_seq=0 out_seq=0 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.201+0000 7f6f2effd640 10 client.25471 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.201+0000 7f6f2effd640 1 -- 172.21.15.31:0/2085345567 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564f98975800 con 0x564f985fb3a0 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.201+0000 7f6f2effd640 10 client.25471 handle_command_reply: tid=0 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.201+0000 7f6f5ca0f740 1 client.25471 shutdown 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 msgr2=0x564f98a31570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a31570 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6f44000c00 tx=0x7f6f44004420 comp rx=0 tx=0).stop 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 shutdown_connections 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x564f985fb3a0 0x564f98aa4c70 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f98667390 0x564f98aa0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f986466b0 0x564f98a31570 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 --2- 172.21.15.31:0/2085345567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f9863e4d0 0x564f98a31030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 >> 172.21.15.31:0/2085345567 conn(0x564f9865b4e0 msgr2=0x564f98a2bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f2effd640 0 client.25471 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:02:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 shutdown_connections 2024-04-25T16:02:20.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 1 -- 172.21.15.31:0/2085345567 wait complete. 2024-04-25T16:02:20.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.202+0000 7f6f5ca0f740 20 client.25471 trim_cache size 0 max 16384 2024-04-25T16:02:20.206 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6f380afaf0 msgr2=0x7f6f380b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6f380afaf0 0x7f6f380b2000 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6f5006af70 tx=0x7f6f48009500 comp rx=0 tx=0).stop 2024-04-25T16:02:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1552470932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f501029e0 msgr2=0x7f6f5006a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f501029e0 0x7f6f5006a8f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6f501022b0 tx=0x7f6f4000ee50 comp rx=0 tx=0).stop 2024-04-25T16:02:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1552470932 shutdown_connections 2024-04-25T16:02:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6f380afaf0 0x7f6f380b2000 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6f50108920 0x7f6f501a1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f50103c20 0x7f6f5019f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 --2- 172.21.15.31:0/1552470932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6f501029e0 0x7f6f5006a8f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.204+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1552470932 >> 172.21.15.31:0/1552470932 conn(0x7f6f500fdbd0 msgr2=0x7f6f50104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.205+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1552470932 shutdown_connections 2024-04-25T16:02:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.205+0000 7f6f55b5e640 1 -- 172.21.15.31:0/1552470932 wait complete. 2024-04-25T16:02:20.221 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:02:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.337+0000 7fc144b96640 1 -- 172.21.15.31:0/2739279684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140102230 msgr2=0x7fc14010bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.337+0000 7fc144b96640 1 --2- 172.21.15.31:0/2739279684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140102230 0x7fc14010bc90 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc138009ca0 tx=0x7fc13802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.337+0000 7fc144b96640 1 -- 172.21.15.31:0/2739279684 shutdown_connections 2024-04-25T16:02:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.337+0000 7fc144b96640 1 --2- 172.21.15.31:0/2739279684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140102230 0x7fc14010bc90 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.337+0000 7fc144b96640 1 --2- 172.21.15.31:0/2739279684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc140101730 0x7fc140101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.337+0000 7fc144b96640 1 --2- 172.21.15.31:0/2739279684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc140105080 0x7fc140105490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.337+0000 7fc144b96640 1 -- 172.21.15.31:0/2739279684 >> 172.21.15.31:0/2739279684 conn(0x7fc1400fd350 msgr2=0x7fc1400ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.337+0000 7fc144b96640 1 -- 172.21.15.31:0/2739279684 shutdown_connections 2024-04-25T16:02:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.337+0000 7fc144b96640 1 -- 172.21.15.31:0/2739279684 wait complete. 2024-04-25T16:02:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc144b96640 1 Processor -- start 2024-04-25T16:02:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc144b96640 1 -- start start 2024-04-25T16:02:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc144b96640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140101730 0x7fc14019a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc144b96640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc140102230 0x7fc14019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc144b96640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc140105080 0x7fc1401a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc144b96640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc14010f660 con 0x7fc140101730 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc144b96640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc14010f4e0 con 0x7fc140102230 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc144b96640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc14010f7e0 con 0x7fc140105080 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc13e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140101730 0x7fc14019a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc13e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140101730 0x7fc14019a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37876/0 (socket says 172.21.15.31:37876) 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc13e59c640 1 -- 172.21.15.31:0/1454913459 learned_addr learned my addr 172.21.15.31:0/1454913459 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc140105080 0x7fc1401a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.338+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc140102230 0x7fc14019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc13e59c640 1 -- 172.21.15.31:0/1454913459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc140105080 msgr2=0x7fc1401a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc13e59c640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc140105080 0x7fc1401a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc13e59c640 1 -- 172.21.15.31:0/1454913459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc140102230 msgr2=0x7fc14019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc13e59c640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc140102230 0x7fc14019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc13e59c640 1 -- 172.21.15.31:0/1454913459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1401a1ce0 con 0x7fc140101730 2024-04-25T16:02:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc13e59c640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140101730 0x7fc14019a1a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc13000e9f0 tx=0x7fc13000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc12f7fe640 1 -- 172.21.15.31:0/1454913459 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13000ccc0 con 0x7fc140101730 2024-04-25T16:02:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc144b96640 1 -- 172.21.15.31:0/1454913459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1401a1fd0 con 0x7fc140101730 2024-04-25T16:02:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc12f7fe640 1 -- 172.21.15.31:0/1454913459 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1300044d0 con 0x7fc140101730 2024-04-25T16:02:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc12f7fe640 1 -- 172.21.15.31:0/1454913459 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc130010430 con 0x7fc140101730 2024-04-25T16:02:20.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.339+0000 7fc144b96640 1 -- 172.21.15.31:0/1454913459 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1401a2430 con 0x7fc140101730 2024-04-25T16:02:20.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.342+0000 7fc14ba49740 1 Processor -- start 2024-04-25T16:02:20.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.342+0000 7fc14ba49740 1 -- start start 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.342+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed5336e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.342+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed15d4d0 0x55f3ed535cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.342+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed1656b0 0x55f3ed538440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.342+0000 7fc14ba49740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f3ed48cd80 con 0x55f3ed186390 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.342+0000 7fc14ba49740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f3ed446a20 con 0x55f3ed15d4d0 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc14ba49740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f3ed488d40 con 0x55f3ed1656b0 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed5336e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed5336e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37892/0 (socket says 172.21.15.31:37892) 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13ed9d640 1 -- 172.21.15.31:0/743773317 learned_addr learned my addr 172.21.15.31:0/743773317 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/743773317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed1656b0 0x55f3ed538440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13ed9d640 1 -- 172.21.15.31:0/743773317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed1656b0 msgr2=0x55f3ed538440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/743773317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed1656b0 0x55f3ed538440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13ed9d640 1 -- 172.21.15.31:0/743773317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed15d4d0 msgr2=0x55f3ed535cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:02:20.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/743773317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed15d4d0 0x55f3ed535cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13ed9d640 1 -- 172.21.15.31:0/743773317 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3ed451c30 con 0x55f3ed186390 2024-04-25T16:02:20.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.343+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/743773317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed5336e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc138035000 tx=0x7fc13802f3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16b1cb318a9ae83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.344+0000 7fc12f7fe640 1 -- 172.21.15.31:0/1454913459 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc130003680 con 0x7fc140101730 2024-04-25T16:02:20.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.344+0000 7fc12cff9640 1 -- 172.21.15.31:0/743773317 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138031650 con 0x55f3ed186390 2024-04-25T16:02:20.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.344+0000 7fc12cff9640 1 -- 172.21.15.31:0/743773317 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc138036030 con 0x55f3ed186390 2024-04-25T16:02:20.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.344+0000 7fc12cff9640 1 -- 172.21.15.31:0/743773317 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138004640 con 0x55f3ed186390 2024-04-25T16:02:20.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc12f7fe640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1280afb70 0x7fc1280b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc13e59c640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1280afb70 0x7fc1280b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc14ba49740 1 -- 172.21.15.31:0/743773317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 msgr2=0x55f3ed5336e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc14ba49740 1 --2- 172.21.15.31:0/743773317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed5336e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fc138035000 tx=0x7fc13802f3a0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc12f7fe640 1 -- 172.21.15.31:0/1454913459 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc1300def00 con 0x7fc140101730 2024-04-25T16:02:20.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc14ba49740 1 -- 172.21.15.31:0/743773317 shutdown_connections 2024-04-25T16:02:20.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc14ba49740 1 --2- 172.21.15.31:0/743773317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed1656b0 0x55f3ed538440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc14ba49740 1 --2- 172.21.15.31:0/743773317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed15d4d0 0x55f3ed535cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc14ba49740 1 --2- 172.21.15.31:0/743773317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed5336e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.345+0000 7fc14ba49740 1 -- 172.21.15.31:0/743773317 >> 172.21.15.31:0/743773317 conn(0x55f3ed17a4e0 msgr2=0x55f3ed523fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.346+0000 7fc13e59c640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1280afb70 0x7fc1280b2080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc13000e9c0 tx=0x7fc13009bd30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.346+0000 7fc14ba49740 1 -- 172.21.15.31:0/743773317 shutdown_connections 2024-04-25T16:02:20.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.346+0000 7fc14ba49740 1 -- 172.21.15.31:0/743773317 wait complete. 2024-04-25T16:02:20.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.346+0000 7fc14ba49740 1 -- start start 2024-04-25T16:02:20.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.346+0000 7fc14ba49740 1 Processor -- start 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 0x55f3ed53b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed186390 0x55f3ed542010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc13ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc13ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37904/0 (socket says 172.21.15.31:37904) 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc13ed9d640 1 -- 172.21.15.31:0/2739760429 learned_addr learned my addr 172.21.15.31:0/2739760429 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f3ed538d80 con 0x55f3ed15d4d0 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f3ed413dc0 con 0x55f3ed1656b0 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f3ed538f00 con 0x55f3ed186390 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc14ba49740 10 client.0 _ll_register_callbacks cb 0x55f3ed42f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed186390 0x55f3ed542010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc13ed9d640 1 -- 172.21.15.31:0/2739760429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed186390 msgr2=0x55f3ed542010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed186390 0x55f3ed542010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc13ed9d640 1 -- 172.21.15.31:0/2739760429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 msgr2=0x55f3ed53b0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 0x55f3ed53b0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.347+0000 7fc13ed9d640 1 -- 172.21.15.31:0/2739760429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3ed19e3f0 con 0x55f3ed15d4d0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.348+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ab70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc138009a60 tx=0x7fc138009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=559738bf9fc1c43f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.348+0000 7fc125ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.348+0000 7fc125ffb640 1 -- 172.21.15.31:0/2739760429 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138059060 con 0x55f3ed15d4d0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.348+0000 7fc125ffb640 1 -- 172.21.15.31:0/2739760429 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc138003ba0 con 0x55f3ed15d4d0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.348+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc138036030 con 0x55f3ed15d4d0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.348+0000 7fc125ffb640 1 -- 172.21.15.31:0/2739760429 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc138003da0 con 0x55f3ed15d4d0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.348+0000 7fc14ba49740 10 client.15807 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.348+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f3ed2687d0 con 0x55f3ed15d4d0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.348+0000 7fc125ffb640 1 -- 172.21.15.31:0/2739760429 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc138036030 con 0x55f3ed15d4d0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.349+0000 7fc125ffb640 1 -- 172.21.15.31:0/2739760429 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc138043070 con 0x55f3ed15d4d0 2024-04-25T16:02:20.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.349+0000 7fc14ba49740 10 client.15807 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.349+0000 7fc14ba49740 10 client.15807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.349+0000 7fc14ba49740 20 client.15807 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.349+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f3ed144f60 0x55f3ed548440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.349+0000 7fc14ba49740 4 client.15807 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.349+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f3ed49a990 con 0x55f3ed144f60 2024-04-25T16:02:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.349+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f3ed144f60 0x55f3ed548440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.350+0000 7fc125ffb640 10 client.15807 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.350+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f3ed144f60 0x55f3ed548440 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b4b36ffa0e3a61f server_cookie=34ad0d9cc7fdffb7 in_seq=0 out_seq=0 2024-04-25T16:02:20.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.350+0000 7fc125ffb640 1 -- 172.21.15.31:0/2739760429 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f3ed49a990 con 0x55f3ed144f60 2024-04-25T16:02:20.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.350+0000 7fc125ffb640 10 client.15807 handle_command_reply: tid=0 2024-04-25T16:02:20.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.350+0000 7fc14ba49740 1 client.15807 shutdown 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.350+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed15d4d0 msgr2=0x55f3ed53ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.350+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ab70 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fc138009a60 tx=0x7fc138009540 comp rx=0 tx=0).stop 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 shutdown_connections 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f3ed144f60 0x55f3ed548440 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed186390 0x55f3ed542010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 0x55f3ed53b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2739760429 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ab70 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 >> 172.21.15.31:0/2739760429 conn(0x55f3ed17a4e0 msgr2=0x55f3ed522610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc125ffb640 0 client.15807 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 shutdown_connections 2024-04-25T16:02:20.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc14ba49740 1 -- 172.21.15.31:0/2739760429 wait complete. 2024-04-25T16:02:20.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.351+0000 7fc14ba49740 20 client.15807 trim_cache size 0 max 16384 2024-04-25T16:02:20.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc14ba49740 1 Processor -- start 2024-04-25T16:02:20.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc14ba49740 1 -- start start 2024-04-25T16:02:20.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 0x55f3ed541fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed536640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc14ba49740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401a8fc0 con 0x55f3ed186390 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc14ba49740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401aab00 con 0x55f3ed1656b0 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc14ba49740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401a9690 con 0x55f3ed15d4d0 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed536640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13dd9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 0x55f3ed541fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed536640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37914/0 (socket says 172.21.15.31:37914) 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13dd9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 0x55f3ed541fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49452/0 (socket says 172.21.15.31:49452) 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13ed9d640 1 -- 172.21.15.31:0/243251811 learned_addr learned my addr 172.21.15.31:0/243251811 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/243251811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13dd9b640 1 -- 172.21.15.31:0/243251811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed15d4d0 msgr2=0x55f3ed53ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/243251811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13dd9b640 1 -- 172.21.15.31:0/243251811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 msgr2=0x55f3ed536640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.365+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/243251811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed536640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.366+0000 7fc13dd9b640 1 -- 172.21.15.31:0/243251811 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f3ed53bcb0 con 0x55f3ed1656b0 2024-04-25T16:02:20.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.366+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/243251811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed536640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:20.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.366+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/243251811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 0x55f3ed541fa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55f3ed548b40 tx=0x7fc134009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=129b73aeed225253 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.366+0000 7fc127fff640 1 -- 172.21.15.31:0/243251811 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc134010e00 con 0x55f3ed1656b0 2024-04-25T16:02:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.366+0000 7fc127fff640 1 -- 172.21.15.31:0/243251811 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc1340035f0 con 0x55f3ed1656b0 2024-04-25T16:02:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.366+0000 7fc127fff640 1 -- 172.21.15.31:0/243251811 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13401c640 con 0x55f3ed1656b0 2024-04-25T16:02:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 -- 172.21.15.31:0/243251811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 msgr2=0x55f3ed541fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 --2- 172.21.15.31:0/243251811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 0x55f3ed541fa0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55f3ed548b40 tx=0x7fc134009f90 comp rx=0 tx=0).stop 2024-04-25T16:02:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 -- 172.21.15.31:0/243251811 shutdown_connections 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 --2- 172.21.15.31:0/243251811 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed186390 0x55f3ed536640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 --2- 172.21.15.31:0/243251811 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed1656b0 0x55f3ed541fa0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 --2- 172.21.15.31:0/243251811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 -- 172.21.15.31:0/243251811 >> 172.21.15.31:0/243251811 conn(0x55f3ed17a4e0 msgr2=0x55f3ed53a740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 -- 172.21.15.31:0/243251811 shutdown_connections 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 -- 172.21.15.31:0/243251811 wait complete. 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 -- start start 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.367+0000 7fc14ba49740 1 Processor -- start 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed1656b0 0x55f3ed53f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc14ba49740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed186390 0x55f3ed5ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc13dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed1656b0 0x55f3ed53f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc13dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed1656b0 0x55f3ed53f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37924/0 (socket says 172.21.15.31:37924) 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc13dd9b640 1 -- 172.21.15.31:0/2581943516 learned_addr learned my addr 172.21.15.31:0/2581943516 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401aa430 con 0x55f3ed1656b0 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401a9d60 con 0x55f3ed186390 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc1401a6fa0 con 0x55f3ed15d4d0 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc13dd9b640 1 -- 172.21.15.31:0/2581943516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed15d4d0 msgr2=0x55f3ed53ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc13dd9b640 1 -- 172.21.15.31:0/2581943516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed186390 msgr2=0x55f3ed5ae7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed186390 0x55f3ed5ae7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.368+0000 7fc13dd9b640 1 -- 172.21.15.31:0/2581943516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f3ed5aece0 con 0x55f3ed1656b0 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.369+0000 7fc14ba49740 10 client.0 _ll_register_callbacks cb 0x7fc138009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.369+0000 7fc13dd9b640 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed1656b0 0x55f3ed53f4a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fc134010c80 tx=0x7fc134010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9625d47cc2adc8b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.369+0000 7fc125ffb640 10 client.15813 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.369+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1340035f0 con 0x55f3ed1656b0 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.369+0000 7fc125ffb640 1 -- 172.21.15.31:0/2581943516 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13402d060 con 0x55f3ed1656b0 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.369+0000 7fc125ffb640 1 -- 172.21.15.31:0/2581943516 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc134003bb0 con 0x55f3ed1656b0 2024-04-25T16:02:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.369+0000 7fc125ffb640 1 -- 172.21.15.31:0/2581943516 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc134005050 con 0x55f3ed1656b0 2024-04-25T16:02:20.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.369+0000 7fc14ba49740 10 client.15813 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:20.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.369+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=27}) -- 0x55f3ed5afd00 con 0x55f3ed1656b0 2024-04-25T16:02:20.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc125ffb640 1 -- 172.21.15.31:0/2581943516 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc1340051f0 con 0x55f3ed1656b0 2024-04-25T16:02:20.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc14ba49740 10 client.15813 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:20.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc14ba49740 10 client.15813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc14ba49740 20 client.15813 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f3ed11a3a0 0x55f3ed5b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc14ba49740 4 client.15813 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f3ed494800 con 0x55f3ed11a3a0 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f3ed11a3a0 0x55f3ed5b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc13ed9d640 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f3ed11a3a0 0x55f3ed5b26a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f51b9c117adbdf1d server_cookie=4f9613ad4d74d0e3 in_seq=0 out_seq=0 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.370+0000 7fc125ffb640 10 client.15813 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc125ffb640 1 -- 172.21.15.31:0/2581943516 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f3ed494800 con 0x55f3ed11a3a0 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc125ffb640 10 client.15813 handle_command_reply: tid=0 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 client.15813 shutdown 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed1656b0 msgr2=0x55f3ed53f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed1656b0 0x55f3ed53f4a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fc134010c80 tx=0x7fc134010cb0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 shutdown_connections 2024-04-25T16:02:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f3ed11a3a0 0x55f3ed5b26a0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc125ffb640 0 client.15813 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f3ed186390 0x55f3ed5ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f3ed1656b0 0x55f3ed53f4a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 --2- 172.21.15.31:0/2581943516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f3ed15d4d0 0x55f3ed53ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 >> 172.21.15.31:0/2581943516 conn(0x55f3ed17a4e0 msgr2=0x55f3ed53aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 shutdown_connections 2024-04-25T16:02:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 1 -- 172.21.15.31:0/2581943516 wait complete. 2024-04-25T16:02:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.371+0000 7fc14ba49740 20 client.15813 trim_cache size 0 max 16384 2024-04-25T16:02:20.376 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 -- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1280afb70 msgr2=0x7fc1280b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1280afb70 0x7fc1280b2080 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc13000e9c0 tx=0x7fc13009bd30 comp rx=0 tx=0).stop 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 -- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140101730 msgr2=0x7fc14019a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140101730 0x7fc14019a1a0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fc13000e9f0 tx=0x7fc13000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 -- 172.21.15.31:0/1454913459 shutdown_connections 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1280afb70 0x7fc1280b2080 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc140105080 0x7fc1401a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc140102230 0x7fc14019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 --2- 172.21.15.31:0/1454913459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc140101730 0x7fc14019a1a0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 -- 172.21.15.31:0/1454913459 >> 172.21.15.31:0/1454913459 conn(0x7fc1400fd350 msgr2=0x7fc140107bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:20.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 -- 172.21.15.31:0/1454913459 shutdown_connections 2024-04-25T16:02:20.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:20.372+0000 7fc144b96640 1 -- 172.21.15.31:0/1454913459 wait complete. 2024-04-25T16:02:20.578 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:20 smithi031 ceph-mon[29140]: pgmap v382: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:20.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:20 smithi063 ceph-mon[32413]: pgmap v382: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:20.911 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:20 smithi186 ceph-mon[33258]: pgmap v382: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:21.384 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:02:21.385 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:02:21.385 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:02:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.494+0000 7fc746c42640 1 -- 172.21.15.31:0/815568976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740102a10 msgr2=0x7fc740103710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.494+0000 7fc746c42640 1 --2- 172.21.15.31:0/815568976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740102a10 0x7fc740103710 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fc734009900 tx=0x7fc73402a410 comp rx=0 tx=0).stop 2024-04-25T16:02:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.494+0000 7fc746c42640 1 -- 172.21.15.31:0/815568976 shutdown_connections 2024-04-25T16:02:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.494+0000 7fc746c42640 1 --2- 172.21.15.31:0/815568976 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc740108950 0x7fc74010ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.494+0000 7fc746c42640 1 --2- 172.21.15.31:0/815568976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc740103c50 0x7fc740104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.494+0000 7fc746c42640 1 --2- 172.21.15.31:0/815568976 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740102a10 0x7fc740103710 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.495+0000 7fc746c42640 1 -- 172.21.15.31:0/815568976 >> 172.21.15.31:0/815568976 conn(0x7fc7400fdc00 msgr2=0x7fc7401000b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.495+0000 7fc746c42640 1 -- 172.21.15.31:0/815568976 shutdown_connections 2024-04-25T16:02:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.495+0000 7fc746c42640 1 -- 172.21.15.31:0/815568976 wait complete. 2024-04-25T16:02:21.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.496+0000 7fc746c42640 1 Processor -- start 2024-04-25T16:02:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.496+0000 7fc746c42640 1 -- start start 2024-04-25T16:02:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc746c42640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc740102a10 0x7fc740198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc746c42640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740103c50 0x7fc740199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc73ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740103c50 0x7fc740199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc7449de640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc740102a10 0x7fc740198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc7449de640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc740102a10 0x7fc740198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49490/0 (socket says 172.21.15.31:49490) 2024-04-25T16:02:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc73ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740103c50 0x7fc740199490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37936/0 (socket says 172.21.15.31:37936) 2024-04-25T16:02:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc746c42640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc740108950 0x7fc7401a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc7449de640 1 -- 172.21.15.31:0/1004349344 learned_addr learned my addr 172.21.15.31:0/1004349344 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc74010e540 con 0x7fc740103c50 2024-04-25T16:02:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc74010e3c0 con 0x7fc740102a10 2024-04-25T16:02:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc74010e6c0 con 0x7fc740108950 2024-04-25T16:02:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.497+0000 7fc7451df640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc740108950 0x7fc7401a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc73ffff640 1 -- 172.21.15.31:0/1004349344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc740108950 msgr2=0x7fc7401a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc73ffff640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc740108950 0x7fc7401a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc73ffff640 1 -- 172.21.15.31:0/1004349344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc740102a10 msgr2=0x7fc740198f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc73ffff640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc740102a10 0x7fc740198f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc73ffff640 1 -- 172.21.15.31:0/1004349344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7401a09e0 con 0x7fc740103c50 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc73ffff640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740103c50 0x7fc740199490 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00c970 tx=0x7fc72c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc73dffb640 1 -- 172.21.15.31:0/1004349344 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c007bb0 con 0x7fc740103c50 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc73dffb640 1 -- 172.21.15.31:0/1004349344 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc72c007d50 con 0x7fc740103c50 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7401a0cd0 con 0x7fc740103c50 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.498+0000 7fc73dffb640 1 -- 172.21.15.31:0/1004349344 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c018780 con 0x7fc740103c50 2024-04-25T16:02:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.499+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7401a11c0 con 0x7fc740103c50 2024-04-25T16:02:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.502+0000 7fc74dbc8740 1 Processor -- start 2024-04-25T16:02:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.502+0000 7fc74dbc8740 1 -- start start 2024-04-25T16:02:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 0x5641dff48d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff49280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 0x5641dff58970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc74dbc8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5641dfeae460 con 0x5641dfb7f2f0 2024-04-25T16:02:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc74dbc8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5641dfe68590 con 0x5641dfb874d0 2024-04-25T16:02:21.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc74dbc8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5641dfeaa420 con 0x5641dfba81b0 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc73dffb640 1 -- 172.21.15.31:0/1004349344 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc72c00f450 con 0x7fc740103c50 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7449de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff49280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7449de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff49280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37952/0 (socket says 172.21.15.31:37952) 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7449de640 1 -- 172.21.15.31:0/721949025 learned_addr learned my addr 172.21.15.31:0/721949025 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7451df640 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 0x5641dff48d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc73ffff640 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 0x5641dff58970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7449de640 1 -- 172.21.15.31:0/721949025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 msgr2=0x5641dff48d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7449de640 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 0x5641dff48d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7449de640 1 -- 172.21.15.31:0/721949025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 msgr2=0x5641dff58970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7449de640 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 0x5641dff58970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7449de640 1 -- 172.21.15.31:0/721949025 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641dfe750d0 con 0x5641dfb7f2f0 2024-04-25T16:02:21.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.503+0000 7fc7451df640 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 0x5641dff48d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.504+0000 7fc7449de640 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff49280 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc734004230 tx=0x7fc734004260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29544cad959ba058 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.504+0000 7fc73dffb640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7300afbe0 0x7fc7300b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.504+0000 7fc72affd640 1 -- 172.21.15.31:0/721949025 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734035040 con 0x5641dfb7f2f0 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.504+0000 7fc72affd640 1 -- 172.21.15.31:0/721949025 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc734005450 con 0x5641dfb7f2f0 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.504+0000 7fc73dffb640 1 -- 172.21.15.31:0/1004349344 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc72c0debc0 con 0x7fc740103c50 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.504+0000 7fc72affd640 1 -- 172.21.15.31:0/721949025 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734003a90 con 0x5641dfb7f2f0 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.505+0000 7fc7451df640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7300afbe0 0x7fc7300b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.505+0000 7fc74dbc8740 1 -- 172.21.15.31:0/721949025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 msgr2=0x5641dff49280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.505+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff49280 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fc734004230 tx=0x7fc734004260 comp rx=0 tx=0).stop 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.505+0000 7fc7451df640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7300afbe0 0x7fc7300b20f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc738009fd0 tx=0x7fc738009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.506+0000 7fc74dbc8740 1 -- 172.21.15.31:0/721949025 shutdown_connections 2024-04-25T16:02:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.506+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 0x5641dff58970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.506+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff49280 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.506+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/721949025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 0x5641dff48d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.506+0000 7fc74dbc8740 1 -- 172.21.15.31:0/721949025 >> 172.21.15.31:0/721949025 conn(0x5641dfb9c300 msgr2=0x5641dff43c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.506+0000 7fc74dbc8740 1 -- 172.21.15.31:0/721949025 shutdown_connections 2024-04-25T16:02:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.506+0000 7fc74dbc8740 1 -- 172.21.15.31:0/721949025 wait complete. 2024-04-25T16:02:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.506+0000 7fc74dbc8740 1 -- start start 2024-04-25T16:02:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.506+0000 7fc74dbc8740 1 Processor -- start 2024-04-25T16:02:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff5aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff5b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfba81b0 0x5641dff62380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc74dbc8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5641dff59050 con 0x5641dfb7f2f0 2024-04-25T16:02:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc74dbc8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5641dfe35b60 con 0x5641dfba81b0 2024-04-25T16:02:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc74dbc8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5641dff591d0 con 0x5641dfb874d0 2024-04-25T16:02:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc7449de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff5aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc7449de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff5aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37964/0 (socket says 172.21.15.31:37964) 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc74dbc8740 10 client.0 _ll_register_callbacks cb 0x5641dfe5a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc7449de640 1 -- 172.21.15.31:0/235972309 learned_addr learned my addr 172.21.15.31:0/235972309 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc73ffff640 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff5b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.507+0000 7fc7451df640 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfba81b0 0x5641dff62380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc7449de640 1 -- 172.21.15.31:0/235972309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 msgr2=0x5641dff5b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc7449de640 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff5b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc7449de640 1 -- 172.21.15.31:0/235972309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfba81b0 msgr2=0x5641dff62380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc7449de640 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfba81b0 0x5641dff62380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc7449de640 1 -- 172.21.15.31:0/235972309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641dfbc0210 con 0x5641dfb7f2f0 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc73ffff640 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff5b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc7449de640 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff5aee0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc73402a930 tx=0x7fc734003a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4c28ac7ad6d7c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc71ffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc71ffff640 1 -- 172.21.15.31:0/235972309 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734052060 con 0x5641dfb7f2f0 2024-04-25T16:02:21.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc71ffff640 1 -- 172.21.15.31:0/235972309 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc734005250 con 0x5641dfb7f2f0 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.508+0000 7fc74dbc8740 1 -- 172.21.15.31:0/235972309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc734005450 con 0x5641dfb7f2f0 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.509+0000 7fc71ffff640 1 -- 172.21.15.31:0/235972309 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc734035040 con 0x5641dfb7f2f0 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.509+0000 7fc74dbc8740 10 client.15825 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.509+0000 7fc74dbc8740 1 -- 172.21.15.31:0/235972309 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5641dfc8c620 con 0x5641dfb7f2f0 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.509+0000 7fc71ffff640 1 -- 172.21.15.31:0/235972309 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc734005450 con 0x5641dfb7f2f0 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.509+0000 7fc71ffff640 1 -- 172.21.15.31:0/235972309 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc7340314a0 con 0x5641dfb7f2f0 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.510+0000 7fc74dbc8740 10 client.15825 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.510+0000 7fc74dbc8740 10 client.15825 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.510+0000 7fc74dbc8740 20 client.15825 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.510+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641dfb66d80 0x5641dff687d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.510+0000 7fc74dbc8740 4 client.15825 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:21.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.510+0000 7fc74dbc8740 1 -- 172.21.15.31:0/235972309 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5641dfebc070 con 0x5641dfb66d80 2024-04-25T16:02:21.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.510+0000 7fc73ffff640 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641dfb66d80 0x5641dff687d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.510+0000 7fc71ffff640 10 client.15825 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.511+0000 7fc73ffff640 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641dfb66d80 0x5641dff687d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=269cb03257752f5b server_cookie=aa07dc8b2c239cc7 in_seq=0 out_seq=0 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc71ffff640 1 -- 172.21.15.31:0/235972309 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5641dfebc070 con 0x5641dfb66d80 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc71ffff640 10 client.15825 handle_command_reply: tid=0 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 client.15825 shutdown 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 -- 172.21.15.31:0/235972309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 msgr2=0x5641dff5aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff5aee0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fc73402a930 tx=0x7fc734003a60 comp rx=0 tx=0).stop 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 -- 172.21.15.31:0/235972309 shutdown_connections 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641dfb66d80 0x5641dff687d0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc71ffff640 0 client.15825 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfba81b0 0x5641dff62380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff5b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/235972309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb7f2f0 0x5641dff5aee0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 -- 172.21.15.31:0/235972309 >> 172.21.15.31:0/235972309 conn(0x5641dfb9c300 msgr2=0x5641dff43250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 -- 172.21.15.31:0/235972309 shutdown_connections 2024-04-25T16:02:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 1 -- 172.21.15.31:0/235972309 wait complete. 2024-04-25T16:02:21.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.512+0000 7fc74dbc8740 20 client.15825 trim_cache size 0 max 16384 2024-04-25T16:02:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.543+0000 7fc74dbc8740 1 Processor -- start 2024-04-25T16:02:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.543+0000 7fc74dbc8740 1 -- start start 2024-04-25T16:02:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb66d80 0x5641dff633b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff59d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfba81b0 0x5641dff5aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc74dbc8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401a8db0 con 0x5641dfba81b0 2024-04-25T16:02:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc74dbc8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401aa8f0 con 0x5641dfb66d80 2024-04-25T16:02:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc74dbc8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401a9480 con 0x5641dfb874d0 2024-04-25T16:02:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc7451df640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfba81b0 0x5641dff5aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc7449de640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb66d80 0x5641dff633b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc73ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff59d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc7451df640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfba81b0 0x5641dff5aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37968/0 (socket says 172.21.15.31:37968) 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc7451df640 1 -- 172.21.15.31:0/1654857427 learned_addr learned my addr 172.21.15.31:0/1654857427 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc73ffff640 1 -- 172.21.15.31:0/1654857427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb66d80 msgr2=0x5641dff633b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc73ffff640 1 --2- 172.21.15.31:0/1654857427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb66d80 0x5641dff633b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc73ffff640 1 -- 172.21.15.31:0/1654857427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfba81b0 msgr2=0x5641dff5aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc73ffff640 1 --2- 172.21.15.31:0/1654857427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfba81b0 0x5641dff5aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.544+0000 7fc73ffff640 1 -- 172.21.15.31:0/1654857427 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641dff5a280 con 0x5641dfb874d0 2024-04-25T16:02:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.545+0000 7fc73ffff640 1 --2- 172.21.15.31:0/1654857427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff59d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00ffd0 tx=0x7fc72c0e21f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab7dfb4e0a064ba5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.545+0000 7fc72a7fc640 1 -- 172.21.15.31:0/1654857427 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c0a6b60 con 0x5641dfb874d0 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.545+0000 7fc72a7fc640 1 -- 172.21.15.31:0/1654857427 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc72c0a7750 con 0x5641dfb874d0 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.545+0000 7fc72a7fc640 1 -- 172.21.15.31:0/1654857427 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc72c00f740 con 0x5641dfb874d0 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.546+0000 7fc74dbc8740 1 -- 172.21.15.31:0/1654857427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 msgr2=0x5641dff59d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.546+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/1654857427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff59d40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00ffd0 tx=0x7fc72c0e21f0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.546+0000 7fc74dbc8740 1 -- 172.21.15.31:0/1654857427 shutdown_connections 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.546+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/1654857427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfba81b0 0x5641dff5aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.546+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/1654857427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfb874d0 0x5641dff59d40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.546+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/1654857427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb66d80 0x5641dff633b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.546+0000 7fc74dbc8740 1 -- 172.21.15.31:0/1654857427 >> 172.21.15.31:0/1654857427 conn(0x5641dfb9c300 msgr2=0x5641dff59630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.546+0000 7fc74dbc8740 1 -- 172.21.15.31:0/1654857427 shutdown_connections 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.546+0000 7fc74dbc8740 1 -- 172.21.15.31:0/1654857427 wait complete. 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.547+0000 7fc74dbc8740 1 -- start start 2024-04-25T16:02:21.548 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.547+0000 7fc74dbc8740 1 Processor -- start 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb66d80 0x5641dff5cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 0x5641dff5d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc74dbc8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 0x5641dff60100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc7449de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb66d80 0x5641dff5cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc7449de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb66d80 0x5641dff5cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37982/0 (socket says 172.21.15.31:37982) 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc73ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 0x5641dff5d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc7449de640 1 -- 172.21.15.31:0/3873991864 learned_addr learned my addr 172.21.15.31:0/3873991864 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401aa220 con 0x5641dfb66d80 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401a9b50 con 0x5641dfb874d0 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc7401a6d90 con 0x5641dfba81b0 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.548+0000 7fc7451df640 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 0x5641dff60100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.549+0000 7fc7449de640 1 -- 172.21.15.31:0/3873991864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 msgr2=0x5641dff60100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.549+0000 7fc7449de640 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 0x5641dff60100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.549+0000 7fc7449de640 1 -- 172.21.15.31:0/3873991864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 msgr2=0x5641dff5d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.549+0000 7fc7449de640 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 0x5641dff5d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.549+0000 7fc7449de640 1 -- 172.21.15.31:0/3873991864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641dff606c0 con 0x5641dfb66d80 2024-04-25T16:02:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.549+0000 7fc74dbc8740 10 client.0 _ll_register_callbacks cb 0x7fc72c0e24d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.549+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc72c0a7750 con 0x5641dfb66d80 2024-04-25T16:02:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.549+0000 7fc7449de640 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb66d80 0x5641dff5cc80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc734009230 tx=0x7fc734005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f53543daa3223a51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.550+0000 7fc71ffff640 10 client.15831 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.550+0000 7fc71ffff640 1 -- 172.21.15.31:0/3873991864 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734009520 con 0x5641dfb66d80 2024-04-25T16:02:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.550+0000 7fc71ffff640 1 -- 172.21.15.31:0/3873991864 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc73404a030 con 0x5641dfb66d80 2024-04-25T16:02:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.550+0000 7fc71ffff640 1 -- 172.21.15.31:0/3873991864 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc734003e60 con 0x5641dfb66d80 2024-04-25T16:02:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.550+0000 7fc74dbc8740 10 client.15831 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.550+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5641dff61870 con 0x5641dfb66d80 2024-04-25T16:02:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc71ffff640 1 -- 172.21.15.31:0/3873991864 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc734031ab0 con 0x5641dfb66d80 2024-04-25T16:02:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc74dbc8740 10 client.15831 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc74dbc8740 10 client.15831 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc74dbc8740 20 client.15831 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641dfb3c1c0 0x5641dffd2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc74dbc8740 4 client.15831 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:02:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5641dfeb5ee0 con 0x5641dfb3c1c0 2024-04-25T16:02:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc73ffff640 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641dfb3c1c0 0x5641dffd2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc71ffff640 10 client.15831 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.551+0000 7fc73ffff640 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641dfb3c1c0 0x5641dffd2cd0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83e8ea0c511a1e51 server_cookie=c49d718ae6a00bb3 in_seq=0 out_seq=0 2024-04-25T16:02:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.553+0000 7f9570c84640 1 -- 172.21.15.31:0/3676404357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f956c1029e0 msgr2=0x7f956c1036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.553+0000 7f9570c84640 1 --2- 172.21.15.31:0/3676404357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f956c1029e0 0x7f956c1036e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f955c00d960 tx=0x7f955c02a0a0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.553+0000 7f9570c84640 1 -- 172.21.15.31:0/3676404357 shutdown_connections 2024-04-25T16:02:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.553+0000 7f9570c84640 1 --2- 172.21.15.31:0/3676404357 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c108920 0x7f956c10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.553+0000 7f9570c84640 1 --2- 172.21.15.31:0/3676404357 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f956c103c20 0x7f956c104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.553+0000 7f9570c84640 1 --2- 172.21.15.31:0/3676404357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f956c1029e0 0x7f956c1036e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.553+0000 7f9570c84640 1 -- 172.21.15.31:0/3676404357 >> 172.21.15.31:0/3676404357 conn(0x7f956c0fdbd0 msgr2=0x7f956c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.553+0000 7f9570c84640 1 -- 172.21.15.31:0/3676404357 shutdown_connections 2024-04-25T16:02:21.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.553+0000 7f9570c84640 1 -- 172.21.15.31:0/3676404357 wait complete. 2024-04-25T16:02:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.554+0000 7f9570c84640 1 Processor -- start 2024-04-25T16:02:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.554+0000 7f9570c84640 1 -- start start 2024-04-25T16:02:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.554+0000 7f9570c84640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f956c1029e0 0x7f956c1a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.554+0000 7f9570c84640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f956c103c20 0x7f956c1a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.554+0000 7f9570c84640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c108920 0x7f956c1a7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.554+0000 7f9570c84640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f956c10d3a0 con 0x7f956c108920 2024-04-25T16:02:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.554+0000 7f9570c84640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f956c10d220 con 0x7f956c103c20 2024-04-25T16:02:21.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.554+0000 7f9570c84640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f956c10d520 con 0x7f956c1029e0 2024-04-25T16:02:21.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.554+0000 7f956ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c108920 0x7f956c1a7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f9563fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f956c103c20 0x7f956c1a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.555+0000 7fc71ffff640 1 -- 172.21.15.31:0/3873991864 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5641dfeb5ee0 con 0x5641dfb3c1c0 2024-04-25T16:02:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.555+0000 7fc71ffff640 10 client.15831 handle_command_reply: tid=0 2024-04-25T16:02:21.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.555+0000 7fc74dbc8740 1 client.15831 shutdown 2024-04-25T16:02:21.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f9563fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f956c103c20 0x7f956c1a1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49538/0 (socket says 172.21.15.31:49538) 2024-04-25T16:02:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f9563fff640 1 -- 172.21.15.31:0/2490182676 learned_addr learned my addr 172.21.15.31:0/2490182676 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f956a59c640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f956c1029e0 0x7f956c1a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f9563fff640 1 -- 172.21.15.31:0/2490182676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f956c1029e0 msgr2=0x7f956c1a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f9563fff640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f956c1029e0 0x7f956c1a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f9563fff640 1 -- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c108920 msgr2=0x7f956c1a7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f9563fff640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c108920 0x7f956c1a7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f9563fff640 1 -- 172.21.15.31:0/2490182676 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956c1a8660 con 0x7f956c103c20 2024-04-25T16:02:21.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.555+0000 7f956ad9d640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c108920 0x7f956c1a7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb66d80 msgr2=0x5641dff5cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb66d80 0x5641dff5cc80 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc734009230 tx=0x7fc734005d50 comp rx=0 tx=0).stop 2024-04-25T16:02:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 shutdown_connections 2024-04-25T16:02:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641dfb3c1c0 0x5641dffd2cd0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc71ffff640 0 client.15831 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641dfba81b0 0x5641dff60100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641dfb874d0 0x5641dff5d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 --2- 172.21.15.31:0/3873991864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641dfb66d80 0x5641dff5cc80 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 >> 172.21.15.31:0/3873991864 conn(0x5641dfb9c300 msgr2=0x5641dff5aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.556+0000 7f9563fff640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f956c103c20 0x7f956c1a1060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9554009a10 tx=0x7f9554009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 shutdown_connections 2024-04-25T16:02:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 1 -- 172.21.15.31:0/3873991864 wait complete. 2024-04-25T16:02:21.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.556+0000 7f95637fe640 1 -- 172.21.15.31:0/2490182676 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955400cc60 con 0x7f956c103c20 2024-04-25T16:02:21.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.556+0000 7fc74dbc8740 20 client.15831 trim_cache size 0 max 16384 2024-04-25T16:02:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.556+0000 7f9570c84640 1 -- 172.21.15.31:0/2490182676 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956c1a8950 con 0x7f956c103c20 2024-04-25T16:02:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.556+0000 7f95637fe640 1 -- 172.21.15.31:0/2490182676 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f955400ce00 con 0x7f956c103c20 2024-04-25T16:02:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.556+0000 7f95637fe640 1 -- 172.21.15.31:0/2490182676 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9554017400 con 0x7f956c103c20 2024-04-25T16:02:21.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.556+0000 7f9570c84640 1 -- 172.21.15.31:0/2490182676 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f956c1a8e50 con 0x7f956c103c20 2024-04-25T16:02:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:02:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:02:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:02:21.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:02:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.557+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7300afbe0 msgr2=0x7fc7300b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.557+0000 7fc746c42640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7300afbe0 0x7fc7300b20f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc738009fd0 tx=0x7fc738009500 comp rx=0 tx=0).stop 2024-04-25T16:02:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.557+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740103c50 msgr2=0x7fc740199490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.557+0000 7fc746c42640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740103c50 0x7fc740199490 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00c970 tx=0x7fc72c00ce30 comp rx=0 tx=0).stop 2024-04-25T16:02:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.558+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 shutdown_connections 2024-04-25T16:02:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.558+0000 7fc746c42640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7300afbe0 0x7fc7300b20f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.558+0000 7fc746c42640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc740108950 0x7fc7401a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.558+0000 7fc746c42640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc740103c50 0x7fc740199490 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.558+0000 7fc746c42640 1 --2- 172.21.15.31:0/1004349344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc740102a10 0x7fc740198f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.558+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 >> 172.21.15.31:0/1004349344 conn(0x7fc7400fdc00 msgr2=0x7fc740100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.558+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 shutdown_connections 2024-04-25T16:02:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:21.558+0000 7fc746c42640 1 -- 172.21.15.31:0/1004349344 wait complete. 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.560+0000 7f9577c0c740 1 Processor -- start 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.560+0000 7f9577c0c740 1 -- start start 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae89c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4d2550 0x5606ae89ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4da730 0x5606ae8a3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f9577c0c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5606ae801b00 con 0x5606ae4fb410 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f9577c0c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5606ae79d920 con 0x5606ae4d2550 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f9577c0c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5606ae7fdac0 con 0x5606ae4da730 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f956ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae89c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f956ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae89c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38010/0 (socket says 172.21.15.31:38010) 2024-04-25T16:02:21.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f956ad9d640 1 -- 172.21.15.31:0/4014347831 learned_addr learned my addr 172.21.15.31:0/4014347831 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f956a59c640 1 --2- 172.21.15.31:0/4014347831 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4d2550 0x5606ae89ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f95637fe640 1 -- 172.21.15.31:0/2490182676 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f95540046c0 con 0x7f956c103c20 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f9563fff640 1 --2- 172.21.15.31:0/4014347831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4da730 0x5606ae8a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f956ad9d640 1 -- 172.21.15.31:0/4014347831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4da730 msgr2=0x5606ae8a3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f956ad9d640 1 --2- 172.21.15.31:0/4014347831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4da730 0x5606ae8a3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f956ad9d640 1 -- 172.21.15.31:0/4014347831 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4d2550 msgr2=0x5606ae89ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f956ad9d640 1 --2- 172.21.15.31:0/4014347831 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4d2550 0x5606ae89ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.561+0000 7f956ad9d640 1 -- 172.21.15.31:0/4014347831 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606ae5db7e0 con 0x5606ae4fb410 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.562+0000 7f956ad9d640 1 --2- 172.21.15.31:0/4014347831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae89c4a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9564009fd0 tx=0x7f956400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa0eba4cb1a4c22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.562+0000 7f9560ff9640 1 -- 172.21.15.31:0/4014347831 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f956401d070 con 0x5606ae4fb410 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.562+0000 7f9560ff9640 1 -- 172.21.15.31:0/4014347831 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9564003e20 con 0x5606ae4fb410 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.562+0000 7f9560ff9640 1 -- 172.21.15.31:0/4014347831 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95640047f0 con 0x5606ae4fb410 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.562+0000 7f95637fe640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95580afaf0 0x7f95580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.563+0000 7f956a59c640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95580afaf0 0x7f95580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.563+0000 7f95637fe640 1 -- 172.21.15.31:0/2490182676 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f95540deb10 con 0x7f956c103c20 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.563+0000 7f9577c0c740 1 -- 172.21.15.31:0/4014347831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 msgr2=0x5606ae89c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.563+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4014347831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae89c4a0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f9564009fd0 tx=0x7f956400ec80 comp rx=0 tx=0).stop 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.563+0000 7f956a59c640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95580afaf0 0x7f95580b2000 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f955c00d930 tx=0x7f955c032e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.564+0000 7f9577c0c740 1 -- 172.21.15.31:0/4014347831 shutdown_connections 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.564+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4014347831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4da730 0x5606ae8a3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.564+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4014347831 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4d2550 0x5606ae89ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.564+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4014347831 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae89c4a0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.564+0000 7f9577c0c740 1 -- 172.21.15.31:0/4014347831 >> 172.21.15.31:0/4014347831 conn(0x5606ae4ef560 msgr2=0x5606ae897e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.564+0000 7f9577c0c740 1 -- 172.21.15.31:0/4014347831 shutdown_connections 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.564+0000 7f9577c0c740 1 -- 172.21.15.31:0/4014347831 wait complete. 2024-04-25T16:02:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.564+0000 7f9577c0c740 1 -- start start 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.565+0000 7f9577c0c740 1 Processor -- start 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.565+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4d2550 0x5606ae8a6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.565+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8a68a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.565+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae8ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f9563fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8a68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f9563fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8a68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49542/0 (socket says 172.21.15.31:49542) 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae8ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956ad9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae8ad800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38022/0 (socket says 172.21.15.31:38022) 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4d2550 0x5606ae8a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f9577c0c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5606ae8a4370 con 0x5606ae4fb410 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f9577c0c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5606ae79b5d0 con 0x5606ae4da730 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f9577c0c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5606ae8a44f0 con 0x5606ae4d2550 2024-04-25T16:02:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4d2550 0x5606ae8a6360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48766/0 (socket says 172.21.15.31:48766) 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956a59c640 1 -- 172.21.15.31:0/1432829629 learned_addr learned my addr 172.21.15.31:0/1432829629 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f9577c0c740 10 client.0 _ll_register_callbacks cb 0x5606ae789080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956a59c640 1 -- 172.21.15.31:0/1432829629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 msgr2=0x5606ae8a68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956a59c640 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8a68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956a59c640 1 -- 172.21.15.31:0/1432829629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 msgr2=0x5606ae8ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956a59c640 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae8ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956a59c640 1 -- 172.21.15.31:0/1432829629 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606ae63bf70 con 0x5606ae4d2550 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f9563fff640 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.566+0000 7f956ad9d640 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae8ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.567+0000 7f956a59c640 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4d2550 0x5606ae8a6360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f955c005eb0 tx=0x7f955c02a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b921b575c3a5dc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.567+0000 7f9551ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.567+0000 7f9551ffb640 1 -- 172.21.15.31:0/1432829629 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c037040 con 0x5606ae4d2550 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.567+0000 7f9551ffb640 1 -- 172.21.15.31:0/1432829629 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f955c030cb0 con 0x5606ae4d2550 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.567+0000 7f9577c0c740 1 -- 172.21.15.31:0/1432829629 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9564003e20 con 0x5606ae4d2550 2024-04-25T16:02:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.567+0000 7f9551ffb640 1 -- 172.21.15.31:0/1432829629 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f955c032440 con 0x5606ae4d2550 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.567+0000 7f9577c0c740 10 client.25510 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.567+0000 7f9577c0c740 1 -- 172.21.15.31:0/1432829629 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5606ae4d3170 con 0x5606ae4d2550 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.567+0000 7f9551ffb640 1 -- 172.21.15.31:0/1432829629 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9564003e20 con 0x5606ae4d2550 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.568+0000 7f9551ffb640 1 -- 172.21.15.31:0/1432829629 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f955c003860 con 0x5606ae4d2550 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.568+0000 7f9577c0c740 10 client.25510 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.568+0000 7f9577c0c740 10 client.25510 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.568+0000 7f9577c0c740 20 client.25510 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.568+0000 7f9577c0c740 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5606ae4ba1f0 0x5606ae8b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.568+0000 7f9577c0c740 4 client.25510 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.568+0000 7f9577c0c740 1 -- 172.21.15.31:0/1432829629 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5606ae80f710 con 0x5606ae4ba1f0 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.568+0000 7f9563fff640 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5606ae4ba1f0 0x5606ae8b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.569+0000 7f9551ffb640 10 client.25510 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.569+0000 7f9563fff640 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5606ae4ba1f0 0x5606ae8b3d60 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e9f77f1f47bf034 server_cookie=3e6646905fb0701 in_seq=0 out_seq=0 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.569+0000 7f9551ffb640 1 -- 172.21.15.31:0/1432829629 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5606ae80f710 con 0x5606ae4ba1f0 2024-04-25T16:02:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.569+0000 7f9551ffb640 10 client.25510 handle_command_reply: tid=0 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.569+0000 7f9577c0c740 1 client.25510 shutdown 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 -- 172.21.15.31:0/1432829629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4d2550 msgr2=0x5606ae8a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4d2550 0x5606ae8a6360 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f955c005eb0 tx=0x7f955c02a0a0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 -- 172.21.15.31:0/1432829629 shutdown_connections 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5606ae4ba1f0 0x5606ae8b3d60 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9551ffb640 0 client.25510 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4fb410 0x5606ae8ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8a68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 --2- 172.21.15.31:0/1432829629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4d2550 0x5606ae8a6360 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 -- 172.21.15.31:0/1432829629 >> 172.21.15.31:0/1432829629 conn(0x5606ae4ef560 msgr2=0x5606ae896ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 -- 172.21.15.31:0/1432829629 shutdown_connections 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 1 -- 172.21.15.31:0/1432829629 wait complete. 2024-04-25T16:02:21.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.570+0000 7f9577c0c740 20 client.25510 trim_cache size 0 max 16384 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.607+0000 7f9577c0c740 1 Processor -- start 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.607+0000 7f9577c0c740 1 -- start start 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.607+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8adcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f9563fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f9563fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38038/0 (socket says 172.21.15.31:38038) 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f9563fff640 1 -- 172.21.15.31:0/3539623722 learned_addr learned my addr 172.21.15.31:0/3539623722 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f9577c0c740 1 -- 172.21.15.31:0/3539623722 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1b18a0 con 0x5606ae4d2550 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f9577c0c740 1 -- 172.21.15.31:0/3539623722 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1b33e0 con 0x5606ae4da730 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f9577c0c740 1 -- 172.21.15.31:0/3539623722 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1b1f70 con 0x5606ae4ba1f0 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f956ad9d640 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8adcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f956a59c640 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f956a59c640 1 -- 172.21.15.31:0/3539623722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 msgr2=0x5606ae8adcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f956a59c640 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8adcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f956a59c640 1 -- 172.21.15.31:0/3539623722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 msgr2=0x5606ae8ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f956a59c640 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f956a59c640 1 -- 172.21.15.31:0/3539623722 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606ae8a3ba0 con 0x5606ae4ba1f0 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.608+0000 7f9563fff640 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.609+0000 7f956a59c640 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8a50c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f955c003d00 tx=0x7f955c00d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d10c056b1442b8d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.609+0000 7f9553fff640 1 -- 172.21.15.31:0/3539623722 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c030cb0 con 0x5606ae4ba1f0 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.609+0000 7f9553fff640 1 -- 172.21.15.31:0/3539623722 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f955c046040 con 0x5606ae4ba1f0 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.609+0000 7f9553fff640 1 -- 172.21.15.31:0/3539623722 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955c03d440 con 0x5606ae4ba1f0 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.610+0000 7f9577c0c740 1 -- 172.21.15.31:0/3539623722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 msgr2=0x5606ae8a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.610+0000 7f9577c0c740 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8a50c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f955c003d00 tx=0x7f955c00d6b0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.610+0000 7f9577c0c740 1 -- 172.21.15.31:0/3539623722 shutdown_connections 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.610+0000 7f9577c0c740 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8adcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.610+0000 7f9577c0c740 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.610+0000 7f9577c0c740 1 --2- 172.21.15.31:0/3539623722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8a50c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.610+0000 7f9577c0c740 1 -- 172.21.15.31:0/3539623722 >> 172.21.15.31:0/3539623722 conn(0x5606ae4ef560 msgr2=0x5606ae8a49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.610+0000 7f9577c0c740 1 -- 172.21.15.31:0/3539623722 shutdown_connections 2024-04-25T16:02:21.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.611+0000 7f9577c0c740 1 -- 172.21.15.31:0/3539623722 wait complete. 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.611+0000 7f9577c0c740 1 -- start start 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.611+0000 7f9577c0c740 1 Processor -- start 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ab5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9577c0c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8ac520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9563fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ab5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9563fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ab5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38042/0 (socket says 172.21.15.31:38042) 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9563fff640 1 -- 172.21.15.31:0/4007526925 learned_addr learned my addr 172.21.15.31:0/4007526925 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1b2d10 con 0x5606ae4d2550 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1b2640 con 0x5606ae4da730 2024-04-25T16:02:21.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f956c1af880 con 0x5606ae4ba1f0 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f956a59c640 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9577c0c740 10 client.0 _ll_register_callbacks cb 0x7f9554005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9563fff640 1 -- 172.21.15.31:0/4007526925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 msgr2=0x5606ae8ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9563fff640 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8ab0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f956ad9d640 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8ac520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.612+0000 7f9563fff640 1 -- 172.21.15.31:0/4007526925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 msgr2=0x5606ae8ac520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9563fff640 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8ac520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9563fff640 1 -- 172.21.15.31:0/4007526925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606ae8acae0 con 0x5606ae4d2550 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f955c046040 con 0x5606ae4d2550 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f956a59c640 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8ab0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9563fff640 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ab5f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f95540dc670 tx=0x7f95540de040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd8b79a34992f571 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9551ffb640 10 client.15849 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9551ffb640 1 -- 172.21.15.31:0/4007526925 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95540a6460 con 0x5606ae4d2550 2024-04-25T16:02:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9551ffb640 1 -- 172.21.15.31:0/4007526925 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f95540dcae0 con 0x5606ae4d2550 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9551ffb640 1 -- 172.21.15.31:0/4007526925 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f95540dcce0 con 0x5606ae4d2550 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9577c0c740 10 client.15849 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.613+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5606ae8add10 con 0x5606ae4d2550 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.614+0000 7f9551ffb640 1 -- 172.21.15.31:0/4007526925 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f95540a2970 con 0x5606ae4d2550 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.614+0000 7f9577c0c740 10 client.15849 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.614+0000 7f9577c0c740 10 client.15849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.614+0000 7f9577c0c740 20 client.15849 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.614+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5606ae4fb410 0x5606ae91e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.614+0000 7f9577c0c740 4 client.15849 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.614+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5606ae809580 con 0x5606ae4fb410 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.614+0000 7f956a59c640 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5606ae4fb410 0x5606ae91e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.615+0000 7f9551ffb640 10 client.15849 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.615+0000 7f956a59c640 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5606ae4fb410 0x5606ae91e480 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61387898800a918d server_cookie=f41ee35cdf4a1c6d in_seq=0 out_seq=0 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.616+0000 7f9551ffb640 1 -- 172.21.15.31:0/4007526925 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5606ae809580 con 0x5606ae4fb410 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.616+0000 7f9551ffb640 10 client.15849 handle_command_reply: tid=0 2024-04-25T16:02:21.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.616+0000 7f9577c0c740 1 client.15849 shutdown 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.616+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 msgr2=0x5606ae8ab5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.616+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ab5f0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f95540dc670 tx=0x7f95540de040 comp rx=0 tx=0).stop 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.616+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 shutdown_connections 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5606ae4fb410 0x5606ae91e480 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5606ae4da730 0x5606ae8ac520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5606ae4d2550 0x5606ae8ab5f0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9551ffb640 0 client.15849 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9577c0c740 1 --2- 172.21.15.31:0/4007526925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5606ae4ba1f0 0x5606ae8ab0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 >> 172.21.15.31:0/4007526925 conn(0x5606ae4ef560 msgr2=0x5606ae8a66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 shutdown_connections 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9577c0c740 1 -- 172.21.15.31:0/4007526925 wait complete. 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9577c0c740 20 client.15849 trim_cache size 0 max 16384 2024-04-25T16:02:21.621 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:02:21.622 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:02:21.622 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "d28cc0f9-9eb8-45ab-aa32-0c8f32c41b41", 2024-04-25T16:02:21.622 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:02:21.622 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.617+0000 7f9570c84640 1 -- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95580afaf0 msgr2=0x7f95580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95580afaf0 0x7f95580b2000 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f955c00d930 tx=0x7f955c032e50 comp rx=0 tx=0).stop 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 -- 172.21.15.31:0/2490182676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f956c103c20 msgr2=0x7f956c1a1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f956c103c20 0x7f956c1a1060 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9554009a10 tx=0x7f9554009ed0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 -- 172.21.15.31:0/2490182676 shutdown_connections 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95580afaf0 0x7f95580b2000 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f956c108920 0x7f956c1a7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f956c103c20 0x7f956c1a1060 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 --2- 172.21.15.31:0/2490182676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f956c1029e0 0x7f956c1a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 -- 172.21.15.31:0/2490182676 >> 172.21.15.31:0/2490182676 conn(0x7f956c0fdbd0 msgr2=0x7f956c104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 -- 172.21.15.31:0/2490182676 shutdown_connections 2024-04-25T16:02:21.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.618+0000 7f9570c84640 1 -- 172.21.15.31:0/2490182676 wait complete. 2024-04-25T16:02:21.634 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:02:21.634 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:02:21.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.848+0000 7f96db0d8640 1 -- 172.21.15.31:0/283184858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 msgr2=0x7f96d410ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.848+0000 7f96db0d8640 1 --2- 172.21.15.31:0/283184858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 0x7f96d410ad40 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009ca0 tx=0x7f96cc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.849+0000 7f96db0d8640 1 -- 172.21.15.31:0/283184858 shutdown_connections 2024-04-25T16:02:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.849+0000 7f96db0d8640 1 --2- 172.21.15.31:0/283184858 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 0x7f96d410ad40 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.849+0000 7f96db0d8640 1 --2- 172.21.15.31:0/283184858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96d4103c20 0x7f96d4104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.849+0000 7f96db0d8640 1 --2- 172.21.15.31:0/283184858 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96d41029e0 0x7f96d41036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.849+0000 7f96db0d8640 1 -- 172.21.15.31:0/283184858 >> 172.21.15.31:0/283184858 conn(0x7f96d40fdbd0 msgr2=0x7f96d4100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.849+0000 7f96db0d8640 1 -- 172.21.15.31:0/283184858 shutdown_connections 2024-04-25T16:02:21.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.849+0000 7f96db0d8640 1 -- 172.21.15.31:0/283184858 wait complete. 2024-04-25T16:02:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96db0d8640 1 Processor -- start 2024-04-25T16:02:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96db0d8640 1 -- start start 2024-04-25T16:02:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96db0d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96d41029e0 0x7f96d41993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96db0d8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96d4103c20 0x7f96d4199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96db0d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 0x7f96d41a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96d9675640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 0x7f96d41a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96d9675640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 0x7f96d41a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38058/0 (socket says 172.21.15.31:38058) 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96d9675640 1 -- 172.21.15.31:0/149028577 learned_addr learned my addr 172.21.15.31:0/149028577 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96db0d8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96d410e560 con 0x7f96d4108920 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f96d410e3e0 con 0x7f96d41029e0 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f96d410e6e0 con 0x7f96d4103c20 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96d8e74640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96d41029e0 0x7f96d41993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.850+0000 7f96d3fff640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96d4103c20 0x7f96d4199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d8e74640 1 -- 172.21.15.31:0/149028577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96d4103c20 msgr2=0x7f96d4199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d8e74640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96d4103c20 0x7f96d4199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d8e74640 1 -- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 msgr2=0x7f96d41a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d8e74640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 0x7f96d41a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d8e74640 1 -- 172.21.15.31:0/149028577 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d41a0f10 con 0x7f96d41029e0 2024-04-25T16:02:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d9675640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 0x7f96d41a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d3fff640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96d4103c20 0x7f96d4199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d8e74640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96d41029e0 0x7f96d41993d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f96d41022b0 tx=0x7f96c800ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d1ffb640 1 -- 172.21.15.31:0/149028577 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c800cc40 con 0x7f96d41029e0 2024-04-25T16:02:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d4069ef0 con 0x7f96d41029e0 2024-04-25T16:02:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d1ffb640 1 -- 172.21.15.31:0/149028577 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96c800cde0 con 0x7f96d41029e0 2024-04-25T16:02:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96d1ffb640 1 -- 172.21.15.31:0/149028577 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c8010430 con 0x7f96d41029e0 2024-04-25T16:02:21.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.851+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96d406a490 con 0x7f96d41029e0 2024-04-25T16:02:21.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.855+0000 7f96e1f84740 1 Processor -- start 2024-04-25T16:02:21.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.855+0000 7f96e1f84740 1 -- start start 2024-04-25T16:02:21.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.855+0000 7f96d1ffb640 1 -- 172.21.15.31:0/149028577 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f96c8003ed0 con 0x7f96d41029e0 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.855+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 0x562089b0ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.855+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.855+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b0cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.855+0000 7f96e1f84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562089a70d80 con 0x5620897414d0 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.855+0000 7f96e1f84740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562089a2a8b0 con 0x5620897496b0 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.855+0000 7f96e1f84740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562089a6cd40 con 0x56208976a390 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d8e74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d9675640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 0x562089b0ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d8e74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38070/0 (socket says 172.21.15.31:38070) 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d9675640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 0x562089b0ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48814/0 (socket says 172.21.15.31:48814) 2024-04-25T16:02:21.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d8e74640 1 -- 172.21.15.31:0/2348299022 learned_addr learned my addr 172.21.15.31:0/2348299022 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d3fff640 1 --2- 172.21.15.31:0/2348299022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b0cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d3fff640 1 -- 172.21.15.31:0/2348299022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 msgr2=0x562089b0ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d3fff640 1 --2- 172.21.15.31:0/2348299022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 0x562089b0ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d3fff640 1 -- 172.21.15.31:0/2348299022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 msgr2=0x562089b0bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d3fff640 1 --2- 172.21.15.31:0/2348299022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d3fff640 1 -- 172.21.15.31:0/2348299022 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620899fba40 con 0x5620897496b0 2024-04-25T16:02:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d8e74640 1 --2- 172.21.15.31:0/2348299022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d3fff640 1 --2- 172.21.15.31:0/2348299022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b0cc70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009f20 tx=0x7f96c0005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=467c990ca256f203 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.856+0000 7f96d1ffb640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96c40afb70 0x7f96c40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.857+0000 7f96d9675640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96c40afb70 0x7f96c40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.857+0000 7f96d1ffb640 1 -- 172.21.15.31:0/149028577 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f96c80defe0 con 0x7f96d41029e0 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.857+0000 7f96beffd640 1 -- 172.21.15.31:0/2348299022 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c0004030 con 0x5620897496b0 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.857+0000 7f96beffd640 1 -- 172.21.15.31:0/2348299022 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96c00054d0 con 0x5620897496b0 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.857+0000 7f96beffd640 1 -- 172.21.15.31:0/2348299022 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c0004ce0 con 0x5620897496b0 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96e1f84740 1 -- 172.21.15.31:0/2348299022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 msgr2=0x562089b0cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2348299022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b0cc70 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009f20 tx=0x7f96c0005f90 comp rx=0 tx=0).stop 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96d9675640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96c40afb70 0x7f96c40b2080 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x562089b0cab0 tx=0x7f96cc031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96e1f84740 1 -- 172.21.15.31:0/2348299022 shutdown_connections 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2348299022 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b0cc70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2348299022 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2348299022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 0x562089b0ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96e1f84740 1 -- 172.21.15.31:0/2348299022 >> 172.21.15.31:0/2348299022 conn(0x56208975e4e0 msgr2=0x562089b07270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96e1f84740 1 -- 172.21.15.31:0/2348299022 shutdown_connections 2024-04-25T16:02:21.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.858+0000 7f96e1f84740 1 -- 172.21.15.31:0/2348299022 wait complete. 2024-04-25T16:02:21.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.859+0000 7f96e1f84740 1 -- start start 2024-04-25T16:02:21.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.859+0000 7f96e1f84740 1 Processor -- start 2024-04-25T16:02:21.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b19d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b1a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 0x562089b21210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96e1f84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562089b0d780 con 0x5620897414d0 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96e1f84740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5620899f7d10 con 0x5620897496b0 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96e1f84740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562089b0d900 con 0x56208976a390 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96e1f84740 10 client.0 _ll_register_callbacks cb 0x562089a24210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d8e74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b19d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d8e74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b19d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38082/0 (socket says 172.21.15.31:38082) 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d8e74640 1 -- 172.21.15.31:0/1229516428 learned_addr learned my addr 172.21.15.31:0/1229516428 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d3fff640 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b1a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d9675640 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 0x562089b21210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d8e74640 1 -- 172.21.15.31:0/1229516428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 msgr2=0x562089b21210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d8e74640 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 0x562089b21210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d8e74640 1 -- 172.21.15.31:0/1229516428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 msgr2=0x562089b1a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d8e74640 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b1a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d8e74640 1 -- 172.21.15.31:0/1229516428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56208984c7d0 con 0x5620897414d0 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d3fff640 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b1a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96e1f84740 1 -- 172.21.15.31:0/1229516428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96c00054d0 con 0x5620897414d0 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96d8e74640 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b19d70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f96c80a3a30 tx=0x7f96c80a2c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884e3919edd726e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.860+0000 7f96b3fff640 10 client.15861 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.861+0000 7f96b3fff640 1 -- 172.21.15.31:0/1229516428 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c80a6de0 con 0x5620897414d0 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.861+0000 7f96b3fff640 1 -- 172.21.15.31:0/1229516428 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f96c80e1110 con 0x5620897414d0 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.861+0000 7f96b3fff640 1 -- 172.21.15.31:0/1229516428 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f96c80a6c00 con 0x5620897414d0 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.861+0000 7f96e1f84740 10 client.15861 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.861+0000 7f96e1f84740 1 -- 172.21.15.31:0/1229516428 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5620897823f0 con 0x5620897414d0 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96b3fff640 1 -- 172.21.15.31:0/1229516428 <== mon.0 v2:172.21.15.31:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f96c80afe60 con 0x5620897414d0 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96e1f84740 10 client.15861 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:21.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96e1f84740 10 client.15861 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96e1f84740 20 client.15861 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96e1f84740 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562089728f60 0x562089b27600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96e1f84740 4 client.15861 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96e1f84740 1 -- 172.21.15.31:0/1229516428 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562089a7e990 con 0x562089728f60 2024-04-25T16:02:21.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96d3fff640 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562089728f60 0x562089b27600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96b3fff640 10 client.15861 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.862+0000 7f96d3fff640 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562089728f60 0x562089b27600 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9d3b61d2d66dae5 server_cookie=e2d3e4d2ff5bd8cc in_seq=0 out_seq=0 2024-04-25T16:02:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96b3fff640 1 -- 172.21.15.31:0/1229516428 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562089a7e990 con 0x562089728f60 2024-04-25T16:02:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96b3fff640 10 client.15861 handle_command_reply: tid=0 2024-04-25T16:02:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96e1f84740 1 client.15861 shutdown 2024-04-25T16:02:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96e1f84740 1 -- 172.21.15.31:0/1229516428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 msgr2=0x562089b19d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96e1f84740 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b19d70 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f96c80a3a30 tx=0x7f96c80a2c30 comp rx=0 tx=0).stop 2024-04-25T16:02:21.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96e1f84740 1 -- 172.21.15.31:0/1229516428 shutdown_connections 2024-04-25T16:02:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96e1f84740 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562089728f60 0x562089b27600 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96e1f84740 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56208976a390 0x562089b21210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96e1f84740 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5620897496b0 0x562089b1a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96b3fff640 0 client.15861 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96e1f84740 1 --2- 172.21.15.31:0/1229516428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b19d70 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.863+0000 7f96e1f84740 1 -- 172.21.15.31:0/1229516428 >> 172.21.15.31:0/1229516428 conn(0x56208975e4e0 msgr2=0x562089b063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.864+0000 7f96e1f84740 1 -- 172.21.15.31:0/1229516428 shutdown_connections 2024-04-25T16:02:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.864+0000 7f96e1f84740 1 -- 172.21.15.31:0/1229516428 wait complete. 2024-04-25T16:02:21.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.864+0000 7f96e1f84740 20 client.15861 trim_cache size 0 max 16384 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.900+0000 7f96e1f84740 1 Processor -- start 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.900+0000 7f96e1f84740 1 -- start start 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.900+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.900+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620897496b0 0x562089b211a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.900+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 0x562089b216e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.900+0000 7f96e1f84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41a9010 con 0x5620897414d0 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.900+0000 7f96e1f84740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41aab50 con 0x56208976a390 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.900+0000 7f96e1f84740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41a96e0 con 0x5620897496b0 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d8e74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d8e74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38084/0 (socket says 172.21.15.31:38084) 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d8e74640 1 -- 172.21.15.31:0/4087806277 learned_addr learned my addr 172.21.15.31:0/4087806277 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d9675640 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 0x562089b216e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d3fff640 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620897496b0 0x562089b211a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d8e74640 1 -- 172.21.15.31:0/4087806277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620897496b0 msgr2=0x562089b211a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d8e74640 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620897496b0 0x562089b211a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d8e74640 1 -- 172.21.15.31:0/4087806277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 msgr2=0x562089b216e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d8e74640 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 0x562089b216e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d8e74640 1 -- 172.21.15.31:0/4087806277 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5620897420f0 con 0x5620897414d0 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d3fff640 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620897496b0 0x562089b211a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96d8e74640 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0a780 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f96c8004610 tx=0x7f96c8090000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84927295432aac8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.901+0000 7f96be7fc640 1 -- 172.21.15.31:0/4087806277 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c80a6de0 con 0x5620897414d0 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.902+0000 7f96be7fc640 1 -- 172.21.15.31:0/4087806277 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96c00054d0 con 0x5620897414d0 2024-04-25T16:02:21.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.902+0000 7f96be7fc640 1 -- 172.21.15.31:0/4087806277 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c80b02f0 con 0x5620897414d0 2024-04-25T16:02:21.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.902+0000 7f96e1f84740 1 -- 172.21.15.31:0/4087806277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 msgr2=0x562089b0a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.902+0000 7f96e1f84740 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0a780 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f96c8004610 tx=0x7f96c8090000 comp rx=0 tx=0).stop 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.903+0000 7f96e1f84740 1 -- 172.21.15.31:0/4087806277 shutdown_connections 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.903+0000 7f96e1f84740 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 0x562089b216e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.903+0000 7f96e1f84740 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620897496b0 0x562089b211a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.903+0000 7f96e1f84740 1 --2- 172.21.15.31:0/4087806277 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897414d0 0x562089b0a780 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.903+0000 7f96e1f84740 1 -- 172.21.15.31:0/4087806277 >> 172.21.15.31:0/4087806277 conn(0x56208975e4e0 msgr2=0x562089b0a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.903+0000 7f96e1f84740 1 -- 172.21.15.31:0/4087806277 shutdown_connections 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.903+0000 7f96e1f84740 1 -- 172.21.15.31:0/4087806277 wait complete. 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.903+0000 7f96e1f84740 1 -- start start 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.903+0000 7f96e1f84740 1 Processor -- start 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897496b0 0x562089b1e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 0x562089b1ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96e1f84740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620896fe3a0 0x562089b8d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96e1f84740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41aa480 con 0x5620897496b0 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96e1f84740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41a9db0 con 0x56208976a390 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96e1f84740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f96d41a6ff0 con 0x5620896fe3a0 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d8e74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897496b0 0x562089b1e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d8e74640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897496b0 0x562089b1e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38098/0 (socket says 172.21.15.31:38098) 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d8e74640 1 -- 172.21.15.31:0/2752872211 learned_addr learned my addr 172.21.15.31:0/2752872211 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96e1f84740 10 client.0 _ll_register_callbacks cb 0x7f96c80e2730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d3fff640 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 0x562089b1ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d9675640 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620896fe3a0 0x562089b8d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d8e74640 1 -- 172.21.15.31:0/2752872211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620896fe3a0 msgr2=0x562089b8d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d8e74640 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620896fe3a0 0x562089b8d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d8e74640 1 -- 172.21.15.31:0/2752872211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 msgr2=0x562089b1ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d8e74640 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 0x562089b1ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d8e74640 1 -- 172.21.15.31:0/2752872211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562089a314b0 con 0x5620897496b0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d9675640 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620896fe3a0 0x562089b8d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96d8e74640 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897496b0 0x562089b1e830 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f96c80a4700 tx=0x7f96c80df2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3728240d7c35cfd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.904+0000 7f96b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96b3fff640 1 -- 172.21.15.31:0/2752872211 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96c80a6de0 con 0x5620897496b0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96b3fff640 1 -- 172.21.15.31:0/2752872211 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f96c80e2250 con 0x5620897496b0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96e1f84740 1 -- 172.21.15.31:0/2752872211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96c00054d0 con 0x5620897496b0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96b3fff640 1 -- 172.21.15.31:0/2752872211 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f96c80a3350 con 0x5620897496b0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96e1f84740 10 client.15867 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96e1f84740 1 -- 172.21.15.31:0/2752872211 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x562089b8ecf0 con 0x5620897496b0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96b3fff640 1 -- 172.21.15.31:0/2752872211 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f96c00054d0 con 0x5620897496b0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96b3fff640 1 -- 172.21.15.31:0/2752872211 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f96c80a66f0 con 0x5620897496b0 2024-04-25T16:02:21.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96e1f84740 10 client.15867 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96e1f84740 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96e1f84740 20 client.15867 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562089b8f4f0 0x562089b91910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96e1f84740 4 client.15867 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.905+0000 7f96e1f84740 1 -- 172.21.15.31:0/2752872211 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562089a78800 con 0x562089b8f4f0 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.906+0000 7f96d3fff640 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562089b8f4f0 0x562089b91910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.906+0000 7f96b3fff640 10 client.15867 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.906+0000 7f96d3fff640 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562089b8f4f0 0x562089b91910 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f7c5d473db0fc4e server_cookie=f07a254c8468b66b in_seq=0 out_seq=0 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96b3fff640 1 -- 172.21.15.31:0/2752872211 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562089a78800 con 0x562089b8f4f0 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96b3fff640 10 client.15867 handle_command_reply: tid=0 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 client.15867 shutdown 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 -- 172.21.15.31:0/2752872211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897496b0 msgr2=0x562089b1e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897496b0 0x562089b1e830 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f96c80a4700 tx=0x7f96c80df2f0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 -- 172.21.15.31:0/2752872211 shutdown_connections 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562089b8f4f0 0x562089b91910 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5620896fe3a0 0x562089b8d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56208976a390 0x562089b1ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 --2- 172.21.15.31:0/2752872211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5620897496b0 0x562089b1e830 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 -- 172.21.15.31:0/2752872211 >> 172.21.15.31:0/2752872211 conn(0x56208975e4e0 msgr2=0x562089b0aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96b3fff640 0 client.15867 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 -- 172.21.15.31:0/2752872211 shutdown_connections 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 1 -- 172.21.15.31:0/2752872211 wait complete. 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.907+0000 7f96e1f84740 20 client.15867 trim_cache size 0 max 16384 2024-04-25T16:02:21.913 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:02:21.914 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T16:02:21.914 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:02:21.914 INFO:teuthology.orchestra.run.smithi031.stdout: "d28cc0f9-9eb8-45ab-aa32-0c8f32c41b41": { 2024-04-25T16:02:21.914 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:02:21.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "d28cc0f9-9eb8-45ab-aa32-0c8f32c41b41", 2024-04-25T16:02:21.914 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:02:21.914 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:02:21.914 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:02:21.914 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:02:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96c40afb70 msgr2=0x7f96c40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96c40afb70 0x7f96c40b2080 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x562089b0cab0 tx=0x7f96cc031af0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96d41029e0 msgr2=0x7f96d41993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96d41029e0 0x7f96d41993d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f96d41022b0 tx=0x7f96c800ee50 comp rx=0 tx=0).stop 2024-04-25T16:02:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 shutdown_connections 2024-04-25T16:02:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f96c40afb70 0x7f96c40b2080 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f96d4108920 0x7f96d41a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96d4103c20 0x7f96d4199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 --2- 172.21.15.31:0/149028577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f96d41029e0 0x7f96d41993d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:21.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 >> 172.21.15.31:0/149028577 conn(0x7f96d40fdbd0 msgr2=0x7f96d4104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:21.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 shutdown_connections 2024-04-25T16:02:21.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:21.908+0000 7f96db0d8640 1 -- 172.21.15.31:0/149028577 wait complete. 2024-04-25T16:02:21.920 INFO:tasks.cephfs.filesystem:scrub status for tag:d28cc0f9-9eb8-45ab-aa32-0c8f32c41b41 - {'path': '/', 'tag': 'd28cc0f9-9eb8-45ab-aa32-0c8f32c41b41', 'options': 'recursive,force'} 2024-04-25T16:02:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:22 smithi063 ceph-mon[32413]: pgmap v383: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T16:02:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:22 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:02:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:22 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:02:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:22 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:02:22.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:22 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:02:22.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:22 smithi186 ceph-mon[33258]: pgmap v383: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T16:02:22.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:22 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:02:22.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:22 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:02:22.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:22 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:02:22.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:22 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:02:22.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:22 smithi031 ceph-mon[29140]: pgmap v383: 97 pgs: 97 active+clean; 665 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T16:02:22.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:22 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:02:22.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:22 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:02:22.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:22 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:02:22.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:22 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:02:23.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:23 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:02:23.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:23 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:02:23.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:23 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:02:24.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:24 smithi063 ceph-mon[32413]: pgmap v384: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 511 B/s wr, 13 op/s 2024-04-25T16:02:24.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:24 smithi186 ceph-mon[33258]: pgmap v384: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 511 B/s wr, 13 op/s 2024-04-25T16:02:24.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:24 smithi031 ceph-mon[29140]: pgmap v384: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 511 B/s wr, 13 op/s 2024-04-25T16:02:26.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:26 smithi063 ceph-mon[32413]: pgmap v385: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:26.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:26 smithi186 ceph-mon[33258]: pgmap v385: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:26.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:26 smithi031 ceph-mon[29140]: pgmap v385: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:28.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:28 smithi063 ceph-mon[32413]: pgmap v386: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T16:02:28.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:28 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:02:28.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:28 smithi186 ceph-mon[33258]: pgmap v386: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T16:02:28.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:28 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:02:28.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:28 smithi031 ceph-mon[29140]: pgmap v386: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T16:02:28.949 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:28 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:02:30.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:02:30.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:02:30.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:02:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:02:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:02:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:02:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:02:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:31 smithi063 ceph-mon[32413]: pgmap v387: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T16:02:31.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:31 smithi186 ceph-mon[33258]: pgmap v387: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T16:02:31.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:31 smithi031 ceph-mon[29140]: pgmap v387: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-25T16:02:33.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:33 smithi186 ceph-mon[33258]: pgmap v388: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:02:33.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:33 smithi031 ceph-mon[29140]: pgmap v388: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:02:33.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:33 smithi063 ceph-mon[32413]: pgmap v388: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:02:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.714+0000 7fb0d7c67640 1 -- 172.21.15.31:0/1746695147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 msgr2=0x7fb0d0104850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.714+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/1746695147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 0x7fb0d0104850 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009900 tx=0x7fb0cc02a410 comp rx=0 tx=0).stop 2024-04-25T16:02:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.715+0000 7fb0d7c67640 1 -- 172.21.15.31:0/1746695147 shutdown_connections 2024-04-25T16:02:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.715+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/1746695147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0d0109a90 0x7fb0d010beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.715+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/1746695147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0d0104d90 0x7fb0d01051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.715+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/1746695147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 0x7fb0d0104850 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.715+0000 7fb0d7c67640 1 -- 172.21.15.31:0/1746695147 >> 172.21.15.31:0/1746695147 conn(0x7fb0d00fed20 msgr2=0x7fb0d01011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.715+0000 7fb0d7c67640 1 -- 172.21.15.31:0/1746695147 shutdown_connections 2024-04-25T16:02:34.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.715+0000 7fb0d7c67640 1 -- 172.21.15.31:0/1746695147 wait complete. 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d7c67640 1 Processor -- start 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d7c67640 1 -- start start 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d7c67640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 0x7fb0d019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d7c67640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0d0104d90 0x7fb0d019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d5a03640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 0x7fb0d019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d7c67640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0d0109a90 0x7fb0d01a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d7c67640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d010f6a0 con 0x7fb0d0103b50 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d7c67640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d010f520 con 0x7fb0d0109a90 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d5a03640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 0x7fb0d019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35540/0 (socket says 172.21.15.31:35540) 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d5a03640 1 -- 172.21.15.31:0/3226737500 learned_addr learned my addr 172.21.15.31:0/3226737500 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:34.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.716+0000 7fb0d7c67640 1 -- 172.21.15.31:0/3226737500 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d010f820 con 0x7fb0d0104d90 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.717+0000 7fb0d6204640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0d0109a90 0x7fb0d01a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.717+0000 7fb0d5202640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0d0104d90 0x7fb0d019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.717+0000 7fb0d5a03640 1 -- 172.21.15.31:0/3226737500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0d0104d90 msgr2=0x7fb0d019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.717+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0d0104d90 0x7fb0d019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.717+0000 7fb0d5a03640 1 -- 172.21.15.31:0/3226737500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0d0109a90 msgr2=0x7fb0d01a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.717+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0d0109a90 0x7fb0d01a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.717+0000 7fb0d5a03640 1 -- 172.21.15.31:0/3226737500 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0d01a1d80 con 0x7fb0d0103b50 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.717+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 0x7fb0d019a240 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc031440 tx=0x7fb0cc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.718+0000 7fb0c6ffd640 1 -- 172.21.15.31:0/3226737500 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc03f070 con 0x7fb0d0103b50 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.718+0000 7fb0c6ffd640 1 -- 172.21.15.31:0/3226737500 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0cc031c50 con 0x7fb0d0103b50 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.718+0000 7fb0d7c67640 1 -- 172.21.15.31:0/3226737500 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0d01a2010 con 0x7fb0d0103b50 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.718+0000 7fb0c6ffd640 1 -- 172.21.15.31:0/3226737500 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc035030 con 0x7fb0d0103b50 2024-04-25T16:02:34.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.718+0000 7fb0d7c67640 1 -- 172.21.15.31:0/3226737500 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0d01a2550 con 0x7fb0d0103b50 2024-04-25T16:02:34.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.722+0000 7fb0debfb740 1 Processor -- start 2024-04-25T16:02:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.722+0000 7fb0debfb740 1 -- start start 2024-04-25T16:02:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.722+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 0x5575cf0107d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.722+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf012d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.722+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf015510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.722+0000 7fb0c6ffd640 1 -- 172.21.15.31:0/3226737500 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb0cc042070 con 0x7fb0d0103b50 2024-04-25T16:02:34.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.722+0000 7fb0debfb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5575cef6b460 con 0x5575cec444d0 2024-04-25T16:02:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.722+0000 7fb0debfb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5575cef25590 con 0x5575cec3c2f0 2024-04-25T16:02:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.722+0000 7fb0debfb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5575cef67420 con 0x5575cec651b0 2024-04-25T16:02:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5202640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf015510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5202640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf015510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35542/0 (socket says 172.21.15.31:35542) 2024-04-25T16:02:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5202640 1 -- 172.21.15.31:0/3950745687 learned_addr learned my addr 172.21.15.31:0/3950745687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:34.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d6204640 1 --2- 172.21.15.31:0/3950745687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 0x5575cf0107d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/3950745687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf012d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5202640 1 -- 172.21.15.31:0/3950745687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 msgr2=0x5575cf0107d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5202640 1 --2- 172.21.15.31:0/3950745687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 0x5575cf0107d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5202640 1 -- 172.21.15.31:0/3950745687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 msgr2=0x5575cf012d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5202640 1 --2- 172.21.15.31:0/3950745687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf012d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5202640 1 -- 172.21.15.31:0/3950745687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575cef320d0 con 0x5575cec444d0 2024-04-25T16:02:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.723+0000 7fb0d5202640 1 --2- 172.21.15.31:0/3950745687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf015510 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009ca0 tx=0x7fb0c0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e45f059aa60078 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.724+0000 7fb0c6ffd640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb0bc0afb60 0x7fb0bc0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.724+0000 7fb0d6204640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb0bc0afb60 0x7fb0bc0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.724+0000 7fb0bbfff640 1 -- 172.21.15.31:0/3950745687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c0005220 con 0x5575cec444d0 2024-04-25T16:02:34.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.724+0000 7fb0c6ffd640 1 -- 172.21.15.31:0/3226737500 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb0cc0fef90 con 0x7fb0d0103b50 2024-04-25T16:02:34.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.724+0000 7fb0bbfff640 1 -- 172.21.15.31:0/3950745687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0c0004c50 con 0x5575cec444d0 2024-04-25T16:02:34.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.724+0000 7fb0bbfff640 1 -- 172.21.15.31:0/3950745687 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c00197a0 con 0x5575cec444d0 2024-04-25T16:02:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0d6204640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb0bc0afb60 0x7fb0bc0b2070 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x5575cf004aa0 tx=0x7fb0c8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0debfb740 1 -- 172.21.15.31:0/3950745687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 msgr2=0x5575cf015510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3950745687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf015510 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7fb0c0009ca0 tx=0x7fb0c0011810 comp rx=0 tx=0).stop 2024-04-25T16:02:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0debfb740 1 -- 172.21.15.31:0/3950745687 shutdown_connections 2024-04-25T16:02:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3950745687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf015510 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3950745687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf012d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3950745687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 0x5575cf0107d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0debfb740 1 -- 172.21.15.31:0/3950745687 >> 172.21.15.31:0/3950745687 conn(0x5575cec59300 msgr2=0x5575cf001280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:34.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0debfb740 1 -- 172.21.15.31:0/3950745687 shutdown_connections 2024-04-25T16:02:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.725+0000 7fb0debfb740 1 -- 172.21.15.31:0/3950745687 wait complete. 2024-04-25T16:02:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.726+0000 7fb0debfb740 1 -- start start 2024-04-25T16:02:34.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.726+0000 7fb0debfb740 1 Processor -- start 2024-04-25T16:02:34.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.726+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf017ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf0183e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 0x5575cf01f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0debfb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5575cf016030 con 0x5575cec444d0 2024-04-25T16:02:34.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0debfb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5575ceef2b60 con 0x5575cec3c2f0 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0debfb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5575cf0161b0 con 0x5575cec651b0 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5a03640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf017ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5a03640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf017ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:41410/0 (socket says 172.21.15.31:41410) 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5a03640 1 -- 172.21.15.31:0/3155634731 learned_addr learned my addr 172.21.15.31:0/3155634731 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5202640 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf0183e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d6204640 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 0x5575cf01f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0debfb740 10 client.0 _ll_register_callbacks cb 0x5575cef172b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5a03640 1 -- 172.21.15.31:0/3155634731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 msgr2=0x5575cf01f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 0x5575cf01f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5a03640 1 -- 172.21.15.31:0/3155634731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 msgr2=0x5575cf0183e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf0183e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5a03640 1 -- 172.21.15.31:0/3155634731 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575cec7d210 con 0x5575cec3c2f0 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0debfb740 1 -- 172.21.15.31:0/3155634731 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0c0004c50 con 0x5575cec3c2f0 2024-04-25T16:02:34.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.727+0000 7fb0d5202640 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf0183e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.728+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf017ea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0c74c0 tx=0x7fb0cc0c6e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36ed2347399a84cf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.728+0000 7fb0b97fa640 10 client.25583 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.728+0000 7fb0b97fa640 1 -- 172.21.15.31:0/3155634731 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc049040 con 0x5575cec3c2f0 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.728+0000 7fb0b97fa640 1 -- 172.21.15.31:0/3155634731 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb0cc0c37c0 con 0x5575cec3c2f0 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.728+0000 7fb0b97fa640 1 -- 172.21.15.31:0/3155634731 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0cc0b0070 con 0x5575cec3c2f0 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.728+0000 7fb0debfb740 10 client.25583 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.728+0000 7fb0debfb740 1 -- 172.21.15.31:0/3155634731 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({fsmap=27}) -- 0x5575ced49620 con 0x5575cec3c2f0 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.729+0000 7fb0b97fa640 1 -- 172.21.15.31:0/3155634731 <== mon.2 v2:172.21.15.63:3300/0 4 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb0cc1010e0 con 0x5575cec3c2f0 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.729+0000 7fb0debfb740 10 client.25583 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.729+0000 7fb0debfb740 10 client.25583 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.729+0000 7fb0debfb740 20 client.25583 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.729+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5575cec23d80 0x5575cf025ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.729+0000 7fb0debfb740 4 client.25583 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.729+0000 7fb0debfb740 1 -- 172.21.15.31:0/3155634731 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575cef79070 con 0x5575cec23d80 2024-04-25T16:02:34.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.729+0000 7fb0d5202640 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5575cec23d80 0x5575cf025ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.730+0000 7fb0b97fa640 10 client.25583 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:34.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.730+0000 7fb0d5202640 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5575cec23d80 0x5575cf025ab0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1970e8a221aec239 server_cookie=bef69dd7661406c in_seq=0 out_seq=0 2024-04-25T16:02:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.730+0000 7fb0b97fa640 1 -- 172.21.15.31:0/3155634731 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5575cef79070 con 0x5575cec23d80 2024-04-25T16:02:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.730+0000 7fb0b97fa640 10 client.25583 handle_command_reply: tid=0 2024-04-25T16:02:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.730+0000 7fb0debfb740 1 client.25583 shutdown 2024-04-25T16:02:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.730+0000 7fb0debfb740 1 -- 172.21.15.31:0/3155634731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 msgr2=0x5575cf017ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.730+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf017ea0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0c74c0 tx=0x7fb0cc0c6e80 comp rx=0 tx=0).stop 2024-04-25T16:02:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0debfb740 1 -- 172.21.15.31:0/3155634731 shutdown_connections 2024-04-25T16:02:34.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5575cec23d80 0x5575cf025ab0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec651b0 0x5575cf01f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf0183e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0debfb740 1 --2- 172.21.15.31:0/3155634731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec3c2f0 0x5575cf017ea0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0debfb740 1 -- 172.21.15.31:0/3155634731 >> 172.21.15.31:0/3155634731 conn(0x5575cec59300 msgr2=0x5575cf000250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0b97fa640 0 client.25583 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0debfb740 1 -- 172.21.15.31:0/3155634731 shutdown_connections 2024-04-25T16:02:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0debfb740 1 -- 172.21.15.31:0/3155634731 wait complete. 2024-04-25T16:02:34.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.731+0000 7fb0debfb740 20 client.25583 trim_cache size 0 max 16384 2024-04-25T16:02:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.761+0000 7fb0debfb740 1 Processor -- start 2024-04-25T16:02:34.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.761+0000 7fb0debfb740 1 -- start start 2024-04-25T16:02:34.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 0x5575cf018180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec444d0 0x5575cf01f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec651b0 0x5575cf02a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0debfb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01a8f60 con 0x5575cec651b0 2024-04-25T16:02:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0debfb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01aaaa0 con 0x5575cec444d0 2024-04-25T16:02:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0debfb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01a9630 con 0x5575cec3c2f0 2024-04-25T16:02:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0d6204640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec651b0 0x5575cf02a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0d6204640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec651b0 0x5575cf02a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35558/0 (socket says 172.21.15.31:35558) 2024-04-25T16:02:34.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0d6204640 1 -- 172.21.15.31:0/768487234 learned_addr learned my addr 172.21.15.31:0/768487234 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/768487234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 0x5575cf018180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0d5202640 1 --2- 172.21.15.31:0/768487234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec444d0 0x5575cf01f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0d6204640 1 -- 172.21.15.31:0/768487234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 msgr2=0x5575cf018180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0d6204640 1 --2- 172.21.15.31:0/768487234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 0x5575cf018180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0d6204640 1 -- 172.21.15.31:0/768487234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec444d0 msgr2=0x5575cf01f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.762+0000 7fb0d6204640 1 --2- 172.21.15.31:0/768487234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec444d0 0x5575cf01f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.763+0000 7fb0d6204640 1 -- 172.21.15.31:0/768487234 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575cec23fb0 con 0x5575cec651b0 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.763+0000 7fb0d6204640 1 --2- 172.21.15.31:0/768487234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec651b0 0x5575cf02a2a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8003ef0 tx=0x7fb0c8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58588e7dda850ba4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.763+0000 7fb0bb7fe640 1 -- 172.21.15.31:0/768487234 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c800f4e0 con 0x5575cec651b0 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.763+0000 7fb0bb7fe640 1 -- 172.21.15.31:0/768487234 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0c800fb30 con 0x5575cec651b0 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.763+0000 7fb0bb7fe640 1 -- 172.21.15.31:0/768487234 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0c801da50 con 0x5575cec651b0 2024-04-25T16:02:34.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.764+0000 7fb0debfb740 1 -- 172.21.15.31:0/768487234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec651b0 msgr2=0x5575cf02a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.764+0000 7fb0debfb740 1 --2- 172.21.15.31:0/768487234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec651b0 0x5575cf02a2a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fb0c8003ef0 tx=0x7fb0c8016000 comp rx=0 tx=0).stop 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.764+0000 7fb0debfb740 1 -- 172.21.15.31:0/768487234 shutdown_connections 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.764+0000 7fb0debfb740 1 --2- 172.21.15.31:0/768487234 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec651b0 0x5575cf02a2a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.764+0000 7fb0debfb740 1 --2- 172.21.15.31:0/768487234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec444d0 0x5575cf01f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.764+0000 7fb0debfb740 1 --2- 172.21.15.31:0/768487234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 0x5575cf018180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.764+0000 7fb0debfb740 1 -- 172.21.15.31:0/768487234 >> 172.21.15.31:0/768487234 conn(0x5575cec59300 msgr2=0x5575cf017a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.764+0000 7fb0debfb740 1 -- 172.21.15.31:0/768487234 shutdown_connections 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.764+0000 7fb0debfb740 1 -- 172.21.15.31:0/768487234 wait complete. 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.765+0000 7fb0debfb740 1 -- start start 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.765+0000 7fb0debfb740 1 Processor -- start 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.765+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 0x5575cf01e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf01ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0debfb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec651b0 0x5575cf08c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0debfb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01aa3d0 con 0x5575cec444d0 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0debfb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01a9d00 con 0x5575cec651b0 2024-04-25T16:02:34.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0debfb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0d01a6f40 con 0x5575cec3c2f0 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5202640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf01ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0debfb740 10 client.0 _ll_register_callbacks cb 0x7fb0cc005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5202640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf01ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35564/0 (socket says 172.21.15.31:35564) 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5202640 1 -- 172.21.15.31:0/1957949142 learned_addr learned my addr 172.21.15.31:0/1957949142 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 0x5575cf01e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d6204640 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec651b0 0x5575cf08c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5a03640 1 -- 172.21.15.31:0/1957949142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec651b0 msgr2=0x5575cf08c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec651b0 0x5575cf08c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5a03640 1 -- 172.21.15.31:0/1957949142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 msgr2=0x5575cf01ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf01ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5a03640 1 -- 172.21.15.31:0/1957949142 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575cf08c610 con 0x5575cec3c2f0 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d5202640 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf01ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.766+0000 7fb0d6204640 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec651b0 0x5575cf08c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:34.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.767+0000 7fb0d5a03640 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 0x5575cf01e750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0092b0 tx=0x7fb0cc0fc970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0daf0ea88945579 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.767+0000 7fb0b97fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.767+0000 7fb0b97fa640 1 -- 172.21.15.31:0/1957949142 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0e6060 con 0x5575cec3c2f0 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.767+0000 7fb0b97fa640 1 -- 172.21.15.31:0/1957949142 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb0cc0c7790 con 0x5575cec3c2f0 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.767+0000 7fb0debfb740 1 -- 172.21.15.31:0/1957949142 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0c800fb30 con 0x5575cec3c2f0 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.767+0000 7fb0b97fa640 1 -- 172.21.15.31:0/1957949142 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb0cc0c6ca0 con 0x5575cec3c2f0 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.767+0000 7fb0debfb740 10 client.25552 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.767+0000 7fb0debfb740 1 -- 172.21.15.31:0/1957949142 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5575cf08d910 con 0x5575cec3c2f0 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.767+0000 7fb0b97fa640 1 -- 172.21.15.31:0/1957949142 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0c800fb30 con 0x5575cec3c2f0 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0b97fa640 1 -- 172.21.15.31:0/1957949142 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb0cc0cfe50 con 0x5575cec3c2f0 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0debfb740 10 client.25552 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0debfb740 10 client.25552 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0debfb740 20 client.25552 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0debfb740 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5575cebf91c0 0x5575cf0900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:34.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0debfb740 4 client.25552 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:02:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0debfb740 1 -- 172.21.15.31:0/1957949142 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5575cef72ee0 con 0x5575cebf91c0 2024-04-25T16:02:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0d5202640 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5575cebf91c0 0x5575cf0900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0b97fa640 10 client.25552 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:34.775 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.768+0000 7fb0d5202640 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5575cebf91c0 0x5575cf0900e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f075af7cbad53846 server_cookie=afc894f67437f441 in_seq=0 out_seq=0 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.772+0000 7fb0b97fa640 1 -- 172.21.15.31:0/1957949142 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5575cef72ee0 con 0x5575cebf91c0 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.772+0000 7fb0b97fa640 10 client.25552 handle_command_reply: tid=0 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.772+0000 7fb0debfb740 1 client.25552 shutdown 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.772+0000 7fb0debfb740 1 -- 172.21.15.31:0/1957949142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 msgr2=0x5575cf01e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.772+0000 7fb0debfb740 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 0x5575cf01e750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc0092b0 tx=0x7fb0cc0fc970 comp rx=0 tx=0).stop 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0debfb740 1 -- 172.21.15.31:0/1957949142 shutdown_connections 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0debfb740 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5575cebf91c0 0x5575cf0900e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0debfb740 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5575cec651b0 0x5575cf08c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0debfb740 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5575cec444d0 0x5575cf01ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0debfb740 1 --2- 172.21.15.31:0/1957949142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575cec3c2f0 0x5575cf01e750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0debfb740 1 -- 172.21.15.31:0/1957949142 >> 172.21.15.31:0/1957949142 conn(0x5575cec59300 msgr2=0x5575cefff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0b97fa640 0 client.25552 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0debfb740 1 -- 172.21.15.31:0/1957949142 shutdown_connections 2024-04-25T16:02:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0debfb740 1 -- 172.21.15.31:0/1957949142 wait complete. 2024-04-25T16:02:34.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0debfb740 20 client.25552 trim_cache size 0 max 16384 2024-04-25T16:02:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:02:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:02:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:02:34.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:02:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0d7c67640 1 -- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb0bc0afb60 msgr2=0x7fb0bc0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.773+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb0bc0afb60 0x7fb0bc0b2070 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x5575cf004aa0 tx=0x7fb0c8009500 comp rx=0 tx=0).stop 2024-04-25T16:02:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 -- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 msgr2=0x7fb0d019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 0x7fb0d019a240 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc031440 tx=0x7fb0cc003c10 comp rx=0 tx=0).stop 2024-04-25T16:02:34.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 -- 172.21.15.31:0/3226737500 shutdown_connections 2024-04-25T16:02:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb0bc0afb60 0x7fb0bc0b2070 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb0d0109a90 0x7fb0d01a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0d0104d90 0x7fb0d019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 --2- 172.21.15.31:0/3226737500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb0d0103b50 0x7fb0d019a240 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 -- 172.21.15.31:0/3226737500 >> 172.21.15.31:0/3226737500 conn(0x7fb0d00fed20 msgr2=0x7fb0d0105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 -- 172.21.15.31:0/3226737500 shutdown_connections 2024-04-25T16:02:34.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:34.774+0000 7fb0d7c67640 1 -- 172.21.15.31:0/3226737500 wait complete. 2024-04-25T16:02:35.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:35 smithi186 ceph-mon[33258]: pgmap v389: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T16:02:35.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:35 smithi031 ceph-mon[29140]: pgmap v389: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T16:02:35.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:35 smithi063 ceph-mon[32413]: pgmap v389: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 255 B/s wr, 13 op/s 2024-04-25T16:02:37.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:37 smithi186 ceph-mon[33258]: pgmap v390: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-25T16:02:37.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:37 smithi031 ceph-mon[29140]: pgmap v390: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-25T16:02:37.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:37 smithi063 ceph-mon[32413]: pgmap v390: 97 pgs: 97 active+clean; 666 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-25T16:02:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:39 smithi186 ceph-mon[33258]: pgmap v391: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:39.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:39 smithi031 ceph-mon[29140]: pgmap v391: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:39.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:39 smithi063 ceph-mon[32413]: pgmap v391: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:41.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:41 smithi186 ceph-mon[33258]: pgmap v392: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:41.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:41 smithi031 ceph-mon[29140]: pgmap v392: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:41.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:41 smithi063 ceph-mon[32413]: pgmap v392: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-25T16:02:42.665 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 92% (11301/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-25T16:02:43.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:43 smithi186 ceph-mon[33258]: pgmap v393: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T16:02:43.444 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to '994e2f20102a20fa9a27966c14b625cc01884df8'. 2024-04-25T16:02:43.444 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T16:02:43.445 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:43.446 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at 994e2f20102 Merge PR #56810 into wip-pdonnell-testing-20240425.015853-debug 2024-04-25T16:02:43.449 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:43 smithi031 ceph-mon[29140]: pgmap v393: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T16:02:43.451 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T16:02:43.486 INFO:tasks.workunit.client.0.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-25T16:02:43.488 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T16:02:43.488 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-25T16:02:43.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:43 smithi063 ceph-mon[32413]: pgmap v393: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-25T16:02:44.120 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-25T16:02:44.178 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-25T16:02:44.220 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-25T16:02:44.221 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T16:02:44.222 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-25T16:02:44.261 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-25T16:02:44.265 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T16:02:44.265 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T16:02:44.322 INFO:tasks.workunit:Running workunits matching suites/iogen.sh on client.0... 2024-04-25T16:02:44.323 INFO:tasks.workunit:Running workunit suites/iogen.sh... 2024-04-25T16:02:44.323 DEBUG:teuthology.orchestra.run.smithi031:workunit test suites/iogen.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=994e2f20102a20fa9a27966c14b625cc01884df8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/iogen.sh 2024-04-25T16:02:44.383 INFO:tasks.workunit.client.0.smithi031.stdout:getting iogen 2024-04-25T16:02:44.383 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'getting iogen' 2024-04-25T16:02:44.383 INFO:tasks.workunit.client.0.smithi031.stderr:+ wget http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-25T16:02:44.400 INFO:tasks.workunit.client.0.smithi031.stderr:--2024-04-25 16:02:44-- http://download.ceph.com/qa/iogen_3.1p0.tar 2024-04-25T16:02:44.401 INFO:tasks.workunit.client.0.smithi031.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-04-25T16:02:44.422 INFO:tasks.workunit.client.0.smithi031.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-04-25T16:02:44.443 INFO:tasks.workunit.client.0.smithi031.stderr:HTTP request sent, awaiting response... 200 OK 2024-04-25T16:02:44.443 INFO:tasks.workunit.client.0.smithi031.stderr:Length: 8071 (7.9K) [application/octet-stream] 2024-04-25T16:02:44.444 INFO:tasks.workunit.client.0.smithi031.stderr:Saving to: ‘iogen_3.1p0.tar’ 2024-04-25T16:02:44.447 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:44.447 INFO:tasks.workunit.client.0.smithi031.stderr: 0K ....... 100% 1.19G=0s 2024-04-25T16:02:44.447 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:44.447 INFO:tasks.workunit.client.0.smithi031.stderr:2024-04-25 16:02:44 (1.19 GB/s) - ‘iogen_3.1p0.tar’ saved [8071/8071] 2024-04-25T16:02:44.448 INFO:tasks.workunit.client.0.smithi031.stderr: 2024-04-25T16:02:44.448 INFO:tasks.workunit.client.0.smithi031.stderr:+ tar -xvzf iogen_3.1p0.tar 2024-04-25T16:02:44.463 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0 2024-04-25T16:02:44.464 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/src 2024-04-25T16:02:44.465 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/src/iogen.8 2024-04-25T16:02:44.466 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/src/iogen.c 2024-04-25T16:02:44.467 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/README 2024-04-25T16:02:44.468 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/Makefile.bsd 2024-04-25T16:02:44.469 INFO:tasks.workunit.client.0.smithi031.stdout:iogen_3.1p0/Makefile 2024-04-25T16:02:44.470 INFO:tasks.workunit.client.0.smithi031.stderr:+ cd iogen_3.1p0 2024-04-25T16:02:44.471 INFO:tasks.workunit.client.0.smithi031.stdout:making iogen 2024-04-25T16:02:44.471 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'making iogen' 2024-04-25T16:02:44.471 INFO:tasks.workunit.client.0.smithi031.stderr:+ make 2024-04-25T16:02:44.479 INFO:tasks.workunit.client.0.smithi031.stdout:gcc -D_FILE_OFFSET_BITS=64 -DVERSION=\"3.1p0\" -Dstrlcpy=strncpy -Darc4random=rand -O2 src/iogen.c -o iogen 2024-04-25T16:02:44.701 INFO:tasks.workunit.client.0.smithi031.stdout:nroff -mandoc -Tascii src/iogen.8 > iogen.cat8 2024-04-25T16:02:44.789 INFO:tasks.workunit.client.0.smithi031.stdout:running iogen 2024-04-25T16:02:44.790 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'running iogen' 2024-04-25T16:02:44.790 INFO:tasks.workunit.client.0.smithi031.stderr:+ ./iogen -n 5 -s 2g 2024-04-25T16:02:44.790 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 84969 2024-04-25T16:02:44.790 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 84970 2024-04-25T16:02:44.790 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 84971 2024-04-25T16:02:44.790 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 84972 2024-04-25T16:02:44.790 INFO:tasks.workunit.client.0.smithi031.stderr:child spawned 84973 2024-04-25T16:02:44.791 INFO:tasks.workunit.client.0.smithi031.stdout:sleep for 10 min 2024-04-25T16:02:44.791 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'sleep for 10 min' 2024-04-25T16:02:44.791 INFO:tasks.workunit.client.0.smithi031.stderr:+ sleep 600 2024-04-25T16:02:45.148 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:45 smithi031 ceph-mon[29140]: pgmap v394: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T16:02:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:45 smithi186 ceph-mon[33258]: pgmap v394: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T16:02:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:45 smithi063 ceph-mon[32413]: pgmap v394: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-25T16:02:47.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:47 smithi031 ceph-mon[29140]: pgmap v395: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:02:47.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:47 smithi063 ceph-mon[32413]: pgmap v395: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:02:47.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:47 smithi186 ceph-mon[33258]: pgmap v395: 97 pgs: 97 active+clean; 667 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-25T16:02:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.067+0000 7fec401c7640 1 -- 172.21.15.31:0/509541241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec3810b350 msgr2=0x7fec3810b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:48.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.067+0000 7fec401c7640 1 --2- 172.21.15.31:0/509541241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec3810b350 0x7fec3810b760 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7fec3401a910 tx=0x7fec3403b470 comp rx=0 tx=0).stop 2024-04-25T16:02:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.070+0000 7fec401c7640 1 -- 172.21.15.31:0/509541241 shutdown_connections 2024-04-25T16:02:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.070+0000 7fec401c7640 1 --2- 172.21.15.31:0/509541241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec38110070 0x7fec38110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.070+0000 7fec401c7640 1 --2- 172.21.15.31:0/509541241 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec3810b350 0x7fec3810b760 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.070+0000 7fec401c7640 1 --2- 172.21.15.31:0/509541241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec3810a420 0x7fec3810ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.070+0000 7fec401c7640 1 -- 172.21.15.31:0/509541241 >> 172.21.15.31:0/509541241 conn(0x7fec38070540 msgr2=0x7fec380729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.071+0000 7fec401c7640 1 -- 172.21.15.31:0/509541241 shutdown_connections 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.072+0000 7fec401c7640 1 -- 172.21.15.31:0/509541241 wait complete. 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.073+0000 7fec401c7640 1 Processor -- start 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.074+0000 7fec401c7640 1 -- start start 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.074+0000 7fec401c7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec3810a420 0x7fec38077040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.074+0000 7fec401c7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec3810b350 0x7fec38079590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.074+0000 7fec401c7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec38110070 0x7fec3807bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.074+0000 7fec401c7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec38113e00 con 0x7fec3810a420 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.074+0000 7fec401c7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec38113c80 con 0x7fec38110070 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.074+0000 7fec401c7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec38113f80 con 0x7fec3810b350 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.075+0000 7fec3df63640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec3810a420 0x7fec38077040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.075+0000 7fec3df63640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec3810a420 0x7fec38077040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40268/0 (socket says 172.21.15.31:40268) 2024-04-25T16:02:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.075+0000 7fec3df63640 1 -- 172.21.15.31:0/1716774595 learned_addr learned my addr 172.21.15.31:0/1716774595 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.075+0000 7fec3e764640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec38110070 0x7fec3807bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.076+0000 7fec3d762640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec3810b350 0x7fec38079590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.079+0000 7fec3e764640 1 -- 172.21.15.31:0/1716774595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec3810b350 msgr2=0x7fec38079590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.079+0000 7fec3e764640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec3810b350 0x7fec38079590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.079+0000 7fec3e764640 1 -- 172.21.15.31:0/1716774595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec3810a420 msgr2=0x7fec38077040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.079+0000 7fec3e764640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec3810a420 0x7fec38077040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.079+0000 7fec3e764640 1 -- 172.21.15.31:0/1716774595 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec3807c2e0 con 0x7fec38110070 2024-04-25T16:02:48.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.081+0000 7fec3e764640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec38110070 0x7fec3807bae0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec2800b3e0 tx=0x7fec2800b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.083+0000 7fec2effd640 1 -- 172.21.15.31:0/1716774595 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec28015020 con 0x7fec38110070 2024-04-25T16:02:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.083+0000 7fec2effd640 1 -- 172.21.15.31:0/1716774595 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec28011ce0 con 0x7fec38110070 2024-04-25T16:02:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.083+0000 7fec2effd640 1 -- 172.21.15.31:0/1716774595 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec28010d80 con 0x7fec38110070 2024-04-25T16:02:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.083+0000 7fec401c7640 1 -- 172.21.15.31:0/1716774595 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec3807c5d0 con 0x7fec38110070 2024-04-25T16:02:48.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.083+0000 7fec401c7640 1 -- 172.21.15.31:0/1716774595 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec3807cb50 con 0x7fec38110070 2024-04-25T16:02:48.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.091+0000 7fec47075740 1 Processor -- start 2024-04-25T16:02:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.092+0000 7fec47075740 1 -- start start 2024-04-25T16:02:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.093+0000 7fec47075740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a03970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.093+0000 7fec47075740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x563693a03ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.093+0000 7fec47075740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x563693a0adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.093+0000 7fec47075740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563693969460 con 0x5636936631b0 2024-04-25T16:02:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.093+0000 7fec47075740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563693923590 con 0x56369363a2f0 2024-04-25T16:02:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.093+0000 7fec47075740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563693965420 con 0x5636936424d0 2024-04-25T16:02:48.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.094+0000 7fec3d762640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a03970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.095+0000 7fec3d762640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a03970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40270/0 (socket says 172.21.15.31:40270) 2024-04-25T16:02:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.095+0000 7fec3d762640 1 -- 172.21.15.31:0/1896675088 learned_addr learned my addr 172.21.15.31:0/1896675088 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.095+0000 7fec3d762640 1 --2- 172.21.15.31:0/1896675088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x563693a0adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.095+0000 7fec3df63640 1 --2- 172.21.15.31:0/1896675088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x563693a03ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.096+0000 7fec3d762640 1 -- 172.21.15.31:0/1896675088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 msgr2=0x563693a0adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.096+0000 7fec3d762640 1 --2- 172.21.15.31:0/1896675088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x563693a0adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.096+0000 7fec3d762640 1 -- 172.21.15.31:0/1896675088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 msgr2=0x563693a03ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:48.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.096+0000 7fec3d762640 1 --2- 172.21.15.31:0/1896675088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x563693a03ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.096+0000 7fec3d762640 1 -- 172.21.15.31:0/1896675088 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5636939300d0 con 0x5636936631b0 2024-04-25T16:02:48.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.096+0000 7fec2effd640 1 -- 172.21.15.31:0/1716774595 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fec28018020 con 0x7fec38110070 2024-04-25T16:02:48.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.098+0000 7fec2effd640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec240afcf0 0x7fec240b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.099+0000 7fec2effd640 1 -- 172.21.15.31:0/1716774595 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fec280de130 con 0x7fec38110070 2024-04-25T16:02:48.102 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.101+0000 7fec3df63640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec240afcf0 0x7fec240b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.102+0000 7fec3d762640 1 --2- 172.21.15.31:0/1896675088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a03970 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fec34005f70 tx=0x7fec3403b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5f1d596b870aea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:48.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.102+0000 7fec3df63640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec240afcf0 0x7fec240b2200 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x563693a04d50 tx=0x7fec3000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:48.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.102+0000 7fec23fff640 1 -- 172.21.15.31:0/1896675088 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec34046040 con 0x5636936631b0 2024-04-25T16:02:48.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.103+0000 7fec23fff640 1 -- 172.21.15.31:0/1896675088 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec340052a0 con 0x5636936631b0 2024-04-25T16:02:48.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.105+0000 7fec23fff640 1 -- 172.21.15.31:0/1896675088 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec340039f0 con 0x5636936631b0 2024-04-25T16:02:48.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.105+0000 7fec47075740 1 -- 172.21.15.31:0/1896675088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 msgr2=0x563693a03970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:48.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.105+0000 7fec47075740 1 --2- 172.21.15.31:0/1896675088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a03970 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fec34005f70 tx=0x7fec3403b470 comp rx=0 tx=0).stop 2024-04-25T16:02:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.106+0000 7fec47075740 1 -- 172.21.15.31:0/1896675088 shutdown_connections 2024-04-25T16:02:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.106+0000 7fec47075740 1 --2- 172.21.15.31:0/1896675088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x563693a0adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.106+0000 7fec47075740 1 --2- 172.21.15.31:0/1896675088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x563693a03ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.106+0000 7fec47075740 1 --2- 172.21.15.31:0/1896675088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a03970 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.106+0000 7fec47075740 1 -- 172.21.15.31:0/1896675088 >> 172.21.15.31:0/1896675088 conn(0x563693657300 msgr2=0x5636939ff300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:48.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.106+0000 7fec47075740 1 -- 172.21.15.31:0/1896675088 shutdown_connections 2024-04-25T16:02:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.106+0000 7fec47075740 1 -- 172.21.15.31:0/1896675088 wait complete. 2024-04-25T16:02:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.107+0000 7fec47075740 1 -- start start 2024-04-25T16:02:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.107+0000 7fec47075740 1 Processor -- start 2024-04-25T16:02:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.108+0000 7fec47075740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56369363a2f0 0x563693a0d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.108+0000 7fec47075740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x563693a0dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.108+0000 7fec47075740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5636936631b0 0x563693a14c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.108+0000 7fec47075740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563693a0b830 con 0x56369363a2f0 2024-04-25T16:02:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.108+0000 7fec47075740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5636938f0b60 con 0x5636936631b0 2024-04-25T16:02:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.108+0000 7fec47075740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563693a0b9b0 con 0x5636936424d0 2024-04-25T16:02:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.108+0000 7fec47075740 10 client.0 _ll_register_callbacks cb 0x5636939152b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.109+0000 7fec3df63640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56369363a2f0 0x563693a0d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.109+0000 7fec3df63640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56369363a2f0 0x563693a0d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40278/0 (socket says 172.21.15.31:40278) 2024-04-25T16:02:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.109+0000 7fec3df63640 1 -- 172.21.15.31:0/1408479673 learned_addr learned my addr 172.21.15.31:0/1408479673 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.110+0000 7fec3df63640 1 -- 172.21.15.31:0/1408479673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 msgr2=0x563693a0dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.110+0000 7fec3df63640 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x563693a0dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.110+0000 7fec3df63640 1 -- 172.21.15.31:0/1408479673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5636936631b0 msgr2=0x563693a14c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.110+0000 7fec3df63640 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5636936631b0 0x563693a14c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.110+0000 7fec3df63640 1 -- 172.21.15.31:0/1408479673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56369367b210 con 0x56369363a2f0 2024-04-25T16:02:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.111+0000 7fec3df63640 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56369363a2f0 0x563693a0d770 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fec30016e20 tx=0x7fec30004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2c2ee414c74bd0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:48.114 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.111+0000 7fec217fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.111+0000 7fec217fa640 1 -- 172.21.15.31:0/1408479673 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30025020 con 0x56369363a2f0 2024-04-25T16:02:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.111+0000 7fec47075740 1 -- 172.21.15.31:0/1408479673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec340052a0 con 0x56369363a2f0 2024-04-25T16:02:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.111+0000 7fec217fa640 1 -- 172.21.15.31:0/1408479673 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fec30017d90 con 0x56369363a2f0 2024-04-25T16:02:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.112+0000 7fec217fa640 1 -- 172.21.15.31:0/1408479673 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec3001d040 con 0x56369363a2f0 2024-04-25T16:02:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.112+0000 7fec47075740 10 client.15897 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:48.115 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.112+0000 7fec47075740 1 -- 172.21.15.31:0/1408479673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563693747620 con 0x56369363a2f0 2024-04-25T16:02:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.112+0000 7fec217fa640 1 -- 172.21.15.31:0/1408479673 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec340052a0 con 0x56369363a2f0 2024-04-25T16:02:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.113+0000 7fec217fa640 1 -- 172.21.15.31:0/1408479673 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fec30020030 con 0x56369363a2f0 2024-04-25T16:02:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.113+0000 7fec47075740 10 client.15897 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.113+0000 7fec47075740 10 client.15897 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:48.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.113+0000 7fec47075740 20 client.15897 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.113+0000 7fec47075740 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563693621d80 0x563693a1b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.113+0000 7fec47075740 4 client.15897 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.113+0000 7fec47075740 1 -- 172.21.15.31:0/1408479673 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563693977070 con 0x563693621d80 2024-04-25T16:02:48.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.115+0000 7fec3d762640 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563693621d80 0x563693a1b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.125+0000 7fec217fa640 10 client.15897 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.125+0000 7fec3d762640 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563693621d80 0x563693a1b6e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=225c6007b3d83f36 server_cookie=91f40ea2940bc3ba in_seq=0 out_seq=0 2024-04-25T16:02:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.132+0000 7fec217fa640 1 -- 172.21.15.31:0/1408479673 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563693977070 con 0x563693621d80 2024-04-25T16:02:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.132+0000 7fec217fa640 10 client.15897 handle_command_reply: tid=0 2024-04-25T16:02:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.135+0000 7fec47075740 1 client.15897 shutdown 2024-04-25T16:02:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.135+0000 7fec47075740 1 -- 172.21.15.31:0/1408479673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56369363a2f0 msgr2=0x563693a0d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:48.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.135+0000 7fec47075740 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56369363a2f0 0x563693a0d770 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fec30016e20 tx=0x7fec30004a60 comp rx=0 tx=0).stop 2024-04-25T16:02:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.136+0000 7fec47075740 1 -- 172.21.15.31:0/1408479673 shutdown_connections 2024-04-25T16:02:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.136+0000 7fec47075740 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563693621d80 0x563693a1b6e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.136+0000 7fec47075740 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5636936631b0 0x563693a14c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.136+0000 7fec47075740 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x563693a0dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.136+0000 7fec47075740 1 --2- 172.21.15.31:0/1408479673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56369363a2f0 0x563693a0d770 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.136+0000 7fec47075740 1 -- 172.21.15.31:0/1408479673 >> 172.21.15.31:0/1408479673 conn(0x563693657300 msgr2=0x5636939fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.136+0000 7fec217fa640 0 client.15897 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.136+0000 7fec47075740 1 -- 172.21.15.31:0/1408479673 shutdown_connections 2024-04-25T16:02:48.139 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.136+0000 7fec47075740 1 -- 172.21.15.31:0/1408479673 wait complete. 2024-04-25T16:02:48.141 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.140+0000 7fec47075740 20 client.15897 trim_cache size 0 max 16384 2024-04-25T16:02:48.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.183+0000 7fec47075740 1 Processor -- start 2024-04-25T16:02:48.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.183+0000 7fec47075740 1 -- start start 2024-04-25T16:02:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.184+0000 7fec47075740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x5636939fecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.185+0000 7fec47075740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x5636939ff200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.185+0000 7fec47075740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a0d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.185+0000 7fec3df63640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x5636939fecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.185+0000 7fec3df63640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x5636939fecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:46270/0 (socket says 172.21.15.31:46270) 2024-04-25T16:02:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.185+0000 7fec3df63640 1 -- 172.21.15.31:0/3411681744 learned_addr learned my addr 172.21.15.31:0/3411681744 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.185+0000 7fec3d762640 1 --2- 172.21.15.31:0/3411681744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x5636939ff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.186+0000 7fec47075740 1 -- 172.21.15.31:0/3411681744 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c00e420 con 0x5636936631b0 2024-04-25T16:02:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.186+0000 7fec47075740 1 -- 172.21.15.31:0/3411681744 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c00ff60 con 0x56369363a2f0 2024-04-25T16:02:48.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.186+0000 7fec47075740 1 -- 172.21.15.31:0/3411681744 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c00eaf0 con 0x5636936424d0 2024-04-25T16:02:48.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.188+0000 7fec3e764640 1 --2- 172.21.15.31:0/3411681744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a0d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.189+0000 7fec3df63640 1 -- 172.21.15.31:0/3411681744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 msgr2=0x5636939ff200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.189+0000 7fec3df63640 1 --2- 172.21.15.31:0/3411681744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x5636939ff200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.189+0000 7fec3df63640 1 -- 172.21.15.31:0/3411681744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 msgr2=0x563693a0d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.189+0000 7fec3df63640 1 --2- 172.21.15.31:0/3411681744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a0d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.189+0000 7fec3df63640 1 -- 172.21.15.31:0/3411681744 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563693a15a30 con 0x56369363a2f0 2024-04-25T16:02:48.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.190+0000 7fec3df63640 1 --2- 172.21.15.31:0/3411681744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x5636939fecc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec3002e040 tx=0x7fec300098b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=238d7c04ad5cce91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.192+0000 7fec237fe640 1 -- 172.21.15.31:0/3411681744 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec30025070 con 0x56369363a2f0 2024-04-25T16:02:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.192+0000 7fec237fe640 1 -- 172.21.15.31:0/3411681744 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec30015960 con 0x56369363a2f0 2024-04-25T16:02:48.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.192+0000 7fec237fe640 1 -- 172.21.15.31:0/3411681744 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec3002a400 con 0x56369363a2f0 2024-04-25T16:02:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.194+0000 7fec47075740 1 -- 172.21.15.31:0/3411681744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 msgr2=0x5636939fecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:48.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.194+0000 7fec47075740 1 --2- 172.21.15.31:0/3411681744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x5636939fecc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fec3002e040 tx=0x7fec300098b0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.195+0000 7fec47075740 1 -- 172.21.15.31:0/3411681744 shutdown_connections 2024-04-25T16:02:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.195+0000 7fec47075740 1 --2- 172.21.15.31:0/3411681744 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a0d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.195+0000 7fec47075740 1 --2- 172.21.15.31:0/3411681744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5636936424d0 0x5636939ff200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.195+0000 7fec47075740 1 --2- 172.21.15.31:0/3411681744 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56369363a2f0 0x5636939fecc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.195+0000 7fec47075740 1 -- 172.21.15.31:0/3411681744 >> 172.21.15.31:0/3411681744 conn(0x563693657300 msgr2=0x563693a07b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:48.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.196+0000 7fec47075740 1 -- 172.21.15.31:0/3411681744 shutdown_connections 2024-04-25T16:02:48.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.196+0000 7fec47075740 1 -- 172.21.15.31:0/3411681744 wait complete. 2024-04-25T16:02:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.196+0000 7fec47075740 1 -- start start 2024-04-25T16:02:48.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.197+0000 7fec47075740 1 Processor -- start 2024-04-25T16:02:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.198+0000 7fec47075740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56369363a2f0 0x563693a11360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.198+0000 7fec47075740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5636936424d0 0x563693a118a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.198+0000 7fec47075740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a81a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.198+0000 7fec47075740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c00f890 con 0x5636936631b0 2024-04-25T16:02:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.198+0000 7fec47075740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c00f1c0 con 0x5636936424d0 2024-04-25T16:02:48.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.198+0000 7fec47075740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c00c400 con 0x56369363a2f0 2024-04-25T16:02:48.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.198+0000 7fec47075740 10 client.0 _ll_register_callbacks cb 0x7fec30016ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.202+0000 7fec3d762640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5636936424d0 0x563693a118a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.203+0000 7fec3e764640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a81a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.203+0000 7fec3e764640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a81a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40300/0 (socket says 172.21.15.31:40300) 2024-04-25T16:02:48.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.203+0000 7fec3e764640 1 -- 172.21.15.31:0/1765237032 learned_addr learned my addr 172.21.15.31:0/1765237032 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.204+0000 7fec3df63640 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56369363a2f0 0x563693a11360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.206+0000 7fec3e764640 1 -- 172.21.15.31:0/1765237032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56369363a2f0 msgr2=0x563693a11360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.206+0000 7fec3e764640 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56369363a2f0 0x563693a11360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.206+0000 7fec3e764640 1 -- 172.21.15.31:0/1765237032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5636936424d0 msgr2=0x563693a118a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.206+0000 7fec3e764640 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5636936424d0 0x563693a118a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.208 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.206+0000 7fec3e764640 1 -- 172.21.15.31:0/1765237032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563693a81ff0 con 0x5636936631b0 2024-04-25T16:02:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.207+0000 7fec3e764640 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a81a80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fec280a62c0 tx=0x7fec280a5fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb731dda97cdc9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:48.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.207+0000 7fec217fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.208+0000 7fec217fa640 1 -- 172.21.15.31:0/1765237032 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec280c5060 con 0x5636936631b0 2024-04-25T16:02:48.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.209+0000 7fec217fa640 1 -- 172.21.15.31:0/1765237032 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fec28011a40 con 0x5636936631b0 2024-04-25T16:02:48.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.209+0000 7fec47075740 1 -- 172.21.15.31:0/1765237032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec30015960 con 0x5636936631b0 2024-04-25T16:02:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.210+0000 7fec217fa640 1 -- 172.21.15.31:0/1765237032 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fec280b4050 con 0x5636936631b0 2024-04-25T16:02:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.211+0000 7fec47075740 10 client.15903 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:48.212 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.211+0000 7fec47075740 1 -- 172.21.15.31:0/1765237032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563693a832b0 con 0x5636936631b0 2024-04-25T16:02:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.216+0000 7fec217fa640 1 -- 172.21.15.31:0/1765237032 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec30015960 con 0x5636936631b0 2024-04-25T16:02:48.217 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.216+0000 7fec217fa640 1 -- 172.21.15.31:0/1765237032 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fec28022060 con 0x5636936631b0 2024-04-25T16:02:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.216+0000 7fec47075740 10 client.15903 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.216+0000 7fec47075740 10 client.15903 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.216+0000 7fec47075740 20 client.15903 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.216+0000 7fec47075740 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5636935f71c0 0x563693a85c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.216+0000 7fec47075740 4 client.15903 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:02:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.216+0000 7fec47075740 1 -- 172.21.15.31:0/1765237032 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563693970ee0 con 0x5636935f71c0 2024-04-25T16:02:48.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.217+0000 7fec3df63640 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5636935f71c0 0x563693a85c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.220+0000 7fec217fa640 10 client.15903 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:48.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.220+0000 7fec3df63640 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5636935f71c0 0x563693a85c80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=450d73f5583b313f server_cookie=98aa6db2d768b6e2 in_seq=0 out_seq=0 2024-04-25T16:02:48.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.541+0000 7fec217fa640 1 -- 172.21.15.31:0/1765237032 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563693970ee0 con 0x5636935f71c0 2024-04-25T16:02:48.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.541+0000 7fec217fa640 10 client.15903 handle_command_reply: tid=0 2024-04-25T16:02:48.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.541+0000 7fec47075740 1 client.15903 shutdown 2024-04-25T16:02:48.543 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.541+0000 7fec47075740 1 -- 172.21.15.31:0/1765237032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 msgr2=0x563693a81a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:48.544 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.541+0000 7fec47075740 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a81a80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fec280a62c0 tx=0x7fec280a5fd0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.542+0000 7fec47075740 1 -- 172.21.15.31:0/1765237032 shutdown_connections 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.542+0000 7fec47075740 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5636935f71c0 0x563693a85c80 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.542+0000 7fec47075740 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5636936631b0 0x563693a81a80 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.542+0000 7fec47075740 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5636936424d0 0x563693a118a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.542+0000 7fec47075740 1 --2- 172.21.15.31:0/1765237032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56369363a2f0 0x563693a11360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.542+0000 7fec47075740 1 -- 172.21.15.31:0/1765237032 >> 172.21.15.31:0/1765237032 conn(0x563693657300 msgr2=0x563693a07b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.542+0000 7fec217fa640 0 client.15903 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.542+0000 7fec47075740 1 -- 172.21.15.31:0/1765237032 shutdown_connections 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.542+0000 7fec47075740 1 -- 172.21.15.31:0/1765237032 wait complete. 2024-04-25T16:02:48.545 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.543+0000 7fec47075740 20 client.15903 trim_cache size 0 max 16384 2024-04-25T16:02:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:02:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:02:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:02:48.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 -- 172.21.15.31:0/1716774595 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec240afcf0 msgr2=0x7fec240b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec240afcf0 0x7fec240b2200 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x563693a04d50 tx=0x7fec3000b040 comp rx=0 tx=0).stop 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 -- 172.21.15.31:0/1716774595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec38110070 msgr2=0x7fec3807bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec38110070 0x7fec3807bae0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec2800b3e0 tx=0x7fec2800b8a0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 -- 172.21.15.31:0/1716774595 shutdown_connections 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec240afcf0 0x7fec240b2200 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec38110070 0x7fec3807bae0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec3810b350 0x7fec38079590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 --2- 172.21.15.31:0/1716774595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec3810a420 0x7fec38077040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:48.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.546+0000 7fec401c7640 1 -- 172.21.15.31:0/1716774595 >> 172.21.15.31:0/1716774595 conn(0x7fec38070540 msgr2=0x7fec3810e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:48.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.549+0000 7fec401c7640 1 -- 172.21.15.31:0/1716774595 shutdown_connections 2024-04-25T16:02:48.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:02:48.550+0000 7fec401c7640 1 -- 172.21.15.31:0/1716774595 wait complete. 2024-04-25T16:02:49.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:49 smithi031 ceph-mon[29140]: pgmap v396: 97 pgs: 97 active+clean; 736 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 61 MiB/s wr, 79 op/s 2024-04-25T16:02:49.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:49 smithi063 ceph-mon[32413]: pgmap v396: 97 pgs: 97 active+clean; 736 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 61 MiB/s wr, 79 op/s 2024-04-25T16:02:49.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:49 smithi186 ceph-mon[33258]: pgmap v396: 97 pgs: 97 active+clean; 736 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 61 MiB/s wr, 79 op/s 2024-04-25T16:02:50.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:50 smithi031 ceph-mon[29140]: pgmap v397: 97 pgs: 97 active+clean; 2.0 GiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 171 MiB/s wr, 122 op/s 2024-04-25T16:02:50.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:50 smithi063 ceph-mon[32413]: pgmap v397: 97 pgs: 97 active+clean; 2.0 GiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 171 MiB/s wr, 122 op/s 2024-04-25T16:02:50.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:50 smithi186 ceph-mon[33258]: pgmap v397: 97 pgs: 97 active+clean; 2.0 GiB data, 652 MiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 171 MiB/s wr, 122 op/s 2024-04-25T16:02:51.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:02:51.921 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:02:52.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.298+0000 7fc2f77fe640 1 -- 172.21.15.31:0/2399741064 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0042b90 con 0x7fc2f8107920 2024-04-25T16:02:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.298+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/2399741064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107920 msgr2=0x7fc2f8107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.298+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/2399741064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107920 0x7fc2f8107d30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc2f001acb0 tx=0x7fc2f003b8e0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.299+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/2399741064 shutdown_connections 2024-04-25T16:02:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.299+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/2399741064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107920 0x7fc2f8107d30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.299+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/2399741064 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2f81027b0 0x7fc2f8102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.299+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/2399741064 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f8101d70 0x7fc2f81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.299+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/2399741064 >> 172.21.15.31:0/2399741064 conn(0x7fc2f80fd340 msgr2=0x7fc2f80ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.299+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/2399741064 shutdown_connections 2024-04-25T16:02:52.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.299+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/2399741064 wait complete. 2024-04-25T16:02:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2fe8bd640 1 Processor -- start 2024-04-25T16:02:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2fe8bd640 1 -- start start 2024-04-25T16:02:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2fe8bd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f8101d70 0x7fc2f806c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2fe8bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2f81027b0 0x7fc2f806cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2fe8bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107920 0x7fc2f80736d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2fe8bd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f810b420 con 0x7fc2f8101d70 2024-04-25T16:02:52.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2fe8bd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f810b2a0 con 0x7fc2f81027b0 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2fe8bd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f810b5a0 con 0x7fc2f8107920 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2f7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f8101d70 0x7fc2f806c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2f7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f8101d70 0x7fc2f806c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40312/0 (socket says 172.21.15.31:40312) 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.300+0000 7fc2f7fff640 1 -- 172.21.15.31:0/1200732409 learned_addr learned my addr 172.21.15.31:0/1200732409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.301+0000 7fc2effff640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2f81027b0 0x7fc2f806cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.301+0000 7fc2fce5a640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107920 0x7fc2f80736d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.301+0000 7fc2fce5a640 1 -- 172.21.15.31:0/1200732409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2f81027b0 msgr2=0x7fc2f806cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.301+0000 7fc2fce5a640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2f81027b0 0x7fc2f806cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.301+0000 7fc2fce5a640 1 -- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f8101d70 msgr2=0x7fc2f806c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.301+0000 7fc2fce5a640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f8101d70 0x7fc2f806c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.301+0000 7fc2fce5a640 1 -- 172.21.15.31:0/1200732409 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2f8073d40 con 0x7fc2f8107920 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.301+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f8101d70 0x7fc2f806c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:02:52.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.301+0000 7fc2fce5a640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107920 0x7fc2f80736d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2f001ac00 tx=0x7fc2f0003ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.302+0000 7fc2f5ffb640 1 -- 172.21.15.31:0/1200732409 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0042470 con 0x7fc2f8107920 2024-04-25T16:02:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.302+0000 7fc2f5ffb640 1 -- 172.21.15.31:0/1200732409 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2f0004700 con 0x7fc2f8107920 2024-04-25T16:02:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.302+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/1200732409 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2f8073fd0 con 0x7fc2f8107920 2024-04-25T16:02:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.302+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/1200732409 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2f8074570 con 0x7fc2f8107920 2024-04-25T16:02:52.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.303+0000 7fc2f5ffb640 1 -- 172.21.15.31:0/1200732409 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0054400 con 0x7fc2f8107920 2024-04-25T16:02:52.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.308+0000 7fc305772740 1 Processor -- start 2024-04-25T16:02:52.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.309+0000 7fc2f5ffb640 1 -- 172.21.15.31:0/1200732409 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc2f00545a0 con 0x7fc2f8107920 2024-04-25T16:02:52.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.309+0000 7fc305772740 1 -- start start 2024-04-25T16:02:52.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.309+0000 7fc305772740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae63390 0x55990b204a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.310+0000 7fc305772740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae3a4d0 0x55990b204fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.310+0000 7fc305772740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b205c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.310+0000 7fc305772740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55990b169d80 con 0x55990ae3a4d0 2024-04-25T16:02:52.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.310+0000 7fc2effff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae63390 0x55990b204a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.310+0000 7fc2effff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae63390 0x55990b204a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:46310/0 (socket says 172.21.15.31:46310) 2024-04-25T16:02:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.310+0000 7fc2effff640 1 -- 172.21.15.31:0/471592417 learned_addr learned my addr 172.21.15.31:0/471592417 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.311+0000 7fc305772740 1 -- 172.21.15.31:0/471592417 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55990b1238b0 con 0x55990ae63390 2024-04-25T16:02:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.311+0000 7fc305772740 1 -- 172.21.15.31:0/471592417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55990b165d40 con 0x55990ae426b0 2024-04-25T16:02:52.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.311+0000 7fc2effff640 1 --2- 172.21.15.31:0/471592417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b205c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.312+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/471592417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae3a4d0 0x55990b204fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.312+0000 7fc2effff640 1 -- 172.21.15.31:0/471592417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 msgr2=0x55990b205c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:52.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.312+0000 7fc2effff640 1 --2- 172.21.15.31:0/471592417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b205c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.312+0000 7fc2effff640 1 -- 172.21.15.31:0/471592417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae3a4d0 msgr2=0x55990b204fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.312+0000 7fc2effff640 1 --2- 172.21.15.31:0/471592417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae3a4d0 0x55990b204fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.312+0000 7fc2effff640 1 -- 172.21.15.31:0/471592417 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55990b0f4a40 con 0x55990ae63390 2024-04-25T16:02:52.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.313+0000 7fc2effff640 1 --2- 172.21.15.31:0/471592417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae63390 0x55990b204a60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00127e0 tx=0x7fc2e000b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf11a86c644d99c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:52.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.314+0000 7fc2ee7fc640 1 -- 172.21.15.31:0/471592417 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0012e20 con 0x55990ae63390 2024-04-25T16:02:52.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.315+0000 7fc2f5ffb640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2e40afc30 0x7fc2e40b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.316+0000 7fc2ee7fc640 1 -- 172.21.15.31:0/471592417 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e00120d0 con 0x55990ae63390 2024-04-25T16:02:52.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.316+0000 7fc2ee7fc640 1 -- 172.21.15.31:0/471592417 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e00106c0 con 0x55990ae63390 2024-04-25T16:02:52.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.317+0000 7fc2f5ffb640 1 -- 172.21.15.31:0/1200732409 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc2f01119b0 con 0x7fc2f8107920 2024-04-25T16:02:52.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.317+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2e40afc30 0x7fc2e40b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.320+0000 7fc305772740 1 -- 172.21.15.31:0/471592417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae63390 msgr2=0x55990b204a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:52.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.320+0000 7fc305772740 1 --2- 172.21.15.31:0/471592417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae63390 0x55990b204a60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc2e00127e0 tx=0x7fc2e000b770 comp rx=0 tx=0).stop 2024-04-25T16:02:52.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.321+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2e40afc30 0x7fc2e40b2140 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55990b204780 tx=0x7fc2e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:52.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.325+0000 7fc305772740 1 -- 172.21.15.31:0/471592417 shutdown_connections 2024-04-25T16:02:52.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.325+0000 7fc305772740 1 --2- 172.21.15.31:0/471592417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b205c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.325+0000 7fc305772740 1 --2- 172.21.15.31:0/471592417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae3a4d0 0x55990b204fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.325+0000 7fc305772740 1 --2- 172.21.15.31:0/471592417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae63390 0x55990b204a60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.326+0000 7fc305772740 1 -- 172.21.15.31:0/471592417 >> 172.21.15.31:0/471592417 conn(0x55990ae574e0 msgr2=0x55990b200270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:52.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.326+0000 7fc305772740 1 -- 172.21.15.31:0/471592417 shutdown_connections 2024-04-25T16:02:52.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.326+0000 7fc305772740 1 -- 172.21.15.31:0/471592417 wait complete. 2024-04-25T16:02:52.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.327+0000 7fc305772740 1 -- start start 2024-04-25T16:02:52.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.328+0000 7fc305772740 1 Processor -- start 2024-04-25T16:02:52.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.328+0000 7fc305772740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae3a4d0 0x55990b20e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.329+0000 7fc305772740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae426b0 0x55990b20ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.329+0000 7fc305772740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae63390 0x55990b215ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.329+0000 7fc305772740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55990b206720 con 0x55990ae3a4d0 2024-04-25T16:02:52.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.329+0000 7fc305772740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55990b0f0d10 con 0x55990ae426b0 2024-04-25T16:02:52.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.329+0000 7fc2effff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae426b0 0x55990b20ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.329+0000 7fc2effff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae426b0 0x55990b20ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:46324/0 (socket says 172.21.15.31:46324) 2024-04-25T16:02:52.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.329+0000 7fc2effff640 1 -- 172.21.15.31:0/1413876461 learned_addr learned my addr 172.21.15.31:0/1413876461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:52.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.329+0000 7fc305772740 1 -- 172.21.15.31:0/1413876461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55990b2068a0 con 0x55990ae63390 2024-04-25T16:02:52.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.330+0000 7fc305772740 10 client.0 _ll_register_callbacks cb 0x55990b11d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:52.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.330+0000 7fc2fce5a640 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae63390 0x55990b215ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.332+0000 7fc2effff640 1 -- 172.21.15.31:0/1413876461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae63390 msgr2=0x55990b215ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:52.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.332+0000 7fc2effff640 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae63390 0x55990b215ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.332+0000 7fc2effff640 1 -- 172.21.15.31:0/1413876461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae3a4d0 msgr2=0x55990b20e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:52.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.332+0000 7fc2effff640 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae3a4d0 0x55990b20e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.332+0000 7fc2effff640 1 -- 172.21.15.31:0/1413876461 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55990af457d0 con 0x55990ae426b0 2024-04-25T16:02:52.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.334+0000 7fc2effff640 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae426b0 0x55990b20ed80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000ec80 tx=0x7fc2e000ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=593c334419a3ffc9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:52.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.334+0000 7fc2d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:02:52.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.335+0000 7fc2d77fe640 1 -- 172.21.15.31:0/1413876461 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e000ba60 con 0x55990ae426b0 2024-04-25T16:02:52.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.335+0000 7fc2d77fe640 1 -- 172.21.15.31:0/1413876461 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc2e0005450 con 0x55990ae426b0 2024-04-25T16:02:52.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.335+0000 7fc305772740 1 -- 172.21.15.31:0/1413876461 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc2e00120d0 con 0x55990ae426b0 2024-04-25T16:02:52.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.344+0000 7fc2d77fe640 1 -- 172.21.15.31:0/1413876461 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc2e0005650 con 0x55990ae426b0 2024-04-25T16:02:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.345+0000 7fc305772740 10 client.25613 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.345+0000 7fc305772740 1 -- 172.21.15.31:0/1413876461 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55990ae7b3f0 con 0x55990ae426b0 2024-04-25T16:02:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.352+0000 7fc2d77fe640 1 -- 172.21.15.31:0/1413876461 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e0004b70 con 0x55990ae426b0 2024-04-25T16:02:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.352+0000 7fc2d77fe640 1 -- 172.21.15.31:0/1413876461 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc2e0010a20 con 0x55990ae426b0 2024-04-25T16:02:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.352+0000 7fc305772740 10 client.25613 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.352+0000 7fc305772740 10 client.25613 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.352+0000 7fc305772740 20 client.25613 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.352+0000 7fc305772740 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55990ae21f60 0x55990b21bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.352+0000 7fc305772740 4 client.25613 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.352+0000 7fc305772740 1 -- 172.21.15.31:0/1413876461 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc2e00120d0 con 0x55990ae21f60 2024-04-25T16:02:52.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.356+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55990ae21f60 0x55990b21bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.368+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55990ae21f60 0x55990b21bde0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e4b2331f4bc8c3e server_cookie=3afe01cb88586722 in_seq=0 out_seq=0 2024-04-25T16:02:52.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.372+0000 7fc2d77fe640 10 client.25613 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:52.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.374+0000 7fc2d77fe640 1 -- 172.21.15.31:0/1413876461 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fc2e00120d0 con 0x55990ae21f60 2024-04-25T16:02:52.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.374+0000 7fc2d77fe640 10 client.25613 handle_command_reply: tid=0 2024-04-25T16:02:52.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.374+0000 7fc305772740 1 client.25613 shutdown 2024-04-25T16:02:52.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.375+0000 7fc305772740 1 -- 172.21.15.31:0/1413876461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae426b0 msgr2=0x55990b20ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:52.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.375+0000 7fc305772740 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae426b0 0x55990b20ed80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc2e000ec80 tx=0x7fc2e000ecb0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.375+0000 7fc305772740 1 -- 172.21.15.31:0/1413876461 shutdown_connections 2024-04-25T16:02:52.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.375+0000 7fc305772740 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55990ae21f60 0x55990b21bde0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.375+0000 7fc305772740 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae63390 0x55990b215ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.375+0000 7fc2d77fe640 0 client.25613 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:52.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.376+0000 7fc305772740 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae426b0 0x55990b20ed80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.376+0000 7fc305772740 1 --2- 172.21.15.31:0/1413876461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae3a4d0 0x55990b20e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.376+0000 7fc305772740 1 -- 172.21.15.31:0/1413876461 >> 172.21.15.31:0/1413876461 conn(0x55990ae574e0 msgr2=0x55990b1ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:52.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.377+0000 7fc305772740 1 -- 172.21.15.31:0/1413876461 shutdown_connections 2024-04-25T16:02:52.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.377+0000 7fc305772740 1 -- 172.21.15.31:0/1413876461 wait complete. 2024-04-25T16:02:52.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.377+0000 7fc305772740 20 client.25613 trim_cache size 0 max 16384 2024-04-25T16:02:52.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.453+0000 7fc305772740 1 Processor -- start 2024-04-25T16:02:52.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.456+0000 7fc305772740 1 -- start start 2024-04-25T16:02:52.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.456+0000 7fc305772740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae21f60 0x55990b20cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.456+0000 7fc305772740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 0x55990b20d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.456+0000 7fc305772740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b215c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.456+0000 7fc305772740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81b1870 con 0x55990ae21f60 2024-04-25T16:02:52.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.456+0000 7fc305772740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81b33b0 con 0x55990ae3a4d0 2024-04-25T16:02:52.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.456+0000 7fc305772740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81b1f40 con 0x55990ae426b0 2024-04-25T16:02:52.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.457+0000 7fc2fce5a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b215c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.458+0000 7fc2effff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 0x55990b20d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.458+0000 7fc2f7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae21f60 0x55990b20cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.458+0000 7fc2f7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae21f60 0x55990b20cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40330/0 (socket says 172.21.15.31:40330) 2024-04-25T16:02:52.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.458+0000 7fc2f7fff640 1 -- 172.21.15.31:0/3416780424 learned_addr learned my addr 172.21.15.31:0/3416780424 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:52.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.460+0000 7fc2f7fff640 1 -- 172.21.15.31:0/3416780424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 msgr2=0x55990b215c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:52.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.460+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/3416780424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b215c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.460+0000 7fc2f7fff640 1 -- 172.21.15.31:0/3416780424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 msgr2=0x55990b20d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:52.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.460+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/3416780424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 0x55990b20d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.460+0000 7fc2f7fff640 1 -- 172.21.15.31:0/3416780424 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55990ae3b0f0 con 0x55990ae21f60 2024-04-25T16:02:52.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.462+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/3416780424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae21f60 0x55990b20cf00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8004630 tx=0x7fc2e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1390691a0d87053c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:52.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.464+0000 7fc2edffb640 1 -- 172.21.15.31:0/3416780424 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e802d070 con 0x55990ae21f60 2024-04-25T16:02:52.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.464+0000 7fc2edffb640 1 -- 172.21.15.31:0/3416780424 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc2e80054d0 con 0x55990ae21f60 2024-04-25T16:02:52.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.466+0000 7fc305772740 1 -- 172.21.15.31:0/3416780424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae21f60 msgr2=0x55990b20cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:52.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.466+0000 7fc305772740 1 --2- 172.21.15.31:0/3416780424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae21f60 0x55990b20cf00 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8004630 tx=0x7fc2e8017000 comp rx=0 tx=0).stop 2024-04-25T16:02:52.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.466+0000 7fc305772740 1 -- 172.21.15.31:0/3416780424 shutdown_connections 2024-04-25T16:02:52.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.466+0000 7fc305772740 1 --2- 172.21.15.31:0/3416780424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b215c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.466+0000 7fc305772740 1 --2- 172.21.15.31:0/3416780424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 0x55990b20d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.466+0000 7fc305772740 1 --2- 172.21.15.31:0/3416780424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae21f60 0x55990b20cf00 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.466+0000 7fc305772740 1 -- 172.21.15.31:0/3416780424 >> 172.21.15.31:0/3416780424 conn(0x55990ae574e0 msgr2=0x55990b209590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:52.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.470+0000 7fc305772740 1 -- 172.21.15.31:0/3416780424 shutdown_connections 2024-04-25T16:02:52.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.470+0000 7fc305772740 1 -- 172.21.15.31:0/3416780424 wait complete. 2024-04-25T16:02:52.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.471+0000 7fc305772740 1 -- start start 2024-04-25T16:02:52.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.471+0000 7fc305772740 1 Processor -- start 2024-04-25T16:02:52.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.472+0000 7fc305772740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 0x55990b210d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.472+0000 7fc305772740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b216200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.472+0000 7fc305772740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae63390 0x55990b2112b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.472+0000 7fc305772740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81b2ce0 con 0x55990ae63390 2024-04-25T16:02:52.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.472+0000 7fc305772740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81b2610 con 0x55990ae3a4d0 2024-04-25T16:02:52.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.472+0000 7fc305772740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2f81af850 con 0x55990ae426b0 2024-04-25T16:02:52.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.472+0000 7fc305772740 10 client.0 _ll_register_callbacks cb 0x7fc2e000be50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:52.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.473+0000 7fc2effff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b216200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.473+0000 7fc2effff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b216200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:49334/0 (socket says 172.21.15.31:49334) 2024-04-25T16:02:52.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.473+0000 7fc2effff640 1 -- 172.21.15.31:0/3891503244 learned_addr learned my addr 172.21.15.31:0/3891503244 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:52.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.473+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 0x55990b210d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.474+0000 7fc2fce5a640 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae63390 0x55990b2112b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.474+0000 7fc2f7fff640 1 -- 172.21.15.31:0/3891503244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 msgr2=0x55990b216200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:52.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.474+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b216200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.474+0000 7fc2f7fff640 1 -- 172.21.15.31:0/3891503244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae63390 msgr2=0x55990b2112b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:52.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.474+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae63390 0x55990b2112b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.474+0000 7fc2f7fff640 1 -- 172.21.15.31:0/3891503244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55990b12a4b0 con 0x55990ae3a4d0 2024-04-25T16:02:52.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.477+0000 7fc2f7fff640 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 0x55990b210d70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55990b2090e0 tx=0x7fc2e80042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a33a50e20641bbf3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:52.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.484+0000 7fc2d77fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:02:52.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.484+0000 7fc2d77fe640 1 -- 172.21.15.31:0/3891503244 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2e802d050 con 0x55990ae3a4d0 2024-04-25T16:02:52.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.484+0000 7fc2d77fe640 1 -- 172.21.15.31:0/3891503244 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc2e80299f0 con 0x55990ae3a4d0 2024-04-25T16:02:52.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.484+0000 7fc305772740 1 -- 172.21.15.31:0/3891503244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55990b177990 con 0x55990ae3a4d0 2024-04-25T16:02:52.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.486+0000 7fc2d77fe640 1 -- 172.21.15.31:0/3891503244 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc2e8005030 con 0x55990ae3a4d0 2024-04-25T16:02:52.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.486+0000 7fc305772740 10 client.25619 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:52.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.486+0000 7fc305772740 1 -- 172.21.15.31:0/3891503244 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55990b2830d0 con 0x55990ae3a4d0 2024-04-25T16:02:52.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.487+0000 7fc2d77fe640 1 -- 172.21.15.31:0/3891503244 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55990b177990 con 0x55990ae3a4d0 2024-04-25T16:02:52.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.487+0000 7fc2d77fe640 1 -- 172.21.15.31:0/3891503244 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc2e80035f0 con 0x55990ae3a4d0 2024-04-25T16:02:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.487+0000 7fc305772740 10 client.25619 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.487+0000 7fc305772740 10 client.25619 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.487+0000 7fc305772740 20 client.25619 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.487+0000 7fc305772740 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55990adf73a0 0x55990b2859f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.487+0000 7fc305772740 4 client.25619 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:02:52.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.487+0000 7fc305772740 1 -- 172.21.15.31:0/3891503244 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55990b171800 con 0x55990adf73a0 2024-04-25T16:02:52.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.488+0000 7fc2effff640 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55990adf73a0 0x55990b2859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:52.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.491+0000 7fc2effff640 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55990adf73a0 0x55990b2859f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=143b41012b64a3b1 server_cookie=9047947d55af8451 in_seq=0 out_seq=0 2024-04-25T16:02:52.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.491+0000 7fc2d77fe640 10 client.25619 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:52.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.495+0000 7fc2d77fe640 1 -- 172.21.15.31:0/3891503244 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55990b171800 con 0x55990adf73a0 2024-04-25T16:02:52.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.495+0000 7fc2d77fe640 10 client.25619 handle_command_reply: tid=0 2024-04-25T16:02:52.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.495+0000 7fc305772740 1 client.25619 shutdown 2024-04-25T16:02:52.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 -- 172.21.15.31:0/3891503244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 msgr2=0x55990b210d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:52.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 0x55990b210d70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55990b2090e0 tx=0x7fc2e80042e0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 -- 172.21.15.31:0/3891503244 shutdown_connections 2024-04-25T16:02:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55990adf73a0 0x55990b2859f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55990ae63390 0x55990b2112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55990ae426b0 0x55990b216200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 --2- 172.21.15.31:0/3891503244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55990ae3a4d0 0x55990b210d70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 -- 172.21.15.31:0/3891503244 >> 172.21.15.31:0/3891503244 conn(0x55990ae574e0 msgr2=0x55990b20cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:52.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc2d77fe640 0 client.25619 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:52.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 -- 172.21.15.31:0/3891503244 shutdown_connections 2024-04-25T16:02:52.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 1 -- 172.21.15.31:0/3891503244 wait complete. 2024-04-25T16:02:52.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.496+0000 7fc305772740 20 client.25619 trim_cache size 0 max 16384 2024-04-25T16:02:52.500 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:02:52.500 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:02:52.500 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:02:52.500 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:02:52.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.498+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2e40afc30 msgr2=0x7fc2e40b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.498+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2e40afc30 0x7fc2e40b2140 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55990b204780 tx=0x7fc2e8005f90 comp rx=0 tx=0).stop 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.498+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/1200732409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107920 msgr2=0x7fc2f80736d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.498+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107920 0x7fc2f80736d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc2f001ac00 tx=0x7fc2f0003ec0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.499+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/1200732409 shutdown_connections 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.499+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2e40afc30 0x7fc2e40b2140 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.499+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2f8107920 0x7fc2f80736d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.499+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc2f81027b0 0x7fc2f806cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.499+0000 7fc2fe8bd640 1 --2- 172.21.15.31:0/1200732409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc2f8101d70 0x7fc2f806c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.499+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/1200732409 >> 172.21.15.31:0/1200732409 conn(0x7fc2f80fd340 msgr2=0x7fc2f80fea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.499+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/1200732409 shutdown_connections 2024-04-25T16:02:52.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:52.499+0000 7fc2fe8bd640 1 -- 172.21.15.31:0/1200732409 wait complete. 2024-04-25T16:02:52.531 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:02:52.531 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:02:52.653 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:52 smithi031 ceph-mon[29140]: pgmap v398: 97 pgs: 97 active+clean; 2.5 GiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 211 MiB/s wr, 131 op/s 2024-04-25T16:02:52.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:52 smithi063 ceph-mon[32413]: pgmap v398: 97 pgs: 97 active+clean; 2.5 GiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 211 MiB/s wr, 131 op/s 2024-04-25T16:02:52.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:52 smithi186 ceph-mon[33258]: pgmap v398: 97 pgs: 97 active+clean; 2.5 GiB data, 733 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 211 MiB/s wr, 131 op/s 2024-04-25T16:02:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.015+0000 7f835759e640 1 -- 172.21.15.31:0/666345383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835810fe90 msgr2=0x7f83581102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.015+0000 7f835759e640 1 --2- 172.21.15.31:0/666345383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835810fe90 0x7f83581102a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8350009be0 tx=0x7f835002a6f0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.016+0000 7f835759e640 1 -- 172.21.15.31:0/666345383 shutdown_connections 2024-04-25T16:02:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.016+0000 7f835759e640 1 --2- 172.21.15.31:0/666345383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835810fe90 0x7f83581102a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.016+0000 7f835759e640 1 --2- 172.21.15.31:0/666345383 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f835810b190 0x7f835810b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.016+0000 7f835759e640 1 --2- 172.21.15.31:0/666345383 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f835810a340 0x7f835810ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.016+0000 7f835759e640 1 -- 172.21.15.31:0/666345383 >> 172.21.15.31:0/666345383 conn(0x7f8358070320 msgr2=0x7f83580727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:53.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.016+0000 7f835759e640 1 -- 172.21.15.31:0/666345383 shutdown_connections 2024-04-25T16:02:53.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.017+0000 7f835759e640 1 -- 172.21.15.31:0/666345383 wait complete. 2024-04-25T16:02:53.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.018+0000 7f835759e640 1 Processor -- start 2024-04-25T16:02:53.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.018+0000 7f835759e640 1 -- start start 2024-04-25T16:02:53.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.019+0000 7f835759e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f835810a340 0x7f8358117020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:53.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.019+0000 7f835759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835810b190 0x7f8358117560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:53.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.019+0000 7f835759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f835810fe90 0x7f83581184c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:53.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.019+0000 7f835759e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f835811e2a0 con 0x7f835810fe90 2024-04-25T16:02:53.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.019+0000 7f835759e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f835811e120 con 0x7f835810a340 2024-04-25T16:02:53.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.019+0000 7f835759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f835811e420 con 0x7f835810b190 2024-04-25T16:02:53.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.020+0000 7f8356d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f835810fe90 0x7f83581184c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:53.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.020+0000 7f8356d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f835810fe90 0x7f83581184c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:40346/0 (socket says 172.21.15.31:40346) 2024-04-25T16:02:53.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.020+0000 7f8356d9d640 1 -- 172.21.15.31:0/2130343783 learned_addr learned my addr 172.21.15.31:0/2130343783 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:53.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.020+0000 7f835659c640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f835810a340 0x7f8358117020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:53.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.021+0000 7f8356d9d640 1 -- 172.21.15.31:0/2130343783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835810b190 msgr2=0x7f8358117560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:53.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.021+0000 7f8356d9d640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835810b190 0x7f8358117560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.021+0000 7f8356d9d640 1 -- 172.21.15.31:0/2130343783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f835810a340 msgr2=0x7f8358117020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:53.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.021+0000 7f8356d9d640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f835810a340 0x7f8358117020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.021+0000 7f8356d9d640 1 -- 172.21.15.31:0/2130343783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8358118b30 con 0x7f835810fe90 2024-04-25T16:02:53.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.021+0000 7f8356d9d640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f835810fe90 0x7f83581184c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8350034040 tx=0x7f8350030d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:53.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.023+0000 7f83477fe640 1 -- 172.21.15.31:0/2130343783 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350041020 con 0x7f835810fe90 2024-04-25T16:02:53.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.023+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8358118dc0 con 0x7f835810fe90 2024-04-25T16:02:53.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.023+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8358119330 con 0x7f835810fe90 2024-04-25T16:02:53.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.027+0000 7f83477fe640 1 -- 172.21.15.31:0/2130343783 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83500318f0 con 0x7f835810fe90 2024-04-25T16:02:53.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.029+0000 7f83477fe640 1 -- 172.21.15.31:0/2130343783 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8350036040 con 0x7f835810fe90 2024-04-25T16:02:53.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.031+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83380052c0 con 0x7f835810fe90 2024-04-25T16:02:53.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.034+0000 7f83477fe640 1 -- 172.21.15.31:0/2130343783 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8350044020 con 0x7f835810fe90 2024-04-25T16:02:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.038+0000 7f83477fe640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83400afb10 0x7f83400b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:53.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.039+0000 7f83477fe640 1 -- 172.21.15.31:0/2130343783 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f83500ff490 con 0x7f835810fe90 2024-04-25T16:02:53.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.040+0000 7f835659c640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83400afb10 0x7f83400b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:53.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.041+0000 7f835659c640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83400afb10 0x7f83400b2020 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8348006fd0 tx=0x7f8348008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:53.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.050+0000 7f83477fe640 1 -- 172.21.15.31:0/2130343783 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f835004b070 con 0x7f835810fe90 2024-04-25T16:02:53.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.880+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f83380055b0 con 0x7f835810fe90 2024-04-25T16:02:53.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.885+0000 7f83477fe640 1 -- 172.21.15.31:0/2130343783 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11917 (secure 0 0 0) 0x7f83500c8020 con 0x7f835810fe90 2024-04-25T16:02:53.886 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:02:53.886 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:01:09.454389+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24535,24521,24563,14631,24553]},"id":1}]} 2024-04-25T16:02:53.886 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 27 2024-04-25T16:02:53.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.892+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83400afb10 msgr2=0x7f83400b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:53.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.892+0000 7f835759e640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83400afb10 0x7f83400b2020 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8348006fd0 tx=0x7f8348008040 comp rx=0 tx=0).stop 2024-04-25T16:02:53.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.892+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f835810fe90 msgr2=0x7f83581184c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:53.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.892+0000 7f835759e640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f835810fe90 0x7f83581184c0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8350034040 tx=0x7f8350030d20 comp rx=0 tx=0).stop 2024-04-25T16:02:53.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.893+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 shutdown_connections 2024-04-25T16:02:53.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.893+0000 7f835759e640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83400afb10 0x7f83400b2020 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.893+0000 7f835759e640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f835810fe90 0x7f83581184c0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.893+0000 7f835759e640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835810b190 0x7f8358117560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.893+0000 7f835759e640 1 --2- 172.21.15.31:0/2130343783 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f835810a340 0x7f8358117020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:53.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.893+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 >> 172.21.15.31:0/2130343783 conn(0x7f8358070320 msgr2=0x7f835810be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:53.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.895+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 shutdown_connections 2024-04-25T16:02:53.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:53.895+0000 7f835759e640 1 -- 172.21.15.31:0/2130343783 wait complete. 2024-04-25T16:02:53.922 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:02:54.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.241+0000 7fb298613640 1 -- 172.21.15.31:0/3843559362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2901029f0 msgr2=0x7fb2901036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:54.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.241+0000 7fb298613640 1 --2- 172.21.15.31:0/3843559362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2901029f0 0x7fb2901036f0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009860 tx=0x7fb28c02a340 comp rx=0 tx=0).stop 2024-04-25T16:02:54.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.242+0000 7fb298613640 1 -- 172.21.15.31:0/3843559362 shutdown_connections 2024-04-25T16:02:54.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.242+0000 7fb298613640 1 --2- 172.21.15.31:0/3843559362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb290108930 0x7fb29010ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.242+0000 7fb298613640 1 --2- 172.21.15.31:0/3843559362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb290103c30 0x7fb290104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.242+0000 7fb298613640 1 --2- 172.21.15.31:0/3843559362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2901029f0 0x7fb2901036f0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.242+0000 7fb298613640 1 -- 172.21.15.31:0/3843559362 >> 172.21.15.31:0/3843559362 conn(0x7fb2900fdc00 msgr2=0x7fb290100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.243+0000 7fb298613640 1 -- 172.21.15.31:0/3843559362 shutdown_connections 2024-04-25T16:02:54.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.243+0000 7fb298613640 1 -- 172.21.15.31:0/3843559362 wait complete. 2024-04-25T16:02:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.245+0000 7fb298613640 1 Processor -- start 2024-04-25T16:02:54.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.246+0000 7fb298613640 1 -- start start 2024-04-25T16:02:54.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.252+0000 7fb298613640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2901029f0 0x7fb290198fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.253+0000 7fb298613640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb290103c30 0x7fb290199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.253+0000 7fb2963af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2901029f0 0x7fb290198fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.253+0000 7fb2963af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2901029f0 0x7fb290198fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53478/0 (socket says 172.21.15.31:53478) 2024-04-25T16:02:54.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.254+0000 7fb2963af640 1 -- 172.21.15.31:0/117904960 learned_addr learned my addr 172.21.15.31:0/117904960 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:54.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.254+0000 7fb298613640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb290108930 0x7fb2901a04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.254+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb29010e4f0 con 0x7fb2901029f0 2024-04-25T16:02:54.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.254+0000 7fb295bae640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb290103c30 0x7fb290199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.254+0000 7fb296bb0640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb290108930 0x7fb2901a04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.254+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb29010e370 con 0x7fb290103c30 2024-04-25T16:02:54.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.255+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb29010e670 con 0x7fb290108930 2024-04-25T16:02:54.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.256+0000 7fb296bb0640 1 -- 172.21.15.31:0/117904960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb290103c30 msgr2=0x7fb290199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:54.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.256+0000 7fb296bb0640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb290103c30 0x7fb290199510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.256+0000 7fb296bb0640 1 -- 172.21.15.31:0/117904960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2901029f0 msgr2=0x7fb290198fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:54.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.257+0000 7fb296bb0640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2901029f0 0x7fb290198fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.257+0000 7fb296bb0640 1 -- 172.21.15.31:0/117904960 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2901a0b10 con 0x7fb290108930 2024-04-25T16:02:54.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.262+0000 7fb296bb0640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb290108930 0x7fb2901a04a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb28800ed70 tx=0x7fb28800c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.264+0000 7fb2877fe640 1 -- 172.21.15.31:0/117904960 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28800ee60 con 0x7fb290108930 2024-04-25T16:02:54.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.264+0000 7fb2877fe640 1 -- 172.21.15.31:0/117904960 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb288003b70 con 0x7fb290108930 2024-04-25T16:02:54.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.264+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29006b290 con 0x7fb290108930 2024-04-25T16:02:54.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.265+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29006b7e0 con 0x7fb290108930 2024-04-25T16:02:54.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.265+0000 7fb2877fe640 1 -- 172.21.15.31:0/117904960 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb288017820 con 0x7fb290108930 2024-04-25T16:02:54.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.270+0000 7fb2877fe640 1 -- 172.21.15.31:0/117904960 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb2880179c0 con 0x7fb290108930 2024-04-25T16:02:54.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.274+0000 7fb2877fe640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb27c0afb70 0x7fb27c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.274+0000 7fb2877fe640 1 -- 172.21.15.31:0/117904960 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb2880ded50 con 0x7fb290108930 2024-04-25T16:02:54.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.274+0000 7fb2963af640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb27c0afb70 0x7fb27c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.275+0000 7fb29f5a0740 1 Processor -- start 2024-04-25T16:02:54.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.275+0000 7fb29f5a0740 1 -- start start 2024-04-25T16:02:54.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.276+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 0x5597b3270c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.276+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea14d0 0x5597b32711a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.276+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 0x5597b3271e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.276+0000 7fb29f5a0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5597b31d0d80 con 0x5597b2eca390 2024-04-25T16:02:54.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.276+0000 7fb29f5a0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5597b318aa20 con 0x5597b2ea96b0 2024-04-25T16:02:54.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.276+0000 7fb29f5a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5597b31ccd40 con 0x5597b2ea14d0 2024-04-25T16:02:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.276+0000 7fb2963af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 0x5597b3270c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.277+0000 7fb296bb0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 0x5597b3271e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.277+0000 7fb296bb0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 0x5597b3271e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52624/0 (socket says 172.21.15.31:52624) 2024-04-25T16:02:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.277+0000 7fb296bb0640 1 -- 172.21.15.31:0/2928538447 learned_addr learned my addr 172.21.15.31:0/2928538447 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.277+0000 7fb295bae640 1 --2- 172.21.15.31:0/2928538447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea14d0 0x5597b32711a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.277+0000 7fb2963af640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb27c0afb70 0x7fb27c0b2080 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009830 tx=0x7fb28c030340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.278+0000 7fb2963af640 1 -- 172.21.15.31:0/2928538447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea14d0 msgr2=0x5597b32711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.278+0000 7fb2963af640 1 --2- 172.21.15.31:0/2928538447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea14d0 0x5597b32711a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.278+0000 7fb2963af640 1 -- 172.21.15.31:0/2928538447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 msgr2=0x5597b3271e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.278+0000 7fb2963af640 1 --2- 172.21.15.31:0/2928538447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 0x5597b3271e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.278+0000 7fb2963af640 1 -- 172.21.15.31:0/2928538447 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597b3195c30 con 0x5597b2eca390 2024-04-25T16:02:54.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.279+0000 7fb2963af640 1 --2- 172.21.15.31:0/2928538447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 0x5597b3270c20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb28c02a930 tx=0x7fb28c03b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef073c1a4bdcb67b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:54.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.280+0000 7fb284bfe640 1 -- 172.21.15.31:0/2928538447 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c04c070 con 0x5597b2eca390 2024-04-25T16:02:54.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.280+0000 7fb284bfe640 1 -- 172.21.15.31:0/2928538447 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb28c042070 con 0x5597b2eca390 2024-04-25T16:02:54.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.280+0000 7fb284bfe640 1 -- 172.21.15.31:0/2928538447 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c037040 con 0x5597b2eca390 2024-04-25T16:02:54.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.282+0000 7fb29f5a0740 1 -- 172.21.15.31:0/2928538447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 msgr2=0x5597b3270c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:54.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.282+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/2928538447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 0x5597b3270c20 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fb28c02a930 tx=0x7fb28c03b000 comp rx=0 tx=0).stop 2024-04-25T16:02:54.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.282+0000 7fb29f5a0740 1 -- 172.21.15.31:0/2928538447 shutdown_connections 2024-04-25T16:02:54.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.282+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/2928538447 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 0x5597b3271e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.282+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/2928538447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea14d0 0x5597b32711a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.282+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/2928538447 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 0x5597b3270c20 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.282+0000 7fb29f5a0740 1 -- 172.21.15.31:0/2928538447 >> 172.21.15.31:0/2928538447 conn(0x5597b2ebe4e0 msgr2=0x5597b3268020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:54.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.285+0000 7fb29f5a0740 1 -- 172.21.15.31:0/2928538447 shutdown_connections 2024-04-25T16:02:54.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.286+0000 7fb29f5a0740 1 -- 172.21.15.31:0/2928538447 wait complete. 2024-04-25T16:02:54.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.287+0000 7fb29f5a0740 1 -- start start 2024-04-25T16:02:54.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.288+0000 7fb29f5a0740 1 Processor -- start 2024-04-25T16:02:54.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.289+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea14d0 0x5597b326fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.289+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea96b0 0x5597b326ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.289+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 0x5597b326c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.289+0000 7fb29f5a0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5597b32728e0 con 0x5597b2ea14d0 2024-04-25T16:02:54.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.289+0000 7fb29f5a0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5597b3157dc0 con 0x5597b2eca390 2024-04-25T16:02:54.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.289+0000 7fb29f5a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5597b3272a60 con 0x5597b2ea96b0 2024-04-25T16:02:54.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.289+0000 7fb29f5a0740 10 client.0 _ll_register_callbacks cb 0x5597b3173600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:54.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.290+0000 7fb2963af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea14d0 0x5597b326fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.290+0000 7fb2963af640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea14d0 0x5597b326fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53496/0 (socket says 172.21.15.31:53496) 2024-04-25T16:02:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.290+0000 7fb2963af640 1 -- 172.21.15.31:0/3685709189 learned_addr learned my addr 172.21.15.31:0/3685709189 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.290+0000 7fb295bae640 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea96b0 0x5597b326ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.291+0000 7fb296bb0640 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 0x5597b326c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.291+0000 7fb2963af640 1 -- 172.21.15.31:0/3685709189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea96b0 msgr2=0x5597b326ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.291+0000 7fb2963af640 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea96b0 0x5597b326ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.291+0000 7fb2963af640 1 -- 172.21.15.31:0/3685709189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 msgr2=0x5597b326c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.291+0000 7fb2963af640 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 0x5597b326c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.291+0000 7fb2963af640 1 -- 172.21.15.31:0/3685709189 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597b2ee23f0 con 0x5597b2ea14d0 2024-04-25T16:02:54.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.292+0000 7fb2963af640 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea14d0 0x5597b326fa90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009340 tx=0x7fb28c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f09b09c2e3f988 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:54.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.292+0000 7fb279ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:54.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.293+0000 7fb279ffb640 1 -- 172.21.15.31:0/3685709189 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c058020 con 0x5597b2ea14d0 2024-04-25T16:02:54.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.294+0000 7fb279ffb640 1 -- 172.21.15.31:0/3685709189 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb28c005250 con 0x5597b2ea14d0 2024-04-25T16:02:54.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.296+0000 7fb29f5a0740 1 -- 172.21.15.31:0/3685709189 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb28c042070 con 0x5597b2ea14d0 2024-04-25T16:02:54.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.327+0000 7fb279ffb640 1 -- 172.21.15.31:0/3685709189 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb28c04c040 con 0x5597b2ea14d0 2024-04-25T16:02:54.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.328+0000 7fb29f5a0740 10 client.15936 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:54.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.328+0000 7fb29f5a0740 1 -- 172.21.15.31:0/3685709189 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5597b2fac7d0 con 0x5597b2ea14d0 2024-04-25T16:02:54.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.357+0000 7fb279ffb640 1 -- 172.21.15.31:0/3685709189 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb28c042070 con 0x5597b2ea14d0 2024-04-25T16:02:54.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.357+0000 7fb279ffb640 1 -- 172.21.15.31:0/3685709189 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb28c047070 con 0x5597b2ea14d0 2024-04-25T16:02:54.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.358+0000 7fb29f5a0740 10 client.15936 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:54.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.358+0000 7fb29f5a0740 10 client.15936 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:02:54.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.358+0000 7fb29f5a0740 20 client.15936 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:54.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.358+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5597b2e88f60 0x5597b327fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.358+0000 7fb29f5a0740 4 client.15936 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:54.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.358+0000 7fb29f5a0740 1 -- 172.21.15.31:0/3685709189 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597b31de990 con 0x5597b2e88f60 2024-04-25T16:02:54.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.359+0000 7fb295bae640 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5597b2e88f60 0x5597b327fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.373+0000 7fb279ffb640 10 client.15936 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:02:54.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.373+0000 7fb295bae640 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5597b2e88f60 0x5597b327fe60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c40215f6b9dad64c server_cookie=916d0b8b5670198a in_seq=0 out_seq=0 2024-04-25T16:02:54.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.382+0000 7fb279ffb640 1 -- 172.21.15.31:0/3685709189 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5597b31de990 con 0x5597b2e88f60 2024-04-25T16:02:54.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.382+0000 7fb279ffb640 10 client.15936 handle_command_reply: tid=0 2024-04-25T16:02:54.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.382+0000 7fb29f5a0740 1 client.15936 shutdown 2024-04-25T16:02:54.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.382+0000 7fb29f5a0740 1 -- 172.21.15.31:0/3685709189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea14d0 msgr2=0x5597b326fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:54.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.383+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea14d0 0x5597b326fa90 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009340 tx=0x7fb28c004d00 comp rx=0 tx=0).stop 2024-04-25T16:02:54.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.383+0000 7fb29f5a0740 1 -- 172.21.15.31:0/3685709189 shutdown_connections 2024-04-25T16:02:54.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.383+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5597b2e88f60 0x5597b327fe60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.383+0000 7fb279ffb640 0 client.15936 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:02:54.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.383+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 0x5597b326c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.383+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2ea96b0 0x5597b326ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.383+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/3685709189 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea14d0 0x5597b326fa90 secure :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009340 tx=0x7fb28c004d00 comp rx=0 tx=0).stop 2024-04-25T16:02:54.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.383+0000 7fb29f5a0740 1 -- 172.21.15.31:0/3685709189 >> 172.21.15.31:0/3685709189 conn(0x5597b2ebe4e0 msgr2=0x5597b3267040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:54.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.394+0000 7fb29f5a0740 1 -- 172.21.15.31:0/3685709189 shutdown_connections 2024-04-25T16:02:54.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.394+0000 7fb29f5a0740 1 -- 172.21.15.31:0/3685709189 wait complete. 2024-04-25T16:02:54.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.397+0000 7fb29f5a0740 20 client.15936 trim_cache size 0 max 16384 2024-04-25T16:02:54.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.484+0000 7fb29f5a0740 1 Processor -- start 2024-04-25T16:02:54.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.485+0000 7fb29f5a0740 1 -- start start 2024-04-25T16:02:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.486+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b3279fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.486+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea96b0 0x5597b32864f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.487+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 0x5597b3279130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.487+0000 7fb2963af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b3279fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.487+0000 7fb2963af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b3279fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:46886/0 (socket says 172.21.15.31:46886) 2024-04-25T16:02:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.487+0000 7fb2963af640 1 -- 172.21.15.31:0/1715422869 learned_addr learned my addr 172.21.15.31:0/1715422869 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.487+0000 7fb29f5a0740 1 -- 172.21.15.31:0/1715422869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb27400e500 con 0x5597b2ea96b0 2024-04-25T16:02:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.487+0000 7fb29f5a0740 1 -- 172.21.15.31:0/1715422869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb274010040 con 0x5597b2eca390 2024-04-25T16:02:54.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.487+0000 7fb29f5a0740 1 -- 172.21.15.31:0/1715422869 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb27400ebd0 con 0x5597b2e88f60 2024-04-25T16:02:54.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.490+0000 7fb296bb0640 1 --2- 172.21.15.31:0/1715422869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 0x5597b3279130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.490+0000 7fb295bae640 1 --2- 172.21.15.31:0/1715422869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea96b0 0x5597b32864f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.491+0000 7fb295bae640 1 -- 172.21.15.31:0/1715422869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 msgr2=0x5597b3279fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.491+0000 7fb295bae640 1 --2- 172.21.15.31:0/1715422869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b3279fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.491+0000 7fb295bae640 1 -- 172.21.15.31:0/1715422869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 msgr2=0x5597b3279130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.491+0000 7fb295bae640 1 --2- 172.21.15.31:0/1715422869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 0x5597b3279130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.491+0000 7fb295bae640 1 -- 172.21.15.31:0/1715422869 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597b3278a80 con 0x5597b2ea96b0 2024-04-25T16:02:54.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.492+0000 7fb295bae640 1 --2- 172.21.15.31:0/1715422869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea96b0 0x5597b32864f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x5597b3280460 tx=0x7fb2800094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9294b6367872481 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:54.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.501+0000 7fb27bfff640 1 -- 172.21.15.31:0/1715422869 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280005220 con 0x5597b2ea96b0 2024-04-25T16:02:54.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.501+0000 7fb27bfff640 1 -- 172.21.15.31:0/1715422869 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb280004c50 con 0x5597b2ea96b0 2024-04-25T16:02:54.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.501+0000 7fb27bfff640 1 -- 172.21.15.31:0/1715422869 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28001f760 con 0x5597b2ea96b0 2024-04-25T16:02:54.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.506+0000 7fb29f5a0740 1 -- 172.21.15.31:0/1715422869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea96b0 msgr2=0x5597b32864f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:54.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.506+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/1715422869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea96b0 0x5597b32864f0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x5597b3280460 tx=0x7fb2800094e0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.511+0000 7fb29f5a0740 1 -- 172.21.15.31:0/1715422869 shutdown_connections 2024-04-25T16:02:54.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.511+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/1715422869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2eca390 0x5597b3279130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.511+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/1715422869 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2ea96b0 0x5597b32864f0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.511+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/1715422869 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b3279fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.511+0000 7fb29f5a0740 1 -- 172.21.15.31:0/1715422869 >> 172.21.15.31:0/1715422869 conn(0x5597b2ebe4e0 msgr2=0x5597b32756c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.512+0000 7fb29f5a0740 1 -- 172.21.15.31:0/1715422869 shutdown_connections 2024-04-25T16:02:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.512+0000 7fb29f5a0740 1 -- 172.21.15.31:0/1715422869 wait complete. 2024-04-25T16:02:54.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.513+0000 7fb29f5a0740 1 -- start start 2024-04-25T16:02:54.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.513+0000 7fb29f5a0740 1 Processor -- start 2024-04-25T16:02:54.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.514+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b32868f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.514+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 0x5597b3283470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.515+0000 7fb29f5a0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 0x5597b3283ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.515+0000 7fb29f5a0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb27400f970 con 0x5597b2eca390 2024-04-25T16:02:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.515+0000 7fb29f5a0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb27400f2a0 con 0x5597b2ea96b0 2024-04-25T16:02:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.515+0000 7fb2963af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b32868f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.515+0000 7fb2963af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b32868f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:46902/0 (socket says 172.21.15.31:46902) 2024-04-25T16:02:54.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.515+0000 7fb2963af640 1 -- 172.21.15.31:0/638647566 learned_addr learned my addr 172.21.15.31:0/638647566 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.515+0000 7fb29f5a0740 1 -- 172.21.15.31:0/638647566 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb27400c4e0 con 0x5597b2e88f60 2024-04-25T16:02:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.516+0000 7fb29f5a0740 10 client.0 _ll_register_callbacks cb 0x7fb280009380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:54.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.516+0000 7fb295bae640 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 0x5597b3283470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.517+0000 7fb296bb0640 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 0x5597b3283ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.517+0000 7fb2963af640 1 -- 172.21.15.31:0/638647566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 msgr2=0x5597b3283470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:54.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.517+0000 7fb2963af640 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 0x5597b3283470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.517+0000 7fb2963af640 1 -- 172.21.15.31:0/638647566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 msgr2=0x5597b3283ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:54.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.517+0000 7fb2963af640 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 0x5597b3283ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.517+0000 7fb2963af640 1 -- 172.21.15.31:0/638647566 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597b32fbab0 con 0x5597b2e88f60 2024-04-25T16:02:54.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.521+0000 7fb2963af640 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b32868f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb28c051820 tx=0x7fb28c004fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab80ec858c4f6c33 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:54.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.521+0000 7fb279ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:02:54.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.525+0000 7fb279ffb640 1 -- 172.21.15.31:0/638647566 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c058040 con 0x5597b2e88f60 2024-04-25T16:02:54.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.526+0000 7fb279ffb640 1 -- 172.21.15.31:0/638647566 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb28c005450 con 0x5597b2e88f60 2024-04-25T16:02:54.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.526+0000 7fb29f5a0740 1 -- 172.21.15.31:0/638647566 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb280004c50 con 0x5597b2e88f60 2024-04-25T16:02:54.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.529+0000 7fb279ffb640 1 -- 172.21.15.31:0/638647566 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fb28c037040 con 0x5597b2e88f60 2024-04-25T16:02:54.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.530+0000 7fb29f5a0740 10 client.25603 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:54.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.530+0000 7fb29f5a0740 1 -- 172.21.15.31:0/638647566 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5597b32fcd90 con 0x5597b2e88f60 2024-04-25T16:02:54.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.531+0000 7fb279ffb640 1 -- 172.21.15.31:0/638647566 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb280004c50 con 0x5597b2e88f60 2024-04-25T16:02:54.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.532+0000 7fb279ffb640 1 -- 172.21.15.31:0/638647566 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fb28c047040 con 0x5597b2e88f60 2024-04-25T16:02:54.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.532+0000 7fb29f5a0740 10 client.25603 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:54.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.532+0000 7fb29f5a0740 10 client.25603 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:02:54.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.532+0000 7fb29f5a0740 20 client.25603 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:54.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.532+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5597b2e5e3a0 0x5597b32ff7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:54.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.532+0000 7fb29f5a0740 4 client.25603 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:54.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.532+0000 7fb29f5a0740 1 -- 172.21.15.31:0/638647566 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5597b31d8800 con 0x5597b2e5e3a0 2024-04-25T16:02:54.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.533+0000 7fb295bae640 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5597b2e5e3a0 0x5597b32ff7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:54.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.534+0000 7fb295bae640 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5597b2e5e3a0 0x5597b32ff7b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f092c7bea6a80178 server_cookie=1a4e5729b1400cc7 in_seq=0 out_seq=0 2024-04-25T16:02:54.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.534+0000 7fb279ffb640 10 client.25603 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:02:54.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.535+0000 7fb279ffb640 1 -- 172.21.15.31:0/638647566 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5597b31d8800 con 0x5597b2e5e3a0 2024-04-25T16:02:54.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.535+0000 7fb279ffb640 10 client.25603 handle_command_reply: tid=0 2024-04-25T16:02:54.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.535+0000 7fb29f5a0740 1 client.25603 shutdown 2024-04-25T16:02:54.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.535+0000 7fb29f5a0740 1 -- 172.21.15.31:0/638647566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 msgr2=0x5597b32868f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:54.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.535+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b32868f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb28c051820 tx=0x7fb28c004fa0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.535+0000 7fb29f5a0740 1 -- 172.21.15.31:0/638647566 shutdown_connections 2024-04-25T16:02:54.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.536+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5597b2e5e3a0 0x5597b32ff7b0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.536+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5597b2eca390 0x5597b3283ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.536+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5597b2ea96b0 0x5597b3283470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.536+0000 7fb29f5a0740 1 --2- 172.21.15.31:0/638647566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5597b2e88f60 0x5597b32868f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.536+0000 7fb29f5a0740 1 -- 172.21.15.31:0/638647566 >> 172.21.15.31:0/638647566 conn(0x5597b2ebe4e0 msgr2=0x5597b3279130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:54.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.536+0000 7fb279ffb640 0 client.25603 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:02:54.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.536+0000 7fb29f5a0740 1 -- 172.21.15.31:0/638647566 shutdown_connections 2024-04-25T16:02:54.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.536+0000 7fb29f5a0740 1 -- 172.21.15.31:0/638647566 wait complete. 2024-04-25T16:02:54.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.536+0000 7fb29f5a0740 20 client.25603 trim_cache size 0 max 16384 2024-04-25T16:02:54.541 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:54.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.537+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb27c0afb70 msgr2=0x7fb27c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:54.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.538+0000 7fb298613640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb27c0afb70 0x7fb27c0b2080 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb28c009830 tx=0x7fb28c030340 comp rx=0 tx=0).stop 2024-04-25T16:02:54.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.538+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb290108930 msgr2=0x7fb2901a04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:54.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.538+0000 7fb298613640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb290108930 0x7fb2901a04a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb28800ed70 tx=0x7fb28800c6e0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.538+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 shutdown_connections 2024-04-25T16:02:54.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.538+0000 7fb298613640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb27c0afb70 0x7fb27c0b2080 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.538+0000 7fb298613640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb290108930 0x7fb2901a04a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.538+0000 7fb298613640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb290103c30 0x7fb290199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.538+0000 7fb298613640 1 --2- 172.21.15.31:0/117904960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2901029f0 0x7fb290198fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:54.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.538+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 >> 172.21.15.31:0/117904960 conn(0x7fb2900fdc00 msgr2=0x7fb290104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:54.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.540+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 shutdown_connections 2024-04-25T16:02:54.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:54.540+0000 7fb298613640 1 -- 172.21.15.31:0/117904960 wait complete. 2024-04-25T16:02:54.602 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:02:54.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:54 smithi063 ceph-mon[32413]: pgmap v399: 97 pgs: 97 active+clean; 5.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 449 MiB/s wr, 357 op/s 2024-04-25T16:02:54.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:54 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2130343783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:02:54.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:54 smithi186 ceph-mon[33258]: pgmap v399: 97 pgs: 97 active+clean; 5.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 449 MiB/s wr, 357 op/s 2024-04-25T16:02:54.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:54 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2130343783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:02:54.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:54 smithi031 ceph-mon[29140]: pgmap v399: 97 pgs: 97 active+clean; 5.3 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 449 MiB/s wr, 357 op/s 2024-04-25T16:02:54.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:54 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2130343783' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:02:55.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.060+0000 7f4700a52640 1 -- 172.21.15.31:0/3046074152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 msgr2=0x7f46fc10b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.060+0000 7f4700a52640 1 --2- 172.21.15.31:0/3046074152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 0x7f46fc10b590 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f46ec009900 tx=0x7f46ec02a440 comp rx=0 tx=0).stop 2024-04-25T16:02:55.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.061+0000 7f4700a52640 1 -- 172.21.15.31:0/3046074152 shutdown_connections 2024-04-25T16:02:55.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.061+0000 7f4700a52640 1 --2- 172.21.15.31:0/3046074152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc10fe80 0x7f46fc1122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.061+0000 7f4700a52640 1 --2- 172.21.15.31:0/3046074152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 0x7f46fc10b590 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.061+0000 7f4700a52640 1 --2- 172.21.15.31:0/3046074152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc109f40 0x7f46fc10ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.061+0000 7f4700a52640 1 -- 172.21.15.31:0/3046074152 >> 172.21.15.31:0/3046074152 conn(0x7f46fc078b50 msgr2=0x7f46fc1079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.061+0000 7f4700a52640 1 -- 172.21.15.31:0/3046074152 shutdown_connections 2024-04-25T16:02:55.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.061+0000 7f4700a52640 1 -- 172.21.15.31:0/3046074152 wait complete. 2024-04-25T16:02:55.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.063+0000 7f4700a52640 1 Processor -- start 2024-04-25T16:02:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.065+0000 7f4700a52640 1 -- start start 2024-04-25T16:02:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.065+0000 7f4700a52640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc109f40 0x7f46fc0809a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.065+0000 7f4700a52640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 0x7f46fc080ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.065+0000 7f4700a52640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc10fe80 0x7f46fc081e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.065+0000 7f4700a52640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc115bb0 con 0x7f46fc10b180 2024-04-25T16:02:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.065+0000 7f4700a52640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc115a30 con 0x7f46fc10fe80 2024-04-25T16:02:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.065+0000 7f4700a52640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc115d30 con 0x7f46fc109f40 2024-04-25T16:02:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.066+0000 7f46fa59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 0x7f46fc080ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.066+0000 7f46fa59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 0x7f46fc080ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53544/0 (socket says 172.21.15.31:53544) 2024-04-25T16:02:55.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.066+0000 7f46fa59c640 1 -- 172.21.15.31:0/1686003705 learned_addr learned my addr 172.21.15.31:0/1686003705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.066+0000 7f46fb59e640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc10fe80 0x7f46fc081e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.067+0000 7f46fa59c640 1 -- 172.21.15.31:0/1686003705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc109f40 msgr2=0x7f46fc0809a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:02:55.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.067+0000 7f46fa59c640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc109f40 0x7f46fc0809a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.067+0000 7f46fa59c640 1 -- 172.21.15.31:0/1686003705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc10fe80 msgr2=0x7f46fc081e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.067+0000 7f46fa59c640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc10fe80 0x7f46fc081e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.067+0000 7f46fa59c640 1 -- 172.21.15.31:0/1686003705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46fc0824b0 con 0x7f46fc10b180 2024-04-25T16:02:55.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.067+0000 7f46fa59c640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 0x7f46fc080ee0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f46ec031440 tx=0x7f46ec003dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.068+0000 7f46e3fff640 1 -- 172.21.15.31:0/1686003705 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec03f070 con 0x7f46fc10b180 2024-04-25T16:02:55.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.068+0000 7f46e3fff640 1 -- 172.21.15.31:0/1686003705 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46ec03b460 con 0x7f46fc10b180 2024-04-25T16:02:55.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.068+0000 7f46e3fff640 1 -- 172.21.15.31:0/1686003705 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec035030 con 0x7f46fc10b180 2024-04-25T16:02:55.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.069+0000 7f4700a52640 1 -- 172.21.15.31:0/1686003705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46fc082740 con 0x7f46fc10b180 2024-04-25T16:02:55.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.069+0000 7f4700a52640 1 -- 172.21.15.31:0/1686003705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46fc082c60 con 0x7f46fc10b180 2024-04-25T16:02:55.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.073+0000 7f47081e0740 1 Processor -- start 2024-04-25T16:02:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.073+0000 7f47081e0740 1 -- start start 2024-04-25T16:02:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.073+0000 7f47081e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bf8390 0x557ac8f9aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.073+0000 7f47081e0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 0x557ac8f9af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.074+0000 7f47081e0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bd76b0 0x557ac8f9bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.074+0000 7f47081e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557ac8efed80 con 0x557ac8bf8390 2024-04-25T16:02:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.074+0000 7f47081e0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557ac8eb8a20 con 0x557ac8bd76b0 2024-04-25T16:02:55.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.074+0000 7f47081e0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ac8efad40 con 0x557ac8bcf4d0 2024-04-25T16:02:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.075+0000 7f46fad9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 0x557ac8f9af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.075+0000 7f46fad9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 0x557ac8f9af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:46922/0 (socket says 172.21.15.31:46922) 2024-04-25T16:02:55.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.075+0000 7f46fad9d640 1 -- 172.21.15.31:0/929243134 learned_addr learned my addr 172.21.15.31:0/929243134 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.076+0000 7f46fb59e640 1 --2- 172.21.15.31:0/929243134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bf8390 0x557ac8f9aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.078+0000 7f46fb59e640 1 -- 172.21.15.31:0/929243134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 msgr2=0x557ac8f9af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.078+0000 7f46fb59e640 1 --2- 172.21.15.31:0/929243134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 0x557ac8f9af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.078+0000 7f46fb59e640 1 -- 172.21.15.31:0/929243134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bd76b0 msgr2=0x557ac8f9bc10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.078+0000 7f46fb59e640 1 --2- 172.21.15.31:0/929243134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bd76b0 0x557ac8f9bc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.078+0000 7f46fb59e640 1 -- 172.21.15.31:0/929243134 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ac8ec3c30 con 0x557ac8bf8390 2024-04-25T16:02:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.078+0000 7f46e3fff640 1 -- 172.21.15.31:0/1686003705 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f46ec042070 con 0x7f46fc10b180 2024-04-25T16:02:55.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.078+0000 7f46fb59e640 1 --2- 172.21.15.31:0/929243134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bf8390 0x557ac8f9aa00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f46f000b6d0 tx=0x7f46f000bb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76845db5fe753dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.080+0000 7f46e3fff640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f46e40afbb0 0x7f46e40b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.080+0000 7f46e3fff640 1 -- 172.21.15.31:0/1686003705 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f46ec0ff6a0 con 0x7f46fc10b180 2024-04-25T16:02:55.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.081+0000 7f46e17fa640 1 -- 172.21.15.31:0/929243134 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f000d740 con 0x557ac8bf8390 2024-04-25T16:02:55.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.081+0000 7f46fad9d640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f46e40afbb0 0x7f46e40b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.081+0000 7f46e17fa640 1 -- 172.21.15.31:0/929243134 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46f0019070 con 0x557ac8bf8390 2024-04-25T16:02:55.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.081+0000 7f46e17fa640 1 -- 172.21.15.31:0/929243134 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f0014cf0 con 0x557ac8bf8390 2024-04-25T16:02:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f47081e0740 1 -- 172.21.15.31:0/929243134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bf8390 msgr2=0x557ac8f9aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f47081e0740 1 --2- 172.21.15.31:0/929243134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bf8390 0x557ac8f9aa00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f46f000b6d0 tx=0x7f46f000bb90 comp rx=0 tx=0).stop 2024-04-25T16:02:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f46fad9d640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f46e40afbb0 0x7f46e40b20c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x557ac8f9a720 tx=0x7f46f400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f47081e0740 1 -- 172.21.15.31:0/929243134 shutdown_connections 2024-04-25T16:02:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f47081e0740 1 --2- 172.21.15.31:0/929243134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bd76b0 0x557ac8f9bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f47081e0740 1 --2- 172.21.15.31:0/929243134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 0x557ac8f9af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f47081e0740 1 --2- 172.21.15.31:0/929243134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bf8390 0x557ac8f9aa00 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f47081e0740 1 -- 172.21.15.31:0/929243134 >> 172.21.15.31:0/929243134 conn(0x557ac8bec4e0 msgr2=0x557ac8f96210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f47081e0740 1 -- 172.21.15.31:0/929243134 shutdown_connections 2024-04-25T16:02:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.085+0000 7f47081e0740 1 -- 172.21.15.31:0/929243134 wait complete. 2024-04-25T16:02:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.086+0000 7f47081e0740 1 -- start start 2024-04-25T16:02:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.086+0000 7f47081e0740 1 Processor -- start 2024-04-25T16:02:55.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.087+0000 7f47081e0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.087+0000 7f47081e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac8fa4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.087+0000 7f47081e0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bf8390 0x557ac8fabda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.088+0000 7f46fa59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac8fa4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.088+0000 7f46fa59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac8fa4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53564/0 (socket says 172.21.15.31:53564) 2024-04-25T16:02:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.088+0000 7f46fa59c640 1 -- 172.21.15.31:0/2378422356 learned_addr learned my addr 172.21.15.31:0/2378422356 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.088+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557ac8f9c720 con 0x557ac8bd76b0 2024-04-25T16:02:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.088+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557ac8e85dc0 con 0x557ac8bf8390 2024-04-25T16:02:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.088+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ac8f9c8a0 con 0x557ac8bcf4d0 2024-04-25T16:02:55.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.089+0000 7f47081e0740 10 client.0 _ll_register_callbacks cb 0x557ac8ea1600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.089+0000 7f46fa59c640 1 -- 172.21.15.31:0/2378422356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 msgr2=0x557ac8fa4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.089+0000 7f46fa59c640 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.089+0000 7f46fa59c640 1 -- 172.21.15.31:0/2378422356 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bf8390 msgr2=0x557ac8fabda0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.089+0000 7f46fa59c640 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bf8390 0x557ac8fabda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.089+0000 7f46fa59c640 1 -- 172.21.15.31:0/2378422356 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ac8c103f0 con 0x557ac8bd76b0 2024-04-25T16:02:55.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.090+0000 7f46fa59c640 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac8fa4e40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0c3b20 tx=0x7f46ec0c7360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a802b135f1923a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.090+0000 7f46deff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.091+0000 7f46deff5640 1 -- 172.21.15.31:0/2378422356 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec0c8e60 con 0x557ac8bd76b0 2024-04-25T16:02:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.091+0000 7f46deff5640 1 -- 172.21.15.31:0/2378422356 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f46ec0c4090 con 0x557ac8bd76b0 2024-04-25T16:02:55.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.091+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46f0019070 con 0x557ac8bd76b0 2024-04-25T16:02:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.091+0000 7f46deff5640 1 -- 172.21.15.31:0/2378422356 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f46ec0cbee0 con 0x557ac8bd76b0 2024-04-25T16:02:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.091+0000 7f47081e0740 10 client.15954 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:55.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.091+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557ac8cda7d0 con 0x557ac8bd76b0 2024-04-25T16:02:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.092+0000 7f46deff5640 1 -- 172.21.15.31:0/2378422356 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46f0019070 con 0x557ac8bd76b0 2024-04-25T16:02:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.092+0000 7f46deff5640 1 -- 172.21.15.31:0/2378422356 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f46ec0d0be0 con 0x557ac8bd76b0 2024-04-25T16:02:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.093+0000 7f47081e0740 10 client.15954 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.093+0000 7f47081e0740 10 client.15954 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:02:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.093+0000 7f47081e0740 20 client.15954 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.093+0000 7f47081e0740 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x557ac8bb6f60 0x557ac8fb2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.093+0000 7f47081e0740 4 client.15954 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:55.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.093+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ac8f0c990 con 0x557ac8bb6f60 2024-04-25T16:02:55.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.095+0000 7f46fad9d640 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x557ac8bb6f60 0x557ac8fb2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.105+0000 7f46deff5640 10 client.15954 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:02:55.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.105+0000 7f46fad9d640 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x557ac8bb6f60 0x557ac8fb2590 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f11735a1112aacf9 server_cookie=dc6c71243541d4a4 in_seq=0 out_seq=0 2024-04-25T16:02:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f46deff5640 1 -- 172.21.15.31:0/2378422356 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557ac8f0c990 con 0x557ac8bb6f60 2024-04-25T16:02:55.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f46deff5640 10 client.15954 handle_command_reply: tid=0 2024-04-25T16:02:55.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 client.15954 shutdown 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 msgr2=0x557ac8fa4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac8fa4e40 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0c3b20 tx=0x7f46ec0c7360 comp rx=0 tx=0).stop 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 shutdown_connections 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bf8390 0x557ac8fabda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac8fa4e40 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x557ac8bb6f60 0x557ac8fb2590 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f46deff5640 0 client.15954 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 --2- 172.21.15.31:0/2378422356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 >> 172.21.15.31:0/2378422356 conn(0x557ac8bec4e0 msgr2=0x557ac8f953f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 shutdown_connections 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 1 -- 172.21.15.31:0/2378422356 wait complete. 2024-04-25T16:02:55.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.108+0000 7f47081e0740 20 client.15954 trim_cache size 0 max 16384 2024-04-25T16:02:55.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.133+0000 7f47081e0740 1 Processor -- start 2024-04-25T16:02:55.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f47081e0740 1 -- start start 2024-04-25T16:02:55.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f47081e0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bb6f60 0x557ac8fa2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f47081e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f47081e0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bd76b0 0x557ac8fb6d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f46fa59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f46fa59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53568/0 (socket says 172.21.15.31:53568) 2024-04-25T16:02:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f46fa59c640 1 -- 172.21.15.31:0/2860475644 learned_addr learned my addr 172.21.15.31:0/2860475644 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f47081e0740 1 -- 172.21.15.31:0/2860475644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc1ba0a0 con 0x557ac8bcf4d0 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f47081e0740 1 -- 172.21.15.31:0/2860475644 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc1bbbe0 con 0x557ac8bb6f60 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.134+0000 7f47081e0740 1 -- 172.21.15.31:0/2860475644 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc1ba770 con 0x557ac8bd76b0 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.135+0000 7f46fad9d640 1 --2- 172.21.15.31:0/2860475644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bb6f60 0x557ac8fa2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.135+0000 7f46fa59c640 1 -- 172.21.15.31:0/2860475644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bd76b0 msgr2=0x557ac8fb6d80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.135+0000 7f46fa59c640 1 --2- 172.21.15.31:0/2860475644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bd76b0 0x557ac8fb6d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.135+0000 7f46fa59c640 1 -- 172.21.15.31:0/2860475644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bb6f60 msgr2=0x557ac8fa2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.135+0000 7f46fa59c640 1 --2- 172.21.15.31:0/2860475644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bb6f60 0x557ac8fa2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.135+0000 7f46fa59c640 1 -- 172.21.15.31:0/2860475644 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ac8fa2a20 con 0x557ac8bcf4d0 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.135+0000 7f46fa59c640 1 --2- 172.21.15.31:0/2860475644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa33e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0c7ac0 tx=0x7f46ec0c73d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a93461e4b8b727 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.136+0000 7f46e0ff9640 1 -- 172.21.15.31:0/2860475644 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec0d5070 con 0x557ac8bcf4d0 2024-04-25T16:02:55.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.137+0000 7f46e0ff9640 1 -- 172.21.15.31:0/2860475644 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46ec0d0640 con 0x557ac8bcf4d0 2024-04-25T16:02:55.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.137+0000 7f46e0ff9640 1 -- 172.21.15.31:0/2860475644 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46ec0c8e60 con 0x557ac8bcf4d0 2024-04-25T16:02:55.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.138+0000 7f47081e0740 1 -- 172.21.15.31:0/2860475644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bcf4d0 msgr2=0x557ac8fa33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.138+0000 7f47081e0740 1 --2- 172.21.15.31:0/2860475644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa33e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0c7ac0 tx=0x7f46ec0c73d0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.140+0000 7f47081e0740 1 -- 172.21.15.31:0/2860475644 shutdown_connections 2024-04-25T16:02:55.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.140+0000 7f47081e0740 1 --2- 172.21.15.31:0/2860475644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bd76b0 0x557ac8fb6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.140+0000 7f47081e0740 1 --2- 172.21.15.31:0/2860475644 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa33e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.140+0000 7f47081e0740 1 --2- 172.21.15.31:0/2860475644 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bb6f60 0x557ac8fa2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.140+0000 7f47081e0740 1 -- 172.21.15.31:0/2860475644 >> 172.21.15.31:0/2860475644 conn(0x557ac8bec4e0 msgr2=0x557ac8f9f660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.140+0000 7f47081e0740 1 -- 172.21.15.31:0/2860475644 shutdown_connections 2024-04-25T16:02:55.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.140+0000 7f47081e0740 1 -- 172.21.15.31:0/2860475644 wait complete. 2024-04-25T16:02:55.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.141+0000 7f47081e0740 1 -- start start 2024-04-25T16:02:55.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.141+0000 7f47081e0740 1 Processor -- start 2024-04-25T16:02:55.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f47081e0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bb6f60 0x557ac8fa8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f47081e0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa8860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f47081e0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac9018930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f47081e0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc1bb510 con 0x557ac8bd76b0 2024-04-25T16:02:55.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f47081e0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc1bae40 con 0x557ac8bcf4d0 2024-04-25T16:02:55.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f47081e0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f46fc1b8080 con 0x557ac8bb6f60 2024-04-25T16:02:55.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f47081e0740 10 client.0 _ll_register_callbacks cb 0x7f46ec004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:55.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f46fb59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac9018930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f46fb59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac9018930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53574/0 (socket says 172.21.15.31:53574) 2024-04-25T16:02:55.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.142+0000 7f46fb59e640 1 -- 172.21.15.31:0/604699896 learned_addr learned my addr 172.21.15.31:0/604699896 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.143+0000 7f46fb59e640 1 -- 172.21.15.31:0/604699896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bb6f60 msgr2=0x557ac8fa8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.143+0000 7f46fb59e640 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bb6f60 0x557ac8fa8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.143+0000 7f46fb59e640 1 -- 172.21.15.31:0/604699896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bcf4d0 msgr2=0x557ac8fa8860 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:55.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.143+0000 7f46fb59e640 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa8860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.143+0000 7f46fb59e640 1 -- 172.21.15.31:0/604699896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ac9018e70 con 0x557ac8bd76b0 2024-04-25T16:02:55.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.144+0000 7f46fb59e640 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac9018930 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f46f0000c00 tx=0x7f46f00149d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6a40d261e3c747 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.144+0000 7f46deff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:55.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.144+0000 7f46deff5640 1 -- 172.21.15.31:0/604699896 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46f000d740 con 0x557ac8bd76b0 2024-04-25T16:02:55.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.144+0000 7f46deff5640 1 -- 172.21.15.31:0/604699896 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f46f000d8e0 con 0x557ac8bd76b0 2024-04-25T16:02:55.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.145+0000 7f47081e0740 1 -- 172.21.15.31:0/604699896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f46ec0d0640 con 0x557ac8bd76b0 2024-04-25T16:02:55.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.145+0000 7f46deff5640 1 -- 172.21.15.31:0/604699896 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f46f000db20 con 0x557ac8bd76b0 2024-04-25T16:02:55.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.145+0000 7f47081e0740 10 client.15960 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:55.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.145+0000 7f47081e0740 1 -- 172.21.15.31:0/604699896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557ac901a090 con 0x557ac8bd76b0 2024-04-25T16:02:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.146+0000 7f46deff5640 1 -- 172.21.15.31:0/604699896 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46ec0d0640 con 0x557ac8bd76b0 2024-04-25T16:02:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.146+0000 7f46deff5640 1 -- 172.21.15.31:0/604699896 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f46f0013070 con 0x557ac8bd76b0 2024-04-25T16:02:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.147+0000 7f47081e0740 10 client.15960 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.147+0000 7f47081e0740 10 client.15960 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:02:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.147+0000 7f47081e0740 20 client.15960 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.147+0000 7f47081e0740 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x557ac8bf8390 0x557ac901c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.147+0000 7f47081e0740 4 client.15960 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.147+0000 7f47081e0740 1 -- 172.21.15.31:0/604699896 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ac8f06800 con 0x557ac8bf8390 2024-04-25T16:02:55.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.147+0000 7f46fad9d640 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x557ac8bf8390 0x557ac901c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.149+0000 7f46fad9d640 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x557ac8bf8390 0x557ac901c7c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a82affae9d49bfd server_cookie=8f2befba5e6219d0 in_seq=0 out_seq=0 2024-04-25T16:02:55.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.149+0000 7f46deff5640 10 client.15960 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:02:55.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.149+0000 7f46deff5640 1 -- 172.21.15.31:0/604699896 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ac8f06800 con 0x557ac8bf8390 2024-04-25T16:02:55.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.149+0000 7f46deff5640 10 client.15960 handle_command_reply: tid=0 2024-04-25T16:02:55.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.151+0000 7f47081e0740 1 client.15960 shutdown 2024-04-25T16:02:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.151+0000 7f47081e0740 1 -- 172.21.15.31:0/604699896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 msgr2=0x557ac9018930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.151+0000 7f47081e0740 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac9018930 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f46f0000c00 tx=0x7f46f00149d0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.151+0000 7f47081e0740 1 -- 172.21.15.31:0/604699896 shutdown_connections 2024-04-25T16:02:55.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.151+0000 7f47081e0740 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ac8bd76b0 0x557ac9018930 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.152+0000 7f47081e0740 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ac8bcf4d0 0x557ac8fa8860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.152+0000 7f47081e0740 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x557ac8bf8390 0x557ac901c7c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.152+0000 7f46deff5640 0 client.15960 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:02:55.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.152+0000 7f47081e0740 1 --2- 172.21.15.31:0/604699896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ac8bb6f60 0x557ac8fa8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.152+0000 7f47081e0740 1 -- 172.21.15.31:0/604699896 >> 172.21.15.31:0/604699896 conn(0x557ac8bec4e0 msgr2=0x557ac8fa2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.155+0000 7f47081e0740 1 -- 172.21.15.31:0/604699896 shutdown_connections 2024-04-25T16:02:55.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.156+0000 7f47081e0740 1 -- 172.21.15.31:0/604699896 wait complete. 2024-04-25T16:02:55.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.156+0000 7f47081e0740 20 client.15960 trim_cache size 0 max 16384 2024-04-25T16:02:55.158 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:55.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.159+0000 7f4700a52640 1 -- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f46e40afbb0 msgr2=0x7f46e40b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.159+0000 7f4700a52640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f46e40afbb0 0x7f46e40b20c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x557ac8f9a720 tx=0x7f46f400b040 comp rx=0 tx=0).stop 2024-04-25T16:02:55.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.159+0000 7f4700a52640 1 -- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 msgr2=0x7f46fc080ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.159+0000 7f4700a52640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 0x7f46fc080ee0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f46ec031440 tx=0x7f46ec003dc0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.161+0000 7f4700a52640 1 -- 172.21.15.31:0/1686003705 shutdown_connections 2024-04-25T16:02:55.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.161+0000 7f4700a52640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f46e40afbb0 0x7f46e40b20c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.162+0000 7f4700a52640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46fc10fe80 0x7f46fc081e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.162+0000 7f4700a52640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f46fc10b180 0x7f46fc080ee0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.162+0000 7f4700a52640 1 --2- 172.21.15.31:0/1686003705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc109f40 0x7f46fc0809a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.162+0000 7f4700a52640 1 -- 172.21.15.31:0/1686003705 >> 172.21.15.31:0/1686003705 conn(0x7f46fc078b50 msgr2=0x7f46fc10be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.163+0000 7f4700a52640 1 -- 172.21.15.31:0/1686003705 shutdown_connections 2024-04-25T16:02:55.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.164+0000 7f4700a52640 1 -- 172.21.15.31:0/1686003705 wait complete. 2024-04-25T16:02:55.211 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:02:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.716+0000 7fa4632f9640 1 -- 172.21.15.31:0/4042094267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c10c010 msgr2=0x7fa45c10c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.716+0000 7fa4632f9640 1 --2- 172.21.15.31:0/4042094267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c10c010 0x7fa45c10c420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa458009900 tx=0x7fa45802a440 comp rx=0 tx=0).stop 2024-04-25T16:02:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.717+0000 7fa4632f9640 1 -- 172.21.15.31:0/4042094267 shutdown_connections 2024-04-25T16:02:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.717+0000 7fa4632f9640 1 --2- 172.21.15.31:0/4042094267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c10c010 0x7fa45c10c420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.717+0000 7fa4632f9640 1 --2- 172.21.15.31:0/4042094267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa45c10b310 0x7fa45c10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.717+0000 7fa4632f9640 1 --2- 172.21.15.31:0/4042094267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa45c10a420 0x7fa45c10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.717+0000 7fa4632f9640 1 -- 172.21.15.31:0/4042094267 >> 172.21.15.31:0/4042094267 conn(0x7fa45c0704e0 msgr2=0x7fa45c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.735+0000 7fa4632f9640 1 -- 172.21.15.31:0/4042094267 shutdown_connections 2024-04-25T16:02:55.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.735+0000 7fa4632f9640 1 -- 172.21.15.31:0/4042094267 wait complete. 2024-04-25T16:02:55.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.738+0000 7fa4632f9640 1 Processor -- start 2024-04-25T16:02:55.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.738+0000 7fa4632f9640 1 -- start start 2024-04-25T16:02:55.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.746+0000 7fa4632f9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa45c10a420 0x7fa45c1a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.746+0000 7fa4632f9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c10b310 0x7fa45c1a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.746+0000 7fa4632f9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa45c10c010 0x7fa45c1ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.747+0000 7fa4632f9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c11a080 con 0x7fa45c10c010 2024-04-25T16:02:55.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.747+0000 7fa461896640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa45c10c010 0x7fa45c1ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.747+0000 7fa461896640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa45c10c010 0x7fa45c1ac0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53582/0 (socket says 172.21.15.31:53582) 2024-04-25T16:02:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.747+0000 7fa461896640 1 -- 172.21.15.31:0/3489986986 learned_addr learned my addr 172.21.15.31:0/3489986986 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.747+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c119f00 con 0x7fa45c10a420 2024-04-25T16:02:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.748+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c11a200 con 0x7fa45c10b310 2024-04-25T16:02:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.749+0000 7fa460894640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c10b310 0x7fa45c1a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.749+0000 7fa461095640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa45c10a420 0x7fa45c1a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.749+0000 7fa461896640 1 -- 172.21.15.31:0/3489986986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c10b310 msgr2=0x7fa45c1a5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.749+0000 7fa461896640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c10b310 0x7fa45c1a5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.749+0000 7fa461896640 1 -- 172.21.15.31:0/3489986986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa45c10a420 msgr2=0x7fa45c1a4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.749+0000 7fa461896640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa45c10a420 0x7fa45c1a4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.749+0000 7fa461896640 1 -- 172.21.15.31:0/3489986986 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45c1ac760 con 0x7fa45c10c010 2024-04-25T16:02:55.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.751+0000 7fa461896640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa45c10c010 0x7fa45c1ac0f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa458006010 tx=0x7fa458005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.753+0000 7fa4527fc640 1 -- 172.21.15.31:0/3489986986 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458035030 con 0x7fa45c10c010 2024-04-25T16:02:55.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.753+0000 7fa4527fc640 1 -- 172.21.15.31:0/3489986986 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa458045070 con 0x7fa45c10c010 2024-04-25T16:02:55.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.753+0000 7fa4527fc640 1 -- 172.21.15.31:0/3489986986 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa458031530 con 0x7fa45c10c010 2024-04-25T16:02:55.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.754+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45c10d750 con 0x7fa45c10c010 2024-04-25T16:02:55.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.754+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa45c10dbf0 con 0x7fa45c10c010 2024-04-25T16:02:55.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.769+0000 7fa46a1af740 1 Processor -- start 2024-04-25T16:02:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.772+0000 7fa46a1af740 1 -- start start 2024-04-25T16:02:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.772+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dd9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.772+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671dd9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.772+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671ddaa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.772+0000 7fa46a1af740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560671d3dd80 con 0x560671a37390 2024-04-25T16:02:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.772+0000 7fa46a1af740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560671cf7a20 con 0x560671a0e4d0 2024-04-25T16:02:55.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.772+0000 7fa46a1af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560671d39d40 con 0x560671a166b0 2024-04-25T16:02:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.774+0000 7fa4527fc640 1 -- 172.21.15.31:0/3489986986 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa4580035f0 con 0x7fa45c10c010 2024-04-25T16:02:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.774+0000 7fa460894640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dd9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.774+0000 7fa460894640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dd9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53592/0 (socket says 172.21.15.31:53592) 2024-04-25T16:02:55.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.774+0000 7fa460894640 1 -- 172.21.15.31:0/469984806 learned_addr learned my addr 172.21.15.31:0/469984806 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.775+0000 7fa460894640 1 --2- 172.21.15.31:0/469984806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671ddaa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.775+0000 7fa460894640 1 -- 172.21.15.31:0/469984806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 msgr2=0x560671ddaa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.775+0000 7fa460894640 1 --2- 172.21.15.31:0/469984806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671ddaa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.775+0000 7fa460894640 1 -- 172.21.15.31:0/469984806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 msgr2=0x560671dd9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.775+0000 7fa460894640 1 --2- 172.21.15.31:0/469984806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671dd9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.775+0000 7fa4527fc640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4480afd00 0x7fa4480b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.775+0000 7fa460894640 1 -- 172.21.15.31:0/469984806 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560671d02c30 con 0x560671a37390 2024-04-25T16:02:55.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.776+0000 7fa4527fc640 1 -- 172.21.15.31:0/3489986986 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa458107f30 con 0x7fa45c10c010 2024-04-25T16:02:55.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.776+0000 7fa460894640 1 --2- 172.21.15.31:0/469984806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dd9810 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa44c004650 tx=0x7fa44c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13058e8557c242f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.777+0000 7fa4477fe640 1 -- 172.21.15.31:0/469984806 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0130d0 con 0x560671a37390 2024-04-25T16:02:55.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.777+0000 7fa461095640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4480afd00 0x7fa4480b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.777+0000 7fa4477fe640 1 -- 172.21.15.31:0/469984806 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa44c00fc10 con 0x560671a37390 2024-04-25T16:02:55.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.778+0000 7fa4477fe640 1 -- 172.21.15.31:0/469984806 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c011910 con 0x560671a37390 2024-04-25T16:02:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.779+0000 7fa46a1af740 1 -- 172.21.15.31:0/469984806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 msgr2=0x560671dd9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.779+0000 7fa46a1af740 1 --2- 172.21.15.31:0/469984806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dd9810 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa44c004650 tx=0x7fa44c0097b0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.779+0000 7fa461095640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4480afd00 0x7fa4480b2210 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x560671dd9530 tx=0x7fa454009b00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.779+0000 7fa46a1af740 1 -- 172.21.15.31:0/469984806 shutdown_connections 2024-04-25T16:02:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.779+0000 7fa46a1af740 1 --2- 172.21.15.31:0/469984806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671ddaa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.779+0000 7fa46a1af740 1 --2- 172.21.15.31:0/469984806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671dd9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.779+0000 7fa46a1af740 1 --2- 172.21.15.31:0/469984806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dd9810 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.779+0000 7fa46a1af740 1 -- 172.21.15.31:0/469984806 >> 172.21.15.31:0/469984806 conn(0x560671a2b4e0 msgr2=0x560671dd5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.787+0000 7fa46a1af740 1 -- 172.21.15.31:0/469984806 shutdown_connections 2024-04-25T16:02:55.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.798+0000 7fa46a1af740 1 -- 172.21.15.31:0/469984806 wait complete. 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.798+0000 7fa46a1af740 1 -- start start 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.799+0000 7fa46a1af740 1 Processor -- start 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.800+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671df4640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.800+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671df4b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.800+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dfbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.800+0000 7fa46a1af740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560671ddb520 con 0x560671a37390 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.800+0000 7fa46a1af740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560671cc4dc0 con 0x560671a0e4d0 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.800+0000 7fa46a1af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560671ddb6a0 con 0x560671a166b0 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.800+0000 7fa46a1af740 10 client.0 _ll_register_callbacks cb 0x560671ce0600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.807+0000 7fa461095640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671df4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.807+0000 7fa461896640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dfbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.807+0000 7fa461896640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dfbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53604/0 (socket says 172.21.15.31:53604) 2024-04-25T16:02:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.807+0000 7fa461896640 1 -- 172.21.15.31:0/2070071773 learned_addr learned my addr 172.21.15.31:0/2070071773 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.808+0000 7fa460894640 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671df4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.809+0000 7fa461095640 1 -- 172.21.15.31:0/2070071773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 msgr2=0x560671df4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.809+0000 7fa461095640 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671df4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.809+0000 7fa461095640 1 -- 172.21.15.31:0/2070071773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 msgr2=0x560671dfbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.810+0000 7fa461095640 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dfbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.810+0000 7fa461095640 1 -- 172.21.15.31:0/2070071773 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560671a4f3f0 con 0x560671a0e4d0 2024-04-25T16:02:55.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.812+0000 7fa461095640 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671df4640 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa45c109c60 tx=0x7fa454009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c09085dc9e49dd40 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.817+0000 7fa444ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:02:55.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.817+0000 7fa444ff9640 1 -- 172.21.15.31:0/2070071773 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45400eca0 con 0x560671a0e4d0 2024-04-25T16:02:55.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.817+0000 7fa46a1af740 1 -- 172.21.15.31:0/2070071773 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa44c00fc10 con 0x560671a0e4d0 2024-04-25T16:02:55.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.818+0000 7fa444ff9640 1 -- 172.21.15.31:0/2070071773 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa45400ee40 con 0x560671a0e4d0 2024-04-25T16:02:55.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.820+0000 7fa444ff9640 1 -- 172.21.15.31:0/2070071773 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa45401e420 con 0x560671a0e4d0 2024-04-25T16:02:55.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.820+0000 7fa46a1af740 10 client.25655 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:55.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.820+0000 7fa46a1af740 1 -- 172.21.15.31:0/2070071773 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560671b197d0 con 0x560671a0e4d0 2024-04-25T16:02:55.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.823+0000 7fa444ff9640 1 -- 172.21.15.31:0/2070071773 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa44c00fc10 con 0x560671a0e4d0 2024-04-25T16:02:55.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.823+0000 7fa444ff9640 1 -- 172.21.15.31:0/2070071773 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa454026880 con 0x560671a0e4d0 2024-04-25T16:02:55.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.823+0000 7fa46a1af740 10 client.25655 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:55.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.823+0000 7fa46a1af740 10 client.25655 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:02:55.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.823+0000 7fa46a1af740 20 client.25655 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:55.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.823+0000 7fa46a1af740 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5606719f5f60 0x560671e02350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.824+0000 7fa46a1af740 4 client.25655 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:55.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.824+0000 7fa46a1af740 1 -- 172.21.15.31:0/2070071773 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560671d4b990 con 0x5606719f5f60 2024-04-25T16:02:55.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.826+0000 7fa460894640 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5606719f5f60 0x560671e02350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.831+0000 7fa460894640 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5606719f5f60 0x560671e02350 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3d8d95b23e885f89 server_cookie=be3b60cadfb4ac00 in_seq=0 out_seq=0 2024-04-25T16:02:55.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.831+0000 7fa444ff9640 10 client.25655 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:02:55.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.833+0000 7fa444ff9640 1 -- 172.21.15.31:0/2070071773 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560671d4b990 con 0x5606719f5f60 2024-04-25T16:02:55.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.833+0000 7fa444ff9640 10 client.25655 handle_command_reply: tid=0 2024-04-25T16:02:55.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.834+0000 7fa46a1af740 1 client.25655 shutdown 2024-04-25T16:02:55.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.842+0000 7fa46a1af740 1 -- 172.21.15.31:0/2070071773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 msgr2=0x560671df4640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.842+0000 7fa46a1af740 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671df4640 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa45c109c60 tx=0x7fa454009500 comp rx=0 tx=0).stop 2024-04-25T16:02:55.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.843+0000 7fa46a1af740 1 -- 172.21.15.31:0/2070071773 shutdown_connections 2024-04-25T16:02:55.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.843+0000 7fa46a1af740 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671dfbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.844+0000 7fa46a1af740 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671df4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.844+0000 7fa46a1af740 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5606719f5f60 0x560671e02350 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.844+0000 7fa46a1af740 1 --2- 172.21.15.31:0/2070071773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671df4640 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.844+0000 7fa46a1af740 1 -- 172.21.15.31:0/2070071773 >> 172.21.15.31:0/2070071773 conn(0x560671a2b4e0 msgr2=0x560671dd4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.844+0000 7fa444ff9640 0 client.25655 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:02:55.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.846+0000 7fa46a1af740 1 -- 172.21.15.31:0/2070071773 shutdown_connections 2024-04-25T16:02:55.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.847+0000 7fa46a1af740 1 -- 172.21.15.31:0/2070071773 wait complete. 2024-04-25T16:02:55.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.848+0000 7fa46a1af740 20 client.25655 trim_cache size 0 max 16384 2024-04-25T16:02:55.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.896+0000 7fa46a1af740 1 Processor -- start 2024-04-25T16:02:55.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.897+0000 7fa46a1af740 1 -- start start 2024-04-25T16:02:55.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.897+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671df2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.897+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a166b0 0x560671df3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a37390 0x560671df4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa46a1af740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c1b17f0 con 0x560671a166b0 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa460894640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a166b0 0x560671df3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa460894640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a166b0 0x560671df3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53610/0 (socket says 172.21.15.31:53610) 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa460894640 1 -- 172.21.15.31:0/3731666710 learned_addr learned my addr 172.21.15.31:0/3731666710 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa46a1af740 1 -- 172.21.15.31:0/3731666710 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c1b3330 con 0x560671a0e4d0 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa46a1af740 1 -- 172.21.15.31:0/3731666710 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c1b1ec0 con 0x560671a37390 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa461095640 1 --2- 172.21.15.31:0/3731666710 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671df2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa460894640 1 -- 172.21.15.31:0/3731666710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a37390 msgr2=0x560671df4210 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa460894640 1 --2- 172.21.15.31:0/3731666710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a37390 0x560671df4210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa460894640 1 -- 172.21.15.31:0/3731666710 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 msgr2=0x560671df2cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa460894640 1 --2- 172.21.15.31:0/3731666710 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671df2cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.898+0000 7fa460894640 1 -- 172.21.15.31:0/3731666710 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560671ddb410 con 0x560671a166b0 2024-04-25T16:02:55.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.899+0000 7fa460894640 1 --2- 172.21.15.31:0/3731666710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a166b0 0x560671df3200 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa44c013750 tx=0x7fa44c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9c713ccd2b11022 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.900+0000 7fa446ffd640 1 -- 172.21.15.31:0/3731666710 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0130d0 con 0x560671a166b0 2024-04-25T16:02:55.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.900+0000 7fa446ffd640 1 -- 172.21.15.31:0/3731666710 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa44c00f730 con 0x560671a166b0 2024-04-25T16:02:55.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.900+0000 7fa446ffd640 1 -- 172.21.15.31:0/3731666710 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0052a0 con 0x560671a166b0 2024-04-25T16:02:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.901+0000 7fa46a1af740 1 -- 172.21.15.31:0/3731666710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a166b0 msgr2=0x560671df3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.901+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3731666710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a166b0 0x560671df3200 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa44c013750 tx=0x7fa44c005f90 comp rx=0 tx=0).stop 2024-04-25T16:02:55.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.901+0000 7fa46a1af740 1 -- 172.21.15.31:0/3731666710 shutdown_connections 2024-04-25T16:02:55.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.901+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3731666710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a37390 0x560671df4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.901+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3731666710 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a166b0 0x560671df3200 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.901+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3731666710 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671df2cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.901+0000 7fa46a1af740 1 -- 172.21.15.31:0/3731666710 >> 172.21.15.31:0/3731666710 conn(0x560671a2b4e0 msgr2=0x560671dde390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.902+0000 7fa46a1af740 1 -- 172.21.15.31:0/3731666710 shutdown_connections 2024-04-25T16:02:55.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.902+0000 7fa46a1af740 1 -- 172.21.15.31:0/3731666710 wait complete. 2024-04-25T16:02:55.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.902+0000 7fa46a1af740 1 -- start start 2024-04-25T16:02:55.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.903+0000 7fa46a1af740 1 Processor -- start 2024-04-25T16:02:55.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.903+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671dfb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.903+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671df7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.903+0000 7fa46a1af740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671df8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.903+0000 7fa46a1af740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c1b2c60 con 0x560671a37390 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.903+0000 7fa46a1af740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c1b2590 con 0x560671a0e4d0 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.903+0000 7fa46a1af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa45c1af7f0 con 0x560671a166b0 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.903+0000 7fa46a1af740 10 client.0 _ll_register_callbacks cb 0x7fa454009cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.904+0000 7fa461896640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671df8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.904+0000 7fa461896640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671df8170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53618/0 (socket says 172.21.15.31:53618) 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.904+0000 7fa461896640 1 -- 172.21.15.31:0/3006369292 learned_addr learned my addr 172.21.15.31:0/3006369292 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.904+0000 7fa461896640 1 -- 172.21.15.31:0/3006369292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 msgr2=0x560671df7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.904+0000 7fa461896640 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671df7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.904+0000 7fa461896640 1 -- 172.21.15.31:0/3006369292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 msgr2=0x560671dfb070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.904+0000 7fa461896640 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671dfb070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.904+0000 7fa461896640 1 -- 172.21.15.31:0/3006369292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560671e686f0 con 0x560671a37390 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.905+0000 7fa461896640 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671df8170 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa4580cc440 tx=0x7fa458009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3457b2ba0358f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:55.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.905+0000 7fa444ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.905+0000 7fa444ff9640 1 -- 172.21.15.31:0/3006369292 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4580e9070 con 0x560671a37390 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.905+0000 7fa444ff9640 1 -- 172.21.15.31:0/3006369292 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa4580d8fa0 con 0x560671a37390 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.905+0000 7fa46a1af740 1 -- 172.21.15.31:0/3006369292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa44c00f730 con 0x560671a37390 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.905+0000 7fa444ff9640 1 -- 172.21.15.31:0/3006369292 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa4580cfed0 con 0x560671a37390 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.906+0000 7fa46a1af740 10 client.15978 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.906+0000 7fa46a1af740 1 -- 172.21.15.31:0/3006369292 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560671e69970 con 0x560671a37390 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.906+0000 7fa444ff9640 1 -- 172.21.15.31:0/3006369292 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa44c00f730 con 0x560671a37390 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.906+0000 7fa444ff9640 1 -- 172.21.15.31:0/3006369292 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fa458052070 con 0x560671a37390 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.906+0000 7fa46a1af740 10 client.15978 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.906+0000 7fa46a1af740 10 client.15978 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.906+0000 7fa46a1af740 20 client.15978 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.907+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5606719cb3a0 0x560671e6c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:55.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.907+0000 7fa46a1af740 4 client.15978 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:55.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.907+0000 7fa46a1af740 1 -- 172.21.15.31:0/3006369292 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560671d45800 con 0x5606719cb3a0 2024-04-25T16:02:55.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.908+0000 7fa461095640 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5606719cb3a0 0x560671e6c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:55.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.913+0000 7fa461095640 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5606719cb3a0 0x560671e6c400 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aa283532c2daf3d3 server_cookie=f55390fa2d5be5b1 in_seq=0 out_seq=0 2024-04-25T16:02:55.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.913+0000 7fa444ff9640 10 client.15978 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:02:55.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.915+0000 7fa444ff9640 1 -- 172.21.15.31:0/3006369292 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560671d45800 con 0x5606719cb3a0 2024-04-25T16:02:55.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.915+0000 7fa444ff9640 10 client.15978 handle_command_reply: tid=0 2024-04-25T16:02:55.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.915+0000 7fa46a1af740 1 client.15978 shutdown 2024-04-25T16:02:55.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.916+0000 7fa46a1af740 1 -- 172.21.15.31:0/3006369292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 msgr2=0x560671df8170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.916+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671df8170 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fa4580cc440 tx=0x7fa458009260 comp rx=0 tx=0).stop 2024-04-25T16:02:55.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.916+0000 7fa46a1af740 1 -- 172.21.15.31:0/3006369292 shutdown_connections 2024-04-25T16:02:55.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.916+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560671a37390 0x560671df8170 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.916+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560671a166b0 0x560671df7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.917+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5606719cb3a0 0x560671e6c400 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.917+0000 7fa46a1af740 1 --2- 172.21.15.31:0/3006369292 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560671a0e4d0 0x560671dfb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.917+0000 7fa444ff9640 0 client.15978 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:02:55.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.917+0000 7fa46a1af740 1 -- 172.21.15.31:0/3006369292 >> 172.21.15.31:0/3006369292 conn(0x560671a2b4e0 msgr2=0x560671df4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.917+0000 7fa46a1af740 1 -- 172.21.15.31:0/3006369292 shutdown_connections 2024-04-25T16:02:55.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.917+0000 7fa46a1af740 1 -- 172.21.15.31:0/3006369292 wait complete. 2024-04-25T16:02:55.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.917+0000 7fa46a1af740 20 client.15978 trim_cache size 0 max 16384 2024-04-25T16:02:55.920 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:55.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.920+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4480afd00 msgr2=0x7fa4480b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.920+0000 7fa4632f9640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4480afd00 0x7fa4480b2210 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x560671dd9530 tx=0x7fa454009b00 comp rx=0 tx=0).stop 2024-04-25T16:02:55.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.920+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa45c10c010 msgr2=0x7fa45c1ac0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:55.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.920+0000 7fa4632f9640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa45c10c010 0x7fa45c1ac0f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa458006010 tx=0x7fa458005020 comp rx=0 tx=0).stop 2024-04-25T16:02:55.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.921+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 shutdown_connections 2024-04-25T16:02:55.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.921+0000 7fa4632f9640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa4480afd00 0x7fa4480b2210 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.921+0000 7fa4632f9640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa45c10c010 0x7fa45c1ac0f0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.921+0000 7fa4632f9640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c10b310 0x7fa45c1a5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.921+0000 7fa4632f9640 1 --2- 172.21.15.31:0/3489986986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa45c10a420 0x7fa45c1a4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:55.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.921+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 >> 172.21.15.31:0/3489986986 conn(0x7fa45c0704e0 msgr2=0x7fa45c113b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:55.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.921+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 shutdown_connections 2024-04-25T16:02:55.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:55.921+0000 7fa4632f9640 1 -- 172.21.15.31:0/3489986986 wait complete. 2024-04-25T16:02:56.004 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:02:56.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.524+0000 7f8d04b65640 1 -- 172.21.15.31:0/2815928172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d0010b190 msgr2=0x7f8d0010b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:56.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.524+0000 7f8d04b65640 1 --2- 172.21.15.31:0/2815928172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d0010b190 0x7f8d0010b5a0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8009ca0 tx=0x7f8cf802a8d0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.525+0000 7f8d04b65640 1 -- 172.21.15.31:0/2815928172 shutdown_connections 2024-04-25T16:02:56.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.525+0000 7f8d04b65640 1 --2- 172.21.15.31:0/2815928172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d0010fe90 0x7f8d001102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.525+0000 7f8d04b65640 1 --2- 172.21.15.31:0/2815928172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d0010b190 0x7f8d0010b5a0 secure :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f8cf8009ca0 tx=0x7f8cf802a8d0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.525+0000 7f8d04b65640 1 --2- 172.21.15.31:0/2815928172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d0010a340 0x7f8d0010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.525+0000 7f8d04b65640 1 -- 172.21.15.31:0/2815928172 >> 172.21.15.31:0/2815928172 conn(0x7f8d00070320 msgr2=0x7f8d000727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:56.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.533+0000 7f8d04b65640 1 -- 172.21.15.31:0/2815928172 shutdown_connections 2024-04-25T16:02:56.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.536+0000 7f8d04b65640 1 -- 172.21.15.31:0/2815928172 wait complete. 2024-04-25T16:02:56.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.537+0000 7f8d04b65640 1 Processor -- start 2024-04-25T16:02:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.540+0000 7f8d04b65640 1 -- start start 2024-04-25T16:02:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.540+0000 7f8d04b65640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d0010a340 0x7f8d001171c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.540+0000 7f8d04b65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d0010fe90 0x7f8d00117780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.540+0000 7f8d04b65640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d00118660 0x7f8d00118a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.540+0000 7f8d04b65640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0011e360 con 0x7f8d0010a340 2024-04-25T16:02:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.540+0000 7f8d04b65640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0011e1e0 con 0x7f8d0010fe90 2024-04-25T16:02:56.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.540+0000 7f8d04b65640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d0011e4e0 con 0x7f8d00118660 2024-04-25T16:02:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.542+0000 7f8cfeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d0010fe90 0x7f8d00117780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.544+0000 7f8cfeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d0010fe90 0x7f8d00117780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52792/0 (socket says 172.21.15.31:52792) 2024-04-25T16:02:56.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.544+0000 7f8cfeffd640 1 -- 172.21.15.31:0/270749853 learned_addr learned my addr 172.21.15.31:0/270749853 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.545+0000 7f8cfffff640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d00118660 0x7f8d00118a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.546+0000 7f8cfeffd640 1 -- 172.21.15.31:0/270749853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d00118660 msgr2=0x7f8d00118a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.546+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d00118660 0x7f8d00118a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.546+0000 7f8cfeffd640 1 -- 172.21.15.31:0/270749853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d0010a340 msgr2=0x7f8d001171c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.546+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d0010a340 0x7f8d001171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.546+0000 7f8cfeffd640 1 -- 172.21.15.31:0/270749853 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d001191a0 con 0x7f8d0010fe90 2024-04-25T16:02:56.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.549+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d0010fe90 0x7f8d00117780 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8cf802aec0 tx=0x7f8cf8003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:56.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.553+0000 7f8cfcff9640 1 -- 172.21.15.31:0/270749853 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf8004240 con 0x7f8d0010fe90 2024-04-25T16:02:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.553+0000 7f8cfcff9640 1 -- 172.21.15.31:0/270749853 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cf80305d0 con 0x7f8d0010fe90 2024-04-25T16:02:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.553+0000 7f8cfcff9640 1 -- 172.21.15.31:0/270749853 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf803b670 con 0x7f8d0010fe90 2024-04-25T16:02:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.554+0000 7f8d04b65640 1 -- 172.21.15.31:0/270749853 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d00119430 con 0x7f8d0010fe90 2024-04-25T16:02:56.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.554+0000 7f8d04b65640 1 -- 172.21.15.31:0/270749853 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d001199b0 con 0x7f8d0010fe90 2024-04-25T16:02:56.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.559+0000 7f8d0cc74740 1 Processor -- start 2024-04-25T16:02:56.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.560+0000 7f8d0cc74740 1 -- start start 2024-04-25T16:02:56.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.568+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f63a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.568+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f63af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.568+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 0x56108f63bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.568+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56108f59ed80 con 0x56108f2776b0 2024-04-25T16:02:56.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.568+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56108f558a20 con 0x56108f26f4d0 2024-04-25T16:02:56.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.568+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56108f59ad40 con 0x56108f298390 2024-04-25T16:02:56.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.569+0000 7f8cfffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f63a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.571+0000 7f8cfffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f63a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47040/0 (socket says 172.21.15.31:47040) 2024-04-25T16:02:56.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.571+0000 7f8cfffff640 1 -- 172.21.15.31:0/862267528 learned_addr learned my addr 172.21.15.31:0/862267528 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:56.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.571+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/862267528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f63af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.571+0000 7f8cfcff9640 1 -- 172.21.15.31:0/270749853 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8cf803b8d0 con 0x7f8d0010fe90 2024-04-25T16:02:56.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.572+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/862267528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 0x56108f63bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.572+0000 7f8cfffff640 1 -- 172.21.15.31:0/862267528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 msgr2=0x56108f63af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.572+0000 7f8cfffff640 1 --2- 172.21.15.31:0/862267528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f63af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.572+0000 7f8cfffff640 1 -- 172.21.15.31:0/862267528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 msgr2=0x56108f63bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.572+0000 7f8cfffff640 1 --2- 172.21.15.31:0/862267528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 0x56108f63bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.572+0000 7f8cfffff640 1 -- 172.21.15.31:0/862267528 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56108f563c30 con 0x56108f298390 2024-04-25T16:02:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.573+0000 7f8cfcff9640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8ce80afcf0 0x7f8ce80b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.573+0000 7f8cfcff9640 1 -- 172.21.15.31:0/270749853 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8cf8100690 con 0x7f8d0010fe90 2024-04-25T16:02:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.574+0000 7f8cfffff640 1 --2- 172.21.15.31:0/862267528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f63a9e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000a8a0 tx=0x7f8cf000ad60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14b42fea309b302d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:56.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.575+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8ce80afcf0 0x7f8ce80b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.581+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8ce80afcf0 0x7f8ce80b2200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x56108f63a700 tx=0x7f8cf4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:56.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.582+0000 7f8ce5ffb640 1 -- 172.21.15.31:0/862267528 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf0015070 con 0x56108f298390 2024-04-25T16:02:56.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.582+0000 7f8ce5ffb640 1 -- 172.21.15.31:0/862267528 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cf00113f0 con 0x56108f298390 2024-04-25T16:02:56.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.582+0000 7f8ce5ffb640 1 -- 172.21.15.31:0/862267528 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf00185c0 con 0x56108f298390 2024-04-25T16:02:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.584+0000 7f8d0cc74740 1 -- 172.21.15.31:0/862267528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 msgr2=0x56108f63a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.584+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/862267528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f63a9e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8cf000a8a0 tx=0x7f8cf000ad60 comp rx=0 tx=0).stop 2024-04-25T16:02:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.584+0000 7f8d0cc74740 1 -- 172.21.15.31:0/862267528 shutdown_connections 2024-04-25T16:02:56.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.584+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/862267528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 0x56108f63bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.584+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/862267528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f63af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.584+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/862267528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f63a9e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.584+0000 7f8d0cc74740 1 -- 172.21.15.31:0/862267528 >> 172.21.15.31:0/862267528 conn(0x56108f28c4e0 msgr2=0x56108f6361f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.590+0000 7f8d0cc74740 1 -- 172.21.15.31:0/862267528 shutdown_connections 2024-04-25T16:02:56.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.590+0000 7f8d0cc74740 1 -- 172.21.15.31:0/862267528 wait complete. 2024-04-25T16:02:56.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.591+0000 7f8d0cc74740 1 -- start start 2024-04-25T16:02:56.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.591+0000 7f8d0cc74740 1 Processor -- start 2024-04-25T16:02:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.592+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f6448c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.592+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 0x56108f644e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.592+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f64bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.592+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56108f63c700 con 0x56108f2776b0 2024-04-25T16:02:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.592+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56108f525dc0 con 0x56108f26f4d0 2024-04-25T16:02:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.592+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56108f63c880 con 0x56108f298390 2024-04-25T16:02:56.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.592+0000 7f8d0cc74740 10 client.0 _ll_register_callbacks cb 0x56108f541600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cff7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f6448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cff7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f6448c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52806/0 (socket says 172.21.15.31:52806) 2024-04-25T16:02:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cff7fe640 1 -- 172.21.15.31:0/3724338638 learned_addr learned my addr 172.21.15.31:0/3724338638 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cfffff640 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f64bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 0x56108f644e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cff7fe640 1 -- 172.21.15.31:0/3724338638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 msgr2=0x56108f64bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f64bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cff7fe640 1 -- 172.21.15.31:0/3724338638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 msgr2=0x56108f644e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:56.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 0x56108f644e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.593+0000 7f8cff7fe640 1 -- 172.21.15.31:0/3724338638 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56108f2b03f0 con 0x56108f26f4d0 2024-04-25T16:02:56.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.594+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f6448c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400dd30 tx=0x7f8cf40047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3083416530904c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:56.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.594+0000 7f8cdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:02:56.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.594+0000 7f8cdeffd640 1 -- 172.21.15.31:0/3724338638 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4024060 con 0x56108f26f4d0 2024-04-25T16:02:56.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.594+0000 7f8cdeffd640 1 -- 172.21.15.31:0/3724338638 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8cf400dd60 con 0x56108f26f4d0 2024-04-25T16:02:56.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.595+0000 7f8d0cc74740 1 -- 172.21.15.31:0/3724338638 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8cf00113f0 con 0x56108f26f4d0 2024-04-25T16:02:56.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.595+0000 7f8cdeffd640 1 -- 172.21.15.31:0/3724338638 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8cf401f400 con 0x56108f26f4d0 2024-04-25T16:02:56.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.595+0000 7f8d0cc74740 10 client.25667 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:56.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.595+0000 7f8d0cc74740 1 -- 172.21.15.31:0/3724338638 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56108f37a7d0 con 0x56108f26f4d0 2024-04-25T16:02:56.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.596+0000 7f8cdeffd640 1 -- 172.21.15.31:0/3724338638 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cf401fa50 con 0x56108f26f4d0 2024-04-25T16:02:56.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.596+0000 7f8cdeffd640 1 -- 172.21.15.31:0/3724338638 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8cf4013070 con 0x56108f26f4d0 2024-04-25T16:02:56.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.596+0000 7f8d0cc74740 10 client.25667 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:56.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.596+0000 7f8d0cc74740 10 client.25667 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:02:56.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.596+0000 7f8d0cc74740 20 client.25667 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:56.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.596+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56108f256f60 0x56108f6526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.596+0000 7f8d0cc74740 4 client.25667 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:56.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.596+0000 7f8d0cc74740 1 -- 172.21.15.31:0/3724338638 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8cf00113f0 con 0x56108f256f60 2024-04-25T16:02:56.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.603+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56108f256f60 0x56108f6526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.605+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56108f256f60 0x56108f6526a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb9a26c41c688fed server_cookie=a85db47f1e75e39a in_seq=0 out_seq=0 2024-04-25T16:02:56.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.605+0000 7f8cdeffd640 10 client.25667 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:02:56.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.606+0000 7f8cdeffd640 1 -- 172.21.15.31:0/3724338638 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f8cf00113f0 con 0x56108f256f60 2024-04-25T16:02:56.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.606+0000 7f8cdeffd640 10 client.25667 handle_command_reply: tid=0 2024-04-25T16:02:56.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.606+0000 7f8d0cc74740 1 client.25667 shutdown 2024-04-25T16:02:56.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 -- 172.21.15.31:0/3724338638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 msgr2=0x56108f6448c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:56.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f6448c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8cf400dd30 tx=0x7f8cf40047c0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 -- 172.21.15.31:0/3724338638 shutdown_connections 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f64bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56108f256f60 0x56108f6526a0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f2776b0 0x56108f644e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/3724338638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f26f4d0 0x56108f6448c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8cdeffd640 0 client.25667 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 -- 172.21.15.31:0/3724338638 >> 172.21.15.31:0/3724338638 conn(0x56108f28c4e0 msgr2=0x56108f639100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 -- 172.21.15.31:0/3724338638 shutdown_connections 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.607+0000 7f8d0cc74740 1 -- 172.21.15.31:0/3724338638 wait complete. 2024-04-25T16:02:56.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.608+0000 7f8d0cc74740 20 client.25667 trim_cache size 0 max 16384 2024-04-25T16:02:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.646+0000 7f8d0cc74740 1 Processor -- start 2024-04-25T16:02:56.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.646+0000 7f8d0cc74740 1 -- start start 2024-04-25T16:02:56.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.646+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f26f4d0 0x56108f636330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.647+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f64bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.647+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f298390 0x56108f644490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.647+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8cec00e500 con 0x56108f298390 2024-04-25T16:02:56.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.647+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8cec010040 con 0x56108f2776b0 2024-04-25T16:02:56.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.647+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8cec00ebd0 con 0x56108f26f4d0 2024-04-25T16:02:56.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.647+0000 7f8cfeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f64bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.648+0000 7f8cfeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f64bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52816/0 (socket says 172.21.15.31:52816) 2024-04-25T16:02:56.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.648+0000 7f8cfeffd640 1 -- 172.21.15.31:0/1581317043 learned_addr learned my addr 172.21.15.31:0/1581317043 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:56.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.648+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/1581317043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f26f4d0 0x56108f636330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.648+0000 7f8cfffff640 1 --2- 172.21.15.31:0/1581317043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f298390 0x56108f644490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.648+0000 7f8cfeffd640 1 -- 172.21.15.31:0/1581317043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f26f4d0 msgr2=0x56108f636330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.648+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/1581317043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f26f4d0 0x56108f636330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.648+0000 7f8cfeffd640 1 -- 172.21.15.31:0/1581317043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f298390 msgr2=0x56108f644490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.648+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/1581317043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f298390 0x56108f644490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.648+0000 7f8cfeffd640 1 -- 172.21.15.31:0/1581317043 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56108f6503e0 con 0x56108f2776b0 2024-04-25T16:02:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.649+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/1581317043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f64bcf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8cf80c9410 tx=0x7f8cf803e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=153bce00bdb466b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:56.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.653+0000 7f8ce57fa640 1 -- 172.21.15.31:0/1581317043 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf80c9220 con 0x56108f2776b0 2024-04-25T16:02:56.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.653+0000 7f8ce57fa640 1 -- 172.21.15.31:0/1581317043 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8cf8036030 con 0x56108f2776b0 2024-04-25T16:02:56.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.655+0000 7f8d0cc74740 1 -- 172.21.15.31:0/1581317043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 msgr2=0x56108f64bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:56.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.655+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/1581317043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f64bcf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8cf80c9410 tx=0x7f8cf803e040 comp rx=0 tx=0).stop 2024-04-25T16:02:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.658+0000 7f8d0cc74740 1 -- 172.21.15.31:0/1581317043 shutdown_connections 2024-04-25T16:02:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.658+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/1581317043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f298390 0x56108f644490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.658+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/1581317043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f64bcf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.658+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/1581317043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f26f4d0 0x56108f636330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.658+0000 7f8d0cc74740 1 -- 172.21.15.31:0/1581317043 >> 172.21.15.31:0/1581317043 conn(0x56108f28c4e0 msgr2=0x56108f635c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.665+0000 7f8d0cc74740 1 -- 172.21.15.31:0/1581317043 shutdown_connections 2024-04-25T16:02:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.665+0000 7f8d0cc74740 1 -- 172.21.15.31:0/1581317043 wait complete. 2024-04-25T16:02:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.666+0000 7f8d0cc74740 1 -- start start 2024-04-25T16:02:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.667+0000 7f8d0cc74740 1 Processor -- start 2024-04-25T16:02:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.667+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f26f4d0 0x56108f649830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.667+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f649d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.667+0000 7f8d0cc74740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f64aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.667+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8cec00f970 con 0x56108f26f4d0 2024-04-25T16:02:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.667+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8cec00f2a0 con 0x56108f2776b0 2024-04-25T16:02:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.667+0000 7f8d0cc74740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8cec00c4e0 con 0x56108f298390 2024-04-25T16:02:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.668+0000 7f8d0cc74740 10 client.0 _ll_register_callbacks cb 0x7f8cf4009e20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:56.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.669+0000 7f8cfeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f649d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.669+0000 7f8cfeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f649d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52822/0 (socket says 172.21.15.31:52822) 2024-04-25T16:02:56.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.669+0000 7f8cfeffd640 1 -- 172.21.15.31:0/456252213 learned_addr learned my addr 172.21.15.31:0/456252213 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:56.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.669+0000 7f8cfffff640 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f64aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.670+0000 7f8cfeffd640 1 -- 172.21.15.31:0/456252213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 msgr2=0x56108f64aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.670+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f64aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.670+0000 7f8cfeffd640 1 -- 172.21.15.31:0/456252213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f26f4d0 msgr2=0x56108f649830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.670+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f26f4d0 0x56108f649830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.670+0000 7f8cfeffd640 1 -- 172.21.15.31:0/456252213 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56108f64b260 con 0x56108f2776b0 2024-04-25T16:02:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.671+0000 7f8cfeffd640 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f649d70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8cf802ae90 tx=0x7f8cf8031950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6d5704eb22dfea2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.675+0000 7f8cdeffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:02:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.679+0000 7f8cdeffd640 1 -- 172.21.15.31:0/456252213 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8cf80c9220 con 0x56108f2776b0 2024-04-25T16:02:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.679+0000 7f8cdeffd640 1 -- 172.21.15.31:0/456252213 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8cf80c5d00 con 0x56108f2776b0 2024-04-25T16:02:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.679+0000 7f8d0cc74740 1 -- 172.21.15.31:0/456252213 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56108f5ac990 con 0x56108f2776b0 2024-04-25T16:02:56.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.681+0000 7f8cdeffd640 1 -- 172.21.15.31:0/456252213 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f8cf80c9710 con 0x56108f2776b0 2024-04-25T16:02:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.684+0000 7f8d0cc74740 10 client.25673 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.684+0000 7f8d0cc74740 1 -- 172.21.15.31:0/456252213 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x56108f6badf0 con 0x56108f2776b0 2024-04-25T16:02:56.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.686+0000 7f8cdeffd640 1 -- 172.21.15.31:0/456252213 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56108f5ac990 con 0x56108f2776b0 2024-04-25T16:02:56.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.686+0000 7f8cdeffd640 1 -- 172.21.15.31:0/456252213 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f8cf80d2170 con 0x56108f2776b0 2024-04-25T16:02:56.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.687+0000 7f8d0cc74740 10 client.25673 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:56.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.687+0000 7f8d0cc74740 10 client.25673 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:02:56.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.687+0000 7f8d0cc74740 20 client.25673 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:56.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.687+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56108f22c3a0 0x56108f6bd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:56.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.687+0000 7f8d0cc74740 4 client.25673 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:56.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.687+0000 7f8d0cc74740 1 -- 172.21.15.31:0/456252213 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56108f5a6800 con 0x56108f22c3a0 2024-04-25T16:02:56.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.688+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56108f22c3a0 0x56108f6bd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:56.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.691+0000 7f8cff7fe640 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56108f22c3a0 0x56108f6bd640 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8f155ebc4baa364d server_cookie=d592e7bae90f189e in_seq=0 out_seq=0 2024-04-25T16:02:56.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.691+0000 7f8cdeffd640 10 client.25673 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:02:56.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.692+0000 7f8cdeffd640 1 -- 172.21.15.31:0/456252213 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56108f5a6800 con 0x56108f22c3a0 2024-04-25T16:02:56.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.692+0000 7f8cdeffd640 10 client.25673 handle_command_reply: tid=0 2024-04-25T16:02:56.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.692+0000 7f8d0cc74740 1 client.25673 shutdown 2024-04-25T16:02:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.695+0000 7f8d0cc74740 1 -- 172.21.15.31:0/456252213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 msgr2=0x56108f649d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.695+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f649d70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8cf802ae90 tx=0x7f8cf8031950 comp rx=0 tx=0).stop 2024-04-25T16:02:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.696+0000 7f8d0cc74740 1 -- 172.21.15.31:0/456252213 shutdown_connections 2024-04-25T16:02:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.696+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56108f298390 0x56108f64aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.696+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56108f2776b0 0x56108f649d70 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8cf802ae90 tx=0x7f8cf8031950 comp rx=0 tx=0).stop 2024-04-25T16:02:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.696+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56108f22c3a0 0x56108f6bd640 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.696+0000 7f8d0cc74740 1 --2- 172.21.15.31:0/456252213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56108f26f4d0 0x56108f649830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.696+0000 7f8d0cc74740 1 -- 172.21.15.31:0/456252213 >> 172.21.15.31:0/456252213 conn(0x56108f28c4e0 msgr2=0x56108f638ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:56.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.696+0000 7f8cdeffd640 0 client.25673 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:02:56.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.712+0000 7f8d0cc74740 1 -- 172.21.15.31:0/456252213 shutdown_connections 2024-04-25T16:02:56.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.713+0000 7f8d0cc74740 1 -- 172.21.15.31:0/456252213 wait complete. 2024-04-25T16:02:56.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.713+0000 7f8d0cc74740 20 client.25673 trim_cache size 0 max 16384 2024-04-25T16:02:56.717 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:56.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.721+0000 7f8d04b65640 1 -- 172.21.15.31:0/270749853 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8ce80afcf0 msgr2=0x7f8ce80b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:56.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.721+0000 7f8d04b65640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8ce80afcf0 0x7f8ce80b2200 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x56108f63a700 tx=0x7f8cf4008040 comp rx=0 tx=0).stop 2024-04-25T16:02:56.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.721+0000 7f8d04b65640 1 -- 172.21.15.31:0/270749853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d0010fe90 msgr2=0x7f8d00117780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:56.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.721+0000 7f8d04b65640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d0010fe90 0x7f8d00117780 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f8cf802aec0 tx=0x7f8cf8003ce0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.727+0000 7f8d04b65640 1 -- 172.21.15.31:0/270749853 shutdown_connections 2024-04-25T16:02:56.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.727+0000 7f8d04b65640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8ce80afcf0 0x7f8ce80b2200 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.727+0000 7f8d04b65640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d00118660 0x7f8d00118a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.727+0000 7f8d04b65640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d0010fe90 0x7f8d00117780 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.727+0000 7f8d04b65640 1 --2- 172.21.15.31:0/270749853 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d0010a340 0x7f8d001171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:56.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.727+0000 7f8d04b65640 1 -- 172.21.15.31:0/270749853 >> 172.21.15.31:0/270749853 conn(0x7f8d00070320 msgr2=0x7f8d0010be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:56.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.727+0000 7f8d04b65640 1 -- 172.21.15.31:0/270749853 shutdown_connections 2024-04-25T16:02:56.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:56.727+0000 7f8d04b65640 1 -- 172.21.15.31:0/270749853 wait complete. 2024-04-25T16:02:56.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:56 smithi031 ceph-mon[29140]: pgmap v400: 97 pgs: 97 active+clean; 6.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 539 MiB/s wr, 421 op/s 2024-04-25T16:02:56.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:02:56.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:56 smithi063 ceph-mon[32413]: pgmap v400: 97 pgs: 97 active+clean; 6.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 539 MiB/s wr, 421 op/s 2024-04-25T16:02:56.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:56 smithi186 ceph-mon[33258]: pgmap v400: 97 pgs: 97 active+clean; 6.3 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 539 MiB/s wr, 421 op/s 2024-04-25T16:02:57.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.220+0000 7fc4a3f85640 1 -- 172.21.15.31:0/982380918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 msgr2=0x7fc49c110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:57.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.220+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/982380918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 0x7fc49c110440 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc498009900 tx=0x7fc49802a3f0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.221+0000 7fc4a3f85640 1 -- 172.21.15.31:0/982380918 shutdown_connections 2024-04-25T16:02:57.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.221+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/982380918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 0x7fc49c110440 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.221+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/982380918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc49c10b310 0x7fc49c10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.221+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/982380918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc49c10a420 0x7fc49c10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.221+0000 7fc4a3f85640 1 -- 172.21.15.31:0/982380918 >> 172.21.15.31:0/982380918 conn(0x7fc49c0704e0 msgr2=0x7fc49c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.223+0000 7fc4a3f85640 1 -- 172.21.15.31:0/982380918 shutdown_connections 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.223+0000 7fc4a3f85640 1 -- 172.21.15.31:0/982380918 wait complete. 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.225+0000 7fc4a3f85640 1 Processor -- start 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.225+0000 7fc4a3f85640 1 -- start start 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.226+0000 7fc4a3f85640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc49c10a420 0x7fc49c117050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.226+0000 7fc4a3f85640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc49c10b310 0x7fc49c117590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.226+0000 7fc4a3f85640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 0x7fc49c1184f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.226+0000 7fc4a3f85640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c11e490 con 0x7fc49c10a420 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.226+0000 7fc4a3f85640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c11e310 con 0x7fc49c110030 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.226+0000 7fc4a3f85640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc49c11e610 con 0x7fc49c10b310 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.226+0000 7fc4a2522640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 0x7fc49c1184f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.226+0000 7fc4a2522640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 0x7fc49c1184f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52840/0 (socket says 172.21.15.31:52840) 2024-04-25T16:02:57.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.226+0000 7fc4a2522640 1 -- 172.21.15.31:0/1138189066 learned_addr learned my addr 172.21.15.31:0/1138189066 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:57.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.230+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc49c10a420 0x7fc49c117050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.231+0000 7fc4a2522640 1 -- 172.21.15.31:0/1138189066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc49c10b310 msgr2=0x7fc49c117590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:02:57.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.231+0000 7fc4a2522640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc49c10b310 0x7fc49c117590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.231+0000 7fc4a2522640 1 -- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc49c10a420 msgr2=0x7fc49c117050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:57.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.231+0000 7fc4a2522640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc49c10a420 0x7fc49c117050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.231+0000 7fc4a2522640 1 -- 172.21.15.31:0/1138189066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc49c118b60 con 0x7fc49c110030 2024-04-25T16:02:57.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.237+0000 7fc4a2522640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 0x7fc49c1184f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc498003b90 tx=0x7fc498003bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:57.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.238+0000 7fc492ffd640 1 -- 172.21.15.31:0/1138189066 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49803f070 con 0x7fc49c110030 2024-04-25T16:02:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.238+0000 7fc492ffd640 1 -- 172.21.15.31:0/1138189066 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc498031c50 con 0x7fc49c110030 2024-04-25T16:02:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.247+0000 7fc4a3f85640 1 -- 172.21.15.31:0/1138189066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc49c118df0 con 0x7fc49c110030 2024-04-25T16:02:57.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.248+0000 7fc4a3f85640 1 -- 172.21.15.31:0/1138189066 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc49c119330 con 0x7fc49c110030 2024-04-25T16:02:57.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.248+0000 7fc492ffd640 1 -- 172.21.15.31:0/1138189066 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc498035030 con 0x7fc49c110030 2024-04-25T16:02:57.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.253+0000 7fc492ffd640 1 -- 172.21.15.31:0/1138189066 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc498042070 con 0x7fc49c110030 2024-04-25T16:02:57.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.254+0000 7fc492ffd640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc4880afc30 0x7fc4880b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.256+0000 7fc492ffd640 1 -- 172.21.15.31:0/1138189066 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc498100640 con 0x7fc49c110030 2024-04-25T16:02:57.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.257+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc4880afc30 0x7fc4880b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.261+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc4880afc30 0x7fc4880b2140 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc48c005fd0 tx=0x7fc48c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:57.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.264+0000 7fc4aae30740 1 Processor -- start 2024-04-25T16:02:57.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.267+0000 7fc4aae30740 1 -- start start 2024-04-25T16:02:57.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.267+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 0x55672dbc7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.267+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d7fc4d0 0x55672dbc7d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.267+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d8046b0 0x55672dbc8a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.267+0000 7fc4aae30740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55672db2bd80 con 0x55672d7fc4d0 2024-04-25T16:02:57.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.267+0000 7fc4aae30740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55672dae5a20 con 0x55672d8046b0 2024-04-25T16:02:57.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.267+0000 7fc4aae30740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55672db27d40 con 0x55672d825390 2024-04-25T16:02:57.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.268+0000 7fc4a1d21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 0x55672dbc7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.268+0000 7fc4a1d21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 0x55672dbc7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47118/0 (socket says 172.21.15.31:47118) 2024-04-25T16:02:57.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.268+0000 7fc4a1d21640 1 -- 172.21.15.31:0/753681214 learned_addr learned my addr 172.21.15.31:0/753681214 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:57.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.269+0000 7fc4a1d21640 1 -- 172.21.15.31:0/753681214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d8046b0 msgr2=0x55672dbc8a20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:57.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.269+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/753681214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d8046b0 0x55672dbc8a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.269+0000 7fc4a1d21640 1 -- 172.21.15.31:0/753681214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d7fc4d0 msgr2=0x55672dbc7d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:02:57.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.269+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/753681214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d7fc4d0 0x55672dbc7d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.269+0000 7fc4a1d21640 1 -- 172.21.15.31:0/753681214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55672daf0c30 con 0x55672d825390 2024-04-25T16:02:57.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.270+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/753681214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 0x55672dbc7810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00fe50 tx=0x7fc48c012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c7a6bbb408630b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:57.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.279+0000 7fc487fff640 1 -- 172.21.15.31:0/753681214 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c022070 con 0x55672d825390 2024-04-25T16:02:57.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.279+0000 7fc487fff640 1 -- 172.21.15.31:0/753681214 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc48c00fe80 con 0x55672d825390 2024-04-25T16:02:57.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.281+0000 7fc487fff640 1 -- 172.21.15.31:0/753681214 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc48c010970 con 0x55672d825390 2024-04-25T16:02:57.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.283+0000 7fc4aae30740 1 -- 172.21.15.31:0/753681214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 msgr2=0x55672dbc7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:57.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.283+0000 7fc4aae30740 1 --2- 172.21.15.31:0/753681214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 0x55672dbc7810 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc48c00fe50 tx=0x7fc48c012000 comp rx=0 tx=0).stop 2024-04-25T16:02:57.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.283+0000 7fc4aae30740 1 -- 172.21.15.31:0/753681214 shutdown_connections 2024-04-25T16:02:57.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.283+0000 7fc4aae30740 1 --2- 172.21.15.31:0/753681214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d8046b0 0x55672dbc8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.283+0000 7fc4aae30740 1 --2- 172.21.15.31:0/753681214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d7fc4d0 0x55672dbc7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.283+0000 7fc4aae30740 1 --2- 172.21.15.31:0/753681214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 0x55672dbc7810 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.283+0000 7fc4aae30740 1 -- 172.21.15.31:0/753681214 >> 172.21.15.31:0/753681214 conn(0x55672d8194e0 msgr2=0x55672dbc3020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:57.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.283+0000 7fc4aae30740 1 -- 172.21.15.31:0/753681214 shutdown_connections 2024-04-25T16:02:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.284+0000 7fc4aae30740 1 -- 172.21.15.31:0/753681214 wait complete. 2024-04-25T16:02:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.284+0000 7fc4aae30740 1 -- start start 2024-04-25T16:02:57.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.289+0000 7fc4aae30740 1 Processor -- start 2024-04-25T16:02:57.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.290+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d7fc4d0 0x55672dbd1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.290+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d8046b0 0x55672dbd1b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.290+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 0x55672dbd8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.290+0000 7fc4aae30740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55672dbc9530 con 0x55672d8046b0 2024-04-25T16:02:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.290+0000 7fc4aae30740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55672dab2dc0 con 0x55672d7fc4d0 2024-04-25T16:02:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.290+0000 7fc4aae30740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55672dbc96b0 con 0x55672d825390 2024-04-25T16:02:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.290+0000 7fc4aae30740 10 client.0 _ll_register_callbacks cb 0x55672dace600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.292+0000 7fc4a1520640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d8046b0 0x55672dbd1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.292+0000 7fc4a1520640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d8046b0 0x55672dbd1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53694/0 (socket says 172.21.15.31:53694) 2024-04-25T16:02:57.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.292+0000 7fc4a1520640 1 -- 172.21.15.31:0/1677632415 learned_addr learned my addr 172.21.15.31:0/1677632415 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:57.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.293+0000 7fc4a1520640 1 -- 172.21.15.31:0/1677632415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 msgr2=0x55672dbd8ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:57.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.293+0000 7fc4a1520640 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 0x55672dbd8ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.293+0000 7fc4a1520640 1 -- 172.21.15.31:0/1677632415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d7fc4d0 msgr2=0x55672dbd1630 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:02:57.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.293+0000 7fc4a1520640 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d7fc4d0 0x55672dbd1630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.293+0000 7fc4a1520640 1 -- 172.21.15.31:0/1677632415 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55672d83d3f0 con 0x55672d8046b0 2024-04-25T16:02:57.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.294+0000 7fc4a1520640 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d8046b0 0x55672dbd1b70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc49400e620 tx=0x7fc49400eae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e7fd8f53d12d36 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:57.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.296+0000 7fc4857fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:02:57.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.296+0000 7fc4857fa640 1 -- 172.21.15.31:0/1677632415 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494004d20 con 0x55672d8046b0 2024-04-25T16:02:57.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.296+0000 7fc4857fa640 1 -- 172.21.15.31:0/1677632415 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc4940106a0 con 0x55672d8046b0 2024-04-25T16:02:57.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.296+0000 7fc4aae30740 1 -- 172.21.15.31:0/1677632415 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55672db39990 con 0x55672d8046b0 2024-04-25T16:02:57.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.297+0000 7fc4857fa640 1 -- 172.21.15.31:0/1677632415 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc4940108a0 con 0x55672d8046b0 2024-04-25T16:02:57.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.297+0000 7fc4aae30740 10 client.15999 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:57.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.297+0000 7fc4aae30740 1 -- 172.21.15.31:0/1677632415 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55672d9077d0 con 0x55672d8046b0 2024-04-25T16:02:57.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.298+0000 7fc4857fa640 1 -- 172.21.15.31:0/1677632415 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55672db39990 con 0x55672d8046b0 2024-04-25T16:02:57.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.299+0000 7fc4857fa640 1 -- 172.21.15.31:0/1677632415 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc49400f040 con 0x55672d8046b0 2024-04-25T16:02:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.299+0000 7fc4aae30740 10 client.15999 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.299+0000 7fc4aae30740 10 client.15999 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.299+0000 7fc4aae30740 20 client.15999 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.299+0000 7fc4aae30740 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55672d7e3f60 0x55672dbdf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.299+0000 7fc4aae30740 4 client.15999 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.299+0000 7fc4aae30740 1 -- 172.21.15.31:0/1677632415 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55672db33800 con 0x55672d7e3f60 2024-04-25T16:02:57.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.299+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55672d7e3f60 0x55672dbdf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.311+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55672d7e3f60 0x55672dbdf420 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=337c38017291ad0b server_cookie=59af45e29d6ebf36 in_seq=0 out_seq=0 2024-04-25T16:02:57.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.311+0000 7fc4857fa640 10 client.15999 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.321+0000 7fc4857fa640 1 -- 172.21.15.31:0/1677632415 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55672db33800 con 0x55672d7e3f60 2024-04-25T16:02:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.321+0000 7fc4857fa640 10 client.15999 handle_command_reply: tid=0 2024-04-25T16:02:57.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.321+0000 7fc4aae30740 1 client.15999 shutdown 2024-04-25T16:02:57.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.322+0000 7fc4aae30740 1 -- 172.21.15.31:0/1677632415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d8046b0 msgr2=0x55672dbd1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:57.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.322+0000 7fc4aae30740 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d8046b0 0x55672dbd1b70 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc49400e620 tx=0x7fc49400eae0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.327+0000 7fc4aae30740 1 -- 172.21.15.31:0/1677632415 shutdown_connections 2024-04-25T16:02:57.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.327+0000 7fc4aae30740 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55672d7e3f60 0x55672dbdf420 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.327+0000 7fc4aae30740 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d825390 0x55672dbd8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.327+0000 7fc4aae30740 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d8046b0 0x55672dbd1b70 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.327+0000 7fc4aae30740 1 --2- 172.21.15.31:0/1677632415 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d7fc4d0 0x55672dbd1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.327+0000 7fc4aae30740 1 -- 172.21.15.31:0/1677632415 >> 172.21.15.31:0/1677632415 conn(0x55672d8194e0 msgr2=0x55672dbc2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:57.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.327+0000 7fc4857fa640 0 client.15999 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:57.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.329+0000 7fc4aae30740 1 -- 172.21.15.31:0/1677632415 shutdown_connections 2024-04-25T16:02:57.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.330+0000 7fc4aae30740 1 -- 172.21.15.31:0/1677632415 wait complete. 2024-04-25T16:02:57.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.335+0000 7fc4aae30740 20 client.15999 trim_cache size 0 max 16384 2024-04-25T16:02:57.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.418+0000 7fc4aae30740 1 Processor -- start 2024-04-25T16:02:57.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.419+0000 7fc4aae30740 1 -- start start 2024-04-25T16:02:57.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.419+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d7fc4d0 0x55672dbcc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.419+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbcfd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.420+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d825390 0x55672dbd0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.420+0000 7fc4aae30740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc48000e500 con 0x55672d825390 2024-04-25T16:02:57.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.420+0000 7fc4aae30740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc480010040 con 0x55672d7fc4d0 2024-04-25T16:02:57.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.420+0000 7fc4aae30740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc48000ebd0 con 0x55672d8046b0 2024-04-25T16:02:57.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.422+0000 7fc4a2522640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d825390 0x55672dbd0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.422+0000 7fc4a2522640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d825390 0x55672dbd0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53696/0 (socket says 172.21.15.31:53696) 2024-04-25T16:02:57.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.422+0000 7fc4a2522640 1 -- 172.21.15.31:0/2732063463 learned_addr learned my addr 172.21.15.31:0/2732063463 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:57.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.422+0000 7fc4a1520640 1 --2- 172.21.15.31:0/2732063463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbcfd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.422+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/2732063463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d7fc4d0 0x55672dbcc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.423+0000 7fc4a1520640 1 -- 172.21.15.31:0/2732063463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d7fc4d0 msgr2=0x55672dbcc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:57.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.423+0000 7fc4a1520640 1 --2- 172.21.15.31:0/2732063463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d7fc4d0 0x55672dbcc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.423+0000 7fc4a1520640 1 -- 172.21.15.31:0/2732063463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d825390 msgr2=0x55672dbd0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:57.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.423+0000 7fc4a1520640 1 --2- 172.21.15.31:0/2732063463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d825390 0x55672dbd0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.423+0000 7fc4a1520640 1 -- 172.21.15.31:0/2732063463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55672dbd2230 con 0x55672d8046b0 2024-04-25T16:02:57.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.425+0000 7fc4a1520640 1 --2- 172.21.15.31:0/2732063463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbcfd20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c2c0 tx=0x7fc49400f050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dc56c06e15991af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:57.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.427+0000 7fc4877fe640 1 -- 172.21.15.31:0/2732063463 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc49401a650 con 0x55672d8046b0 2024-04-25T16:02:57.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.428+0000 7fc4877fe640 1 -- 172.21.15.31:0/2732063463 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc494003970 con 0x55672d8046b0 2024-04-25T16:02:57.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.428+0000 7fc4877fe640 1 -- 172.21.15.31:0/2732063463 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4940116f0 con 0x55672d8046b0 2024-04-25T16:02:57.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.431+0000 7fc4aae30740 1 -- 172.21.15.31:0/2732063463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 msgr2=0x55672dbcfd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:57.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.431+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2732063463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbcfd20 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c2c0 tx=0x7fc49400f050 comp rx=0 tx=0).stop 2024-04-25T16:02:57.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.436+0000 7fc4aae30740 1 -- 172.21.15.31:0/2732063463 shutdown_connections 2024-04-25T16:02:57.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.436+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2732063463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d825390 0x55672dbd0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.437+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2732063463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbcfd20 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.437+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2732063463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d7fc4d0 0x55672dbcc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.437+0000 7fc4aae30740 1 -- 172.21.15.31:0/2732063463 >> 172.21.15.31:0/2732063463 conn(0x55672d8194e0 msgr2=0x55672dbc6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:57.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.437+0000 7fc4aae30740 1 -- 172.21.15.31:0/2732063463 shutdown_connections 2024-04-25T16:02:57.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.437+0000 7fc4aae30740 1 -- 172.21.15.31:0/2732063463 wait complete. 2024-04-25T16:02:57.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.439+0000 7fc4aae30740 1 -- start start 2024-04-25T16:02:57.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.443+0000 7fc4aae30740 1 Processor -- start 2024-04-25T16:02:57.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.447+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d7fc4d0 0x55672dbd5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.448+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbd63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.448+0000 7fc4aae30740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d825390 0x55672dbd7310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.448+0000 7fc4aae30740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc48000f970 con 0x55672d7fc4d0 2024-04-25T16:02:57.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.448+0000 7fc4aae30740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc48000f2a0 con 0x55672d825390 2024-04-25T16:02:57.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.448+0000 7fc4aae30740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc48000c4e0 con 0x55672d8046b0 2024-04-25T16:02:57.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.449+0000 7fc4a2522640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d825390 0x55672dbd7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.449+0000 7fc4a2522640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d825390 0x55672dbd7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52882/0 (socket says 172.21.15.31:52882) 2024-04-25T16:02:57.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.449+0000 7fc4a2522640 1 -- 172.21.15.31:0/2041349164 learned_addr learned my addr 172.21.15.31:0/2041349164 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:57.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.449+0000 7fc4aae30740 10 client.0 _ll_register_callbacks cb 0x7fc4940094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:57.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.449+0000 7fc4a1520640 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbd63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.450+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d7fc4d0 0x55672dbd5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.451+0000 7fc4a1520640 1 -- 172.21.15.31:0/2041349164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d825390 msgr2=0x55672dbd7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:57.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.451+0000 7fc4a1520640 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d825390 0x55672dbd7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.451+0000 7fc4a1520640 1 -- 172.21.15.31:0/2041349164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d7fc4d0 msgr2=0x55672dbd5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:57.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.451+0000 7fc4a1520640 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d7fc4d0 0x55672dbd5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.451+0000 7fc4a1520640 1 -- 172.21.15.31:0/2041349164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55672dbd78d0 con 0x55672d8046b0 2024-04-25T16:02:57.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.452+0000 7fc4a1520640 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbd63e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55672dbc95a0 tx=0x7fc49400f240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7c4a483bf0aea71 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:57.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.452+0000 7fc4857fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:02:57.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.453+0000 7fc4857fa640 1 -- 172.21.15.31:0/2041349164 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc494003b20 con 0x55672d8046b0 2024-04-25T16:02:57.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.453+0000 7fc4857fa640 1 -- 172.21.15.31:0/2041349164 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc494003cc0 con 0x55672d8046b0 2024-04-25T16:02:57.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.454+0000 7fc4aae30740 1 -- 172.21.15.31:0/2041349164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc494003970 con 0x55672d8046b0 2024-04-25T16:02:57.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.455+0000 7fc4857fa640 1 -- 172.21.15.31:0/2041349164 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fc494011690 con 0x55672d8046b0 2024-04-25T16:02:57.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.455+0000 7fc4aae30740 10 client.25642 fetch_fsmap learned FSMap version 27 2024-04-25T16:02:57.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.456+0000 7fc4aae30740 1 -- 172.21.15.31:0/2041349164 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55672dc47c60 con 0x55672d8046b0 2024-04-25T16:02:57.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.457+0000 7fc4857fa640 1 -- 172.21.15.31:0/2041349164 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc49401a800 con 0x55672d8046b0 2024-04-25T16:02:57.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.457+0000 7fc4857fa640 1 -- 172.21.15.31:0/2041349164 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fc4940050b0 con 0x55672d8046b0 2024-04-25T16:02:57.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.457+0000 7fc4aae30740 10 client.25642 fetch_fsmap finished waiting for FSMap version 27 2024-04-25T16:02:57.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.457+0000 7fc4aae30740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:57.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.457+0000 7fc4aae30740 20 client.25642 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:57.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.458+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55672d7b93a0 0x55672dc4a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:57.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.458+0000 7fc4aae30740 4 client.25642 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:02:57.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.458+0000 7fc4aae30740 1 -- 172.21.15.31:0/2041349164 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fc494003970 con 0x55672d7b93a0 2024-04-25T16:02:57.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.458+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55672d7b93a0 0x55672dc4a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:57.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.472+0000 7fc4857fa640 10 client.25642 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:57.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.472+0000 7fc4a1d21640 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55672d7b93a0 0x55672dc4a620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e212d0fffa672408 server_cookie=e2ff4133bc29ee5d in_seq=0 out_seq=0 2024-04-25T16:02:57.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.476+0000 7fc4857fa640 1 -- 172.21.15.31:0/2041349164 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fc494003970 con 0x55672d7b93a0 2024-04-25T16:02:57.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.476+0000 7fc4857fa640 10 client.25642 handle_command_reply: tid=0 2024-04-25T16:02:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.480+0000 7fc4aae30740 1 client.25642 shutdown 2024-04-25T16:02:57.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.483+0000 7fc4aae30740 1 -- 172.21.15.31:0/2041349164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 msgr2=0x55672dbd63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:57.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.483+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbd63e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55672dbc95a0 tx=0x7fc49400f240 comp rx=0 tx=0).stop 2024-04-25T16:02:57.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.484+0000 7fc4aae30740 1 -- 172.21.15.31:0/2041349164 shutdown_connections 2024-04-25T16:02:57.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.484+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55672d7b93a0 0x55672dc4a620 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.484+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55672d825390 0x55672dbd7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.484+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55672d8046b0 0x55672dbd63e0 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x55672dbc95a0 tx=0x7fc49400f240 comp rx=0 tx=0).stop 2024-04-25T16:02:57.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.484+0000 7fc4aae30740 1 --2- 172.21.15.31:0/2041349164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55672d7fc4d0 0x55672dbd5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.484+0000 7fc4aae30740 1 -- 172.21.15.31:0/2041349164 >> 172.21.15.31:0/2041349164 conn(0x55672d8194e0 msgr2=0x55672dbc6850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:57.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.484+0000 7fc4857fa640 0 client.25642 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:57.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.489+0000 7fc4aae30740 1 -- 172.21.15.31:0/2041349164 shutdown_connections 2024-04-25T16:02:57.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.489+0000 7fc4aae30740 1 -- 172.21.15.31:0/2041349164 wait complete. 2024-04-25T16:02:57.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.489+0000 7fc4aae30740 20 client.25642 trim_cache size 0 max 16384 2024-04-25T16:02:57.496 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:02:57.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.497+0000 7fc4a3f85640 1 -- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc4880afc30 msgr2=0x7fc4880b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:57.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.497+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc4880afc30 0x7fc4880b2140 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc48c005fd0 tx=0x7fc48c005850 comp rx=0 tx=0).stop 2024-04-25T16:02:57.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.497+0000 7fc4a3f85640 1 -- 172.21.15.31:0/1138189066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 msgr2=0x7fc49c1184f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:57.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.497+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 0x7fc49c1184f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc498003b90 tx=0x7fc498003bc0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.502+0000 7fc4a3f85640 1 -- 172.21.15.31:0/1138189066 shutdown_connections 2024-04-25T16:02:57.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.502+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc4880afc30 0x7fc4880b2140 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.502+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc49c110030 0x7fc49c1184f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.502+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc49c10b310 0x7fc49c117590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.502+0000 7fc4a3f85640 1 --2- 172.21.15.31:0/1138189066 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc49c10a420 0x7fc49c117050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:57.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.502+0000 7fc4a3f85640 1 -- 172.21.15.31:0/1138189066 >> 172.21.15.31:0/1138189066 conn(0x7fc49c0704e0 msgr2=0x7fc49c10c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:57.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.504+0000 7fc4a3f85640 1 -- 172.21.15.31:0/1138189066 shutdown_connections 2024-04-25T16:02:57.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:57.505+0000 7fc4a3f85640 1 -- 172.21.15.31:0/1138189066 wait complete. 2024-04-25T16:02:58.566 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:02:58.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:02:58.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:02:58.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:58 smithi063 ceph-mon[32413]: pgmap v401: 97 pgs: 97 active+clean; 8.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 742 MiB/s wr, 685 op/s 2024-04-25T16:02:58.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:58 smithi186 ceph-mon[33258]: pgmap v401: 97 pgs: 97 active+clean; 8.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 742 MiB/s wr, 685 op/s 2024-04-25T16:02:58.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:58 smithi031 ceph-mon[29140]: pgmap v401: 97 pgs: 97 active+clean; 8.4 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 742 MiB/s wr, 685 op/s 2024-04-25T16:02:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.960+0000 7f645691f640 1 -- 172.21.15.31:0/2363622027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010b6b0 msgr2=0x7f6450109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.960+0000 7f645691f640 1 --2- 172.21.15.31:0/2363622027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010b6b0 0x7f6450109ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6440009860 tx=0x7f644002a360 comp rx=0 tx=0).stop 2024-04-25T16:02:58.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.963+0000 7f645691f640 1 -- 172.21.15.31:0/2363622027 shutdown_connections 2024-04-25T16:02:58.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.963+0000 7f645691f640 1 --2- 172.21.15.31:0/2363622027 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6450078d00 0x7f645007b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:58.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.963+0000 7f645691f640 1 --2- 172.21.15.31:0/2363622027 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f645010a0e0 0x7f645010a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:58.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.963+0000 7f645691f640 1 --2- 172.21.15.31:0/2363622027 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010b6b0 0x7f6450109ba0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:58.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.963+0000 7f645691f640 1 -- 172.21.15.31:0/2363622027 >> 172.21.15.31:0/2363622027 conn(0x7f64500704e0 msgr2=0x7f6450072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:58.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.970+0000 7f645691f640 1 -- 172.21.15.31:0/2363622027 shutdown_connections 2024-04-25T16:02:58.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.970+0000 7f645691f640 1 -- 172.21.15.31:0/2363622027 wait complete. 2024-04-25T16:02:58.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.971+0000 7f645691f640 1 Processor -- start 2024-04-25T16:02:58.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.972+0000 7f645691f640 1 -- start start 2024-04-25T16:02:58.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.974+0000 7f645691f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6450078d00 0x7f6450084a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:58.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.974+0000 7f645691f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010a0e0 0x7f6450084f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:58.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.974+0000 7f645691f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f645010b6b0 0x7f64501c15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:58.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.974+0000 7f645691f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f645007d920 con 0x7f6450078d00 2024-04-25T16:02:58.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.974+0000 7f645691f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f645007d7a0 con 0x7f645010a0e0 2024-04-25T16:02:58.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.974+0000 7f645691f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f645007daa0 con 0x7f645010b6b0 2024-04-25T16:02:58.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.975+0000 7f644f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010a0e0 0x7f6450084f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.975+0000 7f644f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010a0e0 0x7f6450084f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52904/0 (socket says 172.21.15.31:52904) 2024-04-25T16:02:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.975+0000 7f644f7fe640 1 -- 172.21.15.31:0/3608068916 learned_addr learned my addr 172.21.15.31:0/3608068916 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.976+0000 7f6454ebc640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f645010b6b0 0x7f64501c15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.976+0000 7f644f7fe640 1 -- 172.21.15.31:0/3608068916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f645010b6b0 msgr2=0x7f64501c15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.976+0000 7f644f7fe640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f645010b6b0 0x7f64501c15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.976+0000 7f644f7fe640 1 -- 172.21.15.31:0/3608068916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6450078d00 msgr2=0x7f6450084a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.976+0000 7f644f7fe640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6450078d00 0x7f6450084a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.976+0000 7f644f7fe640 1 -- 172.21.15.31:0/3608068916 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64501c1c40 con 0x7f645010a0e0 2024-04-25T16:02:58.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.979+0000 7f644f7fe640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010a0e0 0x7f6450084f60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f644400b450 tx=0x7f644400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:58.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.987+0000 7f644d7fa640 1 -- 172.21.15.31:0/3608068916 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6444015020 con 0x7f645010a0e0 2024-04-25T16:02:58.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.987+0000 7f644d7fa640 1 -- 172.21.15.31:0/3608068916 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6444011d40 con 0x7f645010a0e0 2024-04-25T16:02:58.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.987+0000 7f644d7fa640 1 -- 172.21.15.31:0/3608068916 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644400f340 con 0x7f645010a0e0 2024-04-25T16:02:58.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.987+0000 7f645691f640 1 -- 172.21.15.31:0/3608068916 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64501c1f30 con 0x7f645010a0e0 2024-04-25T16:02:58.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.987+0000 7f645691f640 1 -- 172.21.15.31:0/3608068916 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64501c2340 con 0x7f645010a0e0 2024-04-25T16:02:58.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.995+0000 7f645d7d2740 1 Processor -- start 2024-04-25T16:02:58.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.997+0000 7f645d7d2740 1 -- start start 2024-04-25T16:02:59.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.998+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6ce410 0x55ef2da6f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.998+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da6fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.999+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da76920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.999+0000 7f645d7d2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ef2d9d4b00 con 0x55ef2d6ad730 2024-04-25T16:02:59.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.999+0000 7f645d7d2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ef2d970920 con 0x55ef2d6ce410 2024-04-25T16:02:59.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:58.999+0000 7f645d7d2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ef2d9d0ac0 con 0x55ef2d6a5550 2024-04-25T16:02:59.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.001+0000 7f644ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da6fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.001+0000 7f6454ebc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6ce410 0x55ef2da6f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.001+0000 7f644ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da6fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47168/0 (socket says 172.21.15.31:47168) 2024-04-25T16:02:59.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.001+0000 7f644ffff640 1 -- 172.21.15.31:0/1611703532 learned_addr learned my addr 172.21.15.31:0/1611703532 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:59.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.007+0000 7f644ffff640 1 -- 172.21.15.31:0/1611703532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6ce410 msgr2=0x55ef2da6f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:59.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.007+0000 7f644ffff640 1 --2- 172.21.15.31:0/1611703532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6ce410 0x55ef2da6f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.007+0000 7f644ffff640 1 -- 172.21.15.31:0/1611703532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 msgr2=0x55ef2da76920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:02:59.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.007+0000 7f644ffff640 1 --2- 172.21.15.31:0/1611703532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da76920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.007+0000 7f644ffff640 1 -- 172.21.15.31:0/1611703532 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef2d7ae7e0 con 0x55ef2d6a5550 2024-04-25T16:02:59.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.010+0000 7f644ffff640 1 --2- 172.21.15.31:0/1611703532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da6fa20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f644002a950 tx=0x7f6440030420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bb4d6ed0c43d2c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.015+0000 7f64367fc640 1 -- 172.21.15.31:0/1611703532 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440037050 con 0x55ef2d6a5550 2024-04-25T16:02:59.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.015+0000 7f64367fc640 1 -- 172.21.15.31:0/1611703532 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6440043070 con 0x55ef2d6a5550 2024-04-25T16:02:59.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.017+0000 7f645d7d2740 1 -- 172.21.15.31:0/1611703532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 msgr2=0x55ef2da6fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:59.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.017+0000 7f645d7d2740 1 --2- 172.21.15.31:0/1611703532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da6fa20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f644002a950 tx=0x7f6440030420 comp rx=0 tx=0).stop 2024-04-25T16:02:59.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.020+0000 7f644d7fa640 1 -- 172.21.15.31:0/3608068916 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6444018020 con 0x7f645010a0e0 2024-04-25T16:02:59.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.021+0000 7f644d7fa640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64380afd00 0x7f64380b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.022+0000 7f644d7fa640 1 -- 172.21.15.31:0/3608068916 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f64440de610 con 0x7f645010a0e0 2024-04-25T16:02:59.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.027+0000 7f6454ebc640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64380afd00 0x7f64380b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.033+0000 7f645d7d2740 1 -- 172.21.15.31:0/1611703532 shutdown_connections 2024-04-25T16:02:59.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.033+0000 7f645d7d2740 1 --2- 172.21.15.31:0/1611703532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da76920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.033+0000 7f645d7d2740 1 --2- 172.21.15.31:0/1611703532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da6fa20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.033+0000 7f645d7d2740 1 --2- 172.21.15.31:0/1611703532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6ce410 0x55ef2da6f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.033+0000 7f645d7d2740 1 -- 172.21.15.31:0/1611703532 >> 172.21.15.31:0/1611703532 conn(0x55ef2d6c2560 msgr2=0x55ef2da6ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.036+0000 7f645d7d2740 1 -- 172.21.15.31:0/1611703532 shutdown_connections 2024-04-25T16:02:59.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.039+0000 7f6454ebc640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64380afd00 0x7f64380b2210 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55ef2da70670 tx=0x7f6448011040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.040+0000 7f645d7d2740 1 -- 172.21.15.31:0/1611703532 wait complete. 2024-04-25T16:02:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.040+0000 7f645d7d2740 1 -- start start 2024-04-25T16:02:59.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.041+0000 7f645d7d2740 1 Processor -- start 2024-04-25T16:02:59.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.042+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da792a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.042+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da797e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.042+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6ce410 0x55ef2da80740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.042+0000 7f645d7d2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ef2da773b0 con 0x55ef2d6ad730 2024-04-25T16:02:59.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.042+0000 7f645d7d2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ef2d96e5d0 con 0x55ef2d6ce410 2024-04-25T16:02:59.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.042+0000 7f645d7d2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ef2da77530 con 0x55ef2d6a5550 2024-04-25T16:02:59.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.042+0000 7f645d7d2740 10 client.0 _ll_register_callbacks cb 0x55ef2d95c080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.042+0000 7f644ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da792a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.043+0000 7f644ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da792a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47184/0 (socket says 172.21.15.31:47184) 2024-04-25T16:02:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.043+0000 7f644ffff640 1 -- 172.21.15.31:0/2702695 learned_addr learned my addr 172.21.15.31:0/2702695 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:59.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.043+0000 7f644f7fe640 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da797e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.044+0000 7f644ffff640 1 -- 172.21.15.31:0/2702695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6ce410 msgr2=0x55ef2da80740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:59.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.044+0000 7f644ffff640 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6ce410 0x55ef2da80740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.044+0000 7f644ffff640 1 -- 172.21.15.31:0/2702695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 msgr2=0x55ef2da797e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:59.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.044+0000 7f644ffff640 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da797e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.044+0000 7f644ffff640 1 -- 172.21.15.31:0/2702695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef2d80ef70 con 0x55ef2d6a5550 2024-04-25T16:02:59.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.045+0000 7f644ffff640 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da792a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64400095f0 tx=0x7f6440030cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de0b9e7f217a839c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:59.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.056+0000 7f642f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:02:59.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.056+0000 7f642f7fe640 1 -- 172.21.15.31:0/2702695 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440003a00 con 0x55ef2d6a5550 2024-04-25T16:02:59.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.056+0000 7f642f7fe640 1 -- 172.21.15.31:0/2702695 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f64400324e0 con 0x55ef2d6a5550 2024-04-25T16:02:59.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.058+0000 7f645d7d2740 1 -- 172.21.15.31:0/2702695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6440043070 con 0x55ef2d6a5550 2024-04-25T16:02:59.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.062+0000 7f642f7fe640 1 -- 172.21.15.31:0/2702695 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6440030e50 con 0x55ef2d6a5550 2024-04-25T16:02:59.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.062+0000 7f645d7d2740 10 client.25648 fetch_fsmap learned FSMap version 28 2024-04-25T16:02:59.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.062+0000 7f645d7d2740 1 -- 172.21.15.31:0/2702695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef2d6a6170 con 0x55ef2d6a5550 2024-04-25T16:02:59.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.066+0000 7f642f7fe640 1 -- 172.21.15.31:0/2702695 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6440043070 con 0x55ef2d6a5550 2024-04-25T16:02:59.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.066+0000 7f642f7fe640 1 -- 172.21.15.31:0/2702695 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f6440050620 con 0x55ef2d6a5550 2024-04-25T16:02:59.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.066+0000 7f645d7d2740 10 client.25648 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:02:59.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.066+0000 7f645d7d2740 10 client.25648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:59.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.066+0000 7f645d7d2740 20 client.25648 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:59.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.067+0000 7f645d7d2740 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ef2d68d1f0 0x55ef2da86d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.067+0000 7f645d7d2740 4 client.25648 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:02:59.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.067+0000 7f645d7d2740 1 -- 172.21.15.31:0/2702695 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef2d9e2710 con 0x55ef2d68d1f0 2024-04-25T16:02:59.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.069+0000 7f644f7fe640 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ef2d68d1f0 0x55ef2da86d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.088+0000 7f644f7fe640 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ef2d68d1f0 0x55ef2da86d00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f24078738449b683 server_cookie=9617a3bafba06a8e in_seq=0 out_seq=0 2024-04-25T16:02:59.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.092+0000 7f642f7fe640 10 client.25648 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:59.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.101+0000 7f642f7fe640 1 -- 172.21.15.31:0/2702695 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ef2d9e2710 con 0x55ef2d68d1f0 2024-04-25T16:02:59.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.101+0000 7f642f7fe640 10 client.25648 handle_command_reply: tid=0 2024-04-25T16:02:59.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.101+0000 7f645d7d2740 1 client.25648 shutdown 2024-04-25T16:02:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.104+0000 7f645d7d2740 1 -- 172.21.15.31:0/2702695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 msgr2=0x55ef2da792a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.104+0000 7f645d7d2740 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da792a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f64400095f0 tx=0x7f6440030cb0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.104+0000 7f645d7d2740 1 -- 172.21.15.31:0/2702695 shutdown_connections 2024-04-25T16:02:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.104+0000 7f645d7d2740 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ef2d68d1f0 0x55ef2da86d00 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.104+0000 7f645d7d2740 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6ce410 0x55ef2da80740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.104+0000 7f645d7d2740 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da797e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.104+0000 7f645d7d2740 1 --2- 172.21.15.31:0/2702695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6a5550 0x55ef2da792a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.104+0000 7f645d7d2740 1 -- 172.21.15.31:0/2702695 >> 172.21.15.31:0/2702695 conn(0x55ef2d6c2560 msgr2=0x55ef2da69ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:59.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.104+0000 7f642f7fe640 0 client.25648 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:59.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.105+0000 7f645d7d2740 1 -- 172.21.15.31:0/2702695 shutdown_connections 2024-04-25T16:02:59.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.109+0000 7f645d7d2740 1 -- 172.21.15.31:0/2702695 wait complete. 2024-04-25T16:02:59.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.109+0000 7f645d7d2740 20 client.25648 trim_cache size 0 max 16384 2024-04-25T16:02:59.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.172+0000 7f645d7d2740 1 Processor -- start 2024-04-25T16:02:59.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.172+0000 7f645d7d2740 1 -- start start 2024-04-25T16:02:59.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.173+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 0x55ef2da73ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.173+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da792b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.173+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ce410 0x55ef2da797f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.173+0000 7f645d7d2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f64501cb160 con 0x55ef2d6ad730 2024-04-25T16:02:59.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.173+0000 7f645d7d2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f64501cccd0 con 0x55ef2d6a5550 2024-04-25T16:02:59.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.173+0000 7f645d7d2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64501cb830 con 0x55ef2d6ce410 2024-04-25T16:02:59.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.174+0000 7f644ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 0x55ef2da73ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.174+0000 7f6454ebc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ce410 0x55ef2da797f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.174+0000 7f6454ebc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ce410 0x55ef2da797f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47188/0 (socket says 172.21.15.31:47188) 2024-04-25T16:02:59.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.174+0000 7f6454ebc640 1 -- 172.21.15.31:0/4087521920 learned_addr learned my addr 172.21.15.31:0/4087521920 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:59.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.174+0000 7f644f7fe640 1 --2- 172.21.15.31:0/4087521920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da792b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.175+0000 7f644f7fe640 1 -- 172.21.15.31:0/4087521920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ce410 msgr2=0x55ef2da797f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:59.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.175+0000 7f644f7fe640 1 --2- 172.21.15.31:0/4087521920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ce410 0x55ef2da797f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.175+0000 7f644f7fe640 1 -- 172.21.15.31:0/4087521920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 msgr2=0x55ef2da73ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:59.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.175+0000 7f644f7fe640 1 --2- 172.21.15.31:0/4087521920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 0x55ef2da73ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.175+0000 7f644f7fe640 1 -- 172.21.15.31:0/4087521920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef2da7a3e0 con 0x55ef2d6ad730 2024-04-25T16:02:59.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.178+0000 7f644f7fe640 1 --2- 172.21.15.31:0/4087521920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da792b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f64440a3d90 tx=0x7f64440c9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd63bf5969dad59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:59.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.182+0000 7f6435ffb640 1 -- 172.21.15.31:0/4087521920 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64440a6b20 con 0x55ef2d6ad730 2024-04-25T16:02:59.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.182+0000 7f6435ffb640 1 -- 172.21.15.31:0/4087521920 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64440a7e50 con 0x55ef2d6ad730 2024-04-25T16:02:59.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.182+0000 7f6435ffb640 1 -- 172.21.15.31:0/4087521920 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644400ed50 con 0x55ef2d6ad730 2024-04-25T16:02:59.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.184+0000 7f645d7d2740 1 -- 172.21.15.31:0/4087521920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 msgr2=0x55ef2da792b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:02:59.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.184+0000 7f645d7d2740 1 --2- 172.21.15.31:0/4087521920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da792b0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f64440a3d90 tx=0x7f64440c9040 comp rx=0 tx=0).stop 2024-04-25T16:02:59.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.189+0000 7f645d7d2740 1 -- 172.21.15.31:0/4087521920 shutdown_connections 2024-04-25T16:02:59.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.189+0000 7f645d7d2740 1 --2- 172.21.15.31:0/4087521920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ce410 0x55ef2da797f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.189+0000 7f645d7d2740 1 --2- 172.21.15.31:0/4087521920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ad730 0x55ef2da792b0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.189+0000 7f645d7d2740 1 --2- 172.21.15.31:0/4087521920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 0x55ef2da73ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.189+0000 7f645d7d2740 1 -- 172.21.15.31:0/4087521920 >> 172.21.15.31:0/4087521920 conn(0x55ef2d6c2560 msgr2=0x55ef2da6e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:02:59.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.190+0000 7f645d7d2740 1 -- 172.21.15.31:0/4087521920 shutdown_connections 2024-04-25T16:02:59.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.191+0000 7f645d7d2740 1 -- 172.21.15.31:0/4087521920 wait complete. 2024-04-25T16:02:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.192+0000 7f645d7d2740 1 -- start start 2024-04-25T16:02:59.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.193+0000 7f645d7d2740 1 Processor -- start 2024-04-25T16:02:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.193+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 0x55ef2da7daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ad730 0x55ef2da7dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f645d7d2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ce410 0x55ef2daed0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f645d7d2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f64501cc5d0 con 0x55ef2d6ce410 2024-04-25T16:02:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f645d7d2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f64501cbf00 con 0x55ef2d6a5550 2024-04-25T16:02:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f645d7d2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64501c9140 con 0x55ef2d6ad730 2024-04-25T16:02:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f645d7d2740 10 client.0 _ll_register_callbacks cb 0x7f6440004430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:02:59.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f644f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ad730 0x55ef2da7dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f644f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ad730 0x55ef2da7dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47190/0 (socket says 172.21.15.31:47190) 2024-04-25T16:02:59.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f644f7fe640 1 -- 172.21.15.31:0/110860794 learned_addr learned my addr 172.21.15.31:0/110860794 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:02:59.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.194+0000 7f6454ebc640 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ce410 0x55ef2daed0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.195+0000 7f644ffff640 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 0x55ef2da7daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.195+0000 7f644ffff640 1 -- 172.21.15.31:0/110860794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ad730 msgr2=0x55ef2da7dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:59.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.195+0000 7f644ffff640 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ad730 0x55ef2da7dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.195+0000 7f644ffff640 1 -- 172.21.15.31:0/110860794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ce410 msgr2=0x55ef2daed0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:02:59.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.195+0000 7f644ffff640 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ce410 0x55ef2daed0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:02:59.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.195+0000 7f644ffff640 1 -- 172.21.15.31:0/110860794 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef2da80fa0 con 0x55ef2d6a5550 2024-04-25T16:02:59.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.196+0000 7f644ffff640 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 0x55ef2da7daa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f644002a950 tx=0x7f644002ab10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43ca1b41045f6fec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:02:59.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.196+0000 7f642f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:02:59.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.197+0000 7f642f7fe640 1 -- 172.21.15.31:0/110860794 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440037040 con 0x55ef2d6a5550 2024-04-25T16:02:59.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.197+0000 7f645d7d2740 1 -- 172.21.15.31:0/110860794 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64440a7e50 con 0x55ef2d6a5550 2024-04-25T16:02:59.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.197+0000 7f642f7fe640 1 -- 172.21.15.31:0/110860794 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6440003a10 con 0x55ef2d6a5550 2024-04-25T16:02:59.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.197+0000 7f642f7fe640 1 -- 172.21.15.31:0/110860794 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f64400044e0 con 0x55ef2d6a5550 2024-04-25T16:02:59.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.197+0000 7f645d7d2740 10 client.25700 fetch_fsmap learned FSMap version 28 2024-04-25T16:02:59.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.198+0000 7f645d7d2740 1 -- 172.21.15.31:0/110860794 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef2daee3f0 con 0x55ef2d6a5550 2024-04-25T16:02:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.199+0000 7f642f7fe640 1 -- 172.21.15.31:0/110860794 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64440a7e50 con 0x55ef2d6a5550 2024-04-25T16:02:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.199+0000 7f642f7fe640 1 -- 172.21.15.31:0/110860794 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f64400329b0 con 0x55ef2d6a5550 2024-04-25T16:02:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.199+0000 7f645d7d2740 10 client.25700 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:02:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.199+0000 7f645d7d2740 10 client.25700 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:02:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.199+0000 7f645d7d2740 20 client.25700 populate_metadata read hostname 'smithi031' 2024-04-25T16:02:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.199+0000 7f645d7d2740 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ef2d662630 0x55ef2daf0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:02:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.199+0000 7f645d7d2740 4 client.25700 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:02:59.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.199+0000 7f645d7d2740 1 -- 172.21.15.31:0/110860794 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ef2d9dc580 con 0x55ef2d662630 2024-04-25T16:02:59.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.200+0000 7f644f7fe640 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ef2d662630 0x55ef2daf0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:02:59.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.213+0000 7f642f7fe640 10 client.25700 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:02:59.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:02:59.213+0000 7f644f7fe640 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ef2d662630 0x55ef2daf0db0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2684f6dd2cfb886 server_cookie=aa465c433990339 in_seq=0 out_seq=0 2024-04-25T16:02:59.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:02:59 smithi063 ceph-mon[32413]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:02:59.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:02:59 smithi186 ceph-mon[33258]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:02:59.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:02:59 smithi031 ceph-mon[29140]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:03:00.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.055+0000 7f642f7fe640 1 -- 172.21.15.31:0/110860794 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ef2d9dc580 con 0x55ef2d662630 2024-04-25T16:03:00.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.055+0000 7f642f7fe640 10 client.25700 handle_command_reply: tid=0 2024-04-25T16:03:00.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.055+0000 7f645d7d2740 1 client.25700 shutdown 2024-04-25T16:03:00.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.061+0000 7f645d7d2740 1 -- 172.21.15.31:0/110860794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 msgr2=0x55ef2da7daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.061+0000 7f645d7d2740 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 0x55ef2da7daa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f644002a950 tx=0x7f644002ab10 comp rx=0 tx=0).stop 2024-04-25T16:03:00.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.061+0000 7f645d7d2740 1 -- 172.21.15.31:0/110860794 shutdown_connections 2024-04-25T16:03:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.061+0000 7f645d7d2740 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ef2d662630 0x55ef2daf0db0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.061+0000 7f645d7d2740 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef2d6ce410 0x55ef2daed0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.061+0000 7f645d7d2740 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef2d6ad730 0x55ef2da7dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.061+0000 7f645d7d2740 1 --2- 172.21.15.31:0/110860794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef2d6a5550 0x55ef2da7daa0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.061+0000 7f645d7d2740 1 -- 172.21.15.31:0/110860794 >> 172.21.15.31:0/110860794 conn(0x55ef2d6c2560 msgr2=0x55ef2da79280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.062+0000 7f642f7fe640 0 client.25700 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:00.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.062+0000 7f645d7d2740 1 -- 172.21.15.31:0/110860794 shutdown_connections 2024-04-25T16:03:00.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.062+0000 7f645d7d2740 1 -- 172.21.15.31:0/110860794 wait complete. 2024-04-25T16:03:00.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.063+0000 7f645d7d2740 20 client.25700 trim_cache size 0 max 16384 2024-04-25T16:03:00.066 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:03:00.066 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:03:00.067 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "58771bbb-8acb-4095-81b7-89fae8ffea0e", 2024-04-25T16:03:00.067 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:03:00.067 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:03:00.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.065+0000 7f645691f640 1 -- 172.21.15.31:0/3608068916 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64380afd00 msgr2=0x7f64380b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.065+0000 7f645691f640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64380afd00 0x7f64380b2210 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55ef2da70670 tx=0x7f6448011040 comp rx=0 tx=0).stop 2024-04-25T16:03:00.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.065+0000 7f645691f640 1 -- 172.21.15.31:0/3608068916 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010a0e0 msgr2=0x7f6450084f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.065+0000 7f645691f640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010a0e0 0x7f6450084f60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f644400b450 tx=0x7f644400b910 comp rx=0 tx=0).stop 2024-04-25T16:03:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.070+0000 7f645691f640 1 -- 172.21.15.31:0/3608068916 shutdown_connections 2024-04-25T16:03:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.070+0000 7f645691f640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f64380afd00 0x7f64380b2210 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.070+0000 7f645691f640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f645010b6b0 0x7f64501c15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.070+0000 7f645691f640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f645010a0e0 0x7f6450084f60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.070+0000 7f645691f640 1 --2- 172.21.15.31:0/3608068916 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6450078d00 0x7f6450084a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.070+0000 7f645691f640 1 -- 172.21.15.31:0/3608068916 >> 172.21.15.31:0/3608068916 conn(0x7f64500704e0 msgr2=0x7f6450072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.070+0000 7f645691f640 1 -- 172.21.15.31:0/3608068916 shutdown_connections 2024-04-25T16:03:00.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.070+0000 7f645691f640 1 -- 172.21.15.31:0/3608068916 wait complete. 2024-04-25T16:03:00.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:03:00.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:03:00.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.595+0000 7ff7da2f2640 1 -- 172.21.15.31:0/647850320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7d41120c0 msgr2=0x7ff7d41144e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.595+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/647850320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7d41120c0 0x7ff7d41144e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009900 tx=0x7ff7cc02a3f0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.598+0000 7ff7da2f2640 1 -- 172.21.15.31:0/647850320 shutdown_connections 2024-04-25T16:03:00.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.598+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/647850320 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7d41120c0 0x7ff7d41144e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.598+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/647850320 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7d410f760 0x7ff7d4111b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.598+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/647850320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d4109fc0 0x7ff7d410a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.598+0000 7ff7da2f2640 1 -- 172.21.15.31:0/647850320 >> 172.21.15.31:0/647850320 conn(0x7ff7d4070400 msgr2=0x7ff7d4072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:00.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.601+0000 7ff7da2f2640 1 -- 172.21.15.31:0/647850320 shutdown_connections 2024-04-25T16:03:00.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.602+0000 7ff7da2f2640 1 -- 172.21.15.31:0/647850320 wait complete. 2024-04-25T16:03:00.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.604+0000 7ff7da2f2640 1 Processor -- start 2024-04-25T16:03:00.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.605+0000 7ff7da2f2640 1 -- start start 2024-04-25T16:03:00.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.607+0000 7ff7da2f2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7d4109fc0 0x7ff7d411cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.608+0000 7ff7da2f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d410f760 0x7ff7d411d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.608+0000 7ff7da2f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7d41120c0 0x7ff7d411b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.608+0000 7ff7da2f2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4115a20 con 0x7ff7d41120c0 2024-04-25T16:03:00.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.609+0000 7ff7da2f2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41158a0 con 0x7ff7d4109fc0 2024-04-25T16:03:00.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.609+0000 7ff7da2f2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4115ba0 con 0x7ff7d410f760 2024-04-25T16:03:00.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.609+0000 7ff7d3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7d4109fc0 0x7ff7d411cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.610+0000 7ff7d3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7d4109fc0 0x7ff7d411cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52950/0 (socket says 172.21.15.31:52950) 2024-04-25T16:03:00.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.610+0000 7ff7d3fff640 1 -- 172.21.15.31:0/4294384835 learned_addr learned my addr 172.21.15.31:0/4294384835 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:00.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.610+0000 7ff7cb7fe640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d410f760 0x7ff7d411d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.611+0000 7ff7d3fff640 1 -- 172.21.15.31:0/4294384835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d410f760 msgr2=0x7ff7d411d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.612+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d410f760 0x7ff7d411d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.612+0000 7ff7d3fff640 1 -- 172.21.15.31:0/4294384835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7d41120c0 msgr2=0x7ff7d411b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.612+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7d41120c0 0x7ff7d411b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.612+0000 7ff7d3fff640 1 -- 172.21.15.31:0/4294384835 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d411bf40 con 0x7ff7d4109fc0 2024-04-25T16:03:00.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.623+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7d4109fc0 0x7ff7d411cfd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400ea50 tx=0x7ff7c400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:00.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.625+0000 7ff7d1ffb640 1 -- 172.21.15.31:0/4294384835 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400cd00 con 0x7ff7d4109fc0 2024-04-25T16:03:00.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.625+0000 7ff7d1ffb640 1 -- 172.21.15.31:0/4294384835 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c4011920 con 0x7ff7d4109fc0 2024-04-25T16:03:00.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.625+0000 7ff7d1ffb640 1 -- 172.21.15.31:0/4294384835 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400cd00 con 0x7ff7d4109fc0 2024-04-25T16:03:00.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.626+0000 7ff7da2f2640 1 -- 172.21.15.31:0/4294384835 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d411c230 con 0x7ff7d4109fc0 2024-04-25T16:03:00.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.626+0000 7ff7da2f2640 1 -- 172.21.15.31:0/4294384835 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d41b99a0 con 0x7ff7d4109fc0 2024-04-25T16:03:00.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.632+0000 7ff7e11a7740 1 Processor -- start 2024-04-25T16:03:00.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.632+0000 7ff7e11a7740 1 -- start start 2024-04-25T16:03:00.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.632+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd90bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.633+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5414d0 0x55b0dd90c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.633+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5496b0 0x55b0dd90ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.633+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b0dd870d80 con 0x55b0dd5414d0 2024-04-25T16:03:00.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.633+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b0dd82a8b0 con 0x55b0dd5496b0 2024-04-25T16:03:00.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.633+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b0dd86cd40 con 0x55b0dd56a390 2024-04-25T16:03:00.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.633+0000 7ff7d888f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd90bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.633+0000 7ff7d888f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd90bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47210/0 (socket says 172.21.15.31:47210) 2024-04-25T16:03:00.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.633+0000 7ff7d888f640 1 -- 172.21.15.31:0/2727127155 learned_addr learned my addr 172.21.15.31:0/2727127155 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:00.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.633+0000 7ff7cb7fe640 1 --2- 172.21.15.31:0/2727127155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5496b0 0x55b0dd90ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.634+0000 7ff7d888f640 1 -- 172.21.15.31:0/2727127155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5496b0 msgr2=0x55b0dd90ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:00.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.634+0000 7ff7d888f640 1 --2- 172.21.15.31:0/2727127155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5496b0 0x55b0dd90ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.634+0000 7ff7d888f640 1 -- 172.21.15.31:0/2727127155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5414d0 msgr2=0x55b0dd90c1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:03:00.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.634+0000 7ff7d888f640 1 --2- 172.21.15.31:0/2727127155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5414d0 0x55b0dd90c1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.634+0000 7ff7d888f640 1 -- 172.21.15.31:0/2727127155 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0dd7fba40 con 0x55b0dd56a390 2024-04-25T16:03:00.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.635+0000 7ff7d888f640 1 --2- 172.21.15.31:0/2727127155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd90bc50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009850 tx=0x7ff7cc0050f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5d79873e358353f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:00.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.635+0000 7ff7ca7fc640 1 -- 172.21.15.31:0/2727127155 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc03e070 con 0x55b0dd56a390 2024-04-25T16:03:00.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.636+0000 7ff7ca7fc640 1 -- 172.21.15.31:0/2727127155 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7cc005520 con 0x55b0dd56a390 2024-04-25T16:03:00.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.636+0000 7ff7d1ffb640 1 -- 172.21.15.31:0/4294384835 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff7c4011470 con 0x7ff7d4109fc0 2024-04-25T16:03:00.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.637+0000 7ff7ca7fc640 1 -- 172.21.15.31:0/2727127155 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc035040 con 0x55b0dd56a390 2024-04-25T16:03:00.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.638+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2727127155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 msgr2=0x55b0dd90bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.638+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2727127155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd90bc50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009850 tx=0x7ff7cc0050f0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.638+0000 7ff7d1ffb640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff7c00aff20 0x7ff7c00b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.639+0000 7ff7d1ffb640 1 -- 172.21.15.31:0/4294384835 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff7c400cb00 con 0x7ff7d4109fc0 2024-04-25T16:03:00.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.639+0000 7ff7d888f640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff7c00aff20 0x7ff7c00b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.639+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2727127155 shutdown_connections 2024-04-25T16:03:00.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.639+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2727127155 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5496b0 0x55b0dd90ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.639+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2727127155 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5414d0 0x55b0dd90c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.639+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2727127155 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd90bc50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.639+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2727127155 >> 172.21.15.31:0/2727127155 conn(0x55b0dd55e4e0 msgr2=0x55b0dd907460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.641+0000 7ff7d888f640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff7c00aff20 0x7ff7c00b2430 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc031500 tx=0x7ff7cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:00.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.642+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2727127155 shutdown_connections 2024-04-25T16:03:00.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.643+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2727127155 wait complete. 2024-04-25T16:03:00.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.644+0000 7ff7e11a7740 1 -- start start 2024-04-25T16:03:00.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.645+0000 7ff7e11a7740 1 Processor -- start 2024-04-25T16:03:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.646+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 0x55b0dd915ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.646+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5496b0 0x55b0dd916020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.646+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.646+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b0dd90d960 con 0x55b0dd56a390 2024-04-25T16:03:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.646+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b0dd7f7d10 con 0x55b0dd5414d0 2024-04-25T16:03:00.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.646+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b0dd90dae0 con 0x55b0dd5496b0 2024-04-25T16:03:00.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.646+0000 7ff7e11a7740 10 client.0 _ll_register_callbacks cb 0x55b0dd824210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.649+0000 7ff7cb7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5496b0 0x55b0dd916020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.650+0000 7ff7d888f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.650+0000 7ff7d888f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53774/0 (socket says 172.21.15.31:53774) 2024-04-25T16:03:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.650+0000 7ff7d888f640 1 -- 172.21.15.31:0/90462340 learned_addr learned my addr 172.21.15.31:0/90462340 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:00.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.650+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 0x55b0dd915ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.657+0000 7ff7d888f640 1 -- 172.21.15.31:0/90462340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5496b0 msgr2=0x55b0dd916020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:00.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.657+0000 7ff7d888f640 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5496b0 0x55b0dd916020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.657+0000 7ff7d888f640 1 -- 172.21.15.31:0/90462340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 msgr2=0x55b0dd915ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:00.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.657+0000 7ff7d888f640 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 0x55b0dd915ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.657+0000 7ff7d888f640 1 -- 172.21.15.31:0/90462340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0dd64c7d0 con 0x55b0dd56a390 2024-04-25T16:03:00.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.658+0000 7ff7d888f640 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91cf80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc040000 tx=0x7ff7cc005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e63400a3299b75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.663+0000 7ff7b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.665+0000 7ff7b37fe640 1 -- 172.21.15.31:0/90462340 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc035030 con 0x55b0dd56a390 2024-04-25T16:03:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.665+0000 7ff7e11a7740 1 -- 172.21.15.31:0/90462340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7cc005520 con 0x55b0dd56a390 2024-04-25T16:03:00.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.665+0000 7ff7b37fe640 1 -- 172.21.15.31:0/90462340 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff7cc005320 con 0x55b0dd56a390 2024-04-25T16:03:00.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.671+0000 7ff7b37fe640 1 -- 172.21.15.31:0/90462340 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff7cc03e040 con 0x55b0dd56a390 2024-04-25T16:03:00.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.671+0000 7ff7e11a7740 10 client.16023 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:00.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.671+0000 7ff7e11a7740 1 -- 172.21.15.31:0/90462340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b0dd5823f0 con 0x55b0dd56a390 2024-04-25T16:03:00.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.672+0000 7ff7b37fe640 1 -- 172.21.15.31:0/90462340 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7cc005520 con 0x55b0dd56a390 2024-04-25T16:03:00.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.672+0000 7ff7b37fe640 1 -- 172.21.15.31:0/90462340 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7ff7cc031890 con 0x55b0dd56a390 2024-04-25T16:03:00.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.672+0000 7ff7e11a7740 10 client.16023 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:00.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.672+0000 7ff7e11a7740 10 client.16023 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:00.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.672+0000 7ff7e11a7740 20 client.16023 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:00.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.672+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b0dd528f60 0x55b0dd923040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.672+0000 7ff7e11a7740 4 client.16023 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:00.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.672+0000 7ff7e11a7740 1 -- 172.21.15.31:0/90462340 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0dd87e990 con 0x55b0dd528f60 2024-04-25T16:03:00.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.674+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b0dd528f60 0x55b0dd923040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.699+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b0dd528f60 0x55b0dd923040 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac6b2534ab051b2 server_cookie=b13c5b471e2a7a83 in_seq=0 out_seq=0 2024-04-25T16:03:00.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.699+0000 7ff7b37fe640 10 client.16023 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:00.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.700+0000 7ff7b37fe640 1 -- 172.21.15.31:0/90462340 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b0dd87e990 con 0x55b0dd528f60 2024-04-25T16:03:00.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.700+0000 7ff7b37fe640 10 client.16023 handle_command_reply: tid=0 2024-04-25T16:03:00.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.700+0000 7ff7e11a7740 1 client.16023 shutdown 2024-04-25T16:03:00.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.701+0000 7ff7e11a7740 1 -- 172.21.15.31:0/90462340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 msgr2=0x55b0dd91cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.701+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91cf80 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc040000 tx=0x7ff7cc005a40 comp rx=0 tx=0).stop 2024-04-25T16:03:00.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.702+0000 7ff7e11a7740 1 -- 172.21.15.31:0/90462340 shutdown_connections 2024-04-25T16:03:00.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.702+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b0dd528f60 0x55b0dd923040 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.702+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91cf80 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.702+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5496b0 0x55b0dd916020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.702+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/90462340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 0x55b0dd915ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.702+0000 7ff7e11a7740 1 -- 172.21.15.31:0/90462340 >> 172.21.15.31:0/90462340 conn(0x55b0dd55e4e0 msgr2=0x55b0dd906420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:00.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.702+0000 7ff7b37fe640 0 client.16023 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:00.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.705+0000 7ff7e11a7740 1 -- 172.21.15.31:0/90462340 shutdown_connections 2024-04-25T16:03:00.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.705+0000 7ff7e11a7740 1 -- 172.21.15.31:0/90462340 wait complete. 2024-04-25T16:03:00.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.705+0000 7ff7e11a7740 20 client.16023 trim_cache size 0 max 16384 2024-04-25T16:03:00.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.852+0000 7ff7e11a7740 1 Processor -- start 2024-04-25T16:03:00.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.852+0000 7ff7e11a7740 1 -- start start 2024-04-25T16:03:00.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.853+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 0x55b0dd906d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.853+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5496b0 0x55b0dd91cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.854+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd9072b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.854+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c2660 con 0x55b0dd5496b0 2024-04-25T16:03:00.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.854+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c41d0 con 0x55b0dd5414d0 2024-04-25T16:03:00.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.854+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c2d30 con 0x55b0dd56a390 2024-04-25T16:03:00.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.855+0000 7ff7cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5496b0 0x55b0dd91cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.855+0000 7ff7cb7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5496b0 0x55b0dd91cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53790/0 (socket says 172.21.15.31:53790) 2024-04-25T16:03:00.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.855+0000 7ff7cb7fe640 1 -- 172.21.15.31:0/1732184633 learned_addr learned my addr 172.21.15.31:0/1732184633 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:00.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.855+0000 7ff7d888f640 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd9072b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.855+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 0x55b0dd906d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.856+0000 7ff7cb7fe640 1 -- 172.21.15.31:0/1732184633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 msgr2=0x55b0dd9072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:00.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.856+0000 7ff7cb7fe640 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd9072b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.856+0000 7ff7cb7fe640 1 -- 172.21.15.31:0/1732184633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 msgr2=0x55b0dd906d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:00.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.856+0000 7ff7cb7fe640 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 0x55b0dd906d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.856+0000 7ff7cb7fe640 1 -- 172.21.15.31:0/1732184633 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0dd5420f0 con 0x55b0dd5496b0 2024-04-25T16:03:00.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.857+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 0x55b0dd906d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:03:00.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.858+0000 7ff7cb7fe640 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5496b0 0x55b0dd91cea0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00ae40 tx=0x7ff7bc00a2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0380bdd1e4846d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:00.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.859+0000 7ff7c9ffb640 1 -- 172.21.15.31:0/1732184633 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc01e030 con 0x55b0dd5496b0 2024-04-25T16:03:00.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.860+0000 7ff7c9ffb640 1 -- 172.21.15.31:0/1732184633 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7bc0054d0 con 0x55b0dd5496b0 2024-04-25T16:03:00.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.862+0000 7ff7e11a7740 1 -- 172.21.15.31:0/1732184633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5496b0 msgr2=0x55b0dd91cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.862+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5496b0 0x55b0dd91cea0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc00ae40 tx=0x7ff7bc00a2d0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.863+0000 7ff7e11a7740 1 -- 172.21.15.31:0/1732184633 shutdown_connections 2024-04-25T16:03:00.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.863+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd56a390 0x55b0dd9072b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.863+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd5496b0 0x55b0dd91cea0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.863+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/1732184633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd5414d0 0x55b0dd906d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.863+0000 7ff7e11a7740 1 -- 172.21.15.31:0/1732184633 >> 172.21.15.31:0/1732184633 conn(0x55b0dd55e4e0 msgr2=0x55b0dd9107b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:00.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.864+0000 7ff7e11a7740 1 -- 172.21.15.31:0/1732184633 shutdown_connections 2024-04-25T16:03:00.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.864+0000 7ff7e11a7740 1 -- 172.21.15.31:0/1732184633 wait complete. 2024-04-25T16:03:00.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.865+0000 7ff7e11a7740 1 -- start start 2024-04-25T16:03:00.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.866+0000 7ff7e11a7740 1 Processor -- start 2024-04-25T16:03:00.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.867+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5414d0 0x55b0dd91a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:00 smithi063 ceph-mon[32413]: pgmap v402: 97 pgs: 97 active+clean; 9.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 845 MiB/s wr, 1.00k op/s 2024-04-25T16:03:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:00 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:03:00.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:00 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:00.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:00 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:03:00.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.867+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.868+0000 7ff7d3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5414d0 0x55b0dd91a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.868+0000 7ff7e11a7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd4fe3a0 0x55b0dd91b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.868+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c3ad0 con 0x55b0dd56a390 2024-04-25T16:03:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.868+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c3400 con 0x55b0dd4fe3a0 2024-04-25T16:03:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.868+0000 7ff7e11a7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d41c0640 con 0x55b0dd5414d0 2024-04-25T16:03:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.868+0000 7ff7d3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5414d0 0x55b0dd91a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47236/0 (socket says 172.21.15.31:47236) 2024-04-25T16:03:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.868+0000 7ff7d3fff640 1 -- 172.21.15.31:0/2449797546 learned_addr learned my addr 172.21.15.31:0/2449797546 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:00.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.869+0000 7ff7e11a7740 10 client.0 _ll_register_callbacks cb 0x7ff7bc00a630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:00.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.874+0000 7ff7cb7fe640 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.874+0000 7ff7d888f640 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd4fe3a0 0x55b0dd91b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.875+0000 7ff7d3fff640 1 -- 172.21.15.31:0/2449797546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd4fe3a0 msgr2=0x55b0dd91b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:00.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.875+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd4fe3a0 0x55b0dd91b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.875+0000 7ff7d3fff640 1 -- 172.21.15.31:0/2449797546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 msgr2=0x55b0dd91a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:00.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.875+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.875+0000 7ff7d3fff640 1 -- 172.21.15.31:0/2449797546 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0dd8314b0 con 0x55b0dd5414d0 2024-04-25T16:03:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.878+0000 7ff7d3fff640 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5414d0 0x55b0dd91a310 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400c640 tx=0x7ff7c40a2930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b09d79d322def258 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.881+0000 7ff7b37fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.881+0000 7ff7b37fe640 1 -- 172.21.15.31:0/2449797546 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c40aff40 con 0x55b0dd5414d0 2024-04-25T16:03:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.881+0000 7ff7b37fe640 1 -- 172.21.15.31:0/2449797546 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff7c4011d80 con 0x55b0dd5414d0 2024-04-25T16:03:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.883+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2449797546 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b0dd9102c0 con 0x55b0dd5414d0 2024-04-25T16:03:00.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.887+0000 7ff7b37fe640 1 -- 172.21.15.31:0/2449797546 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff7c4021dd0 con 0x55b0dd5414d0 2024-04-25T16:03:00.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.888+0000 7ff7e11a7740 10 client.25672 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:00.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.888+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2449797546 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b0dd91ced0 con 0x55b0dd5414d0 2024-04-25T16:03:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.890+0000 7ff7b37fe640 1 -- 172.21.15.31:0/2449797546 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7c4011ad0 con 0x55b0dd5414d0 2024-04-25T16:03:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.890+0000 7ff7b37fe640 1 -- 172.21.15.31:0/2449797546 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7ff7c4022030 con 0x55b0dd5414d0 2024-04-25T16:03:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.890+0000 7ff7e11a7740 10 client.25672 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.890+0000 7ff7e11a7740 10 client.25672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.890+0000 7ff7e11a7740 20 client.25672 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.890+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b0dd98b7b0 0x55b0dd98dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.891+0000 7ff7e11a7740 4 client.25672 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:03:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.891+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2449797546 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b0dd9102c0 con 0x55b0dd98b7b0 2024-04-25T16:03:00.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.892+0000 7ff7cb7fe640 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b0dd98b7b0 0x55b0dd98dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:00 smithi186 ceph-mon[33258]: pgmap v402: 97 pgs: 97 active+clean; 9.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 845 MiB/s wr, 1.00k op/s 2024-04-25T16:03:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:00 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:03:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:00 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:00.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:00 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:03:00.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.907+0000 7ff7cb7fe640 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b0dd98b7b0 0x55b0dd98dbd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea7b24b2cb63b5c0 server_cookie=b2085ea6bedd9515 in_seq=0 out_seq=0 2024-04-25T16:03:00.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.907+0000 7ff7b37fe640 10 client.25672 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.910+0000 7ff7b37fe640 1 -- 172.21.15.31:0/2449797546 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55b0dd9102c0 con 0x55b0dd98b7b0 2024-04-25T16:03:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.910+0000 7ff7b37fe640 10 client.25672 handle_command_reply: tid=0 2024-04-25T16:03:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.911+0000 7ff7e11a7740 1 client.25672 shutdown 2024-04-25T16:03:00.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.913+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2449797546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5414d0 msgr2=0x55b0dd91a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.913+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5414d0 0x55b0dd91a310 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400c640 tx=0x7ff7c40a2930 comp rx=0 tx=0).stop 2024-04-25T16:03:00.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.915+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2449797546 shutdown_connections 2024-04-25T16:03:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.915+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b0dd98b7b0 0x55b0dd98dbd0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.915+0000 7ff7b37fe640 0 client.25672 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.915+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b0dd4fe3a0 0x55b0dd91b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.915+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b0dd56a390 0x55b0dd91a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.915+0000 7ff7e11a7740 1 --2- 172.21.15.31:0/2449797546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b0dd5414d0 0x55b0dd91a310 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.915+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2449797546 >> 172.21.15.31:0/2449797546 conn(0x55b0dd55e4e0 msgr2=0x55b0dd906d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.918+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2449797546 shutdown_connections 2024-04-25T16:03:00.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.920+0000 7ff7e11a7740 1 -- 172.21.15.31:0/2449797546 wait complete. 2024-04-25T16:03:00.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.920+0000 7ff7e11a7740 20 client.25672 trim_cache size 0 max 16384 2024-04-25T16:03:00.924 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:03:00.924 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T16:03:00.924 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:03:00.924 INFO:teuthology.orchestra.run.smithi031.stdout: "58771bbb-8acb-4095-81b7-89fae8ffea0e": { 2024-04-25T16:03:00.924 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:03:00.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "58771bbb-8acb-4095-81b7-89fae8ffea0e", 2024-04-25T16:03:00.925 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:03:00.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:03:00.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:03:00.925 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:03:00.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.928+0000 7ff7da2f2640 1 -- 172.21.15.31:0/4294384835 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff7c00aff20 msgr2=0x7ff7c00b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.928+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff7c00aff20 0x7ff7c00b2430 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc031500 tx=0x7ff7cc009260 comp rx=0 tx=0).stop 2024-04-25T16:03:00.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.928+0000 7ff7da2f2640 1 -- 172.21.15.31:0/4294384835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7d4109fc0 msgr2=0x7ff7d411cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:00.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.928+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7d4109fc0 0x7ff7d411cfd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400ea50 tx=0x7ff7c400ee50 comp rx=0 tx=0).stop 2024-04-25T16:03:00.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.931+0000 7ff7da2f2640 1 -- 172.21.15.31:0/4294384835 shutdown_connections 2024-04-25T16:03:00.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.931+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff7c00aff20 0x7ff7c00b2430 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.931+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff7d41120c0 0x7ff7d411b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.931+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff7d410f760 0x7ff7d411d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.931+0000 7ff7da2f2640 1 --2- 172.21.15.31:0/4294384835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7d4109fc0 0x7ff7d411cfd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:00.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.932+0000 7ff7da2f2640 1 -- 172.21.15.31:0/4294384835 >> 172.21.15.31:0/4294384835 conn(0x7ff7d4070400 msgr2=0x7ff7d41105a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:00.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.937+0000 7ff7da2f2640 1 -- 172.21.15.31:0/4294384835 shutdown_connections 2024-04-25T16:03:00.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:00.937+0000 7ff7da2f2640 1 -- 172.21.15.31:0/4294384835 wait complete. 2024-04-25T16:03:00.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:00 smithi031 ceph-mon[29140]: pgmap v402: 97 pgs: 97 active+clean; 9.6 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 845 MiB/s wr, 1.00k op/s 2024-04-25T16:03:00.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:00 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:03:00.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:00 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:00.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:00 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:03:00.981 INFO:tasks.cephfs.filesystem:scrub status for tag:58771bbb-8acb-4095-81b7-89fae8ffea0e - {'path': '/', 'tag': '58771bbb-8acb-4095-81b7-89fae8ffea0e', 'options': 'recursive,force'} 2024-04-25T16:03:01.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:01 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:03:01.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:01 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:01.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:01 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.891+0000 7fbff1ca6640 1 -- 172.21.15.31:0/2143406879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfec10b190 msgr2=0x7fbfec10b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.891+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/2143406879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfec10b190 0x7fbfec10b5a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fbfe401acb0 tx=0x7fbfe403b8e0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.892+0000 7fbff1ca6640 1 -- 172.21.15.31:0/2143406879 shutdown_connections 2024-04-25T16:03:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.892+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/2143406879 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfec10feb0 0x7fbfec1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.892+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/2143406879 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfec10b190 0x7fbfec10b5a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.892+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/2143406879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfec10a340 0x7fbfec10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.892+0000 7fbff1ca6640 1 -- 172.21.15.31:0/2143406879 >> 172.21.15.31:0/2143406879 conn(0x7fbfec070330 msgr2=0x7fbfec0727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:01 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:03:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:01 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:01.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:01 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.898+0000 7fbff1ca6640 1 -- 172.21.15.31:0/2143406879 shutdown_connections 2024-04-25T16:03:01.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.898+0000 7fbff1ca6640 1 -- 172.21.15.31:0/2143406879 wait complete. 2024-04-25T16:03:01.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.900+0000 7fbff1ca6640 1 Processor -- start 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.900+0000 7fbff1ca6640 1 -- start start 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.900+0000 7fbff1ca6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfec10a340 0x7fbfec1a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.900+0000 7fbff1ca6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfec10b190 0x7fbfec1a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.900+0000 7fbff1ca6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfec10feb0 0x7fbfec1a7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.900+0000 7fbff1ca6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbfec113a60 con 0x7fbfec10b190 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.900+0000 7fbff1ca6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbfec1138e0 con 0x7fbfec10feb0 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.900+0000 7fbff1ca6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfec113be0 con 0x7fbfec10a340 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.901+0000 7fbfebfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfec10b190 0x7fbfec1a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.901+0000 7fbfebfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfec10b190 0x7fbfec1a0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53826/0 (socket says 172.21.15.31:53826) 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.901+0000 7fbfebfff640 1 -- 172.21.15.31:0/1816325304 learned_addr learned my addr 172.21.15.31:0/1816325304 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.902+0000 7fbff14a5640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfec10feb0 0x7fbfec1a7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.902+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfec10a340 0x7fbfec1a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.903+0000 7fbfebfff640 1 -- 172.21.15.31:0/1816325304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfec10a340 msgr2=0x7fbfec1a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.903+0000 7fbfebfff640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfec10a340 0x7fbfec1a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.903+0000 7fbfebfff640 1 -- 172.21.15.31:0/1816325304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfec10feb0 msgr2=0x7fbfec1a7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.903+0000 7fbfebfff640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfec10feb0 0x7fbfec1a7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.903+0000 7fbfebfff640 1 -- 172.21.15.31:0/1816325304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfec1a80e0 con 0x7fbfec10b190 2024-04-25T16:03:01.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.906+0000 7fbfebfff640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfec10b190 0x7fbfec1a0ae0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40427f0 tx=0x7fbfe4003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:01.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.910+0000 7fbfe9ffb640 1 -- 172.21.15.31:0/1816325304 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe4004810 con 0x7fbfec10b190 2024-04-25T16:03:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.911+0000 7fbff1ca6640 1 -- 172.21.15.31:0/1816325304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfec1a8370 con 0x7fbfec10b190 2024-04-25T16:03:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.911+0000 7fbff1ca6640 1 -- 172.21.15.31:0/1816325304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfec1a8860 con 0x7fbfec10b190 2024-04-25T16:03:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.911+0000 7fbfe9ffb640 1 -- 172.21.15.31:0/1816325304 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfe4010070 con 0x7fbfec10b190 2024-04-25T16:03:01.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.911+0000 7fbfe9ffb640 1 -- 172.21.15.31:0/1816325304 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400b8a0 con 0x7fbfec10b190 2024-04-25T16:03:01.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.916+0000 7fbff9db7740 1 Processor -- start 2024-04-25T16:03:01.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.917+0000 7fbff9db7740 1 -- start start 2024-04-25T16:03:01.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.917+0000 7fbfe9ffb640 1 -- 172.21.15.31:0/1816325304 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbfe400ba40 con 0x7fbfec10b190 2024-04-25T16:03:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a51b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a520a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96904d0 0x5654f9a58fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff9db7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5654f99b7460 con 0x5654f96904d0 2024-04-25T16:03:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff9db7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5654f9971590 con 0x5654f96882f0 2024-04-25T16:03:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff9db7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5654f99b3420 con 0x5654f96b11b0 2024-04-25T16:03:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff14a5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a51b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff14a5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a51b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47272/0 (socket says 172.21.15.31:47272) 2024-04-25T16:03:01.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff14a5640 1 -- 172.21.15.31:0/1629059334 learned_addr learned my addr 172.21.15.31:0/1629059334 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:01.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.918+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/1629059334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.919+0000 7fbff14a5640 1 -- 172.21.15.31:0/1629059334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 msgr2=0x5654f9a520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:01.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.919+0000 7fbff14a5640 1 --2- 172.21.15.31:0/1629059334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a520a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.919+0000 7fbff14a5640 1 -- 172.21.15.31:0/1629059334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96904d0 msgr2=0x5654f9a58fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:01.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.919+0000 7fbff14a5640 1 --2- 172.21.15.31:0/1629059334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96904d0 0x5654f9a58fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.919+0000 7fbff14a5640 1 -- 172.21.15.31:0/1629059334 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654f997e0d0 con 0x5654f96b11b0 2024-04-25T16:03:01.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.921+0000 7fbff14a5640 1 --2- 172.21.15.31:0/1629059334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a51b20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00bc70 tx=0x7fbfdc00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee48094e3d996119 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:01.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.921+0000 7fbfe9ffb640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfd40afc70 0x7fbfd40b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.922+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfd40afc70 0x7fbfd40b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.922+0000 7fbfd2ffd640 1 -- 172.21.15.31:0/1629059334 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc015070 con 0x5654f96b11b0 2024-04-25T16:03:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.922+0000 7fbfd2ffd640 1 -- 172.21.15.31:0/1629059334 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfdc01a910 con 0x5654f96b11b0 2024-04-25T16:03:01.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.922+0000 7fbfe9ffb640 1 -- 172.21.15.31:0/1816325304 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbfe41222b0 con 0x7fbfec10b190 2024-04-25T16:03:01.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.923+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfd40afc70 0x7fbfd40b2180 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005fd0 tx=0x7fbfe0005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:01.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.923+0000 7fbfd2ffd640 1 -- 172.21.15.31:0/1629059334 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0118e0 con 0x5654f96b11b0 2024-04-25T16:03:01.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.925+0000 7fbff9db7740 1 -- 172.21.15.31:0/1629059334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 msgr2=0x5654f9a51b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:01.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.925+0000 7fbff9db7740 1 --2- 172.21.15.31:0/1629059334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a51b20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00bc70 tx=0x7fbfdc00f730 comp rx=0 tx=0).stop 2024-04-25T16:03:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.929+0000 7fbff9db7740 1 -- 172.21.15.31:0/1629059334 shutdown_connections 2024-04-25T16:03:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.929+0000 7fbff9db7740 1 --2- 172.21.15.31:0/1629059334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96904d0 0x5654f9a58fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.929+0000 7fbff9db7740 1 --2- 172.21.15.31:0/1629059334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.929+0000 7fbff9db7740 1 --2- 172.21.15.31:0/1629059334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a51b20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.929+0000 7fbff9db7740 1 -- 172.21.15.31:0/1629059334 >> 172.21.15.31:0/1629059334 conn(0x5654f96a5300 msgr2=0x5654f9a4d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:01.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.942+0000 7fbff9db7740 1 -- 172.21.15.31:0/1629059334 shutdown_connections 2024-04-25T16:03:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.945+0000 7fbff9db7740 1 -- 172.21.15.31:0/1629059334 wait complete. 2024-04-25T16:03:01.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.945+0000 7fbff9db7740 1 -- start start 2024-04-25T16:03:01.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.946+0000 7fbff9db7740 1 Processor -- start 2024-04-25T16:03:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.946+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a5ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.946+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96904d0 0x5654f9a5bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.946+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 0x5654f9a62ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.946+0000 7fbff9db7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5654f9a59a40 con 0x5654f96b11b0 2024-04-25T16:03:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.946+0000 7fbff9db7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5654f993eb60 con 0x5654f96904d0 2024-04-25T16:03:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.946+0000 7fbff9db7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5654f9a59bc0 con 0x5654f96882f0 2024-04-25T16:03:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.947+0000 7fbff9db7740 10 client.0 _ll_register_callbacks cb 0x5654f99632b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:01.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.947+0000 7fbff14a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 0x5654f9a62ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.947+0000 7fbff14a5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 0x5654f9a62ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53844/0 (socket says 172.21.15.31:53844) 2024-04-25T16:03:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.947+0000 7fbff14a5640 1 -- 172.21.15.31:0/3214669108 learned_addr learned my addr 172.21.15.31:0/3214669108 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:01.950 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.947+0000 7fbfebfff640 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96904d0 0x5654f9a5bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:01 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:03:01.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:01 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:01.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:01 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.948+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a5ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.948+0000 7fbff14a5640 1 -- 172.21.15.31:0/3214669108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 msgr2=0x5654f9a5ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:01.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.948+0000 7fbff14a5640 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a5ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.948+0000 7fbff14a5640 1 -- 172.21.15.31:0/3214669108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96904d0 msgr2=0x5654f9a5bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.948+0000 7fbff14a5640 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96904d0 0x5654f9a5bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.948+0000 7fbff14a5640 1 -- 172.21.15.31:0/3214669108 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654f96c9210 con 0x5654f96b11b0 2024-04-25T16:03:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.948+0000 7fbff14a5640 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 0x5654f9a62ec0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc01a690 tx=0x7fbfdc01a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa417611565ae865 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.948+0000 7fbfcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.949+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3214669108 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc015070 con 0x5654f96b11b0 2024-04-25T16:03:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.949+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3214669108 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbfdc01acd0 con 0x5654f96b11b0 2024-04-25T16:03:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.949+0000 7fbff9db7740 1 -- 172.21.15.31:0/3214669108 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfdc01a910 con 0x5654f96b11b0 2024-04-25T16:03:01.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.950+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3214669108 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbfdc005040 con 0x5654f96b11b0 2024-04-25T16:03:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.950+0000 7fbff9db7740 10 client.16038 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.950+0000 7fbff9db7740 1 -- 172.21.15.31:0/3214669108 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5654f9795620 con 0x5654f96b11b0 2024-04-25T16:03:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.951+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3214669108 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfdc01a910 con 0x5654f96b11b0 2024-04-25T16:03:01.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.951+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3214669108 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fbfdc011660 con 0x5654f96b11b0 2024-04-25T16:03:01.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.952+0000 7fbff9db7740 10 client.16038 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:01.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.952+0000 7fbff9db7740 10 client.16038 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:01.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.952+0000 7fbff9db7740 20 client.16038 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:01.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.952+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5654f966fd80 0x5654f9a693d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:01.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.952+0000 7fbff9db7740 4 client.16038 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:01.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.952+0000 7fbff9db7740 1 -- 172.21.15.31:0/3214669108 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654f99c5070 con 0x5654f966fd80 2024-04-25T16:03:01.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.953+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5654f966fd80 0x5654f9a693d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.956+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5654f966fd80 0x5654f9a693d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d07f01a542a4cc6 server_cookie=db75e24dc2e4c626 in_seq=0 out_seq=0 2024-04-25T16:03:01.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.956+0000 7fbfcbfff640 10 client.16038 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.961+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3214669108 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5654f99c5070 con 0x5654f966fd80 2024-04-25T16:03:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.961+0000 7fbfcbfff640 10 client.16038 handle_command_reply: tid=0 2024-04-25T16:03:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.961+0000 7fbff9db7740 1 client.16038 shutdown 2024-04-25T16:03:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.961+0000 7fbff9db7740 1 -- 172.21.15.31:0/3214669108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 msgr2=0x5654f9a62ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.961+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 0x5654f9a62ec0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc01a690 tx=0x7fbfdc01a6c0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.962+0000 7fbff9db7740 1 -- 172.21.15.31:0/3214669108 shutdown_connections 2024-04-25T16:03:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.962+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5654f966fd80 0x5654f9a693d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.962+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 0x5654f9a62ec0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.962+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96904d0 0x5654f9a5bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.962+0000 7fbfcbfff640 0 client.16038 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.962+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3214669108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a5ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.962+0000 7fbff9db7740 1 -- 172.21.15.31:0/3214669108 >> 172.21.15.31:0/3214669108 conn(0x5654f96a5300 msgr2=0x5654f9a503c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.962+0000 7fbff9db7740 1 -- 172.21.15.31:0/3214669108 shutdown_connections 2024-04-25T16:03:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.962+0000 7fbff9db7740 1 -- 172.21.15.31:0/3214669108 wait complete. 2024-04-25T16:03:01.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:01.965+0000 7fbff9db7740 20 client.16038 trim_cache size 0 max 16384 2024-04-25T16:03:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.020+0000 7fbff9db7740 1 Processor -- start 2024-04-25T16:03:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.020+0000 7fbff9db7740 1 -- start start 2024-04-25T16:03:02.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.020+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a4cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.021+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96904d0 0x5654f9a59ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.021+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 0x5654f9a5a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.021+0000 7fbff9db7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd800e470 con 0x5654f96b11b0 2024-04-25T16:03:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.021+0000 7fbff9db7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd800ffe0 con 0x5654f96904d0 2024-04-25T16:03:02.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.021+0000 7fbff0ca4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a4cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.021+0000 7fbff9db7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd800eb40 con 0x5654f96882f0 2024-04-25T16:03:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.021+0000 7fbff0ca4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a4cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47276/0 (socket says 172.21.15.31:47276) 2024-04-25T16:03:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.021+0000 7fbff0ca4640 1 -- 172.21.15.31:0/4032736055 learned_addr learned my addr 172.21.15.31:0/4032736055 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.022+0000 7fbff0ca4640 1 -- 172.21.15.31:0/4032736055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96904d0 msgr2=0x5654f9a59ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.022+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/4032736055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96904d0 0x5654f9a59ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.022+0000 7fbff0ca4640 1 -- 172.21.15.31:0/4032736055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 msgr2=0x5654f9a5a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:02.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.022+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/4032736055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 0x5654f9a5a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:02.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.022+0000 7fbff0ca4640 1 -- 172.21.15.31:0/4032736055 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654f9a63300 con 0x5654f96882f0 2024-04-25T16:03:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.024+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/4032736055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a4cd30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5654f9a52f00 tx=0x7fbfe0015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61ae85108a8e254 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:02.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.025+0000 7fbfd27fc640 1 -- 172.21.15.31:0/4032736055 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe002b070 con 0x5654f96882f0 2024-04-25T16:03:02.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.025+0000 7fbfd27fc640 1 -- 172.21.15.31:0/4032736055 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfe0026780 con 0x5654f96882f0 2024-04-25T16:03:02.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.026+0000 7fbff9db7740 1 -- 172.21.15.31:0/4032736055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 msgr2=0x5654f9a4cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:02.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.026+0000 7fbff9db7740 1 --2- 172.21.15.31:0/4032736055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a4cd30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x5654f9a52f00 tx=0x7fbfe0015000 comp rx=0 tx=0).stop 2024-04-25T16:03:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.028+0000 7fbff9db7740 1 -- 172.21.15.31:0/4032736055 shutdown_connections 2024-04-25T16:03:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.028+0000 7fbff9db7740 1 --2- 172.21.15.31:0/4032736055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96b11b0 0x5654f9a5a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.028+0000 7fbff9db7740 1 --2- 172.21.15.31:0/4032736055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96904d0 0x5654f9a59ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.028+0000 7fbff9db7740 1 --2- 172.21.15.31:0/4032736055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96882f0 0x5654f9a4cd30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:02.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.028+0000 7fbff9db7740 1 -- 172.21.15.31:0/4032736055 >> 172.21.15.31:0/4032736055 conn(0x5654f96a5300 msgr2=0x5654f9a55dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:02.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.042+0000 7fbff9db7740 1 -- 172.21.15.31:0/4032736055 shutdown_connections 2024-04-25T16:03:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.049+0000 7fbff9db7740 1 -- 172.21.15.31:0/4032736055 wait complete. 2024-04-25T16:03:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.050+0000 7fbff9db7740 1 -- start start 2024-04-25T16:03:02.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.050+0000 7fbff9db7740 1 Processor -- start 2024-04-25T16:03:02.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a5f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:02.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a5fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:02.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff9db7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96451c0 0x5654f9acf5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:02.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff9db7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd800f8e0 con 0x5654f96451c0 2024-04-25T16:03:02.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff9db7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd800f210 con 0x5654f96882f0 2024-04-25T16:03:02.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff9db7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd800c450 con 0x5654f96b11b0 2024-04-25T16:03:02.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff9db7740 10 client.0 _ll_register_callbacks cb 0x7fbfdc00bac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff0ca4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a5f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbfebfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a5fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff0ca4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a5f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53028/0 (socket says 172.21.15.31:53028) 2024-04-25T16:03:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.052+0000 7fbff0ca4640 1 -- 172.21.15.31:0/3933593913 learned_addr learned my addr 172.21.15.31:0/3933593913 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.053+0000 7fbff0ca4640 1 -- 172.21.15.31:0/3933593913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 msgr2=0x5654f9a5fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.053+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a5fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.053+0000 7fbff0ca4640 1 -- 172.21.15.31:0/3933593913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96451c0 msgr2=0x5654f9acf5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.053+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96451c0 0x5654f9acf5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:02.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.053+0000 7fbff0ca4640 1 -- 172.21.15.31:0/3933593913 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654f9acfb40 con 0x5654f96882f0 2024-04-25T16:03:02.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.054+0000 7fbff0ca4640 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a5f5c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0000c00 tx=0x7fbfe001e6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36e9df100cc06640 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.063+0000 7fbfcbfff640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:03:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.063+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3933593913 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe002b070 con 0x5654f96882f0 2024-04-25T16:03:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.063+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3933593913 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbfe0005440 con 0x5654f96882f0 2024-04-25T16:03:02.065 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.064+0000 7fbff9db7740 1 -- 172.21.15.31:0/3933593913 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5654f99beee0 con 0x5654f96882f0 2024-04-25T16:03:02.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.065+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3933593913 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbfe0004660 con 0x5654f96882f0 2024-04-25T16:03:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.066+0000 7fbff9db7740 10 client.25724 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:02.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.066+0000 7fbff9db7740 1 -- 172.21.15.31:0/3933593913 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5654f9ad0eb0 con 0x5654f96882f0 2024-04-25T16:03:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.067+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3933593913 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5654f99beee0 con 0x5654f96882f0 2024-04-25T16:03:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.067+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3933593913 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fbfe0026680 con 0x5654f96882f0 2024-04-25T16:03:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.067+0000 7fbff9db7740 10 client.25724 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.067+0000 7fbff9db7740 10 client.25724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.067+0000 7fbff9db7740 20 client.25724 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:02.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.067+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5654f9ad1810 0x5654f9ad3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.067+0000 7fbff9db7740 4 client.25724 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:03:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.067+0000 7fbff9db7740 1 -- 172.21.15.31:0/3933593913 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5654f993a5f0 con 0x5654f9ad1810 2024-04-25T16:03:02.070 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.069+0000 7fbfebfff640 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5654f9ad1810 0x5654f9ad3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:02.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.075+0000 7fbfcbfff640 10 client.25724 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:02.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:02.075+0000 7fbfebfff640 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5654f9ad1810 0x5654f9ad3c30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b484ec17333f0ba7 server_cookie=4a24f0fc55ffef7e in_seq=0 out_seq=0 2024-04-25T16:03:02.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:02 smithi063 ceph-mon[32413]: pgmap v403: 97 pgs: 97 active+clean; 9.7 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 776 MiB/s wr, 1.09k op/s 2024-04-25T16:03:02.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:02 smithi186 ceph-mon[33258]: pgmap v403: 97 pgs: 97 active+clean; 9.7 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 776 MiB/s wr, 1.09k op/s 2024-04-25T16:03:02.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:02 smithi031 ceph-mon[29140]: pgmap v403: 97 pgs: 97 active+clean; 9.7 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 776 MiB/s wr, 1.09k op/s 2024-04-25T16:03:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.781+0000 7fbfcbfff640 1 -- 172.21.15.31:0/3933593913 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5654f993a5f0 con 0x5654f9ad1810 2024-04-25T16:03:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.781+0000 7fbfcbfff640 10 client.25724 handle_command_reply: tid=0 2024-04-25T16:03:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.781+0000 7fbff9db7740 1 client.25724 shutdown 2024-04-25T16:03:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.782+0000 7fbff9db7740 1 -- 172.21.15.31:0/3933593913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 msgr2=0x5654f9a5f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:04.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.782+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a5f5c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0000c00 tx=0x7fbfe001e6d0 comp rx=0 tx=0).stop 2024-04-25T16:03:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.782+0000 7fbff9db7740 1 -- 172.21.15.31:0/3933593913 shutdown_connections 2024-04-25T16:03:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.783+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5654f9ad1810 0x5654f9ad3c30 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.783+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654f96451c0 0x5654f9acf5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.783+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654f96b11b0 0x5654f9a5fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.783+0000 7fbff9db7740 1 --2- 172.21.15.31:0/3933593913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654f96882f0 0x5654f9a5f5c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.783+0000 7fbff9db7740 1 -- 172.21.15.31:0/3933593913 >> 172.21.15.31:0/3933593913 conn(0x5654f96a5300 msgr2=0x5654f9a55dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.783+0000 7fbfcbfff640 0 client.25724 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.783+0000 7fbff9db7740 1 -- 172.21.15.31:0/3933593913 shutdown_connections 2024-04-25T16:03:04.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.783+0000 7fbff9db7740 1 -- 172.21.15.31:0/3933593913 wait complete. 2024-04-25T16:03:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.783+0000 7fbff9db7740 20 client.25724 trim_cache size 0 max 16384 2024-04-25T16:03:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:03:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:03:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:03:04.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:03:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.788+0000 7fbff1ca6640 1 -- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfd40afc70 msgr2=0x7fbfd40b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.789+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfd40afc70 0x7fbfd40b2180 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbfe0005fd0 tx=0x7fbfe0005e40 comp rx=0 tx=0).stop 2024-04-25T16:03:04.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.789+0000 7fbff1ca6640 1 -- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfec10b190 msgr2=0x7fbfec1a0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.789+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfec10b190 0x7fbfec1a0ae0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40427f0 tx=0x7fbfe4003f80 comp rx=0 tx=0).stop 2024-04-25T16:03:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.789+0000 7fbff1ca6640 1 -- 172.21.15.31:0/1816325304 shutdown_connections 2024-04-25T16:03:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.789+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfd40afc70 0x7fbfd40b2180 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.789+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfec10feb0 0x7fbfec1a7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.789+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfec10b190 0x7fbfec1a0ae0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.789+0000 7fbff1ca6640 1 --2- 172.21.15.31:0/1816325304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfec10a340 0x7fbfec1a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:04.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.790+0000 7fbff1ca6640 1 -- 172.21.15.31:0/1816325304 >> 172.21.15.31:0/1816325304 conn(0x7fbfec070330 msgr2=0x7fbfec10dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.793+0000 7fbff1ca6640 1 -- 172.21.15.31:0/1816325304 shutdown_connections 2024-04-25T16:03:04.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:04.793+0000 7fbff1ca6640 1 -- 172.21.15.31:0/1816325304 wait complete. 2024-04-25T16:03:04.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:04 smithi063 ceph-mon[32413]: pgmap v404: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 945 MiB/s wr, 2.25k op/s 2024-04-25T16:03:04.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:04 smithi186 ceph-mon[33258]: pgmap v404: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 945 MiB/s wr, 2.25k op/s 2024-04-25T16:03:04.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:04 smithi031 ceph-mon[29140]: pgmap v404: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 945 MiB/s wr, 2.25k op/s 2024-04-25T16:03:06.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:06 smithi186 ceph-mon[33258]: pgmap v405: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 794 MiB/s wr, 2.37k op/s 2024-04-25T16:03:06.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:06 smithi031 ceph-mon[29140]: pgmap v405: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 794 MiB/s wr, 2.37k op/s 2024-04-25T16:03:07.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:06 smithi063 ceph-mon[32413]: pgmap v405: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 794 MiB/s wr, 2.37k op/s 2024-04-25T16:03:08.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:08 smithi186 ceph-mon[33258]: pgmap v406: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 902 MiB/s wr, 2.84k op/s 2024-04-25T16:03:08.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:08 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:03:08.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:08 smithi031 ceph-mon[29140]: pgmap v406: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 902 MiB/s wr, 2.84k op/s 2024-04-25T16:03:08.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:08 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:03:09.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:08 smithi063 ceph-mon[32413]: pgmap v406: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 902 MiB/s wr, 2.84k op/s 2024-04-25T16:03:09.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:08 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:03:09.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:09 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:03:10.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:09 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:03:10.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:09 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:03:10.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:10 smithi031 ceph-mon[29140]: pgmap v407: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 867 MiB/s wr, 3.02k op/s 2024-04-25T16:03:11.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:10 smithi063 ceph-mon[32413]: pgmap v407: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 867 MiB/s wr, 3.02k op/s 2024-04-25T16:03:11.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:10 smithi186 ceph-mon[33258]: pgmap v407: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 867 MiB/s wr, 3.02k op/s 2024-04-25T16:03:12.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:12 smithi031 ceph-mon[29140]: pgmap v408: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 703 MiB/s wr, 2.63k op/s 2024-04-25T16:03:13.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:12 smithi063 ceph-mon[32413]: pgmap v408: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 703 MiB/s wr, 2.63k op/s 2024-04-25T16:03:13.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:12 smithi186 ceph-mon[33258]: pgmap v408: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 703 MiB/s wr, 2.63k op/s 2024-04-25T16:03:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:14 smithi063 ceph-mon[32413]: pgmap v409: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 921 MiB/s wr, 3.13k op/s 2024-04-25T16:03:15.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:14 smithi186 ceph-mon[33258]: pgmap v409: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 921 MiB/s wr, 3.13k op/s 2024-04-25T16:03:15.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:14 smithi031 ceph-mon[29140]: pgmap v409: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 921 MiB/s wr, 3.13k op/s 2024-04-25T16:03:17.013 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:16 smithi031 ceph-mon[29140]: pgmap v410: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 804 MiB/s wr, 2.15k op/s 2024-04-25T16:03:17.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:16 smithi063 ceph-mon[32413]: pgmap v410: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 804 MiB/s wr, 2.15k op/s 2024-04-25T16:03:17.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:16 smithi186 ceph-mon[33258]: pgmap v410: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 804 MiB/s wr, 2.15k op/s 2024-04-25T16:03:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.266+0000 7fddb4cc6640 1 -- 172.21.15.31:0/985649345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddb0110070 msgr2=0x7fddb0110480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.266+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/985649345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddb0110070 0x7fddb0110480 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009860 tx=0x7fdda402a360 comp rx=0 tx=0).stop 2024-04-25T16:03:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.274+0000 7fddb4cc6640 1 -- 172.21.15.31:0/985649345 shutdown_connections 2024-04-25T16:03:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.274+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/985649345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddb0110070 0x7fddb0110480 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.274+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/985649345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb010b350 0x7fddb010b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.274+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/985649345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddb010a420 0x7fddb010ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.276 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.274+0000 7fddb4cc6640 1 -- 172.21.15.31:0/985649345 >> 172.21.15.31:0/985649345 conn(0x7fddb0070540 msgr2=0x7fddb00729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.274+0000 7fddb4cc6640 1 -- 172.21.15.31:0/985649345 shutdown_connections 2024-04-25T16:03:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.274+0000 7fddb4cc6640 1 -- 172.21.15.31:0/985649345 wait complete. 2024-04-25T16:03:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.276+0000 7fddb4cc6640 1 Processor -- start 2024-04-25T16:03:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.276+0000 7fddb4cc6640 1 -- start start 2024-04-25T16:03:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.277+0000 7fddb4cc6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddb010a420 0x7fddb0116cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.277+0000 7fddb4cc6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddb010b350 0x7fddb0117230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.277+0000 7fddb4cc6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb01a5c10 0x7fddb01a8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.277+0000 7fddb4cc6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fddb011e520 con 0x7fddb010a420 2024-04-25T16:03:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.277+0000 7fddb4cc6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fddb011e3a0 con 0x7fddb010b350 2024-04-25T16:03:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.277+0000 7fddb4cc6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fddb011e6a0 con 0x7fddb01a5c10 2024-04-25T16:03:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.278+0000 7fddae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddb010a420 0x7fddb0116cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.279+0000 7fddae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddb010a420 0x7fddb0116cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55044/0 (socket says 172.21.15.31:55044) 2024-04-25T16:03:18.280 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.279+0000 7fddae59c640 1 -- 172.21.15.31:0/786873516 learned_addr learned my addr 172.21.15.31:0/786873516 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:18.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.280+0000 7fddaed9d640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb01a5c10 0x7fddb01a8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.280+0000 7fddadd9b640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddb010b350 0x7fddb0117230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.280+0000 7fddae59c640 1 -- 172.21.15.31:0/786873516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb01a5c10 msgr2=0x7fddb01a8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.280+0000 7fddae59c640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb01a5c10 0x7fddb01a8030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.280+0000 7fddae59c640 1 -- 172.21.15.31:0/786873516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddb010b350 msgr2=0x7fddb0117230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.280+0000 7fddae59c640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddb010b350 0x7fddb0117230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.280+0000 7fddae59c640 1 -- 172.21.15.31:0/786873516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddb01a8620 con 0x7fddb010a420 2024-04-25T16:03:18.282 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.281+0000 7fddae59c640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddb010a420 0x7fddb0116cf0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdda800ea50 tx=0x7fdda800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:18.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.282+0000 7fdd9f7fe640 1 -- 172.21.15.31:0/786873516 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda800cd60 con 0x7fddb010a420 2024-04-25T16:03:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.282+0000 7fddb4cc6640 1 -- 172.21.15.31:0/786873516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddb01a8910 con 0x7fddb010a420 2024-04-25T16:03:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.283+0000 7fddb4cc6640 1 -- 172.21.15.31:0/786873516 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddb01a8e60 con 0x7fddb010a420 2024-04-25T16:03:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.283+0000 7fdd9f7fe640 1 -- 172.21.15.31:0/786873516 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdda8005020 con 0x7fddb010a420 2024-04-25T16:03:18.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.283+0000 7fdd9f7fe640 1 -- 172.21.15.31:0/786873516 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda80116d0 con 0x7fddb010a420 2024-04-25T16:03:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.289+0000 7fddbbb77740 1 Processor -- start 2024-04-25T16:03:18.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.290+0000 7fddbbb77740 1 -- start start 2024-04-25T16:03:18.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.290+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388a01b0 0x564f38c40970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.293+0000 7fdd9f7fe640 1 -- 172.21.15.31:0/786873516 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdda80051d0 con 0x7fddb010a420 2024-04-25T16:03:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.295+0000 7fdd9f7fe640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdd980afcf0 0x7fdd980b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.296+0000 7fdd9f7fe640 1 -- 172.21.15.31:0/786873516 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdda80dedc0 con 0x7fddb010a420 2024-04-25T16:03:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.297+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388a01b0 0x564f38c40970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.297+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388a01b0 0x564f38c40970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55060/0 (socket says 172.21.15.31:55060) 2024-04-25T16:03:18.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.297+0000 7fddaed9d640 1 -- 172.21.15.31:0/1965500452 learned_addr learned my addr 172.21.15.31:0/1965500452 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.297+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1965500452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388772f0 0x564f38c40ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.300+0000 7fddae59c640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdd980afcf0 0x7fdd980b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.301+0000 7fddae59c640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdd980afcf0 0x7fdd980b2200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdda80a6f50 tx=0x7fdda80058b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.303+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1965500452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c47df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.303+0000 7fddbbb77740 1 -- 172.21.15.31:0/1965500452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564f38ba6460 con 0x564f388a01b0 2024-04-25T16:03:18.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.303+0000 7fddbbb77740 1 -- 172.21.15.31:0/1965500452 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564f38b60590 con 0x564f3887f4d0 2024-04-25T16:03:18.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.303+0000 7fddbbb77740 1 -- 172.21.15.31:0/1965500452 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564f38ba2420 con 0x564f388772f0 2024-04-25T16:03:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.304+0000 7fddaed9d640 1 -- 172.21.15.31:0/1965500452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388772f0 msgr2=0x564f38c40ef0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.304+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1965500452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388772f0 0x564f38c40ef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.304+0000 7fddaed9d640 1 -- 172.21.15.31:0/1965500452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 msgr2=0x564f38c47df0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:18.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.304+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1965500452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c47df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.304+0000 7fddaed9d640 1 -- 172.21.15.31:0/1965500452 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f38b6d0d0 con 0x564f388a01b0 2024-04-25T16:03:18.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.305+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1965500452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388a01b0 0x564f38c40970 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdda40092e0 tx=0x7fdda4009420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862abb99265bda64 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.306+0000 7fdd9cff9640 1 -- 172.21.15.31:0/1965500452 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4040020 con 0x564f388a01b0 2024-04-25T16:03:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.306+0000 7fdd9cff9640 1 -- 172.21.15.31:0/1965500452 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdda4038040 con 0x564f388a01b0 2024-04-25T16:03:18.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.306+0000 7fdd9cff9640 1 -- 172.21.15.31:0/1965500452 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4037070 con 0x564f388a01b0 2024-04-25T16:03:18.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.309+0000 7fddbbb77740 1 -- 172.21.15.31:0/1965500452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388a01b0 msgr2=0x564f38c40970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:18.316 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.309+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1965500452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388a01b0 0x564f38c40970 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fdda40092e0 tx=0x7fdda4009420 comp rx=0 tx=0).stop 2024-04-25T16:03:18.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.322+0000 7fddbbb77740 1 -- 172.21.15.31:0/1965500452 shutdown_connections 2024-04-25T16:03:18.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.322+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1965500452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c47df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.322+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1965500452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388772f0 0x564f38c40ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.322+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1965500452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388a01b0 0x564f38c40970 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.322+0000 7fddbbb77740 1 -- 172.21.15.31:0/1965500452 >> 172.21.15.31:0/1965500452 conn(0x564f38894300 msgr2=0x564f38c3c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.324+0000 7fddbbb77740 1 -- 172.21.15.31:0/1965500452 shutdown_connections 2024-04-25T16:03:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.324+0000 7fddbbb77740 1 -- 172.21.15.31:0/1965500452 wait complete. 2024-04-25T16:03:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.325+0000 7fddbbb77740 1 -- start start 2024-04-25T16:03:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.325+0000 7fddbbb77740 1 Processor -- start 2024-04-25T16:03:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.326+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.327+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c4abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.327+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 0x564f38c51b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.327+0000 7fddbbb77740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564f38c487e0 con 0x564f388772f0 2024-04-25T16:03:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.327+0000 7fddbbb77740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564f38b2db60 con 0x564f3887f4d0 2024-04-25T16:03:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.327+0000 7fddbbb77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564f38c48960 con 0x564f388a01b0 2024-04-25T16:03:18.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.327+0000 7fddbbb77740 10 client.0 _ll_register_callbacks cb 0x564f38b522b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.329+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.329+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55076/0 (socket says 172.21.15.31:55076) 2024-04-25T16:03:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.329+0000 7fddaed9d640 1 -- 172.21.15.31:0/4034854283 learned_addr learned my addr 172.21.15.31:0/4034854283 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:18.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.330+0000 7fddaed9d640 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 0x564f38c51b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.331+0000 7fddadd9b640 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c4abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.331+0000 7fddaed9d640 1 -- 172.21.15.31:0/4034854283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 msgr2=0x564f38c51b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.331+0000 7fddaed9d640 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 0x564f38c51b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.332+0000 7fddaed9d640 1 -- 172.21.15.31:0/4034854283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 msgr2=0x564f38c4abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.332+0000 7fddaed9d640 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c4abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.332+0000 7fddaed9d640 1 -- 172.21.15.31:0/4034854283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f388b8210 con 0x564f388772f0 2024-04-25T16:03:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.334+0000 7fddaed9d640 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4a680 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdda40095f0 tx=0x7fdda4005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6315f9aae356001 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.335+0000 7fdd95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.339+0000 7fdd95ffb640 1 -- 172.21.15.31:0/4034854283 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4040030 con 0x564f388772f0 2024-04-25T16:03:18.340 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.339+0000 7fddbbb77740 1 -- 172.21.15.31:0/4034854283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdda4038040 con 0x564f388772f0 2024-04-25T16:03:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.340+0000 7fdd95ffb640 1 -- 172.21.15.31:0/4034854283 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdda4005450 con 0x564f388772f0 2024-04-25T16:03:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.343+0000 7fdd95ffb640 1 -- 172.21.15.31:0/4034854283 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdda4037040 con 0x564f388772f0 2024-04-25T16:03:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.343+0000 7fddbbb77740 10 client.16050 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.343+0000 7fddbbb77740 1 -- 172.21.15.31:0/4034854283 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f38984620 con 0x564f388772f0 2024-04-25T16:03:18.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.344+0000 7fdd95ffb640 1 -- 172.21.15.31:0/4034854283 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdda4038040 con 0x564f388772f0 2024-04-25T16:03:18.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.346+0000 7fdd95ffb640 1 -- 172.21.15.31:0/4034854283 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fdda40035f0 con 0x564f388772f0 2024-04-25T16:03:18.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.346+0000 7fddbbb77740 10 client.16050 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:18.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.346+0000 7fddbbb77740 10 client.16050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:18.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.347+0000 7fddbbb77740 20 client.16050 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:18.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.347+0000 7fddbbb77740 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564f3885ed80 0x564f38c58300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.347+0000 7fddbbb77740 4 client.16050 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:18.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.347+0000 7fddbbb77740 1 -- 172.21.15.31:0/4034854283 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564f38bb4070 con 0x564f3885ed80 2024-04-25T16:03:18.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.349+0000 7fddadd9b640 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564f3885ed80 0x564f38c58300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.362+0000 7fddadd9b640 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564f3885ed80 0x564f38c58300 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d26b36d0b00df218 server_cookie=28d308fbca99f85f in_seq=0 out_seq=0 2024-04-25T16:03:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.362+0000 7fdd95ffb640 10 client.16050 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:18.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.369+0000 7fdd95ffb640 1 -- 172.21.15.31:0/4034854283 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564f38bb4070 con 0x564f3885ed80 2024-04-25T16:03:18.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.369+0000 7fdd95ffb640 10 client.16050 handle_command_reply: tid=0 2024-04-25T16:03:18.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.369+0000 7fddbbb77740 1 client.16050 shutdown 2024-04-25T16:03:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.377+0000 7fddbbb77740 1 -- 172.21.15.31:0/4034854283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 msgr2=0x564f38c4a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.377+0000 7fddbbb77740 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4a680 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fdda40095f0 tx=0x7fdda4005f10 comp rx=0 tx=0).stop 2024-04-25T16:03:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fddbbb77740 1 -- 172.21.15.31:0/4034854283 shutdown_connections 2024-04-25T16:03:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fddbbb77740 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564f3885ed80 0x564f38c58300 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fddbbb77740 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 0x564f38c51b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fddbbb77740 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c4abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fddbbb77740 1 --2- 172.21.15.31:0/4034854283 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4a680 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fddbbb77740 1 -- 172.21.15.31:0/4034854283 >> 172.21.15.31:0/4034854283 conn(0x564f38894300 msgr2=0x564f38c3b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fdd95ffb640 0 client.16050 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fddbbb77740 1 -- 172.21.15.31:0/4034854283 shutdown_connections 2024-04-25T16:03:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fddbbb77740 1 -- 172.21.15.31:0/4034854283 wait complete. 2024-04-25T16:03:18.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.378+0000 7fddbbb77740 20 client.16050 trim_cache size 0 max 16384 2024-04-25T16:03:18.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.460+0000 7fddbbb77740 1 Processor -- start 2024-04-25T16:03:18.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.460+0000 7fddbbb77740 1 -- start start 2024-04-25T16:03:18.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.461+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c3bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.461+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3887f4d0 0x564f38c51ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.461+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f388a01b0 0x564f38c3c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.461+0000 7fddbbb77740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9000e3b0 con 0x564f388772f0 2024-04-25T16:03:18.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.461+0000 7fddbbb77740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9000ff20 con 0x564f388a01b0 2024-04-25T16:03:18.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.461+0000 7fddbbb77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9000ea80 con 0x564f3887f4d0 2024-04-25T16:03:18.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.462+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f388a01b0 0x564f38c3c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.462+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f388a01b0 0x564f38c3c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34738/0 (socket says 172.21.15.31:34738) 2024-04-25T16:03:18.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.462+0000 7fddaed9d640 1 -- 172.21.15.31:0/1253056435 learned_addr learned my addr 172.21.15.31:0/1253056435 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.463+0000 7fddaed9d640 1 -- 172.21.15.31:0/1253056435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3887f4d0 msgr2=0x564f38c51ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.463+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1253056435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3887f4d0 0x564f38c51ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.463+0000 7fddaed9d640 1 -- 172.21.15.31:0/1253056435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 msgr2=0x564f38c3bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.463+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1253056435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c3bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.463+0000 7fddaed9d640 1 -- 172.21.15.31:0/1253056435 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564f38c4b6f0 con 0x564f388a01b0 2024-04-25T16:03:18.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.464+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1253056435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f388a01b0 0x564f38c3c150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdda40059d0 tx=0x7fdda4000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ea73a53cf312f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.465+0000 7fdd97fff640 1 -- 172.21.15.31:0/1253056435 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4040070 con 0x564f388a01b0 2024-04-25T16:03:18.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.465+0000 7fdd97fff640 1 -- 172.21.15.31:0/1253056435 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdda404e070 con 0x564f388a01b0 2024-04-25T16:03:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.466+0000 7fddbbb77740 1 -- 172.21.15.31:0/1253056435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f388a01b0 msgr2=0x564f38c3c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.466+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1253056435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f388a01b0 0x564f38c3c150 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdda40059d0 tx=0x7fdda4000bc0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.466+0000 7fddbbb77740 1 -- 172.21.15.31:0/1253056435 shutdown_connections 2024-04-25T16:03:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.466+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1253056435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f388a01b0 0x564f38c3c150 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.466+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1253056435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3887f4d0 0x564f38c51ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.466+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1253056435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c3bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.466+0000 7fddbbb77740 1 -- 172.21.15.31:0/1253056435 >> 172.21.15.31:0/1253056435 conn(0x564f38894300 msgr2=0x564f38c44ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.472+0000 7fddbbb77740 1 -- 172.21.15.31:0/1253056435 shutdown_connections 2024-04-25T16:03:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.472+0000 7fddbbb77740 1 -- 172.21.15.31:0/1253056435 wait complete. 2024-04-25T16:03:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.473+0000 7fddbbb77740 1 -- start start 2024-04-25T16:03:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.473+0000 7fddbbb77740 1 Processor -- start 2024-04-25T16:03:18.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.473+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.473+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c51870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.473+0000 7fddbbb77740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 0x564f38c4c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.473+0000 7fddbbb77740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9000f820 con 0x564f388772f0 2024-04-25T16:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.473+0000 7fddbbb77740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9000f150 con 0x564f3887f4d0 2024-04-25T16:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.474+0000 7fddbbb77740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdd9000c440 con 0x564f388a01b0 2024-04-25T16:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.474+0000 7fddbbb77740 10 client.0 _ll_register_callbacks cb 0x7fdda0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.474+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.474+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:55100/0 (socket says 172.21.15.31:55100) 2024-04-25T16:03:18.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.474+0000 7fddaed9d640 1 -- 172.21.15.31:0/1542452047 learned_addr learned my addr 172.21.15.31:0/1542452047 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.475+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 0x564f38c4c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.476+0000 7fddadd9b640 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c51870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.477+0000 7fddaed9d640 1 -- 172.21.15.31:0/1542452047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 msgr2=0x564f38c4c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.477+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 0x564f38c4c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.477+0000 7fddaed9d640 1 -- 172.21.15.31:0/1542452047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 msgr2=0x564f38c51870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.477+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c51870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:18.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.477+0000 7fddaed9d640 1 -- 172.21.15.31:0/1542452047 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564f38c4d000 con 0x564f388772f0 2024-04-25T16:03:18.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.486+0000 7fddaed9d640 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4c480 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fdda4030960 tx=0x7fdda40094a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57b99e0757bc7dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.486+0000 7fdd95ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.490+0000 7fdd95ffb640 1 -- 172.21.15.31:0/1542452047 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda4040040 con 0x564f388772f0 2024-04-25T16:03:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.491+0000 7fddbbb77740 1 -- 172.21.15.31:0/1542452047 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564f38badee0 con 0x564f388772f0 2024-04-25T16:03:18.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.491+0000 7fdd95ffb640 1 -- 172.21.15.31:0/1542452047 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdda40046c0 con 0x564f388772f0 2024-04-25T16:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.495+0000 7fdd95ffb640 1 -- 172.21.15.31:0/1542452047 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fdda4005c90 con 0x564f388772f0 2024-04-25T16:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.496+0000 7fddbbb77740 10 client.16053 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:18.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.496+0000 7fddbbb77740 1 -- 172.21.15.31:0/1542452047 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564f38cbf780 con 0x564f388772f0 2024-04-25T16:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.500+0000 7fdd95ffb640 1 -- 172.21.15.31:0/1542452047 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564f38badee0 con 0x564f388772f0 2024-04-25T16:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.501+0000 7fdd95ffb640 1 -- 172.21.15.31:0/1542452047 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fdda4069030 con 0x564f388772f0 2024-04-25T16:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.502+0000 7fddbbb77740 10 client.16053 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.502+0000 7fddbbb77740 10 client.16053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.502+0000 7fddbbb77740 20 client.16053 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.502+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564f388341c0 0x564f38cc21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.502+0000 7fddbbb77740 4 client.16053 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.502+0000 7fddbbb77740 1 -- 172.21.15.31:0/1542452047 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564f38b295f0 con 0x564f388341c0 2024-04-25T16:03:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.502+0000 7fddadd9b640 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564f388341c0 0x564f38cc21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.504+0000 7fddadd9b640 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564f388341c0 0x564f38cc21a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c755c78183e64ce4 server_cookie=60449f9af183dd26 in_seq=0 out_seq=0 2024-04-25T16:03:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:18.506+0000 7fdd95ffb640 10 client.16053 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:19.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:18 smithi063 ceph-mon[32413]: pgmap v411: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 913 MiB/s wr, 2.42k op/s 2024-04-25T16:03:19.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:18 smithi186 ceph-mon[33258]: pgmap v411: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 913 MiB/s wr, 2.42k op/s 2024-04-25T16:03:19.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:18 smithi031 ceph-mon[29140]: pgmap v411: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 913 MiB/s wr, 2.42k op/s 2024-04-25T16:03:21.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:20 smithi063 ceph-mon[32413]: pgmap v412: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 875 MiB/s wr, 2.40k op/s 2024-04-25T16:03:21.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:20 smithi186 ceph-mon[33258]: pgmap v412: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 875 MiB/s wr, 2.40k op/s 2024-04-25T16:03:21.203 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:20 smithi031 ceph-mon[29140]: pgmap v412: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 875 MiB/s wr, 2.40k op/s 2024-04-25T16:03:23.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:22 smithi063 ceph-mon[32413]: pgmap v413: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 707 MiB/s wr, 1.95k op/s 2024-04-25T16:03:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:22 smithi186 ceph-mon[33258]: pgmap v413: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 707 MiB/s wr, 1.95k op/s 2024-04-25T16:03:23.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:22 smithi031 ceph-mon[29140]: pgmap v413: 97 pgs: 97 active+clean; 10 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 707 MiB/s wr, 1.95k op/s 2024-04-25T16:03:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.258+0000 7fdd95ffb640 1 -- 172.21.15.31:0/1542452047 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564f38b295f0 con 0x564f388341c0 2024-04-25T16:03:24.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.258+0000 7fdd95ffb640 10 client.16053 handle_command_reply: tid=0 2024-04-25T16:03:24.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.258+0000 7fddbbb77740 1 client.16053 shutdown 2024-04-25T16:03:24.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 -- 172.21.15.31:0/1542452047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 msgr2=0x564f38c4c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:24.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4c480 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fdda4030960 tx=0x7fdda40094a0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 -- 172.21.15.31:0/1542452047 shutdown_connections 2024-04-25T16:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564f388341c0 0x564f38cc21a0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f388a01b0 0x564f38c4c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564f3887f4d0 0x564f38c51870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 --2- 172.21.15.31:0/1542452047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564f388772f0 0x564f38c4c480 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 -- 172.21.15.31:0/1542452047 >> 172.21.15.31:0/1542452047 conn(0x564f38894300 msgr2=0x564f38c3bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fdd95ffb640 0 client.16053 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 -- 172.21.15.31:0/1542452047 shutdown_connections 2024-04-25T16:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.259+0000 7fddbbb77740 1 -- 172.21.15.31:0/1542452047 wait complete. 2024-04-25T16:03:24.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.260+0000 7fddbbb77740 20 client.16053 trim_cache size 0 max 16384 2024-04-25T16:03:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:03:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:03:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:03:24.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:03:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.279+0000 7fddb4cc6640 1 -- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdd980afcf0 msgr2=0x7fdd980b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.280+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdd980afcf0 0x7fdd980b2200 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdda80a6f50 tx=0x7fdda80058b0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.280+0000 7fddb4cc6640 1 -- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddb010a420 msgr2=0x7fddb0116cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:24.281 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.280+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddb010a420 0x7fddb0116cf0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fdda800ea50 tx=0x7fdda800ef10 comp rx=0 tx=0).stop 2024-04-25T16:03:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.280+0000 7fddb4cc6640 1 -- 172.21.15.31:0/786873516 shutdown_connections 2024-04-25T16:03:24.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.283+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdd980afcf0 0x7fdd980b2200 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.284+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb01a5c10 0x7fddb01a8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.284+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fddb010b350 0x7fddb0117230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.284+0000 7fddb4cc6640 1 --2- 172.21.15.31:0/786873516 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fddb010a420 0x7fddb0116cf0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.284+0000 7fddb4cc6640 1 -- 172.21.15.31:0/786873516 >> 172.21.15.31:0/786873516 conn(0x7fddb0070540 msgr2=0x7fddb0071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.284+0000 7fddb4cc6640 1 -- 172.21.15.31:0/786873516 shutdown_connections 2024-04-25T16:03:24.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:24.284+0000 7fddb4cc6640 1 -- 172.21.15.31:0/786873516 wait complete. 2024-04-25T16:03:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:24 smithi063 ceph-mon[32413]: pgmap v414: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 981 MiB/s wr, 2.58k op/s 2024-04-25T16:03:25.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:24 smithi186 ceph-mon[33258]: pgmap v414: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 981 MiB/s wr, 2.58k op/s 2024-04-25T16:03:25.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:24 smithi031 ceph-mon[29140]: pgmap v414: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 981 MiB/s wr, 2.58k op/s 2024-04-25T16:03:27.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:26 smithi063 ceph-mon[32413]: pgmap v415: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 813 MiB/s wr, 2.22k op/s 2024-04-25T16:03:27.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:26 smithi186 ceph-mon[33258]: pgmap v415: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 813 MiB/s wr, 2.22k op/s 2024-04-25T16:03:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:26 smithi031 ceph-mon[29140]: pgmap v415: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 813 MiB/s wr, 2.22k op/s 2024-04-25T16:03:29.121 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:28 smithi063 ceph-mon[32413]: pgmap v416: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 913 MiB/s wr, 2.48k op/s 2024-04-25T16:03:29.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:28 smithi186 ceph-mon[33258]: pgmap v416: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 913 MiB/s wr, 2.48k op/s 2024-04-25T16:03:29.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:28 smithi031 ceph-mon[29140]: pgmap v416: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 913 MiB/s wr, 2.48k op/s 2024-04-25T16:03:30.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:03:30.982 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:03:31.121 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:30 smithi031 ceph-mon[29140]: pgmap v417: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 854 MiB/s wr, 2.10k op/s 2024-04-25T16:03:31.121 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:30 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:03:31.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:30 smithi186 ceph-mon[33258]: pgmap v417: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 854 MiB/s wr, 2.10k op/s 2024-04-25T16:03:31.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:30 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:03:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:30 smithi063 ceph-mon[32413]: pgmap v417: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 854 MiB/s wr, 2.10k op/s 2024-04-25T16:03:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:30 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:03:31.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.670+0000 7f3c2911c640 1 -- 172.21.15.31:0/3656412708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c2410b760 msgr2=0x7f3c2410bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:31.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.670+0000 7f3c2911c640 1 --2- 172.21.15.31:0/3656412708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c2410b760 0x7f3c2410bb70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f3c14008410 tx=0x7f3c1402e580 comp rx=0 tx=0).stop 2024-04-25T16:03:31.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.673+0000 7f3c2911c640 1 -- 172.21.15.31:0/3656412708 shutdown_connections 2024-04-25T16:03:31.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.673+0000 7f3c2911c640 1 --2- 172.21.15.31:0/3656412708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c24110460 0x7f3c24112880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.673+0000 7f3c2911c640 1 --2- 172.21.15.31:0/3656412708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c2410b760 0x7f3c2410bb70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.673+0000 7f3c2911c640 1 --2- 172.21.15.31:0/3656412708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2410a520 0x7f3c2410b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.673+0000 7f3c2911c640 1 -- 172.21.15.31:0/3656412708 >> 172.21.15.31:0/3656412708 conn(0x7f3c24070940 msgr2=0x7f3c24072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:31.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.679+0000 7f3c2911c640 1 -- 172.21.15.31:0/3656412708 shutdown_connections 2024-04-25T16:03:31.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.683+0000 7f3c2911c640 1 -- 172.21.15.31:0/3656412708 wait complete. 2024-04-25T16:03:31.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.685+0000 7f3c2911c640 1 Processor -- start 2024-04-25T16:03:31.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.695+0000 7f3c2911c640 1 -- start start 2024-04-25T16:03:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.699+0000 7f3c2911c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c2410a520 0x7f3c241a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.699+0000 7f3c2911c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2410b760 0x7f3c241a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.699+0000 7f3c2911c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c24110460 0x7f3c241a7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.699+0000 7f3c2911c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c24115cb0 con 0x7f3c24110460 2024-04-25T16:03:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.699+0000 7f3c2911c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3c24115b30 con 0x7f3c2410a520 2024-04-25T16:03:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.699+0000 7f3c2911c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c24115e30 con 0x7f3c2410b760 2024-04-25T16:03:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.701+0000 7f3c2259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2410b760 0x7f3c241a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.701+0000 7f3c2259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2410b760 0x7f3c241a0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:55104/0 (socket says 172.21.15.31:55104) 2024-04-25T16:03:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.701+0000 7f3c2259c640 1 -- 172.21.15.31:0/4235426409 learned_addr learned my addr 172.21.15.31:0/4235426409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.702+0000 7f3c2259c640 1 -- 172.21.15.31:0/4235426409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c2410a520 msgr2=0x7f3c241a08e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:03:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.702+0000 7f3c2259c640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c2410a520 0x7f3c241a08e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.702+0000 7f3c2259c640 1 -- 172.21.15.31:0/4235426409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c24110460 msgr2=0x7f3c241a7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.702+0000 7f3c2259c640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c24110460 0x7f3c241a7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.702+0000 7f3c2259c640 1 -- 172.21.15.31:0/4235426409 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c241a8420 con 0x7f3c2410b760 2024-04-25T16:03:31.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.703+0000 7f3c2259c640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2410b760 0x7f3c241a0e20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400ace0 tx=0x7f3c1400a420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:31.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.703+0000 7f3c0bfff640 1 -- 172.21.15.31:0/4235426409 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1403a040 con 0x7f3c2410b760 2024-04-25T16:03:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.704+0000 7f3c2911c640 1 -- 172.21.15.31:0/4235426409 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c241a86b0 con 0x7f3c2410b760 2024-04-25T16:03:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.704+0000 7f3c2911c640 1 -- 172.21.15.31:0/4235426409 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c241a8c80 con 0x7f3c2410b760 2024-04-25T16:03:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.709+0000 7f3c2ffcb740 1 Processor -- start 2024-04-25T16:03:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.709+0000 7f3c2ffcb740 1 -- start start 2024-04-25T16:03:31.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.710+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be7942da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.710+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be790634d0 0x55be7942dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.710+0000 7f3c2359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be7942da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.710+0000 7f3c2359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be7942da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56458/0 (socket says 172.21.15.31:56458) 2024-04-25T16:03:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.710+0000 7f3c2359e640 1 -- 172.21.15.31:0/2843828626 learned_addr learned my addr 172.21.15.31:0/2843828626 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.710+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7906b6b0 0x55be7942ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.711+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2843828626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55be79392d80 con 0x55be7908c390 2024-04-25T16:03:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.711+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2843828626 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55be7934c8b0 con 0x55be790634d0 2024-04-25T16:03:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.711+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2843828626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55be7938ed40 con 0x55be7906b6b0 2024-04-25T16:03:31.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.711+0000 7f3c2259c640 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7906b6b0 0x55be7942ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.711+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be790634d0 0x55be7942dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.712+0000 7f3c2359e640 1 -- 172.21.15.31:0/2843828626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7906b6b0 msgr2=0x55be7942ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.712+0000 7f3c2359e640 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7906b6b0 0x55be7942ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.712+0000 7f3c2359e640 1 -- 172.21.15.31:0/2843828626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be790634d0 msgr2=0x55be7942dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:31.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.712+0000 7f3c2359e640 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be790634d0 0x55be7942dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.712+0000 7f3c2359e640 1 -- 172.21.15.31:0/2843828626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be7931da40 con 0x55be7908c390 2024-04-25T16:03:31.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.712+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be790634d0 0x55be7942dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:03:31.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.715+0000 7f3c2359e640 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be7942da60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009d80 tx=0x7f3c1c00e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=201f873eeb52b7d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.716+0000 7f3c0bfff640 1 -- 172.21.15.31:0/4235426409 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c14035b80 con 0x7f3c2410b760 2024-04-25T16:03:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.716+0000 7f3c0bfff640 1 -- 172.21.15.31:0/4235426409 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1403e430 con 0x7f3c2410b760 2024-04-25T16:03:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.716+0000 7f3c0bfff640 1 -- 172.21.15.31:0/4235426409 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3c1403e690 con 0x7f3c2410b760 2024-04-25T16:03:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.718+0000 7f3c0bfff640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c0c0afd30 0x7f3c0c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.719+0000 7f3c0bfff640 1 -- 172.21.15.31:0/4235426409 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3c1410b5b0 con 0x7f3c2410b760 2024-04-25T16:03:31.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.719+0000 7f3c097fa640 1 -- 172.21.15.31:0/2843828626 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c00f810 con 0x55be7908c390 2024-04-25T16:03:31.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.719+0000 7f3c097fa640 1 -- 172.21.15.31:0/2843828626 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c1c0173f0 con 0x55be7908c390 2024-04-25T16:03:31.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.719+0000 7f3c097fa640 1 -- 172.21.15.31:0/2843828626 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c00fe60 con 0x55be7908c390 2024-04-25T16:03:31.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.719+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c0c0afd30 0x7f3c0c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.722+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2843828626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 msgr2=0x55be7942da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.722+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be7942da60 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c009d80 tx=0x7f3c1c00e760 comp rx=0 tx=0).stop 2024-04-25T16:03:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.722+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2843828626 shutdown_connections 2024-04-25T16:03:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.722+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7906b6b0 0x55be7942ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.722+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be790634d0 0x55be7942dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.722+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2843828626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be7942da60 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.722+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2843828626 >> 172.21.15.31:0/2843828626 conn(0x55be790804e0 msgr2=0x55be79429270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.723+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c0c0afd30 0x7f3c0c0b2240 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3c18006fd0 tx=0x7f3c18008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.723+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2843828626 shutdown_connections 2024-04-25T16:03:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.724+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2843828626 wait complete. 2024-04-25T16:03:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.724+0000 7f3c2ffcb740 1 -- start start 2024-04-25T16:03:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.725+0000 7f3c2ffcb740 1 Processor -- start 2024-04-25T16:03:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.725+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be790634d0 0x55be79440250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.725+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 0x55be79440790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.725+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be794476f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.725+0000 7f3c2ffcb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55be7942f780 con 0x55be7908c390 2024-04-25T16:03:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.725+0000 7f3c2ffcb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55be79319d10 con 0x55be7906b6b0 2024-04-25T16:03:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.725+0000 7f3c2ffcb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55be7942f900 con 0x55be790634d0 2024-04-25T16:03:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.725+0000 7f3c2ffcb740 10 client.0 _ll_register_callbacks cb 0x55be79346210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.726+0000 7f3c22d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be790634d0 0x55be79440250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.726+0000 7f3c22d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be790634d0 0x55be79440250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:55122/0 (socket says 172.21.15.31:55122) 2024-04-25T16:03:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.726+0000 7f3c22d9d640 1 -- 172.21.15.31:0/2419147836 learned_addr learned my addr 172.21.15.31:0/2419147836 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.726+0000 7f3c2259c640 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 0x55be79440790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.726+0000 7f3c2359e640 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be794476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.727+0000 7f3c22d9d640 1 -- 172.21.15.31:0/2419147836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 msgr2=0x55be79440790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.727+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 0x55be79440790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.727+0000 7f3c22d9d640 1 -- 172.21.15.31:0/2419147836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 msgr2=0x55be794476f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.727+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be794476f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.727+0000 7f3c22d9d640 1 -- 172.21.15.31:0/2419147836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be7916e7d0 con 0x55be790634d0 2024-04-25T16:03:31.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.728+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be790634d0 0x55be79440250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009510 tx=0x7f3c18004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1ef864b80f83808 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:31.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.728+0000 7f3c06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:31.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.728+0000 7f3c06ff5640 1 -- 172.21.15.31:0/2419147836 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c18024060 con 0x55be790634d0 2024-04-25T16:03:31.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.728+0000 7f3c06ff5640 1 -- 172.21.15.31:0/2419147836 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3c1801f400 con 0x55be790634d0 2024-04-25T16:03:31.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.728+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2419147836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c1c0173f0 con 0x55be790634d0 2024-04-25T16:03:31.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.730+0000 7f3c06ff5640 1 -- 172.21.15.31:0/2419147836 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3c1801f600 con 0x55be790634d0 2024-04-25T16:03:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.731+0000 7f3c2ffcb740 10 client.25702 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:31.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.731+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2419147836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55be790a43f0 con 0x55be790634d0 2024-04-25T16:03:31.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.733+0000 7f3c06ff5640 1 -- 172.21.15.31:0/2419147836 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c1c0173f0 con 0x55be790634d0 2024-04-25T16:03:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.733+0000 7f3c06ff5640 1 -- 172.21.15.31:0/2419147836 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f3c18013070 con 0x55be790634d0 2024-04-25T16:03:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.733+0000 7f3c2ffcb740 10 client.25702 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.733+0000 7f3c2ffcb740 10 client.25702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.733+0000 7f3c2ffcb740 20 client.25702 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.733+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55be7904af60 0x55be7944db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.733+0000 7f3c2ffcb740 4 client.25702 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.733+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2419147836 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be793a0990 con 0x55be7904af60 2024-04-25T16:03:31.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.734+0000 7f3c2259c640 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55be7904af60 0x55be7944db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.743+0000 7f3c2259c640 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55be7904af60 0x55be7944db70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb3f43219fa6ed27 server_cookie=a54921c9b370977a in_seq=0 out_seq=0 2024-04-25T16:03:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.743+0000 7f3c06ff5640 10 client.25702 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.746+0000 7f3c06ff5640 1 -- 172.21.15.31:0/2419147836 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55be793a0990 con 0x55be7904af60 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.746+0000 7f3c06ff5640 10 client.25702 handle_command_reply: tid=0 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.746+0000 7f3c2ffcb740 1 client.25702 shutdown 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.749+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2419147836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be790634d0 msgr2=0x55be79440250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.749+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be790634d0 0x55be79440250 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3c18009510 tx=0x7f3c18004750 comp rx=0 tx=0).stop 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.749+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2419147836 shutdown_connections 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.749+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55be7904af60 0x55be7944db70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.749+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be7908c390 0x55be794476f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.749+0000 7f3c06ff5640 0 client.25702 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.749+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 0x55be79440790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.749+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/2419147836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be790634d0 0x55be79440250 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.749+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2419147836 >> 172.21.15.31:0/2419147836 conn(0x55be790804e0 msgr2=0x55be7942c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.750+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2419147836 shutdown_connections 2024-04-25T16:03:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.750+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/2419147836 wait complete. 2024-04-25T16:03:31.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.750+0000 7f3c2ffcb740 20 client.25702 trim_cache size 0 max 16384 2024-04-25T16:03:31.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.884+0000 7f3c2ffcb740 1 Processor -- start 2024-04-25T16:03:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.885+0000 7f3c2ffcb740 1 -- start start 2024-04-25T16:03:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.886+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 0x55be794283f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.887+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79429310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.887+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 0x55be7942c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.887+0000 7f3c2ffcb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1000e4b0 con 0x55be790634d0 2024-04-25T16:03:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.887+0000 7f3c2ffcb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3c10010020 con 0x55be7906b6b0 2024-04-25T16:03:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.888+0000 7f3c2ffcb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1000eb80 con 0x55be7904af60 2024-04-25T16:03:31.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.889+0000 7f3c2259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79429310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.889+0000 7f3c2259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79429310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56482/0 (socket says 172.21.15.31:56482) 2024-04-25T16:03:31.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.889+0000 7f3c2259c640 1 -- 172.21.15.31:0/1078346416 learned_addr learned my addr 172.21.15.31:0/1078346416 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.890+0000 7f3c2359e640 1 --2- 172.21.15.31:0/1078346416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 0x55be7942c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.890+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/1078346416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 0x55be794283f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.893+0000 7f3c2359e640 1 -- 172.21.15.31:0/1078346416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 msgr2=0x55be794283f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.893+0000 7f3c2359e640 1 --2- 172.21.15.31:0/1078346416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 0x55be794283f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.893+0000 7f3c2359e640 1 -- 172.21.15.31:0/1078346416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 msgr2=0x55be79429310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.893+0000 7f3c2359e640 1 --2- 172.21.15.31:0/1078346416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79429310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.893+0000 7f3c2359e640 1 -- 172.21.15.31:0/1078346416 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be790640f0 con 0x55be7906b6b0 2024-04-25T16:03:31.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.896+0000 7f3c2359e640 1 --2- 172.21.15.31:0/1078346416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 0x55be7942c780 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c000f80 tx=0x7f3c1c00ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be2dc4aee4e337da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:31.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.896+0000 7f3c08ff9640 1 -- 172.21.15.31:0/1078346416 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1c026030 con 0x55be7906b6b0 2024-04-25T16:03:31.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.896+0000 7f3c08ff9640 1 -- 172.21.15.31:0/1078346416 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c1c003e20 con 0x55be7906b6b0 2024-04-25T16:03:31.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.908+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/1078346416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 msgr2=0x55be7942c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:31.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.908+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/1078346416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 0x55be7942c780 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c000f80 tx=0x7f3c1c00ee70 comp rx=0 tx=0).stop 2024-04-25T16:03:31.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.920+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/1078346416 shutdown_connections 2024-04-25T16:03:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.920+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/1078346416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7906b6b0 0x55be7942c780 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.921+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/1078346416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79429310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.921+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/1078346416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 0x55be794283f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.921+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/1078346416 >> 172.21.15.31:0/1078346416 conn(0x55be790804e0 msgr2=0x55be7942c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:31.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.927+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/1078346416 shutdown_connections 2024-04-25T16:03:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.927+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/1078346416 wait complete. 2024-04-25T16:03:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.930+0000 7f3c2ffcb740 1 -- start start 2024-04-25T16:03:31.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.930+0000 7f3c2ffcb740 1 Processor -- start 2024-04-25T16:03:31.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.931+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 0x55be79446f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.931+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79447440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.931+0000 7f3c2ffcb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7908c390 0x55be794b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.931+0000 7f3c2ffcb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1000f920 con 0x55be790634d0 2024-04-25T16:03:31.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.931+0000 7f3c2ffcb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1000f250 con 0x55be7908c390 2024-04-25T16:03:31.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.931+0000 7f3c2ffcb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1000c490 con 0x55be7904af60 2024-04-25T16:03:31.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.931+0000 7f3c2ffcb740 10 client.0 _ll_register_callbacks cb 0x7f3c180095a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:31.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.937+0000 7f3c2259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79447440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.937+0000 7f3c2259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79447440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56484/0 (socket says 172.21.15.31:56484) 2024-04-25T16:03:31.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.937+0000 7f3c2259c640 1 -- 172.21.15.31:0/3410909897 learned_addr learned my addr 172.21.15.31:0/3410909897 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:31.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.938+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 0x55be79446f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.938+0000 7f3c2359e640 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7908c390 0x55be794b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.943+0000 7f3c22d9d640 1 -- 172.21.15.31:0/3410909897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7908c390 msgr2=0x55be794b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:31.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.943+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7908c390 0x55be794b3f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.943+0000 7f3c22d9d640 1 -- 172.21.15.31:0/3410909897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 msgr2=0x55be79447440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:31.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.943+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79447440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.943+0000 7f3c22d9d640 1 -- 172.21.15.31:0/3410909897 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be793534b0 con 0x55be7904af60 2024-04-25T16:03:31.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.951+0000 7f3c22d9d640 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 0x55be79446f00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55be79429d30 tx=0x7f3c18028ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36ae3bf7770db76e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:31.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.951+0000 7f3c06ff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:31.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.953+0000 7f3c06ff5640 1 -- 172.21.15.31:0/3410909897 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1802e080 con 0x55be7904af60 2024-04-25T16:03:31.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.953+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/3410909897 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55be7939a800 con 0x55be7904af60 2024-04-25T16:03:31.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.953+0000 7f3c06ff5640 1 -- 172.21.15.31:0/3410909897 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3c1801f400 con 0x55be7904af60 2024-04-25T16:03:31.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.957+0000 7f3c06ff5640 1 -- 172.21.15.31:0/3410909897 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3c18028e90 con 0x55be7904af60 2024-04-25T16:03:31.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.957+0000 7f3c2ffcb740 10 client.25708 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:31.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.957+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/3410909897 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55be794370d0 con 0x55be7904af60 2024-04-25T16:03:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.959+0000 7f3c06ff5640 1 -- 172.21.15.31:0/3410909897 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55be7939a800 con 0x55be7904af60 2024-04-25T16:03:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.960+0000 7f3c06ff5640 1 -- 172.21.15.31:0/3410909897 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f3c18006040 con 0x55be7904af60 2024-04-25T16:03:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.961+0000 7f3c2ffcb740 10 client.25708 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.961+0000 7f3c2ffcb740 10 client.25708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.961+0000 7f3c2ffcb740 20 client.25708 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.961+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55be790203a0 0x55be79439a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.961+0000 7f3c2ffcb740 4 client.25708 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:03:31.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.961+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/3410909897 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55be793154a0 con 0x55be790203a0 2024-04-25T16:03:31.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.965+0000 7f3c2259c640 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55be790203a0 0x55be79439a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:31.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.974+0000 7f3c2259c640 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55be790203a0 0x55be79439a30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de02f81a31d4bb61 server_cookie=b47e4047c7663325 in_seq=0 out_seq=0 2024-04-25T16:03:31.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.975+0000 7f3c06ff5640 10 client.25708 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:31.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.977+0000 7f3c06ff5640 1 -- 172.21.15.31:0/3410909897 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55be793154a0 con 0x55be790203a0 2024-04-25T16:03:31.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.977+0000 7f3c06ff5640 10 client.25708 handle_command_reply: tid=0 2024-04-25T16:03:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.977+0000 7f3c2ffcb740 1 client.25708 shutdown 2024-04-25T16:03:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/3410909897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 msgr2=0x55be79446f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 0x55be79446f00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x55be79429d30 tx=0x7f3c18028ca0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/3410909897 shutdown_connections 2024-04-25T16:03:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55be790203a0 0x55be79439a30 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55be7908c390 0x55be794b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55be790634d0 0x55be79447440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 --2- 172.21.15.31:0/3410909897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55be7904af60 0x55be79446f00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:31.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/3410909897 >> 172.21.15.31:0/3410909897 conn(0x55be790804e0 msgr2=0x55be79428150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c06ff5640 0 client.25708 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/3410909897 shutdown_connections 2024-04-25T16:03:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.978+0000 7f3c2ffcb740 1 -- 172.21.15.31:0/3410909897 wait complete. 2024-04-25T16:03:31.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:31.979+0000 7f3c2ffcb740 20 client.25708 trim_cache size 0 max 16384 2024-04-25T16:03:31.981 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:03:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:03:31.981 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:03:31.981 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:03:32.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.000+0000 7f3c2911c640 1 -- 172.21.15.31:0/4235426409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c0c0afd30 msgr2=0x7f3c0c0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:32.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.000+0000 7f3c2911c640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c0c0afd30 0x7f3c0c0b2240 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3c18006fd0 tx=0x7f3c18008040 comp rx=0 tx=0).stop 2024-04-25T16:03:32.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.000+0000 7f3c2911c640 1 -- 172.21.15.31:0/4235426409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2410b760 msgr2=0x7f3c241a0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:32.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.000+0000 7f3c2911c640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2410b760 0x7f3c241a0e20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c1400ace0 tx=0x7f3c1400a420 comp rx=0 tx=0).stop 2024-04-25T16:03:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.008+0000 7f3c2911c640 1 -- 172.21.15.31:0/4235426409 shutdown_connections 2024-04-25T16:03:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.008+0000 7f3c2911c640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c0c0afd30 0x7f3c0c0b2240 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.008+0000 7f3c2911c640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c24110460 0x7f3c241a7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.008+0000 7f3c2911c640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c2410b760 0x7f3c241a0e20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.008+0000 7f3c2911c640 1 --2- 172.21.15.31:0/4235426409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c2410a520 0x7f3c241a08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.008+0000 7f3c2911c640 1 -- 172.21.15.31:0/4235426409 >> 172.21.15.31:0/4235426409 conn(0x7f3c24070940 msgr2=0x7f3c2410c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.008+0000 7f3c2911c640 1 -- 172.21.15.31:0/4235426409 shutdown_connections 2024-04-25T16:03:32.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.009+0000 7f3c2911c640 1 -- 172.21.15.31:0/4235426409 wait complete. 2024-04-25T16:03:32.111 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:03:32.112 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:03:32.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.595+0000 7f80924fd640 1 -- 172.21.15.31:0/149871892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808c10a340 msgr2=0x7f808c10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:32.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.595+0000 7f80924fd640 1 --2- 172.21.15.31:0/149871892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808c10a340 0x7f808c10ac50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f8084009ca0 tx=0x7f808402a8f0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.595+0000 7f80924fd640 1 -- 172.21.15.31:0/149871892 shutdown_connections 2024-04-25T16:03:32.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.595+0000 7f80924fd640 1 --2- 172.21.15.31:0/149871892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f808c10fe90 0x7f808c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.595+0000 7f80924fd640 1 --2- 172.21.15.31:0/149871892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808c10b190 0x7f808c10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.595+0000 7f80924fd640 1 --2- 172.21.15.31:0/149871892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808c10a340 0x7f808c10ac50 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.595+0000 7f80924fd640 1 -- 172.21.15.31:0/149871892 >> 172.21.15.31:0/149871892 conn(0x7f808c070320 msgr2=0x7f808c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:32.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.609+0000 7f80924fd640 1 -- 172.21.15.31:0/149871892 shutdown_connections 2024-04-25T16:03:32.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.616+0000 7f80924fd640 1 -- 172.21.15.31:0/149871892 wait complete. 2024-04-25T16:03:32.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.617+0000 7f80924fd640 1 Processor -- start 2024-04-25T16:03:32.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.618+0000 7f80924fd640 1 -- start start 2024-04-25T16:03:32.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.618+0000 7f80924fd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808c10a340 0x7f808c11bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:32.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.618+0000 7f80924fd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f808c10b190 0x7f808c11a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:32.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.618+0000 7f80924fd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808c10fe90 0x7f808c11a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:32.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.618+0000 7f80924fd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f808c112460 con 0x7f808c10fe90 2024-04-25T16:03:32.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.618+0000 7f80924fd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f808c1122e0 con 0x7f808c10b190 2024-04-25T16:03:32.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.618+0000 7f80924fd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f808c1125e0 con 0x7f808c10a340 2024-04-25T16:03:32.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.620+0000 7f80914fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808c10a340 0x7f808c11bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:32.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.621+0000 7f80914fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808c10a340 0x7f808c11bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:55136/0 (socket says 172.21.15.31:55136) 2024-04-25T16:03:32.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.621+0000 7f80914fb640 1 -- 172.21.15.31:0/1045897343 learned_addr learned my addr 172.21.15.31:0/1045897343 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:32.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.623+0000 7f80914fb640 1 -- 172.21.15.31:0/1045897343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f808c10b190 msgr2=0x7f808c11a160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:03:32.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.623+0000 7f80914fb640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f808c10b190 0x7f808c11a160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.623+0000 7f80914fb640 1 -- 172.21.15.31:0/1045897343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808c10fe90 msgr2=0x7f808c11a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:32.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.623+0000 7f80914fb640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808c10fe90 0x7f808c11a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:32.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.623+0000 7f80914fb640 1 -- 172.21.15.31:0/1045897343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808c11ace0 con 0x7f808c10a340 2024-04-25T16:03:32.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.625+0000 7f80914fb640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808c10a340 0x7f808c11bac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f808402aee0 tx=0x7f808402f870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:32.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.628+0000 7f80827fc640 1 -- 172.21.15.31:0/1045897343 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f808402fe70 con 0x7f808c10a340 2024-04-25T16:03:32.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.628+0000 7f80827fc640 1 -- 172.21.15.31:0/1045897343 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f808402d070 con 0x7f808c10a340 2024-04-25T16:03:32.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.628+0000 7f80827fc640 1 -- 172.21.15.31:0/1045897343 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8084046870 con 0x7f808c10a340 2024-04-25T16:03:32.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.630+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f808c11af70 con 0x7f808c10a340 2024-04-25T16:03:32.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.631+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f808c1b9ab0 con 0x7f808c10a340 2024-04-25T16:03:32.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.634+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f808c1b9ce0 con 0x7f808c10a340 2024-04-25T16:03:32.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.646+0000 7f80827fc640 1 -- 172.21.15.31:0/1045897343 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8084004b40 con 0x7f808c10a340 2024-04-25T16:03:32.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.647+0000 7f80827fc640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80780afcf0 0x7f80780b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:32.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.648+0000 7f80827fc640 1 -- 172.21.15.31:0/1045897343 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f80841006d0 con 0x7f808c10a340 2024-04-25T16:03:32.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.653+0000 7f8090cfa640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80780afcf0 0x7f80780b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:32.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.656+0000 7f80827fc640 1 -- 172.21.15.31:0/1045897343 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f808404b060 con 0x7f808c10a340 2024-04-25T16:03:32.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:32.669+0000 7f8090cfa640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80780afcf0 0x7f80780b2200 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f808c11b460 tx=0x7f8088006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:33.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:32 smithi186 ceph-mon[33258]: pgmap v418: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 728 MiB/s wr, 1.65k op/s 2024-04-25T16:03:33.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:32 smithi031 ceph-mon[29140]: pgmap v418: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 728 MiB/s wr, 1.65k op/s 2024-04-25T16:03:33.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:32 smithi063 ceph-mon[32413]: pgmap v418: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 728 MiB/s wr, 1.65k op/s 2024-04-25T16:03:33.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.401+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f808c1ba020 con 0x7f808c10a340 2024-04-25T16:03:33.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.412+0000 7f80827fc640 1 -- 172.21.15.31:0/1045897343 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11918 (secure 0 0 0) 0x7f80840c92e0 con 0x7f808c10a340 2024-04-25T16:03:33.413 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:03:33.413 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:02:58.299084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24553,14631,24563,24535,24521]},"id":1}]} 2024-04-25T16:03:33.414 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-04-25T16:03:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.419+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80780afcf0 msgr2=0x7f80780b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.420+0000 7f80924fd640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80780afcf0 0x7f80780b2200 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f808c11b460 tx=0x7f8088006bd0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.420+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808c10a340 msgr2=0x7f808c11bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:33.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.420+0000 7f80924fd640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808c10a340 0x7f808c11bac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f808402aee0 tx=0x7f808402f870 comp rx=0 tx=0).stop 2024-04-25T16:03:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.432+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 shutdown_connections 2024-04-25T16:03:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.432+0000 7f80924fd640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80780afcf0 0x7f80780b2200 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.432+0000 7f80924fd640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f808c10fe90 0x7f808c11a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.432+0000 7f80924fd640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f808c10b190 0x7f808c11a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.432+0000 7f80924fd640 1 --2- 172.21.15.31:0/1045897343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f808c10a340 0x7f808c11bac0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.432+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 >> 172.21.15.31:0/1045897343 conn(0x7f808c070320 msgr2=0x7f808c10de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.435+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 shutdown_connections 2024-04-25T16:03:33.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.435+0000 7f80924fd640 1 -- 172.21.15.31:0/1045897343 wait complete. 2024-04-25T16:03:33.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:03:33.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.873+0000 7fef9ffff640 1 -- 172.21.15.31:0/809301166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa010b190 msgr2=0x7fefa010b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:33.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.874+0000 7fef9ffff640 1 --2- 172.21.15.31:0/809301166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa010b190 0x7fefa010b5a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fef94009860 tx=0x7fef9402a380 comp rx=0 tx=0).stop 2024-04-25T16:03:33.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.874+0000 7fef9ffff640 1 -- 172.21.15.31:0/809301166 shutdown_connections 2024-04-25T16:03:33.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.874+0000 7fef9ffff640 1 --2- 172.21.15.31:0/809301166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa010fe90 0x7fefa01102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.874+0000 7fef9ffff640 1 --2- 172.21.15.31:0/809301166 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa010b190 0x7fefa010b5a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.874+0000 7fef9ffff640 1 --2- 172.21.15.31:0/809301166 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefa010a340 0x7fefa010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.874+0000 7fef9ffff640 1 -- 172.21.15.31:0/809301166 >> 172.21.15.31:0/809301166 conn(0x7fefa0070320 msgr2=0x7fefa00727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:33.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.875+0000 7fef9ffff640 1 -- 172.21.15.31:0/809301166 shutdown_connections 2024-04-25T16:03:33.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.875+0000 7fef9ffff640 1 -- 172.21.15.31:0/809301166 wait complete. 2024-04-25T16:03:33.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.876+0000 7fef9ffff640 1 Processor -- start 2024-04-25T16:03:33.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.876+0000 7fef9ffff640 1 -- start start 2024-04-25T16:03:33.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.877+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefa010a340 0x7fefa01a8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.877+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa010b190 0x7fefa01a9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.877+0000 7fef9ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa010fe90 0x7fefa007ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.877+0000 7fef9ffff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fefa011e2e0 con 0x7fefa010a340 2024-04-25T16:03:33.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.877+0000 7fef9ffff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fefa011e160 con 0x7fefa010fe90 2024-04-25T16:03:33.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.877+0000 7fef9ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fefa011e460 con 0x7fefa010b190 2024-04-25T16:03:33.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.878+0000 7fef9effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefa010a340 0x7fefa01a8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:33.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.878+0000 7fef9effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefa010a340 0x7fefa01a8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44668/0 (socket says 172.21.15.31:44668) 2024-04-25T16:03:33.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.878+0000 7fef9effd640 1 -- 172.21.15.31:0/1047538608 learned_addr learned my addr 172.21.15.31:0/1047538608 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:33.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.879+0000 7fef9e7fc640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa010b190 0x7fefa01a9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:33.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.879+0000 7fef9f7fe640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa010fe90 0x7fefa007ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:33.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.879+0000 7fef9effd640 1 -- 172.21.15.31:0/1047538608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa010b190 msgr2=0x7fefa01a9440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:33.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.879+0000 7fef9effd640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa010b190 0x7fefa01a9440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.879+0000 7fef9effd640 1 -- 172.21.15.31:0/1047538608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa010fe90 msgr2=0x7fefa007ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.879+0000 7fef9effd640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa010fe90 0x7fefa007ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.879+0000 7fef9effd640 1 -- 172.21.15.31:0/1047538608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa007b470 con 0x7fefa010a340 2024-04-25T16:03:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.880+0000 7fef9effd640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefa010a340 0x7fefa01a8ec0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fef90006fd0 tx=0x7fef9000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.881+0000 7fefa4898640 1 -- 172.21.15.31:0/1047538608 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef90011830 con 0x7fefa010a340 2024-04-25T16:03:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.881+0000 7fef9ffff640 1 -- 172.21.15.31:0/1047538608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa007b760 con 0x7fefa010a340 2024-04-25T16:03:33.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.881+0000 7fef9ffff640 1 -- 172.21.15.31:0/1047538608 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa007bc80 con 0x7fefa010a340 2024-04-25T16:03:33.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.881+0000 7fefa4898640 1 -- 172.21.15.31:0/1047538608 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef90016070 con 0x7fefa010a340 2024-04-25T16:03:33.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.887+0000 7fefac820740 1 Processor -- start 2024-04-25T16:03:33.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.890+0000 7fefac820740 1 -- start start 2024-04-25T16:03:33.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.891+0000 7fefac820740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f45390 0x564a162e79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.891+0000 7fefac820740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162e7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.891+0000 7fefac820740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f246b0 0x564a162e8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.891+0000 7fefac820740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564a1624bd80 con 0x564a15f246b0 2024-04-25T16:03:33.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.891+0000 7fefac820740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564a16205a20 con 0x564a15f45390 2024-04-25T16:03:33.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.891+0000 7fefac820740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564a16247d40 con 0x564a15f1c4d0 2024-04-25T16:03:33.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.892+0000 7fefa4898640 1 -- 172.21.15.31:0/1047538608 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef90010e00 con 0x7fefa010a340 2024-04-25T16:03:33.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.892+0000 7fefa4898640 1 -- 172.21.15.31:0/1047538608 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fef9001b4a0 con 0x7fefa010a340 2024-04-25T16:03:33.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.892+0000 7fef9effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162e7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:33.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.892+0000 7fef9effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162e7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38376/0 (socket says 172.21.15.31:38376) 2024-04-25T16:03:33.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.892+0000 7fef9effd640 1 -- 172.21.15.31:0/390831705 learned_addr learned my addr 172.21.15.31:0/390831705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:33.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.893+0000 7fef9effd640 1 -- 172.21.15.31:0/390831705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f45390 msgr2=0x564a162e79e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:33.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.893+0000 7fef9effd640 1 --2- 172.21.15.31:0/390831705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f45390 0x564a162e79e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.893+0000 7fef9effd640 1 -- 172.21.15.31:0/390831705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f246b0 msgr2=0x564a162e8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:33.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.893+0000 7fef9effd640 1 --2- 172.21.15.31:0/390831705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f246b0 0x564a162e8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.894+0000 7fef9effd640 1 -- 172.21.15.31:0/390831705 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a16210c30 con 0x564a15f1c4d0 2024-04-25T16:03:33.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.899+0000 7fef9effd640 1 --2- 172.21.15.31:0/390831705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162e7f60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fef90010930 tx=0x7fef90011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88d643353255a2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.908+0000 7fefa4898640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef880afd50 0x7fef880b2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.909+0000 7fefa4898640 1 -- 172.21.15.31:0/1047538608 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fef900df1e0 con 0x7fefa010a340 2024-04-25T16:03:33.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.909+0000 7fef85ffb640 1 -- 172.21.15.31:0/390831705 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef900a35b0 con 0x564a15f1c4d0 2024-04-25T16:03:33.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.909+0000 7fef85ffb640 1 -- 172.21.15.31:0/390831705 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef900a7200 con 0x564a15f1c4d0 2024-04-25T16:03:33.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.909+0000 7fef85ffb640 1 -- 172.21.15.31:0/390831705 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef900a6e60 con 0x564a15f1c4d0 2024-04-25T16:03:33.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.911+0000 7fefac820740 1 -- 172.21.15.31:0/390831705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 msgr2=0x564a162e7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:33.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.911+0000 7fefac820740 1 --2- 172.21.15.31:0/390831705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162e7f60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fef90010930 tx=0x7fef90011000 comp rx=0 tx=0).stop 2024-04-25T16:03:33.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.913+0000 7fefac820740 1 -- 172.21.15.31:0/390831705 shutdown_connections 2024-04-25T16:03:33.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.913+0000 7fefac820740 1 --2- 172.21.15.31:0/390831705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f246b0 0x564a162e8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.913+0000 7fefac820740 1 --2- 172.21.15.31:0/390831705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162e7f60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.913+0000 7fefac820740 1 --2- 172.21.15.31:0/390831705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f45390 0x564a162e79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.913+0000 7fefac820740 1 -- 172.21.15.31:0/390831705 >> 172.21.15.31:0/390831705 conn(0x564a15f394e0 msgr2=0x564a162e31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:33.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.913+0000 7fef9f7fe640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef880afd50 0x7fef880b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:33.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.915+0000 7fefac820740 1 -- 172.21.15.31:0/390831705 shutdown_connections 2024-04-25T16:03:33.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.915+0000 7fefac820740 1 -- 172.21.15.31:0/390831705 wait complete. 2024-04-25T16:03:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.916+0000 7fef9f7fe640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef880afd50 0x7fef880b2260 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x564a162e8a30 tx=0x7fef980095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.916+0000 7fefac820740 1 -- start start 2024-04-25T16:03:33.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.917+0000 7fefac820740 1 Processor -- start 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.918+0000 7fefac820740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f1c4d0 0x564a162f1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.918+0000 7fefac820740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f246b0 0x564a162f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.918+0000 7fefac820740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162f8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.918+0000 7fefac820740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564a162e9580 con 0x564a15f246b0 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.918+0000 7fefac820740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564a161d2dc0 con 0x564a15f1c4d0 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.918+0000 7fefac820740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564a162e9700 con 0x564a15f45390 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.918+0000 7fefac820740 10 client.0 _ll_register_callbacks cb 0x564a161ee600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.919+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162f8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.919+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162f8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38390/0 (socket says 172.21.15.31:38390) 2024-04-25T16:03:33.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.919+0000 7fef9f7fe640 1 -- 172.21.15.31:0/1770613062 learned_addr learned my addr 172.21.15.31:0/1770613062 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:33.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.926+0000 7fef9e7fc640 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f246b0 0x564a162f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:33.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.927+0000 7fef9effd640 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f1c4d0 0x564a162f1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:33.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.929+0000 7fef9f7fe640 1 -- 172.21.15.31:0/1770613062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f1c4d0 msgr2=0x564a162f1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:33.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.929+0000 7fef9f7fe640 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f1c4d0 0x564a162f1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.929+0000 7fef9f7fe640 1 -- 172.21.15.31:0/1770613062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f246b0 msgr2=0x564a162f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:33.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.929+0000 7fef9f7fe640 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f246b0 0x564a162f1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.929+0000 7fef9f7fe640 1 -- 172.21.15.31:0/1770613062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a15f5d3f0 con 0x564a15f45390 2024-04-25T16:03:33.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.930+0000 7fef9f7fe640 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162f8b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef9801bc10 tx=0x7fef98004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=954ce887033ab0b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:33.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.930+0000 7fef7effd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:33.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.930+0000 7fef7effd640 1 -- 172.21.15.31:0/1770613062 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9802b060 con 0x564a15f45390 2024-04-25T16:03:33.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.930+0000 7fef7effd640 1 -- 172.21.15.31:0/1770613062 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fef98015030 con 0x564a15f45390 2024-04-25T16:03:33.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.931+0000 7fefac820740 1 -- 172.21.15.31:0/1770613062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef900a7200 con 0x564a15f45390 2024-04-25T16:03:33.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.931+0000 7fef7effd640 1 -- 172.21.15.31:0/1770613062 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fef9801bdc0 con 0x564a15f45390 2024-04-25T16:03:33.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.931+0000 7fefac820740 10 client.25723 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:33.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.931+0000 7fefac820740 1 -- 172.21.15.31:0/1770613062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564a160277d0 con 0x564a15f45390 2024-04-25T16:03:33.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.937+0000 7fef7effd640 1 -- 172.21.15.31:0/1770613062 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef900a7200 con 0x564a15f45390 2024-04-25T16:03:33.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.937+0000 7fef7effd640 1 -- 172.21.15.31:0/1770613062 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fef9800b040 con 0x564a15f45390 2024-04-25T16:03:33.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.937+0000 7fefac820740 10 client.25723 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.937+0000 7fefac820740 10 client.25723 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:03:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.937+0000 7fefac820740 20 client.25723 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.937+0000 7fefac820740 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564a15f03f60 0x564a162ff370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.937+0000 7fefac820740 4 client.25723 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:33.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.937+0000 7fefac820740 1 -- 172.21.15.31:0/1770613062 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564a16259990 con 0x564a15f03f60 2024-04-25T16:03:33.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.939+0000 7fef9effd640 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564a15f03f60 0x564a162ff370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:33.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.942+0000 7fef9effd640 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564a15f03f60 0x564a162ff370 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=76735a7c44a79f3c server_cookie=23a71ec8d3137cd in_seq=0 out_seq=0 2024-04-25T16:03:33.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.942+0000 7fef7effd640 10 client.25723 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:03:33.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.943+0000 7fef7effd640 1 -- 172.21.15.31:0/1770613062 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564a16259990 con 0x564a15f03f60 2024-04-25T16:03:33.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.943+0000 7fef7effd640 10 client.25723 handle_command_reply: tid=0 2024-04-25T16:03:33.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.944+0000 7fefac820740 1 client.25723 shutdown 2024-04-25T16:03:33.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.945+0000 7fefac820740 1 -- 172.21.15.31:0/1770613062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 msgr2=0x564a162f8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:33.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.945+0000 7fefac820740 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162f8b30 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fef9801bc10 tx=0x7fef98004a40 comp rx=0 tx=0).stop 2024-04-25T16:03:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.946+0000 7fefac820740 1 -- 172.21.15.31:0/1770613062 shutdown_connections 2024-04-25T16:03:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.946+0000 7fefac820740 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564a15f03f60 0x564a162ff370 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.946+0000 7fefac820740 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162f8b30 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.946+0000 7fefac820740 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f246b0 0x564a162f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.946+0000 7fefac820740 1 --2- 172.21.15.31:0/1770613062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f1c4d0 0x564a162f1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:33.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.946+0000 7fefac820740 1 -- 172.21.15.31:0/1770613062 >> 172.21.15.31:0/1770613062 conn(0x564a15f394e0 msgr2=0x564a162e23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:33.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.947+0000 7fef7effd640 0 client.25723 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:03:33.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.949+0000 7fefac820740 1 -- 172.21.15.31:0/1770613062 shutdown_connections 2024-04-25T16:03:33.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.951+0000 7fefac820740 1 -- 172.21.15.31:0/1770613062 wait complete. 2024-04-25T16:03:33.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:33.952+0000 7fefac820740 20 client.25723 trim_cache size 0 max 16384 2024-04-25T16:03:34.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.022+0000 7fefac820740 1 Processor -- start 2024-04-25T16:03:34.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.026+0000 7fefac820740 1 -- start start 2024-04-25T16:03:34.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.026+0000 7fefac820740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f1c4d0 0x564a162f0e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.027+0000 7fefac820740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162e6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.027+0000 7fefac820740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162e6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.027+0000 7fefac820740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c00e510 con 0x564a15f1c4d0 2024-04-25T16:03:34.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.027+0000 7fefac820740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c010080 con 0x564a15f246b0 2024-04-25T16:03:34.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.027+0000 7fefac820740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c00ebe0 con 0x564a15f45390 2024-04-25T16:03:34.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.027+0000 7fef9effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f1c4d0 0x564a162f0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.027+0000 7fef9effd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f1c4d0 0x564a162f0e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44696/0 (socket says 172.21.15.31:44696) 2024-04-25T16:03:34.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.027+0000 7fef9effd640 1 -- 172.21.15.31:0/4229125851 learned_addr learned my addr 172.21.15.31:0/4229125851 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:34.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.029+0000 7fef9f7fe640 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162e6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.029+0000 7fef9e7fc640 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162e6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.030+0000 7fef9effd640 1 -- 172.21.15.31:0/4229125851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 msgr2=0x564a162e6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.030+0000 7fef9effd640 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162e6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.030+0000 7fef9effd640 1 -- 172.21.15.31:0/4229125851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 msgr2=0x564a162e6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.030+0000 7fef9effd640 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162e6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.030+0000 7fef9effd640 1 -- 172.21.15.31:0/4229125851 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564a162ece90 con 0x564a15f1c4d0 2024-04-25T16:03:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.031+0000 7fef9e7fc640 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162e6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:03:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.031+0000 7fef9effd640 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f1c4d0 0x564a162f0e20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fef900a3770 tx=0x7fef9000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2186cdadac700b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.031+0000 7fef857fa640 1 -- 172.21.15.31:0/4229125851 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef900a7040 con 0x564a15f1c4d0 2024-04-25T16:03:34.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.031+0000 7fef857fa640 1 -- 172.21.15.31:0/4229125851 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef900a7200 con 0x564a15f1c4d0 2024-04-25T16:03:34.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.033+0000 7fefac820740 1 -- 172.21.15.31:0/4229125851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f1c4d0 msgr2=0x564a162f0e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.033+0000 7fefac820740 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f1c4d0 0x564a162f0e20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fef900a3770 tx=0x7fef9000bf90 comp rx=0 tx=0).stop 2024-04-25T16:03:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.039+0000 7fefac820740 1 -- 172.21.15.31:0/4229125851 shutdown_connections 2024-04-25T16:03:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.039+0000 7fefac820740 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f45390 0x564a162e6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.039+0000 7fefac820740 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162e6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.039+0000 7fefac820740 1 --2- 172.21.15.31:0/4229125851 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f1c4d0 0x564a162f0e20 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.039+0000 7fefac820740 1 -- 172.21.15.31:0/4229125851 >> 172.21.15.31:0/4229125851 conn(0x564a15f394e0 msgr2=0x564a162e5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.043+0000 7fefac820740 1 -- 172.21.15.31:0/4229125851 shutdown_connections 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.044+0000 7fefac820740 1 -- 172.21.15.31:0/4229125851 wait complete. 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.044+0000 7fefac820740 1 -- start start 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.045+0000 7fefac820740 1 Processor -- start 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fefac820740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162f5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fefac820740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fefac820740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f45390 0x564a162f68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fefac820740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c00f980 con 0x564a15f45390 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f45390 0x564a162f68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fef9f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f45390 0x564a162f68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44700/0 (socket says 172.21.15.31:44700) 2024-04-25T16:03:34.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fef9f7fe640 1 -- 172.21.15.31:0/2107504204 learned_addr learned my addr 172.21.15.31:0/2107504204 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:34.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c00f2b0 con 0x564a15f246b0 2024-04-25T16:03:34.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fef9effd640 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162f5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.046+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef8c00c4f0 con 0x564a15f1c4d0 2024-04-25T16:03:34.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.047+0000 7fef9e7fc640 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.047+0000 7fefac820740 10 client.0 _ll_register_callbacks cb 0x7fef9800ff00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:34.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.047+0000 7fef9e7fc640 1 -- 172.21.15.31:0/2107504204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 msgr2=0x564a162f5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.047+0000 7fef9e7fc640 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162f5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.047+0000 7fef9e7fc640 1 -- 172.21.15.31:0/2107504204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f45390 msgr2=0x564a162f68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.047+0000 7fef9e7fc640 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f45390 0x564a162f68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.047+0000 7fef9e7fc640 1 -- 172.21.15.31:0/2107504204 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564a162f6ea0 con 0x564a15f246b0 2024-04-25T16:03:34.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.053+0000 7fef9e7fc640 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162f59b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fef9402a970 tx=0x7fef94030c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5bfbd1a4734d63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:34.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.053+0000 7fef7effd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:03:34.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.065+0000 7fef7effd640 1 -- 172.21.15.31:0/2107504204 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef94041020 con 0x564a15f246b0 2024-04-25T16:03:34.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.065+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef900a7200 con 0x564a15f246b0 2024-04-25T16:03:34.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.065+0000 7fef7effd640 1 -- 172.21.15.31:0/2107504204 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fef94004450 con 0x564a15f246b0 2024-04-25T16:03:34.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.070+0000 7fef7effd640 1 -- 172.21.15.31:0/2107504204 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fef94037040 con 0x564a15f246b0 2024-04-25T16:03:34.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.070+0000 7fefac820740 10 client.25763 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:34.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.070+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564a162f8130 con 0x564a15f246b0 2024-04-25T16:03:34.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.073+0000 7fef7effd640 1 -- 172.21.15.31:0/2107504204 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef900a7200 con 0x564a15f246b0 2024-04-25T16:03:34.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.073+0000 7fef7effd640 1 -- 172.21.15.31:0/2107504204 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fef94004c60 con 0x564a15f246b0 2024-04-25T16:03:34.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.073+0000 7fefac820740 10 client.25763 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:34.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.073+0000 7fefac820740 10 client.25763 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:03:34.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.073+0000 7fefac820740 20 client.25763 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:34.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.073+0000 7fefac820740 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564a15ed93a0 0x564a16369ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.074+0000 7fefac820740 4 client.25763 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:03:34.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.074+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564a16253800 con 0x564a15ed93a0 2024-04-25T16:03:34.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.082+0000 7fef9effd640 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564a15ed93a0 0x564a16369ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.096+0000 7fef9effd640 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564a15ed93a0 0x564a16369ba0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ed18bc483184527 server_cookie=ba2fd5e24c498725 in_seq=0 out_seq=0 2024-04-25T16:03:34.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.097+0000 7fef7effd640 10 client.25763 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:03:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.101+0000 7fef7effd640 1 -- 172.21.15.31:0/2107504204 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564a16253800 con 0x564a15ed93a0 2024-04-25T16:03:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.101+0000 7fef7effd640 10 client.25763 handle_command_reply: tid=0 2024-04-25T16:03:34.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.101+0000 7fefac820740 1 client.25763 shutdown 2024-04-25T16:03:34.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.108+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 msgr2=0x564a162f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.108+0000 7fefac820740 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162f59b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fef9402a970 tx=0x7fef94030c90 comp rx=0 tx=0).stop 2024-04-25T16:03:34.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.115+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 shutdown_connections 2024-04-25T16:03:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.115+0000 7fefac820740 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564a15ed93a0 0x564a16369ba0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.115+0000 7fefac820740 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564a15f45390 0x564a162f68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.115+0000 7fefac820740 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564a15f246b0 0x564a162f59b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.115+0000 7fefac820740 1 --2- 172.21.15.31:0/2107504204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564a15f1c4d0 0x564a162f5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.115+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 >> 172.21.15.31:0/2107504204 conn(0x564a15f394e0 msgr2=0x564a162e6a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:34.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.117+0000 7fef7effd640 0 client.25763 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:03:34.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.117+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 shutdown_connections 2024-04-25T16:03:34.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.117+0000 7fefac820740 1 -- 172.21.15.31:0/2107504204 wait complete. 2024-04-25T16:03:34.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.117+0000 7fefac820740 20 client.25763 trim_cache size 0 max 16384 2024-04-25T16:03:34.119 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:03:34.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.124+0000 7fef9ffff640 1 -- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef880afd50 msgr2=0x7fef880b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.124+0000 7fef9ffff640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef880afd50 0x7fef880b2260 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x564a162e8a30 tx=0x7fef980095c0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.124+0000 7fef9ffff640 1 -- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefa010a340 msgr2=0x7fefa01a8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.124+0000 7fef9ffff640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefa010a340 0x7fefa01a8ec0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fef90006fd0 tx=0x7fef9000b8e0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.125+0000 7fef9ffff640 1 -- 172.21.15.31:0/1047538608 shutdown_connections 2024-04-25T16:03:34.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.125+0000 7fef9ffff640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fef880afd50 0x7fef880b2260 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.125+0000 7fef9ffff640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fefa010fe90 0x7fefa007ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.125+0000 7fef9ffff640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefa010b190 0x7fefa01a9440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.125+0000 7fef9ffff640 1 --2- 172.21.15.31:0/1047538608 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fefa010a340 0x7fefa01a8ec0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.125+0000 7fef9ffff640 1 -- 172.21.15.31:0/1047538608 >> 172.21.15.31:0/1047538608 conn(0x7fefa0070320 msgr2=0x7fefa010be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:34.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.133+0000 7fef9ffff640 1 -- 172.21.15.31:0/1047538608 shutdown_connections 2024-04-25T16:03:34.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.135+0000 7fef9ffff640 1 -- 172.21.15.31:0/1047538608 wait complete. 2024-04-25T16:03:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:33 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:03:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:33 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:03:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:33 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:03:34.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:33 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1045897343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:03:34.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:33 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:03:34.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:33 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:03:34.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:33 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:03:34.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:33 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1045897343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:03:34.202 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:03:34.270 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:33 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:03:34.270 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:33 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:03:34.270 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:33 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:03:34.270 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:33 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1045897343' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:03:34.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.635+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/2079759083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a1810b580 msgr2=0x7f2a1810b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.635+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/2079759083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a1810b580 0x7f2a1810b990 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009ca0 tx=0x7f2a1002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.635+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/2079759083 shutdown_connections 2024-04-25T16:03:34.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.635+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/2079759083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2a18110310 0x7f2a18112730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.635+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/2079759083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a1810b580 0x7f2a1810b990 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.635+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/2079759083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a1810a2d0 0x7f2a1810b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.635+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/2079759083 >> 172.21.15.31:0/2079759083 conn(0x7f2a180707b0 msgr2=0x7f2a18072c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:34.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.639+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/2079759083 shutdown_connections 2024-04-25T16:03:34.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.646+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/2079759083 wait complete. 2024-04-25T16:03:34.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.648+0000 7f2a1c8d0640 1 Processor -- start 2024-04-25T16:03:34.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.648+0000 7f2a1c8d0640 1 -- start start 2024-04-25T16:03:34.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.648+0000 7f2a1c8d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a1810a2d0 0x7f2a1811b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.648+0000 7f2a1c8d0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2a1810b580 0x7f2a1811bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.648+0000 7f2a1c8d0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a18110310 0x7f2a1811ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.648+0000 7f2a1c8d0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2a18114e10 con 0x7f2a18110310 2024-04-25T16:03:34.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.648+0000 7f2a1c8d0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2a18114c90 con 0x7f2a1810b580 2024-04-25T16:03:34.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.648+0000 7f2a1c8d0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2a18114f90 con 0x7f2a1810a2d0 2024-04-25T16:03:34.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.649+0000 7f2a1759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a18110310 0x7f2a1811ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.649+0000 7f2a1759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a18110310 0x7f2a1811ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44726/0 (socket says 172.21.15.31:44726) 2024-04-25T16:03:34.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.649+0000 7f2a1759e640 1 -- 172.21.15.31:0/3621953030 learned_addr learned my addr 172.21.15.31:0/3621953030 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:34.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.649+0000 7f2a16d9d640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a1810a2d0 0x7f2a1811b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.650+0000 7f2a1759e640 1 -- 172.21.15.31:0/3621953030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a1810a2d0 msgr2=0x7f2a1811b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.650+0000 7f2a1759e640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a1810a2d0 0x7f2a1811b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.650+0000 7f2a1759e640 1 -- 172.21.15.31:0/3621953030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2a1810b580 msgr2=0x7f2a1811bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.650+0000 7f2a1759e640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2a1810b580 0x7f2a1811bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.650+0000 7f2a1759e640 1 -- 172.21.15.31:0/3621953030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a1811f480 con 0x7f2a18110310 2024-04-25T16:03:34.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.651+0000 7f2a1759e640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a18110310 0x7f2a1811ee10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800b3f0 tx=0x7f2a0800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:34.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.651+0000 7f29fffff640 1 -- 172.21.15.31:0/3621953030 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08015020 con 0x7f2a18110310 2024-04-25T16:03:34.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.651+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/3621953030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a1811f770 con 0x7f2a18110310 2024-04-25T16:03:34.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.652+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/3621953030 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a1811fd40 con 0x7f2a18110310 2024-04-25T16:03:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.652+0000 7f29fffff640 1 -- 172.21.15.31:0/3621953030 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a08011ce0 con 0x7f2a18110310 2024-04-25T16:03:34.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.653+0000 7f29fffff640 1 -- 172.21.15.31:0/3621953030 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a08010da0 con 0x7f2a18110310 2024-04-25T16:03:34.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.659+0000 7f2a23f7e740 1 Processor -- start 2024-04-25T16:03:34.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.662+0000 7f29fffff640 1 -- 172.21.15.31:0/3621953030 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2a08018020 con 0x7f2a18110310 2024-04-25T16:03:34.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.663+0000 7f29fffff640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2a000afbb0 0x7f2a000b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.664+0000 7f29fffff640 1 -- 172.21.15.31:0/3621953030 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2a080de500 con 0x7f2a18110310 2024-04-25T16:03:34.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.666+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2a000afbb0 0x7f2a000b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.669+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2a000afbb0 0x7f2a000b20c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2a10004af0 tx=0x7f2a10031a50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:34.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.669+0000 7f2a23f7e740 1 -- start start 2024-04-25T16:03:34.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.669+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 0x55ef31308a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.670+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef31308f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.670+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef31309c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.670+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ef3126cd80 con 0x55ef30f66390 2024-04-25T16:03:34.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.670+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ef31226a20 con 0x55ef30f456b0 2024-04-25T16:03:34.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.670+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ef31268d40 con 0x55ef30f3d4d0 2024-04-25T16:03:34.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.670+0000 7f2a1659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef31308f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.670+0000 7f2a1659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef31308f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38420/0 (socket says 172.21.15.31:38420) 2024-04-25T16:03:34.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.670+0000 7f2a1659c640 1 -- 172.21.15.31:0/3243680509 learned_addr learned my addr 172.21.15.31:0/3243680509 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:34.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.671+0000 7f2a1659c640 1 -- 172.21.15.31:0/3243680509 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 msgr2=0x55ef31309c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:03:34.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.671+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3243680509 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef31309c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.671+0000 7f2a1659c640 1 -- 172.21.15.31:0/3243680509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 msgr2=0x55ef31308a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.671+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3243680509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 0x55ef31308a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.671+0000 7f2a1659c640 1 -- 172.21.15.31:0/3243680509 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef31231c30 con 0x55ef30f3d4d0 2024-04-25T16:03:34.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.680+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3243680509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef31308f80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2a10035000 tx=0x7f2a1003d070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=942abc4cb2b838c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:34.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.681+0000 7f29fd7fa640 1 -- 172.21.15.31:0/3243680509 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1004c070 con 0x55ef30f3d4d0 2024-04-25T16:03:34.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.681+0000 7f29fd7fa640 1 -- 172.21.15.31:0/3243680509 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a10036050 con 0x55ef30f3d4d0 2024-04-25T16:03:34.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.681+0000 7f29fd7fa640 1 -- 172.21.15.31:0/3243680509 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a10030d40 con 0x55ef30f3d4d0 2024-04-25T16:03:34.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.687+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3243680509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 msgr2=0x55ef31308f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.687+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3243680509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef31308f80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2a10035000 tx=0x7f2a1003d070 comp rx=0 tx=0).stop 2024-04-25T16:03:34.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.694+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3243680509 shutdown_connections 2024-04-25T16:03:34.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.694+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3243680509 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef31309c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.694+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3243680509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef31308f80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.694+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3243680509 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 0x55ef31308a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.694+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3243680509 >> 172.21.15.31:0/3243680509 conn(0x55ef30f5a4e0 msgr2=0x55ef31304210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:34.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.694+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3243680509 shutdown_connections 2024-04-25T16:03:34.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.694+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3243680509 wait complete. 2024-04-25T16:03:34.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.695+0000 7f2a23f7e740 1 -- start start 2024-04-25T16:03:34.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.695+0000 7f2a23f7e740 1 Processor -- start 2024-04-25T16:03:34.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.696+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef313127f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.696+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef31312d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.696+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f66390 0x55ef31319c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.696+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ef3130a6c0 con 0x55ef30f3d4d0 2024-04-25T16:03:34.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.696+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ef311f3dc0 con 0x55ef30f456b0 2024-04-25T16:03:34.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.696+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ef3130a840 con 0x55ef30f66390 2024-04-25T16:03:34.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.696+0000 7f2a23f7e740 10 client.0 _ll_register_callbacks cb 0x55ef3120f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:34.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.697+0000 7f2a16d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef313127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.697+0000 7f2a16d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef313127f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44740/0 (socket says 172.21.15.31:44740) 2024-04-25T16:03:34.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.697+0000 7f2a16d9d640 1 -- 172.21.15.31:0/3413390331 learned_addr learned my addr 172.21.15.31:0/3413390331 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:34.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.698+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef31312d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.699+0000 7f2a16d9d640 1 -- 172.21.15.31:0/3413390331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f66390 msgr2=0x55ef31319c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.699+0000 7f2a16d9d640 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f66390 0x55ef31319c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.699+0000 7f2a16d9d640 1 -- 172.21.15.31:0/3413390331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 msgr2=0x55ef31312d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.699+0000 7f2a16d9d640 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef31312d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.699+0000 7f2a16d9d640 1 -- 172.21.15.31:0/3413390331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef30f7e3f0 con 0x55ef30f3d4d0 2024-04-25T16:03:34.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.702+0000 7f2a16d9d640 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef313127f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00bf50 tx=0x7f2a0c00d5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aca1a5c29eb5e8f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:34.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.703+0000 7f29faff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:34.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.704+0000 7f29faff5640 1 -- 172.21.15.31:0/3413390331 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a0c014430 con 0x55ef30f3d4d0 2024-04-25T16:03:34.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.705+0000 7f29faff5640 1 -- 172.21.15.31:0/3413390331 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2a0c0145d0 con 0x55ef30f3d4d0 2024-04-25T16:03:34.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.705+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3413390331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a10036050 con 0x55ef30f3d4d0 2024-04-25T16:03:34.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.706+0000 7f29faff5640 1 -- 172.21.15.31:0/3413390331 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2a0c0147d0 con 0x55ef30f3d4d0 2024-04-25T16:03:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.708+0000 7f2a23f7e740 10 client.16092 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:34.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.708+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3413390331 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef310487d0 con 0x55ef30f3d4d0 2024-04-25T16:03:34.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.710+0000 7f29faff5640 1 -- 172.21.15.31:0/3413390331 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a10036050 con 0x55ef30f3d4d0 2024-04-25T16:03:34.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.710+0000 7f29faff5640 1 -- 172.21.15.31:0/3413390331 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f2a0c013070 con 0x55ef30f3d4d0 2024-04-25T16:03:34.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.711+0000 7f2a23f7e740 10 client.16092 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:34.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.711+0000 7f2a23f7e740 10 client.16092 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:03:34.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.711+0000 7f2a23f7e740 20 client.16092 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:34.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.711+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55ef30f24f60 0x55ef31320b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.711+0000 7f2a23f7e740 4 client.16092 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:34.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.711+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3413390331 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef3127a990 con 0x55ef30f24f60 2024-04-25T16:03:34.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.713+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55ef30f24f60 0x55ef31320b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.720+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55ef30f24f60 0x55ef31320b00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cc3fe0a13a6cb389 server_cookie=6e5246007d9c370d in_seq=0 out_seq=0 2024-04-25T16:03:34.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.720+0000 7f29faff5640 10 client.16092 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:03:34.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.723+0000 7f29faff5640 1 -- 172.21.15.31:0/3413390331 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ef3127a990 con 0x55ef30f24f60 2024-04-25T16:03:34.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.723+0000 7f29faff5640 10 client.16092 handle_command_reply: tid=0 2024-04-25T16:03:34.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.723+0000 7f2a23f7e740 1 client.16092 shutdown 2024-04-25T16:03:34.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.728+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3413390331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 msgr2=0x55ef313127f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.728+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef313127f0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00bf50 tx=0x7f2a0c00d5f0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.728+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3413390331 shutdown_connections 2024-04-25T16:03:34.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.728+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55ef30f24f60 0x55ef31320b00 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.728+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f66390 0x55ef31319c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.728+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef31312d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.728+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3413390331 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef313127f0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.728+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3413390331 >> 172.21.15.31:0/3413390331 conn(0x55ef30f5a4e0 msgr2=0x55ef31307120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:34.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.729+0000 7f29faff5640 0 client.16092 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:03:34.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.731+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3413390331 shutdown_connections 2024-04-25T16:03:34.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.731+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3413390331 wait complete. 2024-04-25T16:03:34.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.735+0000 7f2a23f7e740 20 client.16092 trim_cache size 0 max 16384 2024-04-25T16:03:34.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.807+0000 7f2a23f7e740 1 Processor -- start 2024-04-25T16:03:34.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.807+0000 7f2a23f7e740 1 -- start start 2024-04-25T16:03:34.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.808+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef3130d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.809+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef3130da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.809+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 0x55ef31311dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.810+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0400e550 con 0x55ef30f66390 2024-04-25T16:03:34.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.810+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2a040100c0 con 0x55ef30f456b0 2024-04-25T16:03:34.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.810+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0400ec20 con 0x55ef30f3d4d0 2024-04-25T16:03:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.811+0000 7f2a1659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef3130da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.812+0000 7f2a1759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 0x55ef31311dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.812+0000 7f2a1759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 0x55ef31311dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44746/0 (socket says 172.21.15.31:44746) 2024-04-25T16:03:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.812+0000 7f2a1759e640 1 -- 172.21.15.31:0/3266338890 learned_addr learned my addr 172.21.15.31:0/3266338890 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:34.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.812+0000 7f2a16d9d640 1 --2- 172.21.15.31:0/3266338890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef3130d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.815+0000 7f2a1659c640 1 -- 172.21.15.31:0/3266338890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 msgr2=0x55ef3130d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.815+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3266338890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef3130d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.815+0000 7f2a1659c640 1 -- 172.21.15.31:0/3266338890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 msgr2=0x55ef31311dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.815+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3266338890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 0x55ef31311dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.815+0000 7f2a1659c640 1 -- 172.21.15.31:0/3266338890 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef30f25190 con 0x55ef30f456b0 2024-04-25T16:03:34.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.817+0000 7f2a1659c640 1 --2- 172.21.15.31:0/3266338890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef3130da60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009ca0 tx=0x7f2a1003d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfd1c6e86b6fde5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.817+0000 7f29fcff9640 1 -- 172.21.15.31:0/3266338890 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1004c040 con 0x55ef30f456b0 2024-04-25T16:03:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.823+0000 7f29fcff9640 1 -- 172.21.15.31:0/3266338890 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a10036040 con 0x55ef30f456b0 2024-04-25T16:03:34.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.823+0000 7f29fcff9640 1 -- 172.21.15.31:0/3266338890 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a100041e0 con 0x55ef30f456b0 2024-04-25T16:03:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.826+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3266338890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 msgr2=0x55ef3130da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.826+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3266338890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef3130da60 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009ca0 tx=0x7f2a1003d000 comp rx=0 tx=0).stop 2024-04-25T16:03:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.826+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3266338890 shutdown_connections 2024-04-25T16:03:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.826+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3266338890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f66390 0x55ef31311dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.826+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3266338890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30f456b0 0x55ef3130da60 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2a10009ca0 tx=0x7f2a1003d000 comp rx=0 tx=0).stop 2024-04-25T16:03:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.826+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/3266338890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f3d4d0 0x55ef3130d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.826+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3266338890 >> 172.21.15.31:0/3266338890 conn(0x55ef30f5a4e0 msgr2=0x55ef31320a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:34.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.852+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3266338890 shutdown_connections 2024-04-25T16:03:34.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.856+0000 7f2a23f7e740 1 -- 172.21.15.31:0/3266338890 wait complete. 2024-04-25T16:03:34.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.856+0000 7f2a23f7e740 1 -- start start 2024-04-25T16:03:34.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.856+0000 7f2a23f7e740 1 Processor -- start 2024-04-25T16:03:34.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.857+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef31315ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.857+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f66390 0x55ef31316430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.857+0000 7f2a23f7e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30efa3a0 0x55ef31386ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.858+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0400f9c0 con 0x55ef30f3d4d0 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.858+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0400f2f0 con 0x55ef30efa3a0 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.858+0000 7f2a23f7e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0400c530 con 0x55ef30f66390 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.859+0000 7f2a1759e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30efa3a0 0x55ef31386ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.860+0000 7f2a23f7e740 10 client.0 _ll_register_callbacks cb 0x7f2a0c00b0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.862+0000 7f2a16d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef31315ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.862+0000 7f2a16d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef31315ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44750/0 (socket says 172.21.15.31:44750) 2024-04-25T16:03:34.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.862+0000 7f2a16d9d640 1 -- 172.21.15.31:0/2335130652 learned_addr learned my addr 172.21.15.31:0/2335130652 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:34.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.866+0000 7f2a1659c640 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f66390 0x55ef31316430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.868+0000 7f2a1759e640 1 -- 172.21.15.31:0/2335130652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f66390 msgr2=0x55ef31316430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.868+0000 7f2a1759e640 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f66390 0x55ef31316430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.868+0000 7f2a1759e640 1 -- 172.21.15.31:0/2335130652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 msgr2=0x55ef31315ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:34.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.868+0000 7f2a1759e640 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef31315ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.868+0000 7f2a1759e640 1 -- 172.21.15.31:0/2335130652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef313193f0 con 0x55ef30efa3a0 2024-04-25T16:03:34.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.874+0000 7f2a1759e640 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30efa3a0 0x55ef31386ea0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2a080a6920 tx=0x7f2a080a64a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73ff26cb989d188d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:34.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.874+0000 7f29faff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:03:34.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.877+0000 7f29faff5640 1 -- 172.21.15.31:0/2335130652 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a080a6990 con 0x55ef30efa3a0 2024-04-25T16:03:34.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.877+0000 7f29faff5640 1 -- 172.21.15.31:0/2335130652 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2a080de0f0 con 0x55ef30efa3a0 2024-04-25T16:03:34.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.878+0000 7f2a23f7e740 1 -- 172.21.15.31:0/2335130652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a10036040 con 0x55ef30efa3a0 2024-04-25T16:03:34.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.879+0000 7f29faff5640 1 -- 172.21.15.31:0/2335130652 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2a080de2f0 con 0x55ef30efa3a0 2024-04-25T16:03:34.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.882+0000 7f2a23f7e740 10 client.25772 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:34.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.882+0000 7f2a23f7e740 1 -- 172.21.15.31:0/2335130652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef313882a0 con 0x55ef30efa3a0 2024-04-25T16:03:34.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.883+0000 7f29faff5640 1 -- 172.21.15.31:0/2335130652 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a080aad40 con 0x55ef30efa3a0 2024-04-25T16:03:34.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.885+0000 7f29faff5640 1 -- 172.21.15.31:0/2335130652 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f2a080a5c80 con 0x55ef30efa3a0 2024-04-25T16:03:34.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.885+0000 7f2a23f7e740 10 client.25772 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:34.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.885+0000 7f2a23f7e740 10 client.25772 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:03:34.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.886+0000 7f2a23f7e740 20 client.25772 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:34.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.886+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55ef31388c20 0x55ef3138b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:34.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.886+0000 7f2a23f7e740 4 client.25772 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:03:34.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.886+0000 7f2a23f7e740 1 -- 172.21.15.31:0/2335130652 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f2a10036040 con 0x55ef31388c20 2024-04-25T16:03:34.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.889+0000 7f2a16d9d640 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55ef31388c20 0x55ef3138b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:34.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.892+0000 7f2a16d9d640 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55ef31388c20 0x55ef3138b040 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c87edac9dea3fa8a server_cookie=a485fa701de6779 in_seq=0 out_seq=0 2024-04-25T16:03:34.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.893+0000 7f29faff5640 10 client.25772 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:03:34.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.893+0000 7f29faff5640 1 -- 172.21.15.31:0/2335130652 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f2a10036040 con 0x55ef31388c20 2024-04-25T16:03:34.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.893+0000 7f29faff5640 10 client.25772 handle_command_reply: tid=0 2024-04-25T16:03:34.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.893+0000 7f2a23f7e740 1 client.25772 shutdown 2024-04-25T16:03:34.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.902+0000 7f2a23f7e740 1 -- 172.21.15.31:0/2335130652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30efa3a0 msgr2=0x55ef31386ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.902+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30efa3a0 0x55ef31386ea0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2a080a6920 tx=0x7f2a080a64a0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.902+0000 7f2a23f7e740 1 -- 172.21.15.31:0/2335130652 shutdown_connections 2024-04-25T16:03:34.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.902+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ef30efa3a0 0x55ef31386ea0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.902+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55ef31388c20 0x55ef3138b040 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.902+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ef30f66390 0x55ef31316430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.903+0000 7f2a23f7e740 1 --2- 172.21.15.31:0/2335130652 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ef30f3d4d0 0x55ef31315ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.903+0000 7f2a23f7e740 1 -- 172.21.15.31:0/2335130652 >> 172.21.15.31:0/2335130652 conn(0x55ef30f5a4e0 msgr2=0x55ef31320a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:34.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.905+0000 7f29faff5640 0 client.25772 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:03:34.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.908+0000 7f2a23f7e740 1 -- 172.21.15.31:0/2335130652 shutdown_connections 2024-04-25T16:03:34.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.908+0000 7f2a23f7e740 1 -- 172.21.15.31:0/2335130652 wait complete. 2024-04-25T16:03:34.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.908+0000 7f2a23f7e740 20 client.25772 trim_cache size 0 max 16384 2024-04-25T16:03:34.912 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:03:34.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.913+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2a000afbb0 msgr2=0x7f2a000b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.913+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2a000afbb0 0x7f2a000b20c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f2a10004af0 tx=0x7f2a10031a50 comp rx=0 tx=0).stop 2024-04-25T16:03:34.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.914+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a18110310 msgr2=0x7f2a1811ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:34.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.914+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a18110310 0x7f2a1811ee10 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2a0800b3f0 tx=0x7f2a0800b8b0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.914+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/3621953030 shutdown_connections 2024-04-25T16:03:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.914+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2a000afbb0 0x7f2a000b20c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.914+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2a18110310 0x7f2a1811ee10 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.915+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2a1810b580 0x7f2a1811bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.915+0000 7f2a1c8d0640 1 --2- 172.21.15.31:0/3621953030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a1810a2d0 0x7f2a1811b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.915+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/3621953030 >> 172.21.15.31:0/3621953030 conn(0x7f2a180707b0 msgr2=0x7f2a18072bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.915+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/3621953030 shutdown_connections 2024-04-25T16:03:34.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:34.915+0000 7f2a1c8d0640 1 -- 172.21.15.31:0/3621953030 wait complete. 2024-04-25T16:03:34.978 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:03:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:35 smithi063 ceph-mon[32413]: pgmap v419: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 991 MiB/s wr, 2.59k op/s 2024-04-25T16:03:35.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.374+0000 7fe12a622640 1 -- 172.21.15.31:0/1988346992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe12410acf0 msgr2=0x7fe12410b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:35.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.374+0000 7fe12a622640 1 --2- 172.21.15.31:0/1988346992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe12410acf0 0x7fe12410b4e0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe11c009ca0 tx=0x7fe11c02a980 comp rx=0 tx=0).stop 2024-04-25T16:03:35.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.377+0000 7fe12a622640 1 -- 172.21.15.31:0/1988346992 shutdown_connections 2024-04-25T16:03:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.377+0000 7fe12a622640 1 --2- 172.21.15.31:0/1988346992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe12410ba20 0x7fe12410be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.377+0000 7fe12a622640 1 --2- 172.21.15.31:0/1988346992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe12410acf0 0x7fe12410b4e0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.377+0000 7fe12a622640 1 --2- 172.21.15.31:0/1988346992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe124109fc0 0x7fe12410a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.377+0000 7fe12a622640 1 -- 172.21.15.31:0/1988346992 >> 172.21.15.31:0/1988346992 conn(0x7fe124070320 msgr2=0x7fe1240727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.378+0000 7fe12a622640 1 -- 172.21.15.31:0/1988346992 shutdown_connections 2024-04-25T16:03:35.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.378+0000 7fe12a622640 1 -- 172.21.15.31:0/1988346992 wait complete. 2024-04-25T16:03:35.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.380+0000 7fe12a622640 1 Processor -- start 2024-04-25T16:03:35.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.380+0000 7fe12a622640 1 -- start start 2024-04-25T16:03:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.381+0000 7fe12a622640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124109fc0 0x7fe1241a2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.381+0000 7fe12a622640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe12410acf0 0x7fe1241a3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.381+0000 7fe12a622640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe12410ba20 0x7fe1241a9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.381+0000 7fe12a622640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe124117d90 con 0x7fe124109fc0 2024-04-25T16:03:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.381+0000 7fe12a622640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe124117c10 con 0x7fe12410acf0 2024-04-25T16:03:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.381+0000 7fe12a622640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe124117f10 con 0x7fe12410ba20 2024-04-25T16:03:35.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.382+0000 7fe129620640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124109fc0 0x7fe1241a2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.382+0000 7fe129620640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124109fc0 0x7fe1241a2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44768/0 (socket says 172.21.15.31:44768) 2024-04-25T16:03:35.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.382+0000 7fe129620640 1 -- 172.21.15.31:0/555917354 learned_addr learned my addr 172.21.15.31:0/555917354 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:35.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.383+0000 7fe129e21640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe12410ba20 0x7fe1241a9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.383+0000 7fe128e1f640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe12410acf0 0x7fe1241a3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.384+0000 7fe129e21640 1 -- 172.21.15.31:0/555917354 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe12410acf0 msgr2=0x7fe1241a3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:35.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.384+0000 7fe129e21640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe12410acf0 0x7fe1241a3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.384+0000 7fe129e21640 1 -- 172.21.15.31:0/555917354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124109fc0 msgr2=0x7fe1241a2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:35.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.384+0000 7fe129e21640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124109fc0 0x7fe1241a2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.384+0000 7fe129e21640 1 -- 172.21.15.31:0/555917354 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1241aa640 con 0x7fe12410ba20 2024-04-25T16:03:35.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.385+0000 7fe129e21640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe12410ba20 0x7fe1241a9fd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe12000b3e0 tx=0x7fe12000b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:35.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.387+0000 7fe11a7fc640 1 -- 172.21.15.31:0/555917354 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120015020 con 0x7fe12410ba20 2024-04-25T16:03:35.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.388+0000 7fe11a7fc640 1 -- 172.21.15.31:0/555917354 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe120011ce0 con 0x7fe12410ba20 2024-04-25T16:03:35.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.388+0000 7fe11a7fc640 1 -- 172.21.15.31:0/555917354 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe120010e00 con 0x7fe12410ba20 2024-04-25T16:03:35.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.390+0000 7fe12a622640 1 -- 172.21.15.31:0/555917354 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1241aa930 con 0x7fe12410ba20 2024-04-25T16:03:35.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.390+0000 7fe12a622640 1 -- 172.21.15.31:0/555917354 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1241aae00 con 0x7fe12410ba20 2024-04-25T16:03:35.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:35 smithi186 ceph-mon[33258]: pgmap v419: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 991 MiB/s wr, 2.59k op/s 2024-04-25T16:03:35.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.406+0000 7fe13280b740 1 Processor -- start 2024-04-25T16:03:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.408+0000 7fe11a7fc640 1 -- 172.21.15.31:0/555917354 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe120018020 con 0x7fe12410ba20 2024-04-25T16:03:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.410+0000 7fe13280b740 1 -- start start 2024-04-25T16:03:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.410+0000 7fe11a7fc640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1100afcb0 0x7fe1100b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.411+0000 7fe13280b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe1a1b390 0x55dfe1dbd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.411+0000 7fe13280b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dbde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.411+0000 7fe13280b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dbead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.411+0000 7fe13280b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dfe1d21d80 con 0x55dfe1a1b390 2024-04-25T16:03:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.411+0000 7fe13280b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dfe1cdba20 con 0x55dfe19f24d0 2024-04-25T16:03:35.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.411+0000 7fe13280b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dfe1d1dd40 con 0x55dfe19fa6b0 2024-04-25T16:03:35.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.411+0000 7fe11a7fc640 1 -- 172.21.15.31:0/555917354 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe1200de280 con 0x7fe12410ba20 2024-04-25T16:03:35.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.412+0000 7fe128e1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dbde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.412+0000 7fe128e1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dbde50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37082/0 (socket says 172.21.15.31:37082) 2024-04-25T16:03:35.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.412+0000 7fe128e1f640 1 -- 172.21.15.31:0/4076205004 learned_addr learned my addr 172.21.15.31:0/4076205004 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:35.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.413+0000 7fe128e1f640 1 -- 172.21.15.31:0/4076205004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19fa6b0 msgr2=0x55dfe1dbead0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:35.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.413+0000 7fe129e21640 1 --2- 172.21.15.31:0/4076205004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dbead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.413+0000 7fe128e1f640 1 --2- 172.21.15.31:0/4076205004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dbead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.413+0000 7fe128e1f640 1 -- 172.21.15.31:0/4076205004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe1a1b390 msgr2=0x55dfe1dbd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:35.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.413+0000 7fe128e1f640 1 --2- 172.21.15.31:0/4076205004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe1a1b390 0x55dfe1dbd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.413+0000 7fe128e1f640 1 -- 172.21.15.31:0/4076205004 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfe1ce6c30 con 0x55dfe19f24d0 2024-04-25T16:03:35.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.414+0000 7fe128e1f640 1 --2- 172.21.15.31:0/4076205004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dbde50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe11c02aea0 tx=0x7fe11c02fd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdae3125f78376bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:35.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.414+0000 7fe128e1f640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1100afcb0 0x7fe1100b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.415+0000 7fe128e1f640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1100afcb0 0x7fe1100b21c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe11c04a000 tx=0x7fe11c004890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.416+0000 7fe10f7fe640 1 -- 172.21.15.31:0/4076205004 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe11c04d070 con 0x55dfe19f24d0 2024-04-25T16:03:35.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.416+0000 7fe10f7fe640 1 -- 172.21.15.31:0/4076205004 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe11c003e20 con 0x55dfe19f24d0 2024-04-25T16:03:35.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.418+0000 7fe13280b740 1 -- 172.21.15.31:0/4076205004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 msgr2=0x55dfe1dbde50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:35.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.418+0000 7fe13280b740 1 --2- 172.21.15.31:0/4076205004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dbde50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe11c02aea0 tx=0x7fe11c02fd10 comp rx=0 tx=0).stop 2024-04-25T16:03:35.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.426+0000 7fe13280b740 1 -- 172.21.15.31:0/4076205004 shutdown_connections 2024-04-25T16:03:35.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.426+0000 7fe13280b740 1 --2- 172.21.15.31:0/4076205004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dbead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.426+0000 7fe13280b740 1 --2- 172.21.15.31:0/4076205004 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dbde50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.426+0000 7fe13280b740 1 --2- 172.21.15.31:0/4076205004 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe1a1b390 0x55dfe1dbd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.426+0000 7fe13280b740 1 -- 172.21.15.31:0/4076205004 >> 172.21.15.31:0/4076205004 conn(0x55dfe1a0f4e0 msgr2=0x55dfe1db91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:35.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.426+0000 7fe13280b740 1 -- 172.21.15.31:0/4076205004 shutdown_connections 2024-04-25T16:03:35.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.426+0000 7fe13280b740 1 -- 172.21.15.31:0/4076205004 wait complete. 2024-04-25T16:03:35.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.427+0000 7fe13280b740 1 -- start start 2024-04-25T16:03:35.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.427+0000 7fe13280b740 1 Processor -- start 2024-04-25T16:03:35.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.429+0000 7fe13280b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 0x55dfe1dc77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.429+0000 7fe13280b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.429+0000 7fe13280b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe1a1b390 0x55dfe1dcec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.429+0000 7fe13280b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55dfe1dbf640 con 0x55dfe19fa6b0 2024-04-25T16:03:35.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.429+0000 7fe13280b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55dfe1ca8dc0 con 0x55dfe1a1b390 2024-04-25T16:03:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.429+0000 7fe13280b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55dfe1dbf7c0 con 0x55dfe19f24d0 2024-04-25T16:03:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.429+0000 7fe13280b740 10 client.0 _ll_register_callbacks cb 0x55dfe1cc4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.430+0000 7fe129620640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 0x55dfe1dc77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.430+0000 7fe129620640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 0x55dfe1dc77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38476/0 (socket says 172.21.15.31:38476) 2024-04-25T16:03:35.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.430+0000 7fe129620640 1 -- 172.21.15.31:0/243977204 learned_addr learned my addr 172.21.15.31:0/243977204 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:35.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.431+0000 7fe128e1f640 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.431+0000 7fe129620640 1 -- 172.21.15.31:0/243977204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe1a1b390 msgr2=0x55dfe1dcec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:35.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.431+0000 7fe129620640 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe1a1b390 0x55dfe1dcec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.431+0000 7fe129620640 1 -- 172.21.15.31:0/243977204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 msgr2=0x55dfe1dc7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:35.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.431+0000 7fe129620640 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.431+0000 7fe129620640 1 -- 172.21.15.31:0/243977204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfe1a333f0 con 0x55dfe19f24d0 2024-04-25T16:03:35.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.435+0000 7fe129620640 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 0x55dfe1dc77a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe11400d1e0 tx=0x7fe11400d6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42115e04fc2306d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:35.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.449+0000 7fe10cff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:35.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.449+0000 7fe10cff9640 1 -- 172.21.15.31:0/243977204 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114018020 con 0x55dfe19f24d0 2024-04-25T16:03:35.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.449+0000 7fe10cff9640 1 -- 172.21.15.31:0/243977204 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe114014620 con 0x55dfe19f24d0 2024-04-25T16:03:35.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:35 smithi031 ceph-mon[29140]: pgmap v419: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 991 MiB/s wr, 2.59k op/s 2024-04-25T16:03:35.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.449+0000 7fe13280b740 1 -- 172.21.15.31:0/243977204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe11c003e20 con 0x55dfe19f24d0 2024-04-25T16:03:35.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.451+0000 7fe10cff9640 1 -- 172.21.15.31:0/243977204 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe114014820 con 0x55dfe19f24d0 2024-04-25T16:03:35.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.451+0000 7fe13280b740 10 client.25741 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:35.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.451+0000 7fe13280b740 1 -- 172.21.15.31:0/243977204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dfe1afd7d0 con 0x55dfe19f24d0 2024-04-25T16:03:35.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.465+0000 7fe10cff9640 1 -- 172.21.15.31:0/243977204 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe11c003e20 con 0x55dfe19f24d0 2024-04-25T16:03:35.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.467+0000 7fe10cff9640 1 -- 172.21.15.31:0/243977204 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fe114025020 con 0x55dfe19f24d0 2024-04-25T16:03:35.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.470+0000 7fe13280b740 10 client.25741 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:35.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.470+0000 7fe13280b740 10 client.25741 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:03:35.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.470+0000 7fe13280b740 20 client.25741 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:35.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.470+0000 7fe13280b740 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55dfe19d9f60 0x55dfe1de6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.470+0000 7fe13280b740 4 client.25741 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:35.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.470+0000 7fe13280b740 1 -- 172.21.15.31:0/243977204 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfe1d2f990 con 0x55dfe19d9f60 2024-04-25T16:03:35.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.476+0000 7fe128e1f640 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55dfe19d9f60 0x55dfe1de6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.487+0000 7fe128e1f640 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55dfe19d9f60 0x55dfe1de6670 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8608b8a43503a06f server_cookie=94ca50b889cc22ff in_seq=0 out_seq=0 2024-04-25T16:03:35.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.487+0000 7fe10cff9640 10 client.25741 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:03:35.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.488+0000 7fe10cff9640 1 -- 172.21.15.31:0/243977204 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55dfe1d2f990 con 0x55dfe19d9f60 2024-04-25T16:03:35.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.488+0000 7fe10cff9640 10 client.25741 handle_command_reply: tid=0 2024-04-25T16:03:35.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.488+0000 7fe13280b740 1 client.25741 shutdown 2024-04-25T16:03:35.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe13280b740 1 -- 172.21.15.31:0/243977204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 msgr2=0x55dfe1dc77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:35.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe13280b740 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 0x55dfe1dc77a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe11400d1e0 tx=0x7fe11400d6a0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe13280b740 1 -- 172.21.15.31:0/243977204 shutdown_connections 2024-04-25T16:03:35.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe13280b740 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55dfe19d9f60 0x55dfe1de6670 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe13280b740 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe1a1b390 0x55dfe1dcec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe13280b740 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe13280b740 1 --2- 172.21.15.31:0/243977204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 0x55dfe1dc77a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe13280b740 1 -- 172.21.15.31:0/243977204 >> 172.21.15.31:0/243977204 conn(0x55dfe1a0f4e0 msgr2=0x55dfe1dbc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:35.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe10cff9640 0 client.25741 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:03:35.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.491+0000 7fe13280b740 1 -- 172.21.15.31:0/243977204 shutdown_connections 2024-04-25T16:03:35.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.492+0000 7fe13280b740 1 -- 172.21.15.31:0/243977204 wait complete. 2024-04-25T16:03:35.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.506+0000 7fe13280b740 20 client.25741 trim_cache size 0 max 16384 2024-04-25T16:03:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.576+0000 7fe13280b740 1 Processor -- start 2024-04-25T16:03:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.576+0000 7fe13280b740 1 -- start start 2024-04-25T16:03:35.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.577+0000 7fe13280b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe1a0f4e0 0x55dfe1debe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.577+0000 7fe13280b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 0x55dfe1dec370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.578+0000 7fe13280b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.578+0000 7fe13280b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1241b1760 con 0x55dfe19fa6b0 2024-04-25T16:03:35.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.578+0000 7fe13280b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe1241b32d0 con 0x55dfe1a0f4e0 2024-04-25T16:03:35.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.578+0000 7fe13280b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe1241b1e30 con 0x55dfe19f24d0 2024-04-25T16:03:35.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.581+0000 7fe129e21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.582+0000 7fe129e21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44800/0 (socket says 172.21.15.31:44800) 2024-04-25T16:03:35.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.582+0000 7fe129e21640 1 -- 172.21.15.31:0/2846388012 learned_addr learned my addr 172.21.15.31:0/2846388012 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:35.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.583+0000 7fe129e21640 1 -- 172.21.15.31:0/2846388012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 msgr2=0x55dfe1dec370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:35.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.583+0000 7fe129e21640 1 --2- 172.21.15.31:0/2846388012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 0x55dfe1dec370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.583+0000 7fe129e21640 1 -- 172.21.15.31:0/2846388012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe1a0f4e0 msgr2=0x55dfe1debe30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:35.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.583+0000 7fe129e21640 1 --2- 172.21.15.31:0/2846388012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe1a0f4e0 0x55dfe1debe30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.583+0000 7fe129e21640 1 -- 172.21.15.31:0/2846388012 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfe1daaba0 con 0x55dfe19fa6b0 2024-04-25T16:03:35.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.584+0000 7fe129e21640 1 --2- 172.21.15.31:0/2846388012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc5dc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe1200a2790 tx=0x7fe1200a5e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3912ea032d767df server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:35.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.585+0000 7fe10effd640 1 -- 172.21.15.31:0/2846388012 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1200bf020 con 0x55dfe19fa6b0 2024-04-25T16:03:35.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.586+0000 7fe10effd640 1 -- 172.21.15.31:0/2846388012 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1200a6e10 con 0x55dfe19fa6b0 2024-04-25T16:03:35.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.588+0000 7fe13280b740 1 -- 172.21.15.31:0/2846388012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 msgr2=0x55dfe1dc5dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:35.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.588+0000 7fe13280b740 1 --2- 172.21.15.31:0/2846388012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc5dc0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fe1200a2790 tx=0x7fe1200a5e80 comp rx=0 tx=0).stop 2024-04-25T16:03:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.588+0000 7fe13280b740 1 -- 172.21.15.31:0/2846388012 shutdown_connections 2024-04-25T16:03:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.588+0000 7fe13280b740 1 --2- 172.21.15.31:0/2846388012 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe19fa6b0 0x55dfe1dc5dc0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.588+0000 7fe13280b740 1 --2- 172.21.15.31:0/2846388012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe19f24d0 0x55dfe1dec370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.589+0000 7fe13280b740 1 --2- 172.21.15.31:0/2846388012 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe1a0f4e0 0x55dfe1debe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.589+0000 7fe13280b740 1 -- 172.21.15.31:0/2846388012 >> 172.21.15.31:0/2846388012 conn(0x55dfe1db44b0 msgr2=0x55dfe1dbc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.589+0000 7fe13280b740 1 -- 172.21.15.31:0/2846388012 shutdown_connections 2024-04-25T16:03:35.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.589+0000 7fe13280b740 1 -- 172.21.15.31:0/2846388012 wait complete. 2024-04-25T16:03:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.602+0000 7fe13280b740 1 -- start start 2024-04-25T16:03:35.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.603+0000 7fe13280b740 1 Processor -- start 2024-04-25T16:03:35.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.604+0000 7fe13280b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dcbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.604+0000 7fe13280b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe1a0f4e0 0x55dfe1dcc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.604+0000 7fe13280b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe1a1b390 0x55dfe1dcd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.605+0000 7fe13280b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1241b2bd0 con 0x55dfe1a0f4e0 2024-04-25T16:03:35.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.605+0000 7fe13280b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe1241b2500 con 0x55dfe19f24d0 2024-04-25T16:03:35.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.605+0000 7fe13280b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe1241af7f0 con 0x55dfe1a1b390 2024-04-25T16:03:35.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.605+0000 7fe13280b740 10 client.0 _ll_register_callbacks cb 0x7fe11400b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.609+0000 7fe129620640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dcbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.609+0000 7fe129620640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dcbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37114/0 (socket says 172.21.15.31:37114) 2024-04-25T16:03:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.609+0000 7fe129620640 1 -- 172.21.15.31:0/1358335247 learned_addr learned my addr 172.21.15.31:0/1358335247 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:35.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.609+0000 7fe129e21640 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe1a1b390 0x55dfe1dcd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.610+0000 7fe129e21640 1 -- 172.21.15.31:0/1358335247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 msgr2=0x55dfe1dcbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.610+0000 7fe129e21640 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dcbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.610+0000 7fe129e21640 1 -- 172.21.15.31:0/1358335247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe1a0f4e0 msgr2=0x55dfe1dcc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.610+0000 7fe129e21640 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe1a0f4e0 0x55dfe1dcc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.610+0000 7fe129e21640 1 -- 172.21.15.31:0/1358335247 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfe1dcd6f0 con 0x55dfe1a1b390 2024-04-25T16:03:35.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.611+0000 7fe129e21640 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe1a1b390 0x55dfe1dcd130 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55dfe1db9b50 tx=0x7fe120010820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8877ec88d0df984 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:35.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.620+0000 7fe10cff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:35.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.620+0000 7fe10cff9640 1 -- 172.21.15.31:0/1358335247 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1200bf070 con 0x55dfe1a1b390 2024-04-25T16:03:35.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.620+0000 7fe13280b740 1 -- 172.21.15.31:0/1358335247 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55dfe1d29800 con 0x55dfe1a1b390 2024-04-25T16:03:35.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.620+0000 7fe10cff9640 1 -- 172.21.15.31:0/1358335247 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe1200bbd70 con 0x55dfe1a1b390 2024-04-25T16:03:35.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.621+0000 7fe10cff9640 1 -- 172.21.15.31:0/1358335247 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe1200aaac0 con 0x55dfe1a1b390 2024-04-25T16:03:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.625+0000 7fe13280b740 10 client.25744 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:35.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.625+0000 7fe13280b740 1 -- 172.21.15.31:0/1358335247 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55dfe1e4ec80 con 0x55dfe1a1b390 2024-04-25T16:03:35.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.626+0000 7fe10cff9640 1 -- 172.21.15.31:0/1358335247 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55dfe1d29800 con 0x55dfe1a1b390 2024-04-25T16:03:35.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.626+0000 7fe10cff9640 1 -- 172.21.15.31:0/1358335247 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fe120022060 con 0x55dfe1a1b390 2024-04-25T16:03:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.626+0000 7fe13280b740 10 client.25744 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.626+0000 7fe13280b740 10 client.25744 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:03:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.626+0000 7fe13280b740 20 client.25744 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.626+0000 7fe13280b740 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55dfe19af3a0 0x55dfe1e51670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.626+0000 7fe13280b740 4 client.25744 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:03:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.626+0000 7fe13280b740 1 -- 172.21.15.31:0/1358335247 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dfe1ca4610 con 0x55dfe19af3a0 2024-04-25T16:03:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.627+0000 7fe129620640 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55dfe19af3a0 0x55dfe1e51670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.627+0000 7fe10cff9640 10 client.25744 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:03:35.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.628+0000 7fe129620640 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55dfe19af3a0 0x55dfe1e51670 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7592a3dc863ef415 server_cookie=61b4a91cb0c42434 in_seq=0 out_seq=0 2024-04-25T16:03:35.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.631+0000 7fe10cff9640 1 -- 172.21.15.31:0/1358335247 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dfe1ca4610 con 0x55dfe19af3a0 2024-04-25T16:03:35.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.631+0000 7fe10cff9640 10 client.25744 handle_command_reply: tid=0 2024-04-25T16:03:35.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.632+0000 7fe13280b740 1 client.25744 shutdown 2024-04-25T16:03:35.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 -- 172.21.15.31:0/1358335247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe1a1b390 msgr2=0x55dfe1dcd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:35.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe1a1b390 0x55dfe1dcd130 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55dfe1db9b50 tx=0x7fe120010820 comp rx=0 tx=0).stop 2024-04-25T16:03:35.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 -- 172.21.15.31:0/1358335247 shutdown_connections 2024-04-25T16:03:35.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dfe1a1b390 0x55dfe1dcd130 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55dfe1a0f4e0 0x55dfe1dcc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55dfe19af3a0 0x55dfe1e51670 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 --2- 172.21.15.31:0/1358335247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55dfe19f24d0 0x55dfe1dcbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 -- 172.21.15.31:0/1358335247 >> 172.21.15.31:0/1358335247 conn(0x55dfe1db44b0 msgr2=0x55dfe1dbc660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:35.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe10cff9640 0 client.25744 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:03:35.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 -- 172.21.15.31:0/1358335247 shutdown_connections 2024-04-25T16:03:35.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.637+0000 7fe13280b740 1 -- 172.21.15.31:0/1358335247 wait complete. 2024-04-25T16:03:35.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.638+0000 7fe13280b740 20 client.25744 trim_cache size 0 max 16384 2024-04-25T16:03:35.641 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:03:35.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 -- 172.21.15.31:0/555917354 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1100afcb0 msgr2=0x7fe1100b21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:35.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1100afcb0 0x7fe1100b21c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe11c04a000 tx=0x7fe11c004890 comp rx=0 tx=0).stop 2024-04-25T16:03:35.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 -- 172.21.15.31:0/555917354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe12410ba20 msgr2=0x7fe1241a9fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:35.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe12410ba20 0x7fe1241a9fd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe12000b3e0 tx=0x7fe12000b8a0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 -- 172.21.15.31:0/555917354 shutdown_connections 2024-04-25T16:03:35.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1100afcb0 0x7fe1100b21c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe12410ba20 0x7fe1241a9fd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe12410acf0 0x7fe1241a3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 --2- 172.21.15.31:0/555917354 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe124109fc0 0x7fe1241a2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:35.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.657+0000 7fe12a622640 1 -- 172.21.15.31:0/555917354 >> 172.21.15.31:0/555917354 conn(0x7fe124070320 msgr2=0x7fe124111a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:35.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.660+0000 7fe12a622640 1 -- 172.21.15.31:0/555917354 shutdown_connections 2024-04-25T16:03:35.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:35.661+0000 7fe12a622640 1 -- 172.21.15.31:0/555917354 wait complete. 2024-04-25T16:03:35.756 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:03:36.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.342+0000 7f1e56c92640 1 -- 172.21.15.31:0/3603209532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e5010b1d0 msgr2=0x7f1e5010b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:36.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.342+0000 7f1e56c92640 1 --2- 172.21.15.31:0/3603209532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e5010b1d0 0x7f1e5010b5e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1e44009900 tx=0x7f1e4402a400 comp rx=0 tx=0).stop 2024-04-25T16:03:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.343+0000 7f1e56c92640 1 -- 172.21.15.31:0/3603209532 shutdown_connections 2024-04-25T16:03:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.343+0000 7f1e56c92640 1 --2- 172.21.15.31:0/3603209532 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e5010fef0 0x7f1e50110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.343+0000 7f1e56c92640 1 --2- 172.21.15.31:0/3603209532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e5010b1d0 0x7f1e5010b5e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.343+0000 7f1e56c92640 1 --2- 172.21.15.31:0/3603209532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5010a3e0 0x7f1e5010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.343+0000 7f1e56c92640 1 -- 172.21.15.31:0/3603209532 >> 172.21.15.31:0/3603209532 conn(0x7f1e50070400 msgr2=0x7f1e50072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.343+0000 7f1e56c92640 1 -- 172.21.15.31:0/3603209532 shutdown_connections 2024-04-25T16:03:36.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.343+0000 7f1e56c92640 1 -- 172.21.15.31:0/3603209532 wait complete. 2024-04-25T16:03:36.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.345+0000 7f1e56c92640 1 Processor -- start 2024-04-25T16:03:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.345+0000 7f1e56c92640 1 -- start start 2024-04-25T16:03:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.345+0000 7f1e56c92640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5010a3e0 0x7f1e50116ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.345+0000 7f1e56c92640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e5010b1d0 0x7f1e50117570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.345+0000 7f1e56c92640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e5010fef0 0x7f1e50118510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.345+0000 7f1e56c92640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5011e280 con 0x7f1e5010b1d0 2024-04-25T16:03:36.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.345+0000 7f1e56c92640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5011e100 con 0x7f1e5010fef0 2024-04-25T16:03:36.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.345+0000 7f1e56c92640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5011e400 con 0x7f1e5010a3e0 2024-04-25T16:03:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.349+0000 7f1e54a2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5010a3e0 0x7f1e50116ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.349+0000 7f1e54a2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5010a3e0 0x7f1e50116ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38522/0 (socket says 172.21.15.31:38522) 2024-04-25T16:03:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.349+0000 7f1e54a2e640 1 -- 172.21.15.31:0/2257899409 learned_addr learned my addr 172.21.15.31:0/2257899409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:36.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.349+0000 7f1e5522f640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e5010fef0 0x7f1e50118510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.350+0000 7f1e4ffff640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e5010b1d0 0x7f1e50117570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.350+0000 7f1e5522f640 1 -- 172.21.15.31:0/2257899409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5010a3e0 msgr2=0x7f1e50116ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.350+0000 7f1e5522f640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5010a3e0 0x7f1e50116ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.350+0000 7f1e5522f640 1 -- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e5010b1d0 msgr2=0x7f1e50117570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.350+0000 7f1e5522f640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e5010b1d0 0x7f1e50117570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.350+0000 7f1e5522f640 1 -- 172.21.15.31:0/2257899409 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e50118b00 con 0x7f1e5010fef0 2024-04-25T16:03:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.350+0000 7f1e4ffff640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e5010b1d0 0x7f1e50117570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:03:36.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.351+0000 7f1e5522f640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e5010fef0 0x7f1e50118510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800da10 tx=0x7f1e4800ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.358+0000 7f1e4dffb640 1 -- 172.21.15.31:0/2257899409 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4800bd70 con 0x7f1e5010fef0 2024-04-25T16:03:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.358+0000 7f1e4dffb640 1 -- 172.21.15.31:0/2257899409 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e48003e20 con 0x7f1e5010fef0 2024-04-25T16:03:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.358+0000 7f1e4dffb640 1 -- 172.21.15.31:0/2257899409 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e480106c0 con 0x7f1e5010fef0 2024-04-25T16:03:36.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.358+0000 7f1e56c92640 1 -- 172.21.15.31:0/2257899409 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e50118df0 con 0x7f1e5010fef0 2024-04-25T16:03:36.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.358+0000 7f1e56c92640 1 -- 172.21.15.31:0/2257899409 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e501192f0 con 0x7f1e5010fef0 2024-04-25T16:03:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.366+0000 7f1e4dffb640 1 -- 172.21.15.31:0/2257899409 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1e48010860 con 0x7f1e5010fef0 2024-04-25T16:03:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.368+0000 7f1e4dffb640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e380afd00 0x7f1e380b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.368+0000 7f1e4dffb640 1 -- 172.21.15.31:0/2257899409 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1e480dfef0 con 0x7f1e5010fef0 2024-04-25T16:03:36.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.370+0000 7f1e5dc1b740 1 Processor -- start 2024-04-25T16:03:36.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.371+0000 7f1e54a2e640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e380afd00 0x7f1e380b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.372+0000 7f1e5dc1b740 1 -- start start 2024-04-25T16:03:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.372+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fe4390 0x55ebc93869e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.372+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fbb4d0 0x55ebc9386f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.372+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fc36b0 0x55ebc9387bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.372+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ebc92ead80 con 0x55ebc8fc36b0 2024-04-25T16:03:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.372+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ebc92a4a20 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.372+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebc92e6d40 con 0x55ebc8fe4390 2024-04-25T16:03:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.374+0000 7f1e54a2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fe4390 0x55ebc93869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.374+0000 7f1e5522f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fc36b0 0x55ebc9387bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.374+0000 7f1e5522f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fc36b0 0x55ebc9387bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44840/0 (socket says 172.21.15.31:44840) 2024-04-25T16:03:36.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.374+0000 7f1e5522f640 1 -- 172.21.15.31:0/2926615892 learned_addr learned my addr 172.21.15.31:0/2926615892 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.377+0000 7f1e5522f640 1 -- 172.21.15.31:0/2926615892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fe4390 msgr2=0x55ebc93869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.377+0000 7f1e5522f640 1 --2- 172.21.15.31:0/2926615892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fe4390 0x55ebc93869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.377+0000 7f1e5522f640 1 -- 172.21.15.31:0/2926615892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fbb4d0 msgr2=0x55ebc9386f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.377+0000 7f1e5522f640 1 --2- 172.21.15.31:0/2926615892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fbb4d0 0x55ebc9386f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.377+0000 7f1e5522f640 1 -- 172.21.15.31:0/2926615892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc92afc30 con 0x55ebc8fc36b0 2024-04-25T16:03:36.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.401+0000 7f1e54a2e640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e380afd00 0x7f1e380b2210 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1e501184a0 tx=0x7f1e40013940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:36.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.402+0000 7f1e5522f640 1 --2- 172.21.15.31:0/2926615892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fc36b0 0x55ebc9387bf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1e480047c0 tx=0x7f1e480a8280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa89665d9b40542 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.407+0000 7f1e36ffd640 1 -- 172.21.15.31:0/2926615892 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e480c2070 con 0x55ebc8fc36b0 2024-04-25T16:03:36.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.407+0000 7f1e36ffd640 1 -- 172.21.15.31:0/2926615892 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e480a8380 con 0x55ebc8fc36b0 2024-04-25T16:03:36.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.407+0000 7f1e36ffd640 1 -- 172.21.15.31:0/2926615892 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48014070 con 0x55ebc8fc36b0 2024-04-25T16:03:36.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.409+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/2926615892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fc36b0 msgr2=0x55ebc9387bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.409+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/2926615892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fc36b0 0x55ebc9387bf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f1e480047c0 tx=0x7f1e480a8280 comp rx=0 tx=0).stop 2024-04-25T16:03:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.410+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/2926615892 shutdown_connections 2024-04-25T16:03:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.410+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/2926615892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fc36b0 0x55ebc9387bf0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.410+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/2926615892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fbb4d0 0x55ebc9386f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.410+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/2926615892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fe4390 0x55ebc93869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.410+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/2926615892 >> 172.21.15.31:0/2926615892 conn(0x55ebc8fd84e0 msgr2=0x55ebc93821f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:36.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.410+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/2926615892 shutdown_connections 2024-04-25T16:03:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.410+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/2926615892 wait complete. 2024-04-25T16:03:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.411+0000 7f1e5dc1b740 1 -- start start 2024-04-25T16:03:36.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.411+0000 7f1e5dc1b740 1 Processor -- start 2024-04-25T16:03:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.413+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fbb4d0 0x55ebc93907a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.413+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 0x55ebc9390ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.413+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9397c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.413+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ebc9388700 con 0x55ebc8fe4390 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.413+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ebc9271dc0 con 0x55ebc8fc36b0 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.413+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebc9388880 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.413+0000 7f1e5dc1b740 10 client.0 _ll_register_callbacks cb 0x55ebc928d600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.416+0000 7f1e5522f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9397c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.416+0000 7f1e5522f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9397c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44854/0 (socket says 172.21.15.31:44854) 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.416+0000 7f1e5522f640 1 -- 172.21.15.31:0/3134706243 learned_addr learned my addr 172.21.15.31:0/3134706243 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.416+0000 7f1e4ffff640 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 0x55ebc9390ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.417+0000 7f1e5522f640 1 -- 172.21.15.31:0/3134706243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fbb4d0 msgr2=0x55ebc93907a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.417+0000 7f1e5522f640 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fbb4d0 0x55ebc93907a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.417+0000 7f1e5522f640 1 -- 172.21.15.31:0/3134706243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 msgr2=0x55ebc9390ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.417+0000 7f1e5522f640 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 0x55ebc9390ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.418+0000 7f1e5522f640 1 -- 172.21.15.31:0/3134706243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc8ffc3f0 con 0x55ebc8fe4390 2024-04-25T16:03:36.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.418+0000 7f1e5522f640 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9397c40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1e480a4400 tx=0x7f1e480b0ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a0a739980317e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:36.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.419+0000 7f1e2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.419+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3134706243 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48014070 con 0x55ebc8fe4390 2024-04-25T16:03:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.419+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3134706243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e480a8380 con 0x55ebc8fe4390 2024-04-25T16:03:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.420+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3134706243 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1e48010d60 con 0x55ebc8fe4390 2024-04-25T16:03:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.421+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3134706243 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e480c2070 con 0x55ebc8fe4390 2024-04-25T16:03:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.421+0000 7f1e5dc1b740 10 client.16119 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.421+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3134706243 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ebc90c67d0 con 0x55ebc8fe4390 2024-04-25T16:03:36.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.422+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3134706243 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e480a8380 con 0x55ebc8fe4390 2024-04-25T16:03:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.422+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3134706243 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f1e48024060 con 0x55ebc8fe4390 2024-04-25T16:03:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.423+0000 7f1e5dc1b740 10 client.16119 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.423+0000 7f1e5dc1b740 10 client.16119 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:03:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.423+0000 7f1e5dc1b740 20 client.16119 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.423+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ebc8fa2f60 0x55ebc939e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.423+0000 7f1e5dc1b740 4 client.16119 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.423+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3134706243 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebc92f8990 con 0x55ebc8fa2f60 2024-04-25T16:03:36.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.424+0000 7f1e54a2e640 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ebc8fa2f60 0x55ebc939e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.430+0000 7f1e54a2e640 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ebc8fa2f60 0x55ebc939e490 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c87c6db92020c09d server_cookie=c8dc17f9afe6f924 in_seq=0 out_seq=0 2024-04-25T16:03:36.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.430+0000 7f1e2ffff640 10 client.16119 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:03:36.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.435+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3134706243 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ebc92f8990 con 0x55ebc8fa2f60 2024-04-25T16:03:36.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.435+0000 7f1e2ffff640 10 client.16119 handle_command_reply: tid=0 2024-04-25T16:03:36.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.435+0000 7f1e5dc1b740 1 client.16119 shutdown 2024-04-25T16:03:36.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.438+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3134706243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 msgr2=0x55ebc9397c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:36.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.438+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9397c40 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f1e480a4400 tx=0x7f1e480b0ff0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.438+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3134706243 shutdown_connections 2024-04-25T16:03:36.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.438+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ebc8fa2f60 0x55ebc939e490 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.438+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9397c40 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.438+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 0x55ebc9390ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.438+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3134706243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fbb4d0 0x55ebc93907a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.439+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3134706243 >> 172.21.15.31:0/3134706243 conn(0x55ebc8fd84e0 msgr2=0x55ebc93813f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:36.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.439+0000 7f1e2ffff640 0 client.16119 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:03:36.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.440+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3134706243 shutdown_connections 2024-04-25T16:03:36.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.440+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3134706243 wait complete. 2024-04-25T16:03:36.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.441+0000 7f1e5dc1b740 20 client.16119 trim_cache size 0 max 16384 2024-04-25T16:03:36.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.535+0000 7f1e5dc1b740 1 Processor -- start 2024-04-25T16:03:36.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.540+0000 7f1e5dc1b740 1 -- start start 2024-04-25T16:03:36.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.542+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fbb4d0 0x55ebc938b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.542+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 0x55ebc938ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.542+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9386960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.542+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c00e4c0 con 0x55ebc8fe4390 2024-04-25T16:03:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.542+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c010030 con 0x55ebc8fc36b0 2024-04-25T16:03:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.542+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c00eb90 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.543+0000 7f1e5522f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9386960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.543+0000 7f1e5522f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9386960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44866/0 (socket says 172.21.15.31:44866) 2024-04-25T16:03:36.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.543+0000 7f1e5522f640 1 -- 172.21.15.31:0/467263409 learned_addr learned my addr 172.21.15.31:0/467263409 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.544+0000 7f1e4ffff640 1 --2- 172.21.15.31:0/467263409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 0x55ebc938ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.544+0000 7f1e5522f640 1 -- 172.21.15.31:0/467263409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fbb4d0 msgr2=0x55ebc938b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.544+0000 7f1e5522f640 1 --2- 172.21.15.31:0/467263409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fbb4d0 0x55ebc938b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.544+0000 7f1e5522f640 1 -- 172.21.15.31:0/467263409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 msgr2=0x55ebc938ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:36.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.544+0000 7f1e5522f640 1 --2- 172.21.15.31:0/467263409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 0x55ebc938ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.544+0000 7f1e5522f640 1 -- 172.21.15.31:0/467263409 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc9388970 con 0x55ebc8fe4390 2024-04-25T16:03:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.545+0000 7f1e5522f640 1 --2- 172.21.15.31:0/467263409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9386960 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1e480ac8f0 tx=0x7f1e480b0d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235a2d425d3a82a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.545+0000 7f1e367fc640 1 -- 172.21.15.31:0/467263409 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48014050 con 0x55ebc8fe4390 2024-04-25T16:03:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.545+0000 7f1e367fc640 1 -- 172.21.15.31:0/467263409 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e480c8030 con 0x55ebc8fe4390 2024-04-25T16:03:36.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.546+0000 7f1e367fc640 1 -- 172.21.15.31:0/467263409 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e480dde60 con 0x55ebc8fe4390 2024-04-25T16:03:36.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.548+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/467263409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 msgr2=0x55ebc9386960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:36.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.548+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/467263409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9386960 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f1e480ac8f0 tx=0x7f1e480b0d50 comp rx=0 tx=0).stop 2024-04-25T16:03:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.549+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/467263409 shutdown_connections 2024-04-25T16:03:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.549+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/467263409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fe4390 0x55ebc9386960 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.549+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/467263409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fc36b0 0x55ebc938ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.549+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/467263409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fbb4d0 0x55ebc938b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.549+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/467263409 >> 172.21.15.31:0/467263409 conn(0x55ebc8fd84e0 msgr2=0x55ebc9381b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:36.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.554+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/467263409 shutdown_connections 2024-04-25T16:03:36.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.554+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/467263409 wait complete. 2024-04-25T16:03:36.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.555+0000 7f1e5dc1b740 1 -- start start 2024-04-25T16:03:36.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.555+0000 7f1e5dc1b740 1 Processor -- start 2024-04-25T16:03:36.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.557+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fbb4d0 0x55ebc9396e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.557+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fc36b0 0x55ebc93939f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.557+0000 7f1e5dc1b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fe4390 0x55ebc9393f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.557+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c00f930 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.558+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c00f260 con 0x55ebc8fe4390 2024-04-25T16:03:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.558+0000 7f1e5dc1b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3c00c4a0 con 0x55ebc8fc36b0 2024-04-25T16:03:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.558+0000 7f1e4ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fc36b0 0x55ebc93939f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.558+0000 7f1e4ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fc36b0 0x55ebc93939f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38554/0 (socket says 172.21.15.31:38554) 2024-04-25T16:03:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.558+0000 7f1e4ffff640 1 -- 172.21.15.31:0/3783474048 learned_addr learned my addr 172.21.15.31:0/3783474048 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.558+0000 7f1e5dc1b740 10 client.0 _ll_register_callbacks cb 0x7f1e480c0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:36.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.558+0000 7f1e54a2e640 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fbb4d0 0x55ebc9396e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.559+0000 7f1e54a2e640 1 -- 172.21.15.31:0/3783474048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fc36b0 msgr2=0x55ebc93939f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:36.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.559+0000 7f1e54a2e640 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fc36b0 0x55ebc93939f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.559+0000 7f1e54a2e640 1 -- 172.21.15.31:0/3783474048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fe4390 msgr2=0x55ebc9393f30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:36.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.559+0000 7f1e54a2e640 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fe4390 0x55ebc9393f30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.559+0000 7f1e54a2e640 1 -- 172.21.15.31:0/3783474048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc93974c0 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.560+0000 7f1e54a2e640 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fbb4d0 0x55ebc9396e20 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x55ebc939e8f0 tx=0x7f1e400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d037cef2bf31855 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:36.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.560+0000 7f1e2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:36.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.562+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3783474048 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e40026370 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.563+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3783474048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e480c8030 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.563+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3783474048 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1e40026510 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.564+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3783474048 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1e40026710 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.564+0000 7f1e5dc1b740 10 client.16125 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:36.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.564+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3783474048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ebc9405710 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.566+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3783474048 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e480c8030 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.568+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3783474048 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f1e40026ce0 con 0x55ebc8fbb4d0 2024-04-25T16:03:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.569+0000 7f1e5dc1b740 10 client.16125 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.569+0000 7f1e5dc1b740 10 client.16125 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:03:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.569+0000 7f1e5dc1b740 20 client.16125 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.569+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ebc8f783a0 0x55ebc94080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.569+0000 7f1e5dc1b740 4 client.16125 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:03:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.569+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3783474048 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ebc92f2800 con 0x55ebc8f783a0 2024-04-25T16:03:36.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.573+0000 7f1e4ffff640 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ebc8f783a0 0x55ebc94080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.576+0000 7f1e4ffff640 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ebc8f783a0 0x55ebc94080b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed3254bd51ed3cb8 server_cookie=116220a03d0fe361 in_seq=0 out_seq=0 2024-04-25T16:03:36.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.576+0000 7f1e2ffff640 10 client.16125 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:03:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.576+0000 7f1e2ffff640 1 -- 172.21.15.31:0/3783474048 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ebc92f2800 con 0x55ebc8f783a0 2024-04-25T16:03:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.577+0000 7f1e2ffff640 10 client.16125 handle_command_reply: tid=0 2024-04-25T16:03:36.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.577+0000 7f1e5dc1b740 1 client.16125 shutdown 2024-04-25T16:03:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.580+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3783474048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fbb4d0 msgr2=0x55ebc9396e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.580+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fbb4d0 0x55ebc9396e20 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x55ebc939e8f0 tx=0x7f1e400046c0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.580+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3783474048 shutdown_connections 2024-04-25T16:03:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.580+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ebc8fe4390 0x55ebc9393f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.580+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebc8fc36b0 0x55ebc93939f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.580+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ebc8f783a0 0x55ebc94080b0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.580+0000 7f1e5dc1b740 1 --2- 172.21.15.31:0/3783474048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ebc8fbb4d0 0x55ebc9396e20 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.580+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3783474048 >> 172.21.15.31:0/3783474048 conn(0x55ebc8fd84e0 msgr2=0x55ebc9381a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.582+0000 7f1e2ffff640 0 client.16125 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:03:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.582+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3783474048 shutdown_connections 2024-04-25T16:03:36.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.583+0000 7f1e5dc1b740 1 -- 172.21.15.31:0/3783474048 wait complete. 2024-04-25T16:03:36.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.583+0000 7f1e5dc1b740 20 client.16125 trim_cache size 0 max 16384 2024-04-25T16:03:36.586 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:03:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 -- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e380afd00 msgr2=0x7f1e380b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e380afd00 0x7f1e380b2210 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1e501184a0 tx=0x7f1e40013940 comp rx=0 tx=0).stop 2024-04-25T16:03:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 -- 172.21.15.31:0/2257899409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e5010fef0 msgr2=0x7f1e50118510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e5010fef0 0x7f1e50118510 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1e4800da10 tx=0x7f1e4800ded0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 -- 172.21.15.31:0/2257899409 shutdown_connections 2024-04-25T16:03:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e380afd00 0x7f1e380b2210 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e5010fef0 0x7f1e50118510 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e5010b1d0 0x7f1e50117570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 --2- 172.21.15.31:0/2257899409 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5010a3e0 0x7f1e50116ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:36.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.587+0000 7f1e56c92640 1 -- 172.21.15.31:0/2257899409 >> 172.21.15.31:0/2257899409 conn(0x7f1e50070400 msgr2=0x7f1e5010bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:36.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.590+0000 7f1e56c92640 1 -- 172.21.15.31:0/2257899409 shutdown_connections 2024-04-25T16:03:36.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:36.594+0000 7f1e56c92640 1 -- 172.21.15.31:0/2257899409 wait complete. 2024-04-25T16:03:36.646 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:03:37.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.099+0000 7f95cc9ba640 1 -- 172.21.15.31:0/3405860771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c810a0e0 msgr2=0x7f95c810a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.099+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/3405860771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c810a0e0 0x7f95c810a4f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009ca0 tx=0x7f95c002ada0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.099+0000 7f95cc9ba640 1 -- 172.21.15.31:0/3405860771 shutdown_connections 2024-04-25T16:03:37.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.099+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/3405860771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95c8078df0 0x7f95c807b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.099+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/3405860771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c810a0e0 0x7f95c810a4f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.099+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/3405860771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c810b4c0 0x7f95c8109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.099+0000 7f95cc9ba640 1 -- 172.21.15.31:0/3405860771 >> 172.21.15.31:0/3405860771 conn(0x7f95c8070400 msgr2=0x7f95c8072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.099+0000 7f95cc9ba640 1 -- 172.21.15.31:0/3405860771 shutdown_connections 2024-04-25T16:03:37.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.099+0000 7f95cc9ba640 1 -- 172.21.15.31:0/3405860771 wait complete. 2024-04-25T16:03:37.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.101+0000 7f95cc9ba640 1 Processor -- start 2024-04-25T16:03:37.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.101+0000 7f95cc9ba640 1 -- start start 2024-04-25T16:03:37.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.101+0000 7f95cc9ba640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c8078df0 0x7f95c80847a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.101+0000 7f95cc9ba640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95c810a0e0 0x7f95c8084ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.102+0000 7f95cc9ba640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c810b4c0 0x7f95c8133f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.102+0000 7f95cc9ba640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95c807e9a0 con 0x7f95c8078df0 2024-04-25T16:03:37.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.102+0000 7f95cc9ba640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95c807e820 con 0x7f95c810b4c0 2024-04-25T16:03:37.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.102+0000 7f95cc9ba640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95c807eb20 con 0x7f95c810a0e0 2024-04-25T16:03:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.105+0000 7f95c6d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c810b4c0 0x7f95c8133f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.105+0000 7f95c5d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95c810a0e0 0x7f95c8084ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.105+0000 7f95c659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c8078df0 0x7f95c80847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.105+0000 7f95c659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c8078df0 0x7f95c80847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44892/0 (socket says 172.21.15.31:44892) 2024-04-25T16:03:37.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.105+0000 7f95c659c640 1 -- 172.21.15.31:0/334181568 learned_addr learned my addr 172.21.15.31:0/334181568 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.119+0000 7f95c6d9d640 1 -- 172.21.15.31:0/334181568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95c810a0e0 msgr2=0x7f95c8084ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.119+0000 7f95c6d9d640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95c810a0e0 0x7f95c8084ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.119+0000 7f95c6d9d640 1 -- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c8078df0 msgr2=0x7f95c80847a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.119+0000 7f95c6d9d640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c8078df0 0x7f95c80847a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.119+0000 7f95c6d9d640 1 -- 172.21.15.31:0/334181568 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c8134440 con 0x7f95c810b4c0 2024-04-25T16:03:37.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.120+0000 7f95c659c640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c8078df0 0x7f95c80847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:03:37.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.120+0000 7f95c6d9d640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c810b4c0 0x7f95c8133f00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00e970 tx=0x7f95bc00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.126+0000 7f95b77fe640 1 -- 172.21.15.31:0/334181568 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc00ccc0 con 0x7f95c810b4c0 2024-04-25T16:03:37.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.142+0000 7f95cc9ba640 1 -- 172.21.15.31:0/334181568 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c8134730 con 0x7f95c810b4c0 2024-04-25T16:03:37.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.142+0000 7f95cc9ba640 1 -- 172.21.15.31:0/334181568 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c8134c80 con 0x7f95c810b4c0 2024-04-25T16:03:37.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.157+0000 7f95d3860740 1 Processor -- start 2024-04-25T16:03:37.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.158+0000 7f95d3860740 1 -- start start 2024-04-25T16:03:37.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.158+0000 7f95d3860740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e59a390 0x55c11e93c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.158+0000 7f95d3860740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e93cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.158+0000 7f95d3860740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 0x55c11e93dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.158+0000 7f95d3860740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c11e8a0d80 con 0x55c11e5714d0 2024-04-25T16:03:37.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.158+0000 7f95d3860740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c11e85aa20 con 0x55c11e5796b0 2024-04-25T16:03:37.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.158+0000 7f95d3860740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c11e89cd40 con 0x55c11e59a390 2024-04-25T16:03:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.159+0000 7f95b77fe640 1 -- 172.21.15.31:0/334181568 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95bc005020 con 0x7f95c810b4c0 2024-04-25T16:03:37.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.159+0000 7f95b77fe640 1 -- 172.21.15.31:0/334181568 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95bc004a00 con 0x7f95c810b4c0 2024-04-25T16:03:37.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.160+0000 7f95c5d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e59a390 0x55c11e93c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.160+0000 7f95c5d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e59a390 0x55c11e93c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38590/0 (socket says 172.21.15.31:38590) 2024-04-25T16:03:37.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.161+0000 7f95c5d9b640 1 -- 172.21.15.31:0/1640103387 learned_addr learned my addr 172.21.15.31:0/1640103387 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.161+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/1640103387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 0x55c11e93dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.164+0000 7f95c5d9b640 1 -- 172.21.15.31:0/1640103387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 msgr2=0x55c11e93dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.164+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/1640103387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 0x55c11e93dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.164+0000 7f95c5d9b640 1 -- 172.21.15.31:0/1640103387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 msgr2=0x55c11e93cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.164+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/1640103387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e93cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.164+0000 7f95c5d9b640 1 -- 172.21.15.31:0/1640103387 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c11e865c30 con 0x55c11e59a390 2024-04-25T16:03:37.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.164+0000 7f95b77fe640 1 -- 172.21.15.31:0/334181568 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f95bc004ba0 con 0x7f95c810b4c0 2024-04-25T16:03:37.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.165+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/1640103387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e59a390 0x55c11e93c9e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f95c0036040 tx=0x7f95c002d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c31e3f6561325ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.168+0000 7f95b4ff9640 1 -- 172.21.15.31:0/1640103387 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0047070 con 0x55c11e59a390 2024-04-25T16:03:37.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.169+0000 7f95b4ff9640 1 -- 172.21.15.31:0/1640103387 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c000b570 con 0x55c11e59a390 2024-04-25T16:03:37.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.177+0000 7f95d3860740 1 -- 172.21.15.31:0/1640103387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e59a390 msgr2=0x55c11e93c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.177+0000 7f95d3860740 1 --2- 172.21.15.31:0/1640103387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e59a390 0x55c11e93c9e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f95c0036040 tx=0x7f95c002d040 comp rx=0 tx=0).stop 2024-04-25T16:03:37.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.178+0000 7f95d3860740 1 -- 172.21.15.31:0/1640103387 shutdown_connections 2024-04-25T16:03:37.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.178+0000 7f95d3860740 1 --2- 172.21.15.31:0/1640103387 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 0x55c11e93dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.178+0000 7f95d3860740 1 --2- 172.21.15.31:0/1640103387 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e93cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.178+0000 7f95d3860740 1 --2- 172.21.15.31:0/1640103387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e59a390 0x55c11e93c9e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.178+0000 7f95d3860740 1 -- 172.21.15.31:0/1640103387 >> 172.21.15.31:0/1640103387 conn(0x55c11e58e4e0 msgr2=0x55c11e9381f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.179+0000 7f95b77fe640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95b00afbd0 0x7f95b00b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.181+0000 7f95b77fe640 1 -- 172.21.15.31:0/334181568 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f95bc0dea80 con 0x7f95c810b4c0 2024-04-25T16:03:37.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.189+0000 7f95c659c640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95b00afbd0 0x7f95b00b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.189+0000 7f95d3860740 1 -- 172.21.15.31:0/1640103387 shutdown_connections 2024-04-25T16:03:37.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.189+0000 7f95d3860740 1 -- 172.21.15.31:0/1640103387 wait complete. 2024-04-25T16:03:37.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.190+0000 7f95d3860740 1 -- start start 2024-04-25T16:03:37.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.190+0000 7f95d3860740 1 Processor -- start 2024-04-25T16:03:37.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.191+0000 7f95d3860740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5714d0 0x55c11e9465b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.191+0000 7f95d3860740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 0x55c11e946af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.191+0000 7f95d3860740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e558f60 0x55c11e94da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.191+0000 7f95d3860740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c11e93e6f0 con 0x55c11e558f60 2024-04-25T16:03:37.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.191+0000 7f95d3860740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c11e827dc0 con 0x55c11e5796b0 2024-04-25T16:03:37.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.191+0000 7f95d3860740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c11e93e870 con 0x55c11e5714d0 2024-04-25T16:03:37.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.191+0000 7f95d3860740 10 client.0 _ll_register_callbacks cb 0x55c11e843600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:37.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.192+0000 7f95c659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5714d0 0x55c11e9465b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.192+0000 7f95c659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5714d0 0x55c11e9465b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38598/0 (socket says 172.21.15.31:38598) 2024-04-25T16:03:37.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.192+0000 7f95c659c640 1 -- 172.21.15.31:0/222458975 learned_addr learned my addr 172.21.15.31:0/222458975 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.192+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 0x55c11e946af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.193+0000 7f95c659c640 1 -- 172.21.15.31:0/222458975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 msgr2=0x55c11e946af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.193+0000 7f95c659c640 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 0x55c11e946af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.193+0000 7f95c659c640 1 -- 172.21.15.31:0/222458975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e558f60 msgr2=0x55c11e94da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.193+0000 7f95c659c640 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e558f60 0x55c11e94da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.193+0000 7f95c659c640 1 -- 172.21.15.31:0/222458975 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11e5b23f0 con 0x55c11e5714d0 2024-04-25T16:03:37.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.195+0000 7f95c659c640 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5714d0 0x55c11e9465b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f95b8013880 tx=0x7f95b8013d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98ad60865a852288 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.206+0000 7f95adffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.206+0000 7f95adffb640 1 -- 172.21.15.31:0/222458975 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b8022060 con 0x55c11e5714d0 2024-04-25T16:03:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.206+0000 7f95d3860740 1 -- 172.21.15.31:0/222458975 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55c11e8ae990 con 0x55c11e5714d0 2024-04-25T16:03:37.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.206+0000 7f95adffb640 1 -- 172.21.15.31:0/222458975 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f95b8010b00 con 0x55c11e5714d0 2024-04-25T16:03:37.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.207+0000 7f95adffb640 1 -- 172.21.15.31:0/222458975 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f95b801d420 con 0x55c11e5714d0 2024-04-25T16:03:37.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.207+0000 7f95d3860740 10 client.25765 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:37.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.208+0000 7f95d3860740 1 -- 172.21.15.31:0/222458975 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c11e67c7d0 con 0x55c11e5714d0 2024-04-25T16:03:37.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.209+0000 7f95c659c640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95b00afbd0 0x7f95b00b20e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f95b8026870 tx=0x7f95b8026c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.209+0000 7f95adffb640 1 -- 172.21.15.31:0/222458975 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55c11e8ae990 con 0x55c11e5714d0 2024-04-25T16:03:37.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.210+0000 7f95adffb640 1 -- 172.21.15.31:0/222458975 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f95b802b9a0 con 0x55c11e5714d0 2024-04-25T16:03:37.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.210+0000 7f95d3860740 10 client.25765 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:37.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.210+0000 7f95d3860740 10 client.25765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:37.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.210+0000 7f95d3860740 20 client.25765 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:37.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.210+0000 7f95d3860740 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c11e52e3a0 0x55c11e9548d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.210+0000 7f95d3860740 4 client.25765 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:37.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.210+0000 7f95d3860740 1 -- 172.21.15.31:0/222458975 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c11e8a8800 con 0x55c11e52e3a0 2024-04-25T16:03:37.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.215+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c11e52e3a0 0x55c11e9548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.243+0000 7f95adffb640 10 client.25765 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:37.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.243+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c11e52e3a0 0x55c11e9548d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f0d6df939288e450 server_cookie=90ed40dcccc83716 in_seq=0 out_seq=0 2024-04-25T16:03:37.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.250+0000 7f95adffb640 1 -- 172.21.15.31:0/222458975 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c11e8a8800 con 0x55c11e52e3a0 2024-04-25T16:03:37.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.250+0000 7f95adffb640 10 client.25765 handle_command_reply: tid=0 2024-04-25T16:03:37.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.250+0000 7f95d3860740 1 client.25765 shutdown 2024-04-25T16:03:37.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.251+0000 7f95d3860740 1 -- 172.21.15.31:0/222458975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5714d0 msgr2=0x55c11e9465b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.251+0000 7f95d3860740 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5714d0 0x55c11e9465b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f95b8013880 tx=0x7f95b8013d40 comp rx=0 tx=0).stop 2024-04-25T16:03:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.258+0000 7f95d3860740 1 -- 172.21.15.31:0/222458975 shutdown_connections 2024-04-25T16:03:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.258+0000 7f95d3860740 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c11e52e3a0 0x55c11e9548d0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.258+0000 7f95d3860740 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e558f60 0x55c11e94da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.258+0000 7f95d3860740 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e5796b0 0x55c11e946af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.258+0000 7f95d3860740 1 --2- 172.21.15.31:0/222458975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5714d0 0x55c11e9465b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.258+0000 7f95d3860740 1 -- 172.21.15.31:0/222458975 >> 172.21.15.31:0/222458975 conn(0x55c11e58e4e0 msgr2=0x55c11e9373f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.258+0000 7f95adffb640 0 client.25765 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.261+0000 7f95d3860740 1 -- 172.21.15.31:0/222458975 shutdown_connections 2024-04-25T16:03:37.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.261+0000 7f95d3860740 1 -- 172.21.15.31:0/222458975 wait complete. 2024-04-25T16:03:37.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.262+0000 7f95d3860740 20 client.25765 trim_cache size 0 max 16384 2024-04-25T16:03:37.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.360+0000 7f95d3860740 1 Processor -- start 2024-04-25T16:03:37.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.360+0000 7f95d3860740 1 -- start start 2024-04-25T16:03:37.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.360+0000 7f95d3860740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 0x55c11e938330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.360+0000 7f95d3860740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e9449a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.360+0000 7f95d3860740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5796b0 0x55c11e944ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.360+0000 7f95d3860740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95a800e570 con 0x55c11e5714d0 2024-04-25T16:03:37.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.360+0000 7f95d3860740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95a80100e0 con 0x55c11e558f60 2024-04-25T16:03:37.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.360+0000 7f95d3860740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95a800ec40 con 0x55c11e5796b0 2024-04-25T16:03:37.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.361+0000 7f95c5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e9449a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.361+0000 7f95c5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e9449a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44906/0 (socket says 172.21.15.31:44906) 2024-04-25T16:03:37.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.361+0000 7f95c5d9b640 1 -- 172.21.15.31:0/4179904936 learned_addr learned my addr 172.21.15.31:0/4179904936 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.362+0000 7f95c5d9b640 1 -- 172.21.15.31:0/4179904936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5796b0 msgr2=0x55c11e944ee0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:37.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.362+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/4179904936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5796b0 0x55c11e944ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.362+0000 7f95c5d9b640 1 -- 172.21.15.31:0/4179904936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 msgr2=0x55c11e938330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:37.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.362+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/4179904936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 0x55c11e938330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.362+0000 7f95c5d9b640 1 -- 172.21.15.31:0/4179904936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c11e941c90 con 0x55c11e5714d0 2024-04-25T16:03:37.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.363+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/4179904936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e9449a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f95c0036010 tx=0x7f95c002ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5098e02806e69cd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.367+0000 7f95affff640 1 -- 172.21.15.31:0/4179904936 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0047040 con 0x55c11e5714d0 2024-04-25T16:03:37.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.367+0000 7f95affff640 1 -- 172.21.15.31:0/4179904936 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c0053070 con 0x55c11e5714d0 2024-04-25T16:03:37.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.367+0000 7f95affff640 1 -- 172.21.15.31:0/4179904936 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0031800 con 0x55c11e5714d0 2024-04-25T16:03:37.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:37 smithi063 ceph-mon[32413]: pgmap v420: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 794 MiB/s wr, 2.41k op/s 2024-04-25T16:03:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.370+0000 7f95d3860740 1 -- 172.21.15.31:0/4179904936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 msgr2=0x55c11e9449a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.370+0000 7f95d3860740 1 --2- 172.21.15.31:0/4179904936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e9449a0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f95c0036010 tx=0x7f95c002ada0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.375+0000 7f95d3860740 1 -- 172.21.15.31:0/4179904936 shutdown_connections 2024-04-25T16:03:37.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.375+0000 7f95d3860740 1 --2- 172.21.15.31:0/4179904936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5796b0 0x55c11e944ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.375+0000 7f95d3860740 1 --2- 172.21.15.31:0/4179904936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e9449a0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.375+0000 7f95d3860740 1 --2- 172.21.15.31:0/4179904936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 0x55c11e938330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.375+0000 7f95d3860740 1 -- 172.21.15.31:0/4179904936 >> 172.21.15.31:0/4179904936 conn(0x55c11e58e4e0 msgr2=0x55c11e937c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.376+0000 7f95d3860740 1 -- 172.21.15.31:0/4179904936 shutdown_connections 2024-04-25T16:03:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.376+0000 7f95d3860740 1 -- 172.21.15.31:0/4179904936 wait complete. 2024-04-25T16:03:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.377+0000 7f95d3860740 1 -- start start 2024-04-25T16:03:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.377+0000 7f95d3860740 1 Processor -- start 2024-04-25T16:03:37.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.378+0000 7f95d3860740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 0x55c11e94ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.378+0000 7f95d3860740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e94cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.378+0000 7f95d3860740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5796b0 0x55c11e9495e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.378+0000 7f95d3860740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f95a800f9e0 con 0x55c11e5714d0 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.378+0000 7f95d3860740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f95a800f310 con 0x55c11e558f60 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.378+0000 7f95d3860740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f95a800c550 con 0x55c11e5796b0 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.378+0000 7f95d3860740 10 client.0 _ll_register_callbacks cb 0x7f95b8009890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.379+0000 7f95c5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e94cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.379+0000 7f95c5d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e94cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44910/0 (socket says 172.21.15.31:44910) 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.379+0000 7f95c5d9b640 1 -- 172.21.15.31:0/1811672785 learned_addr learned my addr 172.21.15.31:0/1811672785 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.379+0000 7f95c6d9d640 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5796b0 0x55c11e9495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.380+0000 7f95c659c640 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 0x55c11e94ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.380+0000 7f95c5d9b640 1 -- 172.21.15.31:0/1811672785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5796b0 msgr2=0x55c11e9495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.380+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5796b0 0x55c11e9495e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.380+0000 7f95c5d9b640 1 -- 172.21.15.31:0/1811672785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 msgr2=0x55c11e94ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.380+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 0x55c11e94ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.380+0000 7f95c5d9b640 1 -- 172.21.15.31:0/1811672785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c11e949d20 con 0x55c11e5714d0 2024-04-25T16:03:37.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.381+0000 7f95c5d9b640 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e94cfa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009a60 tx=0x7f95c002bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f4a6bf980f98ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.381+0000 7f95adffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:37.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.383+0000 7f95adffb640 1 -- 172.21.15.31:0/1811672785 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0047070 con 0x55c11e5714d0 2024-04-25T16:03:37.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.383+0000 7f95adffb640 1 -- 172.21.15.31:0/1811672785 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f95c000b3f0 con 0x55c11e5714d0 2024-04-25T16:03:37.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.383+0000 7f95d3860740 1 -- 172.21.15.31:0/1811672785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95c0053070 con 0x55c11e5714d0 2024-04-25T16:03:37.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.385+0000 7f95adffb640 1 -- 172.21.15.31:0/1811672785 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f95c0003970 con 0x55c11e5714d0 2024-04-25T16:03:37.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.385+0000 7f95d3860740 10 client.16137 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:37.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.385+0000 7f95d3860740 1 -- 172.21.15.31:0/1811672785 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c11e9bb8c0 con 0x55c11e5714d0 2024-04-25T16:03:37.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.385+0000 7f95c659c640 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 0x55c11e94ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:03:37.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.387+0000 7f95adffb640 1 -- 172.21.15.31:0/1811672785 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c0053070 con 0x55c11e5714d0 2024-04-25T16:03:37.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.387+0000 7f95adffb640 1 -- 172.21.15.31:0/1811672785 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f95c000b050 con 0x55c11e5714d0 2024-04-25T16:03:37.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.387+0000 7f95d3860740 10 client.16137 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:37.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.387+0000 7f95d3860740 10 client.16137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:37.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.387+0000 7f95d3860740 20 client.16137 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:37.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.387+0000 7f95d3860740 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c11e9bbdd0 0x55c11e9be1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.387+0000 7f95d3860740 4 client.16137 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:03:37.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.387+0000 7f95d3860740 1 -- 172.21.15.31:0/1811672785 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c11e823610 con 0x55c11e9bbdd0 2024-04-25T16:03:37.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.391+0000 7f95c659c640 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c11e9bbdd0 0x55c11e9be1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:37 smithi186 ceph-mon[33258]: pgmap v420: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 794 MiB/s wr, 2.41k op/s 2024-04-25T16:03:37.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.399+0000 7f95c659c640 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c11e9bbdd0 0x55c11e9be1f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddd6ab6779c6f675 server_cookie=3eb03bdb4d93638d in_seq=0 out_seq=0 2024-04-25T16:03:37.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.400+0000 7f95adffb640 10 client.16137 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.407+0000 7f95adffb640 1 -- 172.21.15.31:0/1811672785 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c11e823610 con 0x55c11e9bbdd0 2024-04-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.407+0000 7f95adffb640 10 client.16137 handle_command_reply: tid=0 2024-04-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.407+0000 7f95d3860740 1 client.16137 shutdown 2024-04-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.407+0000 7f95d3860740 1 -- 172.21.15.31:0/1811672785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 msgr2=0x55c11e94cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.407+0000 7f95d3860740 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e94cfa0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009a60 tx=0x7f95c002bab0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.408+0000 7f95d3860740 1 -- 172.21.15.31:0/1811672785 shutdown_connections 2024-04-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.408+0000 7f95d3860740 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c11e9bbdd0 0x55c11e9be1f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.408+0000 7f95d3860740 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c11e5796b0 0x55c11e9495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.408+0000 7f95d3860740 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c11e5714d0 0x55c11e94cfa0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.408+0000 7f95d3860740 1 --2- 172.21.15.31:0/1811672785 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c11e558f60 0x55c11e94ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.408+0000 7f95d3860740 1 -- 172.21.15.31:0/1811672785 >> 172.21.15.31:0/1811672785 conn(0x55c11e58e4e0 msgr2=0x55c11e93aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.408+0000 7f95adffb640 0 client.16137 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:37.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.408+0000 7f95d3860740 1 -- 172.21.15.31:0/1811672785 shutdown_connections 2024-04-25T16:03:37.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.409+0000 7f95d3860740 1 -- 172.21.15.31:0/1811672785 wait complete. 2024-04-25T16:03:37.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.409+0000 7f95d3860740 20 client.16137 trim_cache size 0 max 16384 2024-04-25T16:03:37.412 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.411+0000 7f95cc9ba640 1 -- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95b00afbd0 msgr2=0x7f95b00b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.411+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95b00afbd0 0x7f95b00b20e0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f95b8026870 tx=0x7f95b8026c70 comp rx=0 tx=0).stop 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 -- 172.21.15.31:0/334181568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c810b4c0 msgr2=0x7f95c8133f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c810b4c0 0x7f95c8133f00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f95bc00e970 tx=0x7f95bc00ee30 comp rx=0 tx=0).stop 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 -- 172.21.15.31:0/334181568 shutdown_connections 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f95b00afbd0 0x7f95b00b20e0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95c810b4c0 0x7f95c8133f00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95c810a0e0 0x7f95c8084ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 --2- 172.21.15.31:0/334181568 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f95c8078df0 0x7f95c80847a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 -- 172.21.15.31:0/334181568 >> 172.21.15.31:0/334181568 conn(0x7f95c8070400 msgr2=0x7f95c8074d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 -- 172.21.15.31:0/334181568 shutdown_connections 2024-04-25T16:03:37.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:37.412+0000 7f95cc9ba640 1 -- 172.21.15.31:0/334181568 wait complete. 2024-04-25T16:03:37.441 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:37 smithi031 ceph-mon[29140]: pgmap v420: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 794 MiB/s wr, 2.41k op/s 2024-04-25T16:03:37.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.731+0000 7fda56d9d640 1 -- 172.21.15.31:0/4164050140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda5010fef0 msgr2=0x7fda50110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.731+0000 7fda56d9d640 1 --2- 172.21.15.31:0/4164050140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda5010fef0 0x7fda50110300 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fda44009900 tx=0x7fda4402a410 comp rx=0 tx=0).stop 2024-04-25T16:03:37.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.732+0000 7fda56d9d640 1 -- 172.21.15.31:0/4164050140 shutdown_connections 2024-04-25T16:03:37.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.732+0000 7fda56d9d640 1 --2- 172.21.15.31:0/4164050140 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda5010fef0 0x7fda50110300 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.732+0000 7fda56d9d640 1 --2- 172.21.15.31:0/4164050140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda5010b1d0 0x7fda5010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.732+0000 7fda56d9d640 1 --2- 172.21.15.31:0/4164050140 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda5010a350 0x7fda5010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.732+0000 7fda56d9d640 1 -- 172.21.15.31:0/4164050140 >> 172.21.15.31:0/4164050140 conn(0x7fda50070390 msgr2=0x7fda50072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.732+0000 7fda56d9d640 1 -- 172.21.15.31:0/4164050140 shutdown_connections 2024-04-25T16:03:37.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.733+0000 7fda56d9d640 1 -- 172.21.15.31:0/4164050140 wait complete. 2024-04-25T16:03:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.734+0000 7fda56d9d640 1 Processor -- start 2024-04-25T16:03:37.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.734+0000 7fda56d9d640 1 -- start start 2024-04-25T16:03:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.734+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda5010a350 0x7fda501a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.734+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda5010b1d0 0x7fda501a9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.734+0000 7fda56d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda5010fef0 0x7fda501b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.734+0000 7fda56d9d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda5011e270 con 0x7fda5010b1d0 2024-04-25T16:03:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.734+0000 7fda56d9d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fda5011e0f0 con 0x7fda5010fef0 2024-04-25T16:03:37.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.734+0000 7fda56d9d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda5011e3f0 con 0x7fda5010a350 2024-04-25T16:03:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.738+0000 7fda5533a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda5010fef0 0x7fda501b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.738+0000 7fda5533a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda5010fef0 0x7fda501b02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37230/0 (socket says 172.21.15.31:37230) 2024-04-25T16:03:37.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.738+0000 7fda5533a640 1 -- 172.21.15.31:0/2870956695 learned_addr learned my addr 172.21.15.31:0/2870956695 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.739+0000 7fda54b39640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda5010a350 0x7fda501a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.739+0000 7fda54b39640 1 -- 172.21.15.31:0/2870956695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda5010fef0 msgr2=0x7fda501b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.740+0000 7fda54b39640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda5010fef0 0x7fda501b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.740+0000 7fda54b39640 1 -- 172.21.15.31:0/2870956695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda5010b1d0 msgr2=0x7fda501a9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.740+0000 7fda54b39640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda5010b1d0 0x7fda501a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.740+0000 7fda54b39640 1 -- 172.21.15.31:0/2870956695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda501b0930 con 0x7fda5010a350 2024-04-25T16:03:37.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.740+0000 7fda5533a640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda5010fef0 0x7fda501b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:03:37.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.742+0000 7fda54b39640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda5010a350 0x7fda501a8df0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fda40005fd0 tx=0x7fda4000dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.745+0000 7fda4dffb640 1 -- 172.21.15.31:0/2870956695 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40009c30 con 0x7fda5010a350 2024-04-25T16:03:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.745+0000 7fda4dffb640 1 -- 172.21.15.31:0/2870956695 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda40009dd0 con 0x7fda5010a350 2024-04-25T16:03:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.745+0000 7fda56d9d640 1 -- 172.21.15.31:0/2870956695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda501b0c20 con 0x7fda5010a350 2024-04-25T16:03:37.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.746+0000 7fda56d9d640 1 -- 172.21.15.31:0/2870956695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda501b1170 con 0x7fda5010a350 2024-04-25T16:03:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.751+0000 7fda5dc4e740 1 Processor -- start 2024-04-25T16:03:37.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.752+0000 7fda4dffb640 1 -- 172.21.15.31:0/2870956695 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40010460 con 0x7fda5010a350 2024-04-25T16:03:37.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.754+0000 7fda5dc4e740 1 -- start start 2024-04-25T16:03:37.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.755+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0d201b0 0x5593a10c0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.755+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10c10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.756+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cff4d0 0x5593a10c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.756+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5593a1026460 con 0x5593a0d201b0 2024-04-25T16:03:37.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.756+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5593a0fe0590 con 0x5593a0cff4d0 2024-04-25T16:03:37.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.756+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5593a1022420 con 0x5593a0cf72f0 2024-04-25T16:03:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.760+0000 7fda4ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cff4d0 0x5593a10c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.761+0000 7fda54b39640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10c10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.761+0000 7fda5533a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0d201b0 0x5593a10c0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.761+0000 7fda5533a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0d201b0 0x5593a10c0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44956/0 (socket says 172.21.15.31:44956) 2024-04-25T16:03:37.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.761+0000 7fda5533a640 1 -- 172.21.15.31:0/4080673848 learned_addr learned my addr 172.21.15.31:0/4080673848 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.763+0000 7fda4ffff640 1 -- 172.21.15.31:0/4080673848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 msgr2=0x5593a10c10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.763+0000 7fda4ffff640 1 --2- 172.21.15.31:0/4080673848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10c10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.763+0000 7fda4ffff640 1 -- 172.21.15.31:0/4080673848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0d201b0 msgr2=0x5593a10c0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.763+0000 7fda4ffff640 1 --2- 172.21.15.31:0/4080673848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0d201b0 0x5593a10c0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.764+0000 7fda4ffff640 1 -- 172.21.15.31:0/4080673848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593a0fed0d0 con 0x5593a0cff4d0 2024-04-25T16:03:37.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.765+0000 7fda4ffff640 1 --2- 172.21.15.31:0/4080673848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cff4d0 0x5593a10c7fa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fda48005fd0 tx=0x7fda4800e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d78619eeec0c39a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.765+0000 7fda4dffb640 1 -- 172.21.15.31:0/2870956695 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fda400035f0 con 0x7fda5010a350 2024-04-25T16:03:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.767+0000 7fda4dffb640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda380afbf0 0x7fda380b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.768 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.767+0000 7fda4dffb640 1 -- 172.21.15.31:0/2870956695 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fda400e1de0 con 0x7fda5010a350 2024-04-25T16:03:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.768+0000 7fda36ffd640 1 -- 172.21.15.31:0/4080673848 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda48004b20 con 0x5593a0cff4d0 2024-04-25T16:03:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.768+0000 7fda36ffd640 1 -- 172.21.15.31:0/4080673848 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda48003e20 con 0x5593a0cff4d0 2024-04-25T16:03:37.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.768+0000 7fda5533a640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda380afbf0 0x7fda380b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.770+0000 7fda5dc4e740 1 -- 172.21.15.31:0/4080673848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cff4d0 msgr2=0x5593a10c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.770+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/4080673848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cff4d0 0x5593a10c7fa0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fda48005fd0 tx=0x7fda4800e720 comp rx=0 tx=0).stop 2024-04-25T16:03:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.776+0000 7fda5dc4e740 1 -- 172.21.15.31:0/4080673848 shutdown_connections 2024-04-25T16:03:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.776+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/4080673848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cff4d0 0x5593a10c7fa0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.776+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/4080673848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10c10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.776+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/4080673848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0d201b0 0x5593a10c0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.776+0000 7fda5dc4e740 1 -- 172.21.15.31:0/4080673848 >> 172.21.15.31:0/4080673848 conn(0x5593a0d14300 msgr2=0x5593a10bc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.777+0000 7fda5dc4e740 1 -- 172.21.15.31:0/4080673848 shutdown_connections 2024-04-25T16:03:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.777+0000 7fda5dc4e740 1 -- 172.21.15.31:0/4080673848 wait complete. 2024-04-25T16:03:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.778+0000 7fda5dc4e740 1 -- start start 2024-04-25T16:03:37.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.778+0000 7fda5dc4e740 1 Processor -- start 2024-04-25T16:03:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.781+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10ca770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.781+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.781+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 0x5593a10d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.781+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5593a10c8a40 con 0x5593a0cff4d0 2024-04-25T16:03:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.781+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5593a0fadb60 con 0x5593a0cf72f0 2024-04-25T16:03:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.781+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5593a10c8bc0 con 0x5593a0d201b0 2024-04-25T16:03:37.783 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.781+0000 7fda5dc4e740 10 client.0 _ll_register_callbacks cb 0x5593a0fd22b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.783+0000 7fda54b39640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.783+0000 7fda54b39640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10ca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37252/0 (socket says 172.21.15.31:37252) 2024-04-25T16:03:37.784 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.783+0000 7fda54b39640 1 -- 172.21.15.31:0/2913719164 learned_addr learned my addr 172.21.15.31:0/2913719164 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.784+0000 7fda5533a640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda380afbf0 0x7fda380b2100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fda44004650 tx=0x7fda440314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.784+0000 7fda5533a640 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 0x5593a10d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.784+0000 7fda54b39640 1 -- 172.21.15.31:0/2913719164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 msgr2=0x5593a10d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.784+0000 7fda54b39640 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 0x5593a10d1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.784+0000 7fda54b39640 1 -- 172.21.15.31:0/2913719164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 msgr2=0x5593a10cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.784+0000 7fda54b39640 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.784+0000 7fda54b39640 1 -- 172.21.15.31:0/2913719164 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593a0d38210 con 0x5593a0cf72f0 2024-04-25T16:03:37.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.785+0000 7fda54b39640 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10ca770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fda400a6350 tx=0x7fda400a9de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d830867f52efc2b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.797+0000 7fda2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:03:37.798 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.797+0000 7fda2ffff640 1 -- 172.21.15.31:0/2913719164 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40020040 con 0x5593a0cf72f0 2024-04-25T16:03:37.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.797+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2913719164 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda48003e20 con 0x5593a0cf72f0 2024-04-25T16:03:37.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.797+0000 7fda2ffff640 1 -- 172.21.15.31:0/2913719164 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fda400b2ca0 con 0x5593a0cf72f0 2024-04-25T16:03:37.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.798+0000 7fda2ffff640 1 -- 172.21.15.31:0/2913719164 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda400ae680 con 0x5593a0cf72f0 2024-04-25T16:03:37.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.798+0000 7fda5dc4e740 10 client.25817 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:37.799 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.798+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2913719164 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5593a0e04620 con 0x5593a0cf72f0 2024-04-25T16:03:37.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.799+0000 7fda2ffff640 1 -- 172.21.15.31:0/2913719164 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda48003e20 con 0x5593a0cf72f0 2024-04-25T16:03:37.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.801+0000 7fda2ffff640 1 -- 172.21.15.31:0/2913719164 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fda400a5c40 con 0x5593a0cf72f0 2024-04-25T16:03:37.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.801+0000 7fda5dc4e740 10 client.25817 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:37.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.801+0000 7fda5dc4e740 10 client.25817 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:37.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.801+0000 7fda5dc4e740 20 client.25817 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:37.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.801+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5593a0cded80 0x5593a10d8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.801+0000 7fda5dc4e740 4 client.25817 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:37.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.801+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2913719164 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fda48004830 con 0x5593a0cded80 2024-04-25T16:03:37.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.802+0000 7fda4ffff640 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5593a0cded80 0x5593a10d8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.803+0000 7fda4ffff640 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5593a0cded80 0x5593a10d8a40 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a61c80b69a37b9a server_cookie=359885698a5c4745 in_seq=0 out_seq=0 2024-04-25T16:03:37.805 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.804+0000 7fda2ffff640 10 client.25817 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:37.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.807+0000 7fda2ffff640 1 -- 172.21.15.31:0/2913719164 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fda48004830 con 0x5593a0cded80 2024-04-25T16:03:37.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.807+0000 7fda2ffff640 10 client.25817 handle_command_reply: tid=0 2024-04-25T16:03:37.808 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.807+0000 7fda5dc4e740 1 client.25817 shutdown 2024-04-25T16:03:37.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.810+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2913719164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 msgr2=0x5593a10ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.810+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10ca770 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fda400a6350 tx=0x7fda400a9de0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.810+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2913719164 shutdown_connections 2024-04-25T16:03:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.810+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5593a0cded80 0x5593a10d8a40 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.810+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 0x5593a10d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.810+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.810+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2913719164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10ca770 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.810+0000 7fda2ffff640 0 client.25817 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:37.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.810+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2913719164 >> 172.21.15.31:0/2913719164 conn(0x5593a0d14300 msgr2=0x5593a10bf3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.816+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2913719164 shutdown_connections 2024-04-25T16:03:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.817+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2913719164 wait complete. 2024-04-25T16:03:37.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.817+0000 7fda5dc4e740 20 client.25817 trim_cache size 0 max 16384 2024-04-25T16:03:37.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.869+0000 7fda5dc4e740 1 Processor -- start 2024-04-25T16:03:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.873+0000 7fda5dc4e740 1 -- start start 2024-04-25T16:03:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.874+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.874+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10d1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.874+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0d201b0 0x5593a10ca880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.874+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c00e3b0 con 0x5593a0cff4d0 2024-04-25T16:03:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.874+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c00ff20 con 0x5593a0d201b0 2024-04-25T16:03:37.875 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.874+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c00ea80 con 0x5593a0cf72f0 2024-04-25T16:03:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.876+0000 7fda54b39640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.876+0000 7fda54b39640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38666/0 (socket says 172.21.15.31:38666) 2024-04-25T16:03:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.876+0000 7fda54b39640 1 -- 172.21.15.31:0/2456742003 learned_addr learned my addr 172.21.15.31:0/2456742003 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.878+0000 7fda4ffff640 1 --2- 172.21.15.31:0/2456742003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10d1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.879+0000 7fda54b39640 1 -- 172.21.15.31:0/2456742003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0d201b0 msgr2=0x5593a10ca880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:03:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.879+0000 7fda54b39640 1 --2- 172.21.15.31:0/2456742003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0d201b0 0x5593a10ca880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.879+0000 7fda54b39640 1 -- 172.21.15.31:0/2456742003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 msgr2=0x5593a10d1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.879+0000 7fda54b39640 1 --2- 172.21.15.31:0/2456742003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10d1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.880 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.879+0000 7fda54b39640 1 -- 172.21.15.31:0/2456742003 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5593a10c7e80 con 0x5593a0cf72f0 2024-04-25T16:03:37.881 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.880+0000 7fda54b39640 1 --2- 172.21.15.31:0/2456742003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10ca340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fda400a6300 tx=0x7fda400a6770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ee6542e2baa3d6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.883+0000 7fda367fc640 1 -- 172.21.15.31:0/2456742003 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40020070 con 0x5593a0cf72f0 2024-04-25T16:03:37.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.884+0000 7fda367fc640 1 -- 172.21.15.31:0/2456742003 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda400cb330 con 0x5593a0cf72f0 2024-04-25T16:03:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.885+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2456742003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 msgr2=0x5593a10ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:37.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.885+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2456742003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10ca340 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fda400a6300 tx=0x7fda400a6770 comp rx=0 tx=0).stop 2024-04-25T16:03:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.890+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2456742003 shutdown_connections 2024-04-25T16:03:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.890+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2456742003 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0d201b0 0x5593a10ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.890+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2456742003 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10d1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.890+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/2456742003 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0cf72f0 0x5593a10ca340 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.890+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2456742003 >> 172.21.15.31:0/2456742003 conn(0x5593a0d14300 msgr2=0x5593a10c4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:37.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.893+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2456742003 shutdown_connections 2024-04-25T16:03:37.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.896+0000 7fda5dc4e740 1 -- 172.21.15.31:0/2456742003 wait complete. 2024-04-25T16:03:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.896+0000 7fda5dc4e740 1 -- start start 2024-04-25T16:03:37.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.896+0000 7fda5dc4e740 1 Processor -- start 2024-04-25T16:03:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.899+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.899+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10cf170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.899+0000 7fda5dc4e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 0x5593a113ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.899+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c00f820 con 0x5593a0cff4d0 2024-04-25T16:03:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.899+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c00f150 con 0x5593a0cf72f0 2024-04-25T16:03:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.899+0000 7fda5dc4e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda3c00c440 con 0x5593a0d201b0 2024-04-25T16:03:37.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.899+0000 7fda5dc4e740 10 client.0 _ll_register_callbacks cb 0x7fda400aa0c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.902+0000 7fda5533a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 0x5593a113ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.902+0000 7fda5533a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 0x5593a113ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38678/0 (socket says 172.21.15.31:38678) 2024-04-25T16:03:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.902+0000 7fda5533a640 1 -- 172.21.15.31:0/318711872 learned_addr learned my addr 172.21.15.31:0/318711872 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:37.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.903+0000 7fda54b39640 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.905+0000 7fda54b39640 1 -- 172.21.15.31:0/318711872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 msgr2=0x5593a113ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.905+0000 7fda54b39640 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 0x5593a113ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.905+0000 7fda54b39640 1 -- 172.21.15.31:0/318711872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 msgr2=0x5593a10cf170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.905+0000 7fda54b39640 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10cf170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:37.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.905+0000 7fda54b39640 1 -- 172.21.15.31:0/318711872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593a113f320 con 0x5593a0cf72f0 2024-04-25T16:03:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.916+0000 7fda54b39640 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10cec30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x5593a10c9cf0 tx=0x7fda4000bef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a16f7222b1a21d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.916+0000 7fda2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:03:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.916+0000 7fda2ffff640 1 -- 172.21.15.31:0/318711872 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda40020030 con 0x5593a0cf72f0 2024-04-25T16:03:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.916+0000 7fda2ffff640 1 -- 172.21.15.31:0/318711872 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fda400a9740 con 0x5593a0cf72f0 2024-04-25T16:03:37.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.917+0000 7fda5dc4e740 1 -- 172.21.15.31:0/318711872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5593a10c9810 con 0x5593a0cf72f0 2024-04-25T16:03:37.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.919+0000 7fda2ffff640 1 -- 172.21.15.31:0/318711872 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fda400cb330 con 0x5593a0cf72f0 2024-04-25T16:03:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.919+0000 7fda5dc4e740 10 client.25820 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.919+0000 7fda5dc4e740 1 -- 172.21.15.31:0/318711872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5593a113f320 con 0x5593a0cf72f0 2024-04-25T16:03:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.920+0000 7fda2ffff640 1 -- 172.21.15.31:0/318711872 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5593a10c9810 con 0x5593a0cf72f0 2024-04-25T16:03:37.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.920+0000 7fda2ffff640 1 -- 172.21.15.31:0/318711872 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fda40003e80 con 0x5593a0cf72f0 2024-04-25T16:03:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.920+0000 7fda5dc4e740 10 client.25820 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.920+0000 7fda5dc4e740 10 client.25820 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.920+0000 7fda5dc4e740 20 client.25820 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.920+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5593a0cb41c0 0x5593a1142b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.920+0000 7fda5dc4e740 4 client.25820 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:03:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.920+0000 7fda5dc4e740 1 -- 172.21.15.31:0/318711872 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5593a1034070 con 0x5593a0cb41c0 2024-04-25T16:03:37.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.920+0000 7fda4ffff640 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5593a0cb41c0 0x5593a1142b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.922+0000 7fda4ffff640 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5593a0cb41c0 0x5593a1142b10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc6eef7c7ca09d3 server_cookie=6f4b9ede5e298239 in_seq=0 out_seq=0 2024-04-25T16:03:37.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:37.922+0000 7fda2ffff640 10 client.25820 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:38.443 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:03:38.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:03:38.444 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:03:39.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.051+0000 7fd7e896c640 1 -- 172.21.15.31:0/1600564682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 msgr2=0x7fd7e410b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:39.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.051+0000 7fd7e896c640 1 --2- 172.21.15.31:0/1600564682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 0x7fd7e410b720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009860 tx=0x7fd7d402a380 comp rx=0 tx=0).stop 2024-04-25T16:03:39.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.052+0000 7fd7e896c640 1 -- 172.21.15.31:0/1600564682 shutdown_connections 2024-04-25T16:03:39.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.052+0000 7fd7e896c640 1 --2- 172.21.15.31:0/1600564682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e4110030 0x7fd7e4110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.052+0000 7fd7e896c640 1 --2- 172.21.15.31:0/1600564682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 0x7fd7e410b720 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.052+0000 7fd7e896c640 1 --2- 172.21.15.31:0/1600564682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e410a420 0x7fd7e410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.052+0000 7fd7e896c640 1 -- 172.21.15.31:0/1600564682 >> 172.21.15.31:0/1600564682 conn(0x7fd7e40704e0 msgr2=0x7fd7e4072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:39.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.060+0000 7fd7e896c640 1 -- 172.21.15.31:0/1600564682 shutdown_connections 2024-04-25T16:03:39.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.060+0000 7fd7e896c640 1 -- 172.21.15.31:0/1600564682 wait complete. 2024-04-25T16:03:39.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.062+0000 7fd7e896c640 1 Processor -- start 2024-04-25T16:03:39.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.062+0000 7fd7e896c640 1 -- start start 2024-04-25T16:03:39.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.063+0000 7fd7e896c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e410a420 0x7fd7e41a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.063+0000 7fd7e896c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 0x7fd7e41a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.064+0000 7fd7e896c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e4110030 0x7fd7e41b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.064+0000 7fd7e896c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e411e4e0 con 0x7fd7e4110030 2024-04-25T16:03:39.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.064+0000 7fd7e896c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e411e360 con 0x7fd7e410b310 2024-04-25T16:03:39.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.064+0000 7fd7e896c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e411e660 con 0x7fd7e410a420 2024-04-25T16:03:39.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.064+0000 7fd7e1d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 0x7fd7e41a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:39.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.065+0000 7fd7e1d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 0x7fd7e41a9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37276/0 (socket says 172.21.15.31:37276) 2024-04-25T16:03:39.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.065+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/3035965740 learned_addr learned my addr 172.21.15.31:0/3035965740 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:39.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.065+0000 7fd7e259c640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e410a420 0x7fd7e41a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:39.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.065+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/3035965740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e410a420 msgr2=0x7fd7e41a90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:39.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.065+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e410a420 0x7fd7e41a90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.065+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/3035965740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e4110030 msgr2=0x7fd7e41b0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.065+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e4110030 0x7fd7e41b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.065+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/3035965740 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e41b0c00 con 0x7fd7e410b310 2024-04-25T16:03:39.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.066+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 0x7fd7e41a9600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009980 tx=0x7fd7d4030d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.079+0000 7fd7d37fe640 1 -- 172.21.15.31:0/3035965740 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4037040 con 0x7fd7e410b310 2024-04-25T16:03:39.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.079+0000 7fd7d37fe640 1 -- 172.21.15.31:0/3035965740 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7d402adc0 con 0x7fd7e410b310 2024-04-25T16:03:39.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.085+0000 7fd7d37fe640 1 -- 172.21.15.31:0/3035965740 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40053b0 con 0x7fd7e410b310 2024-04-25T16:03:39.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.085+0000 7fd7e896c640 1 -- 172.21.15.31:0/3035965740 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e41b0e90 con 0x7fd7e410b310 2024-04-25T16:03:39.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.085+0000 7fd7e896c640 1 -- 172.21.15.31:0/3035965740 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e41b1290 con 0x7fd7e410b310 2024-04-25T16:03:39.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.090+0000 7fd7ef819740 1 Processor -- start 2024-04-25T16:03:39.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.097+0000 7fd7ef819740 1 -- start start 2024-04-25T16:03:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.098+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e5fb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.098+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e5fba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.098+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 0x560a9e602920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.098+0000 7fd7ef819740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560a9e560b00 con 0x560a9e231550 2024-04-25T16:03:39.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.098+0000 7fd7ef819740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560a9e4fc920 con 0x560a9e239730 2024-04-25T16:03:39.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.098+0000 7fd7ef819740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560a9e55cac0 con 0x560a9e25a410 2024-04-25T16:03:39.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.099+0000 7fd7d37fe640 1 -- 172.21.15.31:0/3035965740 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd7d4043020 con 0x7fd7e410b310 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.099+0000 7fd7e2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e5fb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.099+0000 7fd7e2d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e5fb4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:38694/0 (socket says 172.21.15.31:38694) 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.099+0000 7fd7e2d9d640 1 -- 172.21.15.31:0/3573354881 learned_addr learned my addr 172.21.15.31:0/3573354881 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.099+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 0x560a9e602920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.100+0000 7fd7e259c640 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e5fba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.100+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/3573354881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 msgr2=0x560a9e5fb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.100+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e5fb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.100+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/3573354881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 msgr2=0x560a9e5fba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.102+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e5fba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.102+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/3573354881 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a9e33a7e0 con 0x560a9e239730 2024-04-25T16:03:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.102+0000 7fd7e259c640 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e5fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:03:39.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.103+0000 7fd7d37fe640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7cc0afbd0 0x7fd7cc0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.104+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 0x560a9e602920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40fb9d0 tx=0x7fd7d4101eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22f448e99bed3726 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.106+0000 7fd7d0ff9640 1 -- 172.21.15.31:0/3573354881 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40e0070 con 0x560a9e239730 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.106+0000 7fd7d0ff9640 1 -- 172.21.15.31:0/3573354881 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7d40c5f20 con 0x560a9e239730 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.106+0000 7fd7d37fe640 1 -- 172.21.15.31:0/3035965740 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd7d40fdf80 con 0x7fd7e410b310 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.107+0000 7fd7ef819740 1 -- 172.21.15.31:0/3573354881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 msgr2=0x560a9e602920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.107+0000 7fd7ef819740 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 0x560a9e602920 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd7d40fb9d0 tx=0x7fd7d4101eb0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.111+0000 7fd7e2d9d640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7cc0afbd0 0x7fd7cc0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.111+0000 7fd7ef819740 1 -- 172.21.15.31:0/3573354881 shutdown_connections 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.111+0000 7fd7ef819740 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 0x560a9e602920 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.111+0000 7fd7ef819740 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e5fba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.111+0000 7fd7ef819740 1 --2- 172.21.15.31:0/3573354881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e5fb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.111+0000 7fd7ef819740 1 -- 172.21.15.31:0/3573354881 >> 172.21.15.31:0/3573354881 conn(0x560a9e24e560 msgr2=0x560a9e5f6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:39.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.113+0000 7fd7ef819740 1 -- 172.21.15.31:0/3573354881 shutdown_connections 2024-04-25T16:03:39.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.116+0000 7fd7ef819740 1 -- 172.21.15.31:0/3573354881 wait complete. 2024-04-25T16:03:39.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.117+0000 7fd7ef819740 1 -- start start 2024-04-25T16:03:39.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.117+0000 7fd7ef819740 1 Processor -- start 2024-04-25T16:03:39.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.119+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e231550 0x560a9e605310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.119+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 0x560a9e605850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.119+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e25a410 0x560a9e60c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.119+0000 7fd7ef819740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560a9e603130 con 0x560a9e239730 2024-04-25T16:03:39.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.120+0000 7fd7ef819740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560a9e4fa5d0 con 0x560a9e25a410 2024-04-25T16:03:39.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.120+0000 7fd7ef819740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560a9e6032b0 con 0x560a9e231550 2024-04-25T16:03:39.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.120+0000 7fd7ef819740 10 client.0 _ll_register_callbacks cb 0x560a9e4e8080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:39.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.121+0000 7fd7e1d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 0x560a9e605850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:39.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.121+0000 7fd7e1d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 0x560a9e605850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45034/0 (socket says 172.21.15.31:45034) 2024-04-25T16:03:39.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.121+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/2475572619 learned_addr learned my addr 172.21.15.31:0/2475572619 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:39.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.121+0000 7fd7e259c640 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e231550 0x560a9e605310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.121+0000 7fd7e2d9d640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7cc0afbd0 0x7fd7cc0b20e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x560a9e5fc670 tx=0x7fd7d8005a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.122+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/2475572619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e231550 msgr2=0x560a9e605310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.122+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e231550 0x560a9e605310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.122+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/2475572619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e25a410 msgr2=0x560a9e60c7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.122+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e25a410 0x560a9e60c7b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:39.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.122+0000 7fd7e1d9b640 1 -- 172.21.15.31:0/2475572619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a9e39af70 con 0x560a9e239730 2024-04-25T16:03:39.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.123+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 0x560a9e605850 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4030df0 tx=0x7fd7d402ab60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f816f751f7364451 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:39.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.123+0000 7fd7c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:39.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.124+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/2475572619 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40e0050 con 0x560a9e239730 2024-04-25T16:03:39.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.124+0000 7fd7ef819740 1 -- 172.21.15.31:0/2475572619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560a9e56e710 con 0x560a9e239730 2024-04-25T16:03:39.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.125+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/2475572619 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd7d41010d0 con 0x560a9e239730 2024-04-25T16:03:39.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.125+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/2475572619 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd7d40c5d60 con 0x560a9e239730 2024-04-25T16:03:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.126+0000 7fd7ef819740 10 client.16152 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:39.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.126+0000 7fd7ef819740 1 -- 172.21.15.31:0/2475572619 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560a9e232170 con 0x560a9e239730 2024-04-25T16:03:39.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.127+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/2475572619 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x560a9e56e710 con 0x560a9e239730 2024-04-25T16:03:39.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.129+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/2475572619 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fd7d40e7260 con 0x560a9e239730 2024-04-25T16:03:39.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.129+0000 7fd7ef819740 10 client.16152 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:39.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.129+0000 7fd7ef819740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:39.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.129+0000 7fd7ef819740 20 client.16152 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:39.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.129+0000 7fd7ef819740 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560a9e2191f0 0x560a9e612ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:39.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.129+0000 7fd7ef819740 4 client.16152 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:39.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.129+0000 7fd7ef819740 1 -- 172.21.15.31:0/2475572619 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560a9e568580 con 0x560a9e2191f0 2024-04-25T16:03:39.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.130+0000 7fd7e259c640 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560a9e2191f0 0x560a9e612ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:39.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.133+0000 7fd7e259c640 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560a9e2191f0 0x560a9e612ba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6ad860779c2536c server_cookie=fb7143aba46267c6 in_seq=0 out_seq=0 2024-04-25T16:03:39.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:39.133+0000 7fd7c9ffb640 10 client.16152 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:39 smithi063 ceph-mon[32413]: pgmap v421: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 876 MiB/s wr, 2.89k op/s 2024-04-25T16:03:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:39 smithi186 ceph-mon[33258]: pgmap v421: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 876 MiB/s wr, 2.89k op/s 2024-04-25T16:03:39.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:39 smithi031 ceph-mon[29140]: pgmap v421: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 876 MiB/s wr, 2.89k op/s 2024-04-25T16:03:41.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:41 smithi063 ceph-mon[32413]: pgmap v422: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 789 MiB/s wr, 2.75k op/s 2024-04-25T16:03:41.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:41 smithi186 ceph-mon[33258]: pgmap v422: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 789 MiB/s wr, 2.75k op/s 2024-04-25T16:03:41.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:41 smithi031 ceph-mon[29140]: pgmap v422: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 789 MiB/s wr, 2.75k op/s 2024-04-25T16:03:43.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:43 smithi186 ceph-mon[33258]: pgmap v423: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 703 MiB/s wr, 2.57k op/s 2024-04-25T16:03:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:43 smithi031 ceph-mon[29140]: pgmap v423: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 703 MiB/s wr, 2.57k op/s 2024-04-25T16:03:43.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:43 smithi063 ceph-mon[32413]: pgmap v423: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 703 MiB/s wr, 2.57k op/s 2024-04-25T16:03:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.026+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/2475572619 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560a9e568580 con 0x560a9e2191f0 2024-04-25T16:03:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.026+0000 7fd7c9ffb640 10 client.16152 handle_command_reply: tid=0 2024-04-25T16:03:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.026+0000 7fd7ef819740 1 client.16152 shutdown 2024-04-25T16:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.026+0000 7fda2ffff640 1 -- 172.21.15.31:0/318711872 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5593a1034070 con 0x5593a0cb41c0 2024-04-25T16:03:44.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.026+0000 7fda2ffff640 10 client.25820 handle_command_reply: tid=0 2024-04-25T16:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.029+0000 7fda5dc4e740 1 client.25820 shutdown 2024-04-25T16:03:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fda5dc4e740 1 -- 172.21.15.31:0/318711872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 msgr2=0x5593a10cec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10cec30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x5593a10c9cf0 tx=0x7fda4000bef0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fd7ef819740 1 -- 172.21.15.31:0/2475572619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 msgr2=0x560a9e605850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fd7ef819740 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 0x560a9e605850 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4030df0 tx=0x7fd7d402ab60 comp rx=0 tx=0).stop 2024-04-25T16:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fda5dc4e740 1 -- 172.21.15.31:0/318711872 shutdown_connections 2024-04-25T16:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5593a0cb41c0 0x5593a1142b10 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593a0d201b0 0x5593a113ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593a0cff4d0 0x5593a10cf170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fda5dc4e740 1 --2- 172.21.15.31:0/318711872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593a0cf72f0 0x5593a10cec30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fda5dc4e740 1 -- 172.21.15.31:0/318711872 >> 172.21.15.31:0/318711872 conn(0x5593a0d14300 msgr2=0x5593a10ca2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.030+0000 7fda2ffff640 0 client.25820 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7ef819740 1 -- 172.21.15.31:0/2475572619 shutdown_connections 2024-04-25T16:03:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7ef819740 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560a9e2191f0 0x560a9e612ba0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7ef819740 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e25a410 0x560a9e60c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7ef819740 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 0x560a9e605850 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7ef819740 1 --2- 172.21.15.31:0/2475572619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e231550 0x560a9e605310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7ef819740 1 -- 172.21.15.31:0/2475572619 >> 172.21.15.31:0/2475572619 conn(0x560a9e24e560 msgr2=0x560a9e5f5ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7c9ffb640 0 client.16152 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7ef819740 1 -- 172.21.15.31:0/2475572619 shutdown_connections 2024-04-25T16:03:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7ef819740 1 -- 172.21.15.31:0/2475572619 wait complete. 2024-04-25T16:03:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.035+0000 7fd7ef819740 20 client.16152 trim_cache size 0 max 16384 2024-04-25T16:03:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.036+0000 7fda5dc4e740 1 -- 172.21.15.31:0/318711872 shutdown_connections 2024-04-25T16:03:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.036+0000 7fda5dc4e740 1 -- 172.21.15.31:0/318711872 wait complete. 2024-04-25T16:03:44.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.036+0000 7fda5dc4e740 20 client.25820 trim_cache size 0 max 16384 2024-04-25T16:03:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:03:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:03:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:03:44.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:03:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.050+0000 7fda56d9d640 1 -- 172.21.15.31:0/2870956695 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda380afbf0 msgr2=0x7fda380b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.050+0000 7fda56d9d640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda380afbf0 0x7fda380b2100 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fda44004650 tx=0x7fda440314a0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.050+0000 7fda56d9d640 1 -- 172.21.15.31:0/2870956695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda5010a350 msgr2=0x7fda501a8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.050+0000 7fda56d9d640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda5010a350 0x7fda501a8df0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fda40005fd0 tx=0x7fda4000dde0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.054+0000 7fda56d9d640 1 -- 172.21.15.31:0/2870956695 shutdown_connections 2024-04-25T16:03:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.054+0000 7fda56d9d640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda380afbf0 0x7fda380b2100 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.054+0000 7fda56d9d640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda5010fef0 0x7fda501b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.054+0000 7fda56d9d640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda5010b1d0 0x7fda501a9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.054+0000 7fda56d9d640 1 --2- 172.21.15.31:0/2870956695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda5010a350 0x7fda501a8df0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.054+0000 7fda56d9d640 1 -- 172.21.15.31:0/2870956695 >> 172.21.15.31:0/2870956695 conn(0x7fda50070390 msgr2=0x7fda5010bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.059+0000 7fda56d9d640 1 -- 172.21.15.31:0/2870956695 shutdown_connections 2024-04-25T16:03:44.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:44.059+0000 7fda56d9d640 1 -- 172.21.15.31:0/2870956695 wait complete. 2024-04-25T16:03:44.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.145+0000 7fd7ef819740 1 Processor -- start 2024-04-25T16:03:44.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.145+0000 7fd7ef819740 1 -- start start 2024-04-25T16:03:44.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.146+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e231550 0x560a9e5f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.146+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 0x560a9e60c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.146+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e60cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.146+0000 7fd7ef819740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e41b9e00 con 0x560a9e239730 2024-04-25T16:03:44.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.146+0000 7fd7ef819740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e41bb970 con 0x560a9e231550 2024-04-25T16:03:44.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.146+0000 7fd7ef819740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e41ba4d0 con 0x560a9e25a410 2024-04-25T16:03:44.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.150+0000 7fd7e259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e231550 0x560a9e5f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.150+0000 7fd7e259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e231550 0x560a9e5f6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36314/0 (socket says 172.21.15.31:36314) 2024-04-25T16:03:44.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.150+0000 7fd7e259c640 1 -- 172.21.15.31:0/3961316398 learned_addr learned my addr 172.21.15.31:0/3961316398 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:44.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.153+0000 7fd7e259c640 1 -- 172.21.15.31:0/3961316398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 msgr2=0x560a9e60cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.153+0000 7fd7e259c640 1 --2- 172.21.15.31:0/3961316398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e60cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.153+0000 7fd7e259c640 1 -- 172.21.15.31:0/3961316398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 msgr2=0x560a9e60c6d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:03:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.153+0000 7fd7e259c640 1 --2- 172.21.15.31:0/3961316398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 0x560a9e60c6d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.153+0000 7fd7e259c640 1 -- 172.21.15.31:0/3961316398 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560a9e5fbe70 con 0x560a9e231550 2024-04-25T16:03:44.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.156+0000 7fd7e259c640 1 --2- 172.21.15.31:0/3961316398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e231550 0x560a9e5f6f00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc005fd0 tx=0x7fd7dc009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e1bd066efb1607f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:44.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.159+0000 7fd7cbfff640 1 -- 172.21.15.31:0/3961316398 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc004b20 con 0x560a9e231550 2024-04-25T16:03:44.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.159+0000 7fd7cbfff640 1 -- 172.21.15.31:0/3961316398 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7dc01f030 con 0x560a9e231550 2024-04-25T16:03:44.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.159+0000 7fd7cbfff640 1 -- 172.21.15.31:0/3961316398 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc01b870 con 0x560a9e231550 2024-04-25T16:03:44.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.161+0000 7fd7ef819740 1 -- 172.21.15.31:0/3961316398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e231550 msgr2=0x560a9e5f6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.161+0000 7fd7ef819740 1 --2- 172.21.15.31:0/3961316398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e231550 0x560a9e5f6f00 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc005fd0 tx=0x7fd7dc009610 comp rx=0 tx=0).stop 2024-04-25T16:03:44.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.167+0000 7fd7ef819740 1 -- 172.21.15.31:0/3961316398 shutdown_connections 2024-04-25T16:03:44.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.167+0000 7fd7ef819740 1 --2- 172.21.15.31:0/3961316398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e60cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.167+0000 7fd7ef819740 1 --2- 172.21.15.31:0/3961316398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e239730 0x560a9e60c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.167+0000 7fd7ef819740 1 --2- 172.21.15.31:0/3961316398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e231550 0x560a9e5f6f00 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.167+0000 7fd7ef819740 1 -- 172.21.15.31:0/3961316398 >> 172.21.15.31:0/3961316398 conn(0x560a9e24e560 msgr2=0x560a9e5f67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.180+0000 7fd7ef819740 1 -- 172.21.15.31:0/3961316398 shutdown_connections 2024-04-25T16:03:44.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.181+0000 7fd7ef819740 1 -- 172.21.15.31:0/3961316398 wait complete. 2024-04-25T16:03:44.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.181+0000 7fd7ef819740 1 -- start start 2024-04-25T16:03:44.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.182+0000 7fd7ef819740 1 Processor -- start 2024-04-25T16:03:44.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.182+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e60a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.182+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 0x560a9e60a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.182+0000 7fd7ef819740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e60b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.182+0000 7fd7ef819740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e41bb270 con 0x560a9e231550 2024-04-25T16:03:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.182+0000 7fd7ef819740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e41baba0 con 0x560a9e239730 2024-04-25T16:03:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.182+0000 7fd7ef819740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e41b7e90 con 0x560a9e25a410 2024-04-25T16:03:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.182+0000 7fd7ef819740 10 client.0 _ll_register_callbacks cb 0x7fd7d4101c70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.182+0000 7fd7e259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e60a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.183+0000 7fd7e259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e60a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37266/0 (socket says 172.21.15.31:37266) 2024-04-25T16:03:44.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.183+0000 7fd7e259c640 1 -- 172.21.15.31:0/4161868891 learned_addr learned my addr 172.21.15.31:0/4161868891 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:44.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.185+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 0x560a9e60a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.193+0000 7fd7e2d9d640 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e60b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.194+0000 7fd7e259c640 1 -- 172.21.15.31:0/4161868891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 msgr2=0x560a9e60b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.194+0000 7fd7e259c640 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e60b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.194+0000 7fd7e259c640 1 -- 172.21.15.31:0/4161868891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 msgr2=0x560a9e60a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.194+0000 7fd7e259c640 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 0x560a9e60a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.194+0000 7fd7e259c640 1 -- 172.21.15.31:0/4161868891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560a9e6003d0 con 0x560a9e231550 2024-04-25T16:03:44.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.204+0000 7fd7e259c640 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e60a2b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009d80 tx=0x7fd7dc014fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0531150d9e2eb01 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:44.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.204+0000 7fd7c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:03:44.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.211+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/4161868891 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7dc013030 con 0x560a9e231550 2024-04-25T16:03:44.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.211+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/4161868891 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd7dc0131d0 con 0x560a9e231550 2024-04-25T16:03:44.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.212+0000 7fd7ef819740 1 -- 172.21.15.31:0/4161868891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7dc01f030 con 0x560a9e231550 2024-04-25T16:03:44.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.220+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/4161868891 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd7dc0133d0 con 0x560a9e231550 2024-04-25T16:03:44.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.220+0000 7fd7ef819740 10 client.16155 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:44.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.221+0000 7fd7ef819740 1 -- 172.21.15.31:0/4161868891 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560a9e60cd80 con 0x560a9e231550 2024-04-25T16:03:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.226+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/4161868891 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7dc01f030 con 0x560a9e231550 2024-04-25T16:03:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.226+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/4161868891 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fd7dc005000 con 0x560a9e231550 2024-04-25T16:03:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.226+0000 7fd7ef819740 10 client.16155 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.226+0000 7fd7ef819740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.226+0000 7fd7ef819740 20 client.16155 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.226+0000 7fd7ef819740 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560a9e1ee630 0x560a9e67d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.226+0000 7fd7ef819740 4 client.16155 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:03:44.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.226+0000 7fd7ef819740 1 -- 172.21.15.31:0/4161868891 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560a9e4e3ac0 con 0x560a9e1ee630 2024-04-25T16:03:44.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.228+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560a9e1ee630 0x560a9e67d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.234+0000 7fd7e1d9b640 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560a9e1ee630 0x560a9e67d130 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34f10e94a824ee07 server_cookie=6cc844466cd587b5 in_seq=0 out_seq=0 2024-04-25T16:03:44.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.234+0000 7fd7c9ffb640 10 client.16155 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:44.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.237+0000 7fd7c9ffb640 1 -- 172.21.15.31:0/4161868891 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560a9e4e3ac0 con 0x560a9e1ee630 2024-04-25T16:03:44.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.237+0000 7fd7c9ffb640 10 client.16155 handle_command_reply: tid=0 2024-04-25T16:03:44.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.237+0000 7fd7ef819740 1 client.16155 shutdown 2024-04-25T16:03:44.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.237+0000 7fd7ef819740 1 -- 172.21.15.31:0/4161868891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 msgr2=0x560a9e60a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.237+0000 7fd7ef819740 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e60a2b0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc009d80 tx=0x7fd7dc014fb0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.238+0000 7fd7ef819740 1 -- 172.21.15.31:0/4161868891 shutdown_connections 2024-04-25T16:03:44.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.238+0000 7fd7ef819740 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560a9e1ee630 0x560a9e67d130 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.238+0000 7fd7ef819740 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560a9e25a410 0x560a9e60b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.238+0000 7fd7ef819740 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560a9e239730 0x560a9e60a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.238+0000 7fd7ef819740 1 --2- 172.21.15.31:0/4161868891 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560a9e231550 0x560a9e60a2b0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.238+0000 7fd7ef819740 1 -- 172.21.15.31:0/4161868891 >> 172.21.15.31:0/4161868891 conn(0x560a9e24e560 msgr2=0x560a9e5fa660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.238+0000 7fd7c9ffb640 0 client.16155 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:44.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.239+0000 7fd7ef819740 1 -- 172.21.15.31:0/4161868891 shutdown_connections 2024-04-25T16:03:44.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.239+0000 7fd7ef819740 1 -- 172.21.15.31:0/4161868891 wait complete. 2024-04-25T16:03:44.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.239+0000 7fd7ef819740 20 client.16155 trim_cache size 0 max 16384 2024-04-25T16:03:44.242 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:03:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:03:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "cd2f352b-cd43-4bad-8411-b9ca107cd1e9", 2024-04-25T16:03:44.242 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:03:44.242 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:03:44.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.250+0000 7fd7e896c640 1 -- 172.21.15.31:0/3035965740 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7cc0afbd0 msgr2=0x7fd7cc0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.250+0000 7fd7e896c640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7cc0afbd0 0x7fd7cc0b20e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x560a9e5fc670 tx=0x7fd7d8005a40 comp rx=0 tx=0).stop 2024-04-25T16:03:44.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.250+0000 7fd7e896c640 1 -- 172.21.15.31:0/3035965740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 msgr2=0x7fd7e41a9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.250+0000 7fd7e896c640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 0x7fd7e41a9600 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009980 tx=0x7fd7d4030d80 comp rx=0 tx=0).stop 2024-04-25T16:03:44.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.251+0000 7fd7e896c640 1 -- 172.21.15.31:0/3035965740 shutdown_connections 2024-04-25T16:03:44.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.251+0000 7fd7e896c640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7cc0afbd0 0x7fd7cc0b20e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.251+0000 7fd7e896c640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e4110030 0x7fd7e41b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.251+0000 7fd7e896c640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e410b310 0x7fd7e41a9600 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.251+0000 7fd7e896c640 1 --2- 172.21.15.31:0/3035965740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e410a420 0x7fd7e41a90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.251+0000 7fd7e896c640 1 -- 172.21.15.31:0/3035965740 >> 172.21.15.31:0/3035965740 conn(0x7fd7e40704e0 msgr2=0x7fd7e410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.258+0000 7fd7e896c640 1 -- 172.21.15.31:0/3035965740 shutdown_connections 2024-04-25T16:03:44.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.260+0000 7fd7e896c640 1 -- 172.21.15.31:0/3035965740 wait complete. 2024-04-25T16:03:44.327 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:03:44.327 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:03:44.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.701+0000 7f67516dd640 1 -- 172.21.15.31:0/1536311047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10fe90 msgr2=0x7f674c1102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.701+0000 7f67516dd640 1 --2- 172.21.15.31:0/1536311047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10fe90 0x7f674c1102a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6740009900 tx=0x7f674002a410 comp rx=0 tx=0).stop 2024-04-25T16:03:44.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.701+0000 7f67516dd640 1 -- 172.21.15.31:0/1536311047 shutdown_connections 2024-04-25T16:03:44.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.701+0000 7f67516dd640 1 --2- 172.21.15.31:0/1536311047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10fe90 0x7f674c1102a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.701+0000 7f67516dd640 1 --2- 172.21.15.31:0/1536311047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f674c10b190 0x7f674c10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.701+0000 7f67516dd640 1 --2- 172.21.15.31:0/1536311047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f674c10a340 0x7f674c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.701+0000 7f67516dd640 1 -- 172.21.15.31:0/1536311047 >> 172.21.15.31:0/1536311047 conn(0x7f674c070320 msgr2=0x7f674c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.705+0000 7f67516dd640 1 -- 172.21.15.31:0/1536311047 shutdown_connections 2024-04-25T16:03:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.709+0000 7f67516dd640 1 -- 172.21.15.31:0/1536311047 wait complete. 2024-04-25T16:03:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.710+0000 7f67516dd640 1 Processor -- start 2024-04-25T16:03:44.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.710+0000 7f67516dd640 1 -- start start 2024-04-25T16:03:44.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.711+0000 7f67516dd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f674c10a340 0x7f674c119b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.711+0000 7f67516dd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10b190 0x7f674c11a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.711+0000 7f67516dd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f674c11b020 0x7f674c11b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.711+0000 7f67516dd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f674c112030 con 0x7f674c10a340 2024-04-25T16:03:44.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.712+0000 7f67516dd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f674c111eb0 con 0x7f674c11b020 2024-04-25T16:03:44.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.712+0000 7f67516dd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f674c1121b0 con 0x7f674c10b190 2024-04-25T16:03:44.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.712+0000 7f6750edc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f674c11b020 0x7f674c11b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.712+0000 7f674bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f674c10a340 0x7f674c119b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.712+0000 7f6750edc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f674c11b020 0x7f674c11b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36334/0 (socket says 172.21.15.31:36334) 2024-04-25T16:03:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.712+0000 7f674bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f674c10a340 0x7f674c119b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37276/0 (socket says 172.21.15.31:37276) 2024-04-25T16:03:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.712+0000 7f6750edc640 1 -- 172.21.15.31:0/4051273244 learned_addr learned my addr 172.21.15.31:0/4051273244 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.712+0000 7f674b7fe640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10b190 0x7f674c11a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.713+0000 7f674bfff640 1 -- 172.21.15.31:0/4051273244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10b190 msgr2=0x7f674c11a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.713+0000 7f674bfff640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10b190 0x7f674c11a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.713+0000 7f674bfff640 1 -- 172.21.15.31:0/4051273244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f674c11b020 msgr2=0x7f674c11b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.713+0000 7f674bfff640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f674c11b020 0x7f674c11b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.713+0000 7f674bfff640 1 -- 172.21.15.31:0/4051273244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c11bc00 con 0x7f674c10a340 2024-04-25T16:03:44.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.717+0000 7f674bfff640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f674c10a340 0x7f674c119b80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f674c074a30 tx=0x7f6744009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:44.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.721+0000 7f67497fa640 1 -- 172.21.15.31:0/4051273244 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744019070 con 0x7f674c10a340 2024-04-25T16:03:44.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.722+0000 7f67516dd640 1 -- 172.21.15.31:0/4051273244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674c11bef0 con 0x7f674c10a340 2024-04-25T16:03:44.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.722+0000 7f67516dd640 1 -- 172.21.15.31:0/4051273244 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f674c1b9dc0 con 0x7f674c10a340 2024-04-25T16:03:44.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.743+0000 7f67597f2740 1 Processor -- start 2024-04-25T16:03:44.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.743+0000 7f67497fa640 1 -- 172.21.15.31:0/4051273244 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67440148f0 con 0x7f674c10a340 2024-04-25T16:03:44.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.747+0000 7f67497fa640 1 -- 172.21.15.31:0/4051273244 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6744012840 con 0x7f674c10a340 2024-04-25T16:03:44.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.747+0000 7f67497fa640 1 -- 172.21.15.31:0/4051273244 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f674401c070 con 0x7f674c10a340 2024-04-25T16:03:44.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.748+0000 7f67597f2740 1 -- start start 2024-04-25T16:03:44.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.749+0000 7f67597f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebb1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.750+0000 7f67497fa640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f67340afc50 0x7f67340b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.750+0000 7f67597f2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebb21d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.750+0000 7f67597f2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebb2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.750+0000 7f67597f2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556c0eb16d80 con 0x556c0e7ef6b0 2024-04-25T16:03:44.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.750+0000 7f67597f2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556c0ead08b0 con 0x556c0e7e74d0 2024-04-25T16:03:44.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.750+0000 7f67597f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556c0eb12d40 con 0x556c0e810390 2024-04-25T16:03:44.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.751+0000 7f674bfff640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f67340afc50 0x7f67340b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.752+0000 7f6750edc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebb2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.752+0000 7f67497fa640 1 -- 172.21.15.31:0/4051273244 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f67440df3a0 con 0x7f674c10a340 2024-04-25T16:03:44.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.752+0000 7f6750edc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebb2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37278/0 (socket says 172.21.15.31:37278) 2024-04-25T16:03:44.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.752+0000 7f6750edc640 1 -- 172.21.15.31:0/1722348096 learned_addr learned my addr 172.21.15.31:0/1722348096 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:44.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.753+0000 7f674bfff640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f67340afc50 0x7f67340b2160 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f67440a7660 tx=0x7f6744010000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:44.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.757+0000 7f6750edc640 1 --2- 172.21.15.31:0/1722348096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebb1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.758+0000 7f674b7fe640 1 --2- 172.21.15.31:0/1722348096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebb21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.759+0000 7f6750edc640 1 -- 172.21.15.31:0/1722348096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 msgr2=0x556c0ebb1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.759+0000 7f6750edc640 1 --2- 172.21.15.31:0/1722348096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebb1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.759+0000 7f6750edc640 1 -- 172.21.15.31:0/1722348096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 msgr2=0x556c0ebb21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.759+0000 7f6750edc640 1 --2- 172.21.15.31:0/1722348096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebb21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.759+0000 7f6750edc640 1 -- 172.21.15.31:0/1722348096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c0eaa1a40 con 0x556c0e7ef6b0 2024-04-25T16:03:44.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.762+0000 7f6750edc640 1 --2- 172.21.15.31:0/1722348096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebb2e60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6740031fd0 tx=0x7f6740009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e946dc82b206b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:44.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.763+0000 7f67327fc640 1 -- 172.21.15.31:0/1722348096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6740035040 con 0x556c0e7ef6b0 2024-04-25T16:03:44.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.765+0000 7f67327fc640 1 -- 172.21.15.31:0/1722348096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67400035f0 con 0x556c0e7ef6b0 2024-04-25T16:03:44.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.767+0000 7f67597f2740 1 -- 172.21.15.31:0/1722348096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 msgr2=0x556c0ebb2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.767+0000 7f67597f2740 1 --2- 172.21.15.31:0/1722348096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebb2e60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f6740031fd0 tx=0x7f6740009540 comp rx=0 tx=0).stop 2024-04-25T16:03:44.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.773+0000 7f67597f2740 1 -- 172.21.15.31:0/1722348096 shutdown_connections 2024-04-25T16:03:44.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.773+0000 7f67597f2740 1 --2- 172.21.15.31:0/1722348096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebb2e60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.773+0000 7f67597f2740 1 --2- 172.21.15.31:0/1722348096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebb21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.773+0000 7f67597f2740 1 --2- 172.21.15.31:0/1722348096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebb1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.773+0000 7f67597f2740 1 -- 172.21.15.31:0/1722348096 >> 172.21.15.31:0/1722348096 conn(0x556c0e8044e0 msgr2=0x556c0ebad460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.788+0000 7f67597f2740 1 -- 172.21.15.31:0/1722348096 shutdown_connections 2024-04-25T16:03:44.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.797+0000 7f67597f2740 1 -- 172.21.15.31:0/1722348096 wait complete. 2024-04-25T16:03:44.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.798+0000 7f67597f2740 1 -- start start 2024-04-25T16:03:44.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.798+0000 7f67597f2740 1 Processor -- start 2024-04-25T16:03:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.799+0000 7f67597f2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.799+0000 7f67597f2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbc010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.799+0000 7f67597f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebc2f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.799+0000 7f67597f2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556c0ebb39b0 con 0x556c0e7ef6b0 2024-04-25T16:03:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.799+0000 7f67597f2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556c0ea9dd10 con 0x556c0e7e74d0 2024-04-25T16:03:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.799+0000 7f67597f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556c0ebb3b30 con 0x556c0e810390 2024-04-25T16:03:44.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.799+0000 7f67597f2740 10 client.0 _ll_register_callbacks cb 0x556c0eaca210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:44.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.804+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.804+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbc010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37288/0 (socket says 172.21.15.31:37288) 2024-04-25T16:03:44.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.804+0000 7f674b7fe640 1 -- 172.21.15.31:0/1938093590 learned_addr learned my addr 172.21.15.31:0/1938093590 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:44.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.804+0000 7f6750edc640 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.804+0000 7f6750edc640 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebc2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.810+0000 7f6750edc640 1 -- 172.21.15.31:0/1938093590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 msgr2=0x556c0ebbbad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.810+0000 7f6750edc640 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbbad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.810+0000 7f6750edc640 1 -- 172.21.15.31:0/1938093590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 msgr2=0x556c0ebbc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.810+0000 7f6750edc640 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.810+0000 7f6750edc640 1 -- 172.21.15.31:0/1938093590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c0e8f27d0 con 0x556c0e810390 2024-04-25T16:03:44.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.818+0000 7f6750edc640 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebc2f70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f674002aa00 tx=0x7f6740031ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef8a5efa0c9d2a1c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:44.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.821+0000 7f672b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:44.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.822+0000 7f672b7fe640 1 -- 172.21.15.31:0/1938093590 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6740050020 con 0x556c0e810390 2024-04-25T16:03:44.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.822+0000 7f672b7fe640 1 -- 172.21.15.31:0/1938093590 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f67400039a0 con 0x556c0e810390 2024-04-25T16:03:44.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.822+0000 7f67597f2740 1 -- 172.21.15.31:0/1938093590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67400035f0 con 0x556c0e810390 2024-04-25T16:03:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.823+0000 7f672b7fe640 1 -- 172.21.15.31:0/1938093590 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6740035070 con 0x556c0e810390 2024-04-25T16:03:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.825+0000 7f67597f2740 10 client.25804 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:44.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.825+0000 7f67597f2740 1 -- 172.21.15.31:0/1938093590 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c0e8283f0 con 0x556c0e810390 2024-04-25T16:03:44.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.832+0000 7f672b7fe640 1 -- 172.21.15.31:0/1938093590 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67400035f0 con 0x556c0e810390 2024-04-25T16:03:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.834+0000 7f672b7fe640 1 -- 172.21.15.31:0/1938093590 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f6740049070 con 0x556c0e810390 2024-04-25T16:03:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.835+0000 7f67597f2740 10 client.25804 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.835+0000 7f67597f2740 10 client.25804 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.835+0000 7f67597f2740 20 client.25804 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.835+0000 7f67597f2740 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556c0e7cef60 0x556c0ebc9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.835+0000 7f67597f2740 4 client.25804 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.835+0000 7f67597f2740 1 -- 172.21.15.31:0/1938093590 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c0eb24990 con 0x556c0e7cef60 2024-04-25T16:03:44.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.843+0000 7f674b7fe640 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556c0e7cef60 0x556c0ebc9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.858+0000 7f674b7fe640 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556c0e7cef60 0x556c0ebc9030 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a03bed217bad2016 server_cookie=9e238053e0f08209 in_seq=0 out_seq=0 2024-04-25T16:03:44.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.859+0000 7f672b7fe640 10 client.25804 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:44.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.864+0000 7f672b7fe640 1 -- 172.21.15.31:0/1938093590 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556c0eb24990 con 0x556c0e7cef60 2024-04-25T16:03:44.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.864+0000 7f672b7fe640 10 client.25804 handle_command_reply: tid=0 2024-04-25T16:03:44.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.865+0000 7f67597f2740 1 client.25804 shutdown 2024-04-25T16:03:44.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.868+0000 7f67597f2740 1 -- 172.21.15.31:0/1938093590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 msgr2=0x556c0ebc2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.868+0000 7f67597f2740 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebc2f70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f674002aa00 tx=0x7f6740031ea0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.871+0000 7f67597f2740 1 -- 172.21.15.31:0/1938093590 shutdown_connections 2024-04-25T16:03:44.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.871+0000 7f67597f2740 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556c0e7cef60 0x556c0ebc9030 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.871+0000 7f67597f2740 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebc2f70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.871+0000 7f67597f2740 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.871+0000 7f67597f2740 1 --2- 172.21.15.31:0/1938093590 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.871+0000 7f67597f2740 1 -- 172.21.15.31:0/1938093590 >> 172.21.15.31:0/1938093590 conn(0x556c0e8044e0 msgr2=0x556c0ebac420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.872+0000 7f672b7fe640 0 client.25804 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:44.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.873+0000 7f67597f2740 1 -- 172.21.15.31:0/1938093590 shutdown_connections 2024-04-25T16:03:44.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.873+0000 7f67597f2740 1 -- 172.21.15.31:0/1938093590 wait complete. 2024-04-25T16:03:44.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.873+0000 7f67597f2740 20 client.25804 trim_cache size 0 max 16384 2024-04-25T16:03:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.925+0000 7f67597f2740 1 Processor -- start 2024-04-25T16:03:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.925+0000 7f67597f2740 1 -- start start 2024-04-25T16:03:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.926+0000 7f67597f2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.926+0000 7f67597f2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbb2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.927+0000 7f6750edc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.927+0000 7f6750edc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36356/0 (socket says 172.21.15.31:36356) 2024-04-25T16:03:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.927+0000 7f6750edc640 1 -- 172.21.15.31:0/2538323096 learned_addr learned my addr 172.21.15.31:0/2538323096 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.927+0000 7f674b7fe640 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbb2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.927+0000 7f67597f2740 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebc2f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.928+0000 7f67597f2740 1 -- 172.21.15.31:0/2538323096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f673800e3e0 con 0x556c0e7ef6b0 2024-04-25T16:03:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.928+0000 7f67597f2740 1 -- 172.21.15.31:0/2538323096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f673800ff50 con 0x556c0e7e74d0 2024-04-25T16:03:44.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.928+0000 7f67597f2740 1 -- 172.21.15.31:0/2538323096 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f673800eab0 con 0x556c0e810390 2024-04-25T16:03:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.929+0000 7f674b7fe640 1 -- 172.21.15.31:0/2538323096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 msgr2=0x556c0ebc2f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:03:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.929+0000 7f674b7fe640 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebc2f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.929+0000 7f674b7fe640 1 -- 172.21.15.31:0/2538323096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 msgr2=0x556c0ebbad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.929+0000 7f674b7fe640 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.929+0000 7f674b7fe640 1 -- 172.21.15.31:0/2538323096 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c0e7e80f0 con 0x556c0e7ef6b0 2024-04-25T16:03:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.929+0000 7f6750edc640 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:03:44.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.930+0000 7f674b7fe640 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbb2a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f673c004700 tx=0x7f673c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34b677769be0e51f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.931+0000 7f6731ffb640 1 -- 172.21.15.31:0/2538323096 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c016d70 con 0x556c0e7ef6b0 2024-04-25T16:03:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.931+0000 7f6731ffb640 1 -- 172.21.15.31:0/2538323096 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f673c023070 con 0x556c0e7ef6b0 2024-04-25T16:03:44.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.931+0000 7f6731ffb640 1 -- 172.21.15.31:0/2538323096 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f673c015890 con 0x556c0e7ef6b0 2024-04-25T16:03:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.933+0000 7f67597f2740 1 -- 172.21.15.31:0/2538323096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 msgr2=0x556c0ebbb2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.933+0000 7f67597f2740 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbb2a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f673c004700 tx=0x7f673c008040 comp rx=0 tx=0).stop 2024-04-25T16:03:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.933+0000 7f67597f2740 1 -- 172.21.15.31:0/2538323096 shutdown_connections 2024-04-25T16:03:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.933+0000 7f67597f2740 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ebc2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.933+0000 7f67597f2740 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7ef6b0 0x556c0ebbb2a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.933+0000 7f67597f2740 1 --2- 172.21.15.31:0/2538323096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7e74d0 0x556c0ebbad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.933+0000 7f67597f2740 1 -- 172.21.15.31:0/2538323096 >> 172.21.15.31:0/2538323096 conn(0x556c0e8044e0 msgr2=0x556c0ebb1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.933+0000 7f67597f2740 1 -- 172.21.15.31:0/2538323096 shutdown_connections 2024-04-25T16:03:44.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.934+0000 7f67597f2740 1 -- 172.21.15.31:0/2538323096 wait complete. 2024-04-25T16:03:44.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.934+0000 7f67597f2740 1 -- start start 2024-04-25T16:03:44.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.935+0000 7f67597f2740 1 Processor -- start 2024-04-25T16:03:44.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.936+0000 7f67597f2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7e74d0 0x556c0ebc0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.937+0000 7f67597f2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7ef6b0 0x556c0ebc0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.937+0000 7f67597f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ec2f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.938+0000 7f67597f2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f673800f850 con 0x556c0e7e74d0 2024-04-25T16:03:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.938+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7ef6b0 0x556c0ebc0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.938+0000 7f674b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7ef6b0 0x556c0ebc0ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36370/0 (socket says 172.21.15.31:36370) 2024-04-25T16:03:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.938+0000 7f674b7fe640 1 -- 172.21.15.31:0/450326789 learned_addr learned my addr 172.21.15.31:0/450326789 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.938+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f673800f180 con 0x556c0e7ef6b0 2024-04-25T16:03:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.938+0000 7f6750edc640 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ec2f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.938+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f673800c470 con 0x556c0e810390 2024-04-25T16:03:44.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.939+0000 7f67597f2740 10 client.0 _ll_register_callbacks cb 0x7f6740031d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.939+0000 7f6750edc640 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7e74d0 0x556c0ebc0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.940+0000 7f6750edc640 1 -- 172.21.15.31:0/450326789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7ef6b0 msgr2=0x556c0ebc0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.940+0000 7f6750edc640 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7ef6b0 0x556c0ebc0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.940+0000 7f6750edc640 1 -- 172.21.15.31:0/450326789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7e74d0 msgr2=0x556c0ebc0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.940+0000 7f6750edc640 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7e74d0 0x556c0ebc0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.940+0000 7f6750edc640 1 -- 172.21.15.31:0/450326789 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c0ead74b0 con 0x556c0e810390 2024-04-25T16:03:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.941+0000 7f6750edc640 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ec2f1d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f67400093b0 tx=0x7f6740009540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f382ea3261c3bf9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.941+0000 7f672b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.942+0000 7f672b7fe640 1 -- 172.21.15.31:0/450326789 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6740062030 con 0x556c0e810390 2024-04-25T16:03:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.943+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f673c023070 con 0x556c0e810390 2024-04-25T16:03:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.943+0000 7f672b7fe640 1 -- 172.21.15.31:0/450326789 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6740031400 con 0x556c0e810390 2024-04-25T16:03:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.944+0000 7f672b7fe640 1 -- 172.21.15.31:0/450326789 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6740050040 con 0x556c0e810390 2024-04-25T16:03:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.944+0000 7f67597f2740 10 client.25807 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.944+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c0ec305b0 con 0x556c0e810390 2024-04-25T16:03:44.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.945+0000 7f672b7fe640 1 -- 172.21.15.31:0/450326789 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6740051040 con 0x556c0e810390 2024-04-25T16:03:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.947+0000 7f672b7fe640 1 -- 172.21.15.31:0/450326789 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f6740049070 con 0x556c0e810390 2024-04-25T16:03:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.947+0000 7f67597f2740 10 client.25807 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.947+0000 7f67597f2740 10 client.25807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.947+0000 7f67597f2740 20 client.25807 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.947+0000 7f67597f2740 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556c0e7a43a0 0x556c0ec32e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.947+0000 7f67597f2740 4 client.25807 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:03:44.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.947+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f673c023070 con 0x556c0e7a43a0 2024-04-25T16:03:44.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.952+0000 7f674b7fe640 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556c0e7a43a0 0x556c0ec32e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:44.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.957+0000 7f674b7fe640 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556c0e7a43a0 0x556c0ec32e90 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=faa0d51d3f8f863f server_cookie=2c5ca0ba39065491 in_seq=0 out_seq=0 2024-04-25T16:03:44.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.957+0000 7f672b7fe640 10 client.25807 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:44.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.961+0000 7f672b7fe640 1 -- 172.21.15.31:0/450326789 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x7f673c023070 con 0x556c0e7a43a0 2024-04-25T16:03:44.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.961+0000 7f672b7fe640 10 client.25807 handle_command_reply: tid=0 2024-04-25T16:03:44.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.961+0000 7f67597f2740 1 client.25807 shutdown 2024-04-25T16:03:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 msgr2=0x556c0ec2f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ec2f1d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f67400093b0 tx=0x7f6740009540 comp rx=0 tx=0).stop 2024-04-25T16:03:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 shutdown_connections 2024-04-25T16:03:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556c0e7a43a0 0x556c0ec32e90 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c0e810390 0x556c0ec2f1d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c0e7ef6b0 0x556c0ebc0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 --2- 172.21.15.31:0/450326789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c0e7e74d0 0x556c0ebc0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 >> 172.21.15.31:0/450326789 conn(0x556c0e8044e0 msgr2=0x556c0ebbad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f672b7fe640 0 client.25807 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 shutdown_connections 2024-04-25T16:03:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 1 -- 172.21.15.31:0/450326789 wait complete. 2024-04-25T16:03:44.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.962+0000 7f67597f2740 20 client.25807 trim_cache size 0 max 16384 2024-04-25T16:03:44.964 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:03:44.964 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T16:03:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:03:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "cd2f352b-cd43-4bad-8411-b9ca107cd1e9": { 2024-04-25T16:03:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:03:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "cd2f352b-cd43-4bad-8411-b9ca107cd1e9", 2024-04-25T16:03:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:03:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:03:44.965 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:03:44.965 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 -- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f67340afc50 msgr2=0x7f67340b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f67340afc50 0x7f67340b2160 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f67440a7660 tx=0x7f6744010000 comp rx=0 tx=0).stop 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 -- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f674c10a340 msgr2=0x7f674c119b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f674c10a340 0x7f674c119b80 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f674c074a30 tx=0x7f6744009f70 comp rx=0 tx=0).stop 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 -- 172.21.15.31:0/4051273244 shutdown_connections 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f67340afc50 0x7f67340b2160 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f674c11b020 0x7f674c11b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f674c10b190 0x7f674c11a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 --2- 172.21.15.31:0/4051273244 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f674c10a340 0x7f674c119b80 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 -- 172.21.15.31:0/4051273244 >> 172.21.15.31:0/4051273244 conn(0x7f674c070320 msgr2=0x7f674c10de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 -- 172.21.15.31:0/4051273244 shutdown_connections 2024-04-25T16:03:44.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:03:44.964+0000 7f67516dd640 1 -- 172.21.15.31:0/4051273244 wait complete. 2024-04-25T16:03:45.025 INFO:tasks.cephfs.filesystem:scrub status for tag:cd2f352b-cd43-4bad-8411-b9ca107cd1e9 - {'path': '/', 'tag': 'cd2f352b-cd43-4bad-8411-b9ca107cd1e9', 'options': 'recursive,force'} 2024-04-25T16:03:45.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:45 smithi031 ceph-mon[29140]: pgmap v424: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 943 MiB/s wr, 3.15k op/s 2024-04-25T16:03:45.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:45 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:03:45.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:45 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:45.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:45 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:03:45.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:45 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:03:45.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:45 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:45 smithi063 ceph-mon[32413]: pgmap v424: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 943 MiB/s wr, 3.15k op/s 2024-04-25T16:03:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:45 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:03:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:45 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:45 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:03:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:45 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:03:45.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:45 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:45.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:45 smithi186 ceph-mon[33258]: pgmap v424: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 943 MiB/s wr, 3.15k op/s 2024-04-25T16:03:45.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:45 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:03:45.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:45 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:45.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:45 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:03:45.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:45 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:03:45.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:45 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:46.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:46 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:46.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:46 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:46.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:46 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:03:47.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:47 smithi063 ceph-mon[32413]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 768 MiB/s wr, 2.52k op/s 2024-04-25T16:03:47.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:47 smithi186 ceph-mon[33258]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 768 MiB/s wr, 2.52k op/s 2024-04-25T16:03:47.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:47 smithi031 ceph-mon[29140]: pgmap v425: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 768 MiB/s wr, 2.52k op/s 2024-04-25T16:03:49.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:49 smithi063 ceph-mon[32413]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 874 MiB/s wr, 2.83k op/s 2024-04-25T16:03:49.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:49 smithi186 ceph-mon[33258]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 874 MiB/s wr, 2.83k op/s 2024-04-25T16:03:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:49 smithi031 ceph-mon[29140]: pgmap v426: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 874 MiB/s wr, 2.83k op/s 2024-04-25T16:03:51.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:51 smithi063 ceph-mon[32413]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 816 MiB/s wr, 2.40k op/s 2024-04-25T16:03:51.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:51 smithi186 ceph-mon[33258]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 816 MiB/s wr, 2.40k op/s 2024-04-25T16:03:51.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:51 smithi031 ceph-mon[29140]: pgmap v427: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 816 MiB/s wr, 2.40k op/s 2024-04-25T16:03:53.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:53 smithi063 ceph-mon[32413]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 760 MiB/s wr, 2.21k op/s 2024-04-25T16:03:53.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:53 smithi186 ceph-mon[33258]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 760 MiB/s wr, 2.21k op/s 2024-04-25T16:03:53.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:53 smithi031 ceph-mon[29140]: pgmap v428: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 760 MiB/s wr, 2.21k op/s 2024-04-25T16:03:54.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:54 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:03:54.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:54 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:03:54.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:54 smithi063 ceph-mon[32413]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 986 MiB/s wr, 2.46k op/s 2024-04-25T16:03:54.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:54 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:03:54.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:54 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:03:54.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:54 smithi186 ceph-mon[33258]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 986 MiB/s wr, 2.46k op/s 2024-04-25T16:03:54.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:54 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:03:54.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:54 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:03:54.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:54 smithi031 ceph-mon[29140]: pgmap v429: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 986 MiB/s wr, 2.46k op/s 2024-04-25T16:03:56.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:56 smithi063 ceph-mon[32413]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 804 MiB/s wr, 2.00k op/s 2024-04-25T16:03:56.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:56 smithi186 ceph-mon[33258]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 804 MiB/s wr, 2.00k op/s 2024-04-25T16:03:56.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:56 smithi031 ceph-mon[29140]: pgmap v430: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 804 MiB/s wr, 2.00k op/s 2024-04-25T16:03:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.519+0000 7fe0fd172640 1 -- 172.21.15.31:0/743393525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f810a420 msgr2=0x7fe0f810ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:57.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.519+0000 7fe0fd172640 1 --2- 172.21.15.31:0/743393525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f810a420 0x7fe0f810ae10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009900 tx=0x7fe0ec02a440 comp rx=0 tx=0).stop 2024-04-25T16:03:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.530+0000 7fe0fd172640 1 -- 172.21.15.31:0/743393525 shutdown_connections 2024-04-25T16:03:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.530+0000 7fe0fd172640 1 --2- 172.21.15.31:0/743393525 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f8110070 0x7fe0f8110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.530+0000 7fe0fd172640 1 --2- 172.21.15.31:0/743393525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f810b350 0x7fe0f810b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.530+0000 7fe0fd172640 1 --2- 172.21.15.31:0/743393525 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f810a420 0x7fe0f810ae10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.530+0000 7fe0fd172640 1 -- 172.21.15.31:0/743393525 >> 172.21.15.31:0/743393525 conn(0x7fe0f8070540 msgr2=0x7fe0f80729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:57.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.531+0000 7fe0fd172640 1 -- 172.21.15.31:0/743393525 shutdown_connections 2024-04-25T16:03:57.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.531+0000 7fe0fd172640 1 -- 172.21.15.31:0/743393525 wait complete. 2024-04-25T16:03:57.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.532+0000 7fe0fd172640 1 Processor -- start 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.533+0000 7fe0fd172640 1 -- start start 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.533+0000 7fe0fd172640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f810a420 0x7fe0f81a9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.533+0000 7fe0fd172640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f810b350 0x7fe0f81a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.533+0000 7fe0fd172640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8110070 0x7fe0f81b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.533+0000 7fe0fd172640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f811e580 con 0x7fe0f810a420 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.533+0000 7fe0fd172640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f811e400 con 0x7fe0f8110070 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.533+0000 7fe0fd172640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f811e700 con 0x7fe0f810b350 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.534+0000 7fe0f6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f810a420 0x7fe0f81a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.534+0000 7fe0f6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f810a420 0x7fe0f81a9130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43266/0 (socket says 172.21.15.31:43266) 2024-04-25T16:03:57.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.534+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/2999716786 learned_addr learned my addr 172.21.15.31:0/2999716786 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.535+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/2999716786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f810b350 msgr2=0x7fe0f81a9670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:03:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.535+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f810b350 0x7fe0f81a9670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.535+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/2999716786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8110070 msgr2=0x7fe0f81b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.535+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8110070 0x7fe0f81b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.535+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/2999716786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f81b0c70 con 0x7fe0f810a420 2024-04-25T16:03:57.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.537+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f810a420 0x7fe0f81a9130 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec000c00 tx=0x7fe0ec030960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:57.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.538+0000 7fe0e7fff640 1 -- 172.21.15.31:0/2999716786 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec040020 con 0x7fe0f810a420 2024-04-25T16:03:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.538+0000 7fe0e7fff640 1 -- 172.21.15.31:0/2999716786 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec0035f0 con 0x7fe0f810a420 2024-04-25T16:03:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.538+0000 7fe0e7fff640 1 -- 172.21.15.31:0/2999716786 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec035070 con 0x7fe0f810a420 2024-04-25T16:03:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.538+0000 7fe0fd172640 1 -- 172.21.15.31:0/2999716786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f81b0f00 con 0x7fe0f810a420 2024-04-25T16:03:57.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.538+0000 7fe0fd172640 1 -- 172.21.15.31:0/2999716786 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f81b1370 con 0x7fe0f810a420 2024-04-25T16:03:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.546+0000 7fe0e7fff640 1 -- 172.21.15.31:0/2999716786 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe0ec043020 con 0x7fe0f810a420 2024-04-25T16:03:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.546+0000 7fe104026740 1 Processor -- start 2024-04-25T16:03:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.547+0000 7fe0e7fff640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0e00afcf0 0x7fe0e00b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.548+0000 7fe0e7fff640 1 -- 172.21.15.31:0/2999716786 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe0ec0ffb20 con 0x7fe0f810a420 2024-04-25T16:03:57.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.549+0000 7fe104026740 1 -- start start 2024-04-25T16:03:57.551 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.549+0000 7fe104026740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d5f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.550+0000 7fe104026740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e99962f0 0x55b7e9d5fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.550+0000 7fe104026740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e999e4d0 0x55b7e9d66df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.550+0000 7fe104026740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b7e9cc5460 con 0x55b7e99962f0 2024-04-25T16:03:57.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.550+0000 7fe104026740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b7e9c7f590 con 0x55b7e999e4d0 2024-04-25T16:03:57.553 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.550+0000 7fe104026740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b7e9cc1420 con 0x55b7e99bf1b0 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.552+0000 7fe0f759e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e999e4d0 0x55b7e9d66df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.552+0000 7fe0f759e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e999e4d0 0x55b7e9d66df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:50992/0 (socket says 172.21.15.31:50992) 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.552+0000 7fe0f759e640 1 -- 172.21.15.31:0/2924553496 learned_addr learned my addr 172.21.15.31:0/2924553496 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.552+0000 7fe0f759e640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0e00afcf0 0x7fe0e00b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.552+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2924553496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d5f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.552+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/2924553496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e999e4d0 msgr2=0x55b7e9d66df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.552+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2924553496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e999e4d0 0x55b7e9d66df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.553+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/2924553496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e99962f0 msgr2=0x55b7e9d5fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.553+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2924553496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e99962f0 0x55b7e9d5fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.553+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/2924553496 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7e9c8c0d0 con 0x55b7e99bf1b0 2024-04-25T16:03:57.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.556+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2924553496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d5f970 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0fd5d0 tx=0x7fe0ec0c7ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82c62d65b6866186 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.556+0000 7fe0e57fa640 1 -- 172.21.15.31:0/2924553496 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec0cc360 con 0x55b7e99bf1b0 2024-04-25T16:03:57.559 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.557+0000 7fe0e57fa640 1 -- 172.21.15.31:0/2924553496 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec0d0e10 con 0x55b7e99bf1b0 2024-04-25T16:03:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.559+0000 7fe104026740 1 -- 172.21.15.31:0/2924553496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 msgr2=0x55b7e9d5f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:57.561 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.559+0000 7fe104026740 1 --2- 172.21.15.31:0/2924553496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d5f970 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0fd5d0 tx=0x7fe0ec0c7ac0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.561+0000 7fe0f759e640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0e00afcf0 0x7fe0e00b2200 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00058c0 tx=0x7fe0f0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.563+0000 7fe104026740 1 -- 172.21.15.31:0/2924553496 shutdown_connections 2024-04-25T16:03:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.563+0000 7fe104026740 1 --2- 172.21.15.31:0/2924553496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e999e4d0 0x55b7e9d66df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.564+0000 7fe104026740 1 --2- 172.21.15.31:0/2924553496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e99962f0 0x55b7e9d5fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.564+0000 7fe104026740 1 --2- 172.21.15.31:0/2924553496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d5f970 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.564+0000 7fe104026740 1 -- 172.21.15.31:0/2924553496 >> 172.21.15.31:0/2924553496 conn(0x55b7e99b3300 msgr2=0x55b7e9d5b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:57.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.565+0000 7fe104026740 1 -- 172.21.15.31:0/2924553496 shutdown_connections 2024-04-25T16:03:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.565+0000 7fe104026740 1 -- 172.21.15.31:0/2924553496 wait complete. 2024-04-25T16:03:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.566+0000 7fe104026740 1 -- start start 2024-04-25T16:03:57.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.566+0000 7fe104026740 1 Processor -- start 2024-04-25T16:03:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.567+0000 7fe104026740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d69430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.567+0000 7fe104026740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d69970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.567+0000 7fe104026740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d708d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.567+0000 7fe104026740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b7e9d67600 con 0x55b7e999e4d0 2024-04-25T16:03:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.567+0000 7fe104026740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b7e9c4cb60 con 0x55b7e99962f0 2024-04-25T16:03:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.567+0000 7fe104026740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b7e9d67780 con 0x55b7e99bf1b0 2024-04-25T16:03:57.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.567+0000 7fe104026740 10 client.0 _ll_register_callbacks cb 0x55b7e9c712b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d708d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52596/0 (socket says 172.21.15.31:52596) 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f759e640 1 -- 172.21.15.31:0/2765123538 learned_addr learned my addr 172.21.15.31:0/2765123538 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d69430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f659c640 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d69970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f759e640 1 -- 172.21.15.31:0/2765123538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 msgr2=0x55b7e9d69430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f759e640 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d69430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f759e640 1 -- 172.21.15.31:0/2765123538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 msgr2=0x55b7e9d69970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f759e640 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d69970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.569+0000 7fe0f759e640 1 -- 172.21.15.31:0/2765123538 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7e99d7210 con 0x55b7e99bf1b0 2024-04-25T16:03:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.570+0000 7fe0f759e640 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d708d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe0f000a330 tx=0x7fe0f00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=514f9fde3fd46bd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.570+0000 7fe0da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:57.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.571+0000 7fe0da7fc640 1 -- 172.21.15.31:0/2765123538 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f0024070 con 0x55b7e99bf1b0 2024-04-25T16:03:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.572+0000 7fe104026740 1 -- 172.21.15.31:0/2765123538 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0ec0d0e10 con 0x55b7e99bf1b0 2024-04-25T16:03:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.572+0000 7fe0da7fc640 1 -- 172.21.15.31:0/2765123538 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe0f000d990 con 0x55b7e99bf1b0 2024-04-25T16:03:57.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.572+0000 7fe0da7fc640 1 -- 172.21.15.31:0/2765123538 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0f0030050 con 0x55b7e99bf1b0 2024-04-25T16:03:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.572+0000 7fe104026740 10 client.25813 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:57.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.572+0000 7fe104026740 1 -- 172.21.15.31:0/2765123538 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b7e9aa3620 con 0x55b7e99bf1b0 2024-04-25T16:03:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.574+0000 7fe0da7fc640 1 -- 172.21.15.31:0/2765123538 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec0d0e10 con 0x55b7e99bf1b0 2024-04-25T16:03:57.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.575+0000 7fe0da7fc640 1 -- 172.21.15.31:0/2765123538 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fe0f001d3b0 con 0x55b7e99bf1b0 2024-04-25T16:03:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.575+0000 7fe104026740 10 client.25813 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.575+0000 7fe104026740 10 client.25813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.575+0000 7fe104026740 20 client.25813 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.575+0000 7fe104026740 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b7e997dd80 0x55b7e9d77650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.575+0000 7fe104026740 4 client.25813 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:03:57.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.575+0000 7fe104026740 1 -- 172.21.15.31:0/2765123538 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7e9cd3070 con 0x55b7e997dd80 2024-04-25T16:03:57.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.577+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b7e997dd80 0x55b7e9d77650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.579+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b7e997dd80 0x55b7e9d77650 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b95e53f7da3ef905 server_cookie=7db4036173c5a7de in_seq=0 out_seq=0 2024-04-25T16:03:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.581+0000 7fe0da7fc640 10 client.25813 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.581+0000 7fe0da7fc640 1 -- 172.21.15.31:0/2765123538 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b7e9cd3070 con 0x55b7e997dd80 2024-04-25T16:03:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.581+0000 7fe0da7fc640 10 client.25813 handle_command_reply: tid=0 2024-04-25T16:03:57.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.581+0000 7fe104026740 1 client.25813 shutdown 2024-04-25T16:03:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.613+0000 7fe104026740 1 -- 172.21.15.31:0/2765123538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 msgr2=0x55b7e9d708d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.613+0000 7fe104026740 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d708d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe0f000a330 tx=0x7fe0f00046c0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.613+0000 7fe104026740 1 -- 172.21.15.31:0/2765123538 shutdown_connections 2024-04-25T16:03:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.613+0000 7fe104026740 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b7e997dd80 0x55b7e9d77650 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.613+0000 7fe104026740 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d708d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.613+0000 7fe104026740 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d69970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.613+0000 7fe104026740 1 --2- 172.21.15.31:0/2765123538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d69430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.613+0000 7fe104026740 1 -- 172.21.15.31:0/2765123538 >> 172.21.15.31:0/2765123538 conn(0x55b7e99b3300 msgr2=0x55b7e9d5e210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.615+0000 7fe0da7fc640 0 client.25813 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.615+0000 7fe104026740 1 -- 172.21.15.31:0/2765123538 shutdown_connections 2024-04-25T16:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.615+0000 7fe104026740 1 -- 172.21.15.31:0/2765123538 wait complete. 2024-04-25T16:03:57.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.615+0000 7fe104026740 20 client.25813 trim_cache size 0 max 16384 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.681+0000 7fe104026740 1 Processor -- start 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.681+0000 7fe104026740 1 -- start start 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.682+0000 7fe104026740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d67b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.682+0000 7fe104026740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d68090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.682+0000 7fe104026740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d707f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.682+0000 7fe104026740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f81ba200 con 0x55b7e999e4d0 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.682+0000 7fe104026740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f81bbd70 con 0x55b7e99962f0 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.682+0000 7fe104026740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f81ba8d0 con 0x55b7e99bf1b0 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.683+0000 7fe0f6d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d67b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.683+0000 7fe0f6d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d67b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:51008/0 (socket says 172.21.15.31:51008) 2024-04-25T16:03:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.683+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/3377442533 learned_addr learned my addr 172.21.15.31:0/3377442533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.683+0000 7fe0f759e640 1 --2- 172.21.15.31:0/3377442533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.683+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/3377442533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 msgr2=0x55b7e9d707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.684+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/3377442533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.684+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/3377442533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 msgr2=0x55b7e9d68090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:03:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.684+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/3377442533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d68090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.684+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/3377442533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7e9d70d30 con 0x55b7e99962f0 2024-04-25T16:03:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.684+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/3377442533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d67b50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0fd620 tx=0x7fe0ec0039d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42ce41aedb3c9310 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.685+0000 7fe0e4ff9640 1 -- 172.21.15.31:0/3377442533 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec0cc360 con 0x55b7e99962f0 2024-04-25T16:03:57.690 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.685+0000 7fe0e4ff9640 1 -- 172.21.15.31:0/3377442533 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec0d0e10 con 0x55b7e99962f0 2024-04-25T16:03:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.686+0000 7fe104026740 1 -- 172.21.15.31:0/3377442533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 msgr2=0x55b7e9d67b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:03:57.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.686+0000 7fe104026740 1 --2- 172.21.15.31:0/3377442533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d67b50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0fd620 tx=0x7fe0ec0039d0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.688+0000 7fe104026740 1 -- 172.21.15.31:0/3377442533 shutdown_connections 2024-04-25T16:03:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.688+0000 7fe104026740 1 --2- 172.21.15.31:0/3377442533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.688+0000 7fe104026740 1 --2- 172.21.15.31:0/3377442533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d68090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.688+0000 7fe104026740 1 --2- 172.21.15.31:0/3377442533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d67b50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.688+0000 7fe104026740 1 -- 172.21.15.31:0/3377442533 >> 172.21.15.31:0/3377442533 conn(0x55b7e99b3300 msgr2=0x55b7e9d69430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:03:57.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.694+0000 7fe104026740 1 -- 172.21.15.31:0/3377442533 shutdown_connections 2024-04-25T16:03:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.696+0000 7fe104026740 1 -- 172.21.15.31:0/3377442533 wait complete. 2024-04-25T16:03:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.697+0000 7fe104026740 1 -- start start 2024-04-25T16:03:57.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.697+0000 7fe104026740 1 Processor -- start 2024-04-25T16:03:57.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.698+0000 7fe104026740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d70610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.698+0000 7fe104026740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d6d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.698+0000 7fe104026740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d6d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.698+0000 7fe104026740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f81bb670 con 0x55b7e999e4d0 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.699+0000 7fe104026740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f81bafa0 con 0x55b7e99962f0 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.699+0000 7fe104026740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f81b81e0 con 0x55b7e99bf1b0 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.699+0000 7fe0f6d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d70610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.699+0000 7fe0f6d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d70610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:51018/0 (socket says 172.21.15.31:51018) 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.699+0000 7fe104026740 10 client.0 _ll_register_callbacks cb 0x7fe0f000ad00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.699+0000 7fe0f6d9d640 1 -- 172.21.15.31:0/687737778 learned_addr learned my addr 172.21.15.31:0/687737778 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:03:57.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.699+0000 7fe0f759e640 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d6d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.700+0000 7fe0f759e640 1 -- 172.21.15.31:0/687737778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 msgr2=0x55b7e9d70610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:03:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.700+0000 7fe0f759e640 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d70610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.700+0000 7fe0f759e640 1 -- 172.21.15.31:0/687737778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 msgr2=0x55b7e9d6d200 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:03:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.700+0000 7fe0f759e640 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d6d200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:03:57.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.700+0000 7fe0f759e640 1 -- 172.21.15.31:0/687737778 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7e9ddd9f0 con 0x55b7e99bf1b0 2024-04-25T16:03:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.700+0000 7fe0f759e640 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d6d740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0014010 tx=0x7fe0f000a3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a8689d7636f786e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:03:57.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.701+0000 7fe0da7fc640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:03:57.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.701+0000 7fe0da7fc640 1 -- 172.21.15.31:0/687737778 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f003a020 con 0x55b7e99bf1b0 2024-04-25T16:03:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.703+0000 7fe104026740 1 -- 172.21.15.31:0/687737778 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0ec0d0e10 con 0x55b7e99bf1b0 2024-04-25T16:03:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.703+0000 7fe0da7fc640 1 -- 172.21.15.31:0/687737778 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe0f001d3b0 con 0x55b7e99bf1b0 2024-04-25T16:03:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.704+0000 7fe0da7fc640 1 -- 172.21.15.31:0/687737778 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe0f0024040 con 0x55b7e99bf1b0 2024-04-25T16:03:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.704+0000 7fe104026740 10 client.25819 fetch_fsmap learned FSMap version 28 2024-04-25T16:03:57.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.704+0000 7fe104026740 1 -- 172.21.15.31:0/687737778 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b7e9ddeb30 con 0x55b7e99bf1b0 2024-04-25T16:03:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.706+0000 7fe0da7fc640 1 -- 172.21.15.31:0/687737778 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec0d0e10 con 0x55b7e99bf1b0 2024-04-25T16:03:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.706+0000 7fe0da7fc640 1 -- 172.21.15.31:0/687737778 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fe0f001eb20 con 0x55b7e99bf1b0 2024-04-25T16:03:57.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.706+0000 7fe104026740 10 client.25819 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:03:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.706+0000 7fe104026740 10 client.25819 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:03:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.706+0000 7fe104026740 20 client.25819 populate_metadata read hostname 'smithi031' 2024-04-25T16:03:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.707+0000 7fe104026740 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b7e99531c0 0x55b7e9de1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:03:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.707+0000 7fe104026740 4 client.25819 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:03:57.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.707+0000 7fe104026740 1 -- 172.21.15.31:0/687737778 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b7e9cccee0 con 0x55b7e99531c0 2024-04-25T16:03:57.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.711+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b7e99531c0 0x55b7e9de1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:03:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.719+0000 7fe0da7fc640 10 client.25819 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:03:57.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:03:57.719+0000 7fe0f6d9d640 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b7e99531c0 0x55b7e9de1580 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f013dfa491339b72 server_cookie=b55736b00c36b35 in_seq=0 out_seq=0 2024-04-25T16:03:58.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:03:58 smithi063 ceph-mon[32413]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 909 MiB/s wr, 2.14k op/s 2024-04-25T16:03:58.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:03:58 smithi186 ceph-mon[33258]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 909 MiB/s wr, 2.14k op/s 2024-04-25T16:03:58.955 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:03:58 smithi031 ceph-mon[29140]: pgmap v431: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 909 MiB/s wr, 2.14k op/s 2024-04-25T16:04:00.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:00 smithi186 ceph-mon[33258]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 835 MiB/s wr, 1.65k op/s 2024-04-25T16:04:00.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:00 smithi031 ceph-mon[29140]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 835 MiB/s wr, 1.65k op/s 2024-04-25T16:04:01.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:00 smithi063 ceph-mon[32413]: pgmap v432: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 835 MiB/s wr, 1.65k op/s 2024-04-25T16:04:02.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:02 smithi186 ceph-mon[33258]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 774 MiB/s wr, 1.42k op/s 2024-04-25T16:04:02.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:02 smithi031 ceph-mon[29140]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 774 MiB/s wr, 1.42k op/s 2024-04-25T16:04:03.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:02 smithi063 ceph-mon[32413]: pgmap v433: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 774 MiB/s wr, 1.42k op/s 2024-04-25T16:04:04.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:04 smithi186 ceph-mon[33258]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 998 MiB/s wr, 1.83k op/s 2024-04-25T16:04:04.920 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:04 smithi063 ceph-mon[32413]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 998 MiB/s wr, 1.83k op/s 2024-04-25T16:04:04.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:04 smithi031 ceph-mon[29140]: pgmap v434: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 998 MiB/s wr, 1.83k op/s 2024-04-25T16:04:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.561+0000 7fe0da7fc640 1 -- 172.21.15.31:0/687737778 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b7e9cccee0 con 0x55b7e99531c0 2024-04-25T16:04:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.561+0000 7fe0da7fc640 10 client.25819 handle_command_reply: tid=0 2024-04-25T16:04:05.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.561+0000 7fe104026740 1 client.25819 shutdown 2024-04-25T16:04:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe104026740 1 -- 172.21.15.31:0/687737778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 msgr2=0x55b7e9d6d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe104026740 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d6d740 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fe0f0014010 tx=0x7fe0f000a3d0 comp rx=0 tx=0).stop 2024-04-25T16:04:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe104026740 1 -- 172.21.15.31:0/687737778 shutdown_connections 2024-04-25T16:04:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe104026740 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b7e99531c0 0x55b7e9de1580 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe104026740 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b7e99bf1b0 0x55b7e9d6d740 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe104026740 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b7e999e4d0 0x55b7e9d6d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe104026740 1 --2- 172.21.15.31:0/687737778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b7e99962f0 0x55b7e9d70610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe104026740 1 -- 172.21.15.31:0/687737778 >> 172.21.15.31:0/687737778 conn(0x55b7e99b3300 msgr2=0x55b7e9d693b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe0da7fc640 0 client.25819 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.565+0000 7fe104026740 1 -- 172.21.15.31:0/687737778 shutdown_connections 2024-04-25T16:04:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.566+0000 7fe104026740 1 -- 172.21.15.31:0/687737778 wait complete. 2024-04-25T16:04:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.566+0000 7fe104026740 20 client.25819 trim_cache size 0 max 16384 2024-04-25T16:04:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:04:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:04:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:04:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:04:05.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.580+0000 7fe0fd172640 1 -- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0e00afcf0 msgr2=0x7fe0e00b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:05.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.580+0000 7fe0fd172640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0e00afcf0 0x7fe0e00b2200 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe0f00058c0 tx=0x7fe0f0005850 comp rx=0 tx=0).stop 2024-04-25T16:04:05.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.580+0000 7fe0fd172640 1 -- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f810a420 msgr2=0x7fe0f81a9130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:05.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.580+0000 7fe0fd172640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f810a420 0x7fe0f81a9130 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec000c00 tx=0x7fe0ec030960 comp rx=0 tx=0).stop 2024-04-25T16:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.582+0000 7fe0fd172640 1 -- 172.21.15.31:0/2999716786 shutdown_connections 2024-04-25T16:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.582+0000 7fe0fd172640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0e00afcf0 0x7fe0e00b2200 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.582+0000 7fe0fd172640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f8110070 0x7fe0f81b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.582+0000 7fe0fd172640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f810b350 0x7fe0f81a9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:05.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.582+0000 7fe0fd172640 1 --2- 172.21.15.31:0/2999716786 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f810a420 0x7fe0f81a9130 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:05.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.582+0000 7fe0fd172640 1 -- 172.21.15.31:0/2999716786 >> 172.21.15.31:0/2999716786 conn(0x7fe0f8070540 msgr2=0x7fe0f810c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.587+0000 7fe0fd172640 1 -- 172.21.15.31:0/2999716786 shutdown_connections 2024-04-25T16:04:05.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:05.588+0000 7fe0fd172640 1 -- 172.21.15.31:0/2999716786 wait complete. 2024-04-25T16:04:06.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:06 smithi031 ceph-mon[29140]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 800 MiB/s wr, 1.72k op/s 2024-04-25T16:04:07.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:06 smithi063 ceph-mon[32413]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 800 MiB/s wr, 1.72k op/s 2024-04-25T16:04:07.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:06 smithi186 ceph-mon[33258]: pgmap v435: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 800 MiB/s wr, 1.72k op/s 2024-04-25T16:04:08.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:08 smithi031 ceph-mon[29140]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 901 MiB/s wr, 2.10k op/s 2024-04-25T16:04:09.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:08 smithi063 ceph-mon[32413]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 901 MiB/s wr, 2.10k op/s 2024-04-25T16:04:09.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:08 smithi186 ceph-mon[33258]: pgmap v436: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 901 MiB/s wr, 2.10k op/s 2024-04-25T16:04:10.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:10 smithi031 ceph-mon[29140]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 825 MiB/s wr, 1.95k op/s 2024-04-25T16:04:11.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:10 smithi063 ceph-mon[32413]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 825 MiB/s wr, 1.95k op/s 2024-04-25T16:04:11.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:10 smithi186 ceph-mon[33258]: pgmap v437: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 825 MiB/s wr, 1.95k op/s 2024-04-25T16:04:13.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:12 smithi063 ceph-mon[32413]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 770 MiB/s wr, 1.75k op/s 2024-04-25T16:04:13.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:12 smithi186 ceph-mon[33258]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 770 MiB/s wr, 1.75k op/s 2024-04-25T16:04:13.199 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:12 smithi031 ceph-mon[29140]: pgmap v438: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 770 MiB/s wr, 1.75k op/s 2024-04-25T16:04:15.026 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:04:15.026 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:04:15.115 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:14 smithi031 ceph-mon[29140]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 983 MiB/s wr, 2.11k op/s 2024-04-25T16:04:15.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:14 smithi063 ceph-mon[32413]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 983 MiB/s wr, 2.11k op/s 2024-04-25T16:04:15.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:14 smithi186 ceph-mon[33258]: pgmap v439: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 983 MiB/s wr, 2.11k op/s 2024-04-25T16:04:15.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.488+0000 7fbe9996c640 1 -- 172.21.15.31:0/1885501476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410a0e0 msgr2=0x7fbe9410a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.488+0000 7fbe9996c640 1 --2- 172.21.15.31:0/1885501476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410a0e0 0x7fbe9410a4f0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fbe880074e0 tx=0x7fbe8802b0a0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.488+0000 7fbe9996c640 1 -- 172.21.15.31:0/1885501476 shutdown_connections 2024-04-25T16:04:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.488+0000 7fbe9996c640 1 --2- 172.21.15.31:0/1885501476 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe94078da0 0x7fbe9407b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.488+0000 7fbe9996c640 1 --2- 172.21.15.31:0/1885501476 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410a0e0 0x7fbe9410a4f0 secure :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fbe880074e0 tx=0x7fbe8802b0a0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.488+0000 7fbe9996c640 1 --2- 172.21.15.31:0/1885501476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe9410b3f0 0x7fbe94109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.488+0000 7fbe9996c640 1 -- 172.21.15.31:0/1885501476 >> 172.21.15.31:0/1885501476 conn(0x7fbe94070320 msgr2=0x7fbe940727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:15.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.488+0000 7fbe9996c640 1 -- 172.21.15.31:0/1885501476 shutdown_connections 2024-04-25T16:04:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.488+0000 7fbe9996c640 1 -- 172.21.15.31:0/1885501476 wait complete. 2024-04-25T16:04:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.490+0000 7fbe9996c640 1 Processor -- start 2024-04-25T16:04:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.490+0000 7fbe9996c640 1 -- start start 2024-04-25T16:04:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.491+0000 7fbe9996c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe94078da0 0x7fbe94084950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.491+0000 7fbe9996c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410b3f0 0x7fbe94084ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.491+0000 7fbe9996c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe94131ee0 0x7fbe94134300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.491+0000 7fbe9996c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9407ddf0 con 0x7fbe9410b3f0 2024-04-25T16:04:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.491+0000 7fbe9996c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9407dc70 con 0x7fbe94078da0 2024-04-25T16:04:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.491+0000 7fbe9996c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe9407df70 con 0x7fbe94131ee0 2024-04-25T16:04:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.492+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410b3f0 0x7fbe94084ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.492+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410b3f0 0x7fbe94084ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45676/0 (socket says 172.21.15.31:45676) 2024-04-25T16:04:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.492+0000 7fbe93fff640 1 -- 172.21.15.31:0/2528479700 learned_addr learned my addr 172.21.15.31:0/2528479700 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.492+0000 7fbe9916b640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe94131ee0 0x7fbe94134300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.493+0000 7fbe9896a640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe94078da0 0x7fbe94084950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.493+0000 7fbe93fff640 1 -- 172.21.15.31:0/2528479700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe94131ee0 msgr2=0x7fbe94134300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.493+0000 7fbe93fff640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe94131ee0 0x7fbe94134300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.493+0000 7fbe93fff640 1 -- 172.21.15.31:0/2528479700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe94078da0 msgr2=0x7fbe94084950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.494+0000 7fbe93fff640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe94078da0 0x7fbe94084950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.494+0000 7fbe93fff640 1 -- 172.21.15.31:0/2528479700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe941349e0 con 0x7fbe9410b3f0 2024-04-25T16:04:15.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.495+0000 7fbe93fff640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410b3f0 0x7fbe94084ed0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbe88005f30 tx=0x7fbe88004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:15.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.500+0000 7fbe91ffb640 1 -- 172.21.15.31:0/2528479700 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe88041070 con 0x7fbe9410b3f0 2024-04-25T16:04:15.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.500+0000 7fbe91ffb640 1 -- 172.21.15.31:0/2528479700 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe88005610 con 0x7fbe9410b3f0 2024-04-25T16:04:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.500+0000 7fbe91ffb640 1 -- 172.21.15.31:0/2528479700 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8802b9e0 con 0x7fbe9410b3f0 2024-04-25T16:04:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.501+0000 7fbe9996c640 1 -- 172.21.15.31:0/2528479700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe94134c70 con 0x7fbe9410b3f0 2024-04-25T16:04:15.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.502+0000 7fbe9996c640 1 -- 172.21.15.31:0/2528479700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe94135170 con 0x7fbe9410b3f0 2024-04-25T16:04:15.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.528+0000 7fbea1a80740 1 Processor -- start 2024-04-25T16:04:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.528+0000 7fbea1a80740 1 -- start start 2024-04-25T16:04:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.529+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bee9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.529+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 0x55a31beea1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.529+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31beeae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.529+0000 7fbea1a80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a31be4ed80 con 0x55a31bb1f4d0 2024-04-25T16:04:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.529+0000 7fbea1a80740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a31be088b0 con 0x55a31bb48390 2024-04-25T16:04:15.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.529+0000 7fbea1a80740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a31be4ad40 con 0x55a31bb276b0 2024-04-25T16:04:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.533+0000 7fbe91ffb640 1 -- 172.21.15.31:0/2528479700 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbe8800a0d0 con 0x7fbe9410b3f0 2024-04-25T16:04:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.533+0000 7fbe9916b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bee9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.534+0000 7fbe9916b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bee9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37000/0 (socket says 172.21.15.31:37000) 2024-04-25T16:04:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.534+0000 7fbe9916b640 1 -- 172.21.15.31:0/3431697870 learned_addr learned my addr 172.21.15.31:0/3431697870 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:15.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.534+0000 7fbe93fff640 1 --2- 172.21.15.31:0/3431697870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31beeae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.535+0000 7fbe9916b640 1 -- 172.21.15.31:0/3431697870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 msgr2=0x55a31beeae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.535+0000 7fbe9916b640 1 --2- 172.21.15.31:0/3431697870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31beeae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.535+0000 7fbe9916b640 1 -- 172.21.15.31:0/3431697870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 msgr2=0x55a31beea1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:15.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.535+0000 7fbe9916b640 1 --2- 172.21.15.31:0/3431697870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 0x55a31beea1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.535+0000 7fbe9916b640 1 -- 172.21.15.31:0/3431697870 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a31bdd9a40 con 0x55a31bb48390 2024-04-25T16:04:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.535+0000 7fbe91ffb640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbe7c0afc30 0x7fbe7c0b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.536+0000 7fbe9896a640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbe7c0afc30 0x7fbe7c0b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.536+0000 7fbe9916b640 1 --2- 172.21.15.31:0/3431697870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bee9c50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400b7a0 tx=0x7fbe8400bc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4d4844ae52419e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:15.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.536+0000 7fbe91ffb640 1 -- 172.21.15.31:0/2528479700 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbe880fe110 con 0x7fbe9410b3f0 2024-04-25T16:04:15.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.540+0000 7fbe9896a640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbe7c0afc30 0x7fbe7c0b2140 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55a31bee9970 tx=0x7fbe8c009b20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:15.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.542+0000 7fbe7affd640 1 -- 172.21.15.31:0/3431697870 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8400d950 con 0x55a31bb48390 2024-04-25T16:04:15.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.543+0000 7fbe7affd640 1 -- 172.21.15.31:0/3431697870 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe84012420 con 0x55a31bb48390 2024-04-25T16:04:15.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.543+0000 7fbe7affd640 1 -- 172.21.15.31:0/3431697870 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84014650 con 0x55a31bb48390 2024-04-25T16:04:15.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.548+0000 7fbea1a80740 1 -- 172.21.15.31:0/3431697870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 msgr2=0x55a31bee9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:15.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.548+0000 7fbea1a80740 1 --2- 172.21.15.31:0/3431697870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bee9c50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400b7a0 tx=0x7fbe8400bc60 comp rx=0 tx=0).stop 2024-04-25T16:04:15.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.551+0000 7fbea1a80740 1 -- 172.21.15.31:0/3431697870 shutdown_connections 2024-04-25T16:04:15.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.551+0000 7fbea1a80740 1 --2- 172.21.15.31:0/3431697870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31beeae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.551+0000 7fbea1a80740 1 --2- 172.21.15.31:0/3431697870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 0x55a31beea1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.551+0000 7fbea1a80740 1 --2- 172.21.15.31:0/3431697870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bee9c50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.551+0000 7fbea1a80740 1 -- 172.21.15.31:0/3431697870 >> 172.21.15.31:0/3431697870 conn(0x55a31bb3c4e0 msgr2=0x55a31bee5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:15.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.556+0000 7fbea1a80740 1 -- 172.21.15.31:0/3431697870 shutdown_connections 2024-04-25T16:04:15.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.560+0000 7fbea1a80740 1 -- 172.21.15.31:0/3431697870 wait complete. 2024-04-25T16:04:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.562+0000 7fbea1a80740 1 -- start start 2024-04-25T16:04:15.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.563+0000 7fbea1a80740 1 Processor -- start 2024-04-25T16:04:15.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.564+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb1f4d0 0x55a31bef3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.564+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.564+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb48390 0x55a31befaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.565+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.565+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45764/0 (socket says 172.21.15.31:45764) 2024-04-25T16:04:15.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.565+0000 7fbe93fff640 1 -- 172.21.15.31:0/914185319 learned_addr learned my addr 172.21.15.31:0/914185319 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:15.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.565+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a31beeb970 con 0x55a31bb48390 2024-04-25T16:04:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.565+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a31bdd5d10 con 0x55a31bb1f4d0 2024-04-25T16:04:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.565+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a31beebaf0 con 0x55a31bb276b0 2024-04-25T16:04:15.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.566+0000 7fbe9896a640 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb1f4d0 0x55a31bef3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.566+0000 7fbea1a80740 10 client.0 _ll_register_callbacks cb 0x55a31be02210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.567+0000 7fbe93fff640 1 -- 172.21.15.31:0/914185319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb1f4d0 msgr2=0x55a31bef3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.567+0000 7fbe93fff640 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb1f4d0 0x55a31bef3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.567+0000 7fbe93fff640 1 -- 172.21.15.31:0/914185319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb48390 msgr2=0x55a31befaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.567+0000 7fbe93fff640 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb48390 0x55a31befaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.567+0000 7fbe93fff640 1 -- 172.21.15.31:0/914185319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a31bc2a7d0 con 0x55a31bb276b0 2024-04-25T16:04:15.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.568+0000 7fbe93fff640 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef3f60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbe880c3870 tx=0x7fbe880c2b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=413798224ceeb69 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.568+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe84012420 con 0x55a31bb276b0 2024-04-25T16:04:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.568+0000 7fbe73fff640 10 client.25831 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:04:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.568+0000 7fbe73fff640 1 -- 172.21.15.31:0/914185319 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880c63d0 con 0x55a31bb276b0 2024-04-25T16:04:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.569+0000 7fbe73fff640 1 -- 172.21.15.31:0/914185319 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbe880cf1e0 con 0x55a31bb276b0 2024-04-25T16:04:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.569+0000 7fbe73fff640 1 -- 172.21.15.31:0/914185319 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe880c5d70 con 0x55a31bb276b0 2024-04-25T16:04:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.570+0000 7fbea1a80740 10 client.25831 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:15.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.570+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=28}) -- 0x55a31bb603f0 con 0x55a31bb276b0 2024-04-25T16:04:15.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.571+0000 7fbe73fff640 1 -- 172.21.15.31:0/914185319 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fbe88100110 con 0x55a31bb276b0 2024-04-25T16:04:15.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.572+0000 7fbea1a80740 10 client.25831 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:15.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.572+0000 7fbea1a80740 10 client.25831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:15.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.572+0000 7fbea1a80740 20 client.25831 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:15.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.572+0000 7fbea1a80740 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a31bb06f60 0x55a31bf014e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.572+0000 7fbea1a80740 4 client.25831 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:15.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.572+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbe84012420 con 0x55a31bb06f60 2024-04-25T16:04:15.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.623+0000 7fbe9896a640 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a31bb06f60 0x55a31bf014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.716+0000 7fbe9896a640 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a31bb06f60 0x55a31bf014e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d899b1f1427f72fa server_cookie=4283c15ba60901c6 in_seq=0 out_seq=0 2024-04-25T16:04:15.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.716+0000 7fbe73fff640 10 client.25831 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:15.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.722+0000 7fbe73fff640 1 -- 172.21.15.31:0/914185319 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fbe84012420 con 0x55a31bb06f60 2024-04-25T16:04:15.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.722+0000 7fbe73fff640 10 client.25831 handle_command_reply: tid=0 2024-04-25T16:04:15.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.722+0000 7fbea1a80740 1 client.25831 shutdown 2024-04-25T16:04:15.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.723+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 msgr2=0x55a31bef3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.723+0000 7fbea1a80740 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef3f60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbe880c3870 tx=0x7fbe880c2b10 comp rx=0 tx=0).stop 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.723+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 shutdown_connections 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.723+0000 7fbea1a80740 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a31bb06f60 0x55a31bf014e0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.723+0000 7fbea1a80740 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb48390 0x55a31befaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.723+0000 7fbea1a80740 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef3f60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.723+0000 7fbea1a80740 1 --2- 172.21.15.31:0/914185319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb1f4d0 0x55a31bef3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.723+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 >> 172.21.15.31:0/914185319 conn(0x55a31bb3c4e0 msgr2=0x55a31bee8370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.726+0000 7fbe73fff640 0 client.25831 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.726+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 shutdown_connections 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.726+0000 7fbea1a80740 1 -- 172.21.15.31:0/914185319 wait complete. 2024-04-25T16:04:15.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.727+0000 7fbea1a80740 20 client.25831 trim_cache size 0 max 16384 2024-04-25T16:04:15.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.786+0000 7fbea1a80740 1 Processor -- start 2024-04-25T16:04:15.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.786+0000 7fbea1a80740 1 -- start start 2024-04-25T16:04:15.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.790+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb1f4d0 0x55a31bef33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.792+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb276b0 0x55a31befade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.792+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31befb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.792+0000 7fbea1a80740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8000e450 con 0x55a31bb276b0 2024-04-25T16:04:15.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.792+0000 7fbea1a80740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8000ffc0 con 0x55a31bb48390 2024-04-25T16:04:15.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.792+0000 7fbea1a80740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8000eb20 con 0x55a31bb1f4d0 2024-04-25T16:04:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.795+0000 7fbe9896a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb1f4d0 0x55a31bef33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.795+0000 7fbe9916b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31befb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.795+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb276b0 0x55a31befade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.795+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb276b0 0x55a31befade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45712/0 (socket says 172.21.15.31:45712) 2024-04-25T16:04:15.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.795+0000 7fbe93fff640 1 -- 172.21.15.31:0/3914106663 learned_addr learned my addr 172.21.15.31:0/3914106663 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:15.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.796+0000 7fbe93fff640 1 -- 172.21.15.31:0/3914106663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb1f4d0 msgr2=0x55a31bef33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.796+0000 7fbe93fff640 1 --2- 172.21.15.31:0/3914106663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb1f4d0 0x55a31bef33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.796+0000 7fbe93fff640 1 -- 172.21.15.31:0/3914106663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 msgr2=0x55a31befb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.796+0000 7fbe93fff640 1 --2- 172.21.15.31:0/3914106663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31befb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.796+0000 7fbe93fff640 1 -- 172.21.15.31:0/3914106663 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a31bb200f0 con 0x55a31bb276b0 2024-04-25T16:04:15.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.797+0000 7fbe93fff640 1 --2- 172.21.15.31:0/3914106663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb276b0 0x55a31befade0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fbe88005460 tx=0x7fbe8802d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61906c111b4127a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:15.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.800+0000 7fbe7a7fc640 1 -- 172.21.15.31:0/3914106663 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880c5f50 con 0x55a31bb276b0 2024-04-25T16:04:15.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.800+0000 7fbe7a7fc640 1 -- 172.21.15.31:0/3914106663 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe880c6d00 con 0x55a31bb276b0 2024-04-25T16:04:15.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.800+0000 7fbe7a7fc640 1 -- 172.21.15.31:0/3914106663 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe880c1f70 con 0x55a31bb276b0 2024-04-25T16:04:15.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.802+0000 7fbea1a80740 1 -- 172.21.15.31:0/3914106663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb276b0 msgr2=0x55a31befade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:15.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.802+0000 7fbea1a80740 1 --2- 172.21.15.31:0/3914106663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb276b0 0x55a31befade0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fbe88005460 tx=0x7fbe8802d000 comp rx=0 tx=0).stop 2024-04-25T16:04:15.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.803+0000 7fbea1a80740 1 -- 172.21.15.31:0/3914106663 shutdown_connections 2024-04-25T16:04:15.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.803+0000 7fbea1a80740 1 --2- 172.21.15.31:0/3914106663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31befb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.803+0000 7fbea1a80740 1 --2- 172.21.15.31:0/3914106663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb276b0 0x55a31befade0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.803+0000 7fbea1a80740 1 --2- 172.21.15.31:0/3914106663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb1f4d0 0x55a31bef33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.803+0000 7fbea1a80740 1 -- 172.21.15.31:0/3914106663 >> 172.21.15.31:0/3914106663 conn(0x55a31bb3c4e0 msgr2=0x55a31bef2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:15.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.803+0000 7fbea1a80740 1 -- 172.21.15.31:0/3914106663 shutdown_connections 2024-04-25T16:04:15.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.803+0000 7fbea1a80740 1 -- 172.21.15.31:0/3914106663 wait complete. 2024-04-25T16:04:15.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.804+0000 7fbea1a80740 1 -- start start 2024-04-25T16:04:15.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.805+0000 7fbea1a80740 1 Processor -- start 2024-04-25T16:04:15.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.808+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 0x55a31bef8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.809+0000 7fbe9896a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 0x55a31bef8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.809+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.810+0000 7fbea1a80740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bf67840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.810+0000 7fbe9896a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 0x55a31bef8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45728/0 (socket says 172.21.15.31:45728) 2024-04-25T16:04:15.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.810+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.810+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45780/0 (socket says 172.21.15.31:45780) 2024-04-25T16:04:15.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.811+0000 7fbe9896a640 1 -- 172.21.15.31:0/2484635936 learned_addr learned my addr 172.21.15.31:0/2484635936 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:15.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.811+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8000f8c0 con 0x55a31bb1f4d0 2024-04-25T16:04:15.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.811+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8000f1f0 con 0x55a31bb48390 2024-04-25T16:04:15.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.811+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8000c430 con 0x55a31bb276b0 2024-04-25T16:04:15.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.811+0000 7fbea1a80740 10 client.0 _ll_register_callbacks cb 0x7fbe8800ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:15.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.812+0000 7fbe9916b640 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bf67840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.812+0000 7fbe9896a640 1 -- 172.21.15.31:0/2484635936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 msgr2=0x55a31bef8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:15.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.812+0000 7fbe9896a640 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.812+0000 7fbe9896a640 1 -- 172.21.15.31:0/2484635936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 msgr2=0x55a31bf67840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:15.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.812+0000 7fbe9896a640 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bf67840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.812+0000 7fbe9896a640 1 -- 172.21.15.31:0/2484635936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a31be0f4b0 con 0x55a31bb1f4d0 2024-04-25T16:04:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.816+0000 7fbe9896a640 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 0x55a31bef8430 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbe940859f0 tx=0x7fbe8c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad329e96f7f5669 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:15.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.816+0000 7fbe73fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:04:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.816+0000 7fbe73fff640 1 -- 172.21.15.31:0/2484635936 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe8c027530 con 0x55a31bb1f4d0 2024-04-25T16:04:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.817+0000 7fbe73fff640 1 -- 172.21.15.31:0/2484635936 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbe8c0276d0 con 0x55a31bb1f4d0 2024-04-25T16:04:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.817+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe880c6d00 con 0x55a31bb1f4d0 2024-04-25T16:04:15.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.819+0000 7fbe73fff640 1 -- 172.21.15.31:0/2484635936 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbe8c0278d0 con 0x55a31bb1f4d0 2024-04-25T16:04:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.823+0000 7fbea1a80740 10 client.16191 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:15.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.823+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a31bf68b20 con 0x55a31bb1f4d0 2024-04-25T16:04:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.824+0000 7fbe73fff640 1 -- 172.21.15.31:0/2484635936 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe8c02b030 con 0x55a31bb1f4d0 2024-04-25T16:04:15.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.825+0000 7fbe73fff640 1 -- 172.21.15.31:0/2484635936 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fbe8c03a440 con 0x55a31bb1f4d0 2024-04-25T16:04:15.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.827+0000 7fbea1a80740 10 client.16191 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.827+0000 7fbea1a80740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.827+0000 7fbea1a80740 20 client.16191 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:15.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.828+0000 7fbea1a80740 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a31badc3a0 0x55a31bf6b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.828+0000 7fbea1a80740 4 client.16191 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:04:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.828+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fbe880c6d00 con 0x55a31badc3a0 2024-04-25T16:04:15.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.829+0000 7fbe93fff640 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a31badc3a0 0x55a31bf6b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:15.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.833+0000 7fbe73fff640 10 client.16191 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:15.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.833+0000 7fbe93fff640 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a31badc3a0 0x55a31bf6b540 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f55fbef6ec6b31d server_cookie=b2775c7baad6b273 in_seq=0 out_seq=0 2024-04-25T16:04:15.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.841+0000 7fbe73fff640 1 -- 172.21.15.31:0/2484635936 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fbe880c6d00 con 0x55a31badc3a0 2024-04-25T16:04:15.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.841+0000 7fbe73fff640 10 client.16191 handle_command_reply: tid=0 2024-04-25T16:04:15.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.841+0000 7fbea1a80740 1 client.16191 shutdown 2024-04-25T16:04:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.845+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 msgr2=0x55a31bef8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.845+0000 7fbea1a80740 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 0x55a31bef8430 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fbe940859f0 tx=0x7fbe8c009500 comp rx=0 tx=0).stop 2024-04-25T16:04:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.845+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 shutdown_connections 2024-04-25T16:04:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.845+0000 7fbea1a80740 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a31badc3a0 0x55a31bf6b540 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.845+0000 7fbea1a80740 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a31bb48390 0x55a31bf67840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.845+0000 7fbea1a80740 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a31bb276b0 0x55a31bef8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.845+0000 7fbea1a80740 1 --2- 172.21.15.31:0/2484635936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a31bb1f4d0 0x55a31bef8430 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.845+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 >> 172.21.15.31:0/2484635936 conn(0x55a31bb3c4e0 msgr2=0x55a31bee8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:15.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.845+0000 7fbe73fff640 0 client.16191 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:15.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.846+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 shutdown_connections 2024-04-25T16:04:15.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.847+0000 7fbea1a80740 1 -- 172.21.15.31:0/2484635936 wait complete. 2024-04-25T16:04:15.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.848+0000 7fbea1a80740 20 client.16191 trim_cache size 0 max 16384 2024-04-25T16:04:15.854 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:04:15.854 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:04:15.854 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:04:15.854 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:04:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.856+0000 7fbe9996c640 1 -- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbe7c0afc30 msgr2=0x7fbe7c0b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:15.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.856+0000 7fbe9996c640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbe7c0afc30 0x7fbe7c0b2140 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55a31bee9970 tx=0x7fbe8c009b20 comp rx=0 tx=0).stop 2024-04-25T16:04:15.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.860+0000 7fbe9996c640 1 -- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410b3f0 msgr2=0x7fbe94084ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:15.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.860+0000 7fbe9996c640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410b3f0 0x7fbe94084ed0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fbe88005f30 tx=0x7fbe88004ce0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.867+0000 7fbe9996c640 1 -- 172.21.15.31:0/2528479700 shutdown_connections 2024-04-25T16:04:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.867+0000 7fbe9996c640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbe7c0afc30 0x7fbe7c0b2140 secure :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0x55a31bee9970 tx=0x7fbe8c009b20 comp rx=0 tx=0).stop 2024-04-25T16:04:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.867+0000 7fbe9996c640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe94131ee0 0x7fbe94134300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.867+0000 7fbe9996c640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbe9410b3f0 0x7fbe94084ed0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.867+0000 7fbe9996c640 1 --2- 172.21.15.31:0/2528479700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe94078da0 0x7fbe94084950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:15.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.867+0000 7fbe9996c640 1 -- 172.21.15.31:0/2528479700 >> 172.21.15.31:0/2528479700 conn(0x7fbe94070320 msgr2=0x7fbe94074d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:15.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.871+0000 7fbe9996c640 1 -- 172.21.15.31:0/2528479700 shutdown_connections 2024-04-25T16:04:15.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:15.878+0000 7fbe9996c640 1 -- 172.21.15.31:0/2528479700 wait complete. 2024-04-25T16:04:15.973 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:04:15.974 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:04:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.374+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/3559726452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a3810b310 msgr2=0x7f4a3810b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:16.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.374+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/3559726452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a3810b310 0x7f4a3810b720 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009860 tx=0x7f4a2802a380 comp rx=0 tx=0).stop 2024-04-25T16:04:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.376+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/3559726452 shutdown_connections 2024-04-25T16:04:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.377+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/3559726452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a38110030 0x7f4a38110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.377+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/3559726452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a3810b310 0x7f4a3810b720 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.377+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/3559726452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3810a420 0x7f4a3810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.377+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/3559726452 >> 172.21.15.31:0/3559726452 conn(0x7f4a380704e0 msgr2=0x7f4a38072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:16.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.378+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/3559726452 shutdown_connections 2024-04-25T16:04:16.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.386+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/3559726452 wait complete. 2024-04-25T16:04:16.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.387+0000 7f4a3ffd5640 1 Processor -- start 2024-04-25T16:04:16.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.389+0000 7f4a3ffd5640 1 -- start start 2024-04-25T16:04:16.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.389+0000 7f4a3ffd5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3810a420 0x7f4a381a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:16.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.389+0000 7f4a3ffd5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a3810b310 0x7f4a381a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:16.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.389+0000 7f4a3ffd5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a38110030 0x7f4a381b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:16.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.389+0000 7f4a3ffd5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3811e4e0 con 0x7f4a3810b310 2024-04-25T16:04:16.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.389+0000 7f4a3ffd5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3811e360 con 0x7f4a3810a420 2024-04-25T16:04:16.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.389+0000 7f4a3ffd5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a3811e660 con 0x7f4a38110030 2024-04-25T16:04:16.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.393+0000 7f4a3dd71640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3810a420 0x7f4a381a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:16.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.393+0000 7f4a3dd71640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3810a420 0x7f4a381a90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37040/0 (socket says 172.21.15.31:37040) 2024-04-25T16:04:16.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.393+0000 7f4a3dd71640 1 -- 172.21.15.31:0/2530663106 learned_addr learned my addr 172.21.15.31:0/2530663106 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:16.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.394+0000 7f4a3dd71640 1 -- 172.21.15.31:0/2530663106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a38110030 msgr2=0x7f4a381b0590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:04:16.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.394+0000 7f4a3dd71640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a38110030 0x7f4a381b0590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:16.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.394+0000 7f4a3dd71640 1 -- 172.21.15.31:0/2530663106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a3810b310 msgr2=0x7f4a381a9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:16.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.394+0000 7f4a3dd71640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a3810b310 0x7f4a381a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:16.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.394+0000 7f4a3dd71640 1 -- 172.21.15.31:0/2530663106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a38111c90 con 0x7f4a3810a420 2024-04-25T16:04:16.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.399+0000 7f4a3dd71640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3810a420 0x7f4a381a90c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000eb90 tx=0x7f4a3000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:16.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.400+0000 7f4a2effd640 1 -- 172.21.15.31:0/2530663106 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30010630 con 0x7f4a3810a420 2024-04-25T16:04:16.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.400+0000 7f4a2effd640 1 -- 172.21.15.31:0/2530663106 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a3000ee90 con 0x7f4a3810a420 2024-04-25T16:04:16.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.403+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a38111f80 con 0x7f4a3810a420 2024-04-25T16:04:16.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.403+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a381124d0 con 0x7f4a3810a420 2024-04-25T16:04:16.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.409+0000 7f4a2effd640 1 -- 172.21.15.31:0/2530663106 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a30018b30 con 0x7f4a3810a420 2024-04-25T16:04:16.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.409+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a3810ee40 con 0x7f4a3810a420 2024-04-25T16:04:16.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.442+0000 7f4a2effd640 1 -- 172.21.15.31:0/2530663106 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4a30018cd0 con 0x7f4a3810a420 2024-04-25T16:04:16.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.443+0000 7f4a2effd640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a240afb50 0x7f4a240b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:16.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.444+0000 7f4a3d570640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a240afb50 0x7f4a240b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:16.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.444+0000 7f4a2effd640 1 -- 172.21.15.31:0/2530663106 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4a300df7e0 con 0x7f4a3810a420 2024-04-25T16:04:16.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.445+0000 7f4a2effd640 1 -- 172.21.15.31:0/2530663106 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a300a83f0 con 0x7f4a3810a420 2024-04-25T16:04:16.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:16.474+0000 7f4a3d570640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a240afb50 0x7f4a240b2060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4a28005eb0 tx=0x7f4a28005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:17.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:16 smithi063 ceph-mon[32413]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 804 MiB/s wr, 1.78k op/s 2024-04-25T16:04:17.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:16 smithi186 ceph-mon[33258]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 804 MiB/s wr, 1.78k op/s 2024-04-25T16:04:17.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:16 smithi031 ceph-mon[29140]: pgmap v440: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 804 MiB/s wr, 1.78k op/s 2024-04-25T16:04:17.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.320+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4a3802db50 con 0x7f4a3810a420 2024-04-25T16:04:17.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.322+0000 7f4a2effd640 1 -- 172.21.15.31:0/2530663106 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11918 (secure 0 0 0) 0x7f4a300a7cf0 con 0x7f4a3810a420 2024-04-25T16:04:17.330 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:04:17.330 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:02:58.299084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24553,14631,24563,24535,24521]},"id":1}]} 2024-04-25T16:04:17.330 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-04-25T16:04:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a240afb50 msgr2=0x7f4a240b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a240afb50 0x7f4a240b2060 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f4a28005eb0 tx=0x7f4a28005e40 comp rx=0 tx=0).stop 2024-04-25T16:04:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3810a420 msgr2=0x7f4a381a90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3810a420 0x7f4a381a90c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000eb90 tx=0x7f4a3000c6a0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 shutdown_connections 2024-04-25T16:04:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4a240afb50 0x7f4a240b2060 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a38110030 0x7f4a381b0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4a3810b310 0x7f4a381a9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 --2- 172.21.15.31:0/2530663106 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a3810a420 0x7f4a381a90c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 >> 172.21.15.31:0/2530663106 conn(0x7f4a380704e0 msgr2=0x7f4a3810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 shutdown_connections 2024-04-25T16:04:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.334+0000 7f4a3ffd5640 1 -- 172.21.15.31:0/2530663106 wait complete. 2024-04-25T16:04:17.376 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:04:17.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.800+0000 7ff6121a8640 1 -- 172.21.15.31:0/3914422390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff60c10fef0 msgr2=0x7ff60c110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:17.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.800+0000 7ff6121a8640 1 --2- 172.21.15.31:0/3914422390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff60c10fef0 0x7ff60c110300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff600009860 tx=0x7ff60002a340 comp rx=0 tx=0).stop 2024-04-25T16:04:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.805+0000 7ff6121a8640 1 -- 172.21.15.31:0/3914422390 shutdown_connections 2024-04-25T16:04:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.805+0000 7ff6121a8640 1 --2- 172.21.15.31:0/3914422390 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff60c10fef0 0x7ff60c110300 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.805+0000 7ff6121a8640 1 --2- 172.21.15.31:0/3914422390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c10b1d0 0x7ff60c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.805+0000 7ff6121a8640 1 --2- 172.21.15.31:0/3914422390 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff60c10a3e0 0x7ff60c10ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.805+0000 7ff6121a8640 1 -- 172.21.15.31:0/3914422390 >> 172.21.15.31:0/3914422390 conn(0x7ff60c070400 msgr2=0x7ff60c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:17.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.809+0000 7ff6121a8640 1 -- 172.21.15.31:0/3914422390 shutdown_connections 2024-04-25T16:04:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.809+0000 7ff6121a8640 1 -- 172.21.15.31:0/3914422390 wait complete. 2024-04-25T16:04:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.810+0000 7ff6121a8640 1 Processor -- start 2024-04-25T16:04:17.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.811+0000 7ff6121a8640 1 -- start start 2024-04-25T16:04:17.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.812+0000 7ff6121a8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff60c10a3e0 0x7ff60c117030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.812+0000 7ff6121a8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff60c10b1d0 0x7ff60c117570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.812+0000 7ff6121a8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c10fef0 0x7ff60c1184d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.812+0000 7ff6121a8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c11e250 con 0x7ff60c10b1d0 2024-04-25T16:04:17.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.812+0000 7ff6121a8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c11e0d0 con 0x7ff60c10a3e0 2024-04-25T16:04:17.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.813+0000 7ff6121a8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff60c11e3d0 con 0x7ff60c10fef0 2024-04-25T16:04:17.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.813+0000 7ff60bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c10fef0 0x7ff60c1184d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.813+0000 7ff60bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c10fef0 0x7ff60c1184d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45824/0 (socket says 172.21.15.31:45824) 2024-04-25T16:04:17.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.813+0000 7ff60bfff640 1 -- 172.21.15.31:0/2011303221 learned_addr learned my addr 172.21.15.31:0/2011303221 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:17.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.813+0000 7ff60affd640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff60c10b1d0 0x7ff60c117570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.813+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff60c10a3e0 0x7ff60c117030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.814+0000 7ff60bfff640 1 -- 172.21.15.31:0/2011303221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff60c10a3e0 msgr2=0x7ff60c117030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:17.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.814+0000 7ff60bfff640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff60c10a3e0 0x7ff60c117030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.814+0000 7ff60bfff640 1 -- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff60c10b1d0 msgr2=0x7ff60c117570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:17.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.814+0000 7ff60bfff640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff60c10b1d0 0x7ff60c117570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.814+0000 7ff60bfff640 1 -- 172.21.15.31:0/2011303221 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff60c118b40 con 0x7ff60c10fef0 2024-04-25T16:04:17.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.814+0000 7ff60affd640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff60c10b1d0 0x7ff60c117570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:04:17.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.815+0000 7ff60bfff640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c10fef0 0x7ff60c1184d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6000303f0 tx=0x7ff600032df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:17.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.819+0000 7ff608ff9640 1 -- 172.21.15.31:0/2011303221 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600042070 con 0x7ff60c10fef0 2024-04-25T16:04:17.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.819+0000 7ff6121a8640 1 -- 172.21.15.31:0/2011303221 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff60c118dd0 con 0x7ff60c10fef0 2024-04-25T16:04:17.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.820+0000 7ff6121a8640 1 -- 172.21.15.31:0/2011303221 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff60c1192c0 con 0x7ff60c10fef0 2024-04-25T16:04:17.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.821+0000 7ff608ff9640 1 -- 172.21.15.31:0/2011303221 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff600004050 con 0x7ff60c10fef0 2024-04-25T16:04:17.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.821+0000 7ff608ff9640 1 -- 172.21.15.31:0/2011303221 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff600037040 con 0x7ff60c10fef0 2024-04-25T16:04:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.840+0000 7ff61905a740 1 Processor -- start 2024-04-25T16:04:17.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.834+0000 7ff608ff9640 1 -- 172.21.15.31:0/2011303221 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff600005490 con 0x7ff60c10fef0 2024-04-25T16:04:17.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.845+0000 7ff61905a740 1 -- start start 2024-04-25T16:04:17.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.848+0000 7ff61905a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.848+0000 7ff61905a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aab34d0 0x560c8ae7ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.848+0000 7ff60affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.849+0000 7ff60affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45826/0 (socket says 172.21.15.31:45826) 2024-04-25T16:04:17.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.849+0000 7ff60affd640 1 -- 172.21.15.31:0/882998123 learned_addr learned my addr 172.21.15.31:0/882998123 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:17.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.849+0000 7ff61905a740 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aabb6b0 0x560c8ae7fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.850+0000 7ff61905a740 1 -- 172.21.15.31:0/882998123 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560c8ade2d80 con 0x560c8aab34d0 2024-04-25T16:04:17.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.850+0000 7ff61905a740 1 -- 172.21.15.31:0/882998123 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c8ad9ca20 con 0x560c8aabb6b0 2024-04-25T16:04:17.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.850+0000 7ff61905a740 1 -- 172.21.15.31:0/882998123 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560c8added40 con 0x560c8aadc390 2024-04-25T16:04:17.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.851+0000 7ff608ff9640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5f40afd00 0x7ff5f40b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.851+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aab34d0 0x560c8ae7ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.852+0000 7ff608ff9640 1 -- 172.21.15.31:0/2011303221 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff6000fe590 con 0x7ff60c10fef0 2024-04-25T16:04:17.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.854+0000 7ff60affd640 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aabb6b0 0x560c8ae7fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.854+0000 7ff60b7fe640 1 -- 172.21.15.31:0/882998123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 msgr2=0x560c8ae7e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:17.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.854+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.854+0000 7ff60b7fe640 1 -- 172.21.15.31:0/882998123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aabb6b0 msgr2=0x560c8ae7fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:17.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.854+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aabb6b0 0x560c8ae7fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.854+0000 7ff60b7fe640 1 -- 172.21.15.31:0/882998123 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c8ada7c30 con 0x560c8aab34d0 2024-04-25T16:04:17.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.855+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aab34d0 0x560c8ae7ef60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff60c074760 tx=0x7ff60400f7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb2e17381138218f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:17.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.862+0000 7ff60bfff640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5f40afd00 0x7ff5f40b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.875+0000 7ff5f1ffb640 1 -- 172.21.15.31:0/882998123 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604003970 con 0x560c8aab34d0 2024-04-25T16:04:17.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.875+0000 7ff5f1ffb640 1 -- 172.21.15.31:0/882998123 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff604004030 con 0x560c8aab34d0 2024-04-25T16:04:17.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.875+0000 7ff5f1ffb640 1 -- 172.21.15.31:0/882998123 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff60400c8a0 con 0x560c8aab34d0 2024-04-25T16:04:17.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.877+0000 7ff60bfff640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5f40afd00 0x7ff5f40b2210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff6000c67a0 tx=0x7ff600032f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:17.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.880+0000 7ff61905a740 1 -- 172.21.15.31:0/882998123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aab34d0 msgr2=0x560c8ae7ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:17.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.880+0000 7ff61905a740 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aab34d0 0x560c8ae7ef60 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7ff60c074760 tx=0x7ff60400f7d0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.881+0000 7ff61905a740 1 -- 172.21.15.31:0/882998123 shutdown_connections 2024-04-25T16:04:17.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.881+0000 7ff61905a740 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aabb6b0 0x560c8ae7fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.881+0000 7ff61905a740 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aab34d0 0x560c8ae7ef60 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.881+0000 7ff61905a740 1 --2- 172.21.15.31:0/882998123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.881+0000 7ff61905a740 1 -- 172.21.15.31:0/882998123 >> 172.21.15.31:0/882998123 conn(0x560c8aad04e0 msgr2=0x560c8ae7a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:17.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.886+0000 7ff61905a740 1 -- 172.21.15.31:0/882998123 shutdown_connections 2024-04-25T16:04:17.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.886+0000 7ff61905a740 1 -- 172.21.15.31:0/882998123 wait complete. 2024-04-25T16:04:17.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.887+0000 7ff61905a740 1 -- start start 2024-04-25T16:04:17.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.888+0000 7ff61905a740 1 Processor -- start 2024-04-25T16:04:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.889+0000 7ff61905a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aab34d0 0x560c8ae885a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.889+0000 7ff61905a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 0x560c8ae88ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.890+0000 7ff61905a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae8fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.890+0000 7ff60affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aab34d0 0x560c8ae885a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.890+0000 7ff60affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aab34d0 0x560c8ae885a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37094/0 (socket says 172.21.15.31:37094) 2024-04-25T16:04:17.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.890+0000 7ff60affd640 1 -- 172.21.15.31:0/1489049371 learned_addr learned my addr 172.21.15.31:0/1489049371 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:17.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.891+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560c8ae80700 con 0x560c8aabb6b0 2024-04-25T16:04:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.891+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560c8ad69dc0 con 0x560c8aab34d0 2024-04-25T16:04:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.891+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560c8ae80880 con 0x560c8aadc390 2024-04-25T16:04:17.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.891+0000 7ff61905a740 10 client.0 _ll_register_callbacks cb 0x560c8ad85600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:17.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.892+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 0x560c8ae88ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.895+0000 7ff60affd640 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae8fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.897+0000 7ff60affd640 1 -- 172.21.15.31:0/1489049371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 msgr2=0x560c8ae8fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.897+0000 7ff60affd640 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae8fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.897+0000 7ff60affd640 1 -- 172.21.15.31:0/1489049371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 msgr2=0x560c8ae88ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.897+0000 7ff60affd640 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 0x560c8ae88ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.897+0000 7ff60affd640 1 -- 172.21.15.31:0/1489049371 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c8aaf43f0 con 0x560c8aab34d0 2024-04-25T16:04:17.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.900+0000 7ff60affd640 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aab34d0 0x560c8ae885a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc015ec0 tx=0x7ff5fc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d189d8201b873ab server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.901+0000 7ff5eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.901+0000 7ff5eaffd640 1 -- 172.21.15.31:0/1489049371 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc013110 con 0x560c8aab34d0 2024-04-25T16:04:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.901+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff604004030 con 0x560c8aab34d0 2024-04-25T16:04:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.902+0000 7ff5eaffd640 1 -- 172.21.15.31:0/1489049371 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff5fc015110 con 0x560c8aab34d0 2024-04-25T16:04:17.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.902+0000 7ff5eaffd640 1 -- 172.21.15.31:0/1489049371 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff5fc015310 con 0x560c8aab34d0 2024-04-25T16:04:17.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.905+0000 7ff61905a740 10 client.25892 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:17.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.905+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c8abbe7d0 con 0x560c8aab34d0 2024-04-25T16:04:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.911+0000 7ff5eaffd640 1 -- 172.21.15.31:0/1489049371 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5fc0154b0 con 0x560c8aab34d0 2024-04-25T16:04:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.911+0000 7ff5eaffd640 1 -- 172.21.15.31:0/1489049371 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7ff5fc011a30 con 0x560c8aab34d0 2024-04-25T16:04:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.911+0000 7ff61905a740 10 client.25892 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.911+0000 7ff61905a740 10 client.25892 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:04:17.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.911+0000 7ff61905a740 20 client.25892 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.911+0000 7ff61905a740 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x560c8aa9af60 0x560c8ae967a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.911+0000 7ff61905a740 4 client.25892 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.911+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff604004030 con 0x560c8aa9af60 2024-04-25T16:04:17.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.913+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x560c8aa9af60 0x560c8ae967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.915+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x560c8aa9af60 0x560c8ae967a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c93782ee920f4354 server_cookie=dab7ba993d9bedb6 in_seq=0 out_seq=0 2024-04-25T16:04:17.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.915+0000 7ff5eaffd640 10 client.25892 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:04:17.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.917+0000 7ff5eaffd640 1 -- 172.21.15.31:0/1489049371 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7ff604004030 con 0x560c8aa9af60 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.917+0000 7ff5eaffd640 10 client.25892 handle_command_reply: tid=0 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.917+0000 7ff61905a740 1 client.25892 shutdown 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.917+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aab34d0 msgr2=0x560c8ae885a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.917+0000 7ff61905a740 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aab34d0 0x560c8ae885a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc015ec0 tx=0x7ff5fc004730 comp rx=0 tx=0).stop 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.917+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 shutdown_connections 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.917+0000 7ff61905a740 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x560c8aa9af60 0x560c8ae967a0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.918+0000 7ff61905a740 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae8fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.918+0000 7ff61905a740 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 0x560c8ae88ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.918+0000 7ff61905a740 1 --2- 172.21.15.31:0/1489049371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aab34d0 0x560c8ae885a0 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc015ec0 tx=0x7ff5fc004730 comp rx=0 tx=0).stop 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.918+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 >> 172.21.15.31:0/1489049371 conn(0x560c8aad04e0 msgr2=0x560c8ae7d100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:17.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.918+0000 7ff5eaffd640 0 client.25892 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:04:17.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.925+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 shutdown_connections 2024-04-25T16:04:17.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.925+0000 7ff61905a740 1 -- 172.21.15.31:0/1489049371 wait complete. 2024-04-25T16:04:17.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.927+0000 7ff61905a740 20 client.25892 trim_cache size 0 max 16384 2024-04-25T16:04:17.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.972+0000 7ff61905a740 1 Processor -- start 2024-04-25T16:04:17.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.973+0000 7ff61905a740 1 -- start start 2024-04-25T16:04:17.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.973+0000 7ff61905a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 0x560c8ae7e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.973+0000 7ff61905a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.973+0000 7ff61905a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:17.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.973+0000 7ff61905a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f800e550 con 0x560c8aabb6b0 2024-04-25T16:04:17.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.973+0000 7ff61905a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f80100c0 con 0x560c8aa703a0 2024-04-25T16:04:17.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.973+0000 7ff61905a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f800ec20 con 0x560c8aadc390 2024-04-25T16:04:17.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.978+0000 7ff60b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.978+0000 7ff60affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.978+0000 7ff60affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 0x560c8ae7e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:17.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.979+0000 7ff60affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 0x560c8ae7e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45820/0 (socket says 172.21.15.31:45820) 2024-04-25T16:04:17.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.979+0000 7ff60affd640 1 -- 172.21.15.31:0/2636156271 learned_addr learned my addr 172.21.15.31:0/2636156271 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:17.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.986+0000 7ff60b7fe640 1 -- 172.21.15.31:0/2636156271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 msgr2=0x560c8ae8f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:17.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.986+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/2636156271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.986+0000 7ff60b7fe640 1 -- 172.21.15.31:0/2636156271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 msgr2=0x560c8ae7e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:17.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.986+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/2636156271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 0x560c8ae7e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.986+0000 7ff60b7fe640 1 -- 172.21.15.31:0/2636156271 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c8ae896e0 con 0x560c8aadc390 2024-04-25T16:04:17.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.990+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/2636156271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7eea0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x560c8ae96f40 tx=0x7ff604004eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18a59467dace78ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:17.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.992+0000 7ff5f17fa640 1 -- 172.21.15.31:0/2636156271 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff604003970 con 0x560c8aadc390 2024-04-25T16:04:17.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.992+0000 7ff5f17fa640 1 -- 172.21.15.31:0/2636156271 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff60400c620 con 0x560c8aadc390 2024-04-25T16:04:17.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.992+0000 7ff5f17fa640 1 -- 172.21.15.31:0/2636156271 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6040281b0 con 0x560c8aadc390 2024-04-25T16:04:17.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.994+0000 7ff61905a740 1 -- 172.21.15.31:0/2636156271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 msgr2=0x560c8ae7eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:17.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.994+0000 7ff61905a740 1 --2- 172.21.15.31:0/2636156271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7eea0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x560c8ae96f40 tx=0x7ff604004eb0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.997+0000 7ff61905a740 1 -- 172.21.15.31:0/2636156271 shutdown_connections 2024-04-25T16:04:17.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.997+0000 7ff61905a740 1 --2- 172.21.15.31:0/2636156271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.997+0000 7ff61905a740 1 --2- 172.21.15.31:0/2636156271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aadc390 0x560c8ae7eea0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.997+0000 7ff61905a740 1 --2- 172.21.15.31:0/2636156271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aabb6b0 0x560c8ae7e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:17.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.997+0000 7ff61905a740 1 -- 172.21.15.31:0/2636156271 >> 172.21.15.31:0/2636156271 conn(0x560c8aad04e0 msgr2=0x560c8ae87bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:17.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.998+0000 7ff61905a740 1 -- 172.21.15.31:0/2636156271 shutdown_connections 2024-04-25T16:04:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.998+0000 7ff61905a740 1 -- 172.21.15.31:0/2636156271 wait complete. 2024-04-25T16:04:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.998+0000 7ff61905a740 1 -- start start 2024-04-25T16:04:18.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:17.999+0000 7ff61905a740 1 Processor -- start 2024-04-25T16:04:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff61905a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff61905a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aabb6b0 0x560c8ae8c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff61905a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aadc390 0x560c8ae8c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff61905a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f800f9c0 con 0x560c8aadc390 2024-04-25T16:04:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff61905a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f800f2f0 con 0x560c8aa703a0 2024-04-25T16:04:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff61905a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f800c530 con 0x560c8aabb6b0 2024-04-25T16:04:18.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff61905a740 10 client.0 _ll_register_callbacks cb 0x7ff5fc015ad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff60affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff60affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37116/0 (socket says 172.21.15.31:37116) 2024-04-25T16:04:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.000+0000 7ff60affd640 1 -- 172.21.15.31:0/512381787 learned_addr learned my addr 172.21.15.31:0/512381787 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:18.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.001+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aabb6b0 0x560c8ae8c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.001+0000 7ff60affd640 1 -- 172.21.15.31:0/512381787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aabb6b0 msgr2=0x560c8ae8c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.001+0000 7ff60affd640 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aabb6b0 0x560c8ae8c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.001+0000 7ff60affd640 1 -- 172.21.15.31:0/512381787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aadc390 msgr2=0x560c8ae8c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:18.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.001+0000 7ff60affd640 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aadc390 0x560c8ae8c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.002+0000 7ff60affd640 1 -- 172.21.15.31:0/512381787 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c8aefcb40 con 0x560c8aa703a0 2024-04-25T16:04:18.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.004+0000 7ff60affd640 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f7b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc011f10 tx=0x7ff5fc00ff60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbaf2faa660ad47e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:18.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.004+0000 7ff5eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:18.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.005+0000 7ff5eaffd640 1 -- 172.21.15.31:0/512381787 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc031030 con 0x560c8aa703a0 2024-04-25T16:04:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.005+0000 7ff5eaffd640 1 -- 172.21.15.31:0/512381787 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff5fc01c040 con 0x560c8aa703a0 2024-04-25T16:04:18.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.006+0000 7ff61905a740 1 -- 172.21.15.31:0/512381787 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff60400c620 con 0x560c8aa703a0 2024-04-25T16:04:18.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.009+0000 7ff5eaffd640 1 -- 172.21.15.31:0/512381787 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff5fc03b050 con 0x560c8aa703a0 2024-04-25T16:04:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.015+0000 7ff61905a740 10 client.25898 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:18.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.015+0000 7ff61905a740 1 -- 172.21.15.31:0/512381787 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c8aefdc50 con 0x560c8aa703a0 2024-04-25T16:04:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.017+0000 7ff5eaffd640 1 -- 172.21.15.31:0/512381787 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff60400c620 con 0x560c8aa703a0 2024-04-25T16:04:18.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.018+0000 7ff5eaffd640 1 -- 172.21.15.31:0/512381787 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7ff5fc004270 con 0x560c8aa703a0 2024-04-25T16:04:18.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.021+0000 7ff61905a740 10 client.25898 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:18.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.021+0000 7ff61905a740 10 client.25898 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:04:18.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.021+0000 7ff61905a740 20 client.25898 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:18.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.021+0000 7ff61905a740 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x560c8aefe5a0 0x560c8af009c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.021+0000 7ff61905a740 4 client.25898 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:04:18.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.021+0000 7ff61905a740 1 -- 172.21.15.31:0/512381787 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c8adf0990 con 0x560c8aefe5a0 2024-04-25T16:04:18.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.030+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x560c8aefe5a0 0x560c8af009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.031+0000 7ff5eaffd640 10 client.25898 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:04:18.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.031+0000 7ff60b7fe640 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x560c8aefe5a0 0x560c8af009c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7d640fe1f30f0631 server_cookie=13b0d458adc700ee in_seq=0 out_seq=0 2024-04-25T16:04:18.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.032+0000 7ff5eaffd640 1 -- 172.21.15.31:0/512381787 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c8adf0990 con 0x560c8aefe5a0 2024-04-25T16:04:18.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.032+0000 7ff5eaffd640 10 client.25898 handle_command_reply: tid=0 2024-04-25T16:04:18.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.032+0000 7ff61905a740 1 client.25898 shutdown 2024-04-25T16:04:18.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.033+0000 7ff61905a740 1 -- 172.21.15.31:0/512381787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 msgr2=0x560c8ae8f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.033+0000 7ff61905a740 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f7b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc011f10 tx=0x7ff5fc00ff60 comp rx=0 tx=0).stop 2024-04-25T16:04:18.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.033+0000 7ff61905a740 1 -- 172.21.15.31:0/512381787 shutdown_connections 2024-04-25T16:04:18.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.033+0000 7ff61905a740 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x560c8aefe5a0 0x560c8af009c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.033+0000 7ff61905a740 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560c8aadc390 0x560c8ae8c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.033+0000 7ff61905a740 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560c8aabb6b0 0x560c8ae8c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.033+0000 7ff61905a740 1 --2- 172.21.15.31:0/512381787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560c8aa703a0 0x560c8ae8f7b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.033+0000 7ff5eaffd640 0 client.25898 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:04:18.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.033+0000 7ff61905a740 1 -- 172.21.15.31:0/512381787 >> 172.21.15.31:0/512381787 conn(0x560c8aad04e0 msgr2=0x560c8ae88540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:18.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.035+0000 7ff61905a740 1 -- 172.21.15.31:0/512381787 shutdown_connections 2024-04-25T16:04:18.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.036+0000 7ff61905a740 1 -- 172.21.15.31:0/512381787 wait complete. 2024-04-25T16:04:18.043 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:04:18.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.039+0000 7ff61905a740 20 client.25898 trim_cache size 0 max 16384 2024-04-25T16:04:18.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.049+0000 7ff6121a8640 1 -- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5f40afd00 msgr2=0x7ff5f40b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.050+0000 7ff6121a8640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5f40afd00 0x7ff5f40b2210 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff6000c67a0 tx=0x7ff600032f10 comp rx=0 tx=0).stop 2024-04-25T16:04:18.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.050+0000 7ff6121a8640 1 -- 172.21.15.31:0/2011303221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c10fef0 msgr2=0x7ff60c1184d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.050+0000 7ff6121a8640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c10fef0 0x7ff60c1184d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6000303f0 tx=0x7ff600032df0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.057+0000 7ff6121a8640 1 -- 172.21.15.31:0/2011303221 shutdown_connections 2024-04-25T16:04:18.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.057+0000 7ff6121a8640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff5f40afd00 0x7ff5f40b2210 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.057+0000 7ff6121a8640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c10fef0 0x7ff60c1184d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.057+0000 7ff6121a8640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff60c10b1d0 0x7ff60c117570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.057+0000 7ff6121a8640 1 --2- 172.21.15.31:0/2011303221 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff60c10a3e0 0x7ff60c117030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.057+0000 7ff6121a8640 1 -- 172.21.15.31:0/2011303221 >> 172.21.15.31:0/2011303221 conn(0x7ff60c070400 msgr2=0x7ff60c10bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:18.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.058+0000 7ff6121a8640 1 -- 172.21.15.31:0/2011303221 shutdown_connections 2024-04-25T16:04:18.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.058+0000 7ff6121a8640 1 -- 172.21.15.31:0/2011303221 wait complete. 2024-04-25T16:04:18.095 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:17 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2530663106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:04:18.099 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:04:18.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:17 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2530663106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:04:18.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:17 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2530663106' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:04:18.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.610+0000 7fc72bef2640 1 -- 172.21.15.31:0/3626862593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc72410a420 msgr2=0x7fc72410add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.610+0000 7fc72bef2640 1 --2- 172.21.15.31:0/3626862593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc72410a420 0x7fc72410add0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009ca0 tx=0x7fc71c02a8f0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.613+0000 7fc72bef2640 1 -- 172.21.15.31:0/3626862593 shutdown_connections 2024-04-25T16:04:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.613+0000 7fc72bef2640 1 --2- 172.21.15.31:0/3626862593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc724110030 0x7fc724110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.613+0000 7fc72bef2640 1 --2- 172.21.15.31:0/3626862593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc72410b310 0x7fc72410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.613+0000 7fc72bef2640 1 --2- 172.21.15.31:0/3626862593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc72410a420 0x7fc72410add0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.613+0000 7fc72bef2640 1 -- 172.21.15.31:0/3626862593 >> 172.21.15.31:0/3626862593 conn(0x7fc7240704e0 msgr2=0x7fc724072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:18.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.624+0000 7fc72bef2640 1 -- 172.21.15.31:0/3626862593 shutdown_connections 2024-04-25T16:04:18.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.625+0000 7fc72bef2640 1 -- 172.21.15.31:0/3626862593 wait complete. 2024-04-25T16:04:18.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.626+0000 7fc72bef2640 1 Processor -- start 2024-04-25T16:04:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.628+0000 7fc72bef2640 1 -- start start 2024-04-25T16:04:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.628+0000 7fc72bef2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc72410a420 0x7fc724117010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.628+0000 7fc72bef2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc72410b310 0x7fc724117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.628+0000 7fc72bef2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc724110030 0x7fc7241184b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.628+0000 7fc72bef2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc72411e4a0 con 0x7fc72410a420 2024-04-25T16:04:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.628+0000 7fc72bef2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc72411e320 con 0x7fc72410b310 2024-04-25T16:04:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.628+0000 7fc72bef2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc72411e620 con 0x7fc724110030 2024-04-25T16:04:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.631+0000 7fc72a48f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc724110030 0x7fc7241184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.631+0000 7fc72948d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc72410b310 0x7fc724117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.631+0000 7fc72a48f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc724110030 0x7fc7241184b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45864/0 (socket says 172.21.15.31:45864) 2024-04-25T16:04:18.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.631+0000 7fc72a48f640 1 -- 172.21.15.31:0/1828885996 learned_addr learned my addr 172.21.15.31:0/1828885996 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:18.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.636+0000 7fc729c8e640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc72410a420 0x7fc724117010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.638+0000 7fc729c8e640 1 -- 172.21.15.31:0/1828885996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc724110030 msgr2=0x7fc7241184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.638+0000 7fc729c8e640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc724110030 0x7fc7241184b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.638+0000 7fc729c8e640 1 -- 172.21.15.31:0/1828885996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc72410b310 msgr2=0x7fc724117550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.638+0000 7fc729c8e640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc72410b310 0x7fc724117550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.638+0000 7fc729c8e640 1 -- 172.21.15.31:0/1828885996 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc724118a70 con 0x7fc72410a420 2024-04-25T16:04:18.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.640+0000 7fc729c8e640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc72410a420 0x7fc724117010 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc71c02aee0 tx=0x7fc71c030bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:18.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.641+0000 7fc71affd640 1 -- 172.21.15.31:0/1828885996 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c045620 con 0x7fc72410a420 2024-04-25T16:04:18.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.641+0000 7fc72bef2640 1 -- 172.21.15.31:0/1828885996 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc724118d00 con 0x7fc72410a420 2024-04-25T16:04:18.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.642+0000 7fc72bef2640 1 -- 172.21.15.31:0/1828885996 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc724119160 con 0x7fc72410a420 2024-04-25T16:04:18.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.642+0000 7fc71affd640 1 -- 172.21.15.31:0/1828885996 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71c02d070 con 0x7fc72410a420 2024-04-25T16:04:18.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.642+0000 7fc71affd640 1 -- 172.21.15.31:0/1828885996 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c03bb10 con 0x7fc72410a420 2024-04-25T16:04:18.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.651+0000 7fc71affd640 1 -- 172.21.15.31:0/1828885996 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc71c03bcb0 con 0x7fc72410a420 2024-04-25T16:04:18.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.653+0000 7fc71affd640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7100afcf0 0x7fc7100b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.655+0000 7fc71affd640 1 -- 172.21.15.31:0/1828885996 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc71c100b80 con 0x7fc72410a420 2024-04-25T16:04:18.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.666+0000 7fc72948d640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7100afcf0 0x7fc7100b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.666+0000 7fc732da6740 1 Processor -- start 2024-04-25T16:04:18.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.666+0000 7fc732da6740 1 -- start start 2024-04-25T16:04:18.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.667+0000 7fc732da6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 0x5633dde9d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.667+0000 7fc732da6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633dde9dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.667+0000 7fc732da6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddada6b0 0x5633dde9ea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.667+0000 7fc732da6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5633dde01d80 con 0x5633ddafb390 2024-04-25T16:04:18.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.667+0000 7fc732da6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5633dddbba20 con 0x5633ddada6b0 2024-04-25T16:04:18.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.667+0000 7fc732da6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5633dddfdd40 con 0x5633ddad24d0 2024-04-25T16:04:18.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.668+0000 7fc729c8e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 0x5633dde9d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.668+0000 7fc72948d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633dde9dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.668+0000 7fc72948d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633dde9dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45878/0 (socket says 172.21.15.31:45878) 2024-04-25T16:04:18.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.668+0000 7fc72948d640 1 -- 172.21.15.31:0/3692145463 learned_addr learned my addr 172.21.15.31:0/3692145463 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:18.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.669+0000 7fc72948d640 1 -- 172.21.15.31:0/3692145463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddada6b0 msgr2=0x5633dde9ea20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:04:18.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.669+0000 7fc72948d640 1 --2- 172.21.15.31:0/3692145463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddada6b0 0x5633dde9ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.669+0000 7fc72948d640 1 -- 172.21.15.31:0/3692145463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 msgr2=0x5633dde9d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:18.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.669+0000 7fc72948d640 1 --2- 172.21.15.31:0/3692145463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 0x5633dde9d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.669+0000 7fc72948d640 1 -- 172.21.15.31:0/3692145463 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633dddc6c30 con 0x5633ddad24d0 2024-04-25T16:04:18.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.670+0000 7fc72948d640 1 --2- 172.21.15.31:0/3692145463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633dde9dd90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc714012bb0 tx=0x7fc71400e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=830e4b363513c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:18.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.677+0000 7fc70ffff640 1 -- 172.21.15.31:0/3692145463 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714023070 con 0x5633ddad24d0 2024-04-25T16:04:18.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.677+0000 7fc70ffff640 1 -- 172.21.15.31:0/3692145463 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71400fa70 con 0x5633ddad24d0 2024-04-25T16:04:18.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.677+0000 7fc70ffff640 1 -- 172.21.15.31:0/3692145463 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71401b8d0 con 0x5633ddad24d0 2024-04-25T16:04:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.679+0000 7fc732da6740 1 -- 172.21.15.31:0/3692145463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 msgr2=0x5633dde9dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.679+0000 7fc732da6740 1 --2- 172.21.15.31:0/3692145463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633dde9dd90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc714012bb0 tx=0x7fc71400e720 comp rx=0 tx=0).stop 2024-04-25T16:04:18.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.680+0000 7fc732da6740 1 -- 172.21.15.31:0/3692145463 shutdown_connections 2024-04-25T16:04:18.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.680+0000 7fc732da6740 1 --2- 172.21.15.31:0/3692145463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddada6b0 0x5633dde9ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.680+0000 7fc732da6740 1 --2- 172.21.15.31:0/3692145463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633dde9dd90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.680+0000 7fc732da6740 1 --2- 172.21.15.31:0/3692145463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 0x5633dde9d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.680+0000 7fc732da6740 1 -- 172.21.15.31:0/3692145463 >> 172.21.15.31:0/3692145463 conn(0x5633ddaef4e0 msgr2=0x5633dde99020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:18.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.688+0000 7fc72948d640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7100afcf0 0x7fc7100b2200 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc714020000 tx=0x7fc71401c730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.705+0000 7fc732da6740 1 -- 172.21.15.31:0/3692145463 shutdown_connections 2024-04-25T16:04:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.711+0000 7fc732da6740 1 -- 172.21.15.31:0/3692145463 wait complete. 2024-04-25T16:04:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.712+0000 7fc732da6740 1 -- start start 2024-04-25T16:04:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.712+0000 7fc732da6740 1 Processor -- start 2024-04-25T16:04:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.713+0000 7fc732da6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddad24d0 0x5633ddea7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.713+0000 7fc732da6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddea7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.713+0000 7fc732da6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 0x5633ddeae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.713+0000 7fc732da6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5633dde9f530 con 0x5633ddafb390 2024-04-25T16:04:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.713+0000 7fc732da6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5633ddd88dc0 con 0x5633ddad24d0 2024-04-25T16:04:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.713+0000 7fc732da6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5633dde9f6b0 con 0x5633ddada6b0 2024-04-25T16:04:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.713+0000 7fc732da6740 10 client.0 _ll_register_callbacks cb 0x5633ddda4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.722+0000 7fc72a48f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 0x5633ddeae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.722+0000 7fc729c8e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddad24d0 0x5633ddea7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.727+0000 7fc729c8e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddad24d0 0x5633ddea7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37154/0 (socket says 172.21.15.31:37154) 2024-04-25T16:04:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.727+0000 7fc729c8e640 1 -- 172.21.15.31:0/2445176083 learned_addr learned my addr 172.21.15.31:0/2445176083 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:18.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.727+0000 7fc72948d640 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddea7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.728+0000 7fc729c8e640 1 -- 172.21.15.31:0/2445176083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 msgr2=0x5633ddea7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:18.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.728+0000 7fc729c8e640 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddea7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.728+0000 7fc729c8e640 1 -- 172.21.15.31:0/2445176083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 msgr2=0x5633ddeae9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:18.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.728+0000 7fc729c8e640 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 0x5633ddeae9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.728+0000 7fc729c8e640 1 -- 172.21.15.31:0/2445176083 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633ddb133f0 con 0x5633ddad24d0 2024-04-25T16:04:18.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.734+0000 7fc729c8e640 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddad24d0 0x5633ddea7520 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc71c0c6280 tx=0x7fc71c031ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe894a32d4ab62f8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:18.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.734+0000 7fc70d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:18.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.738+0000 7fc70d7fa640 1 -- 172.21.15.31:0/2445176083 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c0c9710 con 0x5633ddad24d0 2024-04-25T16:04:18.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.738+0000 7fc70d7fa640 1 -- 172.21.15.31:0/2445176083 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc71c0c5600 con 0x5633ddad24d0 2024-04-25T16:04:18.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.739+0000 7fc732da6740 1 -- 172.21.15.31:0/2445176083 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc71400fa70 con 0x5633ddad24d0 2024-04-25T16:04:18.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.742+0000 7fc70d7fa640 1 -- 172.21.15.31:0/2445176083 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc71c0e8050 con 0x5633ddad24d0 2024-04-25T16:04:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.745+0000 7fc732da6740 10 client.25907 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:18.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.746+0000 7fc732da6740 1 -- 172.21.15.31:0/2445176083 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5633ddbdd7d0 con 0x5633ddad24d0 2024-04-25T16:04:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.751+0000 7fc70d7fa640 1 -- 172.21.15.31:0/2445176083 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71400fa70 con 0x5633ddad24d0 2024-04-25T16:04:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.751+0000 7fc70d7fa640 1 -- 172.21.15.31:0/2445176083 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fc71c04d030 con 0x5633ddad24d0 2024-04-25T16:04:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.751+0000 7fc732da6740 10 client.25907 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.751+0000 7fc732da6740 10 client.25907 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:04:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.751+0000 7fc732da6740 20 client.25907 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:18.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.751+0000 7fc732da6740 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5633ddab9f60 0x5633ddeb52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.751+0000 7fc732da6740 4 client.25907 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:18.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.751+0000 7fc732da6740 1 -- 172.21.15.31:0/2445176083 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5633dde0f990 con 0x5633ddab9f60 2024-04-25T16:04:18.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.757+0000 7fc72948d640 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5633ddab9f60 0x5633ddeb52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.773+0000 7fc72948d640 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5633ddab9f60 0x5633ddeb52e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5b847e86c10d0c7f server_cookie=101f28141a8423f8 in_seq=0 out_seq=0 2024-04-25T16:04:18.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.773+0000 7fc70d7fa640 10 client.25907 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:04:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.780+0000 7fc70d7fa640 1 -- 172.21.15.31:0/2445176083 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5633dde0f990 con 0x5633ddab9f60 2024-04-25T16:04:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.780+0000 7fc70d7fa640 10 client.25907 handle_command_reply: tid=0 2024-04-25T16:04:18.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.780+0000 7fc732da6740 1 client.25907 shutdown 2024-04-25T16:04:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.780+0000 7fc732da6740 1 -- 172.21.15.31:0/2445176083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddad24d0 msgr2=0x5633ddea7520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.780+0000 7fc732da6740 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddad24d0 0x5633ddea7520 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc71c0c6280 tx=0x7fc71c031ea0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.781+0000 7fc732da6740 1 -- 172.21.15.31:0/2445176083 shutdown_connections 2024-04-25T16:04:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.781+0000 7fc732da6740 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddafb390 0x5633ddeae9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.781+0000 7fc732da6740 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5633ddab9f60 0x5633ddeb52e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.781+0000 7fc732da6740 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddea7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.781+0000 7fc732da6740 1 --2- 172.21.15.31:0/2445176083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddad24d0 0x5633ddea7520 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.781+0000 7fc732da6740 1 -- 172.21.15.31:0/2445176083 >> 172.21.15.31:0/2445176083 conn(0x5633ddaef4e0 msgr2=0x5633dde9bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:18.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.781+0000 7fc70d7fa640 0 client.25907 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:04:18.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.790+0000 7fc732da6740 1 -- 172.21.15.31:0/2445176083 shutdown_connections 2024-04-25T16:04:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.791+0000 7fc732da6740 1 -- 172.21.15.31:0/2445176083 wait complete. 2024-04-25T16:04:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.791+0000 7fc732da6740 20 client.25907 trim_cache size 0 max 16384 2024-04-25T16:04:18.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.868+0000 7fc732da6740 1 Processor -- start 2024-04-25T16:04:18.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.874+0000 7fc732da6740 1 -- start start 2024-04-25T16:04:18.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.874+0000 7fc732da6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633ddea2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.875+0000 7fc732da6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddada6b0 0x5633ddea2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.875+0000 7fc732da6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddea67d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.875+0000 7fc732da6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800e500 con 0x5633ddada6b0 2024-04-25T16:04:18.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.875+0000 7fc732da6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc708010070 con 0x5633ddafb390 2024-04-25T16:04:18.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.875+0000 7fc732da6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800ebd0 con 0x5633ddad24d0 2024-04-25T16:04:18.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.876+0000 7fc72a48f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddea67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.876+0000 7fc72a48f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddea67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37170/0 (socket says 172.21.15.31:37170) 2024-04-25T16:04:18.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.876+0000 7fc72a48f640 1 -- 172.21.15.31:0/3566292857 learned_addr learned my addr 172.21.15.31:0/3566292857 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:18.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.876+0000 7fc729c8e640 1 --2- 172.21.15.31:0/3566292857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633ddea2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.879+0000 7fc72948d640 1 --2- 172.21.15.31:0/3566292857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddada6b0 0x5633ddea2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.879+0000 7fc729c8e640 1 -- 172.21.15.31:0/3566292857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 msgr2=0x5633ddea67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.879+0000 7fc729c8e640 1 --2- 172.21.15.31:0/3566292857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddea67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.879+0000 7fc729c8e640 1 -- 172.21.15.31:0/3566292857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddada6b0 msgr2=0x5633ddea2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.879+0000 7fc729c8e640 1 --2- 172.21.15.31:0/3566292857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddada6b0 0x5633ddea2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.879+0000 7fc729c8e640 1 -- 172.21.15.31:0/3566292857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5633ddea8660 con 0x5633ddad24d0 2024-04-25T16:04:18.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.881+0000 7fc729c8e640 1 --2- 172.21.15.31:0/3566292857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633ddea2210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc71c006010 tx=0x7fc71c031f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a784c2dadc18dbae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.882+0000 7fc70f7fe640 1 -- 172.21.15.31:0/3566292857 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71c0c8960 con 0x5633ddad24d0 2024-04-25T16:04:18.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.882+0000 7fc70f7fe640 1 -- 172.21.15.31:0/3566292857 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71c036050 con 0x5633ddad24d0 2024-04-25T16:04:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.884+0000 7fc732da6740 1 -- 172.21.15.31:0/3566292857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 msgr2=0x5633ddea2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.884+0000 7fc732da6740 1 --2- 172.21.15.31:0/3566292857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633ddea2210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc71c006010 tx=0x7fc71c031f10 comp rx=0 tx=0).stop 2024-04-25T16:04:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.890+0000 7fc732da6740 1 -- 172.21.15.31:0/3566292857 shutdown_connections 2024-04-25T16:04:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.890+0000 7fc732da6740 1 --2- 172.21.15.31:0/3566292857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddea67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.890+0000 7fc732da6740 1 --2- 172.21.15.31:0/3566292857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddada6b0 0x5633ddea2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.890+0000 7fc732da6740 1 --2- 172.21.15.31:0/3566292857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddad24d0 0x5633ddea2210 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.890+0000 7fc732da6740 1 -- 172.21.15.31:0/3566292857 >> 172.21.15.31:0/3566292857 conn(0x5633ddaef4e0 msgr2=0x5633ddea74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:18.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.890+0000 7fc732da6740 1 -- 172.21.15.31:0/3566292857 shutdown_connections 2024-04-25T16:04:18.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.890+0000 7fc732da6740 1 -- 172.21.15.31:0/3566292857 wait complete. 2024-04-25T16:04:18.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.891+0000 7fc732da6740 1 -- start start 2024-04-25T16:04:18.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.892+0000 7fc732da6740 1 Processor -- start 2024-04-25T16:04:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.892+0000 7fc732da6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddad24d0 0x5633ddeaadc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.892+0000 7fc732da6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddeab300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.892+0000 7fc732da6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddf1b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.892+0000 7fc732da6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800f970 con 0x5633ddad24d0 2024-04-25T16:04:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.892+0000 7fc732da6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800f2a0 con 0x5633ddafb390 2024-04-25T16:04:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.892+0000 7fc732da6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc70800c4e0 con 0x5633ddada6b0 2024-04-25T16:04:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.892+0000 7fc732da6740 10 client.0 _ll_register_callbacks cb 0x7fc71c0c4ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.893+0000 7fc72a48f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddf1b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.893+0000 7fc72948d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddeab300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.893+0000 7fc72a48f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddf1b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37184/0 (socket says 172.21.15.31:37184) 2024-04-25T16:04:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.893+0000 7fc72a48f640 1 -- 172.21.15.31:0/1978370884 learned_addr learned my addr 172.21.15.31:0/1978370884 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:18.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.893+0000 7fc72948d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddeab300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45904/0 (socket says 172.21.15.31:45904) 2024-04-25T16:04:18.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.894+0000 7fc72948d640 1 -- 172.21.15.31:0/1978370884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 msgr2=0x5633ddf1b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:18.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.894+0000 7fc72948d640 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddf1b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.894+0000 7fc72948d640 1 -- 172.21.15.31:0/1978370884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddad24d0 msgr2=0x5633ddeaadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:18.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.894+0000 7fc72948d640 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddad24d0 0x5633ddeaadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.894+0000 7fc72948d640 1 -- 172.21.15.31:0/1978370884 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5633ddeae2c0 con 0x5633ddada6b0 2024-04-25T16:04:18.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.897+0000 7fc72948d640 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddeab300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5633ddea8590 tx=0x7fc714011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea96241aa3e3c1ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:18.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.903+0000 7fc70d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:04:18.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.903+0000 7fc70d7fa640 1 -- 172.21.15.31:0/1978370884 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714023040 con 0x5633ddada6b0 2024-04-25T16:04:18.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.903+0000 7fc732da6740 1 -- 172.21.15.31:0/1978370884 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc71c036050 con 0x5633ddada6b0 2024-04-25T16:04:18.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.903+0000 7fc70d7fa640 1 -- 172.21.15.31:0/1978370884 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc71401cc60 con 0x5633ddada6b0 2024-04-25T16:04:18.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.904+0000 7fc70d7fa640 1 -- 172.21.15.31:0/1978370884 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc71400ac30 con 0x5633ddada6b0 2024-04-25T16:04:18.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.906+0000 7fc732da6740 10 client.25873 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:18.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.906+0000 7fc732da6740 1 -- 172.21.15.31:0/1978370884 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5633ddf1cc10 con 0x5633ddada6b0 2024-04-25T16:04:18.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.908+0000 7fc70d7fa640 1 -- 172.21.15.31:0/1978370884 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc71c036050 con 0x5633ddada6b0 2024-04-25T16:04:18.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.908+0000 7fc70d7fa640 1 -- 172.21.15.31:0/1978370884 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fc7140313d0 con 0x5633ddada6b0 2024-04-25T16:04:18.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.908+0000 7fc732da6740 10 client.25873 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:18.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.908+0000 7fc732da6740 10 client.25873 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:04:18.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.908+0000 7fc732da6740 20 client.25873 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:18.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.908+0000 7fc732da6740 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5633dda8f3a0 0x5633ddf1f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:18.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.908+0000 7fc732da6740 4 client.25873 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:04:18.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.908+0000 7fc732da6740 1 -- 172.21.15.31:0/1978370884 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5633dde09800 con 0x5633dda8f3a0 2024-04-25T16:04:18.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.910+0000 7fc729c8e640 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5633dda8f3a0 0x5633ddf1f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:18.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.916+0000 7fc729c8e640 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5633dda8f3a0 0x5633ddf1f5e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ee1820c02bbf7ddc server_cookie=3c2cbfbecb0eeb3c in_seq=0 out_seq=0 2024-04-25T16:04:18.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.916+0000 7fc70d7fa640 10 client.25873 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:04:18.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.918+0000 7fc70d7fa640 1 -- 172.21.15.31:0/1978370884 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5633dde09800 con 0x5633dda8f3a0 2024-04-25T16:04:18.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.918+0000 7fc70d7fa640 10 client.25873 handle_command_reply: tid=0 2024-04-25T16:04:18.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.918+0000 7fc732da6740 1 client.25873 shutdown 2024-04-25T16:04:18.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.919+0000 7fc732da6740 1 -- 172.21.15.31:0/1978370884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 msgr2=0x5633ddeab300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.919+0000 7fc732da6740 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddeab300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5633ddea8590 tx=0x7fc714011000 comp rx=0 tx=0).stop 2024-04-25T16:04:18.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.920+0000 7fc732da6740 1 -- 172.21.15.31:0/1978370884 shutdown_connections 2024-04-25T16:04:18.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.920+0000 7fc732da6740 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5633ddafb390 0x5633ddf1b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.920+0000 7fc732da6740 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x5633dda8f3a0 0x5633ddf1f5e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.920+0000 7fc732da6740 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5633ddada6b0 0x5633ddeab300 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.920+0000 7fc732da6740 1 --2- 172.21.15.31:0/1978370884 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5633ddad24d0 0x5633ddeaadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.920+0000 7fc732da6740 1 -- 172.21.15.31:0/1978370884 >> 172.21.15.31:0/1978370884 conn(0x5633ddaef4e0 msgr2=0x5633ddea67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:18.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.920+0000 7fc70d7fa640 0 client.25873 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:04:18.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.920+0000 7fc732da6740 1 -- 172.21.15.31:0/1978370884 shutdown_connections 2024-04-25T16:04:18.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.920+0000 7fc732da6740 1 -- 172.21.15.31:0/1978370884 wait complete. 2024-04-25T16:04:18.935 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:04:18.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.930+0000 7fc732da6740 20 client.25873 trim_cache size 0 max 16384 2024-04-25T16:04:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.943+0000 7fc72bef2640 1 -- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7100afcf0 msgr2=0x7fc7100b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.943+0000 7fc72bef2640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7100afcf0 0x7fc7100b2200 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc714020000 tx=0x7fc71401c730 comp rx=0 tx=0).stop 2024-04-25T16:04:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.943+0000 7fc72bef2640 1 -- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc72410a420 msgr2=0x7fc724117010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:18.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.943+0000 7fc72bef2640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc72410a420 0x7fc724117010 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc71c02aee0 tx=0x7fc71c030bb0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.949+0000 7fc72bef2640 1 -- 172.21.15.31:0/1828885996 shutdown_connections 2024-04-25T16:04:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.949+0000 7fc72bef2640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7100afcf0 0x7fc7100b2200 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.949+0000 7fc72bef2640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc724110030 0x7fc7241184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.949+0000 7fc72bef2640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc72410b310 0x7fc724117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.949+0000 7fc72bef2640 1 --2- 172.21.15.31:0/1828885996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc72410a420 0x7fc724117010 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:18.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.949+0000 7fc72bef2640 1 -- 172.21.15.31:0/1828885996 >> 172.21.15.31:0/1828885996 conn(0x7fc7240704e0 msgr2=0x7fc72410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:18.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.955+0000 7fc72bef2640 1 -- 172.21.15.31:0/1828885996 shutdown_connections 2024-04-25T16:04:18.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:18.955+0000 7fc72bef2640 1 -- 172.21.15.31:0/1828885996 wait complete. 2024-04-25T16:04:19.010 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:04:19.064 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.063+0000 7f8d5b492640 1 -- 172.21.15.31:0/845522006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d5410b1e0 msgr2=0x7f8d5410b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.065+0000 7f8d5b492640 1 --2- 172.21.15.31:0/845522006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d5410b1e0 0x7f8d5410b5f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8d44009900 tx=0x7f8d4402a410 comp rx=0 tx=0).stop 2024-04-25T16:04:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.066+0000 7f8d5b492640 1 -- 172.21.15.31:0/845522006 shutdown_connections 2024-04-25T16:04:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.066+0000 7f8d5b492640 1 --2- 172.21.15.31:0/845522006 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5410fee0 0x7f8d54112300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.066+0000 7f8d5b492640 1 --2- 172.21.15.31:0/845522006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d5410b1e0 0x7f8d5410b5f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.066+0000 7f8d5b492640 1 --2- 172.21.15.31:0/845522006 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d54109fa0 0x7f8d5410aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.066+0000 7f8d5b492640 1 -- 172.21.15.31:0/845522006 >> 172.21.15.31:0/845522006 conn(0x7f8d54078bc0 msgr2=0x7f8d541079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.075+0000 7f8d5b492640 1 -- 172.21.15.31:0/845522006 shutdown_connections 2024-04-25T16:04:19.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.076+0000 7f8d5b492640 1 -- 172.21.15.31:0/845522006 wait complete. 2024-04-25T16:04:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.080+0000 7f8d5b492640 1 Processor -- start 2024-04-25T16:04:19.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.080+0000 7f8d5b492640 1 -- start start 2024-04-25T16:04:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.081+0000 7f8d5b492640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d54109fa0 0x7f8d541a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.083+0000 7f8d5b492640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5410b1e0 0x7f8d541a0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.083+0000 7f8d5922e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d54109fa0 0x7f8d541a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.083+0000 7f8d5922e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d54109fa0 0x7f8d541a0630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45926/0 (socket says 172.21.15.31:45926) 2024-04-25T16:04:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.084+0000 7f8d5b492640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d5410fee0 0x7f8d541a7b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.084+0000 7f8d5922e640 1 -- 172.21.15.31:0/1073322758 learned_addr learned my addr 172.21.15.31:0/1073322758 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.084+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54115b10 con 0x7f8d5410b1e0 2024-04-25T16:04:19.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.084+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54115990 con 0x7f8d5410fee0 2024-04-25T16:04:19.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.085+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54115c90 con 0x7f8d54109fa0 2024-04-25T16:04:19.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.087+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5410b1e0 0x7f8d541a0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.088+0000 7f8d58a2d640 1 -- 172.21.15.31:0/1073322758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d54109fa0 msgr2=0x7f8d541a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.088+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d54109fa0 0x7f8d541a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.088+0000 7f8d58a2d640 1 -- 172.21.15.31:0/1073322758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d5410fee0 msgr2=0x7f8d541a7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.088+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d5410fee0 0x7f8d541a7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.088+0000 7f8d58a2d640 1 -- 172.21.15.31:0/1073322758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d541a8170 con 0x7f8d5410b1e0 2024-04-25T16:04:19.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.089+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5410b1e0 0x7f8d541a0b70 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8d4402aa00 tx=0x7f8d44003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.091+0000 7f8d4a7fc640 1 -- 172.21.15.31:0/1073322758 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d4403f070 con 0x7f8d5410b1e0 2024-04-25T16:04:19.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.091+0000 7f8d4a7fc640 1 -- 172.21.15.31:0/1073322758 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d44031c50 con 0x7f8d5410b1e0 2024-04-25T16:04:19.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.092+0000 7f8d4a7fc640 1 -- 172.21.15.31:0/1073322758 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d44035030 con 0x7f8d5410b1e0 2024-04-25T16:04:19.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.093+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d541a8400 con 0x7f8d5410b1e0 2024-04-25T16:04:19.097 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.093+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d541a8870 con 0x7f8d5410b1e0 2024-04-25T16:04:19.109 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.108+0000 7f8d62411740 1 Processor -- start 2024-04-25T16:04:19.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.108+0000 7f8d62411740 1 -- start start 2024-04-25T16:04:19.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.109+0000 7f8d62411740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f008b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:18 smithi063 ceph-mon[32413]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 914 MiB/s wr, 1.94k op/s 2024-04-25T16:04:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.117+0000 7f8d62411740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f0090a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.117+0000 7f8d62411740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 0x55d76f00ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.117+0000 7f8d62411740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d76ef6e460 con 0x55d76ec681b0 2024-04-25T16:04:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.117+0000 7f8d62411740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d76ef28590 con 0x55d76ec474d0 2024-04-25T16:04:19.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.117+0000 7f8d62411740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d76ef6a420 con 0x55d76ec3f2f0 2024-04-25T16:04:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.122+0000 7f8d5922e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f0090a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.123+0000 7f8d5922e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f0090a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45938/0 (socket says 172.21.15.31:45938) 2024-04-25T16:04:19.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.123+0000 7f8d5922e640 1 -- 172.21.15.31:0/521610268 learned_addr learned my addr 172.21.15.31:0/521610268 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.123+0000 7f8d59a2f640 1 --2- 172.21.15.31:0/521610268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f008b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.131+0000 7f8d4a7fc640 1 -- 172.21.15.31:0/1073322758 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8d44042070 con 0x7f8d5410b1e0 2024-04-25T16:04:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.131+0000 7f8d5922e640 1 -- 172.21.15.31:0/521610268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 msgr2=0x55d76f00ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:04:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.131+0000 7f8d5922e640 1 --2- 172.21.15.31:0/521610268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 0x55d76f00ffa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.131+0000 7f8d5922e640 1 -- 172.21.15.31:0/521610268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 msgr2=0x55d76f008b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.131+0000 7f8d5922e640 1 --2- 172.21.15.31:0/521610268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f008b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.131+0000 7f8d5922e640 1 -- 172.21.15.31:0/521610268 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d76ef350d0 con 0x55d76ec3f2f0 2024-04-25T16:04:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.132+0000 7f8d5922e640 1 --2- 172.21.15.31:0/521610268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f0090a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000bdb0 tx=0x7f8d5000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63454eeda96e0739 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.133+0000 7f8d4a7fc640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d400afc30 0x7f8d400b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.133+0000 7f8d3f7fe640 1 -- 172.21.15.31:0/521610268 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d50015070 con 0x55d76ec3f2f0 2024-04-25T16:04:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.133+0000 7f8d3f7fe640 1 -- 172.21.15.31:0/521610268 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d500118f0 con 0x55d76ec3f2f0 2024-04-25T16:04:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.133+0000 7f8d3f7fe640 1 -- 172.21.15.31:0/521610268 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d50010990 con 0x55d76ec3f2f0 2024-04-25T16:04:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.138+0000 7f8d62411740 1 -- 172.21.15.31:0/521610268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 msgr2=0x55d76f0090a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.138+0000 7f8d62411740 1 --2- 172.21.15.31:0/521610268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f0090a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000bdb0 tx=0x7f8d5000f730 comp rx=0 tx=0).stop 2024-04-25T16:04:19.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.143+0000 7f8d59a2f640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d400afc30 0x7f8d400b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.143+0000 7f8d4a7fc640 1 -- 172.21.15.31:0/1073322758 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8d440c3f00 con 0x7f8d5410b1e0 2024-04-25T16:04:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.145+0000 7f8d62411740 1 -- 172.21.15.31:0/521610268 shutdown_connections 2024-04-25T16:04:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.145+0000 7f8d62411740 1 --2- 172.21.15.31:0/521610268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 0x55d76f00ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.145+0000 7f8d62411740 1 --2- 172.21.15.31:0/521610268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f0090a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.145+0000 7f8d62411740 1 --2- 172.21.15.31:0/521610268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f008b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.145+0000 7f8d62411740 1 -- 172.21.15.31:0/521610268 >> 172.21.15.31:0/521610268 conn(0x55d76ec5c300 msgr2=0x55d76f0044b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:18 smithi186 ceph-mon[33258]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 914 MiB/s wr, 1.94k op/s 2024-04-25T16:04:19.154 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:18 smithi031 ceph-mon[29140]: pgmap v441: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 914 MiB/s wr, 1.94k op/s 2024-04-25T16:04:19.173 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.172+0000 7f8d62411740 1 -- 172.21.15.31:0/521610268 shutdown_connections 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.181+0000 7f8d62411740 1 -- 172.21.15.31:0/521610268 wait complete. 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.182+0000 7f8d62411740 1 -- start start 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.182+0000 7f8d62411740 1 Processor -- start 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.183+0000 7f8d62411740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f023860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.183+0000 7f8d62411740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 0x55d76f023da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.183+0000 7f8d62411740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f02ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.183+0000 7f8d62411740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d76f0109f0 con 0x55d76ec681b0 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.183+0000 7f8d62411740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d76eef5b60 con 0x55d76ec474d0 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.183+0000 7f8d62411740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d76f010b70 con 0x55d76ec3f2f0 2024-04-25T16:04:19.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.183+0000 7f8d62411740 10 client.0 _ll_register_callbacks cb 0x55d76ef1a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.187+0000 7f8d58a2d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 0x55d76f023da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.187+0000 7f8d58a2d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 0x55d76f023da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37232/0 (socket says 172.21.15.31:37232) 2024-04-25T16:04:19.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.187+0000 7f8d58a2d640 1 -- 172.21.15.31:0/217254452 learned_addr learned my addr 172.21.15.31:0/217254452 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.190 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.188+0000 7f8d5922e640 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f023860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.190+0000 7f8d58a2d640 1 -- 172.21.15.31:0/217254452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 msgr2=0x55d76f023860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.190+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f023860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.190+0000 7f8d58a2d640 1 -- 172.21.15.31:0/217254452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 msgr2=0x55d76f02ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.190+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f02ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.190+0000 7f8d58a2d640 1 -- 172.21.15.31:0/217254452 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d76ec80210 con 0x55d76ec474d0 2024-04-25T16:04:19.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.192+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 0x55d76f023da0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8d44004010 tx=0x7f8d4402a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f35fab8a4e78625 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.198 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.196+0000 7f8d59a2f640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d400afc30 0x7f8d400b2140 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c005c40 tx=0x7f8d4c005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.203+0000 7f8d3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.203+0000 7f8d3cff9640 1 -- 172.21.15.31:0/217254452 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d44049070 con 0x55d76ec474d0 2024-04-25T16:04:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.204+0000 7f8d3cff9640 1 -- 172.21.15.31:0/217254452 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8d440ea0b0 con 0x55d76ec474d0 2024-04-25T16:04:19.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.204+0000 7f8d62411740 1 -- 172.21.15.31:0/217254452 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d500118f0 con 0x55d76ec474d0 2024-04-25T16:04:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.212+0000 7f8d3cff9640 1 -- 172.21.15.31:0/217254452 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d440c7910 con 0x55d76ec474d0 2024-04-25T16:04:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.212+0000 7f8d62411740 10 client.25919 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.212+0000 7f8d62411740 1 -- 172.21.15.31:0/217254452 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d76ed4c620 con 0x55d76ec474d0 2024-04-25T16:04:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.213+0000 7f8d3cff9640 1 -- 172.21.15.31:0/217254452 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d500118f0 con 0x55d76ec474d0 2024-04-25T16:04:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.213+0000 7f8d3cff9640 1 -- 172.21.15.31:0/217254452 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f8d440c7b70 con 0x55d76ec474d0 2024-04-25T16:04:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.213+0000 7f8d62411740 10 client.25919 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.213+0000 7f8d62411740 10 client.25919 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.213+0000 7f8d62411740 20 client.25919 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.213+0000 7f8d62411740 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d76ec26d80 0x55d76f017540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.213+0000 7f8d62411740 4 client.25919 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:19.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.214+0000 7f8d62411740 1 -- 172.21.15.31:0/217254452 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d76ef7c070 con 0x55d76ec26d80 2024-04-25T16:04:19.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.221+0000 7f8d5922e640 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d76ec26d80 0x55d76f017540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.253+0000 7f8d3cff9640 10 client.25919 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.253+0000 7f8d5922e640 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d76ec26d80 0x55d76f017540 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef5b97c18f25aa17 server_cookie=ef512f12389198d1 in_seq=0 out_seq=0 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.266+0000 7f8d3cff9640 1 -- 172.21.15.31:0/217254452 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d76ef7c070 con 0x55d76ec26d80 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.266+0000 7f8d3cff9640 10 client.25919 handle_command_reply: tid=0 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.266+0000 7f8d62411740 1 client.25919 shutdown 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.266+0000 7f8d62411740 1 -- 172.21.15.31:0/217254452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 msgr2=0x55d76f023da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.266+0000 7f8d62411740 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 0x55d76f023da0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8d44004010 tx=0x7f8d4402a410 comp rx=0 tx=0).stop 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.267+0000 7f8d62411740 1 -- 172.21.15.31:0/217254452 shutdown_connections 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.267+0000 7f8d62411740 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d76ec26d80 0x55d76f017540 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.267+0000 7f8d62411740 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f02ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.267+0000 7f8d62411740 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec474d0 0x55d76f023da0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.267+0000 7f8d62411740 1 --2- 172.21.15.31:0/217254452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec3f2f0 0x55d76f023860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.267+0000 7f8d62411740 1 -- 172.21.15.31:0/217254452 >> 172.21.15.31:0/217254452 conn(0x55d76ec5c300 msgr2=0x55d76f002d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.267+0000 7f8d3cff9640 0 client.25919 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.267+0000 7f8d62411740 1 -- 172.21.15.31:0/217254452 shutdown_connections 2024-04-25T16:04:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.267+0000 7f8d62411740 1 -- 172.21.15.31:0/217254452 wait complete. 2024-04-25T16:04:19.269 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.268+0000 7f8d62411740 20 client.25919 trim_cache size 0 max 16384 2024-04-25T16:04:19.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.312+0000 7f8d62411740 1 Processor -- start 2024-04-25T16:04:19.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.312+0000 7f8d62411740 1 -- start start 2024-04-25T16:04:19.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.312+0000 7f8d62411740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec3f2f0 0x55d76f021d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.312+0000 7f8d62411740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f02ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.312+0000 7f8d62411740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f0222a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.312+0000 7f8d62411740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8d541a8e60 con 0x55d76ec681b0 2024-04-25T16:04:19.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.312+0000 7f8d62411740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d541aa9d0 con 0x55d76ec3f2f0 2024-04-25T16:04:19.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.312+0000 7f8d62411740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d541a9530 con 0x55d76ec474d0 2024-04-25T16:04:19.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.313+0000 7f8d59a2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f0222a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.313+0000 7f8d59a2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f0222a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45932/0 (socket says 172.21.15.31:45932) 2024-04-25T16:04:19.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.313+0000 7f8d59a2f640 1 -- 172.21.15.31:0/2319293985 learned_addr learned my addr 172.21.15.31:0/2319293985 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.317+0000 7f8d5922e640 1 --2- 172.21.15.31:0/2319293985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec3f2f0 0x55d76f021d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.317+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/2319293985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f02ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.320+0000 7f8d5922e640 1 -- 172.21.15.31:0/2319293985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 msgr2=0x55d76f02ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.320+0000 7f8d5922e640 1 --2- 172.21.15.31:0/2319293985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f02ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.320+0000 7f8d5922e640 1 -- 172.21.15.31:0/2319293985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 msgr2=0x55d76f0222a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.320+0000 7f8d5922e640 1 --2- 172.21.15.31:0/2319293985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f0222a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.320+0000 7f8d5922e640 1 -- 172.21.15.31:0/2319293985 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d76f02b160 con 0x55d76ec3f2f0 2024-04-25T16:04:19.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.320+0000 7f8d5922e640 1 --2- 172.21.15.31:0/2319293985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec3f2f0 0x55d76f021d60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000b460 tx=0x7f8d50000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5e0e40ec7b8b327 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.321+0000 7f8d3effd640 1 -- 172.21.15.31:0/2319293985 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d50015040 con 0x55d76ec3f2f0 2024-04-25T16:04:19.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.322+0000 7f8d3effd640 1 -- 172.21.15.31:0/2319293985 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d5002a070 con 0x55d76ec3f2f0 2024-04-25T16:04:19.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.322+0000 7f8d3effd640 1 -- 172.21.15.31:0/2319293985 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d50004680 con 0x55d76ec3f2f0 2024-04-25T16:04:19.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.323+0000 7f8d62411740 1 -- 172.21.15.31:0/2319293985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec3f2f0 msgr2=0x55d76f021d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.323+0000 7f8d62411740 1 --2- 172.21.15.31:0/2319293985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec3f2f0 0x55d76f021d60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000b460 tx=0x7f8d50000bc0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.327+0000 7f8d62411740 1 -- 172.21.15.31:0/2319293985 shutdown_connections 2024-04-25T16:04:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.327+0000 7f8d62411740 1 --2- 172.21.15.31:0/2319293985 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec681b0 0x55d76f0222a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.327+0000 7f8d62411740 1 --2- 172.21.15.31:0/2319293985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f02ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.327+0000 7f8d62411740 1 --2- 172.21.15.31:0/2319293985 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec3f2f0 0x55d76f021d60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.327+0000 7f8d62411740 1 -- 172.21.15.31:0/2319293985 >> 172.21.15.31:0/2319293985 conn(0x55d76ec5c300 msgr2=0x55d76f00ccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.327+0000 7f8d62411740 1 -- 172.21.15.31:0/2319293985 shutdown_connections 2024-04-25T16:04:19.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.327+0000 7f8d62411740 1 -- 172.21.15.31:0/2319293985 wait complete. 2024-04-25T16:04:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.327+0000 7f8d62411740 1 -- start start 2024-04-25T16:04:19.329 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.328+0000 7f8d62411740 1 Processor -- start 2024-04-25T16:04:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d62411740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec3f2f0 0x55d76f026160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d62411740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f0266a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d62411740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec681b0 0x55d76f0295e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d62411740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8d541aa2d0 con 0x55d76ec3f2f0 2024-04-25T16:04:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d62411740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d541a9c00 con 0x55d76ec681b0 2024-04-25T16:04:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d62411740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d54081ea0 con 0x55d76ec474d0 2024-04-25T16:04:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d62411740 10 client.0 _ll_register_callbacks cb 0x7f8d44009230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:19.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d58a2d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f0266a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d58a2d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f0266a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45964/0 (socket says 172.21.15.31:45964) 2024-04-25T16:04:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.329+0000 7f8d58a2d640 1 -- 172.21.15.31:0/477714293 learned_addr learned my addr 172.21.15.31:0/477714293 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.331 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.330+0000 7f8d59a2f640 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec681b0 0x55d76f0295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.332 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.331+0000 7f8d5922e640 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec3f2f0 0x55d76f026160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.335+0000 7f8d58a2d640 1 -- 172.21.15.31:0/477714293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec681b0 msgr2=0x55d76f0295e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.335+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec681b0 0x55d76f0295e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.335+0000 7f8d58a2d640 1 -- 172.21.15.31:0/477714293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec3f2f0 msgr2=0x55d76f026160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.335+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec3f2f0 0x55d76f026160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.335+0000 7f8d58a2d640 1 -- 172.21.15.31:0/477714293 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d76f029ba0 con 0x55d76ec474d0 2024-04-25T16:04:19.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.336+0000 7f8d58a2d640 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f0266a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8d4402a9d0 tx=0x7f8d44031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc52a461fa5dd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.341+0000 7f8d3cff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:04:19.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.341+0000 7f8d3cff9640 1 -- 172.21.15.31:0/477714293 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d440d6030 con 0x55d76ec474d0 2024-04-25T16:04:19.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.341+0000 7f8d62411740 1 -- 172.21.15.31:0/477714293 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d5002a070 con 0x55d76ec474d0 2024-04-25T16:04:19.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.341+0000 7f8d3cff9640 1 -- 172.21.15.31:0/477714293 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8d44031e00 con 0x55d76ec474d0 2024-04-25T16:04:19.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.348+0000 7f8d3cff9640 1 -- 172.21.15.31:0/477714293 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8d44049070 con 0x55d76ec474d0 2024-04-25T16:04:19.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.349+0000 7f8d62411740 10 client.25888 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:19.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.349+0000 7f8d62411740 1 -- 172.21.15.31:0/477714293 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d76f02ae30 con 0x55d76ec474d0 2024-04-25T16:04:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.359+0000 7f8d3cff9640 1 -- 172.21.15.31:0/477714293 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d5002a070 con 0x55d76ec474d0 2024-04-25T16:04:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.359+0000 7f8d3cff9640 1 -- 172.21.15.31:0/477714293 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f8d440e8bc0 con 0x55d76ec474d0 2024-04-25T16:04:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.359+0000 7f8d62411740 10 client.25888 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.359+0000 7f8d62411740 10 client.25888 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.359+0000 7f8d62411740 20 client.25888 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:19.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.359+0000 7f8d62411740 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d76ebfc1c0 0x55d76f0bdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.359+0000 7f8d62411740 4 client.25888 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:04:19.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.359+0000 7f8d62411740 1 -- 172.21.15.31:0/477714293 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d76ef75ee0 con 0x55d76ebfc1c0 2024-04-25T16:04:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.362+0000 7f8d5922e640 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d76ebfc1c0 0x55d76f0bdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.364+0000 7f8d5922e640 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d76ebfc1c0 0x55d76f0bdc80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e65975885d0cc62 server_cookie=96158175e379339a in_seq=0 out_seq=0 2024-04-25T16:04:19.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:19.364+0000 7f8d3cff9640 10 client.25888 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:19.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.505+0000 7f2fe9035640 1 -- 172.21.15.31:0/984146265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe410b180 msgr2=0x7f2fe410b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.505+0000 7f2fe9035640 1 --2- 172.21.15.31:0/984146265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe410b180 0x7f2fe410b590 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8009860 tx=0x7f2fd802a380 comp rx=0 tx=0).stop 2024-04-25T16:04:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.513+0000 7f2fe9035640 1 -- 172.21.15.31:0/984146265 shutdown_connections 2024-04-25T16:04:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.513+0000 7f2fe9035640 1 --2- 172.21.15.31:0/984146265 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe410fe80 0x7f2fe41122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.513+0000 7f2fe9035640 1 --2- 172.21.15.31:0/984146265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe410b180 0x7f2fe410b590 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.513+0000 7f2fe9035640 1 --2- 172.21.15.31:0/984146265 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fe4109f40 0x7f2fe410ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.513+0000 7f2fe9035640 1 -- 172.21.15.31:0/984146265 >> 172.21.15.31:0/984146265 conn(0x7f2fe4078b30 msgr2=0x7f2fe41079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.520+0000 7f2fe9035640 1 -- 172.21.15.31:0/984146265 shutdown_connections 2024-04-25T16:04:19.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.523+0000 7f2fe9035640 1 -- 172.21.15.31:0/984146265 wait complete. 2024-04-25T16:04:19.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.524+0000 7f2fe9035640 1 Processor -- start 2024-04-25T16:04:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.528+0000 7f2fe9035640 1 -- start start 2024-04-25T16:04:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.529+0000 7f2fe9035640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fe4109f40 0x7f2fe4083030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.529+0000 7f2fe9035640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe410b180 0x7f2fe4083570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.529+0000 7f2fe9035640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe410fe80 0x7f2fe40816f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.529+0000 7f2fe9035640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe407bc50 con 0x7f2fe410fe80 2024-04-25T16:04:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.529+0000 7f2fe9035640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe407bad0 con 0x7f2fe4109f40 2024-04-25T16:04:19.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.529+0000 7f2fe9035640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe407bdd0 con 0x7f2fe410b180 2024-04-25T16:04:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.531+0000 7f2fe8834640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe410fe80 0x7f2fe40816f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.531+0000 7f2fe8834640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe410fe80 0x7f2fe40816f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45948/0 (socket says 172.21.15.31:45948) 2024-04-25T16:04:19.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.531+0000 7f2fe8834640 1 -- 172.21.15.31:0/683509093 learned_addr learned my addr 172.21.15.31:0/683509093 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.533+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fe4109f40 0x7f2fe4083030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.533+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe410b180 0x7f2fe4083570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.533+0000 7f2fe8834640 1 -- 172.21.15.31:0/683509093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe410b180 msgr2=0x7f2fe4083570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.533+0000 7f2fe8834640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe410b180 0x7f2fe4083570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.534+0000 7f2fe8834640 1 -- 172.21.15.31:0/683509093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fe4109f40 msgr2=0x7f2fe4083030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.534+0000 7f2fe8834640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fe4109f40 0x7f2fe4083030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.534+0000 7f2fe8834640 1 -- 172.21.15.31:0/683509093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe4081f10 con 0x7f2fe410fe80 2024-04-25T16:04:19.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.535+0000 7f2fe8834640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe410fe80 0x7f2fe40816f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4009fd0 tx=0x7f2fd400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.537+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fe4109f40 0x7f2fe4083030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:04:19.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.537+0000 7f2fe17fa640 1 -- 172.21.15.31:0/683509093 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4011830 con 0x7f2fe410fe80 2024-04-25T16:04:19.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.537+0000 7f2fe9035640 1 -- 172.21.15.31:0/683509093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe4082200 con 0x7f2fe410fe80 2024-04-25T16:04:19.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.537+0000 7f2fe9035640 1 -- 172.21.15.31:0/683509093 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe41b1190 con 0x7f2fe410fe80 2024-04-25T16:04:19.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.539+0000 7f2fe17fa640 1 -- 172.21.15.31:0/683509093 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd4016070 con 0x7f2fe410fe80 2024-04-25T16:04:19.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.539+0000 7f2fe17fa640 1 -- 172.21.15.31:0/683509093 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4010e20 con 0x7f2fe410fe80 2024-04-25T16:04:19.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.543+0000 7f2ff1221740 1 Processor -- start 2024-04-25T16:04:19.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.545+0000 7f2fe17fa640 1 -- 172.21.15.31:0/683509093 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2fd401b4a0 con 0x7f2fe410fe80 2024-04-25T16:04:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.546+0000 7f2ff1221740 1 -- start start 2024-04-25T16:04:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.546+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561673755390 0x561673af79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.546+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56167372c4d0 0x561673af7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.546+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616737346b0 0x561673af8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.546+0000 7f2ff1221740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561673a5bd80 con 0x5616737346b0 2024-04-25T16:04:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.546+0000 7f2ff1221740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561673a15a20 con 0x561673755390 2024-04-25T16:04:19.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.546+0000 7f2ff1221740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561673a57d40 con 0x56167372c4d0 2024-04-25T16:04:19.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.547+0000 7f2fe3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56167372c4d0 0x561673af7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.547+0000 7f2fe3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56167372c4d0 0x561673af7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45982/0 (socket says 172.21.15.31:45982) 2024-04-25T16:04:19.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.547+0000 7f2fe3fff640 1 -- 172.21.15.31:0/2584386187 learned_addr learned my addr 172.21.15.31:0/2584386187 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.548+0000 7f2fe17fa640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fcc0afcb0 0x7f2fcc0b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.548+0000 7f2fe3fff640 1 -- 172.21.15.31:0/2584386187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561673755390 msgr2=0x561673af79c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:04:19.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.548+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/2584386187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561673755390 0x561673af79c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.548+0000 7f2fe3fff640 1 -- 172.21.15.31:0/2584386187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616737346b0 msgr2=0x561673af8bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:04:19.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.548+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/2584386187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616737346b0 0x561673af8bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.548+0000 7f2fe3fff640 1 -- 172.21.15.31:0/2584386187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561673a20c30 con 0x56167372c4d0 2024-04-25T16:04:19.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.548+0000 7f2fe17fa640 1 -- 172.21.15.31:0/683509093 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2fd40de1c0 con 0x7f2fe410fe80 2024-04-25T16:04:19.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.548+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/2584386187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56167372c4d0 0x561673af7f40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009ed0 tx=0x7f2fdc00f7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e98f630363212b39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.550+0000 7f2fe8834640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fcc0afcb0 0x7f2fcc0b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.550+0000 7f2fe9035640 1 -- 172.21.15.31:0/2584386187 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc01c070 con 0x56167372c4d0 2024-04-25T16:04:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.551+0000 7f2fe9035640 1 -- 172.21.15.31:0/2584386187 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fdc00db20 con 0x56167372c4d0 2024-04-25T16:04:19.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.551+0000 7f2fe9035640 1 -- 172.21.15.31:0/2584386187 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc017ca0 con 0x56167372c4d0 2024-04-25T16:04:19.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.553+0000 7f2ff1221740 1 -- 172.21.15.31:0/2584386187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56167372c4d0 msgr2=0x561673af7f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.553+0000 7f2ff1221740 1 --2- 172.21.15.31:0/2584386187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56167372c4d0 0x561673af7f40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009ed0 tx=0x7f2fdc00f7d0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.556+0000 7f2fe8834640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fcc0afcb0 0x7f2fcc0b21c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4006fd0 tx=0x7f2fd400bf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.557+0000 7f2ff1221740 1 -- 172.21.15.31:0/2584386187 shutdown_connections 2024-04-25T16:04:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.557+0000 7f2ff1221740 1 --2- 172.21.15.31:0/2584386187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616737346b0 0x561673af8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.557+0000 7f2ff1221740 1 --2- 172.21.15.31:0/2584386187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56167372c4d0 0x561673af7f40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.557+0000 7f2ff1221740 1 --2- 172.21.15.31:0/2584386187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561673755390 0x561673af79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.557+0000 7f2ff1221740 1 -- 172.21.15.31:0/2584386187 >> 172.21.15.31:0/2584386187 conn(0x5616737494e0 msgr2=0x561673af34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.570+0000 7f2ff1221740 1 -- 172.21.15.31:0/2584386187 shutdown_connections 2024-04-25T16:04:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.571+0000 7f2ff1221740 1 -- 172.21.15.31:0/2584386187 wait complete. 2024-04-25T16:04:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.572+0000 7f2ff1221740 1 -- start start 2024-04-25T16:04:19.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.573+0000 7f2ff1221740 1 Processor -- start 2024-04-25T16:04:19.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.574+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.574+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616737346b0 0x561673b12cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.574+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561673755390 0x561673b19c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.574+0000 7f2ff1221740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561673af96e0 con 0x561673755390 2024-04-25T16:04:19.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.574+0000 7f2ff1221740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5616739e2dc0 con 0x56167372c4d0 2024-04-25T16:04:19.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.574+0000 7f2ff1221740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561673af9860 con 0x5616737346b0 2024-04-25T16:04:19.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.574+0000 7f2ff1221740 10 client.0 _ll_register_callbacks cb 0x5616739fe600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:19.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37278/0 (socket says 172.21.15.31:37278) 2024-04-25T16:04:19.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe37fe640 1 -- 172.21.15.31:0/3038352727 learned_addr learned my addr 172.21.15.31:0/3038352727 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616737346b0 0x561673b12cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe3fff640 1 -- 172.21.15.31:0/3038352727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 msgr2=0x561673b12770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe3fff640 1 -- 172.21.15.31:0/3038352727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561673755390 msgr2=0x561673b19c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561673755390 0x561673b19c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe3fff640 1 -- 172.21.15.31:0/3038352727 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56167376d3f0 con 0x5616737346b0 2024-04-25T16:04:19.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.575+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616737346b0 0x561673b12cb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009ea0 tx=0x7f2fdc0176b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3b4e89945553285 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.576+0000 7f2fc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:04:19.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.576+0000 7f2fc37fe640 1 -- 172.21.15.31:0/3038352727 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc025060 con 0x5616737346b0 2024-04-25T16:04:19.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.576+0000 7f2fc37fe640 1 -- 172.21.15.31:0/3038352727 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2fdc00d7b0 con 0x5616737346b0 2024-04-25T16:04:19.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.576+0000 7f2ff1221740 1 -- 172.21.15.31:0/3038352727 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fdc00db20 con 0x5616737346b0 2024-04-25T16:04:19.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.576+0000 7f2fc37fe640 1 -- 172.21.15.31:0/3038352727 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2fdc01c040 con 0x5616737346b0 2024-04-25T16:04:19.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.577+0000 7f2ff1221740 10 client.25897 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:19.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.577+0000 7f2ff1221740 1 -- 172.21.15.31:0/3038352727 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5616738377d0 con 0x5616737346b0 2024-04-25T16:04:19.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.578+0000 7f2fc37fe640 1 -- 172.21.15.31:0/3038352727 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fdc00db20 con 0x5616737346b0 2024-04-25T16:04:19.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.578+0000 7f2fc37fe640 1 -- 172.21.15.31:0/3038352727 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f2fdc01f070 con 0x5616737346b0 2024-04-25T16:04:19.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.578+0000 7f2ff1221740 10 client.25897 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:19.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.578+0000 7f2ff1221740 10 client.25897 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:04:19.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.578+0000 7f2ff1221740 20 client.25897 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:19.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.578+0000 7f2ff1221740 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561673713f60 0x561673b20780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.578+0000 7f2ff1221740 4 client.25897 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:19.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.578+0000 7f2ff1221740 1 -- 172.21.15.31:0/3038352727 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561673a69990 con 0x561673713f60 2024-04-25T16:04:19.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.582+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561673713f60 0x561673b20780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.587+0000 7f2fc37fe640 10 client.25897 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:04:19.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.587+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561673713f60 0x561673b20780 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b66e35f021782fb1 server_cookie=2c0e2587f94fb8f in_seq=0 out_seq=0 2024-04-25T16:04:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.592+0000 7f2fc37fe640 1 -- 172.21.15.31:0/3038352727 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561673a69990 con 0x561673713f60 2024-04-25T16:04:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.592+0000 7f2fc37fe640 10 client.25897 handle_command_reply: tid=0 2024-04-25T16:04:19.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.592+0000 7f2ff1221740 1 client.25897 shutdown 2024-04-25T16:04:19.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.604+0000 7f2ff1221740 1 -- 172.21.15.31:0/3038352727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616737346b0 msgr2=0x561673b12cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.604+0000 7f2ff1221740 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616737346b0 0x561673b12cb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc009ea0 tx=0x7f2fdc0176b0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.606+0000 7f2ff1221740 1 -- 172.21.15.31:0/3038352727 shutdown_connections 2024-04-25T16:04:19.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.606+0000 7f2ff1221740 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561673755390 0x561673b19c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.606+0000 7f2ff1221740 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616737346b0 0x561673b12cb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.606+0000 7f2ff1221740 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561673713f60 0x561673b20780 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.606+0000 7f2ff1221740 1 --2- 172.21.15.31:0/3038352727 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.606+0000 7f2ff1221740 1 -- 172.21.15.31:0/3038352727 >> 172.21.15.31:0/3038352727 conn(0x5616737494e0 msgr2=0x561673af63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.607+0000 7f2fc37fe640 0 client.25897 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:04:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.607+0000 7f2ff1221740 1 -- 172.21.15.31:0/3038352727 shutdown_connections 2024-04-25T16:04:19.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.607+0000 7f2ff1221740 1 -- 172.21.15.31:0/3038352727 wait complete. 2024-04-25T16:04:19.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.607+0000 7f2ff1221740 20 client.25897 trim_cache size 0 max 16384 2024-04-25T16:04:19.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.642+0000 7f2ff1221740 1 Processor -- start 2024-04-25T16:04:19.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.643+0000 7f2ff1221740 1 -- start start 2024-04-25T16:04:19.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.643+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.644+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616737346b0 0x561673b12cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.644+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 0x561673b19b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.644+0000 7f2fe3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616737346b0 0x561673b12cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.644+0000 7f2fe3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616737346b0 0x561673b12cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37286/0 (socket says 172.21.15.31:37286) 2024-04-25T16:04:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.644+0000 7f2fe3fff640 1 -- 172.21.15.31:0/343015346 learned_addr learned my addr 172.21.15.31:0/343015346 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.644+0000 7f2ff1221740 1 -- 172.21.15.31:0/343015346 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd000e550 con 0x56167372c4d0 2024-04-25T16:04:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.645+0000 7f2ff1221740 1 -- 172.21.15.31:0/343015346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd00100c0 con 0x5616737346b0 2024-04-25T16:04:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.645+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 0x561673b19b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.645+0000 7f2ff1221740 1 -- 172.21.15.31:0/343015346 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd000ec20 con 0x561673755390 2024-04-25T16:04:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.645+0000 7f2fe3fff640 1 -- 172.21.15.31:0/343015346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 msgr2=0x561673b19b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.645+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 0x561673b19b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.646+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.645+0000 7f2fe3fff640 1 -- 172.21.15.31:0/343015346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56167372c4d0 msgr2=0x561673b12770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.646+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.646+0000 7f2fe3fff640 1 -- 172.21.15.31:0/343015346 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561673affb90 con 0x5616737346b0 2024-04-25T16:04:19.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.647+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:04:19.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.648+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616737346b0 0x561673b12cb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc002230 tx=0x7f2fdc009bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab7e9e075d7a41c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.651+0000 7f2fc9ffb640 1 -- 172.21.15.31:0/343015346 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fdc030020 con 0x5616737346b0 2024-04-25T16:04:19.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.652+0000 7f2fc9ffb640 1 -- 172.21.15.31:0/343015346 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fdc005830 con 0x5616737346b0 2024-04-25T16:04:19.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.654+0000 7f2ff1221740 1 -- 172.21.15.31:0/343015346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616737346b0 msgr2=0x561673b12cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.654+0000 7f2ff1221740 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616737346b0 0x561673b12cb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc002230 tx=0x7f2fdc009bc0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.654+0000 7f2ff1221740 1 -- 172.21.15.31:0/343015346 shutdown_connections 2024-04-25T16:04:19.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.654+0000 7f2ff1221740 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 0x561673b19b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.654+0000 7f2ff1221740 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616737346b0 0x561673b12cb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.654+0000 7f2ff1221740 1 --2- 172.21.15.31:0/343015346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56167372c4d0 0x561673b12770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.654+0000 7f2ff1221740 1 -- 172.21.15.31:0/343015346 >> 172.21.15.31:0/343015346 conn(0x5616737494e0 msgr2=0x561673af7940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.665+0000 7f2ff1221740 1 -- 172.21.15.31:0/343015346 shutdown_connections 2024-04-25T16:04:19.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.665+0000 7f2ff1221740 1 -- 172.21.15.31:0/343015346 wait complete. 2024-04-25T16:04:19.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.667+0000 7f2ff1221740 1 -- start start 2024-04-25T16:04:19.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.669+0000 7f2ff1221740 1 Processor -- start 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.669+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b15b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.669+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 0x561673b16040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.669+0000 7f2ff1221740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616736e93a0 0x561673b18f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.669+0000 7f2ff1221740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd000f9c0 con 0x5616736e93a0 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.669+0000 7f2ff1221740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd000f2f0 con 0x56167372c4d0 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.669+0000 7f2ff1221740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd000c530 con 0x561673755390 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2ff1221740 10 client.0 _ll_register_callbacks cb 0x7f2fdc01a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b15b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b15b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37292/0 (socket says 172.21.15.31:37292) 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe37fe640 1 -- 172.21.15.31:0/545770500 learned_addr learned my addr 172.21.15.31:0/545770500 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:19.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 0x561673b16040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe37fe640 1 -- 172.21.15.31:0/545770500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 msgr2=0x561673b16040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 0x561673b16040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe37fe640 1 -- 172.21.15.31:0/545770500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616736e93a0 msgr2=0x561673b18f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:19.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616736e93a0 0x561673b18f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe37fe640 1 -- 172.21.15.31:0/545770500 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561673b1a070 con 0x56167372c4d0 2024-04-25T16:04:19.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.670+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 0x561673b16040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:04:19.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.676+0000 7f2fe37fe640 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b15b00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8035040 tx=0x7f2fd8030f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbb5cf614bb0ab03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:19.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.677+0000 7f2fc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:19.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.682+0000 7f2fc37fe640 1 -- 172.21.15.31:0/545770500 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8037070 con 0x56167372c4d0 2024-04-25T16:04:19.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.682+0000 7f2fc37fe640 1 -- 172.21.15.31:0/545770500 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2fd8004230 con 0x56167372c4d0 2024-04-25T16:04:19.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.682+0000 7f2ff1221740 1 -- 172.21.15.31:0/545770500 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561673a63800 con 0x56167372c4d0 2024-04-25T16:04:19.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.687+0000 7f2fc37fe640 1 -- 172.21.15.31:0/545770500 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2fd8004430 con 0x56167372c4d0 2024-04-25T16:04:19.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.687+0000 7f2ff1221740 10 client.25937 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:19.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.687+0000 7f2ff1221740 1 -- 172.21.15.31:0/545770500 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561673b870a0 con 0x56167372c4d0 2024-04-25T16:04:19.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.692+0000 7f2fc37fe640 1 -- 172.21.15.31:0/545770500 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561673a63800 con 0x56167372c4d0 2024-04-25T16:04:19.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.692+0000 7f2fc37fe640 1 -- 172.21.15.31:0/545770500 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f2fd803d050 con 0x56167372c4d0 2024-04-25T16:04:19.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.692+0000 7f2ff1221740 10 client.25937 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:19.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.692+0000 7f2ff1221740 10 client.25937 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:04:19.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.692+0000 7f2ff1221740 20 client.25937 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:19.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.692+0000 7f2ff1221740 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561673b879d0 0x561673b89df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:19.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.692+0000 7f2ff1221740 4 client.25937 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:04:19.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.692+0000 7f2ff1221740 1 -- 172.21.15.31:0/545770500 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5616739de610 con 0x561673b879d0 2024-04-25T16:04:19.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.699+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561673b879d0 0x561673b89df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:19.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.711+0000 7f2fe3fff640 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561673b879d0 0x561673b89df0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2953259c7e48a26d server_cookie=5d4746ac6fc619a4 in_seq=0 out_seq=0 2024-04-25T16:04:19.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.712+0000 7f2fc37fe640 10 client.25937 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:04:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.713+0000 7f2fc37fe640 1 -- 172.21.15.31:0/545770500 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5616739de610 con 0x561673b879d0 2024-04-25T16:04:19.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.713+0000 7f2fc37fe640 10 client.25937 handle_command_reply: tid=0 2024-04-25T16:04:19.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.714+0000 7f2ff1221740 1 client.25937 shutdown 2024-04-25T16:04:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.722+0000 7f2ff1221740 1 -- 172.21.15.31:0/545770500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 msgr2=0x561673b15b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.722+0000 7f2ff1221740 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b15b00 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8035040 tx=0x7f2fd8030f90 comp rx=0 tx=0).stop 2024-04-25T16:04:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.722+0000 7f2ff1221740 1 -- 172.21.15.31:0/545770500 shutdown_connections 2024-04-25T16:04:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.722+0000 7f2ff1221740 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616736e93a0 0x561673b18f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.722+0000 7f2ff1221740 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561673755390 0x561673b16040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.722+0000 7f2ff1221740 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x561673b879d0 0x561673b89df0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.722+0000 7f2ff1221740 1 --2- 172.21.15.31:0/545770500 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56167372c4d0 0x561673b15b00 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.722+0000 7f2ff1221740 1 -- 172.21.15.31:0/545770500 >> 172.21.15.31:0/545770500 conn(0x5616737494e0 msgr2=0x561673afc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.723+0000 7f2fc37fe640 0 client.25937 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:04:19.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.724+0000 7f2ff1221740 1 -- 172.21.15.31:0/545770500 shutdown_connections 2024-04-25T16:04:19.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.724+0000 7f2ff1221740 1 -- 172.21.15.31:0/545770500 wait complete. 2024-04-25T16:04:19.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.724+0000 7f2ff1221740 20 client.25937 trim_cache size 0 max 16384 2024-04-25T16:04:19.726 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:04:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.738+0000 7f2fcaffd640 1 -- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fcc0afcb0 msgr2=0x7f2fcc0b21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.738+0000 7f2fcaffd640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fcc0afcb0 0x7f2fcc0b21c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4006fd0 tx=0x7f2fd400bf90 comp rx=0 tx=0).stop 2024-04-25T16:04:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.738+0000 7f2fcaffd640 1 -- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe410fe80 msgr2=0x7f2fe40816f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:19.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.738+0000 7f2fcaffd640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe410fe80 0x7f2fe40816f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4009fd0 tx=0x7f2fd400b8e0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.747+0000 7f2fcaffd640 1 -- 172.21.15.31:0/683509093 shutdown_connections 2024-04-25T16:04:19.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.747+0000 7f2fcaffd640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fcc0afcb0 0x7f2fcc0b21c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.747+0000 7f2fcaffd640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fe410fe80 0x7f2fe40816f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.747+0000 7f2fcaffd640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe410b180 0x7f2fe4083570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.747+0000 7f2fcaffd640 1 --2- 172.21.15.31:0/683509093 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fe4109f40 0x7f2fe4083030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:19.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.747+0000 7f2fcaffd640 1 -- 172.21.15.31:0/683509093 >> 172.21.15.31:0/683509093 conn(0x7f2fe4078b30 msgr2=0x7f2fe410be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:19.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.760+0000 7f2fcaffd640 1 -- 172.21.15.31:0/683509093 shutdown_connections 2024-04-25T16:04:19.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:19.770+0000 7f2fcaffd640 1 -- 172.21.15.31:0/683509093 wait complete. 2024-04-25T16:04:19.878 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:04:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.318+0000 7fd552563640 1 -- 172.21.15.31:0/2298954579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 msgr2=0x7fd54c110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:20.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.318+0000 7fd552563640 1 --2- 172.21.15.31:0/2298954579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 0x7fd54c110300 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd53c009900 tx=0x7fd53c02a440 comp rx=0 tx=0).stop 2024-04-25T16:04:20.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.321+0000 7fd552563640 1 -- 172.21.15.31:0/2298954579 shutdown_connections 2024-04-25T16:04:20.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.321+0000 7fd552563640 1 --2- 172.21.15.31:0/2298954579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 0x7fd54c110300 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.321+0000 7fd552563640 1 --2- 172.21.15.31:0/2298954579 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd54c10b1d0 0x7fd54c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.321+0000 7fd552563640 1 --2- 172.21.15.31:0/2298954579 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c10a350 0x7fd54c10ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.321+0000 7fd552563640 1 -- 172.21.15.31:0/2298954579 >> 172.21.15.31:0/2298954579 conn(0x7fd54c070400 msgr2=0x7fd54c072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:20.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.326+0000 7fd552563640 1 -- 172.21.15.31:0/2298954579 shutdown_connections 2024-04-25T16:04:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.333+0000 7fd552563640 1 -- 172.21.15.31:0/2298954579 wait complete. 2024-04-25T16:04:20.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.335+0000 7fd552563640 1 Processor -- start 2024-04-25T16:04:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.335+0000 7fd552563640 1 -- start start 2024-04-25T16:04:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.336+0000 7fd552563640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c10a350 0x7fd54c1170f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.336+0000 7fd552563640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd54c10b1d0 0x7fd54c117630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.336+0000 7fd552563640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 0x7fd54c118590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.336+0000 7fd552563640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd54c11e2e0 con 0x7fd54c10a350 2024-04-25T16:04:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.336+0000 7fd552563640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd54c11e160 con 0x7fd54c10fef0 2024-04-25T16:04:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.336+0000 7fd552563640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd54c11e460 con 0x7fd54c10b1d0 2024-04-25T16:04:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.337+0000 7fd551d62640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 0x7fd54c118590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.337+0000 7fd551d62640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 0x7fd54c118590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37304/0 (socket says 172.21.15.31:37304) 2024-04-25T16:04:20.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.337+0000 7fd551d62640 1 -- 172.21.15.31:0/1763059910 learned_addr learned my addr 172.21.15.31:0/1763059910 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:20.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.338+0000 7fd550d60640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd54c10b1d0 0x7fd54c117630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.338+0000 7fd551d62640 1 -- 172.21.15.31:0/1763059910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd54c10b1d0 msgr2=0x7fd54c117630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.338+0000 7fd551d62640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd54c10b1d0 0x7fd54c117630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.338+0000 7fd551d62640 1 -- 172.21.15.31:0/1763059910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c10a350 msgr2=0x7fd54c1170f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.338+0000 7fd551d62640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c10a350 0x7fd54c1170f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.338+0000 7fd551d62640 1 -- 172.21.15.31:0/1763059910 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd54c118c00 con 0x7fd54c10fef0 2024-04-25T16:04:20.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.339+0000 7fd550d60640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd54c10b1d0 0x7fd54c117630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:04:20.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.340+0000 7fd551d62640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 0x7fd54c118590 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd53c02aa30 tx=0x7fd53c003d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.343+0000 7fd5427fc640 1 -- 172.21.15.31:0/1763059910 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c03f070 con 0x7fd54c10fef0 2024-04-25T16:04:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.343+0000 7fd5427fc640 1 -- 172.21.15.31:0/1763059910 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd53c003f90 con 0x7fd54c10fef0 2024-04-25T16:04:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.343+0000 7fd5427fc640 1 -- 172.21.15.31:0/1763059910 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd53c035030 con 0x7fd54c10fef0 2024-04-25T16:04:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.343+0000 7fd552563640 1 -- 172.21.15.31:0/1763059910 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd54c118e90 con 0x7fd54c10fef0 2024-04-25T16:04:20.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.343+0000 7fd552563640 1 -- 172.21.15.31:0/1763059910 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd54c1193b0 con 0x7fd54c10fef0 2024-04-25T16:04:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.350+0000 7fd5427fc640 1 -- 172.21.15.31:0/1763059910 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd53c042070 con 0x7fd54c10fef0 2024-04-25T16:04:20.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.351+0000 7fd5427fc640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5380afc30 0x7fd5380b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.352+0000 7fd5427fc640 1 -- 172.21.15.31:0/1763059910 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd53c100680 con 0x7fd54c10fef0 2024-04-25T16:04:20.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.360+0000 7fd551561640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5380afc30 0x7fd5380b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.370+0000 7fd55a67a740 1 Processor -- start 2024-04-25T16:04:20.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.370+0000 7fd55a67a740 1 -- start start 2024-04-25T16:04:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.371+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e231a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.371+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e231f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.371+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e232c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.371+0000 7fd55a67a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56011e195d80 con 0x56011de664d0 2024-04-25T16:04:20.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.372+0000 7fd55a67a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56011e14fa20 con 0x56011de8f390 2024-04-25T16:04:20.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.372+0000 7fd55a67a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56011e191d40 con 0x56011de6e6b0 2024-04-25T16:04:20.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.373+0000 7fd550d60640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e231f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.373+0000 7fd550d60640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e231f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46004/0 (socket says 172.21.15.31:46004) 2024-04-25T16:04:20.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.373+0000 7fd550d60640 1 -- 172.21.15.31:0/3123749352 learned_addr learned my addr 172.21.15.31:0/3123749352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:20.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.374+0000 7fd551561640 1 --2- 172.21.15.31:0/3123749352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e231a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.375+0000 7fd550d60640 1 -- 172.21.15.31:0/3123749352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 msgr2=0x56011e232c10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:04:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.375+0000 7fd550d60640 1 --2- 172.21.15.31:0/3123749352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e232c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.375+0000 7fd550d60640 1 -- 172.21.15.31:0/3123749352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 msgr2=0x56011e231a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.375+0000 7fd550d60640 1 --2- 172.21.15.31:0/3123749352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e231a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.375+0000 7fd550d60640 1 -- 172.21.15.31:0/3123749352 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56011e15ac30 con 0x56011de664d0 2024-04-25T16:04:20.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.376+0000 7fd551561640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5380afc30 0x7fd5380b2140 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd548009710 tx=0x7fd548006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:20.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.380+0000 7fd550d60640 1 --2- 172.21.15.31:0/3123749352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e231f80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd544007fa0 tx=0x7fd54400c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d5dc6b179ccbe0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:20.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.382+0000 7fd5377fe640 1 -- 172.21.15.31:0/3123749352 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544014070 con 0x56011de664d0 2024-04-25T16:04:20.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.383+0000 7fd5377fe640 1 -- 172.21.15.31:0/3123749352 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd54400fd00 con 0x56011de664d0 2024-04-25T16:04:20.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.383+0000 7fd5377fe640 1 -- 172.21.15.31:0/3123749352 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd54400ecb0 con 0x56011de664d0 2024-04-25T16:04:20.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.384+0000 7fd55a67a740 1 -- 172.21.15.31:0/3123749352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 msgr2=0x56011e231f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:20.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.384+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3123749352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e231f80 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fd544007fa0 tx=0x7fd54400c750 comp rx=0 tx=0).stop 2024-04-25T16:04:20.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.386+0000 7fd55a67a740 1 -- 172.21.15.31:0/3123749352 shutdown_connections 2024-04-25T16:04:20.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.386+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3123749352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e232c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.386+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3123749352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e231f80 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.386+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3123749352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e231a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.386+0000 7fd55a67a740 1 -- 172.21.15.31:0/3123749352 >> 172.21.15.31:0/3123749352 conn(0x56011de834e0 msgr2=0x56011e22d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:20.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.388+0000 7fd55a67a740 1 -- 172.21.15.31:0/3123749352 shutdown_connections 2024-04-25T16:04:20.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.388+0000 7fd55a67a740 1 -- 172.21.15.31:0/3123749352 wait complete. 2024-04-25T16:04:20.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.389+0000 7fd55a67a740 1 -- start start 2024-04-25T16:04:20.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.389+0000 7fd55a67a740 1 Processor -- start 2024-04-25T16:04:20.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.390+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e23b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.390+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.390+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e242c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.390+0000 7fd55a67a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56011e233720 con 0x56011de664d0 2024-04-25T16:04:20.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.390+0000 7fd55a67a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56011e11cdc0 con 0x56011de8f390 2024-04-25T16:04:20.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.390+0000 7fd55a67a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56011e2338a0 con 0x56011de6e6b0 2024-04-25T16:04:20.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.391+0000 7fd55a67a740 10 client.0 _ll_register_callbacks cb 0x56011e138600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:20.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.395+0000 7fd551d62640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e242c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.395+0000 7fd551d62640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e242c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37322/0 (socket says 172.21.15.31:37322) 2024-04-25T16:04:20.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.395+0000 7fd551d62640 1 -- 172.21.15.31:0/2187068300 learned_addr learned my addr 172.21.15.31:0/2187068300 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:20.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.395+0000 7fd550d60640 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.396+0000 7fd550d60640 1 -- 172.21.15.31:0/2187068300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 msgr2=0x56011e242c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:20.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.396+0000 7fd550d60640 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e242c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.396+0000 7fd550d60640 1 -- 172.21.15.31:0/2187068300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 msgr2=0x56011e23b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:20.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.396+0000 7fd550d60640 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e23b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.396+0000 7fd550d60640 1 -- 172.21.15.31:0/2187068300 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56011dea73f0 con 0x56011de6e6b0 2024-04-25T16:04:20.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.397+0000 7fd550d60640 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x56011e22dfa0 tx=0x7fd54400e5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c359441725aad9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:20.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.401+0000 7fd534ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:04:20.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.401+0000 7fd534ff9640 1 -- 172.21.15.31:0/2187068300 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544014040 con 0x56011de6e6b0 2024-04-25T16:04:20.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.401+0000 7fd534ff9640 1 -- 172.21.15.31:0/2187068300 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd54400f850 con 0x56011de6e6b0 2024-04-25T16:04:20.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.404+0000 7fd55a67a740 1 -- 172.21.15.31:0/2187068300 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd54400fd00 con 0x56011de6e6b0 2024-04-25T16:04:20.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.411+0000 7fd534ff9640 1 -- 172.21.15.31:0/2187068300 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd54400fa50 con 0x56011de6e6b0 2024-04-25T16:04:20.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.411+0000 7fd55a67a740 10 client.25906 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:20.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.411+0000 7fd55a67a740 1 -- 172.21.15.31:0/2187068300 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56011df717d0 con 0x56011de6e6b0 2024-04-25T16:04:20.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.419+0000 7fd534ff9640 1 -- 172.21.15.31:0/2187068300 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd54400fd00 con 0x56011de6e6b0 2024-04-25T16:04:20.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.419+0000 7fd534ff9640 1 -- 172.21.15.31:0/2187068300 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fd544021420 con 0x56011de6e6b0 2024-04-25T16:04:20.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.420+0000 7fd55a67a740 10 client.25906 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:20.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.420+0000 7fd55a67a740 10 client.25906 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:04:20.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.420+0000 7fd55a67a740 20 client.25906 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:20.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.420+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56011de4df60 0x56011e249300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.420+0000 7fd55a67a740 4 client.25906 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:20.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.420+0000 7fd55a67a740 1 -- 172.21.15.31:0/2187068300 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56011e1a3990 con 0x56011de4df60 2024-04-25T16:04:20.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.421+0000 7fd551561640 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56011de4df60 0x56011e249300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.422+0000 7fd551561640 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56011de4df60 0x56011e249300 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4a615a3bd151da97 server_cookie=ea535eae65906c70 in_seq=0 out_seq=0 2024-04-25T16:04:20.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.423+0000 7fd534ff9640 10 client.25906 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:04:20.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.425+0000 7fd534ff9640 1 -- 172.21.15.31:0/2187068300 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56011e1a3990 con 0x56011de4df60 2024-04-25T16:04:20.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.425+0000 7fd534ff9640 10 client.25906 handle_command_reply: tid=0 2024-04-25T16:04:20.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.426+0000 7fd55a67a740 1 client.25906 shutdown 2024-04-25T16:04:20.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 -- 172.21.15.31:0/2187068300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 msgr2=0x56011e23bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:20.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x56011e22dfa0 tx=0x7fd54400e5d0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 -- 172.21.15.31:0/2187068300 shutdown_connections 2024-04-25T16:04:20.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e242c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56011de4df60 0x56011e249300 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2187068300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e23b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 -- 172.21.15.31:0/2187068300 >> 172.21.15.31:0/2187068300 conn(0x56011de834e0 msgr2=0x56011e230120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd534ff9640 0 client.25906 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:04:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 -- 172.21.15.31:0/2187068300 shutdown_connections 2024-04-25T16:04:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.427+0000 7fd55a67a740 1 -- 172.21.15.31:0/2187068300 wait complete. 2024-04-25T16:04:20.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.428+0000 7fd55a67a740 20 client.25906 trim_cache size 0 max 16384 2024-04-25T16:04:20.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.489+0000 7fd55a67a740 1 Processor -- start 2024-04-25T16:04:20.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.492+0000 7fd55a67a740 1 -- start start 2024-04-25T16:04:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.493+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e23b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.493+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.493+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de8f390 0x56011e242b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.493+0000 7fd55a67a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd53000e510 con 0x56011de664d0 2024-04-25T16:04:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.493+0000 7fd55a67a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd530010080 con 0x56011de6e6b0 2024-04-25T16:04:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.493+0000 7fd55a67a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd53000ebe0 con 0x56011de8f390 2024-04-25T16:04:20.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.493+0000 7fd550d60640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.493+0000 7fd550d60640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37328/0 (socket says 172.21.15.31:37328) 2024-04-25T16:04:20.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.494+0000 7fd550d60640 1 -- 172.21.15.31:0/2613976416 learned_addr learned my addr 172.21.15.31:0/2613976416 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.495+0000 7fd550d60640 1 -- 172.21.15.31:0/2613976416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de8f390 msgr2=0x56011e242b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.495+0000 7fd550d60640 1 --2- 172.21.15.31:0/2613976416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de8f390 0x56011e242b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.495+0000 7fd550d60640 1 -- 172.21.15.31:0/2613976416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 msgr2=0x56011e23b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:20.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.495+0000 7fd550d60640 1 --2- 172.21.15.31:0/2613976416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e23b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.495+0000 7fd550d60640 1 -- 172.21.15.31:0/2613976416 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56011e243090 con 0x56011de6e6b0 2024-04-25T16:04:20.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.496+0000 7fd550d60640 1 --2- 172.21.15.31:0/2613976416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd544000f80 tx=0x7fd544007cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de546758f54bdf45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:20.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.502+0000 7fd536ffd640 1 -- 172.21.15.31:0/2613976416 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544037020 con 0x56011de6e6b0 2024-04-25T16:04:20.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.503+0000 7fd536ffd640 1 -- 172.21.15.31:0/2613976416 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd544026070 con 0x56011de6e6b0 2024-04-25T16:04:20.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.503+0000 7fd536ffd640 1 -- 172.21.15.31:0/2613976416 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544014040 con 0x56011de6e6b0 2024-04-25T16:04:20.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.511+0000 7fd55a67a740 1 -- 172.21.15.31:0/2613976416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de6e6b0 msgr2=0x56011e23bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:20.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.511+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2613976416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd544000f80 tx=0x7fd544007cc0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.519+0000 7fd55a67a740 1 -- 172.21.15.31:0/2613976416 shutdown_connections 2024-04-25T16:04:20.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.519+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2613976416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de8f390 0x56011e242b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.519+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2613976416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de6e6b0 0x56011e23bcd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.519+0000 7fd55a67a740 1 --2- 172.21.15.31:0/2613976416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e23b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.519+0000 7fd55a67a740 1 -- 172.21.15.31:0/2613976416 >> 172.21.15.31:0/2613976416 conn(0x56011de834e0 msgr2=0x56011e22cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:20.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.527+0000 7fd55a67a740 1 -- 172.21.15.31:0/2613976416 shutdown_connections 2024-04-25T16:04:20.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.530+0000 7fd55a67a740 1 -- 172.21.15.31:0/2613976416 wait complete. 2024-04-25T16:04:20.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.532+0000 7fd55a67a740 1 -- start start 2024-04-25T16:04:20.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.532+0000 7fd55a67a740 1 Processor -- start 2024-04-25T16:04:20.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.533+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e2429e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.533+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.533+0000 7fd55a67a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e23faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.533+0000 7fd55a67a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd53000f980 con 0x56011de664d0 2024-04-25T16:04:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.533+0000 7fd55a67a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd53000f2b0 con 0x56011de8f390 2024-04-25T16:04:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.533+0000 7fd55a67a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd53000c4f0 con 0x56011de6e6b0 2024-04-25T16:04:20.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.533+0000 7fd55a67a740 10 client.0 _ll_register_callbacks cb 0x7fd544007df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.535+0000 7fd551d62640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e23faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.535+0000 7fd551d62640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e23faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37342/0 (socket says 172.21.15.31:37342) 2024-04-25T16:04:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.535+0000 7fd551d62640 1 -- 172.21.15.31:0/3769010417 learned_addr learned my addr 172.21.15.31:0/3769010417 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.535+0000 7fd551561640 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e2429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.535+0000 7fd550d60640 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.539+0000 7fd550d60640 1 -- 172.21.15.31:0/3769010417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 msgr2=0x56011e23faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.539+0000 7fd550d60640 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e23faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.539+0000 7fd550d60640 1 -- 172.21.15.31:0/3769010417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 msgr2=0x56011e2429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:20.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.539+0000 7fd550d60640 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e2429e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.539+0000 7fd550d60640 1 -- 172.21.15.31:0/3769010417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56011e2af660 con 0x56011de6e6b0 2024-04-25T16:04:20.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.541+0000 7fd550d60640 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23f560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x56011e23ca80 tx=0x7fd5440078a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f165670d2b0f3a81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:20.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.542+0000 7fd534ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:04:20.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.544+0000 7fd534ff9640 1 -- 172.21.15.31:0/3769010417 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544041060 con 0x56011de6e6b0 2024-04-25T16:04:20.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.545+0000 7fd55a67a740 1 -- 172.21.15.31:0/3769010417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd544026070 con 0x56011de6e6b0 2024-04-25T16:04:20.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.545+0000 7fd534ff9640 1 -- 172.21.15.31:0/3769010417 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd54400f650 con 0x56011de6e6b0 2024-04-25T16:04:20.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.549+0000 7fd534ff9640 1 -- 172.21.15.31:0/3769010417 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd544037030 con 0x56011de6e6b0 2024-04-25T16:04:20.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.553+0000 7fd55a67a740 10 client.25909 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:20.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.553+0000 7fd55a67a740 1 -- 172.21.15.31:0/3769010417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56011e2b0800 con 0x56011de6e6b0 2024-04-25T16:04:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.555+0000 7fd534ff9640 1 -- 172.21.15.31:0/3769010417 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd544026070 con 0x56011de6e6b0 2024-04-25T16:04:20.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.556+0000 7fd534ff9640 1 -- 172.21.15.31:0/3769010417 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fd544032030 con 0x56011de6e6b0 2024-04-25T16:04:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.558+0000 7fd55a67a740 10 client.25909 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.558+0000 7fd55a67a740 10 client.25909 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:04:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.558+0000 7fd55a67a740 20 client.25909 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.558+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56011de233a0 0x56011e2b3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.558+0000 7fd55a67a740 4 client.25909 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:04:20.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.558+0000 7fd55a67a740 1 -- 172.21.15.31:0/3769010417 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56011e19d800 con 0x56011de233a0 2024-04-25T16:04:20.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.560+0000 7fd551561640 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56011de233a0 0x56011e2b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:20.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.569+0000 7fd534ff9640 10 client.25909 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:04:20.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.569+0000 7fd551561640 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56011de233a0 0x56011e2b3140 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf510f6a4854c5b8 server_cookie=806fd3a659a6c4c in_seq=0 out_seq=0 2024-04-25T16:04:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.577+0000 7fd534ff9640 1 -- 172.21.15.31:0/3769010417 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56011e19d800 con 0x56011de233a0 2024-04-25T16:04:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.577+0000 7fd534ff9640 10 client.25909 handle_command_reply: tid=0 2024-04-25T16:04:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.577+0000 7fd55a67a740 1 client.25909 shutdown 2024-04-25T16:04:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.577+0000 7fd55a67a740 1 -- 172.21.15.31:0/3769010417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 msgr2=0x56011e23f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:20.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.577+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23f560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x56011e23ca80 tx=0x7fd5440078a0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.581+0000 7fd55a67a740 1 -- 172.21.15.31:0/3769010417 shutdown_connections 2024-04-25T16:04:20.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.581+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56011de8f390 0x56011e23faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.581+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56011de6e6b0 0x56011e23f560 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.581+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56011de233a0 0x56011e2b3140 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.581+0000 7fd55a67a740 1 --2- 172.21.15.31:0/3769010417 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56011de664d0 0x56011e2429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.581+0000 7fd55a67a740 1 -- 172.21.15.31:0/3769010417 >> 172.21.15.31:0/3769010417 conn(0x56011de834e0 msgr2=0x56011e236480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:20.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.581+0000 7fd534ff9640 0 client.25909 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:04:20.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.590+0000 7fd55a67a740 1 -- 172.21.15.31:0/3769010417 shutdown_connections 2024-04-25T16:04:20.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.590+0000 7fd55a67a740 1 -- 172.21.15.31:0/3769010417 wait complete. 2024-04-25T16:04:20.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.594+0000 7fd55a67a740 20 client.25909 trim_cache size 0 max 16384 2024-04-25T16:04:20.603 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.610+0000 7fd552563640 1 -- 172.21.15.31:0/1763059910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5380afc30 msgr2=0x7fd5380b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.610+0000 7fd552563640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5380afc30 0x7fd5380b2140 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fd548009710 tx=0x7fd548006c90 comp rx=0 tx=0).stop 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.610+0000 7fd552563640 1 -- 172.21.15.31:0/1763059910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 msgr2=0x7fd54c118590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.610+0000 7fd552563640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 0x7fd54c118590 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd53c02aa30 tx=0x7fd53c003d60 comp rx=0 tx=0).stop 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.611+0000 7fd552563640 1 -- 172.21.15.31:0/1763059910 shutdown_connections 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.611+0000 7fd552563640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd5380afc30 0x7fd5380b2140 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.611+0000 7fd552563640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd54c10fef0 0x7fd54c118590 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.611+0000 7fd552563640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd54c10b1d0 0x7fd54c117630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.611+0000 7fd552563640 1 --2- 172.21.15.31:0/1763059910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd54c10a350 0x7fd54c1170f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:20.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.611+0000 7fd552563640 1 -- 172.21.15.31:0/1763059910 >> 172.21.15.31:0/1763059910 conn(0x7fd54c070400 msgr2=0x7fd54c10bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:20.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.618+0000 7fd552563640 1 -- 172.21.15.31:0/1763059910 shutdown_connections 2024-04-25T16:04:20.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:20.622+0000 7fd552563640 1 -- 172.21.15.31:0/1763059910 wait complete. 2024-04-25T16:04:20.676 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:04:20.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:20 smithi031 ceph-mon[29140]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 797 MiB/s wr, 1.53k op/s 2024-04-25T16:04:21.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:20 smithi063 ceph-mon[32413]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 797 MiB/s wr, 1.53k op/s 2024-04-25T16:04:21.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:20 smithi186 ceph-mon[33258]: pgmap v442: 97 pgs: 97 active+clean; 10 GiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 797 MiB/s wr, 1.53k op/s 2024-04-25T16:04:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.152+0000 7f9a159ff640 1 -- 172.21.15.31:0/3478930900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1010fe90 msgr2=0x7f9a101102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:21.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.152+0000 7f9a159ff640 1 --2- 172.21.15.31:0/3478930900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1010fe90 0x7f9a101102a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9a04009900 tx=0x7f9a0402a440 comp rx=0 tx=0).stop 2024-04-25T16:04:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.154+0000 7f9a159ff640 1 -- 172.21.15.31:0/3478930900 shutdown_connections 2024-04-25T16:04:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.154+0000 7f9a159ff640 1 --2- 172.21.15.31:0/3478930900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1010fe90 0x7f9a101102a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.154+0000 7f9a159ff640 1 --2- 172.21.15.31:0/3478930900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a1010b190 0x7f9a1010b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.154+0000 7f9a159ff640 1 --2- 172.21.15.31:0/3478930900 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a1010a340 0x7f9a1010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.154+0000 7f9a159ff640 1 -- 172.21.15.31:0/3478930900 >> 172.21.15.31:0/3478930900 conn(0x7f9a10070320 msgr2=0x7f9a100727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:21.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.156+0000 7f9a159ff640 1 -- 172.21.15.31:0/3478930900 shutdown_connections 2024-04-25T16:04:21.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.157+0000 7f9a159ff640 1 -- 172.21.15.31:0/3478930900 wait complete. 2024-04-25T16:04:21.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.158+0000 7f9a159ff640 1 Processor -- start 2024-04-25T16:04:21.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.158+0000 7f9a159ff640 1 -- start start 2024-04-25T16:04:21.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.159+0000 7f9a159ff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a1010a340 0x7f9a101a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.159+0000 7f9a159ff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1010b190 0x7f9a101a9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.159+0000 7f9a159ff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a1010fe90 0x7f9a101b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.159+0000 7f9a159ff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1011e280 con 0x7f9a1010a340 2024-04-25T16:04:21.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.159+0000 7f9a159ff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1011e100 con 0x7f9a1010fe90 2024-04-25T16:04:21.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.159+0000 7f9a159ff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a1011e400 con 0x7f9a1010b190 2024-04-25T16:04:21.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.161+0000 7f9a151fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a1010fe90 0x7f9a101b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.162+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1010b190 0x7f9a101a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.162+0000 7f9a151fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a1010fe90 0x7f9a101b02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37360/0 (socket says 172.21.15.31:37360) 2024-04-25T16:04:21.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.162+0000 7f9a151fe640 1 -- 172.21.15.31:0/2172095700 learned_addr learned my addr 172.21.15.31:0/2172095700 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:21.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.164+0000 7f9a149fd640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a1010a340 0x7f9a101a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.168+0000 7f9a151fe640 1 -- 172.21.15.31:0/2172095700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1010b190 msgr2=0x7f9a101a9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:21.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.168+0000 7f9a151fe640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1010b190 0x7f9a101a9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.168+0000 7f9a151fe640 1 -- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a1010a340 msgr2=0x7f9a101a8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:21.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.168+0000 7f9a151fe640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a1010a340 0x7f9a101a8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.168+0000 7f9a151fe640 1 -- 172.21.15.31:0/2172095700 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a101b0930 con 0x7f9a1010fe90 2024-04-25T16:04:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.173+0000 7f9a151fe640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a1010fe90 0x7f9a101b02c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9a04005ab0 tx=0x7f9a04004340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:21.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.177+0000 7f9a0dffb640 1 -- 172.21.15.31:0/2172095700 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a04040030 con 0x7f9a1010fe90 2024-04-25T16:04:21.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.177+0000 7f9a159ff640 1 -- 172.21.15.31:0/2172095700 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a101b0bc0 con 0x7f9a1010fe90 2024-04-25T16:04:21.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.177+0000 7f9a159ff640 1 -- 172.21.15.31:0/2172095700 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a101b10b0 con 0x7f9a1010fe90 2024-04-25T16:04:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.184+0000 7f9a1dbeb740 1 Processor -- start 2024-04-25T16:04:21.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.184+0000 7f9a1dbeb740 1 -- start start 2024-04-25T16:04:21.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.188+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 0x5569179ff9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.188+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x5569179fff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.188+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a00bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.188+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556917963d80 con 0x55691763c6b0 2024-04-25T16:04:21.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.188+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55691791da20 con 0x5569176344d0 2024-04-25T16:04:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.188+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55691795fd40 con 0x55691765d390 2024-04-25T16:04:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.189+0000 7f9a0dffb640 1 -- 172.21.15.31:0/2172095700 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a040050f0 con 0x7f9a1010fe90 2024-04-25T16:04:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.189+0000 7f9a0dffb640 1 -- 172.21.15.31:0/2172095700 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0403c880 con 0x7f9a1010fe90 2024-04-25T16:04:21.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.190+0000 7f9a149fd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x5569179fff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.190+0000 7f9a149fd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x5569179fff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37376/0 (socket says 172.21.15.31:37376) 2024-04-25T16:04:21.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.190+0000 7f9a149fd640 1 -- 172.21.15.31:0/1788361178 learned_addr learned my addr 172.21.15.31:0/1788361178 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.190+0000 7f9a0ffff640 1 --2- 172.21.15.31:0/1788361178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 0x5569179ff9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.190+0000 7f9a0dffb640 1 -- 172.21.15.31:0/2172095700 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9a04043030 con 0x7f9a1010fe90 2024-04-25T16:04:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.191+0000 7f9a0ffff640 1 -- 172.21.15.31:0/1788361178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 msgr2=0x5569179fff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.191+0000 7f9a0ffff640 1 --2- 172.21.15.31:0/1788361178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x5569179fff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.191+0000 7f9a0ffff640 1 -- 172.21.15.31:0/1788361178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 msgr2=0x556917a00bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:21.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.191+0000 7f9a0ffff640 1 --2- 172.21.15.31:0/1788361178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a00bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.193+0000 7f9a0ffff640 1 -- 172.21.15.31:0/1788361178 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556917928c30 con 0x55691765d390 2024-04-25T16:04:21.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.194+0000 7f9a0dffb640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99f80afd30 0x7f99f80b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.194+0000 7f9a0ffff640 1 --2- 172.21.15.31:0/1788361178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 0x5569179ff9e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9a101a9e90 tx=0x7f9a00011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f5d770464ec16a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:21.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.195+0000 7f9a0dffb640 1 -- 172.21.15.31:0/2172095700 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9a04100310 con 0x7f9a1010fe90 2024-04-25T16:04:21.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.197+0000 7f99f6ffd640 1 -- 172.21.15.31:0/1788361178 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a00004220 con 0x55691765d390 2024-04-25T16:04:21.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.197+0000 7f99f6ffd640 1 -- 172.21.15.31:0/1788361178 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a000043c0 con 0x55691765d390 2024-04-25T16:04:21.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.198+0000 7f9a149fd640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99f80afd30 0x7f99f80b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.199+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/1788361178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 msgr2=0x5569179ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:21.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.199+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/1788361178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 0x5569179ff9e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9a101a9e90 tx=0x7f9a00011810 comp rx=0 tx=0).stop 2024-04-25T16:04:21.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.201+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/1788361178 shutdown_connections 2024-04-25T16:04:21.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.201+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/1788361178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a00bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.201+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/1788361178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x5569179fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.201+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/1788361178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 0x5569179ff9e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.201+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/1788361178 >> 172.21.15.31:0/1788361178 conn(0x5569176514e0 msgr2=0x5569179fb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:21.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.203+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/1788361178 shutdown_connections 2024-04-25T16:04:21.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.203+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/1788361178 wait complete. 2024-04-25T16:04:21.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.204+0000 7f9a1dbeb740 1 -- start start 2024-04-25T16:04:21.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.204+0000 7f9a1dbeb740 1 Processor -- start 2024-04-25T16:04:21.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.206+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x556917a09620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.206+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a09b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.206+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 0x556917a10ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.206+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556917a01750 con 0x55691763c6b0 2024-04-25T16:04:21.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.206+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5569178eadc0 con 0x5569176344d0 2024-04-25T16:04:21.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.206+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556917a018d0 con 0x55691765d390 2024-04-25T16:04:21.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.206+0000 7f9a1dbeb740 10 client.0 _ll_register_callbacks cb 0x556917906600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:21.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.207+0000 7f9a151fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 0x556917a10ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.208+0000 7f9a149fd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x556917a09620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.208+0000 7f9a149fd640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99f80afd30 0x7f99f80b2240 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x5569179ff700 tx=0x7f9a08009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:21.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.208+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a09b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.208+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a09b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46056/0 (socket says 172.21.15.31:46056) 2024-04-25T16:04:21.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.208+0000 7f9a0ffff640 1 -- 172.21.15.31:0/3323716170 learned_addr learned my addr 172.21.15.31:0/3323716170 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.213+0000 7f9a149fd640 1 -- 172.21.15.31:0/3323716170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 msgr2=0x556917a10ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.213+0000 7f9a149fd640 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 0x556917a10ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.213+0000 7f9a149fd640 1 -- 172.21.15.31:0/3323716170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 msgr2=0x556917a09b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.213+0000 7f9a149fd640 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a09b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:21.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.213+0000 7f9a149fd640 1 -- 172.21.15.31:0/3323716170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569176753f0 con 0x5569176344d0 2024-04-25T16:04:21.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.215+0000 7f9a149fd640 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x556917a09620 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x5569179fbf80 tx=0x7f9a08004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c1d4e145f6dd063 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:21.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.215+0000 7f99f47f8640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:21.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.217+0000 7f99f47f8640 1 -- 172.21.15.31:0/3323716170 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a08004b20 con 0x5569176344d0 2024-04-25T16:04:21.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.217+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3323716170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a000043c0 con 0x5569176344d0 2024-04-25T16:04:21.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.217+0000 7f99f47f8640 1 -- 172.21.15.31:0/3323716170 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9a08004cc0 con 0x5569176344d0 2024-04-25T16:04:21.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.218+0000 7f99f47f8640 1 -- 172.21.15.31:0/3323716170 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9a0801b670 con 0x5569176344d0 2024-04-25T16:04:21.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.221+0000 7f9a1dbeb740 10 client.25961 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:21.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.221+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3323716170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55691773f7d0 con 0x5569176344d0 2024-04-25T16:04:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.222+0000 7f99f47f8640 1 -- 172.21.15.31:0/3323716170 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a000043c0 con 0x5569176344d0 2024-04-25T16:04:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.222+0000 7f99f47f8640 1 -- 172.21.15.31:0/3323716170 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f9a08019040 con 0x5569176344d0 2024-04-25T16:04:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.222+0000 7f9a1dbeb740 10 client.25961 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.222+0000 7f9a1dbeb740 10 client.25961 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.222+0000 7f9a1dbeb740 20 client.25961 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:21.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.222+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55691761bf60 0x556917a288a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:21.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.222+0000 7f9a1dbeb740 4 client.25961 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:21.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.222+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3323716170 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556917971990 con 0x55691761bf60 2024-04-25T16:04:21.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.227+0000 7f9a0ffff640 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55691761bf60 0x556917a288a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:21.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.235+0000 7f99f47f8640 10 client.25961 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:21.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:21.235+0000 7f9a0ffff640 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55691761bf60 0x556917a288a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f55b5de72d7a693b server_cookie=ee5de5adf3b034f0 in_seq=0 out_seq=0 2024-04-25T16:04:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:22 smithi186 ceph-mon[33258]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 769 MiB/s wr, 1.39k op/s 2024-04-25T16:04:23.207 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:22 smithi031 ceph-mon[29140]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 769 MiB/s wr, 1.39k op/s 2024-04-25T16:04:23.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:22 smithi063 ceph-mon[32413]: pgmap v443: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 769 MiB/s wr, 1.39k op/s 2024-04-25T16:04:23.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.666+0000 7f99f47f8640 1 -- 172.21.15.31:0/3323716170 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556917971990 con 0x55691761bf60 2024-04-25T16:04:23.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.666+0000 7f99f47f8640 10 client.25961 handle_command_reply: tid=0 2024-04-25T16:04:23.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.666+0000 7f9a1dbeb740 1 client.25961 shutdown 2024-04-25T16:04:23.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.666+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3323716170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 msgr2=0x556917a09620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:23.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.666+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x556917a09620 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x5569179fbf80 tx=0x7f9a08004a40 comp rx=0 tx=0).stop 2024-04-25T16:04:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d3cff9640 1 -- 172.21.15.31:0/477714293 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d76ef75ee0 con 0x55d76ebfc1c0 2024-04-25T16:04:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d3cff9640 10 client.25888 handle_command_reply: tid=0 2024-04-25T16:04:23.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d62411740 1 client.25888 shutdown 2024-04-25T16:04:23.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3323716170 shutdown_connections 2024-04-25T16:04:23.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55691761bf60 0x556917a288a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55691765d390 0x556917a10ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a09b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3323716170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5569176344d0 0x556917a09620 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3323716170 >> 172.21.15.31:0/3323716170 conn(0x5569176514e0 msgr2=0x5569179fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:23.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f99f47f8640 0 client.25961 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d62411740 1 -- 172.21.15.31:0/477714293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 msgr2=0x55d76f0266a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:23.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d62411740 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f0266a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8d4402a9d0 tx=0x7f8d44031400 comp rx=0 tx=0).stop 2024-04-25T16:04:23.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3323716170 shutdown_connections 2024-04-25T16:04:23.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d62411740 1 -- 172.21.15.31:0/477714293 shutdown_connections 2024-04-25T16:04:23.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d62411740 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d76ebfc1c0 0x55d76f0bdc80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d62411740 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d76ec681b0 0x55d76f0295e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d62411740 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d76ec474d0 0x55d76f0266a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d62411740 1 --2- 172.21.15.31:0/477714293 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d76ec3f2f0 0x55d76f026160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d62411740 1 -- 172.21.15.31:0/477714293 >> 172.21.15.31:0/477714293 conn(0x55d76ec5c300 msgr2=0x55d76f0072b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:23.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f8d3cff9640 0 client.25888 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:23.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.668+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3323716170 wait complete. 2024-04-25T16:04:23.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.669+0000 7f9a1dbeb740 20 client.25961 trim_cache size 0 max 16384 2024-04-25T16:04:23.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:04:23.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:04:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:04:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:04:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.671+0000 7f8d62411740 1 -- 172.21.15.31:0/477714293 shutdown_connections 2024-04-25T16:04:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.671+0000 7f8d62411740 1 -- 172.21.15.31:0/477714293 wait complete. 2024-04-25T16:04:23.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.680+0000 7f8d62411740 20 client.25888 trim_cache size 0 max 16384 2024-04-25T16:04:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.699+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d400afc30 msgr2=0x7f8d400b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.699+0000 7f8d5b492640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d400afc30 0x7f8d400b2140 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8d4c005c40 tx=0x7f8d4c005bd0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.699+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5410b1e0 msgr2=0x7f8d541a0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.699+0000 7f8d5b492640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5410b1e0 0x7f8d541a0b70 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f8d4402aa00 tx=0x7f8d44003c10 comp rx=0 tx=0).stop 2024-04-25T16:04:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.700+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 shutdown_connections 2024-04-25T16:04:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.700+0000 7f8d5b492640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d400afc30 0x7f8d400b2140 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.700+0000 7f8d5b492640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d5410fee0 0x7f8d541a7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.700+0000 7f8d5b492640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d5410b1e0 0x7f8d541a0b70 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.700+0000 7f8d5b492640 1 --2- 172.21.15.31:0/1073322758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d54109fa0 0x7f8d541a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.700+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 >> 172.21.15.31:0/1073322758 conn(0x7f8d54078bc0 msgr2=0x7f8d5410bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:23.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.704+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 shutdown_connections 2024-04-25T16:04:23.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:23.706+0000 7f8d5b492640 1 -- 172.21.15.31:0/1073322758 wait complete. 2024-04-25T16:04:23.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.722+0000 7f9a1dbeb740 1 Processor -- start 2024-04-25T16:04:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.725+0000 7f9a1dbeb740 1 -- start start 2024-04-25T16:04:23.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 0x556917a07fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:23.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a084f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:23.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 0x556917a08a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:23.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc00e590 con 0x55691763c6b0 2024-04-25T16:04:23.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc010100 con 0x55691765d390 2024-04-25T16:04:23.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc00ec60 con 0x5569176344d0 2024-04-25T16:04:23.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a084f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53246/0 (socket says 172.21.15.31:53246) 2024-04-25T16:04:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a0ffff640 1 -- 172.21.15.31:0/2589936771 learned_addr learned my addr 172.21.15.31:0/2589936771 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.726+0000 7f9a151fe640 1 --2- 172.21.15.31:0/2589936771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 0x556917a08a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.727+0000 7f9a149fd640 1 --2- 172.21.15.31:0/2589936771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 0x556917a07fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:23.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.727+0000 7f9a151fe640 1 -- 172.21.15.31:0/2589936771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 msgr2=0x556917a07fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.727+0000 7f9a151fe640 1 --2- 172.21.15.31:0/2589936771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 0x556917a07fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.727+0000 7f9a151fe640 1 -- 172.21.15.31:0/2589936771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 msgr2=0x556917a084f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.727+0000 7f9a151fe640 1 --2- 172.21.15.31:0/2589936771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a084f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.728+0000 7f9a151fe640 1 -- 172.21.15.31:0/2589936771 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556917a00650 con 0x55691765d390 2024-04-25T16:04:23.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.728+0000 7f9a151fe640 1 --2- 172.21.15.31:0/2589936771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 0x556917a08a30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9a040c8860 tx=0x7f9a0402a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1748063d16035e2a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:23.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.729+0000 7f99f67fc640 1 -- 172.21.15.31:0/2589936771 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a040e2070 con 0x55691765d390 2024-04-25T16:04:23.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.729+0000 7f99f67fc640 1 -- 172.21.15.31:0/2589936771 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a040c88d0 con 0x55691765d390 2024-04-25T16:04:23.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.729+0000 7f99f67fc640 1 -- 172.21.15.31:0/2589936771 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a0404a040 con 0x55691765d390 2024-04-25T16:04:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.731+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/2589936771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 msgr2=0x556917a08a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.731+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/2589936771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 0x556917a08a30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9a040c8860 tx=0x7f9a0402a440 comp rx=0 tx=0).stop 2024-04-25T16:04:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.731+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/2589936771 shutdown_connections 2024-04-25T16:04:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.731+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/2589936771 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 0x556917a08a30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.731+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/2589936771 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.731+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/2589936771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 0x556917a07fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.731+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/2589936771 >> 172.21.15.31:0/2589936771 conn(0x5569176514e0 msgr2=0x556917a04210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:23.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.732+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/2589936771 shutdown_connections 2024-04-25T16:04:23.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.732+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/2589936771 wait complete. 2024-04-25T16:04:23.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.733+0000 7f9a1dbeb740 1 -- start start 2024-04-25T16:04:23.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.733+0000 7f9a1dbeb740 1 Processor -- start 2024-04-25T16:04:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.734+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 0x556917a0fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.734+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a0ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.734+0000 7f9a1dbeb740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 0x556917a0cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.734+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc00fa00 con 0x55691763c6b0 2024-04-25T16:04:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.734+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc00f330 con 0x55691765d390 2024-04-25T16:04:23.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.734+0000 7f9a1dbeb740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f99fc00c570 con 0x5569176344d0 2024-04-25T16:04:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.734+0000 7f9a1dbeb740 10 client.0 _ll_register_callbacks cb 0x7f9a08009d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.735+0000 7f9a149fd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 0x556917a0fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.734+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a0ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.735+0000 7f9a0ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a0ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53254/0 (socket says 172.21.15.31:53254) 2024-04-25T16:04:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.735+0000 7f9a0ffff640 1 -- 172.21.15.31:0/3779927215 learned_addr learned my addr 172.21.15.31:0/3779927215 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:23.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.736+0000 7f9a149fd640 1 -- 172.21.15.31:0/3779927215 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 msgr2=0x556917a0cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.736+0000 7f9a149fd640 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 0x556917a0cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.736+0000 7f9a149fd640 1 -- 172.21.15.31:0/3779927215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 msgr2=0x556917a0ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.736+0000 7f9a149fd640 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a0ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.736+0000 7f9a149fd640 1 -- 172.21.15.31:0/3779927215 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556917a0d5c0 con 0x5569176344d0 2024-04-25T16:04:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.736+0000 7f9a0ffff640 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a0ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:04:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.737+0000 7f9a149fd640 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 0x556917a0fec0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a08025bb0 tx=0x7f9a08025f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd3fc86497ff0fa2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:23.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.737+0000 7f99f47f8640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:04:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.737+0000 7f99f47f8640 1 -- 172.21.15.31:0/3779927215 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a080254e0 con 0x5569176344d0 2024-04-25T16:04:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.737+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3779927215 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a040c88d0 con 0x5569176344d0 2024-04-25T16:04:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.737+0000 7f99f47f8640 1 -- 172.21.15.31:0/3779927215 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9a08025680 con 0x5569176344d0 2024-04-25T16:04:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.738+0000 7f99f47f8640 1 -- 172.21.15.31:0/3779927215 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9a08025880 con 0x5569176344d0 2024-04-25T16:04:23.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.738+0000 7f9a1dbeb740 10 client.25927 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.738+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3779927215 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556917a8fcf0 con 0x5569176344d0 2024-04-25T16:04:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.739+0000 7f99f47f8640 1 -- 172.21.15.31:0/3779927215 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a040c88d0 con 0x5569176344d0 2024-04-25T16:04:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.739+0000 7f99f47f8640 1 -- 172.21.15.31:0/3779927215 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f9a08019070 con 0x5569176344d0 2024-04-25T16:04:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.739+0000 7f9a1dbeb740 10 client.25927 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.739+0000 7f9a1dbeb740 10 client.25927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.739+0000 7f9a1dbeb740 20 client.25927 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.739+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5569175f13a0 0x556917a926f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.739+0000 7f9a1dbeb740 4 client.25927 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:04:23.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.739+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3779927215 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55691796b800 con 0x5569175f13a0 2024-04-25T16:04:23.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.745+0000 7f9a0ffff640 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5569175f13a0 0x556917a926f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:23.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.753+0000 7f9a0ffff640 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5569175f13a0 0x556917a926f0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92a45356fa27fe1c server_cookie=7a7a608086c36c85 in_seq=0 out_seq=0 2024-04-25T16:04:23.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.754+0000 7f99f47f8640 10 client.25927 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:23.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.756+0000 7f99f47f8640 1 -- 172.21.15.31:0/3779927215 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55691796b800 con 0x5569175f13a0 2024-04-25T16:04:23.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.756+0000 7f99f47f8640 10 client.25927 handle_command_reply: tid=0 2024-04-25T16:04:23.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.756+0000 7f9a1dbeb740 1 client.25927 shutdown 2024-04-25T16:04:23.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.756+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3779927215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 msgr2=0x556917a0fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:23.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.756+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 0x556917a0fec0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a08025bb0 tx=0x7f9a08025f00 comp rx=0 tx=0).stop 2024-04-25T16:04:23.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.758+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3779927215 shutdown_connections 2024-04-25T16:04:23.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.758+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5569175f13a0 0x556917a926f0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.758+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55691765d390 0x556917a0cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.758+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55691763c6b0 0x556917a0ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.758+0000 7f9a1dbeb740 1 --2- 172.21.15.31:0/3779927215 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5569176344d0 0x556917a0fec0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.758+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3779927215 >> 172.21.15.31:0/3779927215 conn(0x5569176514e0 msgr2=0x5569179fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:23.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.758+0000 7f99f47f8640 0 client.25927 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:23.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.761+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3779927215 shutdown_connections 2024-04-25T16:04:23.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.762+0000 7f9a1dbeb740 1 -- 172.21.15.31:0/3779927215 wait complete. 2024-04-25T16:04:23.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.763+0000 7f9a1dbeb740 20 client.25927 trim_cache size 0 max 16384 2024-04-25T16:04:23.765 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:04:23.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.766+0000 7f9a159ff640 1 -- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99f80afd30 msgr2=0x7f99f80b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:23.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.767+0000 7f9a159ff640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99f80afd30 0x7f99f80b2240 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x5569179ff700 tx=0x7f9a08009680 comp rx=0 tx=0).stop 2024-04-25T16:04:23.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.767+0000 7f9a159ff640 1 -- 172.21.15.31:0/2172095700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a1010fe90 msgr2=0x7f9a101b02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:23.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.767+0000 7f9a159ff640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a1010fe90 0x7f9a101b02c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9a04005ab0 tx=0x7f9a04004340 comp rx=0 tx=0).stop 2024-04-25T16:04:23.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.767+0000 7f9a159ff640 1 -- 172.21.15.31:0/2172095700 shutdown_connections 2024-04-25T16:04:23.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.767+0000 7f9a159ff640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f99f80afd30 0x7f99f80b2240 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.767+0000 7f9a159ff640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a1010fe90 0x7f9a101b02c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.767+0000 7f9a159ff640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a1010b190 0x7f9a101a9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.767+0000 7f9a159ff640 1 --2- 172.21.15.31:0/2172095700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a1010a340 0x7f9a101a8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:23.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.768+0000 7f9a159ff640 1 -- 172.21.15.31:0/2172095700 >> 172.21.15.31:0/2172095700 conn(0x7f9a10070320 msgr2=0x7f9a1010be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:23.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.768+0000 7f9a159ff640 1 -- 172.21.15.31:0/2172095700 shutdown_connections 2024-04-25T16:04:23.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:23.768+0000 7f9a159ff640 1 -- 172.21.15.31:0/2172095700 wait complete. 2024-04-25T16:04:24.844 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:04:24.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:04:24.845 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:04:25.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:24 smithi186 ceph-mon[33258]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 971 MiB/s wr, 1.82k op/s 2024-04-25T16:04:25.203 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:24 smithi031 ceph-mon[29140]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 971 MiB/s wr, 1.82k op/s 2024-04-25T16:04:25.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.318+0000 7f59e359e640 1 -- 172.21.15.31:0/1825071795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e410a350 msgr2=0x7f59e410ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:25.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.318+0000 7f59e359e640 1 --2- 172.21.15.31:0/1825071795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e410a350 0x7f59e410ac90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009860 tx=0x7f59d402a360 comp rx=0 tx=0).stop 2024-04-25T16:04:25.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.326+0000 7f59e359e640 1 -- 172.21.15.31:0/1825071795 shutdown_connections 2024-04-25T16:04:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.326+0000 7f59e359e640 1 --2- 172.21.15.31:0/1825071795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59e410fef0 0x7f59e4110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.326+0000 7f59e359e640 1 --2- 172.21.15.31:0/1825071795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e410b1d0 0x7f59e410b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.326+0000 7f59e359e640 1 --2- 172.21.15.31:0/1825071795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e410a350 0x7f59e410ac90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.326+0000 7f59e359e640 1 -- 172.21.15.31:0/1825071795 >> 172.21.15.31:0/1825071795 conn(0x7f59e4070400 msgr2=0x7f59e4072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:25.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.328+0000 7f59e359e640 1 -- 172.21.15.31:0/1825071795 shutdown_connections 2024-04-25T16:04:25.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.329+0000 7f59e359e640 1 -- 172.21.15.31:0/1825071795 wait complete. 2024-04-25T16:04:25.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.331+0000 7f59e359e640 1 Processor -- start 2024-04-25T16:04:25.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.332+0000 7f59e359e640 1 -- start start 2024-04-25T16:04:25.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.332+0000 7f59e359e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59e410a350 0x7f59e41a8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.333+0000 7f59e359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e410b1d0 0x7f59e41a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.334+0000 7f59e1d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e410b1d0 0x7f59e41a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.334+0000 7f59e259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59e410a350 0x7f59e41a8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.334+0000 7f59e359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e410fef0 0x7f59e41b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.334+0000 7f59e259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59e410a350 0x7f59e41a8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52130/0 (socket says 172.21.15.31:52130) 2024-04-25T16:04:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.334+0000 7f59e259c640 1 -- 172.21.15.31:0/764424304 learned_addr learned my addr 172.21.15.31:0/764424304 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.334+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f59e411e290 con 0x7f59e410b1d0 2024-04-25T16:04:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.335+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f59e411e110 con 0x7f59e410a350 2024-04-25T16:04:25.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.335+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59e411e410 con 0x7f59e410fef0 2024-04-25T16:04:25.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.338+0000 7f59e2d9d640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e410fef0 0x7f59e41b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.339+0000 7f59e259c640 1 -- 172.21.15.31:0/764424304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e410fef0 msgr2=0x7f59e41b0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:25.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.339+0000 7f59e259c640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e410fef0 0x7f59e41b0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.339+0000 7f59e259c640 1 -- 172.21.15.31:0/764424304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e410b1d0 msgr2=0x7f59e41a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:25.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.339+0000 7f59e259c640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e410b1d0 0x7f59e41a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.339+0000 7f59e259c640 1 -- 172.21.15.31:0/764424304 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59e41b0a00 con 0x7f59e410a350 2024-04-25T16:04:25.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.340+0000 7f59e259c640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59e410a350 0x7f59e41a8ec0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009830 tx=0x7f59d4033e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:25.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.360+0000 7f59d37fe640 1 -- 172.21.15.31:0/764424304 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d403f030 con 0x7f59e410a350 2024-04-25T16:04:25.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.361+0000 7f59d37fe640 1 -- 172.21.15.31:0/764424304 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d4004020 con 0x7f59e410a350 2024-04-25T16:04:25.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.361+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59e41b0c90 con 0x7f59e410a350 2024-04-25T16:04:25.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.361+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59e41b10c0 con 0x7f59e410a350 2024-04-25T16:04:25.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.361+0000 7f59d37fe640 1 -- 172.21.15.31:0/764424304 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4037040 con 0x7f59e410a350 2024-04-25T16:04:25.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:24 smithi063 ceph-mon[32413]: pgmap v444: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 971 MiB/s wr, 1.82k op/s 2024-04-25T16:04:25.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.368+0000 7f59ef767740 1 Processor -- start 2024-04-25T16:04:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.373+0000 7f59ef767740 1 -- start start 2024-04-25T16:04:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.373+0000 7f59ef767740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ee0410 0x55f4922816a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.373+0000 7f59ef767740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f492281c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.373+0000 7f59ef767740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f492288b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.374+0000 7f59ef767740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f4921e6b00 con 0x55f491ee0410 2024-04-25T16:04:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.374+0000 7f59ef767740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f492182920 con 0x55f491ebf730 2024-04-25T16:04:25.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.374+0000 7f59ef767740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f4921e2ac0 con 0x55f491eb7550 2024-04-25T16:04:25.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.374+0000 7f59d37fe640 1 -- 172.21.15.31:0/764424304 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f59d4042070 con 0x7f59e410a350 2024-04-25T16:04:25.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.375+0000 7f59d37fe640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59cc0afd00 0x7f59cc0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.377+0000 7f59e1d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f492288b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.377+0000 7f59e259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f492281c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.377+0000 7f59e1d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f492288b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52132/0 (socket says 172.21.15.31:52132) 2024-04-25T16:04:25.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.377+0000 7f59e1d9b640 1 -- 172.21.15.31:0/3956560382 learned_addr learned my addr 172.21.15.31:0/3956560382 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:25.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.381+0000 7f59e1d9b640 1 -- 172.21.15.31:0/3956560382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 msgr2=0x55f492281c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.381+0000 7f59e1d9b640 1 --2- 172.21.15.31:0/3956560382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f492281c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.381+0000 7f59e1d9b640 1 -- 172.21.15.31:0/3956560382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ee0410 msgr2=0x55f4922816a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.381+0000 7f59e1d9b640 1 --2- 172.21.15.31:0/3956560382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ee0410 0x55f4922816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.381+0000 7f59e1d9b640 1 -- 172.21.15.31:0/3956560382 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f491fc07e0 con 0x55f491ebf730 2024-04-25T16:04:25.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.381+0000 7f59e1d9b640 1 --2- 172.21.15.31:0/3956560382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f492288b20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f59d800b710 tx=0x7f59d800bbd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e43862d3d2e287de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:25.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.382+0000 7f59d0ff9640 1 -- 172.21.15.31:0/3956560382 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d8003dd0 con 0x55f491ebf730 2024-04-25T16:04:25.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.383+0000 7f59d37fe640 1 -- 172.21.15.31:0/764424304 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f59d40fe9f0 con 0x7f59e410a350 2024-04-25T16:04:25.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.383+0000 7f59d0ff9640 1 -- 172.21.15.31:0/3956560382 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d8015070 con 0x55f491ebf730 2024-04-25T16:04:25.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.384+0000 7f59ef767740 1 -- 172.21.15.31:0/3956560382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 msgr2=0x55f492288b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:25.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.384+0000 7f59ef767740 1 --2- 172.21.15.31:0/3956560382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f492288b20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f59d800b710 tx=0x7f59d800bbd0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.386+0000 7f59e2d9d640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59cc0afd00 0x7f59cc0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.387+0000 7f59ef767740 1 -- 172.21.15.31:0/3956560382 shutdown_connections 2024-04-25T16:04:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.387+0000 7f59ef767740 1 --2- 172.21.15.31:0/3956560382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f492288b20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.387+0000 7f59ef767740 1 --2- 172.21.15.31:0/3956560382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f492281c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.387+0000 7f59ef767740 1 --2- 172.21.15.31:0/3956560382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ee0410 0x55f4922816a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.387+0000 7f59ef767740 1 -- 172.21.15.31:0/3956560382 >> 172.21.15.31:0/3956560382 conn(0x55f491ed4560 msgr2=0x55f49227d030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:25.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.392+0000 7f59e2d9d640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59cc0afd00 0x7f59cc0b2210 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55f492282870 tx=0x7f59dc009490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:25.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.396+0000 7f59ef767740 1 -- 172.21.15.31:0/3956560382 shutdown_connections 2024-04-25T16:04:25.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.408+0000 7f59ef767740 1 -- 172.21.15.31:0/3956560382 wait complete. 2024-04-25T16:04:25.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.408+0000 7f59ef767740 1 -- start start 2024-04-25T16:04:25.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.408+0000 7f59ef767740 1 Processor -- start 2024-04-25T16:04:25.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.409+0000 7f59ef767740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f49228b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.409+0000 7f59ef767740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ebf730 0x55f49228ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.409+0000 7f59ef767740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f4922929d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.409+0000 7f59ef767740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f4922895c0 con 0x55f491ebf730 2024-04-25T16:04:25.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.409+0000 7f59ef767740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f4921805d0 con 0x55f491ee0410 2024-04-25T16:04:25.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.409+0000 7f59ef767740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f492289740 con 0x55f491eb7550 2024-04-25T16:04:25.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.409+0000 7f59ef767740 10 client.0 _ll_register_callbacks cb 0x55f49216e080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:25.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.410+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f4922929d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.411+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f4922929d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52136/0 (socket says 172.21.15.31:52136) 2024-04-25T16:04:25.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.411+0000 7f59e2d9d640 1 -- 172.21.15.31:0/4109328502 learned_addr learned my addr 172.21.15.31:0/4109328502 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:25.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.411+0000 7f59e1d9b640 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ebf730 0x55f49228ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.412+0000 7f59e2d9d640 1 -- 172.21.15.31:0/4109328502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 msgr2=0x55f49228b530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:04:25.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.412+0000 7f59e2d9d640 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f49228b530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.412+0000 7f59e2d9d640 1 -- 172.21.15.31:0/4109328502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ebf730 msgr2=0x55f49228ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:25.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.412+0000 7f59e2d9d640 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ebf730 0x55f49228ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.412+0000 7f59e2d9d640 1 -- 172.21.15.31:0/4109328502 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f492020f70 con 0x55f491ee0410 2024-04-25T16:04:25.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.413+0000 7f59e2d9d640 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f4922929d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f59dc017c00 tx=0x7f59dc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b54f984a23fef9b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:25.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.413+0000 7f59c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:25.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.413+0000 7f59c9ffb640 1 -- 172.21.15.31:0/4109328502 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc027060 con 0x55f491ee0410 2024-04-25T16:04:25.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.413+0000 7f59c9ffb640 1 -- 172.21.15.31:0/4109328502 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f59dc012030 con 0x55f491ee0410 2024-04-25T16:04:25.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.413+0000 7f59ef767740 1 -- 172.21.15.31:0/4109328502 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d8015070 con 0x55f491ee0410 2024-04-25T16:04:25.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.415+0000 7f59c9ffb640 1 -- 172.21.15.31:0/4109328502 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f59dc017cf0 con 0x55f491ee0410 2024-04-25T16:04:25.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.415+0000 7f59ef767740 10 client.25973 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:25.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.415+0000 7f59ef767740 1 -- 172.21.15.31:0/4109328502 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f491eb8170 con 0x55f491ee0410 2024-04-25T16:04:25.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.416+0000 7f59c9ffb640 1 -- 172.21.15.31:0/4109328502 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d8015070 con 0x55f491ee0410 2024-04-25T16:04:25.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.416+0000 7f59c9ffb640 1 -- 172.21.15.31:0/4109328502 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f59dc02b800 con 0x55f491ee0410 2024-04-25T16:04:25.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.416+0000 7f59ef767740 10 client.25973 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:25.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.416+0000 7f59ef767740 10 client.25973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:25.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.416+0000 7f59ef767740 20 client.25973 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:25.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.416+0000 7f59ef767740 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f491e9f1f0 0x55f492299390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.417+0000 7f59ef767740 4 client.25973 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:25.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.417+0000 7f59ef767740 1 -- 172.21.15.31:0/4109328502 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4921f4710 con 0x55f491e9f1f0 2024-04-25T16:04:25.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.418+0000 7f59e259c640 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f491e9f1f0 0x55f492299390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.419+0000 7f59c9ffb640 10 client.25973 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:25.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.419+0000 7f59e259c640 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f491e9f1f0 0x55f492299390 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc7147e9f08f5cb8 server_cookie=731c26812dd8e29b in_seq=0 out_seq=0 2024-04-25T16:04:25.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.427+0000 7f59c9ffb640 1 -- 172.21.15.31:0/4109328502 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f4921f4710 con 0x55f491e9f1f0 2024-04-25T16:04:25.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.427+0000 7f59c9ffb640 10 client.25973 handle_command_reply: tid=0 2024-04-25T16:04:25.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.427+0000 7f59ef767740 1 client.25973 shutdown 2024-04-25T16:04:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.439+0000 7f59ef767740 1 -- 172.21.15.31:0/4109328502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 msgr2=0x55f4922929d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:25.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.439+0000 7f59ef767740 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f4922929d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f59dc017c00 tx=0x7f59dc004a40 comp rx=0 tx=0).stop 2024-04-25T16:04:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59ef767740 1 -- 172.21.15.31:0/4109328502 shutdown_connections 2024-04-25T16:04:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59ef767740 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f491e9f1f0 0x55f492299390 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59ef767740 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f4922929d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59ef767740 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ebf730 0x55f49228ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59ef767740 1 --2- 172.21.15.31:0/4109328502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f49228b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59ef767740 1 -- 172.21.15.31:0/4109328502 >> 172.21.15.31:0/4109328502 conn(0x55f491ed4560 msgr2=0x55f49227c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59c9ffb640 0 client.25973 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:25.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59ef767740 1 -- 172.21.15.31:0/4109328502 shutdown_connections 2024-04-25T16:04:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59ef767740 1 -- 172.21.15.31:0/4109328502 wait complete. 2024-04-25T16:04:25.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.440+0000 7f59ef767740 20 client.25973 trim_cache size 0 max 16384 2024-04-25T16:04:25.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.491+0000 7f59ef767740 1 Processor -- start 2024-04-25T16:04:25.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.491+0000 7f59ef767740 1 -- start start 2024-04-25T16:04:25.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.492+0000 7f59ef767740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f49227b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.492+0000 7f59ef767740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f4922928f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.492+0000 7f59ef767740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ee0410 0x55f492292e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.493+0000 7f59e259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f49227b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.493+0000 7f59ef767740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f59c400e540 con 0x55f491ee0410 2024-04-25T16:04:25.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.493+0000 7f59ef767740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f59c40100b0 con 0x55f491ebf730 2024-04-25T16:04:25.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.493+0000 7f59ef767740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59c400ec10 con 0x55f491eb7550 2024-04-25T16:04:25.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.493+0000 7f59e259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f49227b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37100/0 (socket says 172.21.15.31:37100) 2024-04-25T16:04:25.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.493+0000 7f59e259c640 1 -- 172.21.15.31:0/3806759126 learned_addr learned my addr 172.21.15.31:0/3806759126 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:25.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.493+0000 7f59e1d9b640 1 --2- 172.21.15.31:0/3806759126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f4922928f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.494+0000 7f59e259c640 1 -- 172.21.15.31:0/3806759126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 msgr2=0x55f4922928f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:25.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.494+0000 7f59e259c640 1 --2- 172.21.15.31:0/3806759126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f4922928f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.494+0000 7f59e259c640 1 -- 172.21.15.31:0/3806759126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ee0410 msgr2=0x55f492292e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:25.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.494+0000 7f59e259c640 1 --2- 172.21.15.31:0/3806759126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ee0410 0x55f492292e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.494+0000 7f59e259c640 1 -- 172.21.15.31:0/3806759126 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f49228c130 con 0x55f491eb7550 2024-04-25T16:04:25.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.498+0000 7f59e259c640 1 --2- 172.21.15.31:0/3806759126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f49227b620 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59d4005c40 tx=0x7f59d40fe040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48a572a1297cd6a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:25.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.499+0000 7f59cbfff640 1 -- 172.21.15.31:0/3806759126 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d4004480 con 0x55f491eb7550 2024-04-25T16:04:25.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.499+0000 7f59cbfff640 1 -- 172.21.15.31:0/3806759126 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d40c7060 con 0x55f491eb7550 2024-04-25T16:04:25.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.499+0000 7f59cbfff640 1 -- 172.21.15.31:0/3806759126 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59d40c40f0 con 0x55f491eb7550 2024-04-25T16:04:25.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.500+0000 7f59ef767740 1 -- 172.21.15.31:0/3806759126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 msgr2=0x55f49227b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:25.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.500+0000 7f59ef767740 1 --2- 172.21.15.31:0/3806759126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f49227b620 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f59d4005c40 tx=0x7f59d40fe040 comp rx=0 tx=0).stop 2024-04-25T16:04:25.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.502+0000 7f59ef767740 1 -- 172.21.15.31:0/3806759126 shutdown_connections 2024-04-25T16:04:25.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.502+0000 7f59ef767740 1 --2- 172.21.15.31:0/3806759126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491ee0410 0x55f492292e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.502+0000 7f59ef767740 1 --2- 172.21.15.31:0/3806759126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ebf730 0x55f4922928f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.502+0000 7f59ef767740 1 --2- 172.21.15.31:0/3806759126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491eb7550 0x55f49227b620 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.502+0000 7f59ef767740 1 -- 172.21.15.31:0/3806759126 >> 172.21.15.31:0/3806759126 conn(0x55f491ed4560 msgr2=0x55f492285880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:25.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.503+0000 7f59ef767740 1 -- 172.21.15.31:0/3806759126 shutdown_connections 2024-04-25T16:04:25.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.503+0000 7f59ef767740 1 -- 172.21.15.31:0/3806759126 wait complete. 2024-04-25T16:04:25.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.503+0000 7f59ef767740 1 -- start start 2024-04-25T16:04:25.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.503+0000 7f59ef767740 1 Processor -- start 2024-04-25T16:04:25.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.504+0000 7f59ef767740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491eb7550 0x55f492290ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.504+0000 7f59ef767740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491ebf730 0x55f492291010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.504+0000 7f59ef767740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f492291f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.504+0000 7f59ef767740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f59c400f9b0 con 0x55f491eb7550 2024-04-25T16:04:25.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.504+0000 7f59ef767740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f59c400f2e0 con 0x55f491ee0410 2024-04-25T16:04:25.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.504+0000 7f59ef767740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f59c400c520 con 0x55f491ebf730 2024-04-25T16:04:25.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.504+0000 7f59ef767740 10 client.0 _ll_register_callbacks cb 0x7f59dc017880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:25.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.505+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f492291f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.505+0000 7f59e2d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f492291f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52160/0 (socket says 172.21.15.31:52160) 2024-04-25T16:04:25.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.505+0000 7f59e2d9d640 1 -- 172.21.15.31:0/1258496840 learned_addr learned my addr 172.21.15.31:0/1258496840 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:25.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.505+0000 7f59e2d9d640 1 -- 172.21.15.31:0/1258496840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491ebf730 msgr2=0x55f492291010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:25.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.505+0000 7f59e2d9d640 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491ebf730 0x55f492291010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.505+0000 7f59e2d9d640 1 -- 172.21.15.31:0/1258496840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491eb7550 msgr2=0x55f492290ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:25.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.505+0000 7f59e2d9d640 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491eb7550 0x55f492290ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.505+0000 7f59e2d9d640 1 -- 172.21.15.31:0/1258496840 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f492292500 con 0x55f491ee0410 2024-04-25T16:04:25.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.506+0000 7f59e2d9d640 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f492291f40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f59dc017bd0 tx=0x7f59dc02ba80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8500d0a65d0395f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:25.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.507+0000 7f59c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:25.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.508+0000 7f59c9ffb640 1 -- 172.21.15.31:0/1258496840 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59dc031080 con 0x55f491ee0410 2024-04-25T16:04:25.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.509+0000 7f59c9ffb640 1 -- 172.21.15.31:0/1258496840 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f59dc016770 con 0x55f491ee0410 2024-04-25T16:04:25.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.510+0000 7f59ef767740 1 -- 172.21.15.31:0/1258496840 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59d40c7060 con 0x55f491ee0410 2024-04-25T16:04:25.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.512+0000 7f59c9ffb640 1 -- 172.21.15.31:0/1258496840 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f59dc027030 con 0x55f491ee0410 2024-04-25T16:04:25.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.512+0000 7f59ef767740 10 client.25979 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:25.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.512+0000 7f59ef767740 1 -- 172.21.15.31:0/1258496840 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f4923127b0 con 0x55f491ee0410 2024-04-25T16:04:25.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.513+0000 7f59c9ffb640 1 -- 172.21.15.31:0/1258496840 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59d40c7060 con 0x55f491ee0410 2024-04-25T16:04:25.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.513+0000 7f59c9ffb640 1 -- 172.21.15.31:0/1258496840 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f59dc02a8b0 con 0x55f491ee0410 2024-04-25T16:04:25.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.513+0000 7f59ef767740 10 client.25979 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:25.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.513+0000 7f59ef767740 10 client.25979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:25.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.513+0000 7f59ef767740 20 client.25979 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:25.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.513+0000 7f59ef767740 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f491e74630 0x55f4923151b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:25.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.513+0000 7f59ef767740 4 client.25979 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:04:25.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.513+0000 7f59ef767740 1 -- 172.21.15.31:0/1258496840 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f4921ee580 con 0x55f491e74630 2024-04-25T16:04:25.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.514+0000 7f59e259c640 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f491e74630 0x55f4923151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:25.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.515+0000 7f59e259c640 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f491e74630 0x55f4923151b0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6ad04c277764401 server_cookie=de52b98418dbe257 in_seq=0 out_seq=0 2024-04-25T16:04:25.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.515+0000 7f59c9ffb640 10 client.25979 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.520+0000 7f59c9ffb640 1 -- 172.21.15.31:0/1258496840 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f4921ee580 con 0x55f491e74630 2024-04-25T16:04:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.520+0000 7f59c9ffb640 10 client.25979 handle_command_reply: tid=0 2024-04-25T16:04:25.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.520+0000 7f59ef767740 1 client.25979 shutdown 2024-04-25T16:04:25.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.522+0000 7f59ef767740 1 -- 172.21.15.31:0/1258496840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 msgr2=0x55f492291f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:25.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.522+0000 7f59ef767740 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f492291f40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f59dc017bd0 tx=0x7f59dc02ba80 comp rx=0 tx=0).stop 2024-04-25T16:04:25.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.522+0000 7f59ef767740 1 -- 172.21.15.31:0/1258496840 shutdown_connections 2024-04-25T16:04:25.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.522+0000 7f59ef767740 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f491e74630 0x55f4923151b0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.522+0000 7f59ef767740 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f491ee0410 0x55f492291f40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.523+0000 7f59ef767740 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f491ebf730 0x55f492291010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.523+0000 7f59ef767740 1 --2- 172.21.15.31:0/1258496840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f491eb7550 0x55f492290ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.523+0000 7f59ef767740 1 -- 172.21.15.31:0/1258496840 >> 172.21.15.31:0/1258496840 conn(0x55f491ed4560 msgr2=0x55f49229db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:25.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.523+0000 7f59c9ffb640 0 client.25979 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:25.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.523+0000 7f59ef767740 1 -- 172.21.15.31:0/1258496840 shutdown_connections 2024-04-25T16:04:25.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.523+0000 7f59ef767740 1 -- 172.21.15.31:0/1258496840 wait complete. 2024-04-25T16:04:25.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.524+0000 7f59ef767740 20 client.25979 trim_cache size 0 max 16384 2024-04-25T16:04:25.525 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:04:25.525 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:04:25.525 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "0078e654-b65d-45c3-b910-4dcb224ab0d5", 2024-04-25T16:04:25.525 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:04:25.525 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:04:25.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.536+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59cc0afd00 msgr2=0x7f59cc0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:25.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.536+0000 7f59e359e640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59cc0afd00 0x7f59cc0b2210 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55f492282870 tx=0x7f59dc009490 comp rx=0 tx=0).stop 2024-04-25T16:04:25.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.536+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59e410a350 msgr2=0x7f59e41a8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:25.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.536+0000 7f59e359e640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59e410a350 0x7f59e41a8ec0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f59d4009830 tx=0x7f59d4033e20 comp rx=0 tx=0).stop 2024-04-25T16:04:25.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.539+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 shutdown_connections 2024-04-25T16:04:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.540+0000 7f59e359e640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59cc0afd00 0x7f59cc0b2210 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.540+0000 7f59e359e640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59e410fef0 0x7f59e41b0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.540+0000 7f59e359e640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f59e410b1d0 0x7f59e41a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.540+0000 7f59e359e640 1 --2- 172.21.15.31:0/764424304 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59e410a350 0x7f59e41a8ec0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:25.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.540+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 >> 172.21.15.31:0/764424304 conn(0x7f59e4070400 msgr2=0x7f59e410bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:25.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.546+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 shutdown_connections 2024-04-25T16:04:25.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:25.546+0000 7f59e359e640 1 -- 172.21.15.31:0/764424304 wait complete. 2024-04-25T16:04:25.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:04:25.599 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:04:26.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.122+0000 7f6c47fff640 1 -- 172.21.15.31:0/941799743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c4810a350 msgr2=0x7f6c4810ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:26.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.122+0000 7f6c47fff640 1 --2- 172.21.15.31:0/941799743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c4810a350 0x7f6c4810ac90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6c40009ca0 tx=0x7f6c4002a8b0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.132+0000 7f6c47fff640 1 -- 172.21.15.31:0/941799743 shutdown_connections 2024-04-25T16:04:26.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.132+0000 7f6c47fff640 1 --2- 172.21.15.31:0/941799743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4810fef0 0x7f6c48110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.132+0000 7f6c47fff640 1 --2- 172.21.15.31:0/941799743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c4810b1d0 0x7f6c4810b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.132+0000 7f6c47fff640 1 --2- 172.21.15.31:0/941799743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c4810a350 0x7f6c4810ac90 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.132+0000 7f6c47fff640 1 -- 172.21.15.31:0/941799743 >> 172.21.15.31:0/941799743 conn(0x7f6c48070400 msgr2=0x7f6c48072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:26.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.137+0000 7f6c47fff640 1 -- 172.21.15.31:0/941799743 shutdown_connections 2024-04-25T16:04:26.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.141+0000 7f6c47fff640 1 -- 172.21.15.31:0/941799743 wait complete. 2024-04-25T16:04:26.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.142+0000 7f6c47fff640 1 Processor -- start 2024-04-25T16:04:26.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.144+0000 7f6c47fff640 1 -- start start 2024-04-25T16:04:26.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.144+0000 7f6c47fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c4810a350 0x7f6c481a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.144+0000 7f6c47fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4810b1d0 0x7f6c481a9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.144+0000 7f6c47fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c4810fef0 0x7f6c481b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.145+0000 7f6c47fff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4811e210 con 0x7f6c4810a350 2024-04-25T16:04:26.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.145+0000 7f6c47fff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4811e090 con 0x7f6c4810b1d0 2024-04-25T16:04:26.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.145+0000 7f6c47fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6c4811e390 con 0x7f6c4810fef0 2024-04-25T16:04:26.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.145+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c4810a350 0x7f6c481a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.145+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c4810a350 0x7f6c481a8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53316/0 (socket says 172.21.15.31:53316) 2024-04-25T16:04:26.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.145+0000 7f6c46ffd640 1 -- 172.21.15.31:0/2312834592 learned_addr learned my addr 172.21.15.31:0/2312834592 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:26.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.148+0000 7f6c477fe640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c4810fef0 0x7f6c481b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.148+0000 7f6c467fc640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4810b1d0 0x7f6c481a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.151+0000 7f6c46ffd640 1 -- 172.21.15.31:0/2312834592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c4810fef0 msgr2=0x7f6c481b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:26.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.151+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c4810fef0 0x7f6c481b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.151+0000 7f6c46ffd640 1 -- 172.21.15.31:0/2312834592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4810b1d0 msgr2=0x7f6c481a9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.151+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4810b1d0 0x7f6c481a9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.151+0000 7f6c46ffd640 1 -- 172.21.15.31:0/2312834592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c48111c90 con 0x7f6c4810a350 2024-04-25T16:04:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.152+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c4810a350 0x7f6c481a8e10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6c4002aea0 tx=0x7f6c40003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.170+0000 7f6c2ffff640 1 -- 172.21.15.31:0/2312834592 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c40004450 con 0x7f6c4810a350 2024-04-25T16:04:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.170+0000 7f6c2ffff640 1 -- 172.21.15.31:0/2312834592 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c40040070 con 0x7f6c4810a350 2024-04-25T16:04:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.171+0000 7f6c2ffff640 1 -- 172.21.15.31:0/2312834592 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c40031440 con 0x7f6c4810a350 2024-04-25T16:04:26.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.172+0000 7f6c47fff640 1 -- 172.21.15.31:0/2312834592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c48111f20 con 0x7f6c4810a350 2024-04-25T16:04:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.172+0000 7f6c47fff640 1 -- 172.21.15.31:0/2312834592 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c481123f0 con 0x7f6c4810a350 2024-04-25T16:04:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.180+0000 7f6c54779740 1 Processor -- start 2024-04-25T16:04:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.180+0000 7f6c54779740 1 -- start start 2024-04-25T16:04:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.181+0000 7f6c54779740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc875bf390 0x55fc87960c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.181+0000 7f6c54779740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc879611f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.181+0000 7f6c54779740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc8759e6b0 0x55fc87961e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.181+0000 7f6c54779740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fc878c5d80 con 0x55fc8759e6b0 2024-04-25T16:04:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.181+0000 7f6c54779740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fc8787f8b0 con 0x55fc875964d0 2024-04-25T16:04:26.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.181+0000 7f6c54779740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fc878c1d40 con 0x55fc875bf390 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.181+0000 7f6c2ffff640 1 -- 172.21.15.31:0/2312834592 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6c400045f0 con 0x7f6c4810a350 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.181+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc879611f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.182+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc879611f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52188/0 (socket says 172.21.15.31:52188) 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.182+0000 7f6c46ffd640 1 -- 172.21.15.31:0/586810185 learned_addr learned my addr 172.21.15.31:0/586810185 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.182+0000 7f6c467fc640 1 --2- 172.21.15.31:0/586810185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc8759e6b0 0x55fc87961e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.183+0000 7f6c2ffff640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c300afbd0 0x7f6c300b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.183+0000 7f6c467fc640 1 -- 172.21.15.31:0/586810185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc875bf390 msgr2=0x55fc87960c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.183+0000 7f6c467fc640 1 --2- 172.21.15.31:0/586810185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc875bf390 0x55fc87960c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.183+0000 7f6c467fc640 1 -- 172.21.15.31:0/586810185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 msgr2=0x55fc879611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.183+0000 7f6c467fc640 1 --2- 172.21.15.31:0/586810185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc879611f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.183+0000 7f6c467fc640 1 -- 172.21.15.31:0/586810185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc87850a40 con 0x55fc8759e6b0 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.183+0000 7f6c2ffff640 1 -- 172.21.15.31:0/2312834592 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6c400ffe40 con 0x7f6c4810a350 2024-04-25T16:04:26.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.185+0000 7f6c467fc640 1 --2- 172.21.15.31:0/586810185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc8759e6b0 0x55fc87961e80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6c380058c0 tx=0x7f6c3800e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6635da969a6de417 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.188+0000 7f6c2d7fa640 1 -- 172.21.15.31:0/586810185 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c38005020 con 0x55fc8759e6b0 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.188+0000 7f6c2d7fa640 1 -- 172.21.15.31:0/586810185 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c380047a0 con 0x55fc8759e6b0 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.189+0000 7f6c477fe640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c300afbd0 0x7f6c300b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.190+0000 7f6c54779740 1 -- 172.21.15.31:0/586810185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc8759e6b0 msgr2=0x55fc87961e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.190+0000 7f6c54779740 1 --2- 172.21.15.31:0/586810185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc8759e6b0 0x55fc87961e80 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f6c380058c0 tx=0x7f6c3800e350 comp rx=0 tx=0).stop 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.192+0000 7f6c54779740 1 -- 172.21.15.31:0/586810185 shutdown_connections 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.192+0000 7f6c54779740 1 --2- 172.21.15.31:0/586810185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc8759e6b0 0x55fc87961e80 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.192+0000 7f6c54779740 1 --2- 172.21.15.31:0/586810185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc879611f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.192+0000 7f6c54779740 1 --2- 172.21.15.31:0/586810185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc875bf390 0x55fc87960c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.192+0000 7f6c54779740 1 -- 172.21.15.31:0/586810185 >> 172.21.15.31:0/586810185 conn(0x55fc875b34e0 msgr2=0x55fc8795c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.195+0000 7f6c54779740 1 -- 172.21.15.31:0/586810185 shutdown_connections 2024-04-25T16:04:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.195+0000 7f6c54779740 1 -- 172.21.15.31:0/586810185 wait complete. 2024-04-25T16:04:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.196+0000 7f6c54779740 1 -- start start 2024-04-25T16:04:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.196+0000 7f6c54779740 1 Processor -- start 2024-04-25T16:04:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.197+0000 7f6c54779740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc875964d0 0x55fc8796a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.197+0000 7f6c54779740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc8759e6b0 0x55fc8796ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.197+0000 7f6c54779740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.197+0000 7f6c54779740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fc879629e0 con 0x55fc875bf390 2024-04-25T16:04:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.197+0000 7f6c54779740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fc8784cd10 con 0x55fc8759e6b0 2024-04-25T16:04:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.197+0000 7f6c54779740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fc87962b60 con 0x55fc875964d0 2024-04-25T16:04:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.198+0000 7f6c54779740 10 client.0 _ll_register_callbacks cb 0x55fc87879210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:26.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.198+0000 7f6c477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.198+0000 7f6c477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53344/0 (socket says 172.21.15.31:53344) 2024-04-25T16:04:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.199+0000 7f6c477fe640 1 -- 172.21.15.31:0/795900655 learned_addr learned my addr 172.21.15.31:0/795900655 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:26.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.199+0000 7f6c477fe640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c300afbd0 0x7f6c300b20e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c005fd0 tx=0x7f6c3c005ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.199+0000 7f6c477fe640 1 -- 172.21.15.31:0/795900655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc875964d0 msgr2=0x55fc8796a8d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:04:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.199+0000 7f6c477fe640 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc875964d0 0x55fc8796a8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.200+0000 7f6c477fe640 1 -- 172.21.15.31:0/795900655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc8759e6b0 msgr2=0x55fc8796ae10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:04:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.200+0000 7f6c477fe640 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc8759e6b0 0x55fc8796ae10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.200+0000 7f6c477fe640 1 -- 172.21.15.31:0/795900655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc876a17d0 con 0x55fc875bf390 2024-04-25T16:04:26.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.202+0000 7f6c477fe640 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971d70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c010790 tx=0x7f6c3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1d3ef935b1b871 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:26.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.202+0000 7f6c2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:04:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.203+0000 7f6c2aff5640 1 -- 172.21.15.31:0/795900655 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c020070 con 0x55fc875bf390 2024-04-25T16:04:26.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.204+0000 7f6c54779740 1 -- 172.21.15.31:0/795900655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c380047a0 con 0x55fc875bf390 2024-04-25T16:04:26.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.205+0000 7f6c2aff5640 1 -- 172.21.15.31:0/795900655 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6c3c00fac0 con 0x55fc875bf390 2024-04-25T16:04:26.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.208+0000 7f6c2aff5640 1 -- 172.21.15.31:0/795900655 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6c3c00fd00 con 0x55fc875bf390 2024-04-25T16:04:26.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.210+0000 7f6c54779740 10 client.16275 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:26.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.210+0000 7f6c54779740 1 -- 172.21.15.31:0/795900655 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fc875d73f0 con 0x55fc875bf390 2024-04-25T16:04:26.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.211+0000 7f6c2aff5640 1 -- 172.21.15.31:0/795900655 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c380047a0 con 0x55fc875bf390 2024-04-25T16:04:26.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.214+0000 7f6c2aff5640 1 -- 172.21.15.31:0/795900655 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f6c3c02f420 con 0x55fc875bf390 2024-04-25T16:04:26.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.215+0000 7f6c54779740 10 client.16275 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:26.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.215+0000 7f6c54779740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:26.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.215+0000 7f6c54779740 20 client.16275 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:26.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.215+0000 7f6c54779740 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55fc8757df60 0x55fc879787b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.215+0000 7f6c54779740 4 client.16275 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:26.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.215+0000 7f6c54779740 1 -- 172.21.15.31:0/795900655 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc878d3990 con 0x55fc8757df60 2024-04-25T16:04:26.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.217+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55fc8757df60 0x55fc879787b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.221+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55fc8757df60 0x55fc879787b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33d7a2fad4bc4608 server_cookie=620f393022abc4b4 in_seq=0 out_seq=0 2024-04-25T16:04:26.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.226+0000 7f6c2aff5640 10 client.16275 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:26.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.226+0000 7f6c2aff5640 1 -- 172.21.15.31:0/795900655 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fc878d3990 con 0x55fc8757df60 2024-04-25T16:04:26.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.226+0000 7f6c2aff5640 10 client.16275 handle_command_reply: tid=0 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.226+0000 7f6c54779740 1 client.16275 shutdown 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.227+0000 7f6c54779740 1 -- 172.21.15.31:0/795900655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 msgr2=0x55fc87971d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.227+0000 7f6c54779740 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971d70 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c010790 tx=0x7f6c3c0046c0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.227+0000 7f6c54779740 1 -- 172.21.15.31:0/795900655 shutdown_connections 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.227+0000 7f6c54779740 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55fc8757df60 0x55fc879787b0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.227+0000 7f6c54779740 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971d70 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.227+0000 7f6c54779740 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc8759e6b0 0x55fc8796ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.227+0000 7f6c54779740 1 --2- 172.21.15.31:0/795900655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc875964d0 0x55fc8796a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.227+0000 7f6c54779740 1 -- 172.21.15.31:0/795900655 >> 172.21.15.31:0/795900655 conn(0x55fc875b34e0 msgr2=0x55fc8795b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:26.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.227+0000 7f6c2aff5640 0 client.16275 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.228+0000 7f6c54779740 1 -- 172.21.15.31:0/795900655 shutdown_connections 2024-04-25T16:04:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.228+0000 7f6c54779740 1 -- 172.21.15.31:0/795900655 wait complete. 2024-04-25T16:04:26.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.228+0000 7f6c54779740 20 client.16275 trim_cache size 0 max 16384 2024-04-25T16:04:26.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.286+0000 7f6c54779740 1 Processor -- start 2024-04-25T16:04:26.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.289+0000 7f6c54779740 1 -- start start 2024-04-25T16:04:26.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.290+0000 7f6c54779740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc8795bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.290+0000 7f6c54779740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 0x55fc8795c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.290+0000 7f6c54779740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.290+0000 7f6c54779740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c481c24c0 con 0x55fc875bf390 2024-04-25T16:04:26.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.290+0000 7f6c54779740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6c481c4030 con 0x55fc875964d0 2024-04-25T16:04:26.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.290+0000 7f6c54779740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6c481c2b90 con 0x55fc8759e6b0 2024-04-25T16:04:26.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.294+0000 7f6c467fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 0x55fc8795c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.294+0000 7f6c467fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 0x55fc8795c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37140/0 (socket says 172.21.15.31:37140) 2024-04-25T16:04:26.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.294+0000 7f6c467fc640 1 -- 172.21.15.31:0/693381788 learned_addr learned my addr 172.21.15.31:0/693381788 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:26.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.294+0000 7f6c477fe640 1 --2- 172.21.15.31:0/693381788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.295+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/693381788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc8795bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.298+0000 7f6c46ffd640 1 -- 172.21.15.31:0/693381788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 msgr2=0x55fc8795c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:26.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.298+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/693381788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 0x55fc8795c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.298+0000 7f6c46ffd640 1 -- 172.21.15.31:0/693381788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 msgr2=0x55fc87971c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:26.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.298+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/693381788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.298+0000 7f6c46ffd640 1 -- 172.21.15.31:0/693381788 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc875970f0 con 0x55fc875964d0 2024-04-25T16:04:26.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.302+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/693381788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc8795bd70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6c400c4350 tx=0x7f6c40030890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f44cdd51f37cbc6e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:26.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.303+0000 7f6c2cff9640 1 -- 172.21.15.31:0/693381788 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c400e2070 con 0x55fc875964d0 2024-04-25T16:04:26.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.303+0000 7f6c2cff9640 1 -- 172.21.15.31:0/693381788 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c40036020 con 0x55fc875964d0 2024-04-25T16:04:26.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.305+0000 7f6c54779740 1 -- 172.21.15.31:0/693381788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 msgr2=0x55fc8795bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:26.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.305+0000 7f6c54779740 1 --2- 172.21.15.31:0/693381788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc8795bd70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6c400c4350 tx=0x7f6c40030890 comp rx=0 tx=0).stop 2024-04-25T16:04:26.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.328+0000 7f6c54779740 1 -- 172.21.15.31:0/693381788 shutdown_connections 2024-04-25T16:04:26.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.328+0000 7f6c54779740 1 --2- 172.21.15.31:0/693381788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875bf390 0x55fc87971c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.328+0000 7f6c54779740 1 --2- 172.21.15.31:0/693381788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 0x55fc8795c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.328+0000 7f6c54779740 1 --2- 172.21.15.31:0/693381788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875964d0 0x55fc8795bd70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.328+0000 7f6c54779740 1 -- 172.21.15.31:0/693381788 >> 172.21.15.31:0/693381788 conn(0x55fc875b34e0 msgr2=0x55fc879655b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:26.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.334+0000 7f6c54779740 1 -- 172.21.15.31:0/693381788 shutdown_connections 2024-04-25T16:04:26.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.334+0000 7f6c54779740 1 -- 172.21.15.31:0/693381788 wait complete. 2024-04-25T16:04:26.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.334+0000 7f6c54779740 1 -- start start 2024-04-25T16:04:26.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.335+0000 7f6c54779740 1 Processor -- start 2024-04-25T16:04:26.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.335+0000 7f6c54779740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875964d0 0x55fc8796f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.335+0000 7f6c54779740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 0x55fc8796f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.335+0000 7f6c54779740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875bf390 0x55fc879deb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.335+0000 7f6c54779740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c481c3930 con 0x55fc875964d0 2024-04-25T16:04:26.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.335+0000 7f6c54779740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6c481c3260 con 0x55fc875bf390 2024-04-25T16:04:26.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.335+0000 7f6c54779740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6c48118fd0 con 0x55fc8759e6b0 2024-04-25T16:04:26.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.335+0000 7f6c54779740 10 client.0 _ll_register_callbacks cb 0x7f6c3c014050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:26.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.341+0000 7f6c477fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875bf390 0x55fc879deb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.342+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875964d0 0x55fc8796f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.342+0000 7f6c46ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875964d0 0x55fc8796f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53360/0 (socket says 172.21.15.31:53360) 2024-04-25T16:04:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.342+0000 7f6c46ffd640 1 -- 172.21.15.31:0/1080268104 learned_addr learned my addr 172.21.15.31:0/1080268104 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.342+0000 7f6c467fc640 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 0x55fc8796f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.346+0000 7f6c46ffd640 1 -- 172.21.15.31:0/1080268104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 msgr2=0x55fc8796f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.346+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 0x55fc8796f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.346+0000 7f6c46ffd640 1 -- 172.21.15.31:0/1080268104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875bf390 msgr2=0x55fc879deb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.346+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875bf390 0x55fc879deb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.346+0000 7f6c46ffd640 1 -- 172.21.15.31:0/1080268104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc878864b0 con 0x55fc875964d0 2024-04-25T16:04:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.346+0000 7f6c46ffd640 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875964d0 0x55fc8796f300 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6c400c8260 tx=0x7f6c400c47f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881454ee3d58adec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:26.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.353+0000 7f6c2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:04:26.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.353+0000 7f6c2aff5640 1 -- 172.21.15.31:0/1080268104 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c400dc030 con 0x55fc875964d0 2024-04-25T16:04:26.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.354+0000 7f6c54779740 1 -- 172.21.15.31:0/1080268104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c40036020 con 0x55fc875964d0 2024-04-25T16:04:26.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.356+0000 7f6c2aff5640 1 -- 172.21.15.31:0/1080268104 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6c400c48c0 con 0x55fc875964d0 2024-04-25T16:04:26.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.375+0000 7f6c2aff5640 1 -- 172.21.15.31:0/1080268104 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6c400e2050 con 0x55fc875964d0 2024-04-25T16:04:26.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.379+0000 7f6c54779740 10 client.16281 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:26.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.379+0000 7f6c54779740 1 -- 172.21.15.31:0/1080268104 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fc879dfe40 con 0x55fc875964d0 2024-04-25T16:04:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.389+0000 7f6c2aff5640 1 -- 172.21.15.31:0/1080268104 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c40036020 con 0x55fc875964d0 2024-04-25T16:04:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.391+0000 7f6c2aff5640 1 -- 172.21.15.31:0/1080268104 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f6c4004d060 con 0x55fc875964d0 2024-04-25T16:04:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.391+0000 7f6c54779740 10 client.16281 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.391+0000 7f6c54779740 10 client.16281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.391+0000 7f6c54779740 20 client.16281 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.391+0000 7f6c54779740 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55fc875533a0 0x55fc879e26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.391+0000 7f6c54779740 4 client.16281 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:04:26.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.391+0000 7f6c54779740 1 -- 172.21.15.31:0/1080268104 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fc878cd800 con 0x55fc875533a0 2024-04-25T16:04:26.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.392+0000 7f6c467fc640 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55fc875533a0 0x55fc879e26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:26.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.394+0000 7f6c467fc640 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55fc875533a0 0x55fc879e26f0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de3abe404c5abce server_cookie=36870ae1dd9b868f in_seq=0 out_seq=0 2024-04-25T16:04:26.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.394+0000 7f6c2aff5640 10 client.16281 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:26.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.395+0000 7f6c2aff5640 1 -- 172.21.15.31:0/1080268104 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55fc878cd800 con 0x55fc875533a0 2024-04-25T16:04:26.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.395+0000 7f6c2aff5640 10 client.16281 handle_command_reply: tid=0 2024-04-25T16:04:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.395+0000 7f6c54779740 1 client.16281 shutdown 2024-04-25T16:04:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.396+0000 7f6c54779740 1 -- 172.21.15.31:0/1080268104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875964d0 msgr2=0x55fc8796f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.396+0000 7f6c54779740 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875964d0 0x55fc8796f300 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f6c400c8260 tx=0x7f6c400c47f0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.403 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:04:26.403 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T16:04:26.403 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:04:26.403 INFO:teuthology.orchestra.run.smithi031.stdout: "0078e654-b65d-45c3-b910-4dcb224ab0d5": { 2024-04-25T16:04:26.403 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:04:26.403 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "0078e654-b65d-45c3-b910-4dcb224ab0d5", 2024-04-25T16:04:26.403 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:04:26.403 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:04:26.403 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.399+0000 7f6c54779740 1 -- 172.21.15.31:0/1080268104 shutdown_connections 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.399+0000 7f6c54779740 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55fc875533a0 0x55fc879e26f0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.399+0000 7f6c54779740 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fc875bf390 0x55fc879deb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.399+0000 7f6c54779740 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fc8759e6b0 0x55fc8796f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.399+0000 7f6c54779740 1 --2- 172.21.15.31:0/1080268104 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fc875964d0 0x55fc8796f300 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.399+0000 7f6c54779740 1 -- 172.21.15.31:0/1080268104 >> 172.21.15.31:0/1080268104 conn(0x55fc875b34e0 msgr2=0x55fc8795bd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.400+0000 7f6c2aff5640 0 client.16281 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.400+0000 7f6c54779740 1 -- 172.21.15.31:0/1080268104 shutdown_connections 2024-04-25T16:04:26.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.400+0000 7f6c54779740 1 -- 172.21.15.31:0/1080268104 wait complete. 2024-04-25T16:04:26.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.400+0000 7f6c54779740 20 client.16281 trim_cache size 0 max 16384 2024-04-25T16:04:26.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.412+0000 7f6c47fff640 1 -- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c300afbd0 msgr2=0x7f6c300b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:26.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.412+0000 7f6c47fff640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c300afbd0 0x7f6c300b20e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c005fd0 tx=0x7f6c3c005ce0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.412+0000 7f6c47fff640 1 -- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c4810a350 msgr2=0x7f6c481a8e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:26.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.412+0000 7f6c47fff640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c4810a350 0x7f6c481a8e10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f6c4002aea0 tx=0x7f6c40003c10 comp rx=0 tx=0).stop 2024-04-25T16:04:26.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.417+0000 7f6c47fff640 1 -- 172.21.15.31:0/2312834592 shutdown_connections 2024-04-25T16:04:26.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.417+0000 7f6c47fff640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c300afbd0 0x7f6c300b20e0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.417+0000 7f6c47fff640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c4810fef0 0x7f6c481b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.417+0000 7f6c47fff640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4810b1d0 0x7f6c481a9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.417+0000 7f6c47fff640 1 --2- 172.21.15.31:0/2312834592 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c4810a350 0x7f6c481a8e10 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:26.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.417+0000 7f6c47fff640 1 -- 172.21.15.31:0/2312834592 >> 172.21.15.31:0/2312834592 conn(0x7f6c48070400 msgr2=0x7f6c4810bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:26.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.422+0000 7f6c47fff640 1 -- 172.21.15.31:0/2312834592 shutdown_connections 2024-04-25T16:04:26.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:26.422+0000 7f6c47fff640 1 -- 172.21.15.31:0/2312834592 wait complete. 2024-04-25T16:04:26.528 INFO:tasks.cephfs.filesystem:scrub status for tag:0078e654-b65d-45c3-b910-4dcb224ab0d5 - {'path': '/', 'tag': '0078e654-b65d-45c3-b910-4dcb224ab0d5', 'options': 'recursive,force'} 2024-04-25T16:04:27.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:26 smithi063 ceph-mon[32413]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 761 MiB/s wr, 1.48k op/s 2024-04-25T16:04:27.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:26 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:04:27.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:26 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:04:27.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:26 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:04:27.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:26 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:04:27.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:26 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:04:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:26 smithi031 ceph-mon[29140]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 761 MiB/s wr, 1.48k op/s 2024-04-25T16:04:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:26 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:04:27.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:26 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:04:27.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:26 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:04:27.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:26 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:04:27.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:26 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:04:27.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:26 smithi186 ceph-mon[33258]: pgmap v445: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 761 MiB/s wr, 1.48k op/s 2024-04-25T16:04:27.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:26 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:04:27.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:26 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:04:27.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:26 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:04:27.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:26 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:04:27.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:26 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:04:28.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:27 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:04:28.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:27 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:04:28.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:27 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:04:29.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:28 smithi031 ceph-mon[29140]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 910 MiB/s wr, 1.74k op/s 2024-04-25T16:04:29.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:28 smithi063 ceph-mon[32413]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 910 MiB/s wr, 1.74k op/s 2024-04-25T16:04:29.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:28 smithi186 ceph-mon[33258]: pgmap v446: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 910 MiB/s wr, 1.74k op/s 2024-04-25T16:04:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:30 smithi063 ceph-mon[32413]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 836 MiB/s wr, 1.57k op/s 2024-04-25T16:04:31.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:30 smithi186 ceph-mon[33258]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 836 MiB/s wr, 1.57k op/s 2024-04-25T16:04:31.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:30 smithi031 ceph-mon[29140]: pgmap v447: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 836 MiB/s wr, 1.57k op/s 2024-04-25T16:04:33.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:32 smithi063 ceph-mon[32413]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 810 MiB/s wr, 1.52k op/s 2024-04-25T16:04:33.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:32 smithi186 ceph-mon[33258]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 810 MiB/s wr, 1.52k op/s 2024-04-25T16:04:33.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:32 smithi031 ceph-mon[29140]: pgmap v448: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 810 MiB/s wr, 1.52k op/s 2024-04-25T16:04:34.270 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:33 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:04:34.270 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:33 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:04:34.299 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:33 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:04:34.299 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:33 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:04:34.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:33 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:04:34.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:33 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:04:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:34 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:04:35.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:34 smithi063 ceph-mon[32413]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 983 MiB/s wr, 1.83k op/s 2024-04-25T16:04:35.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:34 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:04:35.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:34 smithi186 ceph-mon[33258]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 983 MiB/s wr, 1.83k op/s 2024-04-25T16:04:35.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:34 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:04:35.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:34 smithi031 ceph-mon[29140]: pgmap v449: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 983 MiB/s wr, 1.83k op/s 2024-04-25T16:04:37.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.021+0000 7ff85d59a640 1 -- 172.21.15.31:0/439931712 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8540308f0 con 0x7ff8580a5940 2024-04-25T16:04:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.022+0000 7ff85f59e640 1 -- 172.21.15.31:0/439931712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8580a5940 msgr2=0x7ff8580a6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:37.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.022+0000 7ff85f59e640 1 --2- 172.21.15.31:0/439931712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8580a5940 0x7ff8580a6500 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff854009860 tx=0x7ff85402a380 comp rx=0 tx=0).stop 2024-04-25T16:04:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.023+0000 7ff85f59e640 1 -- 172.21.15.31:0/439931712 shutdown_connections 2024-04-25T16:04:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.023+0000 7ff85f59e640 1 --2- 172.21.15.31:0/439931712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8580ab760 0x7ff8580abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.023+0000 7ff85f59e640 1 --2- 172.21.15.31:0/439931712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8580a6a40 0x7ff8580a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.023+0000 7ff85f59e640 1 --2- 172.21.15.31:0/439931712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8580a5940 0x7ff8580a6500 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.023+0000 7ff85f59e640 1 -- 172.21.15.31:0/439931712 >> 172.21.15.31:0/439931712 conn(0x7ff8580a09b0 msgr2=0x7ff8580a2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.023+0000 7ff85f59e640 1 -- 172.21.15.31:0/439931712 shutdown_connections 2024-04-25T16:04:37.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.023+0000 7ff85f59e640 1 -- 172.21.15.31:0/439931712 wait complete. 2024-04-25T16:04:37.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.025+0000 7ff85f59e640 1 Processor -- start 2024-04-25T16:04:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85f59e640 1 -- start start 2024-04-25T16:04:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8580a5940 0x7ff858144790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85f59e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8580a6a40 0x7ff858144cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8580ab760 0x7ff85814bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85f59e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff8580b9b80 con 0x7ff8580a5940 2024-04-25T16:04:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85f59e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff8580b9a00 con 0x7ff8580a6a40 2024-04-25T16:04:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff8580b9d00 con 0x7ff8580ab760 2024-04-25T16:04:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8580a5940 0x7ff858144790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8580a5940 0x7ff858144790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58548/0 (socket says 172.21.15.31:58548) 2024-04-25T16:04:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.026+0000 7ff85e59c640 1 -- 172.21.15.31:0/357992789 learned_addr learned my addr 172.21.15.31:0/357992789 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.027+0000 7ff85dd9b640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8580a6a40 0x7ff858144cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.027+0000 7ff85dd9b640 1 -- 172.21.15.31:0/357992789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8580ab760 msgr2=0x7ff85814bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.027+0000 7ff85dd9b640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8580ab760 0x7ff85814bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.027+0000 7ff85dd9b640 1 -- 172.21.15.31:0/357992789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8580a5940 msgr2=0x7ff858144790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.027+0000 7ff85dd9b640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8580a5940 0x7ff858144790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.027+0000 7ff85dd9b640 1 -- 172.21.15.31:0/357992789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8580ad240 con 0x7ff8580a6a40 2024-04-25T16:04:37.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.028+0000 7ff85dd9b640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8580a6a40 0x7ff858144cd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff85000e9f0 tx=0x7ff85000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:37.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.029+0000 7ff84f7fe640 1 -- 172.21.15.31:0/357992789 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff85000ccc0 con 0x7ff8580a6a40 2024-04-25T16:04:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.029+0000 7ff85f59e640 1 -- 172.21.15.31:0/357992789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8580ad530 con 0x7ff8580a6a40 2024-04-25T16:04:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.030+0000 7ff85f59e640 1 -- 172.21.15.31:0/357992789 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8580ada80 con 0x7ff8580a6a40 2024-04-25T16:04:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.030+0000 7ff84f7fe640 1 -- 172.21.15.31:0/357992789 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff8500108e0 con 0x7ff8580a6a40 2024-04-25T16:04:37.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.030+0000 7ff84f7fe640 1 -- 172.21.15.31:0/357992789 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff850018880 con 0x7ff8580a6a40 2024-04-25T16:04:37.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.034+0000 7ff86b824740 1 Processor -- start 2024-04-25T16:04:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.036+0000 7ff86b824740 1 -- start start 2024-04-25T16:04:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.036+0000 7ff86b824740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113ad3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.036+0000 7ff86b824740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411370a2f0 0x564113ad40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.036+0000 7ff86b824740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113adafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.036+0000 7ff86b824740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564113a39460 con 0x56411370a2f0 2024-04-25T16:04:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.036+0000 7ff86b824740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5641139f3590 con 0x5641137331b0 2024-04-25T16:04:37.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.036+0000 7ff86b824740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564113a35420 con 0x5641137124d0 2024-04-25T16:04:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.037+0000 7ff85ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113ad3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.037+0000 7ff85ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113ad3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39826/0 (socket says 172.21.15.31:39826) 2024-04-25T16:04:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.037+0000 7ff85ed9d640 1 -- 172.21.15.31:0/208138922 learned_addr learned my addr 172.21.15.31:0/208138922 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:37.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.037+0000 7ff85e59c640 1 --2- 172.21.15.31:0/208138922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411370a2f0 0x564113ad40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.038+0000 7ff84f7fe640 1 -- 172.21.15.31:0/357992789 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff850010430 con 0x7ff8580a6a40 2024-04-25T16:04:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.039+0000 7ff85ed9d640 1 -- 172.21.15.31:0/208138922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 msgr2=0x564113adafa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:04:37.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.039+0000 7ff85ed9d640 1 --2- 172.21.15.31:0/208138922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113adafa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.039+0000 7ff85ed9d640 1 -- 172.21.15.31:0/208138922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411370a2f0 msgr2=0x564113ad40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.039+0000 7ff85ed9d640 1 --2- 172.21.15.31:0/208138922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411370a2f0 0x564113ad40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.039+0000 7ff85ed9d640 1 -- 172.21.15.31:0/208138922 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564113a000d0 con 0x5641137331b0 2024-04-25T16:04:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.040+0000 7ff85ed9d640 1 --2- 172.21.15.31:0/208138922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113ad3b20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff8600519b0 tx=0x7ff86006aae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e92592ff9f21b5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.040+0000 7ff84f7fe640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff8480afc80 0x7ff8480b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.041+0000 7ff84f7fe640 1 -- 172.21.15.31:0/357992789 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff8500def60 con 0x7ff8580a6a40 2024-04-25T16:04:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.041+0000 7ff84cff9640 1 -- 172.21.15.31:0/208138922 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8600747d0 con 0x5641137331b0 2024-04-25T16:04:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.041+0000 7ff84cff9640 1 -- 172.21.15.31:0/208138922 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff860074e20 con 0x5641137331b0 2024-04-25T16:04:37.042 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.041+0000 7ff84cff9640 1 -- 172.21.15.31:0/208138922 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86007e400 con 0x5641137331b0 2024-04-25T16:04:37.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.042+0000 7ff86b824740 1 -- 172.21.15.31:0/208138922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 msgr2=0x564113ad3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.042+0000 7ff86b824740 1 --2- 172.21.15.31:0/208138922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113ad3b20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff8600519b0 tx=0x7ff86006aae0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.043+0000 7ff86b824740 1 -- 172.21.15.31:0/208138922 shutdown_connections 2024-04-25T16:04:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.043+0000 7ff86b824740 1 --2- 172.21.15.31:0/208138922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113adafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.043+0000 7ff86b824740 1 --2- 172.21.15.31:0/208138922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56411370a2f0 0x564113ad40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.043+0000 7ff86b824740 1 --2- 172.21.15.31:0/208138922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113ad3b20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.043+0000 7ff86b824740 1 -- 172.21.15.31:0/208138922 >> 172.21.15.31:0/208138922 conn(0x564113727300 msgr2=0x564113acf4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:37.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.047+0000 7ff86b824740 1 -- 172.21.15.31:0/208138922 shutdown_connections 2024-04-25T16:04:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.047+0000 7ff86b824740 1 -- 172.21.15.31:0/208138922 wait complete. 2024-04-25T16:04:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.048+0000 7ff86b824740 1 -- start start 2024-04-25T16:04:37.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.048+0000 7ff86b824740 1 Processor -- start 2024-04-25T16:04:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.049+0000 7ff86b824740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113add8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.049+0000 7ff86b824740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 0x564113adde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.049+0000 7ff86b824740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113ae4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.049+0000 7ff86b824740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564113adb9f0 con 0x5641137331b0 2024-04-25T16:04:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.049+0000 7ff86b824740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5641139c0b60 con 0x5641136f1d80 2024-04-25T16:04:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.049+0000 7ff86b824740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564113adbb70 con 0x5641137124d0 2024-04-25T16:04:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.049+0000 7ff86b824740 10 client.0 _ll_register_callbacks cb 0x5641139e52b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:37.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.049+0000 7ff85e59c640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff8480afc80 0x7ff8480b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.050+0000 7ff85dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 0x564113adde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.051+0000 7ff85dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 0x564113adde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58552/0 (socket says 172.21.15.31:58552) 2024-04-25T16:04:37.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.051+0000 7ff85dd9b640 1 -- 172.21.15.31:0/1472110699 learned_addr learned my addr 172.21.15.31:0/1472110699 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.051+0000 7ff85ed9d640 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113ae4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.052+0000 7ff85ed9d640 1 -- 172.21.15.31:0/1472110699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 msgr2=0x564113add8f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:04:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.052+0000 7ff85ed9d640 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113add8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.052+0000 7ff85ed9d640 1 -- 172.21.15.31:0/1472110699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 msgr2=0x564113adde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.052+0000 7ff85ed9d640 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 0x564113adde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.052+0000 7ff85ed9d640 1 -- 172.21.15.31:0/1472110699 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56411374b210 con 0x5641136f1d80 2024-04-25T16:04:37.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.052+0000 7ff85ed9d640 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113ae4d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff860068a60 tx=0x7ff860076730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3a6823c02c90e66 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.055+0000 7ff845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.055+0000 7ff845ffb640 1 -- 172.21.15.31:0/1472110699 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff86006a250 con 0x5641136f1d80 2024-04-25T16:04:37.056 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.056+0000 7ff845ffb640 1 -- 172.21.15.31:0/1472110699 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff86006a3f0 con 0x5641136f1d80 2024-04-25T16:04:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.056+0000 7ff86b824740 1 -- 172.21.15.31:0/1472110699 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff860074e20 con 0x5641136f1d80 2024-04-25T16:04:37.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.056+0000 7ff845ffb640 1 -- 172.21.15.31:0/1472110699 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff8600745d0 con 0x5641136f1d80 2024-04-25T16:04:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.057+0000 7ff86b824740 10 client.26003 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.057+0000 7ff86b824740 1 -- 172.21.15.31:0/1472110699 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564113817620 con 0x5641136f1d80 2024-04-25T16:04:37.058 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.058+0000 7ff845ffb640 1 -- 172.21.15.31:0/1472110699 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff860074e20 con 0x5641136f1d80 2024-04-25T16:04:37.059 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.059+0000 7ff845ffb640 1 -- 172.21.15.31:0/1472110699 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7ff860069c80 con 0x5641136f1d80 2024-04-25T16:04:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.059+0000 7ff86b824740 10 client.26003 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.059+0000 7ff86b824740 10 client.26003 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.059+0000 7ff86b824740 20 client.26003 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.059+0000 7ff86b824740 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641136c71c0 0x564113aeb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.059+0000 7ff86b824740 4 client.26003 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:37.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.059+0000 7ff86b824740 1 -- 172.21.15.31:0/1472110699 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564113a47070 con 0x5641136c71c0 2024-04-25T16:04:37.067 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.067+0000 7ff85e59c640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff8480afc80 0x7ff8480b2190 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x564113ad4f00 tx=0x7ff854005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:37.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.067+0000 7ff85e59c640 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641136c71c0 0x564113aeb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.075+0000 7ff85e59c640 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641136c71c0 0x564113aeb900 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b45476a7d9250aba server_cookie=37dbb7ec14a1aad0 in_seq=0 out_seq=0 2024-04-25T16:04:37.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.075+0000 7ff845ffb640 10 client.26003 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.079+0000 7ff845ffb640 1 -- 172.21.15.31:0/1472110699 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564113a47070 con 0x5641136c71c0 2024-04-25T16:04:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.079+0000 7ff845ffb640 10 client.26003 handle_command_reply: tid=0 2024-04-25T16:04:37.080 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.079+0000 7ff86b824740 1 client.26003 shutdown 2024-04-25T16:04:37.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.080+0000 7ff86b824740 1 -- 172.21.15.31:0/1472110699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 msgr2=0x564113ae4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:37.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.080+0000 7ff86b824740 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113ae4d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff860068a60 tx=0x7ff860076730 comp rx=0 tx=0).stop 2024-04-25T16:04:37.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.083+0000 7ff86b824740 1 -- 172.21.15.31:0/1472110699 shutdown_connections 2024-04-25T16:04:37.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.083+0000 7ff86b824740 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5641136c71c0 0x564113aeb900 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.083+0000 7ff86b824740 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113ae4d90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.083+0000 7ff86b824740 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 0x564113adde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.083+0000 7ff86b824740 1 --2- 172.21.15.31:0/1472110699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113add8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.083+0000 7ff86b824740 1 -- 172.21.15.31:0/1472110699 >> 172.21.15.31:0/1472110699 conn(0x564113727300 msgr2=0x564113acf490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.085+0000 7ff845ffb640 0 client.26003 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:37.086 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.086+0000 7ff86b824740 1 -- 172.21.15.31:0/1472110699 shutdown_connections 2024-04-25T16:04:37.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.086+0000 7ff86b824740 1 -- 172.21.15.31:0/1472110699 wait complete. 2024-04-25T16:04:37.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.090+0000 7ff86b824740 20 client.26003 trim_cache size 0 max 16384 2024-04-25T16:04:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.152+0000 7ff86b824740 1 Processor -- start 2024-04-25T16:04:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.152+0000 7ff86b824740 1 -- start start 2024-04-25T16:04:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.152+0000 7ff86b824740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113adc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.152+0000 7ff86b824740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113ae4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.152+0000 7ff86b824740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 0x564113acdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.152+0000 7ff86b824740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff84000e510 con 0x5641137331b0 2024-04-25T16:04:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.152+0000 7ff86b824740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff840010080 con 0x5641136f1d80 2024-04-25T16:04:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.152+0000 7ff86b824740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff84000ebe0 con 0x5641137124d0 2024-04-25T16:04:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.153+0000 7ff85ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 0x564113acdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.153+0000 7ff85e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113adc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.153+0000 7ff85e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113adc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39846/0 (socket says 172.21.15.31:39846) 2024-04-25T16:04:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.153+0000 7ff85e59c640 1 -- 172.21.15.31:0/1482989397 learned_addr learned my addr 172.21.15.31:0/1482989397 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:37.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.154+0000 7ff85dd9b640 1 --2- 172.21.15.31:0/1482989397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113ae4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.154+0000 7ff85e59c640 1 -- 172.21.15.31:0/1482989397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 msgr2=0x564113ae4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.154+0000 7ff85e59c640 1 --2- 172.21.15.31:0/1482989397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113ae4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.154+0000 7ff85e59c640 1 -- 172.21.15.31:0/1482989397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 msgr2=0x564113acdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.154+0000 7ff85e59c640 1 --2- 172.21.15.31:0/1482989397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 0x564113acdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.154+0000 7ff85e59c640 1 -- 172.21.15.31:0/1482989397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564113ad8740 con 0x5641136f1d80 2024-04-25T16:04:37.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.155+0000 7ff85e59c640 1 --2- 172.21.15.31:0/1482989397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113adc5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff854009830 tx=0x7ff85403c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e41378ffe4b1d16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.157+0000 7ff847fff640 1 -- 172.21.15.31:0/1482989397 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854037040 con 0x5641136f1d80 2024-04-25T16:04:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.157+0000 7ff847fff640 1 -- 172.21.15.31:0/1482989397 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff854042030 con 0x5641136f1d80 2024-04-25T16:04:37.158 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.158+0000 7ff847fff640 1 -- 172.21.15.31:0/1482989397 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854004ae0 con 0x5641136f1d80 2024-04-25T16:04:37.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.158+0000 7ff86b824740 1 -- 172.21.15.31:0/1482989397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 msgr2=0x564113adc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:37.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.158+0000 7ff86b824740 1 --2- 172.21.15.31:0/1482989397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113adc5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff854009830 tx=0x7ff85403c000 comp rx=0 tx=0).stop 2024-04-25T16:04:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.166+0000 7ff86b824740 1 -- 172.21.15.31:0/1482989397 shutdown_connections 2024-04-25T16:04:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.166+0000 7ff86b824740 1 --2- 172.21.15.31:0/1482989397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641137331b0 0x564113acdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.166+0000 7ff86b824740 1 --2- 172.21.15.31:0/1482989397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113ae4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.166+0000 7ff86b824740 1 --2- 172.21.15.31:0/1482989397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641136f1d80 0x564113adc5e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.166+0000 7ff86b824740 1 -- 172.21.15.31:0/1482989397 >> 172.21.15.31:0/1482989397 conn(0x564113727300 msgr2=0x564113add890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:37.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.175+0000 7ff86b824740 1 -- 172.21.15.31:0/1482989397 shutdown_connections 2024-04-25T16:04:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.178+0000 7ff86b824740 1 -- 172.21.15.31:0/1482989397 wait complete. 2024-04-25T16:04:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.178+0000 7ff86b824740 1 -- start start 2024-04-25T16:04:37.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.178+0000 7ff86b824740 1 Processor -- start 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.179+0000 7ff86b824740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641136f1d80 0x564113ae1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.179+0000 7ff86b824740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113ae2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.179+0000 7ff86b824740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113b51c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.179+0000 7ff86b824740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff84000f980 con 0x5641136f1d80 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.179+0000 7ff86b824740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff84000f2b0 con 0x5641137331b0 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.179+0000 7ff86b824740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff84000c4f0 con 0x5641137124d0 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.179+0000 7ff86b824740 10 client.0 _ll_register_callbacks cb 0x7ff860069440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.180+0000 7ff85ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113b51c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.180+0000 7ff85ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113b51c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39848/0 (socket says 172.21.15.31:39848) 2024-04-25T16:04:37.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.180+0000 7ff85ed9d640 1 -- 172.21.15.31:0/1188693484 learned_addr learned my addr 172.21.15.31:0/1188693484 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:37.182 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.180+0000 7ff85dd9b640 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113ae2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.183+0000 7ff85dd9b640 1 -- 172.21.15.31:0/1188693484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 msgr2=0x564113b51c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.183+0000 7ff85dd9b640 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113b51c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.183+0000 7ff85dd9b640 1 -- 172.21.15.31:0/1188693484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641136f1d80 msgr2=0x564113ae1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.183+0000 7ff85dd9b640 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641136f1d80 0x564113ae1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:37.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.183+0000 7ff85dd9b640 1 -- 172.21.15.31:0/1188693484 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564113ae2880 con 0x5641137124d0 2024-04-25T16:04:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.185+0000 7ff85dd9b640 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113ae2240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff85000e9c0 tx=0x7ff850010720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53b9cc6947506f7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.186+0000 7ff845ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:04:37.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.186+0000 7ff845ffb640 1 -- 172.21.15.31:0/1188693484 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8500a7450 con 0x5641137124d0 2024-04-25T16:04:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.186+0000 7ff86b824740 1 -- 172.21.15.31:0/1188693484 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff854042030 con 0x5641137124d0 2024-04-25T16:04:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.186+0000 7ff845ffb640 1 -- 172.21.15.31:0/1188693484 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff8500aed90 con 0x5641137124d0 2024-04-25T16:04:37.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.186+0000 7ff845ffb640 1 -- 172.21.15.31:0/1188693484 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff8500ab800 con 0x5641137124d0 2024-04-25T16:04:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.188+0000 7ff86b824740 10 client.25960 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:37.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.188+0000 7ff86b824740 1 -- 172.21.15.31:0/1188693484 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564113b53320 con 0x5641137124d0 2024-04-25T16:04:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.190+0000 7ff845ffb640 1 -- 172.21.15.31:0/1188693484 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff854042030 con 0x5641137124d0 2024-04-25T16:04:37.191 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.190+0000 7ff845ffb640 1 -- 172.21.15.31:0/1188693484 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7ff850010ca0 con 0x5641137124d0 2024-04-25T16:04:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.190+0000 7ff86b824740 10 client.25960 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.190+0000 7ff86b824740 10 client.25960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.190+0000 7ff86b824740 20 client.25960 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.190+0000 7ff86b824740 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564113b53e20 0x564113b56270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:37.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.190+0000 7ff86b824740 4 client.25960 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:04:37.194 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.190+0000 7ff86b824740 1 -- 172.21.15.31:0/1188693484 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564113a40ee0 con 0x564113b53e20 2024-04-25T16:04:37.197 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.192+0000 7ff85e59c640 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564113b53e20 0x564113b56270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.204+0000 7ff85e59c640 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564113b53e20 0x564113b56270 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc8cc10647799219 server_cookie=742d3d3dbb6c2d56 in_seq=0 out_seq=0 2024-04-25T16:04:37.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:37.204+0000 7ff845ffb640 10 client.25960 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:37.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:37 smithi063 ceph-mon[32413]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 779 MiB/s wr, 1.42k op/s 2024-04-25T16:04:37.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:37 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:04:37.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:37 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:04:37.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:37 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:04:37.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:37 smithi186 ceph-mon[33258]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 779 MiB/s wr, 1.42k op/s 2024-04-25T16:04:37.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:37 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:04:37.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:37 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:04:37.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:37 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:04:37.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:37 smithi031 ceph-mon[29140]: pgmap v450: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 779 MiB/s wr, 1.42k op/s 2024-04-25T16:04:37.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:37 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:04:37.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:37 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:04:37.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:37 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:04:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:39 smithi063 ceph-mon[32413]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 915 MiB/s wr, 1.64k op/s 2024-04-25T16:04:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:39 smithi186 ceph-mon[33258]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 915 MiB/s wr, 1.64k op/s 2024-04-25T16:04:39.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:39 smithi031 ceph-mon[29140]: pgmap v451: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 915 MiB/s wr, 1.64k op/s 2024-04-25T16:04:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.371+0000 7ff845ffb640 1 -- 172.21.15.31:0/1188693484 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564113a40ee0 con 0x564113b53e20 2024-04-25T16:04:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.371+0000 7ff845ffb640 10 client.25960 handle_command_reply: tid=0 2024-04-25T16:04:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.372+0000 7ff86b824740 1 client.25960 shutdown 2024-04-25T16:04:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.373+0000 7ff86b824740 1 -- 172.21.15.31:0/1188693484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 msgr2=0x564113ae2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.373+0000 7ff86b824740 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113ae2240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff85000e9c0 tx=0x7ff850010720 comp rx=0 tx=0).stop 2024-04-25T16:04:40.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.374+0000 7ff86b824740 1 -- 172.21.15.31:0/1188693484 shutdown_connections 2024-04-25T16:04:40.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.374+0000 7ff86b824740 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564113b53e20 0x564113b56270 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.374+0000 7ff86b824740 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641137331b0 0x564113b51c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.374+0000 7ff86b824740 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641137124d0 0x564113ae2240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.374+0000 7ff86b824740 1 --2- 172.21.15.31:0/1188693484 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641136f1d80 0x564113ae1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.374+0000 7ff86b824740 1 -- 172.21.15.31:0/1188693484 >> 172.21.15.31:0/1188693484 conn(0x564113727300 msgr2=0x564113ad22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:40.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.374+0000 7ff845ffb640 0 client.25960 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:40.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.374+0000 7ff86b824740 1 -- 172.21.15.31:0/1188693484 shutdown_connections 2024-04-25T16:04:40.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.374+0000 7ff86b824740 1 -- 172.21.15.31:0/1188693484 wait complete. 2024-04-25T16:04:40.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.375+0000 7ff86b824740 20 client.25960 trim_cache size 0 max 16384 2024-04-25T16:04:40.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:04:40.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:04:40.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:04:40.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:04:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.376+0000 7ff85f59e640 1 -- 172.21.15.31:0/357992789 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff8480afc80 msgr2=0x7ff8480b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.376+0000 7ff85f59e640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff8480afc80 0x7ff8480b2190 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x564113ad4f00 tx=0x7ff854005db0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.376+0000 7ff85f59e640 1 -- 172.21.15.31:0/357992789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8580a6a40 msgr2=0x7ff858144cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.376+0000 7ff85f59e640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8580a6a40 0x7ff858144cd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff85000e9f0 tx=0x7ff85000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.376+0000 7ff85f59e640 1 -- 172.21.15.31:0/357992789 shutdown_connections 2024-04-25T16:04:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.377+0000 7ff85f59e640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff8480afc80 0x7ff8480b2190 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.377+0000 7ff85f59e640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8580ab760 0x7ff85814bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.377+0000 7ff85f59e640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff8580a6a40 0x7ff858144cd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.377+0000 7ff85f59e640 1 --2- 172.21.15.31:0/357992789 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff8580a5940 0x7ff858144790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.377+0000 7ff85f59e640 1 -- 172.21.15.31:0/357992789 >> 172.21.15.31:0/357992789 conn(0x7ff8580a09b0 msgr2=0x7ff8580a7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.378+0000 7ff85f59e640 1 -- 172.21.15.31:0/357992789 shutdown_connections 2024-04-25T16:04:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:40.378+0000 7ff85f59e640 1 -- 172.21.15.31:0/357992789 wait complete. 2024-04-25T16:04:41.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:41 smithi063 ceph-mon[32413]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 782 MiB/s wr, 1.48k op/s 2024-04-25T16:04:41.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:41 smithi186 ceph-mon[33258]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 782 MiB/s wr, 1.48k op/s 2024-04-25T16:04:41.469 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:41 smithi031 ceph-mon[29140]: pgmap v452: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 782 MiB/s wr, 1.48k op/s 2024-04-25T16:04:43.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:43 smithi063 ceph-mon[32413]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 753 MiB/s wr, 1.54k op/s 2024-04-25T16:04:43.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:43 smithi186 ceph-mon[33258]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 753 MiB/s wr, 1.54k op/s 2024-04-25T16:04:43.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:43 smithi031 ceph-mon[29140]: pgmap v453: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 753 MiB/s wr, 1.54k op/s 2024-04-25T16:04:45.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:45 smithi063 ceph-mon[32413]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 952 MiB/s wr, 2.30k op/s 2024-04-25T16:04:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:45 smithi186 ceph-mon[33258]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 952 MiB/s wr, 2.30k op/s 2024-04-25T16:04:45.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:45 smithi031 ceph-mon[29140]: pgmap v454: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 952 MiB/s wr, 2.30k op/s 2024-04-25T16:04:47.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:47 smithi063 ceph-mon[32413]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 743 MiB/s wr, 2.02k op/s 2024-04-25T16:04:47.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:47 smithi186 ceph-mon[33258]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 743 MiB/s wr, 2.02k op/s 2024-04-25T16:04:47.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:47 smithi031 ceph-mon[29140]: pgmap v455: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 743 MiB/s wr, 2.02k op/s 2024-04-25T16:04:49.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:49 smithi031 ceph-mon[29140]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 873 MiB/s wr, 2.52k op/s 2024-04-25T16:04:49.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:49 smithi063 ceph-mon[32413]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 873 MiB/s wr, 2.52k op/s 2024-04-25T16:04:49.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:49 smithi186 ceph-mon[33258]: pgmap v456: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 873 MiB/s wr, 2.52k op/s 2024-04-25T16:04:51.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:51 smithi031 ceph-mon[29140]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 786 MiB/s wr, 2.58k op/s 2024-04-25T16:04:51.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:51 smithi063 ceph-mon[32413]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 786 MiB/s wr, 2.58k op/s 2024-04-25T16:04:51.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:51 smithi186 ceph-mon[33258]: pgmap v457: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 786 MiB/s wr, 2.58k op/s 2024-04-25T16:04:53.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:53 smithi031 ceph-mon[29140]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 755 MiB/s wr, 2.49k op/s 2024-04-25T16:04:53.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:53 smithi063 ceph-mon[32413]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 755 MiB/s wr, 2.49k op/s 2024-04-25T16:04:53.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:53 smithi186 ceph-mon[33258]: pgmap v458: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 755 MiB/s wr, 2.49k op/s 2024-04-25T16:04:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.784+0000 7f1772d52640 1 -- 172.21.15.31:0/2882174859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f176c10b190 msgr2=0x7f176c10b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:53.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.784+0000 7f1772d52640 1 --2- 172.21.15.31:0/2882174859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f176c10b190 0x7f176c10b5a0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f1768009900 tx=0x7f176802a420 comp rx=0 tx=0).stop 2024-04-25T16:04:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.789+0000 7f1772d52640 1 -- 172.21.15.31:0/2882174859 shutdown_connections 2024-04-25T16:04:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.789+0000 7f1772d52640 1 --2- 172.21.15.31:0/2882174859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f176c10feb0 0x7f176c1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.789+0000 7f1772d52640 1 --2- 172.21.15.31:0/2882174859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f176c10b190 0x7f176c10b5a0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.789+0000 7f1772d52640 1 --2- 172.21.15.31:0/2882174859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c10a340 0x7f176c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.789+0000 7f1772d52640 1 -- 172.21.15.31:0/2882174859 >> 172.21.15.31:0/2882174859 conn(0x7f176c070330 msgr2=0x7f176c0727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:53.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.790+0000 7f1772d52640 1 -- 172.21.15.31:0/2882174859 shutdown_connections 2024-04-25T16:04:53.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.793+0000 7f1772d52640 1 -- 172.21.15.31:0/2882174859 wait complete. 2024-04-25T16:04:53.797 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.795+0000 7f1772d52640 1 Processor -- start 2024-04-25T16:04:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.798+0000 7f1772d52640 1 -- start start 2024-04-25T16:04:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.798+0000 7f1772d52640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c10a340 0x7f176c1a8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.798+0000 7f1772d52640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f176c10b190 0x7f176c1a9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.798+0000 7f1772d52640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f176c10feb0 0x7f176c1b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.798+0000 7f1772d52640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f176c11e310 con 0x7f176c10b190 2024-04-25T16:04:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.798+0000 7f1772d52640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f176c11e190 con 0x7f176c10a340 2024-04-25T16:04:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.798+0000 7f1772d52640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f176c11e490 con 0x7f176c10feb0 2024-04-25T16:04:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.799+0000 7f1771d50640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c10a340 0x7f176c1a8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.801 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.799+0000 7f1771d50640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c10a340 0x7f176c1a8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36754/0 (socket says 172.21.15.31:36754) 2024-04-25T16:04:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.799+0000 7f1771d50640 1 -- 172.21.15.31:0/3485451077 learned_addr learned my addr 172.21.15.31:0/3485451077 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.799+0000 7f177154f640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f176c10b190 0x7f176c1a9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.800+0000 7f1771d50640 1 -- 172.21.15.31:0/3485451077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f176c10feb0 msgr2=0x7f176c1b03f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:04:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.800+0000 7f1771d50640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f176c10feb0 0x7f176c1b03f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.800+0000 7f1771d50640 1 -- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f176c10b190 msgr2=0x7f176c1a9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:53.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.800+0000 7f1771d50640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f176c10b190 0x7f176c1a9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.800+0000 7f1771d50640 1 -- 172.21.15.31:0/3485451077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f176c1b0a60 con 0x7f176c10a340 2024-04-25T16:04:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.800+0000 7f1771d50640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c10a340 0x7f176c1a8f20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f175c00c970 tx=0x7f175c00ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:53.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.801+0000 7f1762ffd640 1 -- 172.21.15.31:0/3485451077 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c007bb0 con 0x7f176c10a340 2024-04-25T16:04:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.801+0000 7f1762ffd640 1 -- 172.21.15.31:0/3485451077 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f175c007d50 con 0x7f176c10a340 2024-04-25T16:04:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.801+0000 7f1762ffd640 1 -- 172.21.15.31:0/3485451077 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c00f450 con 0x7f176c10a340 2024-04-25T16:04:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.801+0000 7f1772d52640 1 -- 172.21.15.31:0/3485451077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f176c1b0d50 con 0x7f176c10a340 2024-04-25T16:04:53.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.802+0000 7f1772d52640 1 -- 172.21.15.31:0/3485451077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f176c1b1270 con 0x7f176c10a340 2024-04-25T16:04:53.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.811+0000 7f1762ffd640 1 -- 172.21.15.31:0/3485451077 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f175c00f5f0 con 0x7f176c10a340 2024-04-25T16:04:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.813+0000 7f1762ffd640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17580afcf0 0x7f17580b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.813+0000 7f177ae61740 1 Processor -- start 2024-04-25T16:04:53.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.813+0000 7f177ae61740 1 -- start start 2024-04-25T16:04:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.814+0000 7f177ae61740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab881b0 0x55e01af28b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.814+0000 7f177ae61740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab5f2f0 0x55e01af290a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.814+0000 7f177ae61740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 0x55e01af2ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.814+0000 7f177ae61740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e01ae8e460 con 0x55e01ab881b0 2024-04-25T16:04:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.814+0000 7f177ae61740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e01ae48590 con 0x55e01ab5f2f0 2024-04-25T16:04:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.814+0000 7f177ae61740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e01ae8a420 con 0x55e01ab674d0 2024-04-25T16:04:53.817 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.814+0000 7f177154f640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17580afcf0 0x7f17580b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.815+0000 7f177154f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab5f2f0 0x55e01af290a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.816+0000 7f177154f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab5f2f0 0x55e01af290a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36764/0 (socket says 172.21.15.31:36764) 2024-04-25T16:04:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.816+0000 7f177154f640 1 -- 172.21.15.31:0/276569584 learned_addr learned my addr 172.21.15.31:0/276569584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.816+0000 7f1771d50640 1 --2- 172.21.15.31:0/276569584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab881b0 0x55e01af28b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.816+0000 7f1772551640 1 --2- 172.21.15.31:0/276569584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 0x55e01af2ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.816+0000 7f1762ffd640 1 -- 172.21.15.31:0/3485451077 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f175c0dec60 con 0x7f176c10a340 2024-04-25T16:04:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.817+0000 7f1772551640 1 -- 172.21.15.31:0/276569584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab5f2f0 msgr2=0x55e01af290a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:53.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.817+0000 7f1772551640 1 --2- 172.21.15.31:0/276569584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab5f2f0 0x55e01af290a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.817+0000 7f1772551640 1 -- 172.21.15.31:0/276569584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab881b0 msgr2=0x55e01af28b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.817+0000 7f1772551640 1 --2- 172.21.15.31:0/276569584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab881b0 0x55e01af28b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.817+0000 7f1772551640 1 -- 172.21.15.31:0/276569584 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01ae550d0 con 0x55e01ab674d0 2024-04-25T16:04:53.821 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.819+0000 7f1772551640 1 --2- 172.21.15.31:0/276569584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 0x55e01af2ffa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1764009ce0 tx=0x7f176400b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dee6312239b2509a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.820+0000 7f1757fff640 1 -- 172.21.15.31:0/276569584 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764018070 con 0x55e01ab674d0 2024-04-25T16:04:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.820+0000 7f1757fff640 1 -- 172.21.15.31:0/276569584 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17640119d0 con 0x55e01ab674d0 2024-04-25T16:04:53.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.820+0000 7f1757fff640 1 -- 172.21.15.31:0/276569584 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764013950 con 0x55e01ab674d0 2024-04-25T16:04:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.824+0000 7f177ae61740 1 -- 172.21.15.31:0/276569584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 msgr2=0x55e01af2ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.824+0000 7f177ae61740 1 --2- 172.21.15.31:0/276569584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 0x55e01af2ffa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1764009ce0 tx=0x7f176400b7e0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.825+0000 7f177154f640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17580afcf0 0x7f17580b2200 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1768009230 tx=0x7f17680040f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.825+0000 7f177ae61740 1 -- 172.21.15.31:0/276569584 shutdown_connections 2024-04-25T16:04:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.825+0000 7f177ae61740 1 --2- 172.21.15.31:0/276569584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 0x55e01af2ffa0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.825+0000 7f177ae61740 1 --2- 172.21.15.31:0/276569584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab5f2f0 0x55e01af290a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.825+0000 7f177ae61740 1 --2- 172.21.15.31:0/276569584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab881b0 0x55e01af28b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.825+0000 7f177ae61740 1 -- 172.21.15.31:0/276569584 >> 172.21.15.31:0/276569584 conn(0x55e01ab7c300 msgr2=0x55e01af244b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:53.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.826+0000 7f177ae61740 1 -- 172.21.15.31:0/276569584 shutdown_connections 2024-04-25T16:04:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.826+0000 7f177ae61740 1 -- 172.21.15.31:0/276569584 wait complete. 2024-04-25T16:04:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.827+0000 7f177ae61740 1 -- start start 2024-04-25T16:04:53.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.827+0000 7f177ae61740 1 Processor -- start 2024-04-25T16:04:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f177ae61740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 0x55e01af32880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f177ae61740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 0x55e01af32dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f177ae61740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 0x55e01af39cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f177ae61740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e01af309f0 con 0x55e01ab5f2f0 2024-04-25T16:04:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f177ae61740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e01ae15b60 con 0x55e01ab881b0 2024-04-25T16:04:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f177ae61740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e01af30b70 con 0x55e01ab674d0 2024-04-25T16:04:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f177ae61740 10 client.0 _ll_register_callbacks cb 0x55e01ae3a2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f1771d50640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 0x55e01af32880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f1771d50640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 0x55e01af32880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38616/0 (socket says 172.21.15.31:38616) 2024-04-25T16:04:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.828+0000 7f1771d50640 1 -- 172.21.15.31:0/1872736718 learned_addr learned my addr 172.21.15.31:0/1872736718 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.829+0000 7f1771d50640 1 -- 172.21.15.31:0/1872736718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 msgr2=0x55e01af32dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:04:53.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.829+0000 7f1771d50640 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 0x55e01af32dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.829+0000 7f1771d50640 1 -- 172.21.15.31:0/1872736718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 msgr2=0x55e01af39cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.829+0000 7f1771d50640 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 0x55e01af39cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.829+0000 7f1771d50640 1 -- 172.21.15.31:0/1872736718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01aba0210 con 0x55e01ab5f2f0 2024-04-25T16:04:53.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.830+0000 7f1771d50640 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 0x55e01af32880 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f175c0a5320 tx=0x7f175c072950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedf5cd37914225 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.835+0000 7f17557fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:04:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.835+0000 7f17557fa640 1 -- 172.21.15.31:0/1872736718 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f175c01d020 con 0x55e01ab5f2f0 2024-04-25T16:04:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.835+0000 7f17557fa640 1 -- 172.21.15.31:0/1872736718 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f175c0bcc70 con 0x55e01ab5f2f0 2024-04-25T16:04:53.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.835+0000 7f177ae61740 1 -- 172.21.15.31:0/1872736718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17640119d0 con 0x55e01ab5f2f0 2024-04-25T16:04:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.836+0000 7f17557fa640 1 -- 172.21.15.31:0/1872736718 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f175c0a7bc0 con 0x55e01ab5f2f0 2024-04-25T16:04:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.836+0000 7f177ae61740 10 client.16305 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:53.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.836+0000 7f177ae61740 1 -- 172.21.15.31:0/1872736718 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e01ac6c620 con 0x55e01ab5f2f0 2024-04-25T16:04:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.839+0000 7f17557fa640 1 -- 172.21.15.31:0/1872736718 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17640119d0 con 0x55e01ab5f2f0 2024-04-25T16:04:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.839+0000 7f17557fa640 1 -- 172.21.15.31:0/1872736718 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f175c098ce0 con 0x55e01ab5f2f0 2024-04-25T16:04:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.839+0000 7f177ae61740 10 client.16305 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.839+0000 7f177ae61740 10 client.16305 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:53.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.839+0000 7f177ae61740 20 client.16305 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.839+0000 7f177ae61740 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e01ab46d80 0x55e01af40af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.839+0000 7f177ae61740 4 client.16305 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.839+0000 7f177ae61740 1 -- 172.21.15.31:0/1872736718 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e01ae9c070 con 0x55e01ab46d80 2024-04-25T16:04:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.839+0000 7f177154f640 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e01ab46d80 0x55e01af40af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.840+0000 7f177154f640 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e01ab46d80 0x55e01af40af0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c175089ccf5ad6c server_cookie=f06465255ccb360f in_seq=0 out_seq=0 2024-04-25T16:04:53.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.840+0000 7f17557fa640 10 client.16305 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.843+0000 7f17557fa640 1 -- 172.21.15.31:0/1872736718 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e01ae9c070 con 0x55e01ab46d80 2024-04-25T16:04:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.843+0000 7f17557fa640 10 client.16305 handle_command_reply: tid=0 2024-04-25T16:04:53.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.843+0000 7f177ae61740 1 client.16305 shutdown 2024-04-25T16:04:53.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.844+0000 7f177ae61740 1 -- 172.21.15.31:0/1872736718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 msgr2=0x55e01af32880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:53.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.844+0000 7f177ae61740 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 0x55e01af32880 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f175c0a5320 tx=0x7f175c072950 comp rx=0 tx=0).stop 2024-04-25T16:04:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.846+0000 7f177ae61740 1 -- 172.21.15.31:0/1872736718 shutdown_connections 2024-04-25T16:04:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.846+0000 7f177ae61740 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e01ab46d80 0x55e01af40af0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.846+0000 7f177ae61740 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 0x55e01af39cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.846+0000 7f177ae61740 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab674d0 0x55e01af32dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.846+0000 7f177ae61740 1 --2- 172.21.15.31:0/1872736718 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 0x55e01af32880 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.846+0000 7f177ae61740 1 -- 172.21.15.31:0/1872736718 >> 172.21.15.31:0/1872736718 conn(0x55e01ab7c300 msgr2=0x55e01af273c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.847+0000 7f17557fa640 0 client.16305 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.847+0000 7f177ae61740 1 -- 172.21.15.31:0/1872736718 shutdown_connections 2024-04-25T16:04:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.847+0000 7f177ae61740 1 -- 172.21.15.31:0/1872736718 wait complete. 2024-04-25T16:04:53.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.848+0000 7f177ae61740 20 client.16305 trim_cache size 0 max 16384 2024-04-25T16:04:53.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.893+0000 7f177ae61740 1 Processor -- start 2024-04-25T16:04:53.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.893+0000 7f177ae61740 1 -- start start 2024-04-25T16:04:53.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.894+0000 7f177ae61740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab5f2f0 0x55e01af2cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.894+0000 7f177ae61740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab674d0 0x55e01af2d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.894+0000 7f177ae61740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 0x55e01af452e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f177ae61740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f175000e470 con 0x55e01ab674d0 2024-04-25T16:04:53.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f177ae61740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f175000ffe0 con 0x55e01ab881b0 2024-04-25T16:04:53.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f177ae61740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f175000eb40 con 0x55e01ab5f2f0 2024-04-25T16:04:53.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f1771d50640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab5f2f0 0x55e01af2cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f1771d50640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab5f2f0 0x55e01af2cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:51986/0 (socket says 172.21.15.31:51986) 2024-04-25T16:04:53.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f1771d50640 1 -- 172.21.15.31:0/3987403077 learned_addr learned my addr 172.21.15.31:0/3987403077 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:53.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f1772551640 1 --2- 172.21.15.31:0/3987403077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 0x55e01af452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f177154f640 1 --2- 172.21.15.31:0/3987403077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab674d0 0x55e01af2d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f1772551640 1 -- 172.21.15.31:0/3987403077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab5f2f0 msgr2=0x55e01af2cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:53.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f1772551640 1 --2- 172.21.15.31:0/3987403077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab5f2f0 0x55e01af2cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f1772551640 1 -- 172.21.15.31:0/3987403077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab674d0 msgr2=0x55e01af2d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:53.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f1772551640 1 --2- 172.21.15.31:0/3987403077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab674d0 0x55e01af2d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.895+0000 7f1772551640 1 -- 172.21.15.31:0/3987403077 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e01af33920 con 0x55e01ab881b0 2024-04-25T16:04:53.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.896+0000 7f1772551640 1 --2- 172.21.15.31:0/3987403077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 0x55e01af452e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55e01af16460 tx=0x7f17640098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=936d545f0fc9633a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.905+0000 7f17577fe640 1 -- 172.21.15.31:0/3987403077 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764023070 con 0x55e01ab881b0 2024-04-25T16:04:53.908 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.905+0000 7f17577fe640 1 -- 172.21.15.31:0/3987403077 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1764013520 con 0x55e01ab881b0 2024-04-25T16:04:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.907+0000 7f17577fe640 1 -- 172.21.15.31:0/3987403077 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764018070 con 0x55e01ab881b0 2024-04-25T16:04:53.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.908+0000 7f177ae61740 1 -- 172.21.15.31:0/3987403077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 msgr2=0x55e01af452e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:53.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.908+0000 7f177ae61740 1 --2- 172.21.15.31:0/3987403077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 0x55e01af452e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x55e01af16460 tx=0x7f17640098d0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.914+0000 7f177ae61740 1 -- 172.21.15.31:0/3987403077 shutdown_connections 2024-04-25T16:04:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.914+0000 7f177ae61740 1 --2- 172.21.15.31:0/3987403077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab881b0 0x55e01af452e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.914+0000 7f177ae61740 1 --2- 172.21.15.31:0/3987403077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab674d0 0x55e01af2d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.914+0000 7f177ae61740 1 --2- 172.21.15.31:0/3987403077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab5f2f0 0x55e01af2cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.914+0000 7f177ae61740 1 -- 172.21.15.31:0/3987403077 >> 172.21.15.31:0/3987403077 conn(0x55e01ab7c300 msgr2=0x55e01af32820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.918+0000 7f177ae61740 1 -- 172.21.15.31:0/3987403077 shutdown_connections 2024-04-25T16:04:53.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.918+0000 7f177ae61740 1 -- 172.21.15.31:0/3987403077 wait complete. 2024-04-25T16:04:53.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.918+0000 7f177ae61740 1 -- start start 2024-04-25T16:04:53.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.918+0000 7f177ae61740 1 Processor -- start 2024-04-25T16:04:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.919+0000 7f177ae61740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 0x55e01af36070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.919+0000 7f177ae61740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab674d0 0x55e01af365b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.919+0000 7f177ae61740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab881b0 0x55e01afa6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.919+0000 7f177ae61740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f175000f8e0 con 0x55e01ab5f2f0 2024-04-25T16:04:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.919+0000 7f177ae61740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f175000f210 con 0x55e01ab674d0 2024-04-25T16:04:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.919+0000 7f177ae61740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f175000c450 con 0x55e01ab881b0 2024-04-25T16:04:53.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.920+0000 7f177ae61740 10 client.0 _ll_register_callbacks cb 0x7f175c00a4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:53.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.922+0000 7f1772551640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab881b0 0x55e01afa6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.922+0000 7f1772551640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab881b0 0x55e01afa6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:51988/0 (socket says 172.21.15.31:51988) 2024-04-25T16:04:53.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.922+0000 7f1772551640 1 -- 172.21.15.31:0/1493855334 learned_addr learned my addr 172.21.15.31:0/1493855334 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:53.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.923+0000 7f177154f640 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab674d0 0x55e01af365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.923+0000 7f177154f640 1 -- 172.21.15.31:0/1493855334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab881b0 msgr2=0x55e01afa6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.923+0000 7f177154f640 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab881b0 0x55e01afa6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.923+0000 7f177154f640 1 -- 172.21.15.31:0/1493855334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 msgr2=0x55e01af36070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:53.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.923+0000 7f177154f640 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 0x55e01af36070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:53.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.923+0000 7f177154f640 1 -- 172.21.15.31:0/1493855334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e01af39570 con 0x55e01ab674d0 2024-04-25T16:04:53.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.924+0000 7f177154f640 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab674d0 0x55e01af365b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55e01af29f00 tx=0x7f1768046000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e8ccc7721550481 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.924+0000 7f17557fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.933+0000 7f17557fa640 1 -- 172.21.15.31:0/1493855334 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768031b30 con 0x55e01ab674d0 2024-04-25T16:04:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.933+0000 7f177ae61740 1 -- 172.21.15.31:0/1493855334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1764013520 con 0x55e01ab674d0 2024-04-25T16:04:53.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.934+0000 7f17557fa640 1 -- 172.21.15.31:0/1493855334 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1768031cd0 con 0x55e01ab674d0 2024-04-25T16:04:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.934+0000 7f17557fa640 1 -- 172.21.15.31:0/1493855334 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f17680521a0 con 0x55e01ab674d0 2024-04-25T16:04:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.934+0000 7f177ae61740 10 client.26021 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:53.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.934+0000 7f177ae61740 1 -- 172.21.15.31:0/1493855334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e01afa8400 con 0x55e01ab674d0 2024-04-25T16:04:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.938+0000 7f17557fa640 1 -- 172.21.15.31:0/1493855334 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1764013520 con 0x55e01ab674d0 2024-04-25T16:04:53.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.938+0000 7f17557fa640 1 -- 172.21.15.31:0/1493855334 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f176805b590 con 0x55e01ab674d0 2024-04-25T16:04:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.939+0000 7f177ae61740 10 client.26021 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.939+0000 7f177ae61740 10 client.26021 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.939+0000 7f177ae61740 20 client.26021 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.939+0000 7f177ae61740 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e01ab1c1c0 0x55e01afaadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.939+0000 7f177ae61740 4 client.26021 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:04:53.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.939+0000 7f177ae61740 1 -- 172.21.15.31:0/1493855334 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e01ae95ee0 con 0x55e01ab1c1c0 2024-04-25T16:04:53.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.942+0000 7f1771d50640 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e01ab1c1c0 0x55e01afaadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.950+0000 7f1771d50640 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e01ab1c1c0 0x55e01afaadd0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c5b0e1caa8bfea8 server_cookie=e073e574afeea98b in_seq=0 out_seq=0 2024-04-25T16:04:53.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:53.950+0000 7f17557fa640 10 client.26021 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:55.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:55 smithi031 ceph-mon[29140]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 932 MiB/s wr, 2.80k op/s 2024-04-25T16:04:55.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:55 smithi063 ceph-mon[32413]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 932 MiB/s wr, 2.80k op/s 2024-04-25T16:04:55.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:55 smithi186 ceph-mon[33258]: pgmap v459: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 932 MiB/s wr, 2.80k op/s 2024-04-25T16:04:56.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:04:56.530 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:04:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.990+0000 7f79772d9640 1 -- 172.21.15.31:0/1344889799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 msgr2=0x7f7970110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.990+0000 7f79772d9640 1 --2- 172.21.15.31:0/1344889799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 0x7f7970110440 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f7968009ca0 tx=0x7f796802a8d0 comp rx=0 tx=0).stop 2024-04-25T16:04:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.991+0000 7f79772d9640 1 -- 172.21.15.31:0/1344889799 shutdown_connections 2024-04-25T16:04:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.991+0000 7f79772d9640 1 --2- 172.21.15.31:0/1344889799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 0x7f7970110440 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.991+0000 7f79772d9640 1 --2- 172.21.15.31:0/1344889799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f797010b310 0x7f797010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.991+0000 7f79772d9640 1 --2- 172.21.15.31:0/1344889799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f797010a420 0x7f797010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.991+0000 7f79772d9640 1 -- 172.21.15.31:0/1344889799 >> 172.21.15.31:0/1344889799 conn(0x7f79700704e0 msgr2=0x7f7970072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.991+0000 7f79772d9640 1 -- 172.21.15.31:0/1344889799 shutdown_connections 2024-04-25T16:04:56.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.991+0000 7f79772d9640 1 -- 172.21.15.31:0/1344889799 wait complete. 2024-04-25T16:04:56.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.993+0000 7f79772d9640 1 Processor -- start 2024-04-25T16:04:56.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.993+0000 7f79772d9640 1 -- start start 2024-04-25T16:04:56.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f79772d9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f797010a420 0x7f7970119db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f79772d9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f797010b310 0x7f797011a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f79772d9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 0x7f797011b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f79772d9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7970111f90 con 0x7f7970110030 2024-04-25T16:04:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f79772d9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7970111e10 con 0x7f797010b310 2024-04-25T16:04:56.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f79772d9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7970112110 con 0x7f797010a420 2024-04-25T16:04:56.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f7975876640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 0x7f797011b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:56.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f7975876640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 0x7f797011b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38656/0 (socket says 172.21.15.31:38656) 2024-04-25T16:04:56.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f7975876640 1 -- 172.21.15.31:0/2496373922 learned_addr learned my addr 172.21.15.31:0/2496373922 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:56.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.994+0000 7f7975075640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f797010a420 0x7f7970119db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.995+0000 7f7975876640 1 -- 172.21.15.31:0/2496373922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f797010a420 msgr2=0x7f7970119db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.995+0000 7f7975876640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f797010a420 0x7f7970119db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.995+0000 7f7975876640 1 -- 172.21.15.31:0/2496373922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f797010b310 msgr2=0x7f797011a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.995+0000 7f7975876640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f797010b310 0x7f797011a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.995+0000 7f7975876640 1 -- 172.21.15.31:0/2496373922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f797011b8c0 con 0x7f7970110030 2024-04-25T16:04:56.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.995+0000 7f7975876640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 0x7f797011b250 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7968003f40 tx=0x7f7968009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:56.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.997+0000 7f79667fc640 1 -- 172.21.15.31:0/2496373922 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7968004380 con 0x7f7970110030 2024-04-25T16:04:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.997+0000 7f79667fc640 1 -- 172.21.15.31:0/2496373922 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f796802f620 con 0x7f7970110030 2024-04-25T16:04:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.998+0000 7f79772d9640 1 -- 172.21.15.31:0/2496373922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f797011bb50 con 0x7f7970110030 2024-04-25T16:04:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.998+0000 7f79772d9640 1 -- 172.21.15.31:0/2496373922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f797011c0a0 con 0x7f7970110030 2024-04-25T16:04:57.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:56.998+0000 7f79667fc640 1 -- 172.21.15.31:0/2496373922 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7968030830 con 0x7f7970110030 2024-04-25T16:04:57.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.008+0000 7f79667fc640 1 -- 172.21.15.31:0/2496373922 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7968030a90 con 0x7f7970110030 2024-04-25T16:04:57.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.010+0000 7f79667fc640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795c0afc70 0x7f795c0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:57.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.011+0000 7f797e182740 1 Processor -- start 2024-04-25T16:04:57.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.012+0000 7f797e182740 1 -- start start 2024-04-25T16:04:57.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.012+0000 7f797e182740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed20a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:57.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.013+0000 7f797e182740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed20fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:57.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.013+0000 7f797e182740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed21c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:57.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.013+0000 7f797e182740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ec85d80 con 0x55ae0e97f390 2024-04-25T16:04:57.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.013+0000 7f797e182740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ec3f8b0 con 0x55ae0e95e6b0 2024-04-25T16:04:57.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.013+0000 7f797e182740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ec81d40 con 0x55ae0e9564d0 2024-04-25T16:04:57.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.014+0000 7f79667fc640 1 -- 172.21.15.31:0/2496373922 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7968101ba0 con 0x7f7970110030 2024-04-25T16:04:57.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.014+0000 7f7974874640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed20fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:57.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.014+0000 7f7974874640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed20fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52012/0 (socket says 172.21.15.31:52012) 2024-04-25T16:04:57.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.014+0000 7f7974874640 1 -- 172.21.15.31:0/359562767 learned_addr learned my addr 172.21.15.31:0/359562767 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:57.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.015+0000 7f7974874640 1 -- 172.21.15.31:0/359562767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e95e6b0 msgr2=0x55ae0ed21c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:57.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.015+0000 7f7974874640 1 --2- 172.21.15.31:0/359562767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed21c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:57.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.015+0000 7f7974874640 1 -- 172.21.15.31:0/359562767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 msgr2=0x55ae0ed20a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:57.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.015+0000 7f7974874640 1 --2- 172.21.15.31:0/359562767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed20a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:57.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.015+0000 7f7974874640 1 -- 172.21.15.31:0/359562767 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae0ec10a40 con 0x55ae0e9564d0 2024-04-25T16:04:57.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.015+0000 7f7975075640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795c0afc70 0x7f795c0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.019+0000 7f7975075640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795c0afc70 0x7f795c0b2180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55ae0ed21ab0 tx=0x7f7960005ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:57.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.020+0000 7f7974874640 1 --2- 172.21.15.31:0/359562767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed20fe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f796c0097b0 tx=0x7f796c009c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4e3bdd5b3fcdd22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:57.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.021+0000 7f795b7fe640 1 -- 172.21.15.31:0/359562767 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796c015020 con 0x55ae0e9564d0 2024-04-25T16:04:57.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.021+0000 7f795b7fe640 1 -- 172.21.15.31:0/359562767 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f796c011420 con 0x55ae0e9564d0 2024-04-25T16:04:57.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.021+0000 7f795b7fe640 1 -- 172.21.15.31:0/359562767 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796c0105c0 con 0x55ae0e9564d0 2024-04-25T16:04:57.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.025+0000 7f797e182740 1 -- 172.21.15.31:0/359562767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 msgr2=0x55ae0ed20fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:57.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.025+0000 7f797e182740 1 --2- 172.21.15.31:0/359562767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed20fe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f796c0097b0 tx=0x7f796c009c70 comp rx=0 tx=0).stop 2024-04-25T16:04:57.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.035+0000 7f797e182740 1 -- 172.21.15.31:0/359562767 shutdown_connections 2024-04-25T16:04:57.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.035+0000 7f797e182740 1 --2- 172.21.15.31:0/359562767 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed21c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:57.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.035+0000 7f797e182740 1 --2- 172.21.15.31:0/359562767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed20fe0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.035+0000 7f797e182740 1 --2- 172.21.15.31:0/359562767 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed20a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.035+0000 7f797e182740 1 -- 172.21.15.31:0/359562767 >> 172.21.15.31:0/359562767 conn(0x55ae0e9734e0 msgr2=0x55ae0ed1c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.035+0000 7f797e182740 1 -- 172.21.15.31:0/359562767 shutdown_connections 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.035+0000 7f797e182740 1 -- 172.21.15.31:0/359562767 wait complete. 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.036+0000 7f797e182740 1 -- start start 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.036+0000 7f797e182740 1 Processor -- start 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.038+0000 7f797e182740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.038+0000 7f7975075640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.038+0000 7f7975075640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36820/0 (socket says 172.21.15.31:36820) 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.038+0000 7f7975075640 1 -- 172.21.15.31:0/303990688 learned_addr learned my addr 172.21.15.31:0/303990688 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.038+0000 7f797e182740 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed2af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.039+0000 7f797e182740 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed31e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.039+0000 7f7974874640 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed2af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:57.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.039+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ed22780 con 0x55ae0e97f390 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.039+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ec0cd10 con 0x55ae0e9564d0 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.039+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ae0ed22900 con 0x55ae0e95e6b0 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.040+0000 7f797e182740 10 client.0 _ll_register_callbacks cb 0x55ae0ec39210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.040+0000 7f7975075640 1 -- 172.21.15.31:0/303990688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 msgr2=0x55ae0ed2af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.040+0000 7f7975075640 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed2af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.040+0000 7f7975075640 1 -- 172.21.15.31:0/303990688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 msgr2=0x55ae0ed31e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.040+0000 7f7975075640 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed31e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.040+0000 7f7975075640 1 -- 172.21.15.31:0/303990688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae0ea617d0 con 0x55ae0e9564d0 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.041+0000 7f7975075640 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2a9d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7960004770 tx=0x7f7960005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf05f82045b52c07 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.041+0000 7f7958ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.041+0000 7f7958ff9640 1 -- 172.21.15.31:0/303990688 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796001aa10 con 0x55ae0e9564d0 2024-04-25T16:04:57.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.042+0000 7f7958ff9640 1 -- 172.21.15.31:0/303990688 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f796001abb0 con 0x55ae0e9564d0 2024-04-25T16:04:57.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.042+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f796c011420 con 0x55ae0e9564d0 2024-04-25T16:04:57.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.051+0000 7f7958ff9640 1 -- 172.21.15.31:0/303990688 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f796001adb0 con 0x55ae0e9564d0 2024-04-25T16:04:57.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.051+0000 7f797e182740 10 client.26024 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:57.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.051+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ae0e9973f0 con 0x55ae0e9564d0 2024-04-25T16:04:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.052+0000 7f7958ff9640 1 -- 172.21.15.31:0/303990688 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f796c011420 con 0x55ae0e9564d0 2024-04-25T16:04:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.052+0000 7f7958ff9640 1 -- 172.21.15.31:0/303990688 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f7960017030 con 0x55ae0e9564d0 2024-04-25T16:04:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.052+0000 7f797e182740 10 client.26024 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.052+0000 7f797e182740 10 client.26024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.052+0000 7f797e182740 20 client.26024 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:57.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.052+0000 7f797e182740 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ae0e93df60 0x55ae0ed38580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:57.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.052+0000 7f797e182740 4 client.26024 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:04:57.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.052+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae0ec93990 con 0x55ae0e93df60 2024-04-25T16:04:57.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.053+0000 7f7974874640 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ae0e93df60 0x55ae0ed38580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:57.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.054+0000 7f7974874640 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ae0e93df60 0x55ae0ed38580 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ad8e485bfa2fded server_cookie=1b59478f176d7bb0 in_seq=0 out_seq=0 2024-04-25T16:04:57.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:57.054+0000 7f7958ff9640 10 client.26024 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:57.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:57 smithi063 ceph-mon[32413]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 740 MiB/s wr, 2.10k op/s 2024-04-25T16:04:57.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:57 smithi186 ceph-mon[33258]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 740 MiB/s wr, 2.10k op/s 2024-04-25T16:04:57.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:57 smithi031 ceph-mon[29140]: pgmap v460: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 740 MiB/s wr, 2.10k op/s 2024-04-25T16:04:58.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.843+0000 7f17557fa640 1 -- 172.21.15.31:0/1493855334 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e01ae95ee0 con 0x55e01ab1c1c0 2024-04-25T16:04:58.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.843+0000 7f17557fa640 10 client.26021 handle_command_reply: tid=0 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.843+0000 7f177ae61740 1 client.26021 shutdown 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 -- 172.21.15.31:0/1493855334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab674d0 msgr2=0x55e01af365b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab674d0 0x55e01af365b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x55e01af29f00 tx=0x7f1768046000 comp rx=0 tx=0).stop 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 -- 172.21.15.31:0/1493855334 shutdown_connections 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e01ab1c1c0 0x55e01afaadd0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e01ab881b0 0x55e01afa6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e01ab674d0 0x55e01af365b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 --2- 172.21.15.31:0/1493855334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e01ab5f2f0 0x55e01af36070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 -- 172.21.15.31:0/1493855334 >> 172.21.15.31:0/1493855334 conn(0x55e01ab7c300 msgr2=0x55e01af32820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f17557fa640 0 client.26021 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:58.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 -- 172.21.15.31:0/1493855334 shutdown_connections 2024-04-25T16:04:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 1 -- 172.21.15.31:0/1493855334 wait complete. 2024-04-25T16:04:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.845+0000 7f177ae61740 20 client.26021 trim_cache size 0 max 16384 2024-04-25T16:04:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:04:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:04:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:04:58.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:04:58.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.847+0000 7f7958ff9640 1 -- 172.21.15.31:0/303990688 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ae0ec93990 con 0x55ae0e93df60 2024-04-25T16:04:58.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.847+0000 7f7958ff9640 10 client.26024 handle_command_reply: tid=0 2024-04-25T16:04:58.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.847+0000 7f797e182740 1 client.26024 shutdown 2024-04-25T16:04:58.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.847+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e9564d0 msgr2=0x55ae0ed2a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:58.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.847+0000 7f797e182740 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2a9d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f7960004770 tx=0x7f7960005e40 comp rx=0 tx=0).stop 2024-04-25T16:04:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 -- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17580afcf0 msgr2=0x7f17580b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17580afcf0 0x7f17580b2200 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1768009230 tx=0x7f17680040f0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 -- 172.21.15.31:0/3485451077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c10a340 msgr2=0x7f176c1a8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:58.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c10a340 0x7f176c1a8f20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f175c00c970 tx=0x7f175c00ce30 comp rx=0 tx=0).stop 2024-04-25T16:04:58.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 shutdown_connections 2024-04-25T16:04:58.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f797e182740 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ae0e93df60 0x55ae0ed38580 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f797e182740 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed31e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f797e182740 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed2af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f797e182740 1 --2- 172.21.15.31:0/303990688 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2a9d0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 >> 172.21.15.31:0/303990688 conn(0x55ae0e9734e0 msgr2=0x55ae0ed1f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:58.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f7958ff9640 0 client.26024 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:58.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 -- 172.21.15.31:0/3485451077 shutdown_connections 2024-04-25T16:04:58.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f17580afcf0 0x7f17580b2200 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f176c10feb0 0x7f176c1b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f176c10b190 0x7f176c1a9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 --2- 172.21.15.31:0/3485451077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f176c10a340 0x7f176c1a8f20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.848+0000 7f1772d52640 1 -- 172.21.15.31:0/3485451077 >> 172.21.15.31:0/3485451077 conn(0x7f176c070330 msgr2=0x7f176c10be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:58.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.849+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 shutdown_connections 2024-04-25T16:04:58.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.849+0000 7f797e182740 1 -- 172.21.15.31:0/303990688 wait complete. 2024-04-25T16:04:58.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.849+0000 7f797e182740 20 client.26024 trim_cache size 0 max 16384 2024-04-25T16:04:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.852+0000 7f1772d52640 1 -- 172.21.15.31:0/3485451077 shutdown_connections 2024-04-25T16:04:58.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:04:58.852+0000 7f1772d52640 1 -- 172.21.15.31:0/3485451077 wait complete. 2024-04-25T16:04:58.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.927+0000 7f797e182740 1 Processor -- start 2024-04-25T16:04:58.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.934+0000 7f797e182740 1 -- start start 2024-04-25T16:04:58.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.934+0000 7f797e182740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e9564d0 0x55ae0ed295e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.935+0000 7f797e182740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed31e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.935+0000 7f797e182740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e97f390 0x55ae0ed2acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.935+0000 7f7974874640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed31e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.935+0000 7f7974874640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed31e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52040/0 (socket says 172.21.15.31:52040) 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.935+0000 7f7974874640 1 -- 172.21.15.31:0/2661417680 learned_addr learned my addr 172.21.15.31:0/2661417680 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.935+0000 7f797e182740 1 -- 172.21.15.31:0/2661417680 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f795400e420 con 0x55ae0e9564d0 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.935+0000 7f797e182740 1 -- 172.21.15.31:0/2661417680 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f795400ff90 con 0x55ae0e97f390 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.935+0000 7f797e182740 1 -- 172.21.15.31:0/2661417680 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f795400eaf0 con 0x55ae0e95e6b0 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.936+0000 7f7975075640 1 --2- 172.21.15.31:0/2661417680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e9564d0 0x55ae0ed295e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:58.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.936+0000 7f7975876640 1 --2- 172.21.15.31:0/2661417680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e97f390 0x55ae0ed2acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:58.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.936+0000 7f7974874640 1 -- 172.21.15.31:0/2661417680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e97f390 msgr2=0x55ae0ed2acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:58.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.936+0000 7f7974874640 1 --2- 172.21.15.31:0/2661417680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e97f390 0x55ae0ed2acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.936+0000 7f7974874640 1 -- 172.21.15.31:0/2661417680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e9564d0 msgr2=0x55ae0ed295e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:58.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.936+0000 7f7974874640 1 --2- 172.21.15.31:0/2661417680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e9564d0 0x55ae0ed295e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.936+0000 7f7974874640 1 -- 172.21.15.31:0/2661417680 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae0e9570f0 con 0x55ae0e95e6b0 2024-04-25T16:04:58.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.937+0000 7f7974874640 1 --2- 172.21.15.31:0/2661417680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed31e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f796c000c00 tx=0x7f796c009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70c5bb9b482a895f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:58.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.937+0000 7f795affd640 1 -- 172.21.15.31:0/2661417680 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f796c02b030 con 0x55ae0e95e6b0 2024-04-25T16:04:58.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.937+0000 7f795affd640 1 -- 172.21.15.31:0/2661417680 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f796c011da0 con 0x55ae0e95e6b0 2024-04-25T16:04:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.939+0000 7f797e182740 1 -- 172.21.15.31:0/2661417680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 msgr2=0x55ae0ed31e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.939+0000 7f797e182740 1 --2- 172.21.15.31:0/2661417680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed31e00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f796c000c00 tx=0x7f796c009f70 comp rx=0 tx=0).stop 2024-04-25T16:04:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.945+0000 7f797e182740 1 -- 172.21.15.31:0/2661417680 shutdown_connections 2024-04-25T16:04:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.945+0000 7f797e182740 1 --2- 172.21.15.31:0/2661417680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e97f390 0x55ae0ed2acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.945+0000 7f797e182740 1 --2- 172.21.15.31:0/2661417680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed31e00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.945+0000 7f797e182740 1 --2- 172.21.15.31:0/2661417680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e9564d0 0x55ae0ed295e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.945+0000 7f797e182740 1 -- 172.21.15.31:0/2661417680 >> 172.21.15.31:0/2661417680 conn(0x55ae0e9734e0 msgr2=0x55ae0ed2a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:58.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.951+0000 7f797e182740 1 -- 172.21.15.31:0/2661417680 shutdown_connections 2024-04-25T16:04:58.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.951+0000 7f797e182740 1 -- 172.21.15.31:0/2661417680 wait complete. 2024-04-25T16:04:58.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.952+0000 7f797e182740 1 -- start start 2024-04-25T16:04:58.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.952+0000 7f797e182740 1 Processor -- start 2024-04-25T16:04:58.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.953+0000 7f797e182740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:58.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.953+0000 7f797e182740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed2fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:58.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.954+0000 7f797e182740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed309b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:58.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.954+0000 7f797e182740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f795400f890 con 0x55ae0e97f390 2024-04-25T16:04:58.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.954+0000 7f797e182740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f795400f1c0 con 0x55ae0e95e6b0 2024-04-25T16:04:58.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.954+0000 7f797e182740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f795400c400 con 0x55ae0e9564d0 2024-04-25T16:04:58.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.954+0000 7f797e182740 10 client.0 _ll_register_callbacks cb 0x7f796000b830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:04:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.957+0000 7f7975075640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.957+0000 7f7974874640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed2fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.960+0000 7f7975075640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52054/0 (socket says 172.21.15.31:52054) 2024-04-25T16:04:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.960+0000 7f7975075640 1 -- 172.21.15.31:0/2701272450 learned_addr learned my addr 172.21.15.31:0/2701272450 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:58.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.962+0000 7f7975876640 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:58.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.966+0000 7f7975075640 1 -- 172.21.15.31:0/2701272450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e95e6b0 msgr2=0x55ae0ed2fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:58.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.966+0000 7f7975075640 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed2fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.966+0000 7f7975075640 1 -- 172.21.15.31:0/2701272450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 msgr2=0x55ae0ed309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:04:58.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.966+0000 7f7975075640 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed309b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:58.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.966+0000 7f7975075640 1 -- 172.21.15.31:0/2701272450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae0ec464b0 con 0x55ae0e9564d0 2024-04-25T16:04:58.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.972+0000 7f7975075640 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2f540 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7960004770 tx=0x7f796001b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee1a87b76721e7ea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:58.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.976+0000 7f7958ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:04:58.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.976+0000 7f7958ff9640 1 -- 172.21.15.31:0/2701272450 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7960031040 con 0x55ae0e9564d0 2024-04-25T16:04:58.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.976+0000 7f7958ff9640 1 -- 172.21.15.31:0/2701272450 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f796001b9f0 con 0x55ae0e9564d0 2024-04-25T16:04:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.983+0000 7f797e182740 1 -- 172.21.15.31:0/2701272450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f796c011da0 con 0x55ae0e9564d0 2024-04-25T16:04:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.989+0000 7f7958ff9640 1 -- 172.21.15.31:0/2701272450 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f796001bbf0 con 0x55ae0e9564d0 2024-04-25T16:04:58.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.989+0000 7f797e182740 10 client.25984 fetch_fsmap learned FSMap version 28 2024-04-25T16:04:58.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.989+0000 7f797e182740 1 -- 172.21.15.31:0/2701272450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ae0ed32030 con 0x55ae0e9564d0 2024-04-25T16:04:58.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.990+0000 7f7958ff9640 1 -- 172.21.15.31:0/2701272450 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f796001bd90 con 0x55ae0e9564d0 2024-04-25T16:04:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.990+0000 7f7958ff9640 1 -- 172.21.15.31:0/2701272450 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f7960017070 con 0x55ae0e9564d0 2024-04-25T16:04:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.991+0000 7f797e182740 10 client.25984 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:04:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.991+0000 7f797e182740 10 client.25984 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:04:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.991+0000 7f797e182740 20 client.25984 populate_metadata read hostname 'smithi031' 2024-04-25T16:04:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.991+0000 7f797e182740 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ae0e9133a0 0x55ae0eda2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:58.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.991+0000 7f797e182740 4 client.25984 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:04:58.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.991+0000 7f797e182740 1 -- 172.21.15.31:0/2701272450 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f796c011da0 con 0x55ae0e9133a0 2024-04-25T16:04:58.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:58.992+0000 7f7974874640 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ae0e9133a0 0x55ae0eda2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:59.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.001+0000 7f7958ff9640 10 client.25984 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:59.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.003+0000 7f7974874640 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ae0e9133a0 0x55ae0eda2fd0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f083c158f20afc9 server_cookie=943a7d4c4708c134 in_seq=0 out_seq=0 2024-04-25T16:04:59.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.006+0000 7f7958ff9640 1 -- 172.21.15.31:0/2701272450 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f796c011da0 con 0x55ae0e9133a0 2024-04-25T16:04:59.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.006+0000 7f7958ff9640 10 client.25984 handle_command_reply: tid=0 2024-04-25T16:04:59.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.007+0000 7f797e182740 1 client.25984 shutdown 2024-04-25T16:04:59.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.007+0000 7f797e182740 1 -- 172.21.15.31:0/2701272450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 msgr2=0x55ae0ed2f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:59.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.007+0000 7f797e182740 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2f540 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7960004770 tx=0x7f796001b680 comp rx=0 tx=0).stop 2024-04-25T16:04:59.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.008+0000 7f797e182740 1 -- 172.21.15.31:0/2701272450 shutdown_connections 2024-04-25T16:04:59.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.008+0000 7f797e182740 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ae0e9133a0 0x55ae0eda2fd0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.009+0000 7f7958ff9640 0 client.25984 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:04:59.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.009+0000 7f797e182740 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ae0e97f390 0x55ae0ed309b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.009+0000 7f797e182740 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ae0e95e6b0 0x55ae0ed2fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.009+0000 7f797e182740 1 --2- 172.21.15.31:0/2701272450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ae0e9564d0 0x55ae0ed2f540 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.009+0000 7f797e182740 1 -- 172.21.15.31:0/2701272450 >> 172.21.15.31:0/2701272450 conn(0x55ae0e9734e0 msgr2=0x55ae0ed1f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:59.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.010+0000 7f797e182740 1 -- 172.21.15.31:0/2701272450 shutdown_connections 2024-04-25T16:04:59.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.010+0000 7f797e182740 1 -- 172.21.15.31:0/2701272450 wait complete. 2024-04-25T16:04:59.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.010+0000 7f797e182740 20 client.25984 trim_cache size 0 max 16384 2024-04-25T16:04:59.012 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:04:59.012 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:04:59.013 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:04:59.013 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:04:59.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.035+0000 7f79772d9640 1 -- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795c0afc70 msgr2=0x7f795c0b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.035+0000 7f79772d9640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795c0afc70 0x7f795c0b2180 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55ae0ed21ab0 tx=0x7f7960005ef0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.035+0000 7f79772d9640 1 -- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 msgr2=0x7f797011b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.035+0000 7f79772d9640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 0x7f797011b250 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f7968003f40 tx=0x7f7968009880 comp rx=0 tx=0).stop 2024-04-25T16:04:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.036+0000 7f79772d9640 1 -- 172.21.15.31:0/2496373922 shutdown_connections 2024-04-25T16:04:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.036+0000 7f79772d9640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f795c0afc70 0x7f795c0b2180 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.036+0000 7f79772d9640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7970110030 0x7f797011b250 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.036+0000 7f79772d9640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f797010b310 0x7f797011a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.036+0000 7f79772d9640 1 --2- 172.21.15.31:0/2496373922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f797010a420 0x7f7970119db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.036+0000 7f79772d9640 1 -- 172.21.15.31:0/2496373922 >> 172.21.15.31:0/2496373922 conn(0x7f79700704e0 msgr2=0x7f797010e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:59.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.038+0000 7f79772d9640 1 -- 172.21.15.31:0/2496373922 shutdown_connections 2024-04-25T16:04:59.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.038+0000 7f79772d9640 1 -- 172.21.15.31:0/2496373922 wait complete. 2024-04-25T16:04:59.118 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:04:59.118 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:04:59.455 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:04:59 smithi031 ceph-mon[29140]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 854 MiB/s wr, 2.60k op/s 2024-04-25T16:04:59.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:04:59 smithi063 ceph-mon[32413]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 854 MiB/s wr, 2.60k op/s 2024-04-25T16:04:59.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:04:59 smithi186 ceph-mon[33258]: pgmap v461: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 854 MiB/s wr, 2.60k op/s 2024-04-25T16:04:59.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.668+0000 7fcbe291b640 1 -- 172.21.15.31:0/3282176528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10a340 msgr2=0x7fcbdc10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:59.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.668+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3282176528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10a340 0x7fcbdc10ac50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcbd401acb0 tx=0x7fcbd403b900 comp rx=0 tx=0).stop 2024-04-25T16:04:59.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.674+0000 7fcbe291b640 1 -- 172.21.15.31:0/3282176528 shutdown_connections 2024-04-25T16:04:59.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.674+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3282176528 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbdc10fe90 0x7fcbdc1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.674+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3282176528 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbdc10b190 0x7fcbdc10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.674+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3282176528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10a340 0x7fcbdc10ac50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.674+0000 7fcbe291b640 1 -- 172.21.15.31:0/3282176528 >> 172.21.15.31:0/3282176528 conn(0x7fcbdc070320 msgr2=0x7fcbdc0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:04:59.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.675+0000 7fcbe291b640 1 -- 172.21.15.31:0/3282176528 shutdown_connections 2024-04-25T16:04:59.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.679+0000 7fcbe291b640 1 -- 172.21.15.31:0/3282176528 wait complete. 2024-04-25T16:04:59.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.682+0000 7fcbe291b640 1 Processor -- start 2024-04-25T16:04:59.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.684+0000 7fcbe291b640 1 -- start start 2024-04-25T16:04:59.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.684+0000 7fcbe291b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbdc10a340 0x7fcbdc1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:59.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.685+0000 7fcbe291b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10b190 0x7fcbdc1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:59.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.685+0000 7fcbe1118640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10b190 0x7fcbdc1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:59.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.685+0000 7fcbe1118640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10b190 0x7fcbdc1a0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36860/0 (socket says 172.21.15.31:36860) 2024-04-25T16:04:59.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.685+0000 7fcbe291b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbdc10fe90 0x7fcbdc1a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:59.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.686+0000 7fcbe1118640 1 -- 172.21.15.31:0/3449579584 learned_addr learned my addr 172.21.15.31:0/3449579584 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:04:59.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.686+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc113c10 con 0x7fcbdc10a340 2024-04-25T16:04:59.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.686+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc113a90 con 0x7fcbdc10b190 2024-04-25T16:04:59.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.686+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc113d90 con 0x7fcbdc10fe90 2024-04-25T16:04:59.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.686+0000 7fcbe1919640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbdc10a340 0x7fcbdc1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:59.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.688+0000 7fcbe211a640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbdc10fe90 0x7fcbdc1a7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:59.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.689+0000 7fcbe1919640 1 -- 172.21.15.31:0/3449579584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbdc10fe90 msgr2=0x7fcbdc1a7c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:59.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.689+0000 7fcbe1919640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbdc10fe90 0x7fcbdc1a7c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.689+0000 7fcbe1919640 1 -- 172.21.15.31:0/3449579584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10b190 msgr2=0x7fcbdc1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:04:59.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.689+0000 7fcbe1919640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10b190 0x7fcbdc1a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:04:59.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.689+0000 7fcbe1919640 1 -- 172.21.15.31:0/3449579584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc1a8300 con 0x7fcbdc10a340 2024-04-25T16:04:59.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.689+0000 7fcbe1118640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10b190 0x7fcbdc1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:04:59.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.693+0000 7fcbe1919640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbdc10a340 0x7fcbdc1a07c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc109c10 tx=0x7fcbd4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:59.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.696+0000 7fcbd2ffd640 1 -- 172.21.15.31:0/3449579584 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4045070 con 0x7fcbdc10a340 2024-04-25T16:04:59.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.696+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbdc1a8590 con 0x7fcbdc10a340 2024-04-25T16:04:59.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.696+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbdc1a8b00 con 0x7fcbdc10a340 2024-04-25T16:04:59.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.697+0000 7fcbd2ffd640 1 -- 172.21.15.31:0/3449579584 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcbd4003d90 con 0x7fcbdc10a340 2024-04-25T16:04:59.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.700+0000 7fcbd2ffd640 1 -- 172.21.15.31:0/3449579584 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd40045d0 con 0x7fcbdc10a340 2024-04-25T16:04:59.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.703+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbdc1a8e20 con 0x7fcbdc10a340 2024-04-25T16:04:59.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.705+0000 7fcbd2ffd640 1 -- 172.21.15.31:0/3449579584 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcbd4048020 con 0x7fcbdc10a340 2024-04-25T16:04:59.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.707+0000 7fcbd2ffd640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcbc80afc70 0x7fcbc80b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:04:59.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.712+0000 7fcbd2ffd640 1 -- 172.21.15.31:0/3449579584 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcbd40d5580 con 0x7fcbdc10a340 2024-04-25T16:04:59.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.715+0000 7fcbe1118640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcbc80afc70 0x7fcbc80b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:04:59.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.720+0000 7fcbe1118640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcbc80afc70 0x7fcbc80b2180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc1a1860 tx=0x7fcbcc0091c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:04:59.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:04:59.721+0000 7fcbd2ffd640 1 -- 172.21.15.31:0/3449579584 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcbd40d9690 con 0x7fcbdc10a340 2024-04-25T16:05:00.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.457+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcbdc1a8ff0 con 0x7fcbdc10a340 2024-04-25T16:05:00.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.461+0000 7fcbd2ffd640 1 -- 172.21.15.31:0/3449579584 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11918 (secure 0 0 0) 0x7fcbd40d8f90 con 0x7fcbdc10a340 2024-04-25T16:05:00.463 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:05:00.463 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:02:58.299084+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24553,14631,24563,24535,24521]},"id":1}]} 2024-04-25T16:05:00.463 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 28 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.481+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcbc80afc70 msgr2=0x7fcbc80b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.481+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcbc80afc70 0x7fcbc80b2180 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc1a1860 tx=0x7fcbcc0091c0 comp rx=0 tx=0).stop 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.481+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbdc10a340 msgr2=0x7fcbdc1a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.481+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbdc10a340 0x7fcbdc1a07c0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc109c10 tx=0x7fcbd4003ba0 comp rx=0 tx=0).stop 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.482+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 shutdown_connections 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.482+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcbc80afc70 0x7fcbc80b2180 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.482+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbdc10fe90 0x7fcbdc1a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.482+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcbdc10b190 0x7fcbdc1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.482+0000 7fcbe291b640 1 --2- 172.21.15.31:0/3449579584 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcbdc10a340 0x7fcbdc1a07c0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:00.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.482+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 >> 172.21.15.31:0/3449579584 conn(0x7fcbdc070320 msgr2=0x7fcbdc10de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:00.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.482+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 shutdown_connections 2024-04-25T16:05:00.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:00.482+0000 7fcbe291b640 1 -- 172.21.15.31:0/3449579584 wait complete. 2024-04-25T16:05:00.561 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:05:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.054+0000 7f61c2a3d640 1 -- 172.21.15.31:0/2557934931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 msgr2=0x7f61bc10b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.054+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/2557934931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 0x7f61bc10b5a0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009900 tx=0x7f61ac02a420 comp rx=0 tx=0).stop 2024-04-25T16:05:01.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.059+0000 7f61c2a3d640 1 -- 172.21.15.31:0/2557934931 shutdown_connections 2024-04-25T16:05:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.059+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/2557934931 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61bc10fe90 0x7f61bc1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.059+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/2557934931 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 0x7f61bc10b5a0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.059+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/2557934931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61bc10a340 0x7f61bc10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.059+0000 7f61c2a3d640 1 -- 172.21.15.31:0/2557934931 >> 172.21.15.31:0/2557934931 conn(0x7f61bc070320 msgr2=0x7f61bc0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.066+0000 7f61c2a3d640 1 -- 172.21.15.31:0/2557934931 shutdown_connections 2024-04-25T16:05:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.069+0000 7f61c2a3d640 1 -- 172.21.15.31:0/2557934931 wait complete. 2024-04-25T16:05:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.071+0000 7f61c2a3d640 1 Processor -- start 2024-04-25T16:05:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.073+0000 7f61c2a3d640 1 -- start start 2024-04-25T16:05:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.073+0000 7f61c2a3d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61bc10a340 0x7f61bc11ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.073+0000 7f61c2a3d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 0x7f61bc11a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.073+0000 7f61c2a3d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61bc10fe90 0x7f61bc11a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.073+0000 7f61c2a3d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc112420 con 0x7f61bc10b190 2024-04-25T16:05:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.073+0000 7f61c2a3d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1122a0 con 0x7f61bc10a340 2024-04-25T16:05:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.073+0000 7f61c2a3d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1125a0 con 0x7f61bc10fe90 2024-04-25T16:05:01.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.076+0000 7f61c123a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 0x7f61bc11a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.076+0000 7f61c123a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 0x7f61bc11a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38758/0 (socket says 172.21.15.31:38758) 2024-04-25T16:05:01.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.076+0000 7f61c123a640 1 -- 172.21.15.31:0/3012807533 learned_addr learned my addr 172.21.15.31:0/3012807533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:01.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.076+0000 7f61c223c640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61bc10fe90 0x7f61bc11a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.078+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61bc10a340 0x7f61bc11ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.078+0000 7f61c1a3b640 1 -- 172.21.15.31:0/3012807533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61bc10fe90 msgr2=0x7f61bc11a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.078+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61bc10fe90 0x7f61bc11a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.078+0000 7f61c1a3b640 1 -- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 msgr2=0x7f61bc11a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.078+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 0x7f61bc11a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.078+0000 7f61c1a3b640 1 -- 172.21.15.31:0/3012807533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61bc11ab70 con 0x7f61bc10a340 2024-04-25T16:05:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.078+0000 7f61c123a640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 0x7f61bc11a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:05:01.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.081+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61bc10a340 0x7f61bc11ba20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f61bc074a30 tx=0x7f61b800cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.092+0000 7f61b2ffd640 1 -- 172.21.15.31:0/3012807533 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b8007b50 con 0x7f61bc10a340 2024-04-25T16:05:01.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.093+0000 7f61b2ffd640 1 -- 172.21.15.31:0/3012807533 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61b8007cf0 con 0x7f61bc10a340 2024-04-25T16:05:01.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.093+0000 7f61b2ffd640 1 -- 172.21.15.31:0/3012807533 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b8004d80 con 0x7f61bc10a340 2024-04-25T16:05:01.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.097+0000 7f61c2a3d640 1 -- 172.21.15.31:0/3012807533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61bc11ae60 con 0x7f61bc10a340 2024-04-25T16:05:01.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.097+0000 7f61c2a3d640 1 -- 172.21.15.31:0/3012807533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61bc1b99b0 con 0x7f61bc10a340 2024-04-25T16:05:01.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.102+0000 7f61cac2e740 1 Processor -- start 2024-04-25T16:05:01.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.106+0000 7f61b2ffd640 1 -- 172.21.15.31:0/3012807533 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f61b8004f20 con 0x7f61bc10a340 2024-04-25T16:05:01.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.108+0000 7f61b2ffd640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61a80afcf0 0x7f61a80b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.108+0000 7f61b2ffd640 1 -- 172.21.15.31:0/3012807533 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f61b80dea80 con 0x7f61bc10a340 2024-04-25T16:05:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.109+0000 7f61cac2e740 1 -- start start 2024-04-25T16:05:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.110+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53f9e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.110+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bd34d0 0x562a53f9ef60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.110+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bdb6b0 0x562a53f9fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.110+0000 7f61cac2e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562a53f02d80 con 0x562a53bdb6b0 2024-04-25T16:05:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.110+0000 7f61cac2e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562a53ebca20 con 0x562a53bfc390 2024-04-25T16:05:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.110+0000 7f61cac2e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562a53efed40 con 0x562a53bd34d0 2024-04-25T16:05:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.111+0000 7f61c1a3b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53f9e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.111+0000 7f61c1a3b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53f9e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36898/0 (socket says 172.21.15.31:36898) 2024-04-25T16:05:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.111+0000 7f61c1a3b640 1 -- 172.21.15.31:0/3853542152 learned_addr learned my addr 172.21.15.31:0/3853542152 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.112+0000 7f61c1a3b640 1 -- 172.21.15.31:0/3853542152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bd34d0 msgr2=0x562a53f9ef60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.112+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/3853542152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bd34d0 0x562a53f9ef60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.112+0000 7f61c1a3b640 1 -- 172.21.15.31:0/3853542152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bdb6b0 msgr2=0x562a53f9fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.112+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/3853542152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bdb6b0 0x562a53f9fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.112+0000 7f61c1a3b640 1 -- 172.21.15.31:0/3853542152 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a53ec7c30 con 0x562a53bfc390 2024-04-25T16:05:01.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.113+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/3853542152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53f9e9e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f61b80058c0 tx=0x7f61b800a470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f11f8752d62a32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.113+0000 7f61a7fff640 1 -- 172.21.15.31:0/3853542152 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b801d040 con 0x562a53bfc390 2024-04-25T16:05:01.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.114+0000 7f61a7fff640 1 -- 172.21.15.31:0/3853542152 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61b80ab2c0 con 0x562a53bfc390 2024-04-25T16:05:01.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.114+0000 7f61a7fff640 1 -- 172.21.15.31:0/3853542152 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b8005470 con 0x562a53bfc390 2024-04-25T16:05:01.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.118+0000 7f61c223c640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61a80afcf0 0x7f61a80b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.119+0000 7f61cac2e740 1 -- 172.21.15.31:0/3853542152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 msgr2=0x562a53f9e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.119+0000 7f61cac2e740 1 --2- 172.21.15.31:0/3853542152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53f9e9e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f61b80058c0 tx=0x7f61b800a470 comp rx=0 tx=0).stop 2024-04-25T16:05:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.121+0000 7f61cac2e740 1 -- 172.21.15.31:0/3853542152 shutdown_connections 2024-04-25T16:05:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.121+0000 7f61cac2e740 1 --2- 172.21.15.31:0/3853542152 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bdb6b0 0x562a53f9fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.121+0000 7f61cac2e740 1 --2- 172.21.15.31:0/3853542152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bd34d0 0x562a53f9ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.121+0000 7f61cac2e740 1 --2- 172.21.15.31:0/3853542152 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53f9e9e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.121+0000 7f61cac2e740 1 -- 172.21.15.31:0/3853542152 >> 172.21.15.31:0/3853542152 conn(0x562a53bf04e0 msgr2=0x562a53f9a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:01.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.123+0000 7f61cac2e740 1 -- 172.21.15.31:0/3853542152 shutdown_connections 2024-04-25T16:05:01.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.123+0000 7f61c223c640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61a80afcf0 0x7f61a80b2200 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f61b40098b0 tx=0x7f61b4009840 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.123+0000 7f61cac2e740 1 -- 172.21.15.31:0/3853542152 wait complete. 2024-04-25T16:05:01.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.124+0000 7f61cac2e740 1 -- start start 2024-04-25T16:05:01.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.124+0000 7f61cac2e740 1 Processor -- start 2024-04-25T16:05:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.125+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.125+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 0x562a53fa8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.125+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53fafc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.125+0000 7f61cac2e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562a53fa0700 con 0x562a53bd34d0 2024-04-25T16:05:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.125+0000 7f61cac2e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562a53e89dc0 con 0x562a53bfc390 2024-04-25T16:05:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.125+0000 7f61cac2e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562a53fa0880 con 0x562a53bdb6b0 2024-04-25T16:05:01.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.125+0000 7f61cac2e740 10 client.0 _ll_register_callbacks cb 0x562a53ea5600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:01.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.125+0000 7f61c1a3b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.126+0000 7f61c1a3b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38764/0 (socket says 172.21.15.31:38764) 2024-04-25T16:05:01.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.126+0000 7f61c1a3b640 1 -- 172.21.15.31:0/664128930 learned_addr learned my addr 172.21.15.31:0/664128930 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:01.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.126+0000 7f61c1a3b640 1 -- 172.21.15.31:0/664128930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 msgr2=0x562a53fa8cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:01.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.126+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 0x562a53fa8cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.127+0000 7f61c1a3b640 1 -- 172.21.15.31:0/664128930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 msgr2=0x562a53fafc30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:01.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.127+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53fafc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.127+0000 7f61c1a3b640 1 -- 172.21.15.31:0/664128930 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a53c143f0 con 0x562a53bd34d0 2024-04-25T16:05:01.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.127+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa8790 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f61b8007f50 tx=0x7f61b80a3480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5d4b2fae7368d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.127+0000 7f61a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:01.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.128+0000 7f61a57fa640 1 -- 172.21.15.31:0/664128930 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b80c5070 con 0x562a53bd34d0 2024-04-25T16:05:01.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.128+0000 7f61cac2e740 1 -- 172.21.15.31:0/664128930 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61b80ab2c0 con 0x562a53bd34d0 2024-04-25T16:05:01.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.128+0000 7f61a57fa640 1 -- 172.21.15.31:0/664128930 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f61b8005420 con 0x562a53bd34d0 2024-04-25T16:05:01.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.129+0000 7f61a57fa640 1 -- 172.21.15.31:0/664128930 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f61b801d040 con 0x562a53bd34d0 2024-04-25T16:05:01.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.129+0000 7f61cac2e740 10 client.16332 fetch_fsmap learned FSMap version 28 2024-04-25T16:05:01.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.129+0000 7f61cac2e740 1 -- 172.21.15.31:0/664128930 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a53cde7d0 con 0x562a53bd34d0 2024-04-25T16:05:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.130+0000 7f61a57fa640 1 -- 172.21.15.31:0/664128930 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61b80ab2c0 con 0x562a53bd34d0 2024-04-25T16:05:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.130+0000 7f61a57fa640 1 -- 172.21.15.31:0/664128930 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f61b80a5e90 con 0x562a53bd34d0 2024-04-25T16:05:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.130+0000 7f61cac2e740 10 client.16332 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:05:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.130+0000 7f61cac2e740 10 client.16332 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:05:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.130+0000 7f61cac2e740 20 client.16332 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.130+0000 7f61cac2e740 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x562a53bbaf60 0x562a53fb65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.130+0000 7f61cac2e740 4 client.16332 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:01.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.130+0000 7f61cac2e740 1 -- 172.21.15.31:0/664128930 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a53f10990 con 0x562a53bbaf60 2024-04-25T16:05:01.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.135+0000 7f61c123a640 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x562a53bbaf60 0x562a53fb65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.136+0000 7f61c123a640 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x562a53bbaf60 0x562a53fb65a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=98db2203d1c2497e server_cookie=fa0fac12cd994921 in_seq=0 out_seq=0 2024-04-25T16:05:01.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.136+0000 7f61a57fa640 10 client.16332 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:05:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.137+0000 7f61a57fa640 1 -- 172.21.15.31:0/664128930 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562a53f10990 con 0x562a53bbaf60 2024-04-25T16:05:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.137+0000 7f61a57fa640 10 client.16332 handle_command_reply: tid=0 2024-04-25T16:05:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.138+0000 7f61cac2e740 1 client.16332 shutdown 2024-04-25T16:05:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.138+0000 7f61cac2e740 1 -- 172.21.15.31:0/664128930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 msgr2=0x562a53fa8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.138+0000 7f61cac2e740 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa8790 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f61b8007f50 tx=0x7f61b80a3480 comp rx=0 tx=0).stop 2024-04-25T16:05:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.138+0000 7f61cac2e740 1 -- 172.21.15.31:0/664128930 shutdown_connections 2024-04-25T16:05:01.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.138+0000 7f61cac2e740 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x562a53bbaf60 0x562a53fb65a0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.138+0000 7f61cac2e740 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53fafc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.138+0000 7f61cac2e740 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 0x562a53fa8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.138+0000 7f61cac2e740 1 --2- 172.21.15.31:0/664128930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa8790 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.138+0000 7f61cac2e740 1 -- 172.21.15.31:0/664128930 >> 172.21.15.31:0/664128930 conn(0x562a53bf04e0 msgr2=0x562a53f993f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.139+0000 7f61a57fa640 0 client.16332 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:05:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.139+0000 7f61cac2e740 1 -- 172.21.15.31:0/664128930 shutdown_connections 2024-04-25T16:05:01.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.139+0000 7f61cac2e740 1 -- 172.21.15.31:0/664128930 wait complete. 2024-04-25T16:05:01.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.139+0000 7f61cac2e740 20 client.16332 trim_cache size 0 max 16384 2024-04-25T16:05:01.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.184+0000 7f61cac2e740 1 Processor -- start 2024-04-25T16:05:01.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.184+0000 7f61cac2e740 1 -- start start 2024-04-25T16:05:01.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.185+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.186+0000 7f61c1a3b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.186+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bdb6b0 0x562a53fa3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.186+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bfc390 0x562a53fa6e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.186+0000 7f61cac2e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1c28a0 con 0x562a53bd34d0 2024-04-25T16:05:01.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.186+0000 7f61cac2e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1c4410 con 0x562a53bdb6b0 2024-04-25T16:05:01.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.186+0000 7f61cac2e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1c2f70 con 0x562a53bfc390 2024-04-25T16:05:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.187+0000 7f61c1a3b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38776/0 (socket says 172.21.15.31:38776) 2024-04-25T16:05:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.187+0000 7f61c1a3b640 1 -- 172.21.15.31:0/2858826379 learned_addr learned my addr 172.21.15.31:0/2858826379 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:01.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.188+0000 7f61c223c640 1 --2- 172.21.15.31:0/2858826379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bfc390 0x562a53fa6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.189+0000 7f61c223c640 1 -- 172.21.15.31:0/2858826379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bdb6b0 msgr2=0x562a53fa3a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:05:01.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.189+0000 7f61c223c640 1 --2- 172.21.15.31:0/2858826379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bdb6b0 0x562a53fa3a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.190+0000 7f61c223c640 1 -- 172.21.15.31:0/2858826379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 msgr2=0x562a53fa3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:01.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.190+0000 7f61c223c640 1 --2- 172.21.15.31:0/2858826379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.190+0000 7f61c223c640 1 -- 172.21.15.31:0/2858826379 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a53f8bba0 con 0x562a53bfc390 2024-04-25T16:05:01.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.190+0000 7f61c223c640 1 --2- 172.21.15.31:0/2858826379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bfc390 0x562a53fa6e80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f61b40099d0 tx=0x7f61b4013000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47589820f5762c52 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.191+0000 7f61a77fe640 1 -- 172.21.15.31:0/2858826379 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4022070 con 0x562a53bfc390 2024-04-25T16:05:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.192+0000 7f61a77fe640 1 -- 172.21.15.31:0/2858826379 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61b4012900 con 0x562a53bfc390 2024-04-25T16:05:01.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.192+0000 7f61a77fe640 1 -- 172.21.15.31:0/2858826379 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b4016070 con 0x562a53bfc390 2024-04-25T16:05:01.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.195+0000 7f61cac2e740 1 -- 172.21.15.31:0/2858826379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bfc390 msgr2=0x562a53fa6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.195+0000 7f61cac2e740 1 --2- 172.21.15.31:0/2858826379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bfc390 0x562a53fa6e80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f61b40099d0 tx=0x7f61b4013000 comp rx=0 tx=0).stop 2024-04-25T16:05:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.195+0000 7f61cac2e740 1 -- 172.21.15.31:0/2858826379 shutdown_connections 2024-04-25T16:05:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.195+0000 7f61cac2e740 1 --2- 172.21.15.31:0/2858826379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bfc390 0x562a53fa6e80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.195+0000 7f61cac2e740 1 --2- 172.21.15.31:0/2858826379 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bdb6b0 0x562a53fa3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.195+0000 7f61cac2e740 1 --2- 172.21.15.31:0/2858826379 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fa3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.196+0000 7f61cac2e740 1 -- 172.21.15.31:0/2858826379 >> 172.21.15.31:0/2858826379 conn(0x562a53bf04e0 msgr2=0x562a53f9e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:01.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.200+0000 7f61cac2e740 1 -- 172.21.15.31:0/2858826379 shutdown_connections 2024-04-25T16:05:01.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.200+0000 7f61cac2e740 1 -- 172.21.15.31:0/2858826379 wait complete. 2024-04-25T16:05:01.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.200+0000 7f61cac2e740 1 -- start start 2024-04-25T16:05:01.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.201+0000 7f61cac2e740 1 Processor -- start 2024-04-25T16:05:01.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fab5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 0x562a53fabb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61cac2e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53fae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61cac2e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1c3d10 con 0x562a53bd34d0 2024-04-25T16:05:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61cac2e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1c3640 con 0x562a53bfc390 2024-04-25T16:05:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61cac2e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61bc1c0880 con 0x562a53bdb6b0 2024-04-25T16:05:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61cac2e740 10 client.0 _ll_register_callbacks cb 0x7f61b80a67c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61c123a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 0x562a53fabb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61c123a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 0x562a53fabb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52136/0 (socket says 172.21.15.31:52136) 2024-04-25T16:05:01.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.202+0000 7f61c123a640 1 -- 172.21.15.31:0/4120435219 learned_addr learned my addr 172.21.15.31:0/4120435219 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:01.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.203+0000 7f61c223c640 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53fae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.203+0000 7f61c123a640 1 -- 172.21.15.31:0/4120435219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 msgr2=0x562a53fae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.203+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fab5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.203+0000 7f61c123a640 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53fae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.203+0000 7f61c123a640 1 -- 172.21.15.31:0/4120435219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 msgr2=0x562a53fab5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.203+0000 7f61c123a640 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fab5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.203+0000 7f61c123a640 1 -- 172.21.15.31:0/4120435219 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a53faef90 con 0x562a53bdb6b0 2024-04-25T16:05:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.204+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:05:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.204+0000 7f61c123a640 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 0x562a53fabb00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f61ac031470 tx=0x7f61ac005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45d217d530ee30c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.204+0000 7f61a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.206+0000 7f61a57fa640 1 -- 172.21.15.31:0/4120435219 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac048020 con 0x562a53bdb6b0 2024-04-25T16:05:01.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.206+0000 7f61a57fa640 1 -- 172.21.15.31:0/4120435219 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f61ac004960 con 0x562a53bdb6b0 2024-04-25T16:05:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.208+0000 7f61cac2e740 1 -- 172.21.15.31:0/4120435219 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61b4012900 con 0x562a53bdb6b0 2024-04-25T16:05:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.213+0000 7f61a57fa640 1 -- 172.21.15.31:0/4120435219 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f61ac035040 con 0x562a53bdb6b0 2024-04-25T16:05:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.213+0000 7f61cac2e740 10 client.25996 fetch_fsmap learned FSMap version 28 2024-04-25T16:05:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.213+0000 7f61cac2e740 1 -- 172.21.15.31:0/4120435219 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x562a5401d420 con 0x562a53bdb6b0 2024-04-25T16:05:01.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.214+0000 7f61a57fa640 1 -- 172.21.15.31:0/4120435219 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61b4012900 con 0x562a53bdb6b0 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.214+0000 7f61a57fa640 1 -- 172.21.15.31:0/4120435219 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f61ac055020 con 0x562a53bdb6b0 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.214+0000 7f61cac2e740 10 client.25996 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.214+0000 7f61cac2e740 10 client.25996 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.214+0000 7f61cac2e740 20 client.25996 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.214+0000 7f61cac2e740 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x562a53b903a0 0x562a5401fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.214+0000 7f61cac2e740 4 client.25996 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.214+0000 7f61cac2e740 1 -- 172.21.15.31:0/4120435219 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a53f0a800 con 0x562a53b903a0 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.215+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x562a53b903a0 0x562a5401fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.216+0000 7f61c1a3b640 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x562a53b903a0 0x562a5401fe20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bb45e413e5ecf711 server_cookie=e7e4dbd824aad470 in_seq=0 out_seq=0 2024-04-25T16:05:01.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.216+0000 7f61a57fa640 10 client.25996 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:05:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.216+0000 7f61a57fa640 1 -- 172.21.15.31:0/4120435219 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a53f0a800 con 0x562a53b903a0 2024-04-25T16:05:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.216+0000 7f61a57fa640 10 client.25996 handle_command_reply: tid=0 2024-04-25T16:05:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.216+0000 7f61cac2e740 1 client.25996 shutdown 2024-04-25T16:05:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.217+0000 7f61cac2e740 1 -- 172.21.15.31:0/4120435219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 msgr2=0x562a53fabb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.217+0000 7f61cac2e740 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 0x562a53fabb00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f61ac031470 tx=0x7f61ac005270 comp rx=0 tx=0).stop 2024-04-25T16:05:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.217+0000 7f61cac2e740 1 -- 172.21.15.31:0/4120435219 shutdown_connections 2024-04-25T16:05:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.217+0000 7f61cac2e740 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x562a53b903a0 0x562a5401fe20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.217+0000 7f61cac2e740 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562a53bfc390 0x562a53fae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.217+0000 7f61cac2e740 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562a53bdb6b0 0x562a53fabb00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.217+0000 7f61cac2e740 1 --2- 172.21.15.31:0/4120435219 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562a53bd34d0 0x562a53fab5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.217+0000 7f61cac2e740 1 -- 172.21.15.31:0/4120435219 >> 172.21.15.31:0/4120435219 conn(0x562a53bf04e0 msgr2=0x562a53f99aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.218+0000 7f61a57fa640 0 client.25996 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:05:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.218+0000 7f61cac2e740 1 -- 172.21.15.31:0/4120435219 shutdown_connections 2024-04-25T16:05:01.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.218+0000 7f61cac2e740 1 -- 172.21.15.31:0/4120435219 wait complete. 2024-04-25T16:05:01.227 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:01.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.218+0000 7f61cac2e740 20 client.25996 trim_cache size 0 max 16384 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 -- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61a80afcf0 msgr2=0x7f61a80b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61a80afcf0 0x7f61a80b2200 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f61b40098b0 tx=0x7f61b4009840 comp rx=0 tx=0).stop 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 -- 172.21.15.31:0/3012807533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61bc10a340 msgr2=0x7f61bc11ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61bc10a340 0x7f61bc11ba20 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f61bc074a30 tx=0x7f61b800cdd0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 -- 172.21.15.31:0/3012807533 shutdown_connections 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f61a80afcf0 0x7f61a80b2200 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61bc10fe90 0x7f61bc11a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f61bc10b190 0x7f61bc11a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 --2- 172.21.15.31:0/3012807533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f61bc10a340 0x7f61bc11ba20 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.220+0000 7f61c2a3d640 1 -- 172.21.15.31:0/3012807533 >> 172.21.15.31:0/3012807533 conn(0x7f61bc070320 msgr2=0x7f61bc10de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:01.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.228+0000 7f61c2a3d640 1 -- 172.21.15.31:0/3012807533 shutdown_connections 2024-04-25T16:05:01.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.228+0000 7f61c2a3d640 1 -- 172.21.15.31:0/3012807533 wait complete. 2024-04-25T16:05:01.295 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:05:01.545 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:01 smithi031 ceph-mon[29140]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 778 MiB/s wr, 2.30k op/s 2024-04-25T16:05:01.546 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:01 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3449579584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:05:01.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:01 smithi063 ceph-mon[32413]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 778 MiB/s wr, 2.30k op/s 2024-04-25T16:05:01.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:01 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3449579584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:05:01.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:01 smithi186 ceph-mon[33258]: pgmap v462: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 778 MiB/s wr, 2.30k op/s 2024-04-25T16:05:01.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:01 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3449579584' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:05:01.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.887+0000 7f4d522f3640 1 -- 172.21.15.31:0/3056426179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 msgr2=0x7f4d4c112c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.887+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3056426179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 0x7f4d4c112c20 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009900 tx=0x7f4d3c02a420 comp rx=0 tx=0).stop 2024-04-25T16:05:01.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.893+0000 7f4d522f3640 1 -- 172.21.15.31:0/3056426179 shutdown_connections 2024-04-25T16:05:01.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.893+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3056426179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 0x7f4d4c112c20 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.893+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3056426179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c10b940 0x7f4d4c1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.893+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3056426179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d4c10aff0 0x7f4d4c10b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.893+0000 7f4d522f3640 1 -- 172.21.15.31:0/3056426179 >> 172.21.15.31:0/3056426179 conn(0x7f4d4c070780 msgr2=0x7f4d4c072c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:01.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.899+0000 7f4d522f3640 1 -- 172.21.15.31:0/3056426179 shutdown_connections 2024-04-25T16:05:01.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.900+0000 7f4d522f3640 1 -- 172.21.15.31:0/3056426179 wait complete. 2024-04-25T16:05:01.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.902+0000 7f4d522f3640 1 Processor -- start 2024-04-25T16:05:01.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.902+0000 7f4d522f3640 1 -- start start 2024-04-25T16:05:01.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.902+0000 7f4d522f3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c10aff0 0x7f4d4c115d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.902+0000 7f4d522f3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d4c10b940 0x7f4d4c1143b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.902+0000 7f4d522f3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 0x7f4d4c1148f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.902+0000 7f4d522f3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c117c40 con 0x7f4d4c110800 2024-04-25T16:05:01.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.902+0000 7f4d522f3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c117ac0 con 0x7f4d4c10aff0 2024-04-25T16:05:01.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.902+0000 7f4d522f3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4c117dc0 con 0x7f4d4c10b940 2024-04-25T16:05:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.906+0000 7f4d512f1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c10aff0 0x7f4d4c115d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.907+0000 7f4d51af2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 0x7f4d4c1148f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.907+0000 7f4d51af2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 0x7f4d4c1148f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38808/0 (socket says 172.21.15.31:38808) 2024-04-25T16:05:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.907+0000 7f4d51af2640 1 -- 172.21.15.31:0/3134020564 learned_addr learned my addr 172.21.15.31:0/3134020564 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.908+0000 7f4d50af0640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d4c10b940 0x7f4d4c1143b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.909+0000 7f4d51af2640 1 -- 172.21.15.31:0/3134020564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d4c10b940 msgr2=0x7f4d4c1143b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.909+0000 7f4d51af2640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d4c10b940 0x7f4d4c1143b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.909+0000 7f4d51af2640 1 -- 172.21.15.31:0/3134020564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c10aff0 msgr2=0x7f4d4c115d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.909+0000 7f4d51af2640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c10aff0 0x7f4d4c115d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.909+0000 7f4d51af2640 1 -- 172.21.15.31:0/3134020564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4c1a6660 con 0x7f4d4c110800 2024-04-25T16:05:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.909+0000 7f4d512f1640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c10aff0 0x7f4d4c115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:05:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.911+0000 7f4d51af2640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 0x7f4d4c1148f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009850 tx=0x7f4d3c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.913+0000 7f4d427fc640 1 -- 172.21.15.31:0/3134020564 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c03f070 con 0x7f4d4c110800 2024-04-25T16:05:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.913+0000 7f4d522f3640 1 -- 172.21.15.31:0/3134020564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d4c114ef0 con 0x7f4d4c110800 2024-04-25T16:05:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.913+0000 7f4d522f3640 1 -- 172.21.15.31:0/3134020564 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d4c1b1240 con 0x7f4d4c110800 2024-04-25T16:05:01.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.920+0000 7f4d5a405740 1 Processor -- start 2024-04-25T16:05:01.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.920+0000 7f4d5a405740 1 -- start start 2024-04-25T16:05:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.920+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662e29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.920+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662e2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.920+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.920+0000 7f4d5a405740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560966246d80 con 0x560965f174d0 2024-04-25T16:05:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.920+0000 7f4d5a405740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560966200a20 con 0x560965f40390 2024-04-25T16:05:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.920+0000 7f4d5a405740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560966242d40 con 0x560965f1f6b0 2024-04-25T16:05:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.920+0000 7f4d427fc640 1 -- 172.21.15.31:0/3134020564 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d3c031bc0 con 0x7f4d4c110800 2024-04-25T16:05:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.921+0000 7f4d427fc640 1 -- 172.21.15.31:0/3134020564 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c035030 con 0x7f4d4c110800 2024-04-25T16:05:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.923+0000 7f4d50af0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.923+0000 7f4d50af0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662e3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52180/0 (socket says 172.21.15.31:52180) 2024-04-25T16:05:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.923+0000 7f4d50af0640 1 -- 172.21.15.31:0/1579157015 learned_addr learned my addr 172.21.15.31:0/1579157015 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.923+0000 7f4d512f1640 1 --2- 172.21.15.31:0/1579157015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662e2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.924+0000 7f4d50af0640 1 --2- 172.21.15.31:0/1579157015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662e29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.925+0000 7f4d50af0640 1 -- 172.21.15.31:0/1579157015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 msgr2=0x5609662e29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.925+0000 7f4d50af0640 1 --2- 172.21.15.31:0/1579157015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662e29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.925+0000 7f4d50af0640 1 -- 172.21.15.31:0/1579157015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 msgr2=0x5609662e2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.925+0000 7f4d50af0640 1 --2- 172.21.15.31:0/1579157015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662e2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.926+0000 7f4d50af0640 1 -- 172.21.15.31:0/1579157015 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56096620bc30 con 0x560965f1f6b0 2024-04-25T16:05:01.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.931+0000 7f4d50af0640 1 --2- 172.21.15.31:0/1579157015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662e3bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009fd0 tx=0x7f4d44009a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3fe27a715c5b7af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.933+0000 7f4d377fe640 1 -- 172.21.15.31:0/1579157015 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44011c40 con 0x560965f1f6b0 2024-04-25T16:05:01.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.933+0000 7f4d377fe640 1 -- 172.21.15.31:0/1579157015 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d44011de0 con 0x560965f1f6b0 2024-04-25T16:05:01.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.933+0000 7f4d377fe640 1 -- 172.21.15.31:0/1579157015 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44004320 con 0x560965f1f6b0 2024-04-25T16:05:01.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.935+0000 7f4d5a405740 1 -- 172.21.15.31:0/1579157015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 msgr2=0x5609662e3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:01.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.935+0000 7f4d5a405740 1 --2- 172.21.15.31:0/1579157015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662e3bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009fd0 tx=0x7f4d44009a90 comp rx=0 tx=0).stop 2024-04-25T16:05:01.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.933+0000 7f4d427fc640 1 -- 172.21.15.31:0/3134020564 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4d3c042070 con 0x7f4d4c110800 2024-04-25T16:05:01.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.936+0000 7f4d5a405740 1 -- 172.21.15.31:0/1579157015 shutdown_connections 2024-04-25T16:05:01.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.937+0000 7f4d5a405740 1 --2- 172.21.15.31:0/1579157015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662e3bf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.937+0000 7f4d5a405740 1 --2- 172.21.15.31:0/1579157015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662e2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.937+0000 7f4d5a405740 1 --2- 172.21.15.31:0/1579157015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662e29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.939+0000 7f4d427fc640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d380afc70 0x7f4d380b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.939+0000 7f4d427fc640 1 -- 172.21.15.31:0/3134020564 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4d3c0ff290 con 0x7f4d4c110800 2024-04-25T16:05:01.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.941+0000 7f4d50af0640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d380afc70 0x7f4d380b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.941+0000 7f4d5a405740 1 -- 172.21.15.31:0/1579157015 >> 172.21.15.31:0/1579157015 conn(0x560965f344e0 msgr2=0x5609662de1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:01.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.946+0000 7f4d5a405740 1 -- 172.21.15.31:0/1579157015 shutdown_connections 2024-04-25T16:05:01.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.952+0000 7f4d5a405740 1 -- 172.21.15.31:0/1579157015 wait complete. 2024-04-25T16:05:01.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.953+0000 7f4d5a405740 1 -- start start 2024-04-25T16:05:01.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.954+0000 7f4d5a405740 1 Processor -- start 2024-04-25T16:05:01.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.954+0000 7f4d50af0640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d380afc70 0x7f4d380b2180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4d44013a40 tx=0x7f4d44015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.955+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662ec6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.955+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662ecc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.955+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.955+0000 7f4d5a405740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5609662e4700 con 0x560965f174d0 2024-04-25T16:05:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.955+0000 7f4d5a405740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5609661cddc0 con 0x560965f40390 2024-04-25T16:05:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.955+0000 7f4d5a405740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5609662e4880 con 0x560965f1f6b0 2024-04-25T16:05:01.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.956+0000 7f4d5a405740 10 client.0 _ll_register_callbacks cb 0x5609661e9600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:01.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.956+0000 7f4d512f1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662ec6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.956+0000 7f4d512f1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662ec6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38830/0 (socket says 172.21.15.31:38830) 2024-04-25T16:05:01.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.956+0000 7f4d512f1640 1 -- 172.21.15.31:0/3922131206 learned_addr learned my addr 172.21.15.31:0/3922131206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:01.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.958+0000 7f4d512f1640 1 -- 172.21.15.31:0/3922131206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 msgr2=0x5609662ecc30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:01.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.958+0000 7f4d512f1640 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662ecc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.958+0000 7f4d512f1640 1 -- 172.21.15.31:0/3922131206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 msgr2=0x5609662f3b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:01.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.958+0000 7f4d512f1640 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662f3b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:01.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.958+0000 7f4d512f1640 1 -- 172.21.15.31:0/3922131206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560965f583f0 con 0x560965f174d0 2024-04-25T16:05:01.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.963+0000 7f4d512f1640 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662ec6f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4d48013790 tx=0x7f4d48013c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5841a1c71c4f49 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:01.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.965+0000 7f4d34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:01.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.965+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3922131206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48011dd0 con 0x560965f174d0 2024-04-25T16:05:01.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.965+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3922131206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4d48004020 con 0x560965f174d0 2024-04-25T16:05:01.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.984+0000 7f4d5a405740 1 -- 172.21.15.31:0/3922131206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d44011de0 con 0x560965f174d0 2024-04-25T16:05:01.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.985+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3922131206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4d48004220 con 0x560965f174d0 2024-04-25T16:05:01.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.985+0000 7f4d5a405740 10 client.16347 fetch_fsmap learned FSMap version 28 2024-04-25T16:05:01.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.985+0000 7f4d5a405740 1 -- 172.21.15.31:0/3922131206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5609660227d0 con 0x560965f174d0 2024-04-25T16:05:01.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.988+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3922131206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d44011de0 con 0x560965f174d0 2024-04-25T16:05:01.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.988+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3922131206 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f4d4801a030 con 0x560965f174d0 2024-04-25T16:05:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.990+0000 7f4d5a405740 10 client.16347 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:05:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.990+0000 7f4d5a405740 10 client.16347 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:05:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.990+0000 7f4d5a405740 20 client.16347 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.990+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x560965efef60 0x56096630b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.990+0000 7f4d5a405740 4 client.16347 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.990+0000 7f4d5a405740 1 -- 172.21.15.31:0/3922131206 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560966254990 con 0x560965efef60 2024-04-25T16:05:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.991+0000 7f4d50af0640 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x560965efef60 0x56096630b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.995+0000 7f4d50af0640 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x560965efef60 0x56096630b600 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=59c3c3d01fcbae72 server_cookie=927b72c8036de79c in_seq=0 out_seq=0 2024-04-25T16:05:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.995+0000 7f4d34ff9640 10 client.16347 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:05:02.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.999+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3922131206 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560966254990 con 0x560965efef60 2024-04-25T16:05:02.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.999+0000 7f4d34ff9640 10 client.16347 handle_command_reply: tid=0 2024-04-25T16:05:02.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:01.999+0000 7f4d5a405740 1 client.16347 shutdown 2024-04-25T16:05:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.002+0000 7f4d5a405740 1 -- 172.21.15.31:0/3922131206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 msgr2=0x5609662ec6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.002+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662ec6f0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4d48013790 tx=0x7f4d48013c50 comp rx=0 tx=0).stop 2024-04-25T16:05:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.002+0000 7f4d5a405740 1 -- 172.21.15.31:0/3922131206 shutdown_connections 2024-04-25T16:05:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.002+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.002+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x560965efef60 0x56096630b600 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.002+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662ecc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.002+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3922131206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662ec6f0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.002+0000 7f4d5a405740 1 -- 172.21.15.31:0/3922131206 >> 172.21.15.31:0/3922131206 conn(0x560965f344e0 msgr2=0x5609662e1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:02.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.004+0000 7f4d34ff9640 0 client.16347 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:05:02.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.008+0000 7f4d5a405740 1 -- 172.21.15.31:0/3922131206 shutdown_connections 2024-04-25T16:05:02.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.012+0000 7f4d5a405740 1 -- 172.21.15.31:0/3922131206 wait complete. 2024-04-25T16:05:02.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.012+0000 7f4d5a405740 20 client.16347 trim_cache size 0 max 16384 2024-04-25T16:05:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.063+0000 7f4d5a405740 1 Processor -- start 2024-04-25T16:05:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.063+0000 7f4d5a405740 1 -- start start 2024-04-25T16:05:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.063+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662de330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.064+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f1f6b0 0x5609662ec2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.065+0000 7f4d512f1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662de330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.065+0000 7f4d512f1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662de330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38834/0 (socket says 172.21.15.31:38834) 2024-04-25T16:05:02.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.071+0000 7f4d512f1640 1 -- 172.21.15.31:0/3341445183 learned_addr learned my addr 172.21.15.31:0/3341445183 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.071+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3341445183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f40390 0x5609662ec800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.071+0000 7f4d5a405740 1 -- 172.21.15.31:0/3341445183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3000e540 con 0x560965f174d0 2024-04-25T16:05:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.071+0000 7f4d5a405740 1 -- 172.21.15.31:0/3341445183 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4d300100b0 con 0x560965f1f6b0 2024-04-25T16:05:02.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.071+0000 7f4d5a405740 1 -- 172.21.15.31:0/3341445183 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3000ec10 con 0x560965f40390 2024-04-25T16:05:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.073+0000 7f4d512f1640 1 -- 172.21.15.31:0/3341445183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f40390 msgr2=0x5609662ec800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:05:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.073+0000 7f4d512f1640 1 --2- 172.21.15.31:0/3341445183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f40390 0x5609662ec800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.073+0000 7f4d512f1640 1 -- 172.21.15.31:0/3341445183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f1f6b0 msgr2=0x5609662ec2c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:02.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.073+0000 7f4d512f1640 1 --2- 172.21.15.31:0/3341445183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f1f6b0 0x5609662ec2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.073+0000 7f4d512f1640 1 -- 172.21.15.31:0/3341445183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5609662e7e10 con 0x560965f174d0 2024-04-25T16:05:02.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.074+0000 7f4d512f1640 1 --2- 172.21.15.31:0/3341445183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662de330 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800f640 tx=0x7f4d48004000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67d01b91807e909 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:02.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.074+0000 7f4d36ffd640 1 -- 172.21.15.31:0/3341445183 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4800fd90 con 0x560965f174d0 2024-04-25T16:05:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.075+0000 7f4d36ffd640 1 -- 172.21.15.31:0/3341445183 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d48005440 con 0x560965f174d0 2024-04-25T16:05:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.075+0000 7f4d36ffd640 1 -- 172.21.15.31:0/3341445183 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d48003aa0 con 0x560965f174d0 2024-04-25T16:05:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.077+0000 7f4d5a405740 1 -- 172.21.15.31:0/3341445183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 msgr2=0x5609662de330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.077+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3341445183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662de330 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f4d4800f640 tx=0x7f4d48004000 comp rx=0 tx=0).stop 2024-04-25T16:05:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.080+0000 7f4d5a405740 1 -- 172.21.15.31:0/3341445183 shutdown_connections 2024-04-25T16:05:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.081+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3341445183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f40390 0x5609662ec800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.081+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3341445183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f1f6b0 0x5609662ec2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.081+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3341445183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662de330 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.081+0000 7f4d5a405740 1 -- 172.21.15.31:0/3341445183 >> 172.21.15.31:0/3341445183 conn(0x560965f344e0 msgr2=0x5609662ddc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.081+0000 7f4d5a405740 1 -- 172.21.15.31:0/3341445183 shutdown_connections 2024-04-25T16:05:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.081+0000 7f4d5a405740 1 -- 172.21.15.31:0/3341445183 wait complete. 2024-04-25T16:05:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.082+0000 7f4d5a405740 1 -- start start 2024-04-25T16:05:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.082+0000 7f4d5a405740 1 Processor -- start 2024-04-25T16:05:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.084+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662f0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.084+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662f14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.084+0000 7f4d5a405740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662f2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.085+0000 7f4d5a405740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3000f9b0 con 0x560965f174d0 2024-04-25T16:05:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.085+0000 7f4d5a405740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3000f2e0 con 0x560965f40390 2024-04-25T16:05:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.085+0000 7f4d5a405740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3000c520 con 0x560965f1f6b0 2024-04-25T16:05:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.085+0000 7f4d512f1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662f0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.085+0000 7f4d512f1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662f0fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38850/0 (socket says 172.21.15.31:38850) 2024-04-25T16:05:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.085+0000 7f4d512f1640 1 -- 172.21.15.31:0/3835616728 learned_addr learned my addr 172.21.15.31:0/3835616728 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.085+0000 7f4d50af0640 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662f14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.086+0000 7f4d51af2640 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.086+0000 7f4d5a405740 10 client.0 _ll_register_callbacks cb 0x7f4d48009690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.087+0000 7f4d50af0640 1 -- 172.21.15.31:0/3835616728 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 msgr2=0x5609662f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.087+0000 7f4d50af0640 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.087+0000 7f4d50af0640 1 -- 172.21.15.31:0/3835616728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 msgr2=0x5609662f0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.087+0000 7f4d50af0640 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662f0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.087+0000 7f4d50af0640 1 -- 172.21.15.31:0/3835616728 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5609662f29e0 con 0x560965f1f6b0 2024-04-25T16:05:02.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.091+0000 7f4d50af0640 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662f14f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009540 tx=0x7f4d44009a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8648dee09b1fa3ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:02.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.091+0000 7f4d34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:02.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.092+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3835616728 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4403d060 con 0x560965f1f6b0 2024-04-25T16:05:02.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.092+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3835616728 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f4d44003e00 con 0x560965f1f6b0 2024-04-25T16:05:02.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.097+0000 7f4d5a405740 1 -- 172.21.15.31:0/3835616728 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d48005440 con 0x560965f1f6b0 2024-04-25T16:05:02.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.099+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3835616728 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4d440373b0 con 0x560965f1f6b0 2024-04-25T16:05:02.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.101+0000 7f4d5a405740 10 client.26005 fetch_fsmap learned FSMap version 28 2024-04-25T16:05:02.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.101+0000 7f4d5a405740 1 -- 172.21.15.31:0/3835616728 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560966373ed0 con 0x560965f1f6b0 2024-04-25T16:05:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.102+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3835616728 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d48005440 con 0x560965f1f6b0 2024-04-25T16:05:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.102+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3835616728 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f4d44005780 con 0x560965f1f6b0 2024-04-25T16:05:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.103+0000 7f4d5a405740 10 client.26005 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:05:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.104+0000 7f4d5a405740 10 client.26005 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:05:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.104+0000 7f4d5a405740 20 client.26005 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.104+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x560965ed43a0 0x560966376700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.104+0000 7f4d5a405740 4 client.26005 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:02.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.104+0000 7f4d5a405740 1 -- 172.21.15.31:0/3835616728 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56096624e800 con 0x560965ed43a0 2024-04-25T16:05:02.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.109+0000 7f4d512f1640 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x560965ed43a0 0x560966376700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.115+0000 7f4d34ff9640 10 client.26005 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:05:02.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.115+0000 7f4d512f1640 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x560965ed43a0 0x560966376700 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5814460455b1ebbf server_cookie=a9ec50e00bb4bc3d in_seq=0 out_seq=0 2024-04-25T16:05:02.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.116+0000 7f4d34ff9640 1 -- 172.21.15.31:0/3835616728 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56096624e800 con 0x560965ed43a0 2024-04-25T16:05:02.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.116+0000 7f4d34ff9640 10 client.26005 handle_command_reply: tid=0 2024-04-25T16:05:02.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.116+0000 7f4d5a405740 1 client.26005 shutdown 2024-04-25T16:05:02.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.118+0000 7f4d5a405740 1 -- 172.21.15.31:0/3835616728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 msgr2=0x5609662f14f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.118+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662f14f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009540 tx=0x7f4d44009a90 comp rx=0 tx=0).stop 2024-04-25T16:05:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.126+0000 7f4d5a405740 1 -- 172.21.15.31:0/3835616728 shutdown_connections 2024-04-25T16:05:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.126+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560965f40390 0x5609662f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.126+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x560965ed43a0 0x560966376700 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.126+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560965f1f6b0 0x5609662f14f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.126+0000 7f4d5a405740 1 --2- 172.21.15.31:0/3835616728 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560965f174d0 0x5609662f0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.126+0000 7f4d5a405740 1 -- 172.21.15.31:0/3835616728 >> 172.21.15.31:0/3835616728 conn(0x560965f344e0 msgr2=0x5609662e0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.127+0000 7f4d34ff9640 0 client.26005 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:05:02.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.133+0000 7f4d5a405740 1 -- 172.21.15.31:0/3835616728 shutdown_connections 2024-04-25T16:05:02.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.134+0000 7f4d5a405740 1 -- 172.21.15.31:0/3835616728 wait complete. 2024-04-25T16:05:02.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.134+0000 7f4d5a405740 20 client.26005 trim_cache size 0 max 16384 2024-04-25T16:05:02.135 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.137+0000 7f4d522f3640 1 -- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d380afc70 msgr2=0x7f4d380b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.137+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d380afc70 0x7f4d380b2180 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f4d44013a40 tx=0x7f4d44015040 comp rx=0 tx=0).stop 2024-04-25T16:05:02.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.138+0000 7f4d522f3640 1 -- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 msgr2=0x7f4d4c1148f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.138+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 0x7f4d4c1148f0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009850 tx=0x7f4d3c003b90 comp rx=0 tx=0).stop 2024-04-25T16:05:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.138+0000 7f4d522f3640 1 -- 172.21.15.31:0/3134020564 shutdown_connections 2024-04-25T16:05:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.138+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f4d380afc70 0x7f4d380b2180 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.138+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f4d4c110800 0x7f4d4c1148f0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.138+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4d4c10b940 0x7f4d4c1143b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.139+0000 7f4d522f3640 1 --2- 172.21.15.31:0/3134020564 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4d4c10aff0 0x7f4d4c115d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.139+0000 7f4d522f3640 1 -- 172.21.15.31:0/3134020564 >> 172.21.15.31:0/3134020564 conn(0x7f4d4c070780 msgr2=0x7f4d4c0715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.141+0000 7f4d522f3640 1 -- 172.21.15.31:0/3134020564 shutdown_connections 2024-04-25T16:05:02.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.141+0000 7f4d522f3640 1 -- 172.21.15.31:0/3134020564 wait complete. 2024-04-25T16:05:02.197 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:05:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.762+0000 7fe7965b2640 1 -- 172.21.15.31:0/1111741134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe79010a3d0 msgr2=0x7fe79010adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.762+0000 7fe7965b2640 1 --2- 172.21.15.31:0/1111741134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe79010a3d0 0x7fe79010adf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fe788009ca0 tx=0x7fe78802a8f0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.771+0000 7fe7965b2640 1 -- 172.21.15.31:0/1111741134 shutdown_connections 2024-04-25T16:05:02.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.771+0000 7fe7965b2640 1 --2- 172.21.15.31:0/1111741134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe790110030 0x7fe790112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.771+0000 7fe7965b2640 1 --2- 172.21.15.31:0/1111741134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe79010b330 0x7fe79010b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.771+0000 7fe7965b2640 1 --2- 172.21.15.31:0/1111741134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe79010a3d0 0x7fe79010adf0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.771+0000 7fe7965b2640 1 -- 172.21.15.31:0/1111741134 >> 172.21.15.31:0/1111741134 conn(0x7fe790078de0 msgr2=0x7fe79007b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:02.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.778+0000 7fe7965b2640 1 -- 172.21.15.31:0/1111741134 shutdown_connections 2024-04-25T16:05:02.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.782+0000 7fe7965b2640 1 -- 172.21.15.31:0/1111741134 wait complete. 2024-04-25T16:05:02.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.783+0000 7fe7965b2640 1 Processor -- start 2024-04-25T16:05:02.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.783+0000 7fe7965b2640 1 -- start start 2024-04-25T16:05:02.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.784+0000 7fe7965b2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe79010a3d0 0x7fe7900744b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.784+0000 7fe7965b2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe79010b330 0x7fe790072b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.784+0000 7fe7965b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe790110030 0x7fe790073090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.784+0000 7fe7965b2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe79006cf70 con 0x7fe790110030 2024-04-25T16:05:02.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.784+0000 7fe7965b2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe79006cdf0 con 0x7fe79010a3d0 2024-04-25T16:05:02.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.784+0000 7fe7965b2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe79006d0f0 con 0x7fe79010b330 2024-04-25T16:05:02.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.785+0000 7fe794b4f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe790110030 0x7fe790073090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.785+0000 7fe794b4f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe790110030 0x7fe790073090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38870/0 (socket says 172.21.15.31:38870) 2024-04-25T16:05:02.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.785+0000 7fe794b4f640 1 -- 172.21.15.31:0/2046753165 learned_addr learned my addr 172.21.15.31:0/2046753165 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.786+0000 7fe78ffff640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe79010a3d0 0x7fe7900744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.786+0000 7fe794b4f640 1 -- 172.21.15.31:0/2046753165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe79010b330 msgr2=0x7fe790072b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.786+0000 7fe794b4f640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe79010b330 0x7fe790072b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.786+0000 7fe794b4f640 1 -- 172.21.15.31:0/2046753165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe79010a3d0 msgr2=0x7fe7900744b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.786+0000 7fe794b4f640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe79010a3d0 0x7fe7900744b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.786+0000 7fe794b4f640 1 -- 172.21.15.31:0/2046753165 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe790073650 con 0x7fe790110030 2024-04-25T16:05:02.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.786+0000 7fe794b4f640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe790110030 0x7fe790073090 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe77800b3f0 tx=0x7fe77800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.814+0000 7fe78d7fa640 1 -- 172.21.15.31:0/2046753165 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe778015020 con 0x7fe790110030 2024-04-25T16:05:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.814+0000 7fe78d7fa640 1 -- 172.21.15.31:0/2046753165 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe778011ce0 con 0x7fe790110030 2024-04-25T16:05:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.814+0000 7fe78d7fa640 1 -- 172.21.15.31:0/2046753165 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe778010de0 con 0x7fe790110030 2024-04-25T16:05:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.819+0000 7fe7965b2640 1 -- 172.21.15.31:0/2046753165 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe790073940 con 0x7fe790110030 2024-04-25T16:05:02.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.820+0000 7fe7965b2640 1 -- 172.21.15.31:0/2046753165 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7901b1480 con 0x7fe790110030 2024-04-25T16:05:02.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.835+0000 7fe79d465740 1 Processor -- start 2024-04-25T16:05:02.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.838+0000 7fe78d7fa640 1 -- 172.21.15.31:0/2046753165 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe778018020 con 0x7fe790110030 2024-04-25T16:05:02.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.838+0000 7fe79d465740 1 -- start start 2024-04-25T16:05:02.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.838+0000 7fe79d465740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91823390 0x556c91bc5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.838+0000 7fe79d465740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91bc5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.838+0000 7fe79d465740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91bc6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.838+0000 7fe79d465740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556c91b29d80 con 0x556c918026b0 2024-04-25T16:05:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.838+0000 7fe79d465740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556c91ae3a20 con 0x556c917fa4d0 2024-04-25T16:05:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.838+0000 7fe79d465740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556c91b25d40 con 0x556c91823390 2024-04-25T16:05:02.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.840+0000 7fe78d7fa640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe77c0afc70 0x7fe77c0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.843+0000 7fe78f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91823390 0x556c91bc5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.843+0000 7fe78f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91bc6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.843+0000 7fe78f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91bc6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38876/0 (socket says 172.21.15.31:38876) 2024-04-25T16:05:02.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.843+0000 7fe78f7fe640 1 -- 172.21.15.31:0/3183972111 learned_addr learned my addr 172.21.15.31:0/3183972111 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:02.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.848+0000 7fe78d7fa640 1 -- 172.21.15.31:0/2046753165 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe7780de2e0 con 0x7fe790110030 2024-04-25T16:05:02.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.851+0000 7fe78f7fe640 1 -- 172.21.15.31:0/3183972111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91823390 msgr2=0x556c91bc5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:02.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.851+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/3183972111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91823390 0x556c91bc5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.851+0000 7fe78f7fe640 1 -- 172.21.15.31:0/3183972111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 msgr2=0x556c91bc5d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:02.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.852+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/3183972111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91bc5d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.852+0000 7fe78f7fe640 1 -- 172.21.15.31:0/3183972111 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c91aeec30 con 0x556c918026b0 2024-04-25T16:05:02.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.856+0000 7fe794b4f640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe77c0afc70 0x7fe77c0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.865+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/3183972111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91bc6a20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe780009fd0 tx=0x7fe780001140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b6e37709445dbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:02.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.866+0000 7fe794b4f640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe77c0afc70 0x7fe77c0b2180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe778011040 tx=0x7fe778010660 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:02.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.870+0000 7fe7867fc640 1 -- 172.21.15.31:0/3183972111 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780012040 con 0x556c918026b0 2024-04-25T16:05:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.870+0000 7fe7867fc640 1 -- 172.21.15.31:0/3183972111 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe780007590 con 0x556c918026b0 2024-04-25T16:05:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.872+0000 7fe79d465740 1 -- 172.21.15.31:0/3183972111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 msgr2=0x556c91bc6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.872+0000 7fe79d465740 1 --2- 172.21.15.31:0/3183972111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91bc6a20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe780009fd0 tx=0x7fe780001140 comp rx=0 tx=0).stop 2024-04-25T16:05:02.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.881+0000 7fe79d465740 1 -- 172.21.15.31:0/3183972111 shutdown_connections 2024-04-25T16:05:02.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.881+0000 7fe79d465740 1 --2- 172.21.15.31:0/3183972111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91bc6a20 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.881+0000 7fe79d465740 1 --2- 172.21.15.31:0/3183972111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91bc5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.881+0000 7fe79d465740 1 --2- 172.21.15.31:0/3183972111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91823390 0x556c91bc5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.881+0000 7fe79d465740 1 -- 172.21.15.31:0/3183972111 >> 172.21.15.31:0/3183972111 conn(0x556c918174e0 msgr2=0x556c91bc1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:02.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.886+0000 7fe79d465740 1 -- 172.21.15.31:0/3183972111 shutdown_connections 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.890+0000 7fe79d465740 1 -- 172.21.15.31:0/3183972111 wait complete. 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.891+0000 7fe79d465740 1 -- start start 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.891+0000 7fe79d465740 1 Processor -- start 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.892+0000 7fe79d465740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c917fa4d0 0x556c91be0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.892+0000 7fe79d465740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c918026b0 0x556c91be0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.892+0000 7fe79d465740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c91823390 0x556c91be7a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.892+0000 7fe79d465740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556c91bc7580 con 0x556c91823390 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.892+0000 7fe79d465740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556c91ab0dc0 con 0x556c918026b0 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.892+0000 7fe79d465740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556c91bc7700 con 0x556c917fa4d0 2024-04-25T16:05:02.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.892+0000 7fe79d465740 10 client.0 _ll_register_callbacks cb 0x556c91acc600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.893+0000 7fe78f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c91823390 0x556c91be7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.893+0000 7fe78f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c91823390 0x556c91be7a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38888/0 (socket says 172.21.15.31:38888) 2024-04-25T16:05:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.893+0000 7fe78f7fe640 1 -- 172.21.15.31:0/1080007845 learned_addr learned my addr 172.21.15.31:0/1080007845 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.894+0000 7fe78f7fe640 1 -- 172.21.15.31:0/1080007845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c917fa4d0 msgr2=0x556c91be0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:02.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.894+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c917fa4d0 0x556c91be0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.894+0000 7fe78f7fe640 1 -- 172.21.15.31:0/1080007845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c918026b0 msgr2=0x556c91be0aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.894+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c918026b0 0x556c91be0aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.894+0000 7fe78f7fe640 1 -- 172.21.15.31:0/1080007845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c9183b3f0 con 0x556c91823390 2024-04-25T16:05:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.894+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c91823390 0x556c91be7a00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe780000f60 tx=0x7fe780015f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b8ac781da3eba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.895+0000 7fe76f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.895+0000 7fe76f7fe640 1 -- 172.21.15.31:0/1080007845 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780001da0 con 0x556c91823390 2024-04-25T16:05:02.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.895+0000 7fe79d465740 1 -- 172.21.15.31:0/1080007845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe780007590 con 0x556c91823390 2024-04-25T16:05:02.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.896+0000 7fe76f7fe640 1 -- 172.21.15.31:0/1080007845 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe780004c30 con 0x556c91823390 2024-04-25T16:05:02.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.897+0000 7fe76f7fe640 1 -- 172.21.15.31:0/1080007845 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe780029050 con 0x556c91823390 2024-04-25T16:05:02.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.898+0000 7fe79d465740 10 client.16365 fetch_fsmap learned FSMap version 28 2024-04-25T16:05:02.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.898+0000 7fe79d465740 1 -- 172.21.15.31:0/1080007845 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c919057d0 con 0x556c91823390 2024-04-25T16:05:02.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.904+0000 7fe76f7fe640 1 -- 172.21.15.31:0/1080007845 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe780007590 con 0x556c91823390 2024-04-25T16:05:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.904+0000 7fe76f7fe640 1 -- 172.21.15.31:0/1080007845 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fe7800036b0 con 0x556c91823390 2024-04-25T16:05:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.904+0000 7fe79d465740 10 client.16365 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:05:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.904+0000 7fe79d465740 10 client.16365 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:05:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.904+0000 7fe79d465740 20 client.16365 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.905+0000 7fe79d465740 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x556c917e1f60 0x556c91c10600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.905+0000 7fe79d465740 4 client.16365 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:02.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.905+0000 7fe79d465740 1 -- 172.21.15.31:0/1080007845 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c91b37990 con 0x556c917e1f60 2024-04-25T16:05:02.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.914+0000 7fe78ffff640 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x556c917e1f60 0x556c91c10600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:02.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.931+0000 7fe78ffff640 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x556c917e1f60 0x556c91c10600 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e720717869970f3 server_cookie=3220f6c95bd12977 in_seq=0 out_seq=0 2024-04-25T16:05:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.931+0000 7fe76f7fe640 10 client.16365 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:05:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.935+0000 7fe76f7fe640 1 -- 172.21.15.31:0/1080007845 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556c91b37990 con 0x556c917e1f60 2024-04-25T16:05:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.935+0000 7fe76f7fe640 10 client.16365 handle_command_reply: tid=0 2024-04-25T16:05:02.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.936+0000 7fe79d465740 1 client.16365 shutdown 2024-04-25T16:05:02.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.945+0000 7fe79d465740 1 -- 172.21.15.31:0/1080007845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c91823390 msgr2=0x556c91be7a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:02.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.945+0000 7fe79d465740 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c91823390 0x556c91be7a00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fe780000f60 tx=0x7fe780015f30 comp rx=0 tx=0).stop 2024-04-25T16:05:02.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.945+0000 7fe79d465740 1 -- 172.21.15.31:0/1080007845 shutdown_connections 2024-04-25T16:05:02.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.945+0000 7fe79d465740 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c91823390 0x556c91be7a00 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.945+0000 7fe79d465740 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x556c917e1f60 0x556c91c10600 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.945+0000 7fe79d465740 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c918026b0 0x556c91be0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.945+0000 7fe79d465740 1 --2- 172.21.15.31:0/1080007845 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c917fa4d0 0x556c91be0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:02.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.945+0000 7fe79d465740 1 -- 172.21.15.31:0/1080007845 >> 172.21.15.31:0/1080007845 conn(0x556c918174e0 msgr2=0x556c91bc0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:02.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.946+0000 7fe76f7fe640 0 client.16365 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:05:02.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.946+0000 7fe79d465740 1 -- 172.21.15.31:0/1080007845 shutdown_connections 2024-04-25T16:05:02.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.948+0000 7fe79d465740 1 -- 172.21.15.31:0/1080007845 wait complete. 2024-04-25T16:05:02.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:02.948+0000 7fe79d465740 20 client.16365 trim_cache size 0 max 16384 2024-04-25T16:05:03.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.032+0000 7fe79d465740 1 Processor -- start 2024-04-25T16:05:03.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.032+0000 7fe79d465740 1 -- start start 2024-04-25T16:05:03.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.033+0000 7fe79d465740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91c10570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.033+0000 7fe79d465740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c918026b0 0x556c91bca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.033+0000 7fe79d465740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c917b73a0 0x556c91bca770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.033+0000 7fe79d465740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe7901155c0 con 0x556c917b73a0 2024-04-25T16:05:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.033+0000 7fe79d465740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe790117130 con 0x556c917fa4d0 2024-04-25T16:05:03.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.033+0000 7fe79d465740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe790115c90 con 0x556c918026b0 2024-04-25T16:05:03.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.036+0000 7fe78f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c917b73a0 0x556c91bca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.036+0000 7fe78f7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c917b73a0 0x556c91bca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38898/0 (socket says 172.21.15.31:38898) 2024-04-25T16:05:03.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.036+0000 7fe78f7fe640 1 -- 172.21.15.31:0/4077181934 learned_addr learned my addr 172.21.15.31:0/4077181934 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:03.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.036+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/4077181934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91c10570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.037+0000 7fe78ffff640 1 --2- 172.21.15.31:0/4077181934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c918026b0 0x556c91bca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.037+0000 7fe78f7fe640 1 -- 172.21.15.31:0/4077181934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c918026b0 msgr2=0x556c91bca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.037+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/4077181934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c918026b0 0x556c91bca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.037+0000 7fe78f7fe640 1 -- 172.21.15.31:0/4077181934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 msgr2=0x556c91c10570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.037+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/4077181934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91c10570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.037+0000 7fe78f7fe640 1 -- 172.21.15.31:0/4077181934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c91be1160 con 0x556c917b73a0 2024-04-25T16:05:03.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.044+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/4077181934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c917b73a0 0x556c91bca770 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe780001800 tx=0x7fe780000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=820625aa3b0f320e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.046+0000 7fe785ffb640 1 -- 172.21.15.31:0/4077181934 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780033070 con 0x556c917b73a0 2024-04-25T16:05:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.046+0000 7fe785ffb640 1 -- 172.21.15.31:0/4077181934 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe780001df0 con 0x556c917b73a0 2024-04-25T16:05:03.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.046+0000 7fe785ffb640 1 -- 172.21.15.31:0/4077181934 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe780004dd0 con 0x556c917b73a0 2024-04-25T16:05:03.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.049+0000 7fe79d465740 1 -- 172.21.15.31:0/4077181934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c917b73a0 msgr2=0x556c91bca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.050+0000 7fe79d465740 1 --2- 172.21.15.31:0/4077181934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c917b73a0 0x556c91bca770 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe780001800 tx=0x7fe780000bc0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.051+0000 7fe79d465740 1 -- 172.21.15.31:0/4077181934 shutdown_connections 2024-04-25T16:05:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.051+0000 7fe79d465740 1 --2- 172.21.15.31:0/4077181934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c917b73a0 0x556c91bca770 secure :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fe780001800 tx=0x7fe780000bc0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.051+0000 7fe79d465740 1 --2- 172.21.15.31:0/4077181934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c918026b0 0x556c91bca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.051+0000 7fe79d465740 1 --2- 172.21.15.31:0/4077181934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91c10570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.051+0000 7fe79d465740 1 -- 172.21.15.31:0/4077181934 >> 172.21.15.31:0/4077181934 conn(0x556c918174e0 msgr2=0x556c91be0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:03.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.065+0000 7fe79d465740 1 -- 172.21.15.31:0/4077181934 shutdown_connections 2024-04-25T16:05:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.068+0000 7fe79d465740 1 -- 172.21.15.31:0/4077181934 wait complete. 2024-04-25T16:05:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.068+0000 7fe79d465740 1 -- start start 2024-04-25T16:05:03.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.069+0000 7fe79d465740 1 Processor -- start 2024-04-25T16:05:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.070+0000 7fe79d465740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91be6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.070+0000 7fe79d465740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91be6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.070+0000 7fe79d465740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91be3630 0x556c91be3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.070+0000 7fe79d465740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe790116a30 con 0x556c918026b0 2024-04-25T16:05:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.070+0000 7fe79d465740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe790116360 con 0x556c917fa4d0 2024-04-25T16:05:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.070+0000 7fe79d465740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe790082f00 con 0x556c91be3630 2024-04-25T16:05:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.070+0000 7fe79d465740 10 client.0 _ll_register_callbacks cb 0x7fe780001a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:03.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.071+0000 7fe78f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91be6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.071+0000 7fe78f7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91be3630 0x556c91be3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.072+0000 7fe78ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91be6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.072+0000 7fe78ffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91be6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38910/0 (socket says 172.21.15.31:38910) 2024-04-25T16:05:03.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.072+0000 7fe78ffff640 1 -- 172.21.15.31:0/3033270615 learned_addr learned my addr 172.21.15.31:0/3033270615 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.074+0000 7fe78ffff640 1 -- 172.21.15.31:0/3033270615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91be3630 msgr2=0x556c91be3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.074+0000 7fe78ffff640 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91be3630 0x556c91be3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.074+0000 7fe78ffff640 1 -- 172.21.15.31:0/3033270615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 msgr2=0x556c91be6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.074+0000 7fe78ffff640 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91be6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.074+0000 7fe78ffff640 1 -- 172.21.15.31:0/3033270615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c91c76960 con 0x556c918026b0 2024-04-25T16:05:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.080+0000 7fe78ffff640 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91be6ff0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe788009bf0 tx=0x7fe788009a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f51d7074ee8b42 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:03.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.080+0000 7fe76f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:03.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.081+0000 7fe76f7fe640 1 -- 172.21.15.31:0/3033270615 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788003b70 con 0x556c918026b0 2024-04-25T16:05:03.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.081+0000 7fe79d465740 1 -- 172.21.15.31:0/3033270615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe780001df0 con 0x556c918026b0 2024-04-25T16:05:03.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.082+0000 7fe76f7fe640 1 -- 172.21.15.31:0/3033270615 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe788003d10 con 0x556c918026b0 2024-04-25T16:05:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.083+0000 7fe76f7fe640 1 -- 172.21.15.31:0/3033270615 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe788052050 con 0x556c918026b0 2024-04-25T16:05:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.083+0000 7fe79d465740 10 client.16371 fetch_fsmap learned FSMap version 28 2024-04-25T16:05:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.083+0000 7fe79d465740 1 -- 172.21.15.31:0/3033270615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556c91c77b20 con 0x556c918026b0 2024-04-25T16:05:03.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.084+0000 7fe76f7fe640 1 -- 172.21.15.31:0/3033270615 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe780001df0 con 0x556c918026b0 2024-04-25T16:05:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.087+0000 7fe76f7fe640 1 -- 172.21.15.31:0/3033270615 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fe788055020 con 0x556c918026b0 2024-04-25T16:05:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.087+0000 7fe79d465740 10 client.16371 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:05:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.087+0000 7fe79d465740 10 client.16371 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:05:03.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.087+0000 7fe79d465740 20 client.16371 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:03.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.088+0000 7fe79d465740 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x556c91c784d0 0x556c91c7a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.088+0000 7fe79d465740 4 client.16371 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:03.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.088+0000 7fe79d465740 1 -- 172.21.15.31:0/3033270615 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c91b31800 con 0x556c91c784d0 2024-04-25T16:05:03.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.096+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x556c91c784d0 0x556c91c7a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.113+0000 7fe78f7fe640 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x556c91c784d0 0x556c91c7a8f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d896d5f1cb3451f server_cookie=a463678be2bb06a2 in_seq=0 out_seq=0 2024-04-25T16:05:03.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.116+0000 7fe76f7fe640 10 client.16371 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:05:03.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.119+0000 7fe76f7fe640 1 -- 172.21.15.31:0/3033270615 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c91b31800 con 0x556c91c784d0 2024-04-25T16:05:03.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.120+0000 7fe76f7fe640 10 client.16371 handle_command_reply: tid=0 2024-04-25T16:05:03.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.120+0000 7fe79d465740 1 client.16371 shutdown 2024-04-25T16:05:03.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.126+0000 7fe79d465740 1 -- 172.21.15.31:0/3033270615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 msgr2=0x556c91be6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.126+0000 7fe79d465740 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91be6ff0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe788009bf0 tx=0x7fe788009a00 comp rx=0 tx=0).stop 2024-04-25T16:05:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.132+0000 7fe79d465740 1 -- 172.21.15.31:0/3033270615 shutdown_connections 2024-04-25T16:05:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.132+0000 7fe79d465740 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556c91be3630 0x556c91be3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.132+0000 7fe79d465740 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556c918026b0 0x556c91be6ff0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.132+0000 7fe79d465740 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x556c91c784d0 0x556c91c7a8f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.132+0000 7fe79d465740 1 --2- 172.21.15.31:0/3033270615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556c917fa4d0 0x556c91be6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.132+0000 7fe79d465740 1 -- 172.21.15.31:0/3033270615 >> 172.21.15.31:0/3033270615 conn(0x556c918174e0 msgr2=0x556c91bca230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.132+0000 7fe76f7fe640 0 client.16371 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:05:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.132+0000 7fe79d465740 1 -- 172.21.15.31:0/3033270615 shutdown_connections 2024-04-25T16:05:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.132+0000 7fe79d465740 1 -- 172.21.15.31:0/3033270615 wait complete. 2024-04-25T16:05:03.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.133+0000 7fe79d465740 20 client.16371 trim_cache size 0 max 16384 2024-04-25T16:05:03.135 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.136+0000 7fe7965b2640 1 -- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe77c0afc70 msgr2=0x7fe77c0b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.136+0000 7fe7965b2640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe77c0afc70 0x7fe77c0b2180 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe778011040 tx=0x7fe778010660 comp rx=0 tx=0).stop 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.136+0000 7fe7965b2640 1 -- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe790110030 msgr2=0x7fe790073090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.136+0000 7fe7965b2640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe790110030 0x7fe790073090 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe77800b3f0 tx=0x7fe77800b8b0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.137+0000 7fe7965b2640 1 -- 172.21.15.31:0/2046753165 shutdown_connections 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.137+0000 7fe7965b2640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe77c0afc70 0x7fe77c0b2180 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.137+0000 7fe7965b2640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe790110030 0x7fe790073090 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.137+0000 7fe7965b2640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe79010b330 0x7fe790072b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.137+0000 7fe7965b2640 1 --2- 172.21.15.31:0/2046753165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe79010a3d0 0x7fe7900744b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.137+0000 7fe7965b2640 1 -- 172.21.15.31:0/2046753165 >> 172.21.15.31:0/2046753165 conn(0x7fe790078de0 msgr2=0x7fe79010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:03.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.138+0000 7fe7965b2640 1 -- 172.21.15.31:0/2046753165 shutdown_connections 2024-04-25T16:05:03.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.141+0000 7fe7965b2640 1 -- 172.21.15.31:0/2046753165 wait complete. 2024-04-25T16:05:03.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:05:03.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:03 smithi031 ceph-mon[29140]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 760 MiB/s wr, 2.10k op/s 2024-04-25T16:05:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.457+0000 7fa7712fc640 1 -- 172.21.15.31:0/905027202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c10e6d0 msgr2=0x7fa76c10f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.457+0000 7fa7712fc640 1 --2- 172.21.15.31:0/905027202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c10e6d0 0x7fa76c10f5c0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fa760009900 tx=0x7fa76002a420 comp rx=0 tx=0).stop 2024-04-25T16:05:03.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.458+0000 7fa7712fc640 1 -- 172.21.15.31:0/905027202 shutdown_connections 2024-04-25T16:05:03.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.458+0000 7fa7712fc640 1 --2- 172.21.15.31:0/905027202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c114290 0x7fa76c1166b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.458+0000 7fa7712fc640 1 --2- 172.21.15.31:0/905027202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c10e6d0 0x7fa76c10f5c0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.458+0000 7fa7712fc640 1 --2- 172.21.15.31:0/905027202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c10e210 0x7fa76c10efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.458+0000 7fa7712fc640 1 -- 172.21.15.31:0/905027202 >> 172.21.15.31:0/905027202 conn(0x7fa76c06e0b0 msgr2=0x7fa76c10bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.465+0000 7fa7712fc640 1 -- 172.21.15.31:0/905027202 shutdown_connections 2024-04-25T16:05:03.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.467+0000 7fa7712fc640 1 -- 172.21.15.31:0/905027202 wait complete. 2024-04-25T16:05:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.468+0000 7fa7712fc640 1 Processor -- start 2024-04-25T16:05:03.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.469+0000 7fa7712fc640 1 -- start start 2024-04-25T16:05:03.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.470+0000 7fa7712fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c10e210 0x7fa76c11b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.470+0000 7fa7712fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c114290 0x7fa76c11bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.470+0000 7fa7712fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c11eaa0 0x7fa76c11eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.471+0000 7fa7712fc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c0703e0 con 0x7fa76c114290 2024-04-25T16:05:03.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.471+0000 7fa7712fc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c070260 con 0x7fa76c10e210 2024-04-25T16:05:03.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.471+0000 7fa7712fc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa76c070560 con 0x7fa76c11eaa0 2024-04-25T16:05:03.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.473+0000 7fa76b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c114290 0x7fa76c11bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.473+0000 7fa76b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c114290 0x7fa76c11bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38932/0 (socket says 172.21.15.31:38932) 2024-04-25T16:05:03.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.473+0000 7fa76b7fe640 1 -- 172.21.15.31:0/2637968031 learned_addr learned my addr 172.21.15.31:0/2637968031 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:03.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.477+0000 7fa76b7fe640 1 -- 172.21.15.31:0/2637968031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c11eaa0 msgr2=0x7fa76c11eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.477+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c11eaa0 0x7fa76c11eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.478+0000 7fa76b7fe640 1 -- 172.21.15.31:0/2637968031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c10e210 msgr2=0x7fa76c11b5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:05:03.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.478+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c10e210 0x7fa76c11b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.478+0000 7fa76b7fe640 1 -- 172.21.15.31:0/2637968031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76c11f520 con 0x7fa76c114290 2024-04-25T16:05:03.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.479+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c114290 0x7fa76c11bb30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa760031440 tx=0x7fa760003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:03.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.480+0000 7fa7697fa640 1 -- 172.21.15.31:0/2637968031 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa76003f070 con 0x7fa76c114290 2024-04-25T16:05:03.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.480+0000 7fa7697fa640 1 -- 172.21.15.31:0/2637968031 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa760003ed0 con 0x7fa76c114290 2024-04-25T16:05:03.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.481+0000 7fa7712fc640 1 -- 172.21.15.31:0/2637968031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa76c11f7b0 con 0x7fa76c114290 2024-04-25T16:05:03.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.481+0000 7fa7712fc640 1 -- 172.21.15.31:0/2637968031 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76c11fc60 con 0x7fa76c114290 2024-04-25T16:05:03.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.485+0000 7fa7697fa640 1 -- 172.21.15.31:0/2637968031 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa760035030 con 0x7fa76c114290 2024-04-25T16:05:03.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.492+0000 7fa7697fa640 1 -- 172.21.15.31:0/2637968031 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa760042070 con 0x7fa76c114290 2024-04-25T16:05:03.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.494+0000 7fa7697fa640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7540afcf0 0x7fa7540b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.496+0000 7fa76bfff640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7540afcf0 0x7fa7540b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.496+0000 7fa7697fa640 1 -- 172.21.15.31:0/2637968031 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa760100690 con 0x7fa76c114290 2024-04-25T16:05:03.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.498+0000 7fa76bfff640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7540afcf0 0x7fa7540b2200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa76c10dba0 tx=0x7fa75c006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:03.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.498+0000 7fa779410740 1 Processor -- start 2024-04-25T16:05:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.500+0000 7fa779410740 1 -- start start 2024-04-25T16:05:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.500+0000 7fa779410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b255390 0x561b7b5f79e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.500+0000 7fa779410740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b5f7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.500+0000 7fa779410740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b2346b0 0x561b7b5f8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.500+0000 7fa779410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561b7b55bd80 con 0x561b7b255390 2024-04-25T16:05:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.500+0000 7fa779410740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561b7b515a20 con 0x561b7b22c4d0 2024-04-25T16:05:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.500+0000 7fa779410740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561b7b557d40 con 0x561b7b2346b0 2024-04-25T16:05:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.501+0000 7fa770afb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b2346b0 0x561b7b5f8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.501+0000 7fa770afb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b2346b0 0x561b7b5f8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52230/0 (socket says 172.21.15.31:52230) 2024-04-25T16:05:03.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.501+0000 7fa770afb640 1 -- 172.21.15.31:0/2092505413 learned_addr learned my addr 172.21.15.31:0/2092505413 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:03.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.502+0000 7fa770afb640 1 -- 172.21.15.31:0/2092505413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 msgr2=0x561b7b5f7f60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:03.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.502+0000 7fa770afb640 1 --2- 172.21.15.31:0/2092505413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b5f7f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.502+0000 7fa770afb640 1 -- 172.21.15.31:0/2092505413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b255390 msgr2=0x561b7b5f79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.502+0000 7fa770afb640 1 --2- 172.21.15.31:0/2092505413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b255390 0x561b7b5f79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.502+0000 7fa770afb640 1 -- 172.21.15.31:0/2092505413 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b7b520c30 con 0x561b7b2346b0 2024-04-25T16:05:03.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.504+0000 7fa770afb640 1 --2- 172.21.15.31:0/2092505413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b2346b0 0x561b7b5f8bf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa764007c50 tx=0x7fa76400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e84b5ba9b5d5524 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.507+0000 7fa7527fc640 1 -- 172.21.15.31:0/2092505413 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa764014070 con 0x561b7b2346b0 2024-04-25T16:05:03.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.507+0000 7fa7527fc640 1 -- 172.21.15.31:0/2092505413 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa76400fb00 con 0x561b7b2346b0 2024-04-25T16:05:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.508+0000 7fa779410740 1 -- 172.21.15.31:0/2092505413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b2346b0 msgr2=0x561b7b5f8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.508+0000 7fa779410740 1 --2- 172.21.15.31:0/2092505413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b2346b0 0x561b7b5f8bf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa764007c50 tx=0x7fa76400c6a0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.509+0000 7fa779410740 1 -- 172.21.15.31:0/2092505413 shutdown_connections 2024-04-25T16:05:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.509+0000 7fa779410740 1 --2- 172.21.15.31:0/2092505413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b2346b0 0x561b7b5f8bf0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.509+0000 7fa779410740 1 --2- 172.21.15.31:0/2092505413 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b5f7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.509+0000 7fa779410740 1 --2- 172.21.15.31:0/2092505413 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b255390 0x561b7b5f79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.509+0000 7fa779410740 1 -- 172.21.15.31:0/2092505413 >> 172.21.15.31:0/2092505413 conn(0x561b7b2494e0 msgr2=0x561b7b5f31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:03.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.511+0000 7fa779410740 1 -- 172.21.15.31:0/2092505413 shutdown_connections 2024-04-25T16:05:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.512+0000 7fa779410740 1 -- 172.21.15.31:0/2092505413 wait complete. 2024-04-25T16:05:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.513+0000 7fa779410740 1 -- start start 2024-04-25T16:05:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.513+0000 7fa779410740 1 Processor -- start 2024-04-25T16:05:03.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.516+0000 7fa779410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b22c4d0 0x561b7b601660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.516+0000 7fa779410740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b255390 0x561b7b601ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.516+0000 7fa779410740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b608b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.516+0000 7fa779410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561b7b5f9750 con 0x561b7b22c4d0 2024-04-25T16:05:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.516+0000 7fa779410740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561b7b4e2dc0 con 0x561b7b255390 2024-04-25T16:05:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.516+0000 7fa779410740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561b7b5f98d0 con 0x561b7b213f60 2024-04-25T16:05:03.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.516+0000 7fa779410740 10 client.0 _ll_register_callbacks cb 0x561b7b4fe600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:03.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.517+0000 7fa770afb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b608b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.517+0000 7fa76b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b255390 0x561b7b601ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.517+0000 7fa76b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b255390 0x561b7b601ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37036/0 (socket says 172.21.15.31:37036) 2024-04-25T16:05:03.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.517+0000 7fa76b7fe640 1 -- 172.21.15.31:0/1015883302 learned_addr learned my addr 172.21.15.31:0/1015883302 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.517+0000 7fa76b7fe640 1 -- 172.21.15.31:0/1015883302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 msgr2=0x561b7b608b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.517+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b608b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.517+0000 7fa76b7fe640 1 -- 172.21.15.31:0/1015883302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b22c4d0 msgr2=0x561b7b601660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.517+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b22c4d0 0x561b7b601660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.517+0000 7fa76b7fe640 1 -- 172.21.15.31:0/1015883302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b7b26d3f0 con 0x561b7b255390 2024-04-25T16:05:03.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.519+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b255390 0x561b7b601ba0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7600c8ab0 tx=0x7fa7600c8630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac3443da05d6e8d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.520+0000 7fa74b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:05:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.520+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1015883302 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa760049040 con 0x561b7b255390 2024-04-25T16:05:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.521+0000 7fa779410740 1 -- 172.21.15.31:0/1015883302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561b7b569990 con 0x561b7b255390 2024-04-25T16:05:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.521+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1015883302 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa7600c4d20 con 0x561b7b255390 2024-04-25T16:05:03.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.521+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1015883302 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa7600e31e0 con 0x561b7b255390 2024-04-25T16:05:03.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.523+0000 7fa779410740 10 client.26066 fetch_fsmap learned FSMap version 28 2024-04-25T16:05:03.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.523+0000 7fa779410740 1 -- 172.21.15.31:0/1015883302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561b7b3377d0 con 0x561b7b255390 2024-04-25T16:05:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.524+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1015883302 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x561b7b569990 con 0x561b7b255390 2024-04-25T16:05:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.524+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1015883302 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fa7600c6a70 con 0x561b7b255390 2024-04-25T16:05:03.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.524+0000 7fa779410740 10 client.26066 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:05:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.524+0000 7fa779410740 10 client.26066 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:05:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.524+0000 7fa779410740 20 client.26066 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.524+0000 7fa779410740 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x561b7b1e93a0 0x561b7b60f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.524+0000 7fa779410740 4 client.26066 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:03.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.524+0000 7fa779410740 1 -- 172.21.15.31:0/1015883302 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b7b563800 con 0x561b7b1e93a0 2024-04-25T16:05:03.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.527+0000 7fa76bfff640 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x561b7b1e93a0 0x561b7b60f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.531+0000 7fa76bfff640 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x561b7b1e93a0 0x561b7b60f580 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bd26223430898fe3 server_cookie=2b4e0dc2f59b7316 in_seq=0 out_seq=0 2024-04-25T16:05:03.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.532+0000 7fa74b7fe640 10 client.26066 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:05:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.534+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1015883302 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561b7b563800 con 0x561b7b1e93a0 2024-04-25T16:05:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.534+0000 7fa74b7fe640 10 client.26066 handle_command_reply: tid=0 2024-04-25T16:05:03.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa779410740 1 client.26066 shutdown 2024-04-25T16:05:03.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa779410740 1 -- 172.21.15.31:0/1015883302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b255390 msgr2=0x561b7b601ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa779410740 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b255390 0x561b7b601ba0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa7600c8ab0 tx=0x7fa7600c8630 comp rx=0 tx=0).stop 2024-04-25T16:05:03.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa779410740 1 -- 172.21.15.31:0/1015883302 shutdown_connections 2024-04-25T16:05:03.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa779410740 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b608b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa779410740 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b255390 0x561b7b601ba0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa779410740 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x561b7b1e93a0 0x561b7b60f580 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa779410740 1 --2- 172.21.15.31:0/1015883302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b22c4d0 0x561b7b601660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa779410740 1 -- 172.21.15.31:0/1015883302 >> 172.21.15.31:0/1015883302 conn(0x561b7b2494e0 msgr2=0x561b7b5f23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:03.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.535+0000 7fa74b7fe640 0 client.26066 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:05:03.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.541+0000 7fa779410740 1 -- 172.21.15.31:0/1015883302 shutdown_connections 2024-04-25T16:05:03.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.547+0000 7fa779410740 1 -- 172.21.15.31:0/1015883302 wait complete. 2024-04-25T16:05:03.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.547+0000 7fa779410740 20 client.26066 trim_cache size 0 max 16384 2024-04-25T16:05:03.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.588+0000 7fa779410740 1 Processor -- start 2024-04-25T16:05:03.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.588+0000 7fa779410740 1 -- start start 2024-04-25T16:05:03.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.589+0000 7fa779410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b213f60 0x561b7b5fc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.589+0000 7fa779410740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b608a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.589+0000 7fa779410740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b255390 0x561b7b5fc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.590+0000 7fa76bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b213f60 0x561b7b5fc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.590+0000 7fa76bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b213f60 0x561b7b5fc290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38946/0 (socket says 172.21.15.31:38946) 2024-04-25T16:05:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.590+0000 7fa76b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b608a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.590+0000 7fa76b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b608a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37042/0 (socket says 172.21.15.31:37042) 2024-04-25T16:05:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.590+0000 7fa76bfff640 1 -- 172.21.15.31:0/2596422846 learned_addr learned my addr 172.21.15.31:0/2596422846 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.590+0000 7fa779410740 1 -- 172.21.15.31:0/2596422846 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa75800e4f0 con 0x561b7b213f60 2024-04-25T16:05:03.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.590+0000 7fa779410740 1 -- 172.21.15.31:0/2596422846 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa758010060 con 0x561b7b22c4d0 2024-04-25T16:05:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.590+0000 7fa779410740 1 -- 172.21.15.31:0/2596422846 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa75800ebc0 con 0x561b7b255390 2024-04-25T16:05:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.591+0000 7fa76b7fe640 1 -- 172.21.15.31:0/2596422846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b255390 msgr2=0x561b7b5fc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.591+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/2596422846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b255390 0x561b7b5fc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.591+0000 7fa76b7fe640 1 -- 172.21.15.31:0/2596422846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b213f60 msgr2=0x561b7b5fc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.591+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/2596422846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b213f60 0x561b7b5fc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.591+0000 7fa76b7fe640 1 -- 172.21.15.31:0/2596422846 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b7b5e4ba0 con 0x561b7b22c4d0 2024-04-25T16:05:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.592+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/2596422846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b608a20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa76003d040 tx=0x7fa76003b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f88b90d7439c52d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.592+0000 7fa751ffb640 1 -- 172.21.15.31:0/2596422846 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa760049070 con 0x561b7b22c4d0 2024-04-25T16:05:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.592+0000 7fa751ffb640 1 -- 172.21.15.31:0/2596422846 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7600c9e50 con 0x561b7b22c4d0 2024-04-25T16:05:03.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.592+0000 7fa751ffb640 1 -- 172.21.15.31:0/2596422846 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7600c4ea0 con 0x561b7b22c4d0 2024-04-25T16:05:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.595+0000 7fa779410740 1 -- 172.21.15.31:0/2596422846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 msgr2=0x561b7b608a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.595+0000 7fa779410740 1 --2- 172.21.15.31:0/2596422846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b608a20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa76003d040 tx=0x7fa76003b880 comp rx=0 tx=0).stop 2024-04-25T16:05:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.595+0000 7fa779410740 1 -- 172.21.15.31:0/2596422846 shutdown_connections 2024-04-25T16:05:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.596+0000 7fa779410740 1 --2- 172.21.15.31:0/2596422846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b255390 0x561b7b5fc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.596+0000 7fa779410740 1 --2- 172.21.15.31:0/2596422846 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b608a20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.596+0000 7fa779410740 1 --2- 172.21.15.31:0/2596422846 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b213f60 0x561b7b5fc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.596+0000 7fa779410740 1 -- 172.21.15.31:0/2596422846 >> 172.21.15.31:0/2596422846 conn(0x561b7b2494e0 msgr2=0x561b7b60f4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.596+0000 7fa779410740 1 -- 172.21.15.31:0/2596422846 shutdown_connections 2024-04-25T16:05:03.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.596+0000 7fa779410740 1 -- 172.21.15.31:0/2596422846 wait complete. 2024-04-25T16:05:03.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.597+0000 7fa779410740 1 -- start start 2024-04-25T16:05:03.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.598+0000 7fa779410740 1 Processor -- start 2024-04-25T16:05:03.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.599+0000 7fa779410740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b605aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.599+0000 7fa779410740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b605fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.599+0000 7fa779410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b255390 0x561b7b6758e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.600+0000 7fa779410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa75800f960 con 0x561b7b255390 2024-04-25T16:05:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.600+0000 7fa76bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b605aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.600+0000 7fa779410740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa75800f290 con 0x561b7b22c4d0 2024-04-25T16:05:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.600+0000 7fa779410740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa75800c4d0 con 0x561b7b213f60 2024-04-25T16:05:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.600+0000 7fa76bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b605aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52262/0 (socket says 172.21.15.31:52262) 2024-04-25T16:05:03.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.600+0000 7fa76bfff640 1 -- 172.21.15.31:0/1658491550 learned_addr learned my addr 172.21.15.31:0/1658491550 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.600+0000 7fa779410740 10 client.0 _ll_register_callbacks cb 0x7fa760005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.600+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b605fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.600+0000 7fa770afb640 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b255390 0x561b7b6758e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.601+0000 7fa76bfff640 1 -- 172.21.15.31:0/1658491550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 msgr2=0x561b7b605fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.601+0000 7fa76bfff640 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b605fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.601+0000 7fa76bfff640 1 -- 172.21.15.31:0/1658491550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b255390 msgr2=0x561b7b6758e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.601+0000 7fa76bfff640 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b255390 0x561b7b6758e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.601+0000 7fa76bfff640 1 -- 172.21.15.31:0/1658491550 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b7b675e70 con 0x561b7b213f60 2024-04-25T16:05:03.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.602+0000 7fa76bfff640 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b605aa0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa75c013c20 tx=0x7fa75c004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b861cf7ed976c208 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.603+0000 7fa74b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.604+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1658491550 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c013e60 con 0x561b7b213f60 2024-04-25T16:05:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.604+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1658491550 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa75c02c680 con 0x561b7b213f60 2024-04-25T16:05:03.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.604+0000 7fa779410740 1 -- 172.21.15.31:0/1658491550 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7600c9e50 con 0x561b7b213f60 2024-04-25T16:05:03.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.614+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1658491550 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa75c02c880 con 0x561b7b213f60 2024-04-25T16:05:03.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.617+0000 7fa779410740 10 client.26020 fetch_fsmap learned FSMap version 28 2024-04-25T16:05:03.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.617+0000 7fa779410740 1 -- 172.21.15.31:0/1658491550 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561b7b6771c0 con 0x561b7b213f60 2024-04-25T16:05:03.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:03 smithi063 ceph-mon[32413]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 760 MiB/s wr, 2.10k op/s 2024-04-25T16:05:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.621+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1658491550 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7600c9e50 con 0x561b7b213f60 2024-04-25T16:05:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.621+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1658491550 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7fa75c012070 con 0x561b7b213f60 2024-04-25T16:05:03.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.621+0000 7fa779410740 10 client.26020 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:05:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.621+0000 7fa779410740 10 client.26020 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:05:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.621+0000 7fa779410740 20 client.26020 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:03.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.621+0000 7fa779410740 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x561b7b677b80 0x561b7b679fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:03.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.623+0000 7fa779410740 4 client.26020 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:03.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.623+0000 7fa779410740 1 -- 172.21.15.31:0/1658491550 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561b7b4de610 con 0x561b7b677b80 2024-04-25T16:05:03.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.624+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x561b7b677b80 0x561b7b679fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:03.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.626+0000 7fa74b7fe640 10 client.26020 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:05:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.626+0000 7fa76b7fe640 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x561b7b677b80 0x561b7b679fa0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a23ca77cef3fd317 server_cookie=745bdf6225df77ed in_seq=0 out_seq=0 2024-04-25T16:05:03.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.627+0000 7fa74b7fe640 1 -- 172.21.15.31:0/1658491550 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561b7b4de610 con 0x561b7b677b80 2024-04-25T16:05:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.627+0000 7fa74b7fe640 10 client.26020 handle_command_reply: tid=0 2024-04-25T16:05:03.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.628+0000 7fa779410740 1 client.26020 shutdown 2024-04-25T16:05:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.629+0000 7fa779410740 1 -- 172.21.15.31:0/1658491550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 msgr2=0x561b7b605aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.629+0000 7fa779410740 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b605aa0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa75c013c20 tx=0x7fa75c004700 comp rx=0 tx=0).stop 2024-04-25T16:05:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.629+0000 7fa779410740 1 -- 172.21.15.31:0/1658491550 shutdown_connections 2024-04-25T16:05:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.629+0000 7fa779410740 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x561b7b677b80 0x561b7b679fa0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.629+0000 7fa779410740 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b7b255390 0x561b7b6758e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.630+0000 7fa779410740 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b7b22c4d0 0x561b7b605fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.630+0000 7fa779410740 1 --2- 172.21.15.31:0/1658491550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b7b213f60 0x561b7b605aa0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.630+0000 7fa779410740 1 -- 172.21.15.31:0/1658491550 >> 172.21.15.31:0/1658491550 conn(0x561b7b2494e0 msgr2=0x561b7b601660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.635+0000 7fa74b7fe640 0 client.26020 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:05:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.635+0000 7fa779410740 1 -- 172.21.15.31:0/1658491550 shutdown_connections 2024-04-25T16:05:03.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.635+0000 7fa779410740 1 -- 172.21.15.31:0/1658491550 wait complete. 2024-04-25T16:05:03.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.636+0000 7fa779410740 20 client.26020 trim_cache size 0 max 16384 2024-04-25T16:05:03.637 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:03.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:03 smithi186 ceph-mon[33258]: pgmap v463: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 760 MiB/s wr, 2.10k op/s 2024-04-25T16:05:03.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.652+0000 7fa7712fc640 1 -- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7540afcf0 msgr2=0x7fa7540b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.652+0000 7fa7712fc640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7540afcf0 0x7fa7540b2200 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa76c10dba0 tx=0x7fa75c006bd0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.652+0000 7fa7712fc640 1 -- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c114290 msgr2=0x7fa76c11bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:03.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.652+0000 7fa7712fc640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c114290 0x7fa76c11bb30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fa760031440 tx=0x7fa760003d60 comp rx=0 tx=0).stop 2024-04-25T16:05:03.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.653+0000 7fa7712fc640 1 -- 172.21.15.31:0/2637968031 shutdown_connections 2024-04-25T16:05:03.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.653+0000 7fa7712fc640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7540afcf0 0x7fa7540b2200 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.653+0000 7fa7712fc640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa76c11eaa0 0x7fa76c11eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.653+0000 7fa7712fc640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa76c114290 0x7fa76c11bb30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.653+0000 7fa7712fc640 1 --2- 172.21.15.31:0/2637968031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa76c10e210 0x7fa76c11b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:03.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.653+0000 7fa7712fc640 1 -- 172.21.15.31:0/2637968031 >> 172.21.15.31:0/2637968031 conn(0x7fa76c06e0b0 msgr2=0x7fa76c115b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.661+0000 7fa7712fc640 1 -- 172.21.15.31:0/2637968031 shutdown_connections 2024-04-25T16:05:03.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:03.661+0000 7fa7712fc640 1 -- 172.21.15.31:0/2637968031 wait complete. 2024-04-25T16:05:03.739 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:05:04.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.148+0000 7f2149664640 1 -- 172.21.15.31:0/110917553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f214410a340 msgr2=0x7f214410ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:04.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.148+0000 7f2149664640 1 --2- 172.21.15.31:0/110917553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f214410a340 0x7f214410ac50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f2138009900 tx=0x7f213802a440 comp rx=0 tx=0).stop 2024-04-25T16:05:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.159+0000 7f2149664640 1 -- 172.21.15.31:0/110917553 shutdown_connections 2024-04-25T16:05:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.159+0000 7f2149664640 1 --2- 172.21.15.31:0/110917553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f214410fe90 0x7f21441102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.159+0000 7f2149664640 1 --2- 172.21.15.31:0/110917553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f214410b190 0x7f214410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.159+0000 7f2149664640 1 --2- 172.21.15.31:0/110917553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f214410a340 0x7f214410ac50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.159+0000 7f2149664640 1 -- 172.21.15.31:0/110917553 >> 172.21.15.31:0/110917553 conn(0x7f2144070320 msgr2=0x7f21440727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:04.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.175+0000 7f2149664640 1 -- 172.21.15.31:0/110917553 shutdown_connections 2024-04-25T16:05:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.186+0000 7f2149664640 1 -- 172.21.15.31:0/110917553 wait complete. 2024-04-25T16:05:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.187+0000 7f2149664640 1 Processor -- start 2024-04-25T16:05:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.187+0000 7f2149664640 1 -- start start 2024-04-25T16:05:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.188+0000 7f2149664640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f214410a340 0x7f21441a8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.188+0000 7f2149664640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f214410b190 0x7f21441a9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.188+0000 7f2149664640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f214410fe90 0x7f214407ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.188+0000 7f2149664640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f214411e380 con 0x7f214410fe90 2024-04-25T16:05:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.188+0000 7f2149664640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f214411e200 con 0x7f214410a340 2024-04-25T16:05:04.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.188+0000 7f2149664640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f214411e500 con 0x7f214410b190 2024-04-25T16:05:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.190+0000 7f2143fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f214410a340 0x7f21441a8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.190+0000 7f2143fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f214410a340 0x7f21441a8ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34720/0 (socket says 172.21.15.31:34720) 2024-04-25T16:05:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.190+0000 7f2143fff640 1 -- 172.21.15.31:0/3374130125 learned_addr learned my addr 172.21.15.31:0/3374130125 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.191+0000 7f21437fe640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f214410b190 0x7f21441a9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.191+0000 7f21437fe640 1 -- 172.21.15.31:0/3374130125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f214410a340 msgr2=0x7f21441a8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.191+0000 7f21437fe640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f214410a340 0x7f21441a8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.192+0000 7f21437fe640 1 -- 172.21.15.31:0/3374130125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f214410fe90 msgr2=0x7f214407ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.192+0000 7f21437fe640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f214410fe90 0x7f214407ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.192+0000 7f21437fe640 1 -- 172.21.15.31:0/3374130125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214407b470 con 0x7f214410b190 2024-04-25T16:05:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.194+0000 7f21437fe640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f214410b190 0x7f21441a9460 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f213400b3f0 tx=0x7f213400b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.195+0000 7f21417fa640 1 -- 172.21.15.31:0/3374130125 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2134015020 con 0x7f214410b190 2024-04-25T16:05:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.196+0000 7f2149664640 1 -- 172.21.15.31:0/3374130125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2144111b60 con 0x7f214410b190 2024-04-25T16:05:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.196+0000 7f2149664640 1 -- 172.21.15.31:0/3374130125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21441120d0 con 0x7f214410b190 2024-04-25T16:05:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.196+0000 7f21417fa640 1 -- 172.21.15.31:0/3374130125 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2134011ce0 con 0x7f214410b190 2024-04-25T16:05:04.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.196+0000 7f21417fa640 1 -- 172.21.15.31:0/3374130125 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2134010d80 con 0x7f214410b190 2024-04-25T16:05:04.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.206+0000 7f21417fa640 1 -- 172.21.15.31:0/3374130125 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2134018020 con 0x7f214410b190 2024-04-25T16:05:04.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.208+0000 7f2151776740 1 Processor -- start 2024-04-25T16:05:04.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.216+0000 7f2151776740 1 -- start start 2024-04-25T16:05:04.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.216+0000 7f2151776740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b927390 0x56385bcc99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.216+0000 7f2151776740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcc9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.216+0000 7f2151776740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b9066b0 0x56385bccabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.216+0000 7f2151776740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56385bc2dd80 con 0x56385b8fe4d0 2024-04-25T16:05:04.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.216+0000 7f2151776740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56385bbe7a20 con 0x56385b927390 2024-04-25T16:05:04.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.216+0000 7f2151776740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56385bc29d40 con 0x56385b9066b0 2024-04-25T16:05:04.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.217+0000 7f2148e63640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b927390 0x56385bcc99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.218+0000 7f2148e63640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b927390 0x56385bcc99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34728/0 (socket says 172.21.15.31:34728) 2024-04-25T16:05:04.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.218+0000 7f2148e63640 1 -- 172.21.15.31:0/2548202267 learned_addr learned my addr 172.21.15.31:0/2548202267 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:04.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.218+0000 7f21417fa640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f212c0afd30 0x7f212c0b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.219+0000 7f2148e63640 1 -- 172.21.15.31:0/2548202267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b9066b0 msgr2=0x56385bccabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:04.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.219+0000 7f2148e63640 1 --2- 172.21.15.31:0/2548202267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b9066b0 0x56385bccabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.219+0000 7f2148e63640 1 -- 172.21.15.31:0/2548202267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 msgr2=0x56385bcc9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:04.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.219+0000 7f2148e63640 1 --2- 172.21.15.31:0/2548202267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcc9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.219+0000 7f2148e63640 1 -- 172.21.15.31:0/2548202267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385bbf2c30 con 0x56385b927390 2024-04-25T16:05:04.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.219+0000 7f2148e63640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f212c0afd30 0x7f212c0b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.220+0000 7f21417fa640 1 -- 172.21.15.31:0/3374130125 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f21340df380 con 0x7f214410b190 2024-04-25T16:05:04.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.225+0000 7f2148e63640 1 --2- 172.21.15.31:0/2548202267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b927390 0x56385bcc99e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f213c009ee0 tx=0x7f213c00b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a470c52d3da20ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.230+0000 7f21327fc640 1 -- 172.21.15.31:0/2548202267 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213c0147f0 con 0x56385b927390 2024-04-25T16:05:04.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.230+0000 7f21327fc640 1 -- 172.21.15.31:0/2548202267 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f213c00be40 con 0x56385b927390 2024-04-25T16:05:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.232+0000 7f2151776740 1 -- 172.21.15.31:0/2548202267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b927390 msgr2=0x56385bcc99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.232+0000 7f2151776740 1 --2- 172.21.15.31:0/2548202267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b927390 0x56385bcc99e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f213c009ee0 tx=0x7f213c00b420 comp rx=0 tx=0).stop 2024-04-25T16:05:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.232+0000 7f2151776740 1 -- 172.21.15.31:0/2548202267 shutdown_connections 2024-04-25T16:05:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.232+0000 7f2151776740 1 --2- 172.21.15.31:0/2548202267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b9066b0 0x56385bccabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.232+0000 7f2151776740 1 --2- 172.21.15.31:0/2548202267 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcc9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.232+0000 7f2151776740 1 --2- 172.21.15.31:0/2548202267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b927390 0x56385bcc99e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.232+0000 7f2151776740 1 -- 172.21.15.31:0/2548202267 >> 172.21.15.31:0/2548202267 conn(0x56385b91b4e0 msgr2=0x56385bcc51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:04.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.237+0000 7f2148e63640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f212c0afd30 0x7f212c0b2240 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f213c014000 tx=0x7f213c016000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:04.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.238+0000 7f2151776740 1 -- 172.21.15.31:0/2548202267 shutdown_connections 2024-04-25T16:05:04.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.239+0000 7f2151776740 1 -- 172.21.15.31:0/2548202267 wait complete. 2024-04-25T16:05:04.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.239+0000 7f2151776740 1 -- start start 2024-04-25T16:05:04.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.240+0000 7f2151776740 1 Processor -- start 2024-04-25T16:05:04.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.245+0000 7f2151776740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8fe4d0 0x56385bcd3680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.246+0000 7f2151776740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b9066b0 0x56385bcd3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.246+0000 7f2151776740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 0x56385bcdab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.246+0000 7f2151776740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56385bccb750 con 0x56385b9066b0 2024-04-25T16:05:04.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.246+0000 7f2151776740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56385bbb4dc0 con 0x56385b8fe4d0 2024-04-25T16:05:04.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.246+0000 7f2151776740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56385bccb8d0 con 0x56385b8e5f60 2024-04-25T16:05:04.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.246+0000 7f2151776740 10 client.0 _ll_register_callbacks cb 0x56385bbd0600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:04.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.248+0000 7f21437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b9066b0 0x56385bcd3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.248+0000 7f21437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b9066b0 0x56385bcd3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53050/0 (socket says 172.21.15.31:53050) 2024-04-25T16:05:04.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.248+0000 7f21437fe640 1 -- 172.21.15.31:0/1471910711 learned_addr learned my addr 172.21.15.31:0/1471910711 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:04.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.249+0000 7f2143fff640 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8fe4d0 0x56385bcd3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.249+0000 7f2148e63640 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 0x56385bcdab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.253+0000 7f2143fff640 1 -- 172.21.15.31:0/1471910711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 msgr2=0x56385bcdab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.253+0000 7f2143fff640 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 0x56385bcdab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.253+0000 7f2143fff640 1 -- 172.21.15.31:0/1471910711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b9066b0 msgr2=0x56385bcd3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.253+0000 7f2143fff640 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b9066b0 0x56385bcd3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.253+0000 7f2143fff640 1 -- 172.21.15.31:0/1471910711 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385b93f3f0 con 0x56385b8fe4d0 2024-04-25T16:05:04.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.257+0000 7f2143fff640 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8fe4d0 0x56385bcd3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f21441aa390 tx=0x7f2138031d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=303fa0998193ada9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:04.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.257+0000 7f21277fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:05:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.257+0000 7f21277fe640 1 -- 172.21.15.31:0/1471910711 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2138040070 con 0x56385b8fe4d0 2024-04-25T16:05:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.258+0000 7f2151776740 1 -- 172.21.15.31:0/1471910711 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56385bc3b990 con 0x56385b8fe4d0 2024-04-25T16:05:04.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.258+0000 7f21277fe640 1 -- 172.21.15.31:0/1471910711 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2138004440 con 0x56385b8fe4d0 2024-04-25T16:05:04.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.260+0000 7f21277fe640 1 -- 172.21.15.31:0/1471910711 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f213804c050 con 0x56385b8fe4d0 2024-04-25T16:05:04.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.260+0000 7f2151776740 10 client.26081 fetch_fsmap learned FSMap version 28 2024-04-25T16:05:04.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.260+0000 7f2151776740 1 -- 172.21.15.31:0/1471910711 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56385ba097d0 con 0x56385b8fe4d0 2024-04-25T16:05:04.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.262+0000 7f21277fe640 1 -- 172.21.15.31:0/1471910711 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56385bc3b990 con 0x56385b8fe4d0 2024-04-25T16:05:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.264+0000 7f21277fe640 1 -- 172.21.15.31:0/1471910711 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 28) ==== 8358+0+0 (secure 0 0 0) 0x7f213803c5c0 con 0x56385b8fe4d0 2024-04-25T16:05:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.264+0000 7f2151776740 10 client.26081 fetch_fsmap finished waiting for FSMap version 28 2024-04-25T16:05:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.264+0000 7f2151776740 10 client.26081 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.264+0000 7f2151776740 20 client.26081 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.264+0000 7f2151776740 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56385b8bb3a0 0x56385bce1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.264+0000 7f2151776740 4 client.26081 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:04.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.264+0000 7f2151776740 1 -- 172.21.15.31:0/1471910711 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56385bc35800 con 0x56385b8bb3a0 2024-04-25T16:05:04.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.270+0000 7f21437fe640 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56385b8bb3a0 0x56385bce1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.296+0000 7f21437fe640 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56385b8bb3a0 0x56385bce1830 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a5d837f79529e0d server_cookie=b5f68a90422ec71c in_seq=0 out_seq=0 2024-04-25T16:05:04.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.298+0000 7f21277fe640 10 client.26081 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:04.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.303+0000 7f21277fe640 1 -- 172.21.15.31:0/1471910711 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56385bc35800 con 0x56385b8bb3a0 2024-04-25T16:05:04.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.303+0000 7f21277fe640 10 client.26081 handle_command_reply: tid=0 2024-04-25T16:05:04.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.303+0000 7f2151776740 1 client.26081 shutdown 2024-04-25T16:05:04.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.304+0000 7f2151776740 1 -- 172.21.15.31:0/1471910711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8fe4d0 msgr2=0x56385bcd3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:04.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.304+0000 7f2151776740 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8fe4d0 0x56385bcd3680 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f21441aa390 tx=0x7f2138031d30 comp rx=0 tx=0).stop 2024-04-25T16:05:04.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.304+0000 7f2151776740 1 -- 172.21.15.31:0/1471910711 shutdown_connections 2024-04-25T16:05:04.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.304+0000 7f2151776740 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56385b8bb3a0 0x56385bce1830 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.304+0000 7f2151776740 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 0x56385bcdab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.304+0000 7f2151776740 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b9066b0 0x56385bcd3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.304+0000 7f2151776740 1 --2- 172.21.15.31:0/1471910711 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8fe4d0 0x56385bcd3680 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.304+0000 7f2151776740 1 -- 172.21.15.31:0/1471910711 >> 172.21.15.31:0/1471910711 conn(0x56385b91b4e0 msgr2=0x56385bcc43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:04.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.304+0000 7f21277fe640 0 client.26081 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:04.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.308+0000 7f2151776740 1 -- 172.21.15.31:0/1471910711 shutdown_connections 2024-04-25T16:05:04.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.308+0000 7f2151776740 1 -- 172.21.15.31:0/1471910711 wait complete. 2024-04-25T16:05:04.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.308+0000 7f2151776740 20 client.26081 trim_cache size 0 max 16384 2024-04-25T16:05:04.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.399+0000 7f2151776740 1 Processor -- start 2024-04-25T16:05:04.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.404+0000 7f2151776740 1 -- start start 2024-04-25T16:05:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.404+0000 7f2151776740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 0x56385bcce910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.404+0000 7f2151776740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcc8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.404+0000 7f2151776740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b9066b0 0x56385bcc8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.404+0000 7f2151776740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f212800e3e0 con 0x56385b8fe4d0 2024-04-25T16:05:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.404+0000 7f2151776740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f212800ff50 con 0x56385b9066b0 2024-04-25T16:05:04.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.404+0000 7f2151776740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f212800eab0 con 0x56385b8e5f60 2024-04-25T16:05:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.407+0000 7f2148e63640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b9066b0 0x56385bcc8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.407+0000 7f2143fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 0x56385bcce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.409+0000 7f2143fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 0x56385bcce910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37216/0 (socket says 172.21.15.31:37216) 2024-04-25T16:05:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.409+0000 7f2143fff640 1 -- 172.21.15.31:0/964222151 learned_addr learned my addr 172.21.15.31:0/964222151 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:04.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.411+0000 7f21437fe640 1 --2- 172.21.15.31:0/964222151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcc8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.415+0000 7f2148e63640 1 -- 172.21.15.31:0/964222151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 msgr2=0x56385bcce910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.415+0000 7f2148e63640 1 --2- 172.21.15.31:0/964222151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 0x56385bcce910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.415+0000 7f2148e63640 1 -- 172.21.15.31:0/964222151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 msgr2=0x56385bcc8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.415+0000 7f2148e63640 1 --2- 172.21.15.31:0/964222151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcc8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.415+0000 7f2148e63640 1 -- 172.21.15.31:0/964222151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56385bcb6ba0 con 0x56385b9066b0 2024-04-25T16:05:04.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.416+0000 7f2148e63640 1 --2- 172.21.15.31:0/964222151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b9066b0 0x56385bcc8c40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x56385bccaa30 tx=0x7f213c004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=184dd18fd8bb9e35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:04.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.418+0000 7f2131ffb640 1 -- 172.21.15.31:0/964222151 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213c01e660 con 0x56385b9066b0 2024-04-25T16:05:04.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.418+0000 7f2131ffb640 1 -- 172.21.15.31:0/964222151 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f213c02e070 con 0x56385b9066b0 2024-04-25T16:05:04.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.419+0000 7f2131ffb640 1 -- 172.21.15.31:0/964222151 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f213c029470 con 0x56385b9066b0 2024-04-25T16:05:04.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.424+0000 7f2151776740 1 -- 172.21.15.31:0/964222151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b9066b0 msgr2=0x56385bcc8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:04.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.424+0000 7f2151776740 1 --2- 172.21.15.31:0/964222151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b9066b0 0x56385bcc8c40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x56385bccaa30 tx=0x7f213c004b40 comp rx=0 tx=0).stop 2024-04-25T16:05:04.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.426+0000 7f2151776740 1 -- 172.21.15.31:0/964222151 shutdown_connections 2024-04-25T16:05:04.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.426+0000 7f2151776740 1 --2- 172.21.15.31:0/964222151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b9066b0 0x56385bcc8c40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.426+0000 7f2151776740 1 --2- 172.21.15.31:0/964222151 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcc8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.426+0000 7f2151776740 1 --2- 172.21.15.31:0/964222151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b8e5f60 0x56385bcce910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.426+0000 7f2151776740 1 -- 172.21.15.31:0/964222151 >> 172.21.15.31:0/964222151 conn(0x56385b91b4e0 msgr2=0x56385bcc7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:04.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.426+0000 7f2151776740 1 -- 172.21.15.31:0/964222151 shutdown_connections 2024-04-25T16:05:04.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.426+0000 7f2151776740 1 -- 172.21.15.31:0/964222151 wait complete. 2024-04-25T16:05:04.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.435+0000 7f2151776740 1 -- start start 2024-04-25T16:05:04.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.436+0000 7f2151776740 1 Processor -- start 2024-04-25T16:05:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f2151776740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8e5f60 0x56385bcd8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f2151776740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcd9320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f2151776740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b9066b0 0x56385bd47b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f21437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcd9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f21437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcd9320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53060/0 (socket says 172.21.15.31:53060) 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f21437fe640 1 -- 172.21.15.31:0/1208458491 learned_addr learned my addr 172.21.15.31:0/1208458491 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f212800f850 con 0x56385b8fe4d0 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f212800f180 con 0x56385b8e5f60 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f212800c470 con 0x56385b9066b0 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f2151776740 10 client.0 _ll_register_callbacks cb 0x7f2138009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f21437fe640 1 -- 172.21.15.31:0/1208458491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b9066b0 msgr2=0x56385bd47b90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f21437fe640 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b9066b0 0x56385bd47b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f21437fe640 1 -- 172.21.15.31:0/1208458491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8e5f60 msgr2=0x56385bcd8de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.437+0000 7f21437fe640 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8e5f60 0x56385bcd8de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.438+0000 7f21437fe640 1 -- 172.21.15.31:0/1208458491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385bcd9bc0 con 0x56385b8fe4d0 2024-04-25T16:05:04.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.443+0000 7f21437fe640 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcd9320 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f21340a3840 tx=0x7f21340b0040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9cb86a29abdc403 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:04.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.445+0000 7f21277fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:04.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.456+0000 7f21277fe640 1 -- 172.21.15.31:0/1208458491 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21340abbc0 con 0x56385b8fe4d0 2024-04-25T16:05:04.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.457+0000 7f21277fe640 1 -- 172.21.15.31:0/1208458491 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f21340a3aa0 con 0x56385b8fe4d0 2024-04-25T16:05:04.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.457+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f213c02e070 con 0x56385b8fe4d0 2024-04-25T16:05:04.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.465+0000 7f21277fe640 1 -- 172.21.15.31:0/1208458491 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f21340b0670 con 0x56385b8fe4d0 2024-04-25T16:05:04.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.465+0000 7f2151776740 10 client.16392 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:04.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.465+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56385bd491e0 con 0x56385b8fe4d0 2024-04-25T16:05:04.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.468+0000 7f21277fe640 1 -- 172.21.15.31:0/1208458491 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f213c02e070 con 0x56385b8fe4d0 2024-04-25T16:05:04.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.470+0000 7f21277fe640 1 -- 172.21.15.31:0/1208458491 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f21340de040 con 0x56385b8fe4d0 2024-04-25T16:05:04.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.470+0000 7f2151776740 10 client.16392 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:04.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.470+0000 7f2151776740 10 client.16392 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:04.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.470+0000 7f2151776740 20 client.16392 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:04.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.470+0000 7f2151776740 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56385bd49b40 0x56385bd4bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:04.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.470+0000 7f2151776740 4 client.16392 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:04.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.470+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56385bbb0610 con 0x56385bd49b40 2024-04-25T16:05:04.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.475+0000 7f2143fff640 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56385bd49b40 0x56385bd4bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.490+0000 7f2143fff640 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56385bd49b40 0x56385bd4bf80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94a88a44377b6955 server_cookie=acbcc1fd85d78a27 in_seq=0 out_seq=0 2024-04-25T16:05:04.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.490+0000 7f21277fe640 10 client.16392 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:04.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.497+0000 7f21277fe640 1 -- 172.21.15.31:0/1208458491 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56385bbb0610 con 0x56385bd49b40 2024-04-25T16:05:04.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.497+0000 7f21277fe640 10 client.16392 handle_command_reply: tid=0 2024-04-25T16:05:04.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.497+0000 7f2151776740 1 client.16392 shutdown 2024-04-25T16:05:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.502+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 msgr2=0x56385bcd9320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.502+0000 7f2151776740 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcd9320 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f21340a3840 tx=0x7f21340b0040 comp rx=0 tx=0).stop 2024-04-25T16:05:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.503+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 shutdown_connections 2024-04-25T16:05:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.503+0000 7f2151776740 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56385bd49b40 0x56385bd4bf80 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.503+0000 7f2151776740 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56385b9066b0 0x56385bd47b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.503+0000 7f2151776740 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56385b8fe4d0 0x56385bcd9320 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.503+0000 7f2151776740 1 --2- 172.21.15.31:0/1208458491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56385b8e5f60 0x56385bcd8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.503+0000 7f21277fe640 0 client.16392 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:04.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.503+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 >> 172.21.15.31:0/1208458491 conn(0x56385b91b4e0 msgr2=0x56385bcc8780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:04.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.509+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 shutdown_connections 2024-04-25T16:05:04.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.513+0000 7f2151776740 1 -- 172.21.15.31:0/1208458491 wait complete. 2024-04-25T16:05:04.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.516+0000 7f2151776740 20 client.16392 trim_cache size 0 max 16384 2024-04-25T16:05:04.521 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:04.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.527+0000 7f2149664640 1 -- 172.21.15.31:0/3374130125 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f212c0afd30 msgr2=0x7f212c0b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:04.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.527+0000 7f2149664640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f212c0afd30 0x7f212c0b2240 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f213c014000 tx=0x7f213c016000 comp rx=0 tx=0).stop 2024-04-25T16:05:04.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.527+0000 7f2149664640 1 -- 172.21.15.31:0/3374130125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f214410b190 msgr2=0x7f21441a9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:04.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.528+0000 7f2149664640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f214410b190 0x7f21441a9460 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f213400b3f0 tx=0x7f213400b8b0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.536+0000 7f2149664640 1 -- 172.21.15.31:0/3374130125 shutdown_connections 2024-04-25T16:05:04.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.536+0000 7f2149664640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f212c0afd30 0x7f212c0b2240 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.536+0000 7f2149664640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f214410fe90 0x7f214407ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.536+0000 7f2149664640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f214410b190 0x7f21441a9460 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.536+0000 7f2149664640 1 --2- 172.21.15.31:0/3374130125 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f214410a340 0x7f21441a8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:04.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.536+0000 7f2149664640 1 -- 172.21.15.31:0/3374130125 >> 172.21.15.31:0/3374130125 conn(0x7f2144070320 msgr2=0x7f214410be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:04.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.539+0000 7f2149664640 1 -- 172.21.15.31:0/3374130125 shutdown_connections 2024-04-25T16:05:04.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:04.541+0000 7f2149664640 1 -- 172.21.15.31:0/3374130125 wait complete. 2024-04-25T16:05:05.589 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:05:05.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:05:05.589 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:05:05.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:05 smithi063 ceph-mon[32413]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 952 MiB/s wr, 2.28k op/s 2024-04-25T16:05:05.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:05 smithi063 ceph-mon[32413]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:05:05.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:05 smithi186 ceph-mon[33258]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 952 MiB/s wr, 2.28k op/s 2024-04-25T16:05:05.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:05 smithi186 ceph-mon[33258]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:05:05.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:05 smithi031 ceph-mon[29140]: pgmap v464: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 952 MiB/s wr, 2.28k op/s 2024-04-25T16:05:05.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:05 smithi031 ceph-mon[29140]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:05:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.122+0000 7f732edc0640 1 -- 172.21.15.31:0/918417750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810b310 msgr2=0x7f732810b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.122+0000 7f732edc0640 1 --2- 172.21.15.31:0/918417750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810b310 0x7f732810b720 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7320009ca0 tx=0x7f732002a8e0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.123+0000 7f732edc0640 1 -- 172.21.15.31:0/918417750 shutdown_connections 2024-04-25T16:05:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.123+0000 7f732edc0640 1 --2- 172.21.15.31:0/918417750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328110030 0x7f7328110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.123+0000 7f732edc0640 1 --2- 172.21.15.31:0/918417750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810b310 0x7f732810b720 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.123+0000 7f732edc0640 1 --2- 172.21.15.31:0/918417750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f732810a420 0x7f732810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.123+0000 7f732edc0640 1 -- 172.21.15.31:0/918417750 >> 172.21.15.31:0/918417750 conn(0x7f73280704e0 msgr2=0x7f7328072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:06.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.131+0000 7f732edc0640 1 -- 172.21.15.31:0/918417750 shutdown_connections 2024-04-25T16:05:06.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.131+0000 7f732edc0640 1 -- 172.21.15.31:0/918417750 wait complete. 2024-04-25T16:05:06.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.132+0000 7f732edc0640 1 Processor -- start 2024-04-25T16:05:06.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.132+0000 7f732edc0640 1 -- start start 2024-04-25T16:05:06.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.133+0000 7f732edc0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810a420 0x7f732811b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.133+0000 7f732edc0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f732810b310 0x7f732811bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.133+0000 7f732edc0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328110030 0x7f7328119c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.133+0000 7f732edc0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7328111f70 con 0x7f7328110030 2024-04-25T16:05:06.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.133+0000 7f732edc0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7328111df0 con 0x7f732810a420 2024-04-25T16:05:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.133+0000 7f732edc0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73281120f0 con 0x7f732810b310 2024-04-25T16:05:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.134+0000 7f732cb5c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810a420 0x7f732811b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.134+0000 7f732cb5c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810a420 0x7f732811b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34784/0 (socket says 172.21.15.31:34784) 2024-04-25T16:05:06.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.134+0000 7f732cb5c640 1 -- 172.21.15.31:0/831455932 learned_addr learned my addr 172.21.15.31:0/831455932 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.135+0000 7f7327fff640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f732810b310 0x7f732811bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.135+0000 7f732cb5c640 1 -- 172.21.15.31:0/831455932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f732810b310 msgr2=0x7f732811bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.135+0000 7f732cb5c640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f732810b310 0x7f732811bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.135+0000 7f732cb5c640 1 -- 172.21.15.31:0/831455932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328110030 msgr2=0x7f7328119c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.135+0000 7f732cb5c640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328110030 0x7f7328119c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.135+0000 7f732cb5c640 1 -- 172.21.15.31:0/831455932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732811a390 con 0x7f732810a420 2024-04-25T16:05:06.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.139+0000 7f732cb5c640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810a420 0x7f732811b5f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f731c00d770 tx=0x7f731c00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:06.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.140+0000 7f7325ffb640 1 -- 172.21.15.31:0/831455932 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c017070 con 0x7f732810a420 2024-04-25T16:05:06.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.140+0000 7f7325ffb640 1 -- 172.21.15.31:0/831455932 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f731c009c30 con 0x7f732810a420 2024-04-25T16:05:06.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.140+0000 7f7325ffb640 1 -- 172.21.15.31:0/831455932 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c00f620 con 0x7f732810a420 2024-04-25T16:05:06.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.142+0000 7f732edc0640 1 -- 172.21.15.31:0/831455932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f732811a680 con 0x7f732810a420 2024-04-25T16:05:06.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.143+0000 7f732edc0640 1 -- 172.21.15.31:0/831455932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f732811aa90 con 0x7f732810a420 2024-04-25T16:05:06.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.149+0000 7f7335c70740 1 Processor -- start 2024-04-25T16:05:06.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.153+0000 7f7335c70740 1 -- start start 2024-04-25T16:05:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.153+0000 7f7335c70740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a958410 0x55f23acf9460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.153+0000 7f7335c70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23acf99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.153+0000 7f7335c70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a937730 0x55f23ad008e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.153+0000 7f7335c70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f23ac5eb00 con 0x55f23a92f550 2024-04-25T16:05:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.153+0000 7f7335c70740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f23abfa920 con 0x55f23a958410 2024-04-25T16:05:06.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.153+0000 7f7335c70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f23ac5aac0 con 0x55f23a937730 2024-04-25T16:05:06.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f7327fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a937730 0x55f23ad008e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f7327fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a937730 0x55f23ad008e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37252/0 (socket says 172.21.15.31:37252) 2024-04-25T16:05:06.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f7327fff640 1 -- 172.21.15.31:0/2068772616 learned_addr learned my addr 172.21.15.31:0/2068772616 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:06.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f732d35d640 1 --2- 172.21.15.31:0/2068772616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a958410 0x55f23acf9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f7325ffb640 1 -- 172.21.15.31:0/831455932 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f731c01a070 con 0x7f732810a420 2024-04-25T16:05:06.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f7327fff640 1 -- 172.21.15.31:0/2068772616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a958410 msgr2=0x55f23acf9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:06.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f7327fff640 1 --2- 172.21.15.31:0/2068772616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a958410 0x55f23acf9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f7327fff640 1 -- 172.21.15.31:0/2068772616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 msgr2=0x55f23acf99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:06.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f7327fff640 1 --2- 172.21.15.31:0/2068772616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23acf99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.155+0000 7f7327fff640 1 -- 172.21.15.31:0/2068772616 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23aa387e0 con 0x55f23a937730 2024-04-25T16:05:06.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.157+0000 7f7325ffb640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73100afcf0 0x7f73100b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.157+0000 7f7325ffb640 1 -- 172.21.15.31:0/831455932 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f731c0de440 con 0x7f732810a420 2024-04-25T16:05:06.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.159+0000 7f732d35d640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73100afcf0 0x7f73100b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.161+0000 7f732d35d640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73100afcf0 0x7f73100b2200 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55f23acfa630 tx=0x7f7318009530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:06.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.162+0000 7f7327fff640 1 --2- 172.21.15.31:0/2068772616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a937730 0x55f23ad008e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f732002ae50 tx=0x7f7320009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfebd84f69e0aba8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:06.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.164+0000 7f732edc0640 1 -- 172.21.15.31:0/2068772616 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320003b70 con 0x55f23a937730 2024-04-25T16:05:06.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.164+0000 7f732edc0640 1 -- 172.21.15.31:0/2068772616 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7320036040 con 0x55f23a937730 2024-04-25T16:05:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.165+0000 7f732edc0640 1 -- 172.21.15.31:0/2068772616 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320004370 con 0x55f23a937730 2024-04-25T16:05:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.168+0000 7f7335c70740 1 -- 172.21.15.31:0/2068772616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a937730 msgr2=0x55f23ad008e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.168+0000 7f7335c70740 1 --2- 172.21.15.31:0/2068772616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a937730 0x55f23ad008e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f732002ae50 tx=0x7f7320009760 comp rx=0 tx=0).stop 2024-04-25T16:05:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.169+0000 7f7335c70740 1 -- 172.21.15.31:0/2068772616 shutdown_connections 2024-04-25T16:05:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.169+0000 7f7335c70740 1 --2- 172.21.15.31:0/2068772616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a937730 0x55f23ad008e0 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f732002ae50 tx=0x7f7320009760 comp rx=0 tx=0).stop 2024-04-25T16:05:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.169+0000 7f7335c70740 1 --2- 172.21.15.31:0/2068772616 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23acf99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.169+0000 7f7335c70740 1 --2- 172.21.15.31:0/2068772616 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a958410 0x55f23acf9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.169+0000 7f7335c70740 1 -- 172.21.15.31:0/2068772616 >> 172.21.15.31:0/2068772616 conn(0x55f23a94c560 msgr2=0x55f23acf50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:06.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.175+0000 7f7335c70740 1 -- 172.21.15.31:0/2068772616 shutdown_connections 2024-04-25T16:05:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.176+0000 7f7335c70740 1 -- 172.21.15.31:0/2068772616 wait complete. 2024-04-25T16:05:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.177+0000 7f7335c70740 1 -- start start 2024-04-25T16:05:06.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.177+0000 7f7335c70740 1 Processor -- start 2024-04-25T16:05:06.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.178+0000 7f7335c70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad03420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.178+0000 7f7335c70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 0x55f23ad03960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.178+0000 7f7335c70740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 0x55f23ad0a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.178+0000 7f7335c70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f23ad012d0 con 0x55f23a92f550 2024-04-25T16:05:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.178+0000 7f7335c70740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f23abf85d0 con 0x55f23a9171f0 2024-04-25T16:05:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.178+0000 7f7335c70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f23ad01450 con 0x55f23a958410 2024-04-25T16:05:06.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.178+0000 7f7335c70740 10 client.0 _ll_register_callbacks cb 0x55f23abe6080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.180+0000 7f732cb5c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad03420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.180+0000 7f732cb5c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad03420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53096/0 (socket says 172.21.15.31:53096) 2024-04-25T16:05:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.180+0000 7f732cb5c640 1 -- 172.21.15.31:0/1180538576 learned_addr learned my addr 172.21.15.31:0/1180538576 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.180+0000 7f732d35d640 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 0x55f23ad0a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.180+0000 7f7327fff640 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 0x55f23ad03960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.181+0000 7f732cb5c640 1 -- 172.21.15.31:0/1180538576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 msgr2=0x55f23ad03960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.181+0000 7f732cb5c640 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 0x55f23ad03960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.181+0000 7f732cb5c640 1 -- 172.21.15.31:0/1180538576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 msgr2=0x55f23ad0a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.181+0000 7f732cb5c640 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 0x55f23ad0a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.181+0000 7f732cb5c640 1 -- 172.21.15.31:0/1180538576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23aa98f70 con 0x55f23a92f550 2024-04-25T16:05:06.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.182+0000 7f732cb5c640 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad03420 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f731c0a6650 tx=0x7f731c0a6180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d3c8115a98c5479 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.182+0000 7f7307fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.183+0000 7f7307fff640 1 -- 172.21.15.31:0/1180538576 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731c0a6950 con 0x55f23a92f550 2024-04-25T16:05:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.183+0000 7f7335c70740 1 -- 172.21.15.31:0/1180538576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7320036040 con 0x55f23a92f550 2024-04-25T16:05:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.183+0000 7f7307fff640 1 -- 172.21.15.31:0/1180538576 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f731c0a2be0 con 0x55f23a92f550 2024-04-25T16:05:06.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.183+0000 7f7307fff640 1 -- 172.21.15.31:0/1180538576 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f731c0aad00 con 0x55f23a92f550 2024-04-25T16:05:06.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.188+0000 7f7335c70740 10 client.16398 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:06.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.188+0000 7f7335c70740 1 -- 172.21.15.31:0/1180538576 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f23a930170 con 0x55f23a92f550 2024-04-25T16:05:06.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.192+0000 7f7307fff640 1 -- 172.21.15.31:0/1180538576 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7320036040 con 0x55f23a92f550 2024-04-25T16:05:06.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.194+0000 7f7307fff640 1 -- 172.21.15.31:0/1180538576 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f731c0a22c0 con 0x55f23a92f550 2024-04-25T16:05:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.194+0000 7f7335c70740 10 client.16398 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.194+0000 7f7335c70740 10 client.16398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.195+0000 7f7335c70740 20 client.16398 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.195+0000 7f7335c70740 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f23a8ec630 0x55f23ad112e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.195+0000 7f7335c70740 4 client.16398 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:06.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.195+0000 7f7335c70740 1 -- 172.21.15.31:0/1180538576 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f23ac6c710 con 0x55f23a8ec630 2024-04-25T16:05:06.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.197+0000 7f7327fff640 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f23a8ec630 0x55f23ad112e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.212+0000 7f7327fff640 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f23a8ec630 0x55f23ad112e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35353f1ba0b00a53 server_cookie=ada763b6da6ec7c in_seq=0 out_seq=0 2024-04-25T16:05:06.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.212+0000 7f7307fff640 10 client.16398 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:06.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.216+0000 7f7307fff640 1 -- 172.21.15.31:0/1180538576 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f23ac6c710 con 0x55f23a8ec630 2024-04-25T16:05:06.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.216+0000 7f7307fff640 10 client.16398 handle_command_reply: tid=0 2024-04-25T16:05:06.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.216+0000 7f7335c70740 1 client.16398 shutdown 2024-04-25T16:05:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.222+0000 7f7335c70740 1 -- 172.21.15.31:0/1180538576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 msgr2=0x55f23ad03420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.222+0000 7f7335c70740 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad03420 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f731c0a6650 tx=0x7f731c0a6180 comp rx=0 tx=0).stop 2024-04-25T16:05:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.223+0000 7f7335c70740 1 -- 172.21.15.31:0/1180538576 shutdown_connections 2024-04-25T16:05:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.223+0000 7f7335c70740 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f23a8ec630 0x55f23ad112e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.223+0000 7f7335c70740 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 0x55f23ad0a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.223+0000 7f7335c70740 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 0x55f23ad03960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.223+0000 7f7335c70740 1 --2- 172.21.15.31:0/1180538576 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad03420 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.223+0000 7f7335c70740 1 -- 172.21.15.31:0/1180538576 >> 172.21.15.31:0/1180538576 conn(0x55f23a94c560 msgr2=0x55f23acf4080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:06.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.223+0000 7f7307fff640 0 client.16398 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:06.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.228+0000 7f7335c70740 1 -- 172.21.15.31:0/1180538576 shutdown_connections 2024-04-25T16:05:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.232+0000 7f7335c70740 1 -- 172.21.15.31:0/1180538576 wait complete. 2024-04-25T16:05:06.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.232+0000 7f7335c70740 20 client.16398 trim_cache size 0 max 16384 2024-04-25T16:05:06.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.291+0000 7f7335c70740 1 Processor -- start 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.291+0000 7f7335c70740 1 -- start start 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.291+0000 7f7335c70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a9171f0 0x55f23acf4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.292+0000 7f7335c70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23acf4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.292+0000 7f7335c70740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a958410 0x55f23ad0a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.292+0000 7f7335c70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73281c2890 con 0x55f23a92f550 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.292+0000 7f7335c70740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73281c43d0 con 0x55f23a958410 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.292+0000 7f7335c70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73281c2f60 con 0x55f23a9171f0 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.292+0000 7f7327fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23acf4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.292+0000 7f7327fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23acf4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53104/0 (socket says 172.21.15.31:53104) 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.292+0000 7f7327fff640 1 -- 172.21.15.31:0/2704275774 learned_addr learned my addr 172.21.15.31:0/2704275774 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.293+0000 7f7327fff640 1 -- 172.21.15.31:0/2704275774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a9171f0 msgr2=0x55f23acf4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.293+0000 7f7327fff640 1 --2- 172.21.15.31:0/2704275774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a9171f0 0x55f23acf4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.293+0000 7f7327fff640 1 -- 172.21.15.31:0/2704275774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a958410 msgr2=0x55f23ad0a880 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.293+0000 7f7327fff640 1 --2- 172.21.15.31:0/2704275774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a958410 0x55f23ad0a880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.293+0000 7f7327fff640 1 -- 172.21.15.31:0/2704275774 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f23acfe300 con 0x55f23a92f550 2024-04-25T16:05:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.293+0000 7f7327fff640 1 --2- 172.21.15.31:0/2704275774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23acf4fe0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f7320004af0 tx=0x7f7320009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55aa1a95cec7cfd4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.294+0000 7f730e7fc640 1 -- 172.21.15.31:0/2704275774 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320057070 con 0x55f23a92f550 2024-04-25T16:05:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.294+0000 7f730e7fc640 1 -- 172.21.15.31:0/2704275774 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7320036040 con 0x55f23a92f550 2024-04-25T16:05:06.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.294+0000 7f730e7fc640 1 -- 172.21.15.31:0/2704275774 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320049040 con 0x55f23a92f550 2024-04-25T16:05:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.302+0000 7f7335c70740 1 -- 172.21.15.31:0/2704275774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 msgr2=0x55f23acf4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.303+0000 7f7335c70740 1 --2- 172.21.15.31:0/2704275774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23acf4fe0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f7320004af0 tx=0x7f7320009850 comp rx=0 tx=0).stop 2024-04-25T16:05:06.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.304+0000 7f7335c70740 1 -- 172.21.15.31:0/2704275774 shutdown_connections 2024-04-25T16:05:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.304+0000 7f7335c70740 1 --2- 172.21.15.31:0/2704275774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a958410 0x55f23ad0a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.304+0000 7f7335c70740 1 --2- 172.21.15.31:0/2704275774 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23acf4fe0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.304+0000 7f7335c70740 1 --2- 172.21.15.31:0/2704275774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a9171f0 0x55f23acf4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.304+0000 7f7335c70740 1 -- 172.21.15.31:0/2704275774 >> 172.21.15.31:0/2704275774 conn(0x55f23a94c560 msgr2=0x55f23ad01a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.304+0000 7f7335c70740 1 -- 172.21.15.31:0/2704275774 shutdown_connections 2024-04-25T16:05:06.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.304+0000 7f7335c70740 1 -- 172.21.15.31:0/2704275774 wait complete. 2024-04-25T16:05:06.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.305+0000 7f7335c70740 1 -- start start 2024-04-25T16:05:06.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.305+0000 7f7335c70740 1 Processor -- start 2024-04-25T16:05:06.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.307+0000 7f7335c70740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 0x55f23ad07eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.307+0000 7f7335c70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad083f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.307+0000 7f7335c70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 0x55f23ad77680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.307+0000 7f7335c70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73281c3d00 con 0x55f23a92f550 2024-04-25T16:05:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.307+0000 7f7335c70740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73281c3630 con 0x55f23a9171f0 2024-04-25T16:05:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.307+0000 7f7335c70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73281c0920 con 0x55f23a958410 2024-04-25T16:05:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.307+0000 7f7335c70740 10 client.0 _ll_register_callbacks cb 0x7f731c0a28a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.307+0000 7f732cb5c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 0x55f23ad07eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.308+0000 7f732cb5c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 0x55f23ad07eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34812/0 (socket says 172.21.15.31:34812) 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.308+0000 7f732cb5c640 1 -- 172.21.15.31:0/393876489 learned_addr learned my addr 172.21.15.31:0/393876489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.308+0000 7f7327fff640 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad083f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.309+0000 7f732d35d640 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 0x55f23ad77680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.309+0000 7f7327fff640 1 -- 172.21.15.31:0/393876489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 msgr2=0x55f23ad77680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.309+0000 7f7327fff640 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 0x55f23ad77680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.309+0000 7f7327fff640 1 -- 172.21.15.31:0/393876489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 msgr2=0x55f23ad07eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.309+0000 7f7327fff640 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 0x55f23ad07eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.309+0000 7f7327fff640 1 -- 172.21.15.31:0/393876489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f23ad08a30 con 0x55f23a92f550 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.310+0000 7f7327fff640 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad083f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x55f23ad0bad0 tx=0x7f7320003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee06493d854f6a91 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.312+0000 7f7307fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.312+0000 7f7307fff640 1 -- 172.21.15.31:0/393876489 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320057070 con 0x55f23a92f550 2024-04-25T16:05:06.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.312+0000 7f7307fff640 1 -- 172.21.15.31:0/393876489 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7320004450 con 0x55f23a92f550 2024-04-25T16:05:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.312+0000 7f7335c70740 1 -- 172.21.15.31:0/393876489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7320036040 con 0x55f23a92f550 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.313+0000 7f7307fff640 1 -- 172.21.15.31:0/393876489 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7320004690 con 0x55f23a92f550 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.313+0000 7f7335c70740 10 client.16404 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.313+0000 7f7335c70740 1 -- 172.21.15.31:0/393876489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f23ad78c60 con 0x55f23a92f550 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.314+0000 7f7307fff640 1 -- 172.21.15.31:0/393876489 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7320004830 con 0x55f23a92f550 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.314+0000 7f7307fff640 1 -- 172.21.15.31:0/393876489 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7320030960 con 0x55f23a92f550 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.314+0000 7f7335c70740 10 client.16404 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.314+0000 7f7335c70740 10 client.16404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.314+0000 7f7335c70740 20 client.16404 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.314+0000 7f7335c70740 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f23ad79040 0x55f23ad7b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.314+0000 7f7335c70740 4 client.16404 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:05:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.314+0000 7f7335c70740 1 -- 172.21.15.31:0/393876489 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7f7320036040 con 0x55f23ad79040 2024-04-25T16:05:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.315+0000 7f732cb5c640 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f23ad79040 0x55f23ad7b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.316+0000 7f7307fff640 10 client.16404 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:06.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.319+0000 7f732cb5c640 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f23ad79040 0x55f23ad7b460 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=805e974782c3a063 server_cookie=47b5f483d6b948aa in_seq=0 out_seq=0 2024-04-25T16:05:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.321+0000 7f7307fff640 1 -- 172.21.15.31:0/393876489 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7f7320036040 con 0x55f23ad79040 2024-04-25T16:05:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.321+0000 7f7307fff640 10 client.16404 handle_command_reply: tid=0 2024-04-25T16:05:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.321+0000 7f7335c70740 1 client.16404 shutdown 2024-04-25T16:05:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.322+0000 7f7335c70740 1 -- 172.21.15.31:0/393876489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 msgr2=0x55f23ad083f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.322+0000 7f7335c70740 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad083f0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x55f23ad0bad0 tx=0x7f7320003d00 comp rx=0 tx=0).stop 2024-04-25T16:05:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.323+0000 7f7335c70740 1 -- 172.21.15.31:0/393876489 shutdown_connections 2024-04-25T16:05:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.323+0000 7f7335c70740 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f23ad79040 0x55f23ad7b460 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.323+0000 7f7335c70740 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f23a958410 0x55f23ad77680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.323+0000 7f7335c70740 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f23a92f550 0x55f23ad083f0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.323+0000 7f7335c70740 1 --2- 172.21.15.31:0/393876489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f23a9171f0 0x55f23ad07eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.323+0000 7f7335c70740 1 -- 172.21.15.31:0/393876489 >> 172.21.15.31:0/393876489 conn(0x55f23a94c560 msgr2=0x55f23acf4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.323+0000 7f7307fff640 0 client.16404 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:06.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.340+0000 7f7335c70740 1 -- 172.21.15.31:0/393876489 shutdown_connections 2024-04-25T16:05:06.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.340+0000 7f7335c70740 1 -- 172.21.15.31:0/393876489 wait complete. 2024-04-25T16:05:06.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.341+0000 7f7335c70740 20 client.16404 trim_cache size 0 max 16384 2024-04-25T16:05:06.344 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:05:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:05:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "18360ad0-a1c6-491f-a421-9c84a4e93d14", 2024-04-25T16:05:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:05:06.345 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:05:06.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.345+0000 7f730f7fe640 1 -- 172.21.15.31:0/831455932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73100afcf0 msgr2=0x7f73100b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.345+0000 7f730f7fe640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73100afcf0 0x7f73100b2200 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55f23acfa630 tx=0x7f7318009530 comp rx=0 tx=0).stop 2024-04-25T16:05:06.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.345+0000 7f730f7fe640 1 -- 172.21.15.31:0/831455932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810a420 msgr2=0x7f732811b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.345+0000 7f730f7fe640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810a420 0x7f732811b5f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f731c00d770 tx=0x7f731c00dc30 comp rx=0 tx=0).stop 2024-04-25T16:05:06.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.345+0000 7f730f7fe640 1 -- 172.21.15.31:0/831455932 shutdown_connections 2024-04-25T16:05:06.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.345+0000 7f730f7fe640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73100afcf0 0x7f73100b2200 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.346+0000 7f730f7fe640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7328110030 0x7f7328119c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.346+0000 7f730f7fe640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f732810b310 0x7f732811bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.346+0000 7f730f7fe640 1 --2- 172.21.15.31:0/831455932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f732810a420 0x7f732811b5f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.346+0000 7f730f7fe640 1 -- 172.21.15.31:0/831455932 >> 172.21.15.31:0/831455932 conn(0x7f73280704e0 msgr2=0x7f732810e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:06.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.346+0000 7f730f7fe640 1 -- 172.21.15.31:0/831455932 shutdown_connections 2024-04-25T16:05:06.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.346+0000 7f730f7fe640 1 -- 172.21.15.31:0/831455932 wait complete. 2024-04-25T16:05:06.407 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:05:06.407 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:05:06.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.935+0000 7f129343d640 1 -- 172.21.15.31:0/1342673747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 msgr2=0x7f128c10ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.935+0000 7f129343d640 1 --2- 172.21.15.31:0/1342673747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 0x7f128c10ac90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f128801a910 tx=0x7f128803b450 comp rx=0 tx=0).stop 2024-04-25T16:05:06.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.945+0000 7f129343d640 1 -- 172.21.15.31:0/1342673747 shutdown_connections 2024-04-25T16:05:06.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.945+0000 7f129343d640 1 --2- 172.21.15.31:0/1342673747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f128c10fef0 0x7f128c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.945+0000 7f129343d640 1 --2- 172.21.15.31:0/1342673747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f128c10b1d0 0x7f128c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.945+0000 7f129343d640 1 --2- 172.21.15.31:0/1342673747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 0x7f128c10ac90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.945+0000 7f129343d640 1 -- 172.21.15.31:0/1342673747 >> 172.21.15.31:0/1342673747 conn(0x7f128c070400 msgr2=0x7f128c072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:06.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.953+0000 7f129343d640 1 -- 172.21.15.31:0/1342673747 shutdown_connections 2024-04-25T16:05:06.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.953+0000 7f129343d640 1 -- 172.21.15.31:0/1342673747 wait complete. 2024-04-25T16:05:06.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.954+0000 7f129343d640 1 Processor -- start 2024-04-25T16:05:06.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.957+0000 7f129343d640 1 -- start start 2024-04-25T16:05:06.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.958+0000 7f129343d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 0x7f128c1a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.958+0000 7f129343d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f128c10b1d0 0x7f128c1a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.958+0000 7f129343d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f128c10fef0 0x7f128c1a7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.959+0000 7f129243b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 0x7f128c1a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.959+0000 7f129243b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 0x7f128c1a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37300/0 (socket says 172.21.15.31:37300) 2024-04-25T16:05:06.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.959+0000 7f129243b640 1 -- 172.21.15.31:0/3015710489 learned_addr learned my addr 172.21.15.31:0/3015710489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:06.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.959+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f128c113aa0 con 0x7f128c10fef0 2024-04-25T16:05:06.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.959+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f128c113920 con 0x7f128c10b1d0 2024-04-25T16:05:06.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.959+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f128c113c20 con 0x7f128c10a350 2024-04-25T16:05:06.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.961+0000 7f1292c3c640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f128c10fef0 0x7f128c1a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.965+0000 7f1291c3a640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f128c10b1d0 0x7f128c1a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.966+0000 7f129243b640 1 -- 172.21.15.31:0/3015710489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f128c10b1d0 msgr2=0x7f128c1a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.966+0000 7f129243b640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f128c10b1d0 0x7f128c1a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.966+0000 7f129243b640 1 -- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f128c10fef0 msgr2=0x7f128c1a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.966+0000 7f129243b640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f128c10fef0 0x7f128c1a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:06.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.966+0000 7f129243b640 1 -- 172.21.15.31:0/3015710489 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f128c1a7fb0 con 0x7f128c10a350 2024-04-25T16:05:06.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.968+0000 7f129243b640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 0x7f128c1a0470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1288004d30 tx=0x7f12880053a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:06.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.972+0000 7f12837fe640 1 -- 172.21.15.31:0/3015710489 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f128804f020 con 0x7f128c10a350 2024-04-25T16:05:06.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.973+0000 7f12837fe640 1 -- 172.21.15.31:0/3015710489 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1288004320 con 0x7f128c10a350 2024-04-25T16:05:06.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.973+0000 7f12837fe640 1 -- 172.21.15.31:0/3015710489 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1288046070 con 0x7f128c10a350 2024-04-25T16:05:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.976+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f128c1a8240 con 0x7f128c10a350 2024-04-25T16:05:06.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.976+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f128c1a86b0 con 0x7f128c10a350 2024-04-25T16:05:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.983+0000 7f129b62c740 1 Processor -- start 2024-04-25T16:05:06.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.983+0000 7f129b62c740 1 -- start start 2024-04-25T16:05:06.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.984+0000 7f129b62c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.985+0000 7f129b62c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.985+0000 7f129b62c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a0066b0 0x55a01a3c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:06.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.985+0000 7f129b62c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a01a32dd80 con 0x55a01a0066b0 2024-04-25T16:05:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.985+0000 7f1292c3c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.985+0000 7f1292c3c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3c8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34834/0 (socket says 172.21.15.31:34834) 2024-04-25T16:05:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.985+0000 7f1292c3c640 1 -- 172.21.15.31:0/631751658 learned_addr learned my addr 172.21.15.31:0/631751658 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.986+0000 7f129b62c740 1 -- 172.21.15.31:0/631751658 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a01a2e78b0 con 0x55a01a027390 2024-04-25T16:05:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.986+0000 7f129b62c740 1 -- 172.21.15.31:0/631751658 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a01a329d40 con 0x55a019ffe4d0 2024-04-25T16:05:06.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.989+0000 7f129243b640 1 --2- 172.21.15.31:0/631751658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.989+0000 7f1291c3a640 1 --2- 172.21.15.31:0/631751658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a0066b0 0x55a01a3c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:06.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:06.990+0000 7f12837fe640 1 -- 172.21.15.31:0/3015710489 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1288052020 con 0x7f128c10a350 2024-04-25T16:05:07.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.003+0000 7f12837fe640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12780afd00 0x7f12780b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.003+0000 7f12837fe640 1 -- 172.21.15.31:0/3015710489 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f12881120e0 con 0x7f128c10a350 2024-04-25T16:05:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.004+0000 7f1292c3c640 1 -- 172.21.15.31:0/631751658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 msgr2=0x55a01a3c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.004+0000 7f1292c3c640 1 --2- 172.21.15.31:0/631751658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3c91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.004+0000 7f1292c3c640 1 -- 172.21.15.31:0/631751658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a0066b0 msgr2=0x55a01a3c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.004+0000 7f1292c3c640 1 --2- 172.21.15.31:0/631751658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a0066b0 0x55a01a3c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.004+0000 7f1292c3c640 1 -- 172.21.15.31:0/631751658 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a01a2b8a40 con 0x55a01a027390 2024-04-25T16:05:07.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.006+0000 7f1292c3c640 1 --2- 172.21.15.31:0/631751658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3c8c70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1284009f40 tx=0x7f1284010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a955a0b74400d4c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.008+0000 7f1280bfe640 1 -- 172.21.15.31:0/631751658 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284016070 con 0x55a01a027390 2024-04-25T16:05:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.008+0000 7f1280bfe640 1 -- 172.21.15.31:0/631751658 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f128401bb10 con 0x55a01a027390 2024-04-25T16:05:07.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.008+0000 7f1280bfe640 1 -- 172.21.15.31:0/631751658 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1284012cc0 con 0x55a01a027390 2024-04-25T16:05:07.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.017+0000 7f129b62c740 1 -- 172.21.15.31:0/631751658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 msgr2=0x55a01a3c8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:07.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.017+0000 7f129b62c740 1 --2- 172.21.15.31:0/631751658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3c8c70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1284009f40 tx=0x7f1284010800 comp rx=0 tx=0).stop 2024-04-25T16:05:07.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.026+0000 7f1292c3c640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12780afd00 0x7f12780b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.030+0000 7f129b62c740 1 -- 172.21.15.31:0/631751658 shutdown_connections 2024-04-25T16:05:07.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.030+0000 7f129b62c740 1 --2- 172.21.15.31:0/631751658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a0066b0 0x55a01a3c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.031+0000 7f129b62c740 1 --2- 172.21.15.31:0/631751658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3c91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.031+0000 7f129b62c740 1 --2- 172.21.15.31:0/631751658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3c8c70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.031+0000 7f129b62c740 1 -- 172.21.15.31:0/631751658 >> 172.21.15.31:0/631751658 conn(0x55a01a01b4e0 msgr2=0x55a01a3c4480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:07.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.041+0000 7f1292c3c640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12780afd00 0x7f12780b2210 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1284009640 tx=0x7f1284010fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:07.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.044+0000 7f129b62c740 1 -- 172.21.15.31:0/631751658 shutdown_connections 2024-04-25T16:05:07.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.044+0000 7f129b62c740 1 -- 172.21.15.31:0/631751658 wait complete. 2024-04-25T16:05:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.045+0000 7f129b62c740 1 -- start start 2024-04-25T16:05:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.045+0000 7f129b62c740 1 Processor -- start 2024-04-25T16:05:07.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.046+0000 7f129b62c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 0x55a01a3d2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.046+0000 7f129b62c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.046+0000 7f129b62c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 0x55a01a3d9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.046+0000 7f129b62c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a01a3ca990 con 0x55a01a027390 2024-04-25T16:05:07.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.046+0000 7f129b62c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a01a2b4d10 con 0x55a019ffe4d0 2024-04-25T16:05:07.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.046+0000 7f129b62c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a01a3cab10 con 0x55a01a0066b0 2024-04-25T16:05:07.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.046+0000 7f129b62c740 10 client.0 _ll_register_callbacks cb 0x55a01a2e1210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:07.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.047+0000 7f1291c3a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.047+0000 7f1291c3a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37320/0 (socket says 172.21.15.31:37320) 2024-04-25T16:05:07.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.047+0000 7f1291c3a640 1 -- 172.21.15.31:0/2371844612 learned_addr learned my addr 172.21.15.31:0/2371844612 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:07.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.052+0000 7f1292c3c640 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 0x55a01a3d9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.052+0000 7f129243b640 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 0x55a01a3d2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.052+0000 7f1291c3a640 1 -- 172.21.15.31:0/2371844612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 msgr2=0x55a01a3d2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.052+0000 7f1291c3a640 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 0x55a01a3d2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.052+0000 7f1291c3a640 1 -- 172.21.15.31:0/2371844612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 msgr2=0x55a01a3d9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.052+0000 7f1291c3a640 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 0x55a01a3d9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.052+0000 7f1291c3a640 1 -- 172.21.15.31:0/2371844612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a01a1097d0 con 0x55a01a0066b0 2024-04-25T16:05:07.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.053+0000 7f1291c3a640 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d3090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f127c00be00 tx=0x7f127c0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10fa2c72ed10054f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.054+0000 7f1275ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.054+0000 7f1275ffb640 1 -- 172.21.15.31:0/2371844612 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127c012650 con 0x55a01a0066b0 2024-04-25T16:05:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.054+0000 7f129b62c740 1 -- 172.21.15.31:0/2371844612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f128401bb10 con 0x55a01a0066b0 2024-04-25T16:05:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.055+0000 7f1275ffb640 1 -- 172.21.15.31:0/2371844612 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f127c0127f0 con 0x55a01a0066b0 2024-04-25T16:05:07.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.055+0000 7f1275ffb640 1 -- 172.21.15.31:0/2371844612 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f127c0129f0 con 0x55a01a0066b0 2024-04-25T16:05:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.055+0000 7f129b62c740 10 client.26056 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.055+0000 7f129b62c740 1 -- 172.21.15.31:0/2371844612 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a01a03f3f0 con 0x55a01a0066b0 2024-04-25T16:05:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.056+0000 7f1275ffb640 1 -- 172.21.15.31:0/2371844612 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f128401bb10 con 0x55a01a0066b0 2024-04-25T16:05:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.056+0000 7f1275ffb640 1 -- 172.21.15.31:0/2371844612 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f127c025c70 con 0x55a01a0066b0 2024-04-25T16:05:07.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.056+0000 7f129b62c740 10 client.26056 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.056+0000 7f129b62c740 10 client.26056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.057+0000 7f129b62c740 20 client.26056 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.057+0000 7f129b62c740 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a019fe5f60 0x55a01a3e0970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.057+0000 7f129b62c740 4 client.26056 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:07.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.057+0000 7f129b62c740 1 -- 172.21.15.31:0/2371844612 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a01a33b990 con 0x55a019fe5f60 2024-04-25T16:05:07.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.058+0000 7f129243b640 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a019fe5f60 0x55a01a3e0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.271+0000 7f129243b640 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a019fe5f60 0x55a01a3e0970 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12debbc66e792fe3 server_cookie=1b23d586aea02b84 in_seq=0 out_seq=0 2024-04-25T16:05:07.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.271+0000 7f1275ffb640 10 client.26056 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:07.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.279+0000 7f1275ffb640 1 -- 172.21.15.31:0/2371844612 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a01a33b990 con 0x55a019fe5f60 2024-04-25T16:05:07.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.279+0000 7f1275ffb640 10 client.26056 handle_command_reply: tid=0 2024-04-25T16:05:07.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.279+0000 7f129b62c740 1 client.26056 shutdown 2024-04-25T16:05:07.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.281+0000 7f129b62c740 1 -- 172.21.15.31:0/2371844612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 msgr2=0x55a01a3d3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:07.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.281+0000 7f129b62c740 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d3090 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f127c00be00 tx=0x7f127c0117b0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.281+0000 7f129b62c740 1 -- 172.21.15.31:0/2371844612 shutdown_connections 2024-04-25T16:05:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.281+0000 7f129b62c740 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a019fe5f60 0x55a01a3e0970 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.281+0000 7f129b62c740 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 0x55a01a3d9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.281+0000 7f129b62c740 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d3090 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.281+0000 7f129b62c740 1 --2- 172.21.15.31:0/2371844612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 0x55a01a3d2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.281+0000 7f129b62c740 1 -- 172.21.15.31:0/2371844612 >> 172.21.15.31:0/2371844612 conn(0x55a01a01b4e0 msgr2=0x55a01a3c3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.281+0000 7f1275ffb640 0 client.26056 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:07.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.282+0000 7f129b62c740 1 -- 172.21.15.31:0/2371844612 shutdown_connections 2024-04-25T16:05:07.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.282+0000 7f129b62c740 1 -- 172.21.15.31:0/2371844612 wait complete. 2024-04-25T16:05:07.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.282+0000 7f129b62c740 20 client.26056 trim_cache size 0 max 16384 2024-04-25T16:05:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.330+0000 7f129b62c740 1 Processor -- start 2024-04-25T16:05:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.330+0000 7f129b62c740 1 -- start start 2024-04-25T16:05:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.330+0000 7f129b62c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.331+0000 7f129b62c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a0066b0 0x55a01a3d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.331+0000 7f129243b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.331+0000 7f129243b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37334/0 (socket says 172.21.15.31:37334) 2024-04-25T16:05:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.331+0000 7f129243b640 1 -- 172.21.15.31:0/1386713479 learned_addr learned my addr 172.21.15.31:0/1386713479 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.331+0000 7f129b62c740 1 --2- 172.21.15.31:0/1386713479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3d9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.331+0000 7f129b62c740 1 -- 172.21.15.31:0/1386713479 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f128c1b15b0 con 0x55a01a0066b0 2024-04-25T16:05:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.331+0000 7f129b62c740 1 -- 172.21.15.31:0/1386713479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f128c1b30f0 con 0x55a01a027390 2024-04-25T16:05:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.331+0000 7f1292c3c640 1 --2- 172.21.15.31:0/1386713479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3d9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.331+0000 7f129b62c740 1 -- 172.21.15.31:0/1386713479 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f128c1b1c80 con 0x55a019ffe4d0 2024-04-25T16:05:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.332+0000 7f129243b640 1 -- 172.21.15.31:0/1386713479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 msgr2=0x55a01a3d9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.332+0000 7f129243b640 1 --2- 172.21.15.31:0/1386713479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3d9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.332+0000 7f129243b640 1 -- 172.21.15.31:0/1386713479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a0066b0 msgr2=0x55a01a3d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.332+0000 7f129243b640 1 --2- 172.21.15.31:0/1386713479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a0066b0 0x55a01a3d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.332+0000 7f129243b640 1 -- 172.21.15.31:0/1386713479 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a019fff0f0 con 0x55a019ffe4d0 2024-04-25T16:05:07.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.336+0000 7f129243b640 1 --2- 172.21.15.31:0/1386713479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3d1110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f128810fb30 tx=0x7f12880c3040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98b5e53838e29405 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:07.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.336+0000 7f1277fff640 1 -- 172.21.15.31:0/1386713479 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12880d9f50 con 0x55a019ffe4d0 2024-04-25T16:05:07.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.336+0000 7f1277fff640 1 -- 172.21.15.31:0/1386713479 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12880da110 con 0x55a019ffe4d0 2024-04-25T16:05:07.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.336+0000 7f1277fff640 1 -- 172.21.15.31:0/1386713479 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12880d9d70 con 0x55a019ffe4d0 2024-04-25T16:05:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.338+0000 7f129b62c740 1 -- 172.21.15.31:0/1386713479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 msgr2=0x55a01a3d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.339+0000 7f129b62c740 1 --2- 172.21.15.31:0/1386713479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3d1110 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f128810fb30 tx=0x7f12880c3040 comp rx=0 tx=0).stop 2024-04-25T16:05:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.339+0000 7f129b62c740 1 -- 172.21.15.31:0/1386713479 shutdown_connections 2024-04-25T16:05:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.339+0000 7f129b62c740 1 --2- 172.21.15.31:0/1386713479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a01a027390 0x55a01a3d9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.339+0000 7f129b62c740 1 --2- 172.21.15.31:0/1386713479 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a0066b0 0x55a01a3d1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.339+0000 7f129b62c740 1 --2- 172.21.15.31:0/1386713479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a019ffe4d0 0x55a01a3d1110 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.339+0000 7f129b62c740 1 -- 172.21.15.31:0/1386713479 >> 172.21.15.31:0/1386713479 conn(0x55a01a01b4e0 msgr2=0x55a01a3c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:07.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.339+0000 7f129b62c740 1 -- 172.21.15.31:0/1386713479 shutdown_connections 2024-04-25T16:05:07.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.339+0000 7f129b62c740 1 -- 172.21.15.31:0/1386713479 wait complete. 2024-04-25T16:05:07.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.340+0000 7f129b62c740 1 -- start start 2024-04-25T16:05:07.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.341+0000 7f129b62c740 1 Processor -- start 2024-04-25T16:05:07.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.342+0000 7f129b62c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 0x55a01a3d7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.342+0000 7f129b62c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.342+0000 7f129b62c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 0x55a01a3d8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.343+0000 7f129243b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 0x55a01a3d7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.343+0000 7f129243b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 0x55a01a3d7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34856/0 (socket says 172.21.15.31:34856) 2024-04-25T16:05:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.343+0000 7f129243b640 1 -- 172.21.15.31:0/3250250721 learned_addr learned my addr 172.21.15.31:0/3250250721 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.343+0000 7f1291c3a640 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.343+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f128c1b2a20 con 0x55a01a027390 2024-04-25T16:05:07.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.343+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f128c1b2350 con 0x55a019ffe4d0 2024-04-25T16:05:07.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.343+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f128c1af590 con 0x55a01a0066b0 2024-04-25T16:05:07.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.344+0000 7f129b62c740 10 client.0 _ll_register_callbacks cb 0x7f127c00b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:07.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.344+0000 7f1292c3c640 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 0x55a01a3d8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.345+0000 7f1291c3a640 1 -- 172.21.15.31:0/3250250721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 msgr2=0x55a01a3d7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:07.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.345+0000 7f1291c3a640 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 0x55a01a3d7ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.345+0000 7f1291c3a640 1 -- 172.21.15.31:0/3250250721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 msgr2=0x55a01a3d8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:07.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.345+0000 7f1291c3a640 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 0x55a01a3d8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.345+0000 7f1291c3a640 1 -- 172.21.15.31:0/3250250721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a01a2ee4b0 con 0x55a01a0066b0 2024-04-25T16:05:07.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.348+0000 7f1291c3a640 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d8020 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x55a01a3bfc40 tx=0x7f127c00b600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdb9b7ac60b033c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:07.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.348+0000 7f1275ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:07.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.350+0000 7f1275ffb640 1 -- 172.21.15.31:0/3250250721 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f127c012d90 con 0x55a01a0066b0 2024-04-25T16:05:07.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.350+0000 7f1275ffb640 1 -- 172.21.15.31:0/3250250721 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f127c00b880 con 0x55a01a0066b0 2024-04-25T16:05:07.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.363+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12880da110 con 0x55a01a0066b0 2024-04-25T16:05:07.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.366+0000 7f1275ffb640 1 -- 172.21.15.31:0/3250250721 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f127c034050 con 0x55a01a0066b0 2024-04-25T16:05:07.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.366+0000 7f129b62c740 10 client.26062 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:07.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.367+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a01a3da5f0 con 0x55a01a0066b0 2024-04-25T16:05:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.369+0000 7f1275ffb640 1 -- 172.21.15.31:0/3250250721 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12880da110 con 0x55a01a0066b0 2024-04-25T16:05:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.369+0000 7f1275ffb640 1 -- 172.21.15.31:0/3250250721 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f127c01a960 con 0x55a01a0066b0 2024-04-25T16:05:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.370+0000 7f129b62c740 10 client.26062 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.370+0000 7f129b62c740 10 client.26062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.370+0000 7f129b62c740 20 client.26062 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.370+0000 7f129b62c740 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a019fbb3a0 0x55a01a44afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.370+0000 7f129b62c740 4 client.26062 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:05:07.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.370+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a01a335800 con 0x55a019fbb3a0 2024-04-25T16:05:07.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.372+0000 7f129243b640 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a019fbb3a0 0x55a01a44afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:07.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.372+0000 7f129243b640 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a019fbb3a0 0x55a01a44afe0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7e004de6942a154 server_cookie=9d355bddbf7194f9 in_seq=0 out_seq=0 2024-04-25T16:05:07.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.373+0000 7f1275ffb640 10 client.26062 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.377+0000 7f1275ffb640 1 -- 172.21.15.31:0/3250250721 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55a01a335800 con 0x55a019fbb3a0 2024-04-25T16:05:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.377+0000 7f1275ffb640 10 client.26062 handle_command_reply: tid=0 2024-04-25T16:05:07.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.377+0000 7f129b62c740 1 client.26062 shutdown 2024-04-25T16:05:07.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.386+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 msgr2=0x55a01a3d8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:07.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.387+0000 7f129b62c740 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d8020 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x55a01a3bfc40 tx=0x7f127c00b600 comp rx=0 tx=0).stop 2024-04-25T16:05:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.399+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 shutdown_connections 2024-04-25T16:05:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.399+0000 7f129b62c740 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a019fbb3a0 0x55a01a44afe0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.399+0000 7f1275ffb640 0 client.26062 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.399+0000 7f129b62c740 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a01a027390 0x55a01a3d8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.399+0000 7f129b62c740 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a01a0066b0 0x55a01a3d8020 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.399+0000 7f129b62c740 1 --2- 172.21.15.31:0/3250250721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a019ffe4d0 0x55a01a3d7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.399+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 >> 172.21.15.31:0/3250250721 conn(0x55a01a01b4e0 msgr2=0x55a01a3c3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.400+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 shutdown_connections 2024-04-25T16:05:07.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.400+0000 7f129b62c740 1 -- 172.21.15.31:0/3250250721 wait complete. 2024-04-25T16:05:07.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.400+0000 7f129b62c740 20 client.26062 trim_cache size 0 max 16384 2024-04-25T16:05:07.402 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:05:07.402 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T16:05:07.402 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:05:07.402 INFO:teuthology.orchestra.run.smithi031.stdout: "18360ad0-a1c6-491f-a421-9c84a4e93d14": { 2024-04-25T16:05:07.402 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:05:07.403 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "18360ad0-a1c6-491f-a421-9c84a4e93d14", 2024-04-25T16:05:07.403 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:05:07.403 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:05:07.403 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:05:07.403 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:05:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.401+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12780afd00 msgr2=0x7f12780b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.402+0000 7f129343d640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12780afd00 0x7f12780b2210 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1284009640 tx=0x7f1284010fd0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.402+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 msgr2=0x7f128c1a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.402+0000 7f129343d640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 0x7f128c1a0470 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1288004d30 tx=0x7f12880053a0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.402+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 shutdown_connections 2024-04-25T16:05:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.402+0000 7f129343d640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f12780afd00 0x7f12780b2210 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.402+0000 7f129343d640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f128c10fef0 0x7f128c1a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.402+0000 7f129343d640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f128c10b1d0 0x7f128c1a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.402+0000 7f129343d640 1 --2- 172.21.15.31:0/3015710489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f128c10a350 0x7f128c1a0470 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:07.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.402+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 >> 172.21.15.31:0/3015710489 conn(0x7f128c070400 msgr2=0x7f128c0725b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:07.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.403+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 shutdown_connections 2024-04-25T16:05:07.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:07.403+0000 7f129343d640 1 -- 172.21.15.31:0/3015710489 wait complete. 2024-04-25T16:05:07.491 INFO:tasks.cephfs.filesystem:scrub status for tag:18360ad0-a1c6-491f-a421-9c84a4e93d14 - {'path': '/', 'tag': '18360ad0-a1c6-491f-a421-9c84a4e93d14', 'options': 'recursive,force'} 2024-04-25T16:05:07.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:07 smithi063 ceph-mon[32413]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 743 MiB/s wr, 1.85k op/s 2024-04-25T16:05:07.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:07 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:05:07.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:07 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:07.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:07 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:05:07.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:07 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:05:07.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:07 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:07.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:07 smithi186 ceph-mon[33258]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 743 MiB/s wr, 1.85k op/s 2024-04-25T16:05:07.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:07 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:05:07.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:07 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:07.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:07 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:05:07.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:07 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:05:07.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:07 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:07.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:07 smithi031 ceph-mon[29140]: pgmap v465: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 743 MiB/s wr, 1.85k op/s 2024-04-25T16:05:07.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:07 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:05:07.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:07 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:07.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:07 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:05:07.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:07 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:05:07.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:07 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:09.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:09 smithi063 ceph-mon[32413]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 861 MiB/s wr, 2.23k op/s 2024-04-25T16:05:09.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:09 smithi186 ceph-mon[33258]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 861 MiB/s wr, 2.23k op/s 2024-04-25T16:05:09.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:09 smithi031 ceph-mon[29140]: pgmap v466: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 861 MiB/s wr, 2.23k op/s 2024-04-25T16:05:11.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:11 smithi186 ceph-mon[33258]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 798 MiB/s wr, 1.81k op/s 2024-04-25T16:05:11.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:11 smithi031 ceph-mon[29140]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 798 MiB/s wr, 1.81k op/s 2024-04-25T16:05:11.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:11 smithi063 ceph-mon[32413]: pgmap v467: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 798 MiB/s wr, 1.81k op/s 2024-04-25T16:05:12.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.248+0000 7f0e44841640 1 -- 172.21.15.31:0/4168004201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e40108770 msgr2=0x7f0e4010ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.248+0000 7f0e44841640 1 --2- 172.21.15.31:0/4168004201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e40108770 0x7f0e4010ab90 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009ca0 tx=0x7f0e3802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.248+0000 7f0e44841640 1 -- 172.21.15.31:0/4168004201 shutdown_connections 2024-04-25T16:05:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.248+0000 7f0e44841640 1 --2- 172.21.15.31:0/4168004201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e40108770 0x7f0e4010ab90 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.249+0000 7f0e44841640 1 --2- 172.21.15.31:0/4168004201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e40103a70 0x7f0e40103e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.249+0000 7f0e44841640 1 --2- 172.21.15.31:0/4168004201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e40102830 0x7f0e40103530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.250 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.249+0000 7f0e44841640 1 -- 172.21.15.31:0/4168004201 >> 172.21.15.31:0/4168004201 conn(0x7f0e400fda20 msgr2=0x7f0e400ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.249+0000 7f0e44841640 1 -- 172.21.15.31:0/4168004201 shutdown_connections 2024-04-25T16:05:12.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.249+0000 7f0e44841640 1 -- 172.21.15.31:0/4168004201 wait complete. 2024-04-25T16:05:12.252 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.252+0000 7f0e44841640 1 Processor -- start 2024-04-25T16:05:12.253 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.252+0000 7f0e44841640 1 -- start start 2024-04-25T16:05:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.253+0000 7f0e44841640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e40102830 0x7f0e40199000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.254 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.253+0000 7f0e44841640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e40103a70 0x7f0e40199540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.253+0000 7f0e44841640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e40108770 0x7f0e401a0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.253+0000 7f0e44841640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4010e420 con 0x7f0e40103a70 2024-04-25T16:05:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.253+0000 7f0e44841640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4010e2a0 con 0x7f0e40108770 2024-04-25T16:05:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.254+0000 7f0e44841640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e4010e5a0 con 0x7f0e40102830 2024-04-25T16:05:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.254+0000 7f0e3f59e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e40108770 0x7f0e401a0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.254+0000 7f0e3f59e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e40108770 0x7f0e401a0460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34864/0 (socket says 172.21.15.31:34864) 2024-04-25T16:05:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.254+0000 7f0e3f59e640 1 -- 172.21.15.31:0/635394965 learned_addr learned my addr 172.21.15.31:0/635394965 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:12.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.254+0000 7f0e3e59c640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e40103a70 0x7f0e40199540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.255+0000 7f0e3f59e640 1 -- 172.21.15.31:0/635394965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e40102830 msgr2=0x7f0e40199000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:05:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.255+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e40102830 0x7f0e40199000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.255+0000 7f0e3f59e640 1 -- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e40103a70 msgr2=0x7f0e40199540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.255+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e40103a70 0x7f0e40199540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.255+0000 7f0e3f59e640 1 -- 172.21.15.31:0/635394965 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e401a0ad0 con 0x7f0e40108770 2024-04-25T16:05:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.256+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e40108770 0x7f0e401a0460 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e3802f520 tx=0x7f0e38031830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:12.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.261+0000 7f0e27fff640 1 -- 172.21.15.31:0/635394965 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e38004040 con 0x7f0e40108770 2024-04-25T16:05:12.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.262+0000 7f0e27fff640 1 -- 172.21.15.31:0/635394965 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e38004690 con 0x7f0e40108770 2024-04-25T16:05:12.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.262+0000 7f0e27fff640 1 -- 172.21.15.31:0/635394965 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3803ede0 con 0x7f0e40108770 2024-04-25T16:05:12.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.262+0000 7f0e44841640 1 -- 172.21.15.31:0/635394965 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e40074dd0 con 0x7f0e40108770 2024-04-25T16:05:12.263 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.262+0000 7f0e44841640 1 -- 172.21.15.31:0/635394965 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e400753a0 con 0x7f0e40108770 2024-04-25T16:05:12.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.269+0000 7f0e4bfcf740 1 Processor -- start 2024-04-25T16:05:12.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.274+0000 7f0e4bfcf740 1 -- start start 2024-04-25T16:05:12.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.275+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64ba1b0 0x55afd685c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.275+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 0x55afd685e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.275+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64994d0 0x55afd6860ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.275+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55afd67c0460 con 0x55afd64994d0 2024-04-25T16:05:12.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.275+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55afd677a590 con 0x55afd64ba1b0 2024-04-25T16:05:12.277 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.275+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55afd67bc420 con 0x55afd64912f0 2024-04-25T16:05:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.276+0000 7f0e3ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 0x55afd685e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.277+0000 7f0e3ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 0x55afd685e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37346/0 (socket says 172.21.15.31:37346) 2024-04-25T16:05:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.277+0000 7f0e3ed9d640 1 -- 172.21.15.31:0/1270254182 learned_addr learned my addr 172.21.15.31:0/1270254182 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.277+0000 7f0e3ed9d640 1 -- 172.21.15.31:0/1270254182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64ba1b0 msgr2=0x55afd685c160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.277+0000 7f0e3ed9d640 1 --2- 172.21.15.31:0/1270254182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64ba1b0 0x55afd685c160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.277+0000 7f0e3ed9d640 1 -- 172.21.15.31:0/1270254182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64994d0 msgr2=0x55afd6860ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.277+0000 7f0e3ed9d640 1 --2- 172.21.15.31:0/1270254182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64994d0 0x55afd6860ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.277+0000 7f0e3ed9d640 1 -- 172.21.15.31:0/1270254182 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd67870d0 con 0x55afd64912f0 2024-04-25T16:05:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.279+0000 7f0e3ed9d640 1 --2- 172.21.15.31:0/1270254182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 0x55afd685e770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000a940 tx=0x7f0e3000ae00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7674ab5bf16b5e58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.279+0000 7f0e257fa640 1 -- 172.21.15.31:0/1270254182 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e3000cd80 con 0x55afd64912f0 2024-04-25T16:05:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.280+0000 7f0e257fa640 1 -- 172.21.15.31:0/1270254182 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e30005020 con 0x55afd64912f0 2024-04-25T16:05:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.280+0000 7f0e257fa640 1 -- 172.21.15.31:0/1270254182 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30004b20 con 0x55afd64912f0 2024-04-25T16:05:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.282+0000 7f0e27fff640 1 -- 172.21.15.31:0/635394965 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0e380041e0 con 0x7f0e40108770 2024-04-25T16:05:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.282+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/1270254182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 msgr2=0x55afd685e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:12.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.282+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/1270254182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 0x55afd685e770 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000a940 tx=0x7f0e3000ae00 comp rx=0 tx=0).stop 2024-04-25T16:05:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.284+0000 7f0e27fff640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e280afbe0 0x7f0e280b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.284+0000 7f0e27fff640 1 -- 172.21.15.31:0/635394965 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0e38100320 con 0x7f0e40108770 2024-04-25T16:05:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.285+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/1270254182 shutdown_connections 2024-04-25T16:05:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.285+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/1270254182 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64994d0 0x55afd6860ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.285+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/1270254182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 0x55afd685e770 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.285+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/1270254182 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64ba1b0 0x55afd685c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.286 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.285+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/1270254182 >> 172.21.15.31:0/1270254182 conn(0x55afd64ae300 msgr2=0x55afd6856430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.291+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/1270254182 shutdown_connections 2024-04-25T16:05:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.294+0000 7f0e3e59c640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e280afbe0 0x7f0e280b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.296+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/1270254182 wait complete. 2024-04-25T16:05:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.297+0000 7f0e4bfcf740 1 -- start start 2024-04-25T16:05:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.297+0000 7f0e4bfcf740 1 Processor -- start 2024-04-25T16:05:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.299+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64912f0 0x55afd6875930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.299+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64994d0 0x55afd6875e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.299+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd687cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.300+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55afd6861a10 con 0x55afd64ba1b0 2024-04-25T16:05:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.300+0000 7f0e3ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64912f0 0x55afd6875930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.300+0000 7f0e3ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64912f0 0x55afd6875930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34892/0 (socket says 172.21.15.31:34892) 2024-04-25T16:05:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.300+0000 7f0e3ed9d640 1 -- 172.21.15.31:0/2769953980 learned_addr learned my addr 172.21.15.31:0/2769953980 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.300+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55afd6747b60 con 0x55afd64912f0 2024-04-25T16:05:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.300+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55afd6861b90 con 0x55afd64994d0 2024-04-25T16:05:12.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.300+0000 7f0e4bfcf740 10 client.0 _ll_register_callbacks cb 0x55afd676c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:12.303 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.302+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd687cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.304+0000 7f0e3e59c640 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64994d0 0x55afd6875e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.307+0000 7f0e3ed9d640 1 -- 172.21.15.31:0/2769953980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64994d0 msgr2=0x55afd6875e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.307+0000 7f0e3ed9d640 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64994d0 0x55afd6875e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.307+0000 7f0e3ed9d640 1 -- 172.21.15.31:0/2769953980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 msgr2=0x55afd687cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.307+0000 7f0e3ed9d640 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd687cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.307+0000 7f0e3ed9d640 1 -- 172.21.15.31:0/2769953980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd64d2210 con 0x55afd64912f0 2024-04-25T16:05:12.311 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.307+0000 7f0e3e59c640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e280afbe0 0x7f0e280b20f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0e34006aa0 tx=0x7f0e34006a30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.314+0000 7f0e3ed9d640 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64912f0 0x55afd6875930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0e30005400 tx=0x7f0e30005430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98874630d007c52a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:12.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.315+0000 7f0e22ff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:05:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.315+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2769953980 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e30027060 con 0x55afd64912f0 2024-04-25T16:05:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.316+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e30005020 con 0x55afd64912f0 2024-04-25T16:05:12.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.316+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2769953980 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0e30004aa0 con 0x55afd64912f0 2024-04-25T16:05:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.317+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2769953980 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0e30004390 con 0x55afd64912f0 2024-04-25T16:05:12.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.318+0000 7f0e4bfcf740 10 client.26123 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:12.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.318+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55afd659e620 con 0x55afd64912f0 2024-04-25T16:05:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.322+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2769953980 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e30005020 con 0x55afd64912f0 2024-04-25T16:05:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.322+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2769953980 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0e30021070 con 0x55afd64912f0 2024-04-25T16:05:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.322+0000 7f0e4bfcf740 10 client.26123 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.322+0000 7f0e4bfcf740 10 client.26123 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.322+0000 7f0e4bfcf740 20 client.26123 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.322+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55afd6478d80 0x55afd6883610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.322+0000 7f0e4bfcf740 4 client.26123 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:12.327 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.322+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afd67ce070 con 0x55afd6478d80 2024-04-25T16:05:12.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.333+0000 7f0e3e59c640 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55afd6478d80 0x55afd6883610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.350+0000 7f0e3e59c640 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55afd6478d80 0x55afd6883610 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad308a9c66f8e594 server_cookie=50f27fa07d695f86 in_seq=0 out_seq=0 2024-04-25T16:05:12.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.351+0000 7f0e22ff5640 10 client.26123 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:12.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.363+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2769953980 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55afd67ce070 con 0x55afd6478d80 2024-04-25T16:05:12.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.364+0000 7f0e22ff5640 10 client.26123 handle_command_reply: tid=0 2024-04-25T16:05:12.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.364+0000 7f0e4bfcf740 1 client.26123 shutdown 2024-04-25T16:05:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.367+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64912f0 msgr2=0x55afd6875930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:12.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.367+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64912f0 0x55afd6875930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0e30005400 tx=0x7f0e30005430 comp rx=0 tx=0).stop 2024-04-25T16:05:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.391+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 shutdown_connections 2024-04-25T16:05:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.391+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55afd6478d80 0x55afd6883610 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.391+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd687cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.391+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64994d0 0x55afd6875e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.391+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2769953980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64912f0 0x55afd6875930 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.391+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 >> 172.21.15.31:0/2769953980 conn(0x55afd64ae300 msgr2=0x55afd6862000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:12.394 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.391+0000 7f0e22ff5640 0 client.26123 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:12.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.394+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 shutdown_connections 2024-04-25T16:05:12.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.400+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2769953980 wait complete. 2024-04-25T16:05:12.405 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.403+0000 7f0e4bfcf740 20 client.26123 trim_cache size 0 max 16384 2024-04-25T16:05:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.499+0000 7f0e4bfcf740 1 Processor -- start 2024-04-25T16:05:12.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.499+0000 7f0e4bfcf740 1 -- start start 2024-04-25T16:05:12.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.499+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 0x55afd6874f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.499+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64994d0 0x55afd6875450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.499+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd6875990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.499+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e40074800 con 0x55afd64ba1b0 2024-04-25T16:05:12.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.499+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0e401b3280 con 0x55afd64994d0 2024-04-25T16:05:12.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.499+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e401b1e30 con 0x55afd64912f0 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.500+0000 7f0e3f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd6875990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.500+0000 7f0e3f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd6875990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53214/0 (socket says 172.21.15.31:53214) 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.500+0000 7f0e3f59e640 1 -- 172.21.15.31:0/550826674 learned_addr learned my addr 172.21.15.31:0/550826674 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.501+0000 7f0e3f59e640 1 -- 172.21.15.31:0/550826674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 msgr2=0x55afd6874f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.501+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/550826674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 0x55afd6874f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.501+0000 7f0e3f59e640 1 -- 172.21.15.31:0/550826674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64994d0 msgr2=0x55afd6875450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.501+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/550826674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64994d0 0x55afd6875450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.501+0000 7f0e3f59e640 1 -- 172.21.15.31:0/550826674 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd6859690 con 0x55afd64ba1b0 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.502+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/550826674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd6875990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009410 tx=0x7f0e380eb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84a789ca142a2bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.503+0000 7f0e24ff9640 1 -- 172.21.15.31:0/550826674 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e380044d0 con 0x55afd64ba1b0 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.503+0000 7f0e24ff9640 1 -- 172.21.15.31:0/550826674 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e380e2070 con 0x55afd64ba1b0 2024-04-25T16:05:12.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.503+0000 7f0e24ff9640 1 -- 172.21.15.31:0/550826674 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e380ccbc0 con 0x55afd64ba1b0 2024-04-25T16:05:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.504+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/550826674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 msgr2=0x55afd6875990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.504+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/550826674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd6875990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0e38009410 tx=0x7f0e380eb040 comp rx=0 tx=0).stop 2024-04-25T16:05:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.507+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/550826674 shutdown_connections 2024-04-25T16:05:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.507+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/550826674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64ba1b0 0x55afd6875990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.507+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/550826674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64994d0 0x55afd6875450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.507+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/550826674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64912f0 0x55afd6874f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.507+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/550826674 >> 172.21.15.31:0/550826674 conn(0x55afd64ae300 msgr2=0x55afd685c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:12.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.517+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/550826674 shutdown_connections 2024-04-25T16:05:12.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.518+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/550826674 wait complete. 2024-04-25T16:05:12.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.519+0000 7f0e4bfcf740 1 -- start start 2024-04-25T16:05:12.521 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.520+0000 7f0e4bfcf740 1 Processor -- start 2024-04-25T16:05:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.520+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64912f0 0x55afd68792b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.520+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64994d0 0x55afd68797f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.520+0000 7f0e4bfcf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64ba1b0 0x55afd68e99b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.521+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0e401b2bb0 con 0x55afd64912f0 2024-04-25T16:05:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.521+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0e401b24e0 con 0x55afd64994d0 2024-04-25T16:05:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.521+0000 7f0e4bfcf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0e400727e0 con 0x55afd64ba1b0 2024-04-25T16:05:12.522 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.521+0000 7f0e4bfcf740 10 client.0 _ll_register_callbacks cb 0x7f0e3000a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.524+0000 7f0e3ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64912f0 0x55afd68792b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.524+0000 7f0e3ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64912f0 0x55afd68792b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53226/0 (socket says 172.21.15.31:53226) 2024-04-25T16:05:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.524+0000 7f0e3ed9d640 1 -- 172.21.15.31:0/2083441163 learned_addr learned my addr 172.21.15.31:0/2083441163 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.524+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64ba1b0 0x55afd68e99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.525 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.524+0000 7f0e3e59c640 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64994d0 0x55afd68797f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.527+0000 7f0e3f59e640 1 -- 172.21.15.31:0/2083441163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64994d0 msgr2=0x55afd68797f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.527+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64994d0 0x55afd68797f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.527+0000 7f0e3f59e640 1 -- 172.21.15.31:0/2083441163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64912f0 msgr2=0x55afd68792b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.527+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64912f0 0x55afd68792b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:12.529 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.527+0000 7f0e3f59e640 1 -- 172.21.15.31:0/2083441163 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd687c680 con 0x55afd64ba1b0 2024-04-25T16:05:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.533+0000 7f0e3f59e640 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64ba1b0 0x55afd68e99b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0e380c5a80 tx=0x7f0e380c3fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a010121d7decc4c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.533+0000 7f0e22ff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:12.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.534+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2083441163 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e380098d0 con 0x55afd64ba1b0 2024-04-25T16:05:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.534+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2083441163 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0e380e2070 con 0x55afd64ba1b0 2024-04-25T16:05:12.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.534+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2083441163 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0e380c4e00 con 0x55afd64ba1b0 2024-04-25T16:05:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.535+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2083441163 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0e38004840 con 0x55afd64ba1b0 2024-04-25T16:05:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.535+0000 7f0e4bfcf740 10 client.26068 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:12.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.535+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2083441163 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55afd68eaf10 con 0x55afd64ba1b0 2024-04-25T16:05:12.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.538+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2083441163 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0e380e2070 con 0x55afd64ba1b0 2024-04-25T16:05:12.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.538+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2083441163 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0e380ff040 con 0x55afd64ba1b0 2024-04-25T16:05:12.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.539+0000 7f0e4bfcf740 10 client.26068 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.539+0000 7f0e4bfcf740 10 client.26068 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.539+0000 7f0e4bfcf740 20 client.26068 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.539+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55afd644e1c0 0x55afd68ed7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.539+0000 7f0e4bfcf740 4 client.26068 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:05:12.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.539+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2083441163 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55afd67c7ee0 con 0x55afd644e1c0 2024-04-25T16:05:12.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.548+0000 7f0e3ed9d640 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55afd644e1c0 0x55afd68ed7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.559+0000 7f0e22ff5640 10 client.26068 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:12.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:12.559+0000 7f0e3ed9d640 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55afd644e1c0 0x55afd68ed7c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0185066b82e6289 server_cookie=35a2e678dd07444b in_seq=0 out_seq=0 2024-04-25T16:05:12.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:12 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:12 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:12.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:12 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:13.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:13 smithi031 ceph-mon[29140]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 791 MiB/s wr, 1.67k op/s 2024-04-25T16:05:13.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:13 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:05:13.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:13 smithi063 ceph-mon[32413]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 791 MiB/s wr, 1.67k op/s 2024-04-25T16:05:13.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:13 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:05:13.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:13 smithi186 ceph-mon[33258]: pgmap v468: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 791 MiB/s wr, 1.67k op/s 2024-04-25T16:05:13.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:13 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:05:14.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:14 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:05:14.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:14 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:05:14.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:14 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:05:15.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:15 smithi063 ceph-mon[32413]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 969 MiB/s wr, 1.99k op/s 2024-04-25T16:05:15.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:15 smithi186 ceph-mon[33258]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 969 MiB/s wr, 1.99k op/s 2024-04-25T16:05:15.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:15 smithi031 ceph-mon[29140]: pgmap v469: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 969 MiB/s wr, 1.99k op/s 2024-04-25T16:05:16.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:16 smithi063 ceph-mon[32413]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 757 MiB/s wr, 1.75k op/s 2024-04-25T16:05:16.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:16 smithi186 ceph-mon[33258]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 757 MiB/s wr, 1.75k op/s 2024-04-25T16:05:16.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:16 smithi031 ceph-mon[29140]: pgmap v470: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 757 MiB/s wr, 1.75k op/s 2024-04-25T16:05:18.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:18 smithi186 ceph-mon[33258]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 904 MiB/s wr, 1.99k op/s 2024-04-25T16:05:18.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:18 smithi031 ceph-mon[29140]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 904 MiB/s wr, 1.99k op/s 2024-04-25T16:05:19.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:18 smithi063 ceph-mon[32413]: pgmap v471: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 904 MiB/s wr, 1.99k op/s 2024-04-25T16:05:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.827+0000 7f0e22ff5640 1 -- 172.21.15.31:0/2083441163 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55afd67c7ee0 con 0x55afd644e1c0 2024-04-25T16:05:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.827+0000 7f0e22ff5640 10 client.26068 handle_command_reply: tid=0 2024-04-25T16:05:20.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.827+0000 7f0e4bfcf740 1 client.26068 shutdown 2024-04-25T16:05:20.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.831+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2083441163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64ba1b0 msgr2=0x55afd68e99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:20.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.831+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64ba1b0 0x55afd68e99b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0e380c5a80 tx=0x7f0e380c3fb0 comp rx=0 tx=0).stop 2024-04-25T16:05:20.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.832+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2083441163 shutdown_connections 2024-04-25T16:05:20.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.832+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55afd644e1c0 0x55afd68ed7c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:20.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.832+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55afd64ba1b0 0x55afd68e99b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:20.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.832+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55afd64994d0 0x55afd68797f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:20.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.832+0000 7f0e4bfcf740 1 --2- 172.21.15.31:0/2083441163 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55afd64912f0 0x55afd68792b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:20.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.832+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2083441163 >> 172.21.15.31:0/2083441163 conn(0x55afd64ae300 msgr2=0x55afd685a0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:20.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.832+0000 7f0e22ff5640 0 client.26068 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:20.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.832+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2083441163 shutdown_connections 2024-04-25T16:05:20.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.832+0000 7f0e4bfcf740 1 -- 172.21.15.31:0/2083441163 wait complete. 2024-04-25T16:05:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.833+0000 7f0e4bfcf740 20 client.26068 trim_cache size 0 max 16384 2024-04-25T16:05:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:05:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:05:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:05:20.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:05:20.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.847+0000 7f0e44841640 1 -- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e280afbe0 msgr2=0x7f0e280b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.847+0000 7f0e44841640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e280afbe0 0x7f0e280b20f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0e34006aa0 tx=0x7f0e34006a30 comp rx=0 tx=0).stop 2024-04-25T16:05:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.847+0000 7f0e44841640 1 -- 172.21.15.31:0/635394965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e40108770 msgr2=0x7f0e401a0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:20.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.847+0000 7f0e44841640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e40108770 0x7f0e401a0460 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0e3802f520 tx=0x7f0e38031830 comp rx=0 tx=0).stop 2024-04-25T16:05:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.852+0000 7f0e44841640 1 -- 172.21.15.31:0/635394965 shutdown_connections 2024-04-25T16:05:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.852+0000 7f0e44841640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0e280afbe0 0x7f0e280b20f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.852+0000 7f0e44841640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e40108770 0x7f0e401a0460 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.852+0000 7f0e44841640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0e40103a70 0x7f0e40199540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.852+0000 7f0e44841640 1 --2- 172.21.15.31:0/635394965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e40102830 0x7f0e40199000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.852+0000 7f0e44841640 1 -- 172.21.15.31:0/635394965 >> 172.21.15.31:0/635394965 conn(0x7f0e400fda20 msgr2=0x7f0e40104750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.852+0000 7f0e44841640 1 -- 172.21.15.31:0/635394965 shutdown_connections 2024-04-25T16:05:20.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:20.853+0000 7f0e44841640 1 -- 172.21.15.31:0/635394965 wait complete. 2024-04-25T16:05:20.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:20 smithi186 ceph-mon[33258]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 838 MiB/s wr, 1.65k op/s 2024-04-25T16:05:20.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:20 smithi031 ceph-mon[29140]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 838 MiB/s wr, 1.65k op/s 2024-04-25T16:05:21.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:20 smithi063 ceph-mon[32413]: pgmap v472: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 838 MiB/s wr, 1.65k op/s 2024-04-25T16:05:22.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:22 smithi031 ceph-mon[29140]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 791 MiB/s wr, 1.48k op/s 2024-04-25T16:05:23.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:22 smithi063 ceph-mon[32413]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 791 MiB/s wr, 1.48k op/s 2024-04-25T16:05:23.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:22 smithi186 ceph-mon[33258]: pgmap v473: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 791 MiB/s wr, 1.48k op/s 2024-04-25T16:05:24.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:24 smithi031 ceph-mon[29140]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 990 MiB/s wr, 1.89k op/s 2024-04-25T16:05:25.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:24 smithi063 ceph-mon[32413]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 990 MiB/s wr, 1.89k op/s 2024-04-25T16:05:25.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:24 smithi186 ceph-mon[33258]: pgmap v474: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 990 MiB/s wr, 1.89k op/s 2024-04-25T16:05:26.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:26 smithi031 ceph-mon[29140]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 777 MiB/s wr, 1.48k op/s 2024-04-25T16:05:27.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:26 smithi063 ceph-mon[32413]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 777 MiB/s wr, 1.48k op/s 2024-04-25T16:05:27.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:26 smithi186 ceph-mon[33258]: pgmap v475: 97 pgs: 97 active+clean; 10 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 777 MiB/s wr, 1.48k op/s 2024-04-25T16:05:28.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:28 smithi031 ceph-mon[29140]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 918 MiB/s wr, 1.96k op/s 2024-04-25T16:05:29.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:28 smithi063 ceph-mon[32413]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 918 MiB/s wr, 1.96k op/s 2024-04-25T16:05:29.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:28 smithi186 ceph-mon[33258]: pgmap v476: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 918 MiB/s wr, 1.96k op/s 2024-04-25T16:05:30.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:30 smithi031 ceph-mon[29140]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 826 MiB/s wr, 1.90k op/s 2024-04-25T16:05:31.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:30 smithi063 ceph-mon[32413]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 826 MiB/s wr, 1.90k op/s 2024-04-25T16:05:31.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:30 smithi186 ceph-mon[33258]: pgmap v477: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 826 MiB/s wr, 1.90k op/s 2024-04-25T16:05:33.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:32 smithi063 ceph-mon[32413]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 776 MiB/s wr, 1.86k op/s 2024-04-25T16:05:33.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:32 smithi186 ceph-mon[33258]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 776 MiB/s wr, 1.86k op/s 2024-04-25T16:05:33.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:32 smithi031 ceph-mon[29140]: pgmap v478: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 776 MiB/s wr, 1.86k op/s 2024-04-25T16:05:34.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.282+0000 7f557759e640 1 -- 172.21.15.31:0/948235486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f557810e2c0 msgr2=0x7f55781106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:34.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.282+0000 7f557759e640 1 --2- 172.21.15.31:0/948235486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f557810e2c0 0x7f55781106e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5568009900 tx=0x7f556802a420 comp rx=0 tx=0).stop 2024-04-25T16:05:34.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.280+0000 7f557559a640 1 -- 172.21.15.31:0/948235486 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f5568031810 con 0x7f557810e2c0 2024-04-25T16:05:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.288+0000 7f557759e640 1 -- 172.21.15.31:0/948235486 shutdown_connections 2024-04-25T16:05:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.288+0000 7f557759e640 1 --2- 172.21.15.31:0/948235486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5578110cb0 0x7f55781131c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.288+0000 7f557759e640 1 --2- 172.21.15.31:0/948235486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f557810e2c0 0x7f55781106e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.288+0000 7f557759e640 1 --2- 172.21.15.31:0/948235486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f557810b960 0x7f557810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.288+0000 7f557759e640 1 -- 172.21.15.31:0/948235486 >> 172.21.15.31:0/948235486 conn(0x7f5578070330 msgr2=0x7f55780727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:34.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.295+0000 7f557759e640 1 -- 172.21.15.31:0/948235486 shutdown_connections 2024-04-25T16:05:34.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.300+0000 7f557759e640 1 -- 172.21.15.31:0/948235486 wait complete. 2024-04-25T16:05:34.305 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.303+0000 7f557759e640 1 Processor -- start 2024-04-25T16:05:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.307+0000 7f557759e640 1 -- start start 2024-04-25T16:05:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.308+0000 7f557759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f557810b960 0x7f557810af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.308+0000 7f557759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f557810e2c0 0x7f55781a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.308+0000 7f557759e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5578110cb0 0x7f55781b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.308+0000 7f557759e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5578116b80 con 0x7f557810e2c0 2024-04-25T16:05:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.308+0000 7f557759e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5578116a00 con 0x7f5578110cb0 2024-04-25T16:05:34.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.308+0000 7f557759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5578116d00 con 0x7f557810b960 2024-04-25T16:05:34.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.311+0000 7f5576d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5578110cb0 0x7f55781b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.311+0000 7f5576d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5578110cb0 0x7f55781b0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52260/0 (socket says 172.21.15.31:52260) 2024-04-25T16:05:34.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.311+0000 7f5576d9d640 1 -- 172.21.15.31:0/2071748964 learned_addr learned my addr 172.21.15.31:0/2071748964 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:34.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.314+0000 7f5576d9d640 1 -- 172.21.15.31:0/2071748964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f557810b960 msgr2=0x7f557810af00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:05:34.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.314+0000 7f5576d9d640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f557810b960 0x7f557810af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.314+0000 7f5576d9d640 1 -- 172.21.15.31:0/2071748964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f557810e2c0 msgr2=0x7f55781a5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:34.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.314+0000 7f5576d9d640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f557810e2c0 0x7f55781a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.317 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.314+0000 7f5576d9d640 1 -- 172.21.15.31:0/2071748964 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55781a7ab0 con 0x7f5578110cb0 2024-04-25T16:05:34.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.320+0000 7f5576d9d640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5578110cb0 0x7f55781b0190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f556c00d750 tx=0x7f556c00dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:34.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.321+0000 7f55677fe640 1 -- 172.21.15.31:0/2071748964 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c017070 con 0x7f5578110cb0 2024-04-25T16:05:34.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.321+0000 7f557759e640 1 -- 172.21.15.31:0/2071748964 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55781a7da0 con 0x7f5578110cb0 2024-04-25T16:05:34.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.322+0000 7f557759e640 1 -- 172.21.15.31:0/2071748964 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55781a82f0 con 0x7f5578110cb0 2024-04-25T16:05:34.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.322+0000 7f55677fe640 1 -- 172.21.15.31:0/2071748964 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f556c0054d0 con 0x7f5578110cb0 2024-04-25T16:05:34.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.322+0000 7f55677fe640 1 -- 172.21.15.31:0/2071748964 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c004c30 con 0x7f5578110cb0 2024-04-25T16:05:34.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.329+0000 7f5583db8740 1 Processor -- start 2024-04-25T16:05:34.333 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.332+0000 7f55677fe640 1 -- 172.21.15.31:0/2071748964 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f556c009bd0 con 0x7f5578110cb0 2024-04-25T16:05:34.334 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.333+0000 7f5583db8740 1 -- start start 2024-04-25T16:05:34.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.334+0000 7f5583db8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef481b0 0x557e4f2e8b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.334+0000 7f5583db8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 0x557e4f2e90a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.334+0000 7f5583db8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef274d0 0x557e4f2effa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.334+0000 7f5583db8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557e4f24e460 con 0x557e4ef274d0 2024-04-25T16:05:34.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.334+0000 7f5583db8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557e4f208590 con 0x557e4ef481b0 2024-04-25T16:05:34.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.334+0000 7f5583db8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557e4f24a420 con 0x557e4ef1f2f0 2024-04-25T16:05:34.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.335+0000 7f5575d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef481b0 0x557e4f2e8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.335+0000 7f5575d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef481b0 0x557e4f2e8b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52264/0 (socket says 172.21.15.31:52264) 2024-04-25T16:05:34.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.335+0000 7f5575d9b640 1 -- 172.21.15.31:0/2040738507 learned_addr learned my addr 172.21.15.31:0/2040738507 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:34.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.336+0000 7f557659c640 1 --2- 172.21.15.31:0/2040738507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 0x557e4f2e90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.337+0000 7f5575d9b640 1 -- 172.21.15.31:0/2040738507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 msgr2=0x557e4f2e90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.337+0000 7f5575d9b640 1 --2- 172.21.15.31:0/2040738507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 0x557e4f2e90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.337+0000 7f5575d9b640 1 -- 172.21.15.31:0/2040738507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef274d0 msgr2=0x557e4f2effa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.337+0000 7f5575d9b640 1 --2- 172.21.15.31:0/2040738507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef274d0 0x557e4f2effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.337+0000 7f5575d9b640 1 -- 172.21.15.31:0/2040738507 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e4f2150d0 con 0x557e4ef481b0 2024-04-25T16:05:34.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.338+0000 7f5575d9b640 1 --2- 172.21.15.31:0/2040738507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef481b0 0x557e4f2e8b20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5568009a20 tx=0x7f5568000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb5487e367fc8633 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:34.339 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.338+0000 7f55677fe640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55600afcf0 0x7f55600b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.341 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.340+0000 7f5564ff9640 1 -- 172.21.15.31:0/2040738507 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568040070 con 0x557e4ef481b0 2024-04-25T16:05:34.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.340+0000 7f5564ff9640 1 -- 172.21.15.31:0/2040738507 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5568031e30 con 0x557e4ef481b0 2024-04-25T16:05:34.342 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.340+0000 7f5564ff9640 1 -- 172.21.15.31:0/2040738507 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568035030 con 0x557e4ef481b0 2024-04-25T16:05:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.346+0000 7f5583db8740 1 -- 172.21.15.31:0/2040738507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef481b0 msgr2=0x557e4f2e8b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.346+0000 7f5583db8740 1 --2- 172.21.15.31:0/2040738507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef481b0 0x557e4f2e8b20 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5568009a20 tx=0x7f5568000bc0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.347+0000 7f5583db8740 1 -- 172.21.15.31:0/2040738507 shutdown_connections 2024-04-25T16:05:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.347+0000 7f5583db8740 1 --2- 172.21.15.31:0/2040738507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef274d0 0x557e4f2effa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.347+0000 7f5583db8740 1 --2- 172.21.15.31:0/2040738507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 0x557e4f2e90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.347+0000 7f5583db8740 1 --2- 172.21.15.31:0/2040738507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef481b0 0x557e4f2e8b20 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.347+0000 7f5583db8740 1 -- 172.21.15.31:0/2040738507 >> 172.21.15.31:0/2040738507 conn(0x557e4ef3c300 msgr2=0x557e4f2e44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:34.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.348+0000 7f55677fe640 1 -- 172.21.15.31:0/2071748964 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f556c0de5f0 con 0x7f5578110cb0 2024-04-25T16:05:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.354+0000 7f557659c640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55600afcf0 0x7f55600b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.355+0000 7f5583db8740 1 -- 172.21.15.31:0/2040738507 shutdown_connections 2024-04-25T16:05:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.358+0000 7f5583db8740 1 -- 172.21.15.31:0/2040738507 wait complete. 2024-04-25T16:05:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.359+0000 7f5583db8740 1 -- start start 2024-04-25T16:05:34.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.359+0000 7f5583db8740 1 Processor -- start 2024-04-25T16:05:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.362+0000 7f5583db8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef1f2f0 0x557e4f2f2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.362+0000 7f5583db8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.362+0000 7f5583db8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef481b0 0x557e4f2f9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.362+0000 7f5583db8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557e4f2f0990 con 0x557e4ef1f2f0 2024-04-25T16:05:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.362+0000 7f5583db8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557e4f1d5b60 con 0x557e4ef274d0 2024-04-25T16:05:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.362+0000 7f5583db8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557e4f2f0b10 con 0x557e4ef481b0 2024-04-25T16:05:34.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.362+0000 7f5583db8740 10 client.0 _ll_register_callbacks cb 0x557e4f1fa2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.368+0000 7f5575d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.368+0000 7f5576d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef481b0 0x557e4f2f9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.368+0000 7f5575d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52280/0 (socket says 172.21.15.31:52280) 2024-04-25T16:05:34.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.368+0000 7f5575d9b640 1 -- 172.21.15.31:0/1783979321 learned_addr learned my addr 172.21.15.31:0/1783979321 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:34.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.371+0000 7f557659c640 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef1f2f0 0x557e4f2f2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.372+0000 7f5575d9b640 1 -- 172.21.15.31:0/1783979321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef481b0 msgr2=0x557e4f2f9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.372+0000 7f5575d9b640 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef481b0 0x557e4f2f9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.372+0000 7f5575d9b640 1 -- 172.21.15.31:0/1783979321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef1f2f0 msgr2=0x557e4f2f2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.372+0000 7f5575d9b640 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef1f2f0 0x557e4f2f2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.372+0000 7f5575d9b640 1 -- 172.21.15.31:0/1783979321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e4ef60210 con 0x557e4ef274d0 2024-04-25T16:05:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.373+0000 7f5575d9b640 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f2ec0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f55680099f0 tx=0x7f55680042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ab18e55132b0dca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.374+0000 7f555dffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:05:34.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.374+0000 7f555dffb640 1 -- 172.21.15.31:0/1783979321 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568035040 con 0x557e4ef274d0 2024-04-25T16:05:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.375+0000 7f5583db8740 1 -- 172.21.15.31:0/1783979321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5568031e30 con 0x557e4ef274d0 2024-04-25T16:05:34.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.375+0000 7f555dffb640 1 -- 172.21.15.31:0/1783979321 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f556803c880 con 0x557e4ef274d0 2024-04-25T16:05:34.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.379+0000 7f555dffb640 1 -- 172.21.15.31:0/1783979321 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5568040040 con 0x557e4ef274d0 2024-04-25T16:05:34.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.379+0000 7f5583db8740 10 client.26138 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:34.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.380+0000 7f5583db8740 1 -- 172.21.15.31:0/1783979321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557e4f02c620 con 0x557e4ef274d0 2024-04-25T16:05:34.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.380+0000 7f557659c640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55600afcf0 0x7f55600b2200 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x557e4f2e5240 tx=0x7f55700138e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:34.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.383+0000 7f555dffb640 1 -- 172.21.15.31:0/1783979321 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5568031e30 con 0x557e4ef274d0 2024-04-25T16:05:34.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.385+0000 7f555dffb640 1 -- 172.21.15.31:0/1783979321 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f556803a040 con 0x557e4ef274d0 2024-04-25T16:05:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.386+0000 7f5583db8740 10 client.26138 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.386+0000 7f5583db8740 10 client.26138 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:34.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.386+0000 7f5583db8740 20 client.26138 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.386+0000 7f5583db8740 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557e4ef06d80 0x557e4f3006b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.386+0000 7f5583db8740 4 client.26138 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:34.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.386+0000 7f5583db8740 1 -- 172.21.15.31:0/1783979321 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557e4f25c070 con 0x557e4ef06d80 2024-04-25T16:05:34.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.389+0000 7f557659c640 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557e4ef06d80 0x557e4f3006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.396+0000 7f557659c640 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557e4ef06d80 0x557e4f3006b0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a83bbea133d4374b server_cookie=91679fa20ea66b85 in_seq=0 out_seq=0 2024-04-25T16:05:34.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.396+0000 7f555dffb640 10 client.26138 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.399+0000 7f555dffb640 1 -- 172.21.15.31:0/1783979321 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557e4f25c070 con 0x557e4ef06d80 2024-04-25T16:05:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.399+0000 7f555dffb640 10 client.26138 handle_command_reply: tid=0 2024-04-25T16:05:34.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.399+0000 7f5583db8740 1 client.26138 shutdown 2024-04-25T16:05:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.406+0000 7f5583db8740 1 -- 172.21.15.31:0/1783979321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 msgr2=0x557e4f2f2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:34.408 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.406+0000 7f5583db8740 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f2ec0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f55680099f0 tx=0x7f55680042e0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.406+0000 7f5583db8740 1 -- 172.21.15.31:0/1783979321 shutdown_connections 2024-04-25T16:05:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.406+0000 7f5583db8740 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557e4ef06d80 0x557e4f3006b0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.406+0000 7f5583db8740 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef481b0 0x557e4f2f9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.406+0000 7f5583db8740 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f2ec0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.406+0000 7f5583db8740 1 --2- 172.21.15.31:0/1783979321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef1f2f0 0x557e4f2f2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.406+0000 7f5583db8740 1 -- 172.21.15.31:0/1783979321 >> 172.21.15.31:0/1783979321 conn(0x557e4ef3c300 msgr2=0x557e4f2e73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:34.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.407+0000 7f555dffb640 0 client.26138 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.407+0000 7f5583db8740 1 -- 172.21.15.31:0/1783979321 shutdown_connections 2024-04-25T16:05:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.407+0000 7f5583db8740 1 -- 172.21.15.31:0/1783979321 wait complete. 2024-04-25T16:05:34.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.408+0000 7f5583db8740 20 client.26138 trim_cache size 0 max 16384 2024-04-25T16:05:34.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.473+0000 7f5583db8740 1 Processor -- start 2024-04-25T16:05:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.473+0000 7f5583db8740 1 -- start start 2024-04-25T16:05:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.474+0000 7f5583db8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 0x557e4f2e3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.474+0000 7f5583db8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.474+0000 7f5583db8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef481b0 0x557e4f2f0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.474+0000 7f5583db8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f555800e3c0 con 0x557e4ef481b0 2024-04-25T16:05:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.474+0000 7f5583db8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f555800ff00 con 0x557e4ef274d0 2024-04-25T16:05:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.474+0000 7f5583db8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f555800ea90 con 0x557e4ef1f2f0 2024-04-25T16:05:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.474+0000 7f5575d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.475+0000 7f557659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 0x557e4f2e3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.475+0000 7f5575d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52282/0 (socket says 172.21.15.31:52282) 2024-04-25T16:05:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.475+0000 7f5575d9b640 1 -- 172.21.15.31:0/2494758039 learned_addr learned my addr 172.21.15.31:0/2494758039 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.477+0000 7f5576d9d640 1 --2- 172.21.15.31:0/2494758039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef481b0 0x557e4f2f0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.478+0000 7f5575d9b640 1 -- 172.21.15.31:0/2494758039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 msgr2=0x557e4f2e3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.478+0000 7f5575d9b640 1 --2- 172.21.15.31:0/2494758039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 0x557e4f2e3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.478+0000 7f5575d9b640 1 -- 172.21.15.31:0/2494758039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef481b0 msgr2=0x557e4f2f0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.478+0000 7f5575d9b640 1 --2- 172.21.15.31:0/2494758039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef481b0 0x557e4f2f0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.478+0000 7f5575d9b640 1 -- 172.21.15.31:0/2494758039 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557e4f2fa210 con 0x557e4ef274d0 2024-04-25T16:05:34.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.484+0000 7f5575d9b640 1 --2- 172.21.15.31:0/2494758039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f9cd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5568004350 tx=0x7f5568000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b35af1a9e1f69e4e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.491+0000 7f555ffff640 1 -- 172.21.15.31:0/2494758039 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556805e030 con 0x557e4ef274d0 2024-04-25T16:05:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.491+0000 7f555ffff640 1 -- 172.21.15.31:0/2494758039 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5568055030 con 0x557e4ef274d0 2024-04-25T16:05:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.491+0000 7f555ffff640 1 -- 172.21.15.31:0/2494758039 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5568035040 con 0x557e4ef274d0 2024-04-25T16:05:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.494+0000 7f5583db8740 1 -- 172.21.15.31:0/2494758039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 msgr2=0x557e4f2f9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.494+0000 7f5583db8740 1 --2- 172.21.15.31:0/2494758039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f9cd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5568004350 tx=0x7f5568000bc0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.494+0000 7f5583db8740 1 -- 172.21.15.31:0/2494758039 shutdown_connections 2024-04-25T16:05:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.494+0000 7f5583db8740 1 --2- 172.21.15.31:0/2494758039 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef481b0 0x557e4f2f0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.494+0000 7f5583db8740 1 --2- 172.21.15.31:0/2494758039 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef274d0 0x557e4f2f9cd0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5568004350 tx=0x7f5568000bc0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.494+0000 7f5583db8740 1 --2- 172.21.15.31:0/2494758039 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4ef1f2f0 0x557e4f2e3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.494+0000 7f5583db8740 1 -- 172.21.15.31:0/2494758039 >> 172.21.15.31:0/2494758039 conn(0x557e4ef3c300 msgr2=0x557e4f2ece20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.501+0000 7f5583db8740 1 -- 172.21.15.31:0/2494758039 shutdown_connections 2024-04-25T16:05:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.502+0000 7f5583db8740 1 -- 172.21.15.31:0/2494758039 wait complete. 2024-04-25T16:05:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.502+0000 7f5583db8740 1 -- start start 2024-04-25T16:05:34.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.502+0000 7f5583db8740 1 Processor -- start 2024-04-25T16:05:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.503+0000 7f5583db8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef1f2f0 0x557e4f2f6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.503+0000 7f5583db8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef481b0 0x557e4f2f7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.503+0000 7f5583db8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4eedc1c0 0x557e4f2f82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.503+0000 7f5583db8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f555800f830 con 0x557e4ef481b0 2024-04-25T16:05:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.503+0000 7f5583db8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f555800f160 con 0x557e4ef1f2f0 2024-04-25T16:05:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.503+0000 7f5583db8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f555800c450 con 0x557e4eedc1c0 2024-04-25T16:05:34.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.503+0000 7f5583db8740 10 client.0 _ll_register_callbacks cb 0x7f5568031510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:34.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.505+0000 7f5576d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4eedc1c0 0x557e4f2f82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.506+0000 7f5576d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4eedc1c0 0x557e4f2f82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56224/0 (socket says 172.21.15.31:56224) 2024-04-25T16:05:34.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.506+0000 7f5576d9d640 1 -- 172.21.15.31:0/646802855 learned_addr learned my addr 172.21.15.31:0/646802855 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:34.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.507+0000 7f5576d9d640 1 -- 172.21.15.31:0/646802855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef1f2f0 msgr2=0x557e4f2f6e50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:34.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.508+0000 7f5576d9d640 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef1f2f0 0x557e4f2f6e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.508+0000 7f5576d9d640 1 -- 172.21.15.31:0/646802855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef481b0 msgr2=0x557e4f2f7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:34.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.508+0000 7f5576d9d640 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef481b0 0x557e4f2f7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:34.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.508+0000 7f5576d9d640 1 -- 172.21.15.31:0/646802855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557e4f2f8880 con 0x557e4eedc1c0 2024-04-25T16:05:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.519+0000 7f5576d9d640 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4eedc1c0 0x557e4f2f82c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f556c0dc0a0 tx=0x7f556c0a6590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4dc83015ecb0dc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.522+0000 7f555dffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.523+0000 7f555dffb640 1 -- 172.21.15.31:0/646802855 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556c0c0060 con 0x557e4eedc1c0 2024-04-25T16:05:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.523+0000 7f555dffb640 1 -- 172.21.15.31:0/646802855 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f556c0c90b0 con 0x557e4eedc1c0 2024-04-25T16:05:34.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.524+0000 7f5583db8740 1 -- 172.21.15.31:0/646802855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5568055030 con 0x557e4eedc1c0 2024-04-25T16:05:34.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.528+0000 7f555dffb640 1 -- 172.21.15.31:0/646802855 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f556c014030 con 0x557e4eedc1c0 2024-04-25T16:05:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.530+0000 7f5583db8740 10 client.26080 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:34.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.530+0000 7f5583db8740 1 -- 172.21.15.31:0/646802855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557e4f2f9b80 con 0x557e4eedc1c0 2024-04-25T16:05:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.533+0000 7f555dffb640 1 -- 172.21.15.31:0/646802855 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5568055030 con 0x557e4eedc1c0 2024-04-25T16:05:34.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.533+0000 7f555dffb640 1 -- 172.21.15.31:0/646802855 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f556c0a5ce0 con 0x557e4eedc1c0 2024-04-25T16:05:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.535+0000 7f5583db8740 10 client.26080 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.535+0000 7f5583db8740 10 client.26080 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.535+0000 7f5583db8740 20 client.26080 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.535+0000 7f5583db8740 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557e4f368ca0 0x557e4f36b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:34.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.535+0000 7f5583db8740 4 client.26080 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:05:34.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.535+0000 7f5583db8740 1 -- 172.21.15.31:0/646802855 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557e4f255ee0 con 0x557e4f368ca0 2024-04-25T16:05:34.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.548+0000 7f557659c640 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557e4f368ca0 0x557e4f36b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:34.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.560+0000 7f555dffb640 10 client.26080 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:34.564 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:34.560+0000 7f557659c640 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557e4f368ca0 0x557e4f36b150 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77b986f28bb0dca6 server_cookie=7be36778adb09b4d in_seq=0 out_seq=0 2024-04-25T16:05:35.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:34 smithi063 ceph-mon[32413]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1009 MiB/s wr, 2.25k op/s 2024-04-25T16:05:35.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:34 smithi186 ceph-mon[33258]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1009 MiB/s wr, 2.25k op/s 2024-04-25T16:05:35.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:34 smithi031 ceph-mon[29140]: pgmap v479: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1009 MiB/s wr, 2.25k op/s 2024-04-25T16:05:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.199+0000 7f555dffb640 1 -- 172.21.15.31:0/646802855 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557e4f255ee0 con 0x557e4f368ca0 2024-04-25T16:05:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.199+0000 7f555dffb640 10 client.26080 handle_command_reply: tid=0 2024-04-25T16:05:36.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.199+0000 7f5583db8740 1 client.26080 shutdown 2024-04-25T16:05:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.200+0000 7f5583db8740 1 -- 172.21.15.31:0/646802855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4eedc1c0 msgr2=0x557e4f2f82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.200+0000 7f5583db8740 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4eedc1c0 0x557e4f2f82c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f556c0dc0a0 tx=0x7f556c0a6590 comp rx=0 tx=0).stop 2024-04-25T16:05:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.200+0000 7f5583db8740 1 -- 172.21.15.31:0/646802855 shutdown_connections 2024-04-25T16:05:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.200+0000 7f5583db8740 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557e4f368ca0 0x557e4f36b150 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.200+0000 7f5583db8740 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557e4eedc1c0 0x557e4f2f82c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:36.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.200+0000 7f5583db8740 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557e4ef481b0 0x557e4f2f7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.200+0000 7f5583db8740 1 --2- 172.21.15.31:0/646802855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557e4ef1f2f0 0x557e4f2f6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.200+0000 7f5583db8740 1 -- 172.21.15.31:0/646802855 >> 172.21.15.31:0/646802855 conn(0x557e4ef3c300 msgr2=0x557e4f2f0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.200+0000 7f555dffb640 0 client.26080 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.201+0000 7f5583db8740 1 -- 172.21.15.31:0/646802855 shutdown_connections 2024-04-25T16:05:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.201+0000 7f5583db8740 1 -- 172.21.15.31:0/646802855 wait complete. 2024-04-25T16:05:36.203 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.201+0000 7f5583db8740 20 client.26080 trim_cache size 0 max 16384 2024-04-25T16:05:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:05:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:05:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:05:36.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.202+0000 7f557759e640 1 -- 172.21.15.31:0/2071748964 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55600afcf0 msgr2=0x7f55600b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.202+0000 7f557759e640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55600afcf0 0x7f55600b2200 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x557e4f2e5240 tx=0x7f55700138e0 comp rx=0 tx=0).stop 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.202+0000 7f557759e640 1 -- 172.21.15.31:0/2071748964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5578110cb0 msgr2=0x7f55781b0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.202+0000 7f557759e640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5578110cb0 0x7f55781b0190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f556c00d750 tx=0x7f556c00dc10 comp rx=0 tx=0).stop 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.203+0000 7f557759e640 1 -- 172.21.15.31:0/2071748964 shutdown_connections 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.203+0000 7f557759e640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f55600afcf0 0x7f55600b2200 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.203+0000 7f557759e640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5578110cb0 0x7f55781b0190 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.203+0000 7f557759e640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f557810e2c0 0x7f55781a5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.203+0000 7f557759e640 1 --2- 172.21.15.31:0/2071748964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f557810b960 0x7f557810af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:36.205 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.203+0000 7f557759e640 1 -- 172.21.15.31:0/2071748964 >> 172.21.15.31:0/2071748964 conn(0x7f5578070330 msgr2=0x7f557810d900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.203+0000 7f557759e640 1 -- 172.21.15.31:0/2071748964 shutdown_connections 2024-04-25T16:05:36.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:36.203+0000 7f557759e640 1 -- 172.21.15.31:0/2071748964 wait complete. 2024-04-25T16:05:37.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:36 smithi063 ceph-mon[32413]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 763 MiB/s wr, 1.86k op/s 2024-04-25T16:05:37.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:36 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:05:37.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:36 smithi186 ceph-mon[33258]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 763 MiB/s wr, 1.86k op/s 2024-04-25T16:05:37.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:36 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:05:37.207 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:36 smithi031 ceph-mon[29140]: pgmap v480: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 763 MiB/s wr, 1.86k op/s 2024-04-25T16:05:37.207 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:36 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:05:37.492 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:05:37.492 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:05:37.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.911+0000 7f7474cb8640 1 -- 172.21.15.31:0/3992977885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747010a3e0 msgr2=0x7f747010ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:37.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.911+0000 7f7474cb8640 1 --2- 172.21.15.31:0/3992977885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747010a3e0 0x7f747010ac90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f746401a910 tx=0x7f746403b470 comp rx=0 tx=0).stop 2024-04-25T16:05:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.912+0000 7f7474cb8640 1 -- 172.21.15.31:0/3992977885 shutdown_connections 2024-04-25T16:05:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.912+0000 7f7474cb8640 1 --2- 172.21.15.31:0/3992977885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f747010fef0 0x7f7470110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.912+0000 7f7474cb8640 1 --2- 172.21.15.31:0/3992977885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747010b1d0 0x7f747010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.912+0000 7f7474cb8640 1 --2- 172.21.15.31:0/3992977885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747010a3e0 0x7f747010ac90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.912+0000 7f7474cb8640 1 -- 172.21.15.31:0/3992977885 >> 172.21.15.31:0/3992977885 conn(0x7f7470070400 msgr2=0x7f7470072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.913+0000 7f7474cb8640 1 -- 172.21.15.31:0/3992977885 shutdown_connections 2024-04-25T16:05:37.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.913+0000 7f7474cb8640 1 -- 172.21.15.31:0/3992977885 wait complete. 2024-04-25T16:05:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.914+0000 7f7474cb8640 1 Processor -- start 2024-04-25T16:05:37.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.914+0000 7f7474cb8640 1 -- start start 2024-04-25T16:05:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.915+0000 7f7474cb8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747010b1d0 0x7f74701a0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.915+0000 7f7474cb8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747010fef0 0x7f74701a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.915+0000 7f7474cb8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74701a5800 0x7f74701a7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.915+0000 7f7474cb8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7470113ac0 con 0x7f747010b1d0 2024-04-25T16:05:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.915+0000 7f7474cb8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7470113940 con 0x7f74701a5800 2024-04-25T16:05:37.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.915+0000 7f7474cb8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7470113c40 con 0x7f747010fef0 2024-04-25T16:05:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.916+0000 7f746e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747010b1d0 0x7f74701a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.916+0000 7f746e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747010b1d0 0x7f74701a0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38456/0 (socket says 172.21.15.31:38456) 2024-04-25T16:05:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.916+0000 7f746e59c640 1 -- 172.21.15.31:0/527858934 learned_addr learned my addr 172.21.15.31:0/527858934 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:37.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.917+0000 7f746dd9b640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747010fef0 0x7f74701a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.917+0000 7f746e59c640 1 -- 172.21.15.31:0/527858934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747010fef0 msgr2=0x7f74701a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:37.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.917+0000 7f746e59c640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747010fef0 0x7f74701a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.917+0000 7f746e59c640 1 -- 172.21.15.31:0/527858934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74701a5800 msgr2=0x7f74701a7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:37.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.917+0000 7f746e59c640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74701a5800 0x7f74701a7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.917+0000 7f746e59c640 1 -- 172.21.15.31:0/527858934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74701a83f0 con 0x7f747010b1d0 2024-04-25T16:05:37.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.918+0000 7f746e59c640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747010b1d0 0x7f74701a0340 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7464044040 tx=0x7f74640051c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.918+0000 7f745f7fe640 1 -- 172.21.15.31:0/527858934 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464050070 con 0x7f747010b1d0 2024-04-25T16:05:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.918+0000 7f745f7fe640 1 -- 172.21.15.31:0/527858934 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74640055c0 con 0x7f747010b1d0 2024-04-25T16:05:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.918+0000 7f745f7fe640 1 -- 172.21.15.31:0/527858934 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7464040890 con 0x7f747010b1d0 2024-04-25T16:05:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.918+0000 7f7474cb8640 1 -- 172.21.15.31:0/527858934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74701a8680 con 0x7f747010b1d0 2024-04-25T16:05:37.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.919+0000 7f7474cb8640 1 -- 172.21.15.31:0/527858934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74701a8b80 con 0x7f747010b1d0 2024-04-25T16:05:37.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.927+0000 7f747bb68740 1 Processor -- start 2024-04-25T16:05:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.927+0000 7f747bb68740 1 -- start start 2024-04-25T16:05:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.927+0000 7f747bb68740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054fa390 0x556f0589bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.928+0000 7f747bb68740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 0x556f0589c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.928+0000 7f747bb68740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f0589ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.928+0000 7f746ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054fa390 0x556f0589bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.929+0000 7f746ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054fa390 0x556f0589bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52314/0 (socket says 172.21.15.31:52314) 2024-04-25T16:05:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.929+0000 7f746ed9d640 1 -- 172.21.15.31:0/644768615 learned_addr learned my addr 172.21.15.31:0/644768615 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.929+0000 7f747bb68740 1 -- 172.21.15.31:0/644768615 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556f05800d80 con 0x556f054d14d0 2024-04-25T16:05:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.929+0000 7f747bb68740 1 -- 172.21.15.31:0/644768615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556f057ba8b0 con 0x556f054fa390 2024-04-25T16:05:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.929+0000 7f747bb68740 1 -- 172.21.15.31:0/644768615 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556f057fcd40 con 0x556f054d96b0 2024-04-25T16:05:37.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.929+0000 7f746dd9b640 1 --2- 172.21.15.31:0/644768615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f0589ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.933+0000 7f746e59c640 1 --2- 172.21.15.31:0/644768615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 0x556f0589c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.935+0000 7f745f7fe640 1 -- 172.21.15.31:0/527858934 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7464053070 con 0x7f747010b1d0 2024-04-25T16:05:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.937+0000 7f745f7fe640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74580afc70 0x7f74580b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.938+0000 7f746ed9d640 1 -- 172.21.15.31:0/644768615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 msgr2=0x556f0589ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.938+0000 7f746ed9d640 1 --2- 172.21.15.31:0/644768615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f0589ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.938+0000 7f746ed9d640 1 -- 172.21.15.31:0/644768615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 msgr2=0x556f0589c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.938+0000 7f746ed9d640 1 --2- 172.21.15.31:0/644768615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 0x556f0589c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.938+0000 7f746ed9d640 1 -- 172.21.15.31:0/644768615 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f0578ba40 con 0x556f054fa390 2024-04-25T16:05:37.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.939+0000 7f746ed9d640 1 --2- 172.21.15.31:0/644768615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054fa390 0x556f0589bc50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7460009c40 tx=0x7f746000e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bf053936e3efe39 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:37.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.943+0000 7f745f7fe640 1 -- 172.21.15.31:0/527858934 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f74640da5c0 con 0x7f747010b1d0 2024-04-25T16:05:37.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.943+0000 7f745cff9640 1 -- 172.21.15.31:0/644768615 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460010430 con 0x556f054fa390 2024-04-25T16:05:37.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.943+0000 7f745cff9640 1 -- 172.21.15.31:0/644768615 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7460010a80 con 0x556f054fa390 2024-04-25T16:05:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.945+0000 7f747bb68740 1 -- 172.21.15.31:0/644768615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054fa390 msgr2=0x556f0589bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.945+0000 7f747bb68740 1 --2- 172.21.15.31:0/644768615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054fa390 0x556f0589bc50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7460009c40 tx=0x7f746000e720 comp rx=0 tx=0).stop 2024-04-25T16:05:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.946+0000 7f747bb68740 1 -- 172.21.15.31:0/644768615 shutdown_connections 2024-04-25T16:05:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.946+0000 7f747bb68740 1 --2- 172.21.15.31:0/644768615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f0589ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.946+0000 7f747bb68740 1 --2- 172.21.15.31:0/644768615 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 0x556f0589c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.946+0000 7f747bb68740 1 --2- 172.21.15.31:0/644768615 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054fa390 0x556f0589bc50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.946+0000 7f747bb68740 1 -- 172.21.15.31:0/644768615 >> 172.21.15.31:0/644768615 conn(0x556f054ee4e0 msgr2=0x556f05897460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:37.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.946+0000 7f747bb68740 1 -- 172.21.15.31:0/644768615 shutdown_connections 2024-04-25T16:05:37.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.946+0000 7f747bb68740 1 -- 172.21.15.31:0/644768615 wait complete. 2024-04-25T16:05:37.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.947+0000 7f747bb68740 1 -- start start 2024-04-25T16:05:37.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.947+0000 7f747bb68740 1 Processor -- start 2024-04-25T16:05:37.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.947+0000 7f747bb68740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 0x556f058a59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.947+0000 7f747bb68740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f058a5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.947+0000 7f747bb68740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 0x556f058ace60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.947+0000 7f747bb68740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556f0589d9c0 con 0x556f054d14d0 2024-04-25T16:05:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.947+0000 7f747bb68740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556f05787d10 con 0x556f054b8f60 2024-04-25T16:05:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.947+0000 7f747bb68740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556f0589db40 con 0x556f054d96b0 2024-04-25T16:05:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.947+0000 7f747bb68740 10 client.0 _ll_register_callbacks cb 0x556f057b4210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.948+0000 7f746dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f058a5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.948+0000 7f746dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f058a5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56256/0 (socket says 172.21.15.31:56256) 2024-04-25T16:05:37.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.948+0000 7f746dd9b640 1 -- 172.21.15.31:0/3220894392 learned_addr learned my addr 172.21.15.31:0/3220894392 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:37.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.951+0000 7f746ed9d640 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 0x556f058ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.951+0000 7f746e59c640 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 0x556f058a59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.951+0000 7f746dd9b640 1 -- 172.21.15.31:0/3220894392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 msgr2=0x556f058ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:37.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.951+0000 7f746dd9b640 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 0x556f058ace60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.951+0000 7f746dd9b640 1 -- 172.21.15.31:0/3220894392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 msgr2=0x556f058a59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:37.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.951+0000 7f746dd9b640 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 0x556f058a59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.951+0000 7f746dd9b640 1 -- 172.21.15.31:0/3220894392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f055dc7d0 con 0x556f054d96b0 2024-04-25T16:05:37.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.952+0000 7f746ed9d640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74580afc70 0x7f74580b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.952+0000 7f746dd9b640 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f058a5f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f746800f740 tx=0x7f746800fb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe0357e7a2f6f7c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.960+0000 7f7455ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.961+0000 7f7455ffb640 1 -- 172.21.15.31:0/3220894392 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7468017070 con 0x556f054d96b0 2024-04-25T16:05:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.961+0000 7f747bb68740 1 -- 172.21.15.31:0/3220894392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556f0580e990 con 0x556f054d96b0 2024-04-25T16:05:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.961+0000 7f7455ffb640 1 -- 172.21.15.31:0/3220894392 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f746800d9c0 con 0x556f054d96b0 2024-04-25T16:05:37.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.961+0000 7f7455ffb640 1 -- 172.21.15.31:0/3220894392 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7468023050 con 0x556f054d96b0 2024-04-25T16:05:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.962+0000 7f747bb68740 10 client.26089 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.962+0000 7f747bb68740 1 -- 172.21.15.31:0/3220894392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f055123f0 con 0x556f054d96b0 2024-04-25T16:05:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.962+0000 7f7455ffb640 1 -- 172.21.15.31:0/3220894392 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x556f0580e990 con 0x556f054d96b0 2024-04-25T16:05:37.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.962+0000 7f7455ffb640 1 -- 172.21.15.31:0/3220894392 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f74680123e0 con 0x556f054d96b0 2024-04-25T16:05:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.963+0000 7f747bb68740 10 client.26089 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.963+0000 7f747bb68740 10 client.26089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.963+0000 7f747bb68740 20 client.26089 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.963+0000 7f747bb68740 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556f0548e3a0 0x556f058b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.963+0000 7f747bb68740 4 client.26089 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:37.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.963+0000 7f747bb68740 1 -- 172.21.15.31:0/3220894392 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f05808800 con 0x556f0548e3a0 2024-04-25T16:05:37.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.965+0000 7f746e59c640 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556f0548e3a0 0x556f058b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:37.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.968+0000 7f746ed9d640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74580afc70 0x7f74580b2180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7460009c10 tx=0x7f7460009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:37.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.972+0000 7f746e59c640 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556f0548e3a0 0x556f058b38d0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c92938c3bdf9d1d7 server_cookie=f42e85aee3defe31 in_seq=0 out_seq=0 2024-04-25T16:05:37.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.972+0000 7f7455ffb640 10 client.26089 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:37.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.977+0000 7f7455ffb640 1 -- 172.21.15.31:0/3220894392 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556f05808800 con 0x556f0548e3a0 2024-04-25T16:05:37.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.977+0000 7f7455ffb640 10 client.26089 handle_command_reply: tid=0 2024-04-25T16:05:37.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.977+0000 7f747bb68740 1 client.26089 shutdown 2024-04-25T16:05:37.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.977+0000 7f747bb68740 1 -- 172.21.15.31:0/3220894392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 msgr2=0x556f058a5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:37.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.977+0000 7f747bb68740 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f058a5f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f746800f740 tx=0x7f746800fb40 comp rx=0 tx=0).stop 2024-04-25T16:05:37.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f747bb68740 1 -- 172.21.15.31:0/3220894392 shutdown_connections 2024-04-25T16:05:37.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f747bb68740 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556f0548e3a0 0x556f058b38d0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f747bb68740 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 0x556f058ace60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f747bb68740 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d96b0 0x556f058a5f00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f747bb68740 1 --2- 172.21.15.31:0/3220894392 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d14d0 0x556f058a59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:37.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f7455ffb640 0 client.26089 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:37.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f747bb68740 1 -- 172.21.15.31:0/3220894392 >> 172.21.15.31:0/3220894392 conn(0x556f054ee4e0 msgr2=0x556f05896420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:37.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f747bb68740 1 -- 172.21.15.31:0/3220894392 shutdown_connections 2024-04-25T16:05:37.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f747bb68740 1 -- 172.21.15.31:0/3220894392 wait complete. 2024-04-25T16:05:37.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:37.978+0000 7f747bb68740 20 client.26089 trim_cache size 0 max 16384 2024-04-25T16:05:38.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.026+0000 7f747bb68740 1 Processor -- start 2024-04-25T16:05:38.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.026+0000 7f747bb68740 1 -- start start 2024-04-25T16:05:38.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.027+0000 7f747bb68740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 0x556f058a4220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.027+0000 7f747bb68740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d14d0 0x556f058ace20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.027+0000 7f747bb68740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d96b0 0x556f058ad360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.030+0000 7f746e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 0x556f058a4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.030+0000 7f746ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d96b0 0x556f058ad360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.030+0000 7f747bb68740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f745000e590 con 0x556f054d96b0 2024-04-25T16:05:38.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.030+0000 7f747bb68740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f74500100d0 con 0x556f054b8f60 2024-04-25T16:05:38.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.030+0000 7f747bb68740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f745000ec60 con 0x556f054d14d0 2024-04-25T16:05:38.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.030+0000 7f746dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d14d0 0x556f058ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.030+0000 7f746ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d96b0 0x556f058ad360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38480/0 (socket says 172.21.15.31:38480) 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.030+0000 7f746ed9d640 1 -- 172.21.15.31:0/2673318945 learned_addr learned my addr 172.21.15.31:0/2673318945 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.031+0000 7f746ed9d640 1 -- 172.21.15.31:0/2673318945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d14d0 msgr2=0x556f058ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.031+0000 7f746ed9d640 1 --2- 172.21.15.31:0/2673318945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d14d0 0x556f058ace20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.031+0000 7f746ed9d640 1 -- 172.21.15.31:0/2673318945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 msgr2=0x556f058a4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.031+0000 7f746ed9d640 1 --2- 172.21.15.31:0/2673318945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 0x556f058a4220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.031+0000 7f746ed9d640 1 -- 172.21.15.31:0/2673318945 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f0589c2f0 con 0x556f054d96b0 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.032+0000 7f746ed9d640 1 --2- 172.21.15.31:0/2673318945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d96b0 0x556f058ad360 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x556f0589cca0 tx=0x7f7460025000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=131df4b45ae16ef0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.033+0000 7f7457fff640 1 -- 172.21.15.31:0/2673318945 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74600035f0 con 0x556f054d96b0 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.033+0000 7f7457fff640 1 -- 172.21.15.31:0/2673318945 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7460034070 con 0x556f054d96b0 2024-04-25T16:05:38.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.033+0000 7f7457fff640 1 -- 172.21.15.31:0/2673318945 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746002f8a0 con 0x556f054d96b0 2024-04-25T16:05:38.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.034+0000 7f747bb68740 1 -- 172.21.15.31:0/2673318945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d96b0 msgr2=0x556f058ad360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:38.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.034+0000 7f747bb68740 1 --2- 172.21.15.31:0/2673318945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d96b0 0x556f058ad360 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x556f0589cca0 tx=0x7f7460025000 comp rx=0 tx=0).stop 2024-04-25T16:05:38.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.034+0000 7f747bb68740 1 -- 172.21.15.31:0/2673318945 shutdown_connections 2024-04-25T16:05:38.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.034+0000 7f747bb68740 1 --2- 172.21.15.31:0/2673318945 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f054d96b0 0x556f058ad360 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.034+0000 7f747bb68740 1 --2- 172.21.15.31:0/2673318945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054d14d0 0x556f058ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.034+0000 7f747bb68740 1 --2- 172.21.15.31:0/2673318945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054b8f60 0x556f058a4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.034+0000 7f747bb68740 1 -- 172.21.15.31:0/2673318945 >> 172.21.15.31:0/2673318945 conn(0x556f054ee4e0 msgr2=0x556f058a05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:38.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.036+0000 7f747bb68740 1 -- 172.21.15.31:0/2673318945 shutdown_connections 2024-04-25T16:05:38.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.039+0000 7f747bb68740 1 -- 172.21.15.31:0/2673318945 wait complete. 2024-04-25T16:05:38.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.039+0000 7f747bb68740 1 -- start start 2024-04-25T16:05:38.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.040+0000 7f747bb68740 1 Processor -- start 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.040+0000 7f747bb68740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054b8f60 0x556f058aa290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.040+0000 7f747bb68740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054d96b0 0x556f058aa7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.040+0000 7f747bb68740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f05917c60 0x556f0591a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.040+0000 7f747bb68740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f745000fa00 con 0x556f05917c60 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.040+0000 7f747bb68740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f745000f330 con 0x556f054d96b0 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.040+0000 7f747bb68740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f745000c570 con 0x556f054b8f60 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.040+0000 7f747bb68740 10 client.0 _ll_register_callbacks cb 0x7f7468007a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.041+0000 7f746ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f05917c60 0x556f0591a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.041+0000 7f746ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f05917c60 0x556f0591a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38484/0 (socket says 172.21.15.31:38484) 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.041+0000 7f746ed9d640 1 -- 172.21.15.31:0/548898258 learned_addr learned my addr 172.21.15.31:0/548898258 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:38.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.041+0000 7f746e59c640 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054b8f60 0x556f058aa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.042+0000 7f746dd9b640 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054d96b0 0x556f058aa7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.043+0000 7f746dd9b640 1 -- 172.21.15.31:0/548898258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054b8f60 msgr2=0x556f058aa290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:38.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.043+0000 7f746dd9b640 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054b8f60 0x556f058aa290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.043+0000 7f746dd9b640 1 -- 172.21.15.31:0/548898258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f05917c60 msgr2=0x556f0591a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:38.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.043+0000 7f746dd9b640 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f05917c60 0x556f0591a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.043+0000 7f746dd9b640 1 -- 172.21.15.31:0/548898258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f054d20f0 con 0x556f054d96b0 2024-04-25T16:05:38.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.044+0000 7f746dd9b640 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054d96b0 0x556f058aa7d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x556f05892c30 tx=0x7f7468012fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cac461851ac21d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:38.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.058+0000 7f7455ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:05:38.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.058+0000 7f7455ffb640 1 -- 172.21.15.31:0/548898258 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f746802b060 con 0x556f054d96b0 2024-04-25T16:05:38.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.058+0000 7f747bb68740 1 -- 172.21.15.31:0/548898258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7460034070 con 0x556f054d96b0 2024-04-25T16:05:38.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.058+0000 7f7455ffb640 1 -- 172.21.15.31:0/548898258 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7468003a30 con 0x556f054d96b0 2024-04-25T16:05:38.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.059+0000 7f7455ffb640 1 -- 172.21.15.31:0/548898258 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7468017070 con 0x556f054d96b0 2024-04-25T16:05:38.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.059+0000 7f747bb68740 10 client.26156 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:38.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.059+0000 7f747bb68740 1 -- 172.21.15.31:0/548898258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f057c14b0 con 0x556f054d96b0 2024-04-25T16:05:38.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.060+0000 7f7455ffb640 1 -- 172.21.15.31:0/548898258 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7460034070 con 0x556f054d96b0 2024-04-25T16:05:38.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.060+0000 7f7455ffb640 1 -- 172.21.15.31:0/548898258 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7468004040 con 0x556f054d96b0 2024-04-25T16:05:38.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.060+0000 7f747bb68740 10 client.26156 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:38.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.060+0000 7f747bb68740 10 client.26156 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:38.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.060+0000 7f747bb68740 20 client.26156 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:38.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.060+0000 7f747bb68740 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556f0591bce0 0x556f0591e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.060+0000 7f747bb68740 4 client.26156 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:05:38.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.060+0000 7f747bb68740 1 -- 172.21.15.31:0/548898258 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556f057834a0 con 0x556f0591bce0 2024-04-25T16:05:38.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.061+0000 7f746e59c640 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556f0591bce0 0x556f0591e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.062+0000 7f746e59c640 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556f0591bce0 0x556f0591e100 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75880ec318e3ce2b server_cookie=79cd64854fd12446 in_seq=0 out_seq=0 2024-04-25T16:05:38.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.062+0000 7f7455ffb640 10 client.26156 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f7455ffb640 1 -- 172.21.15.31:0/548898258 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x556f057834a0 con 0x556f0591bce0 2024-04-25T16:05:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f7455ffb640 10 client.26156 handle_command_reply: tid=0 2024-04-25T16:05:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f747bb68740 1 client.26156 shutdown 2024-04-25T16:05:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f747bb68740 1 -- 172.21.15.31:0/548898258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054d96b0 msgr2=0x556f058aa7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f747bb68740 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054d96b0 0x556f058aa7d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x556f05892c30 tx=0x7f7468012fa0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f747bb68740 1 -- 172.21.15.31:0/548898258 shutdown_connections 2024-04-25T16:05:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f747bb68740 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556f0591bce0 0x556f0591e100 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f747bb68740 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f05917c60 0x556f0591a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f747bb68740 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f054d96b0 0x556f058aa7d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f747bb68740 1 --2- 172.21.15.31:0/548898258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f054b8f60 0x556f058aa290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f747bb68740 1 -- 172.21.15.31:0/548898258 >> 172.21.15.31:0/548898258 conn(0x556f054ee4e0 msgr2=0x556f05896d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:38.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.072+0000 7f7455ffb640 0 client.26156 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:38.081 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:05:38.081 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:05:38.081 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:05:38.081 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:05:38.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.076+0000 7f747bb68740 1 -- 172.21.15.31:0/548898258 shutdown_connections 2024-04-25T16:05:38.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.076+0000 7f747bb68740 1 -- 172.21.15.31:0/548898258 wait complete. 2024-04-25T16:05:38.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.077+0000 7f747bb68740 20 client.26156 trim_cache size 0 max 16384 2024-04-25T16:05:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.080+0000 7f7474cb8640 1 -- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74580afc70 msgr2=0x7f74580b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.080+0000 7f7474cb8640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74580afc70 0x7f74580b2180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f7460009c10 tx=0x7f7460009720 comp rx=0 tx=0).stop 2024-04-25T16:05:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.080+0000 7f7474cb8640 1 -- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747010b1d0 msgr2=0x7f74701a0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:38.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.080+0000 7f7474cb8640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747010b1d0 0x7f74701a0340 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f7464044040 tx=0x7f74640051c0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.083+0000 7f7474cb8640 1 -- 172.21.15.31:0/527858934 shutdown_connections 2024-04-25T16:05:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.083+0000 7f7474cb8640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74580afc70 0x7f74580b2180 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.083+0000 7f7474cb8640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74701a5800 0x7f74701a7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.083+0000 7f7474cb8640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747010fef0 0x7f74701a0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.083+0000 7f7474cb8640 1 --2- 172.21.15.31:0/527858934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f747010b1d0 0x7f74701a0340 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.084+0000 7f7474cb8640 1 -- 172.21.15.31:0/527858934 >> 172.21.15.31:0/527858934 conn(0x7f7470070400 msgr2=0x7f7470072720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:38.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.088+0000 7f7474cb8640 1 -- 172.21.15.31:0/527858934 shutdown_connections 2024-04-25T16:05:38.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.088+0000 7f7474cb8640 1 -- 172.21.15.31:0/527858934 wait complete. 2024-04-25T16:05:38.124 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:05:38.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:05:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.633+0000 7f90e73b2640 1 -- 172.21.15.31:0/2818187639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90e010c010 msgr2=0x7f90e010c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.633+0000 7f90e73b2640 1 --2- 172.21.15.31:0/2818187639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90e010c010 0x7f90e010c420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f90d0009900 tx=0x7f90d002a400 comp rx=0 tx=0).stop 2024-04-25T16:05:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.634+0000 7f90e73b2640 1 -- 172.21.15.31:0/2818187639 shutdown_connections 2024-04-25T16:05:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.634+0000 7f90e73b2640 1 --2- 172.21.15.31:0/2818187639 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90e010c010 0x7f90e010c420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.634+0000 7f90e73b2640 1 --2- 172.21.15.31:0/2818187639 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e010b310 0x7f90e010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.634+0000 7f90e73b2640 1 --2- 172.21.15.31:0/2818187639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e010a420 0x7f90e010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.634+0000 7f90e73b2640 1 -- 172.21.15.31:0/2818187639 >> 172.21.15.31:0/2818187639 conn(0x7f90e00704e0 msgr2=0x7f90e0072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:38.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.634+0000 7f90e73b2640 1 -- 172.21.15.31:0/2818187639 shutdown_connections 2024-04-25T16:05:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.634+0000 7f90e73b2640 1 -- 172.21.15.31:0/2818187639 wait complete. 2024-04-25T16:05:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.636+0000 7f90e73b2640 1 Processor -- start 2024-04-25T16:05:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.636+0000 7f90e73b2640 1 -- start start 2024-04-25T16:05:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.636+0000 7f90e73b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e010a420 0x7f90e01103d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.636+0000 7f90e73b2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90e010b310 0x7f90e0110910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.636+0000 7f90e73b2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e010c010 0x7f90e01b8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.636+0000 7f90e73b2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f90e011a0e0 con 0x7f90e010a420 2024-04-25T16:05:38.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.636+0000 7f90e73b2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f90e0119f60 con 0x7f90e010b310 2024-04-25T16:05:38.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.636+0000 7f90e73b2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f90e011a260 con 0x7f90e010c010 2024-04-25T16:05:38.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.638+0000 7f90e594f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e010c010 0x7f90e01b8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.638+0000 7f90e514e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e010a420 0x7f90e01103d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.638+0000 7f90e514e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e010a420 0x7f90e01103d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38504/0 (socket says 172.21.15.31:38504) 2024-04-25T16:05:38.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.638+0000 7f90e514e640 1 -- 172.21.15.31:0/4139945633 learned_addr learned my addr 172.21.15.31:0/4139945633 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:38.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.638+0000 7f90e494d640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90e010b310 0x7f90e0110910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.640+0000 7f90e594f640 1 -- 172.21.15.31:0/4139945633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90e010b310 msgr2=0x7f90e0110910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.640+0000 7f90e594f640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90e010b310 0x7f90e0110910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.640+0000 7f90e594f640 1 -- 172.21.15.31:0/4139945633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e010a420 msgr2=0x7f90e01103d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.640+0000 7f90e594f640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e010a420 0x7f90e01103d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:38.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.640+0000 7f90e594f640 1 -- 172.21.15.31:0/4139945633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90e01b9520 con 0x7f90e010c010 2024-04-25T16:05:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.641+0000 7f90e594f640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e010c010 0x7f90e01b8fe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f90d002a920 tx=0x7f90d0003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.641+0000 7f90d67fc640 1 -- 172.21.15.31:0/4139945633 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d003f070 con 0x7f90e010c010 2024-04-25T16:05:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.641+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90e01b96f0 con 0x7f90e010c010 2024-04-25T16:05:38.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.642+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90e01b9b50 con 0x7f90e010c010 2024-04-25T16:05:38.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.644+0000 7f90d67fc640 1 -- 172.21.15.31:0/4139945633 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f90d0005d50 con 0x7f90e010c010 2024-04-25T16:05:38.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.644+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90e01a4600 con 0x7f90e010c010 2024-04-25T16:05:38.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.644+0000 7f90d67fc640 1 -- 172.21.15.31:0/4139945633 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90d0035030 con 0x7f90e010c010 2024-04-25T16:05:38.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.668+0000 7f90d67fc640 1 -- 172.21.15.31:0/4139945633 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f90d0042070 con 0x7f90e010c010 2024-04-25T16:05:38.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.669+0000 7f90d67fc640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f90cc0afbf0 0x7f90cc0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:38.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.670+0000 7f90d67fc640 1 -- 172.21.15.31:0/4139945633 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f90d00ff5f0 con 0x7f90e010c010 2024-04-25T16:05:38.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.670+0000 7f90d67fc640 1 -- 172.21.15.31:0/4139945633 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90d00ffab0 con 0x7f90e010c010 2024-04-25T16:05:38.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.671+0000 7f90e514e640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f90cc0afbf0 0x7f90cc0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:38.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:38.672+0000 7f90e514e640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f90cc0afbf0 0x7f90cc0b2100 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f90d8005f40 tx=0x7f90d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:39.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:38 smithi063 ceph-mon[32413]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 887 MiB/s wr, 2.31k op/s 2024-04-25T16:05:39.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:38 smithi186 ceph-mon[33258]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 887 MiB/s wr, 2.31k op/s 2024-04-25T16:05:39.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:38 smithi031 ceph-mon[29140]: pgmap v481: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 887 MiB/s wr, 2.31k op/s 2024-04-25T16:05:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.667+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f90e002db50 con 0x7f90e010c010 2024-04-25T16:05:39.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.674+0000 7f90d67fc640 1 -- 172.21.15.31:0/4139945633 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7f90d00d0070 con 0x7f90e010c010 2024-04-25T16:05:39.685 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:05:39.685 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:05:39.685 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:05:39.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.688+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f90cc0afbf0 msgr2=0x7f90cc0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:39.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.688+0000 7f90e73b2640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f90cc0afbf0 0x7f90cc0b2100 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f90d8005f40 tx=0x7f90d8009500 comp rx=0 tx=0).stop 2024-04-25T16:05:39.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.688+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e010c010 msgr2=0x7f90e01b8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:39.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.688+0000 7f90e73b2640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e010c010 0x7f90e01b8fe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f90d002a920 tx=0x7f90d0003b90 comp rx=0 tx=0).stop 2024-04-25T16:05:39.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.689+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 shutdown_connections 2024-04-25T16:05:39.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.689+0000 7f90e73b2640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f90cc0afbf0 0x7f90cc0b2100 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:39.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.689+0000 7f90e73b2640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f90e010c010 0x7f90e01b8fe0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:39.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.689+0000 7f90e73b2640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f90e010b310 0x7f90e0110910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:39.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.689+0000 7f90e73b2640 1 --2- 172.21.15.31:0/4139945633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f90e010a420 0x7f90e01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:39.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.689+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 >> 172.21.15.31:0/4139945633 conn(0x7f90e00704e0 msgr2=0x7f90e01a17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:39.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.695+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 shutdown_connections 2024-04-25T16:05:39.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:39.695+0000 7f90e73b2640 1 -- 172.21.15.31:0/4139945633 wait complete. 2024-04-25T16:05:39.774 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:05:40.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:39 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4139945633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:05:40.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:39 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4139945633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:05:40.206 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:39 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4139945633' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:05:40.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.248+0000 7f2b79ffb640 1 -- 172.21.15.31:0/274787042 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b700035f0 con 0x7f2b7c10fef0 2024-04-25T16:05:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.250+0000 7f2b8144c640 1 -- 172.21.15.31:0/274787042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b7c10fef0 msgr2=0x7f2b7c110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.250+0000 7f2b8144c640 1 --2- 172.21.15.31:0/274787042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b7c10fef0 0x7f2b7c110300 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2b70009900 tx=0x7f2b7002a440 comp rx=0 tx=0).stop 2024-04-25T16:05:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.250+0000 7f2b8144c640 1 -- 172.21.15.31:0/274787042 shutdown_connections 2024-04-25T16:05:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.250+0000 7f2b8144c640 1 --2- 172.21.15.31:0/274787042 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b7c10fef0 0x7f2b7c110300 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.250+0000 7f2b8144c640 1 --2- 172.21.15.31:0/274787042 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b7c10b1d0 0x7f2b7c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.250+0000 7f2b8144c640 1 --2- 172.21.15.31:0/274787042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b7c10a3e0 0x7f2b7c10ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.250+0000 7f2b8144c640 1 -- 172.21.15.31:0/274787042 >> 172.21.15.31:0/274787042 conn(0x7f2b7c070400 msgr2=0x7f2b7c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.251+0000 7f2b8144c640 1 -- 172.21.15.31:0/274787042 shutdown_connections 2024-04-25T16:05:40.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.251+0000 7f2b8144c640 1 -- 172.21.15.31:0/274787042 wait complete. 2024-04-25T16:05:40.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.252+0000 7f2b8144c640 1 Processor -- start 2024-04-25T16:05:40.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.252+0000 7f2b8144c640 1 -- start start 2024-04-25T16:05:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.253+0000 7f2b8144c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b7c10a3e0 0x7f2b7c1a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.253+0000 7f2b8144c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b7c10b1d0 0x7f2b7c1a9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.253+0000 7f2b8144c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b7c10fef0 0x7f2b7c1b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.253+0000 7f2b8144c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c11e320 con 0x7f2b7c10b1d0 2024-04-25T16:05:40.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.253+0000 7f2b8144c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c11e1a0 con 0x7f2b7c10fef0 2024-04-25T16:05:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.253+0000 7f2b8144c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b7c11e4a0 con 0x7f2b7c10a3e0 2024-04-25T16:05:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.253+0000 7f2b7a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b7c10b1d0 0x7f2b7c1a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.253+0000 7f2b7a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b7c10b1d0 0x7f2b7c1a9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38514/0 (socket says 172.21.15.31:38514) 2024-04-25T16:05:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.253+0000 7f2b7a7fc640 1 -- 172.21.15.31:0/249522306 learned_addr learned my addr 172.21.15.31:0/249522306 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.254+0000 7f2b7a7fc640 1 -- 172.21.15.31:0/249522306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b7c10a3e0 msgr2=0x7f2b7c1a8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.254+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b7c10a3e0 0x7f2b7c1a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.254+0000 7f2b7a7fc640 1 -- 172.21.15.31:0/249522306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b7c10fef0 msgr2=0x7f2b7c1b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.254+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b7c10fef0 0x7f2b7c1b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.254+0000 7f2b7a7fc640 1 -- 172.21.15.31:0/249522306 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b7c1b0980 con 0x7f2b7c10b1d0 2024-04-25T16:05:40.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.255+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b7c10b1d0 0x7f2b7c1a9380 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400ba50 tx=0x7f2b7400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:40.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.263+0000 7f2b63fff640 1 -- 172.21.15.31:0/249522306 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b74019070 con 0x7f2b7c10b1d0 2024-04-25T16:05:40.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.263+0000 7f2b8144c640 1 -- 172.21.15.31:0/249522306 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b7c1b0c70 con 0x7f2b7c10b1d0 2024-04-25T16:05:40.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.263+0000 7f2b8144c640 1 -- 172.21.15.31:0/249522306 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b7c1b1240 con 0x7f2b7c10b1d0 2024-04-25T16:05:40.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.268+0000 7f2b883db740 1 Processor -- start 2024-04-25T16:05:40.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.268+0000 7f2b883db740 1 -- start start 2024-04-25T16:05:40.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.268+0000 7f2b883db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 0x55c55c23a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.269+0000 7f2b883db740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c23af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.271+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 0x55c55c23a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.271+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 0x55c55c23a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52408/0 (socket says 172.21.15.31:52408) 2024-04-25T16:05:40.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.272+0000 7f2b7b7fe640 1 -- 172.21.15.31:0/3746884467 learned_addr learned my addr 172.21.15.31:0/3746884467 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:40.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.272+0000 7f2b883db740 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 0x55c55c23bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.272+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 0x55c55c23bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.272+0000 7f2b883db740 1 -- 172.21.15.31:0/3746884467 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c55c19ed80 con 0x55c55be776b0 2024-04-25T16:05:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.272+0000 7f2b7affd640 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c23af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.272+0000 7f2b883db740 1 -- 172.21.15.31:0/3746884467 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c55c158a20 con 0x55c55be98390 2024-04-25T16:05:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.272+0000 7f2b883db740 1 -- 172.21.15.31:0/3746884467 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c55c19ad40 con 0x55c55be6f4d0 2024-04-25T16:05:40.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.273+0000 7f2b63fff640 1 -- 172.21.15.31:0/249522306 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b7400cbe0 con 0x7f2b7c10b1d0 2024-04-25T16:05:40.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.275+0000 7f2b63fff640 1 -- 172.21.15.31:0/249522306 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b740075d0 con 0x7f2b7c10b1d0 2024-04-25T16:05:40.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.275+0000 7f2b7affd640 1 -- 172.21.15.31:0/3746884467 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 msgr2=0x55c55c23a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:40.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.275+0000 7f2b7affd640 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 0x55c55c23a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.275+0000 7f2b7affd640 1 -- 172.21.15.31:0/3746884467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 msgr2=0x55c55c23bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:40.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.275+0000 7f2b7affd640 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 0x55c55c23bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.275+0000 7f2b7affd640 1 -- 172.21.15.31:0/3746884467 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c55c163c30 con 0x55c55be6f4d0 2024-04-25T16:05:40.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.276+0000 7f2b63fff640 1 -- 172.21.15.31:0/249522306 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2b7401c070 con 0x7f2b7c10b1d0 2024-04-25T16:05:40.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.276+0000 7f2b7affd640 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c23af60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c074760 tx=0x7f2b6c00bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=374567fcfe474220 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:40.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.278+0000 7f2b63fff640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b640afc50 0x7f2b640b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.279+0000 7f2b63fff640 1 -- 172.21.15.31:0/249522306 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2b740df190 con 0x7f2b7c10b1d0 2024-04-25T16:05:40.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.282+0000 7f2b617fa640 1 -- 172.21.15.31:0/3746884467 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c004230 con 0x55c55be6f4d0 2024-04-25T16:05:40.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.282+0000 7f2b7b7fe640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b640afc50 0x7f2b640b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.282+0000 7f2b617fa640 1 -- 172.21.15.31:0/3746884467 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b6c0043d0 con 0x55c55be6f4d0 2024-04-25T16:05:40.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.284+0000 7f2b883db740 1 -- 172.21.15.31:0/3746884467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 msgr2=0x55c55c23af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.284+0000 7f2b883db740 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c23af60 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c074760 tx=0x7f2b6c00bea0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.287+0000 7f2b7b7fe640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b640afc50 0x7f2b640b2160 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2b700096c0 tx=0x7f2b70004b10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:40.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.287+0000 7f2b883db740 1 -- 172.21.15.31:0/3746884467 shutdown_connections 2024-04-25T16:05:40.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.287+0000 7f2b883db740 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 0x55c55c23bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.287+0000 7f2b883db740 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c23af60 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.287+0000 7f2b883db740 1 --2- 172.21.15.31:0/3746884467 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 0x55c55c23a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.287+0000 7f2b883db740 1 -- 172.21.15.31:0/3746884467 >> 172.21.15.31:0/3746884467 conn(0x55c55be8c4e0 msgr2=0x55c55c2361f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:40.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.288+0000 7f2b883db740 1 -- 172.21.15.31:0/3746884467 shutdown_connections 2024-04-25T16:05:40.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.288+0000 7f2b883db740 1 -- 172.21.15.31:0/3746884467 wait complete. 2024-04-25T16:05:40.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.289+0000 7f2b883db740 1 -- start start 2024-04-25T16:05:40.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.289+0000 7f2b883db740 1 Processor -- start 2024-04-25T16:05:40.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.290+0000 7f2b883db740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be6f4d0 0x55c55c2446f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.290+0000 7f2b883db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c244c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.290+0000 7f2b883db740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be98390 0x55c55c24bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.290+0000 7f2b883db740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c55c23c750 con 0x55c55be6f4d0 2024-04-25T16:05:40.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.290+0000 7f2b883db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c55c125dc0 con 0x55c55be776b0 2024-04-25T16:05:40.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.290+0000 7f2b883db740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c55c23c8d0 con 0x55c55be98390 2024-04-25T16:05:40.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.291+0000 7f2b883db740 10 client.0 _ll_register_callbacks cb 0x55c55c141600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:40.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.291+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be98390 0x55c55c24bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.291+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be98390 0x55c55c24bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56336/0 (socket says 172.21.15.31:56336) 2024-04-25T16:05:40.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.292+0000 7f2b7b7fe640 1 -- 172.21.15.31:0/1295881399 learned_addr learned my addr 172.21.15.31:0/1295881399 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:40.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.292+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c244c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.293+0000 7f2b7affd640 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be6f4d0 0x55c55c2446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.294+0000 7f2b7b7fe640 1 -- 172.21.15.31:0/1295881399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 msgr2=0x55c55c244c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:40.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.294+0000 7f2b7b7fe640 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c244c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.294+0000 7f2b7b7fe640 1 -- 172.21.15.31:0/1295881399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be6f4d0 msgr2=0x55c55c2446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:40.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.294+0000 7f2b7b7fe640 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be6f4d0 0x55c55c2446f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.294+0000 7f2b7b7fe640 1 -- 172.21.15.31:0/1295881399 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c55beb03f0 con 0x55c55be98390 2024-04-25T16:05:40.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.294+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c244c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:05:40.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.294+0000 7f2b7affd640 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be6f4d0 0x55c55c2446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:05:40.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.295+0000 7f2b7b7fe640 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be98390 0x55c55c24bb90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2b7002aa30 tx=0x7f2b70009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f0cb2dfb9361f51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:40.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.295+0000 7f2b5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:40.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.296+0000 7f2b5eff5640 1 -- 172.21.15.31:0/1295881399 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b7004a830 con 0x55c55be98390 2024-04-25T16:05:40.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.296+0000 7f2b5eff5640 1 -- 172.21.15.31:0/1295881399 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2b7004a9d0 con 0x55c55be98390 2024-04-25T16:05:40.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.297+0000 7f2b883db740 1 -- 172.21.15.31:0/1295881399 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b6c0043d0 con 0x55c55be98390 2024-04-25T16:05:40.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.300+0000 7f2b5eff5640 1 -- 172.21.15.31:0/1295881399 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2b7004abd0 con 0x55c55be98390 2024-04-25T16:05:40.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.300+0000 7f2b883db740 10 client.26107 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:40.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.300+0000 7f2b883db740 1 -- 172.21.15.31:0/1295881399 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c55bf7a7d0 con 0x55c55be98390 2024-04-25T16:05:40.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.307+0000 7f2b5eff5640 1 -- 172.21.15.31:0/1295881399 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b6c0043d0 con 0x55c55be98390 2024-04-25T16:05:40.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.307+0000 7f2b5eff5640 1 -- 172.21.15.31:0/1295881399 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f2b70031440 con 0x55c55be98390 2024-04-25T16:05:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.318+0000 7f2b883db740 10 client.26107 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.318+0000 7f2b883db740 10 client.26107 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:05:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.318+0000 7f2b883db740 20 client.26107 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.318+0000 7f2b883db740 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55c55be56f60 0x55c55c252430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.318+0000 7f2b883db740 4 client.26107 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:40.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.318+0000 7f2b883db740 1 -- 172.21.15.31:0/1295881399 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c55c1ac990 con 0x55c55be56f60 2024-04-25T16:05:40.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.320+0000 7f2b7affd640 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55c55be56f60 0x55c55c252430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.321+0000 7f2b5eff5640 10 client.26107 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:05:40.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.321+0000 7f2b7affd640 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55c55be56f60 0x55c55c252430 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7c092c270c407fbf server_cookie=ff4617f8288aef51 in_seq=0 out_seq=0 2024-04-25T16:05:40.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.323+0000 7f2b5eff5640 1 -- 172.21.15.31:0/1295881399 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c55c1ac990 con 0x55c55be56f60 2024-04-25T16:05:40.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.323+0000 7f2b5eff5640 10 client.26107 handle_command_reply: tid=0 2024-04-25T16:05:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.332+0000 7f2b883db740 1 client.26107 shutdown 2024-04-25T16:05:40.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.333+0000 7f2b883db740 1 -- 172.21.15.31:0/1295881399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be98390 msgr2=0x55c55c24bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.333+0000 7f2b883db740 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be98390 0x55c55c24bb90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2b7002aa30 tx=0x7f2b70009260 comp rx=0 tx=0).stop 2024-04-25T16:05:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.333+0000 7f2b883db740 1 -- 172.21.15.31:0/1295881399 shutdown_connections 2024-04-25T16:05:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.333+0000 7f2b883db740 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55c55be56f60 0x55c55c252430 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.333+0000 7f2b883db740 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be98390 0x55c55c24bb90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.333+0000 7f2b883db740 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c244c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.333+0000 7f2b883db740 1 --2- 172.21.15.31:0/1295881399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be6f4d0 0x55c55c2446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.333+0000 7f2b883db740 1 -- 172.21.15.31:0/1295881399 >> 172.21.15.31:0/1295881399 conn(0x55c55be8c4e0 msgr2=0x55c55c239100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:40.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.333+0000 7f2b5eff5640 0 client.26107 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:05:40.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.335+0000 7f2b883db740 1 -- 172.21.15.31:0/1295881399 shutdown_connections 2024-04-25T16:05:40.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.340+0000 7f2b883db740 1 -- 172.21.15.31:0/1295881399 wait complete. 2024-04-25T16:05:40.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.341+0000 7f2b883db740 20 client.26107 trim_cache size 0 max 16384 2024-04-25T16:05:40.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.413+0000 7f2b883db740 1 Processor -- start 2024-04-25T16:05:40.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.413+0000 7f2b883db740 1 -- start start 2024-04-25T16:05:40.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.414+0000 7f2b883db740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c235bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.414+0000 7f2b883db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c24bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.414+0000 7f2b883db740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be98390 0x55c55c2360f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.414+0000 7f2b883db740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6800e590 con 0x55c55be98390 2024-04-25T16:05:40.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.414+0000 7f2b883db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2b680100d0 con 0x55c55be776b0 2024-04-25T16:05:40.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.414+0000 7f2b883db740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6800ec60 con 0x55c55be6f4d0 2024-04-25T16:05:40.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.416+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be98390 0x55c55c2360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.416+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be98390 0x55c55c2360f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38542/0 (socket says 172.21.15.31:38542) 2024-04-25T16:05:40.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.416+0000 7f2b7b7fe640 1 -- 172.21.15.31:0/3760521856 learned_addr learned my addr 172.21.15.31:0/3760521856 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.417+0000 7f2b7affd640 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c235bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.417+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c24bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.418+0000 7f2b7affd640 1 -- 172.21.15.31:0/3760521856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 msgr2=0x55c55c24bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:40.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.418+0000 7f2b7affd640 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c24bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.418+0000 7f2b7affd640 1 -- 172.21.15.31:0/3760521856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be98390 msgr2=0x55c55c2360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:40.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.418+0000 7f2b7affd640 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be98390 0x55c55c2360f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.418+0000 7f2b7affd640 1 -- 172.21.15.31:0/3760521856 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c55c23fd50 con 0x55c55be6f4d0 2024-04-25T16:05:40.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.418+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c24bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:05:40.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.418+0000 7f2b7b7fe640 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be98390 0x55c55c2360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:05:40.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.426+0000 7f2b7affd640 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c235bb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005cd0 tx=0x7f2b6c00b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24e77d83a0253baa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.428+0000 7f2b60ff9640 1 -- 172.21.15.31:0/3760521856 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c02c070 con 0x55c55be6f4d0 2024-04-25T16:05:40.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.428+0000 7f2b60ff9640 1 -- 172.21.15.31:0/3760521856 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b6c026880 con 0x55c55be6f4d0 2024-04-25T16:05:40.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.430+0000 7f2b883db740 1 -- 172.21.15.31:0/3760521856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 msgr2=0x55c55c235bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.430+0000 7f2b883db740 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c235bb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005cd0 tx=0x7f2b6c00b730 comp rx=0 tx=0).stop 2024-04-25T16:05:40.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.434+0000 7f2b883db740 1 -- 172.21.15.31:0/3760521856 shutdown_connections 2024-04-25T16:05:40.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.434+0000 7f2b883db740 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be98390 0x55c55c2360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.434+0000 7f2b883db740 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be776b0 0x55c55c24bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.434+0000 7f2b883db740 1 --2- 172.21.15.31:0/3760521856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c235bb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.434+0000 7f2b883db740 1 -- 172.21.15.31:0/3760521856 >> 172.21.15.31:0/3760521856 conn(0x55c55be8c4e0 msgr2=0x55c55c23a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:40.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.434+0000 7f2b883db740 1 -- 172.21.15.31:0/3760521856 shutdown_connections 2024-04-25T16:05:40.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.434+0000 7f2b883db740 1 -- 172.21.15.31:0/3760521856 wait complete. 2024-04-25T16:05:40.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.435+0000 7f2b883db740 1 -- start start 2024-04-25T16:05:40.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.435+0000 7f2b883db740 1 Processor -- start 2024-04-25T16:05:40.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.436+0000 7f2b883db740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c249270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.436+0000 7f2b883db740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 0x55c55c2497b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.436+0000 7f2b883db740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 0x55c55c24a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.436+0000 7f2b883db740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6800fa00 con 0x55c55be776b0 2024-04-25T16:05:40.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.436+0000 7f2b883db740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6800f330 con 0x55c55be98390 2024-04-25T16:05:40.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.436+0000 7f2b883db740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2b6800c570 con 0x55c55be6f4d0 2024-04-25T16:05:40.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.436+0000 7f2b883db740 10 client.0 _ll_register_callbacks cb 0x7f2b70004740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:40.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.438+0000 7f2b7b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 0x55c55c24a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.438+0000 7f2b7affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c249270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.440+0000 7f2b7affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c249270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56354/0 (socket says 172.21.15.31:56354) 2024-04-25T16:05:40.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.440+0000 7f2b7affd640 1 -- 172.21.15.31:0/2245988370 learned_addr learned my addr 172.21.15.31:0/2245988370 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:40.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.442+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 0x55c55c2497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.445+0000 7f2b7affd640 1 -- 172.21.15.31:0/2245988370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 msgr2=0x55c55c24a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.445+0000 7f2b7affd640 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 0x55c55c24a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.445+0000 7f2b7affd640 1 -- 172.21.15.31:0/2245988370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 msgr2=0x55c55c2497b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.445+0000 7f2b7affd640 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 0x55c55c2497b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.445+0000 7f2b7affd640 1 -- 172.21.15.31:0/2245988370 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c55c24aca0 con 0x55c55be6f4d0 2024-04-25T16:05:40.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.452+0000 7f2b7affd640 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c249270 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005d00 tx=0x7f2b6c0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cec81e582d6ae38 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:40.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.452+0000 7f2b5eff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:40.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.455+0000 7f2b5eff5640 1 -- 172.21.15.31:0/2245988370 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b6c02c070 con 0x55c55be6f4d0 2024-04-25T16:05:40.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.455+0000 7f2b883db740 1 -- 172.21.15.31:0/2245988370 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b6c026880 con 0x55c55be6f4d0 2024-04-25T16:05:40.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.455+0000 7f2b5eff5640 1 -- 172.21.15.31:0/2245988370 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2b6c00dbb0 con 0x55c55be6f4d0 2024-04-25T16:05:40.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.459+0000 7f2b5eff5640 1 -- 172.21.15.31:0/2245988370 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2b6c0263d0 con 0x55c55be6f4d0 2024-04-25T16:05:40.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.459+0000 7f2b883db740 10 client.26113 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:40.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.459+0000 7f2b883db740 1 -- 172.21.15.31:0/2245988370 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c55c2ba7a0 con 0x55c55be6f4d0 2024-04-25T16:05:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.465+0000 7f2b5eff5640 1 -- 172.21.15.31:0/2245988370 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2b6c026880 con 0x55c55be6f4d0 2024-04-25T16:05:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.465+0000 7f2b5eff5640 1 -- 172.21.15.31:0/2245988370 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f2b700047f0 con 0x55c55be6f4d0 2024-04-25T16:05:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.465+0000 7f2b883db740 10 client.26113 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:40.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.465+0000 7f2b883db740 10 client.26113 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:05:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.465+0000 7f2b883db740 20 client.26113 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.465+0000 7f2b883db740 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55c55be2c3a0 0x55c55c2bcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.465+0000 7f2b883db740 4 client.26113 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:40.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.465+0000 7f2b883db740 1 -- 172.21.15.31:0/2245988370 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c55c1a6800 con 0x55c55be2c3a0 2024-04-25T16:05:40.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.474+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55c55be2c3a0 0x55c55c2bcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.498+0000 7f2b7a7fc640 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55c55be2c3a0 0x55c55c2bcfd0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=45d0493343ab8ff3 server_cookie=21fddf3a3c9ddaee in_seq=0 out_seq=0 2024-04-25T16:05:40.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.498+0000 7f2b5eff5640 10 client.26113 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:05:40.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.500+0000 7f2b5eff5640 1 -- 172.21.15.31:0/2245988370 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c55c1a6800 con 0x55c55be2c3a0 2024-04-25T16:05:40.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.500+0000 7f2b5eff5640 10 client.26113 handle_command_reply: tid=0 2024-04-25T16:05:40.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.501+0000 7f2b883db740 1 client.26113 shutdown 2024-04-25T16:05:40.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.505+0000 7f2b883db740 1 -- 172.21.15.31:0/2245988370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 msgr2=0x55c55c249270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.505+0000 7f2b883db740 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c249270 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005d00 tx=0x7f2b6c0045e0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.505+0000 7f2b883db740 1 -- 172.21.15.31:0/2245988370 shutdown_connections 2024-04-25T16:05:40.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.505+0000 7f2b883db740 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55c55be2c3a0 0x55c55c2bcfd0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.505+0000 7f2b883db740 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c55be98390 0x55c55c24a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.505+0000 7f2b883db740 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c55be776b0 0x55c55c2497b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.505+0000 7f2b883db740 1 --2- 172.21.15.31:0/2245988370 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c55be6f4d0 0x55c55c249270 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f2b6c005d00 tx=0x7f2b6c0045e0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.505+0000 7f2b883db740 1 -- 172.21.15.31:0/2245988370 >> 172.21.15.31:0/2245988370 conn(0x55c55be8c4e0 msgr2=0x55c55c23a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:40.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.505+0000 7f2b5eff5640 0 client.26113 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:05:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.517+0000 7f2b883db740 1 -- 172.21.15.31:0/2245988370 shutdown_connections 2024-04-25T16:05:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.517+0000 7f2b883db740 1 -- 172.21.15.31:0/2245988370 wait complete. 2024-04-25T16:05:40.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.518+0000 7f2b883db740 20 client.26113 trim_cache size 0 max 16384 2024-04-25T16:05:40.519 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:40.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.520+0000 7f2b8144c640 1 -- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b640afc50 msgr2=0x7f2b640b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.520+0000 7f2b8144c640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b640afc50 0x7f2b640b2160 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f2b700096c0 tx=0x7f2b70004b10 comp rx=0 tx=0).stop 2024-04-25T16:05:40.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.520+0000 7f2b8144c640 1 -- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b7c10b1d0 msgr2=0x7f2b7c1a9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.520+0000 7f2b8144c640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b7c10b1d0 0x7f2b7c1a9380 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f2b7400ba50 tx=0x7f2b7400bf10 comp rx=0 tx=0).stop 2024-04-25T16:05:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.520+0000 7f2b8144c640 1 -- 172.21.15.31:0/249522306 shutdown_connections 2024-04-25T16:05:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.520+0000 7f2b8144c640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2b640afc50 0x7f2b640b2160 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.520+0000 7f2b8144c640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2b7c10fef0 0x7f2b7c1b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.521+0000 7f2b8144c640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2b7c10b1d0 0x7f2b7c1a9380 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.521+0000 7f2b8144c640 1 --2- 172.21.15.31:0/249522306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b7c10a3e0 0x7f2b7c1a8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.521+0000 7f2b8144c640 1 -- 172.21.15.31:0/249522306 >> 172.21.15.31:0/249522306 conn(0x7f2b7c070400 msgr2=0x7f2b7c10bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:40.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.522+0000 7f2b8144c640 1 -- 172.21.15.31:0/249522306 shutdown_connections 2024-04-25T16:05:40.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.523+0000 7f2b8144c640 1 -- 172.21.15.31:0/249522306 wait complete. 2024-04-25T16:05:40.586 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:05:40.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:40 smithi031 ceph-mon[29140]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 793 MiB/s wr, 1.91k op/s 2024-04-25T16:05:40.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:05:40.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:05:40.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:05:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.991+0000 7fecd1f96640 1 -- 172.21.15.31:0/165942963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feccc10a420 msgr2=0x7feccc10add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.991+0000 7fecd1f96640 1 --2- 172.21.15.31:0/165942963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feccc10a420 0x7feccc10add0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fecc4009ca0 tx=0x7fecc402a8b0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.993+0000 7fecd1f96640 1 -- 172.21.15.31:0/165942963 shutdown_connections 2024-04-25T16:05:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.993+0000 7fecd1f96640 1 --2- 172.21.15.31:0/165942963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feccc110030 0x7feccc110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.993+0000 7fecd1f96640 1 --2- 172.21.15.31:0/165942963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feccc10b310 0x7feccc10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.993+0000 7fecd1f96640 1 --2- 172.21.15.31:0/165942963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feccc10a420 0x7feccc10add0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.993+0000 7fecd1f96640 1 -- 172.21.15.31:0/165942963 >> 172.21.15.31:0/165942963 conn(0x7feccc0704e0 msgr2=0x7feccc072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:40.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.997+0000 7fecd1f96640 1 -- 172.21.15.31:0/165942963 shutdown_connections 2024-04-25T16:05:41.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:40.999+0000 7fecd1f96640 1 -- 172.21.15.31:0/165942963 wait complete. 2024-04-25T16:05:41.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.000+0000 7fecd1f96640 1 Processor -- start 2024-04-25T16:05:41.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.000+0000 7fecd1f96640 1 -- start start 2024-04-25T16:05:41.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.001+0000 7fecd1f96640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feccc10a420 0x7feccc119e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.001+0000 7fecd1f96640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feccc10b310 0x7feccc11a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.001+0000 7fecd1f96640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feccc110030 0x7feccc11b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.001+0000 7fecd1f96640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feccc111f70 con 0x7feccc110030 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.001+0000 7fecd1f96640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7feccc111df0 con 0x7feccc10a420 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.001+0000 7fecd1f96640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feccc1120f0 con 0x7feccc10b310 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.003+0000 7feccb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feccc10a420 0x7feccc119e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.003+0000 7feccb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feccc10a420 0x7feccc119e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52442/0 (socket says 172.21.15.31:52442) 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.003+0000 7feccb7fe640 1 -- 172.21.15.31:0/479244430 learned_addr learned my addr 172.21.15.31:0/479244430 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.007+0000 7feccb7fe640 1 -- 172.21.15.31:0/479244430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feccc10b310 msgr2=0x7feccc11a360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.007+0000 7feccb7fe640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feccc10b310 0x7feccc11a360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.007+0000 7feccb7fe640 1 -- 172.21.15.31:0/479244430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feccc110030 msgr2=0x7feccc11b2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.007+0000 7feccb7fe640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feccc110030 0x7feccc11b2c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.007+0000 7feccb7fe640 1 -- 172.21.15.31:0/479244430 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc11b930 con 0x7feccc10a420 2024-04-25T16:05:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.020+0000 7feccb7fe640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feccc10a420 0x7feccc119e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fecc402aea0 tx=0x7fecc4004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.025+0000 7fecc8ff9640 1 -- 172.21.15.31:0/479244430 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4045470 con 0x7feccc10a420 2024-04-25T16:05:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.025+0000 7fecd1f96640 1 -- 172.21.15.31:0/479244430 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feccc11bbc0 con 0x7feccc10a420 2024-04-25T16:05:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.025+0000 7fecd1f96640 1 -- 172.21.15.31:0/479244430 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feccc11c110 con 0x7feccc10a420 2024-04-25T16:05:41.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.025+0000 7fecc8ff9640 1 -- 172.21.15.31:0/479244430 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecc4040070 con 0x7feccc10a420 2024-04-25T16:05:41.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.025+0000 7fecc8ff9640 1 -- 172.21.15.31:0/479244430 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc403c670 con 0x7feccc10a420 2024-04-25T16:05:41.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.030+0000 7fecc8ff9640 1 -- 172.21.15.31:0/479244430 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fecc403c810 con 0x7feccc10a420 2024-04-25T16:05:41.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.034+0000 7fecc8ff9640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecb40afcf0 0x7fecb40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.035+0000 7fecd8e47740 1 Processor -- start 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.035+0000 7fecd8e47740 1 -- start start 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.036+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f60810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.036+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f60d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.036+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f61a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.036+0000 7fecd8e47740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c558ec4d80 con 0x55c558bbe390 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.036+0000 7fecd8e47740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c558e7ea20 con 0x55c558b9d6b0 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.036+0000 7fecd8e47740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c558ec0d40 con 0x55c558b954d0 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.036+0000 7feccbfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f61a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.036+0000 7feccbfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f61a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52450/0 (socket says 172.21.15.31:52450) 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.036+0000 7feccbfff640 1 -- 172.21.15.31:0/3811529739 learned_addr learned my addr 172.21.15.31:0/3811529739 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.037+0000 7feccaffd640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecb40afcf0 0x7fecb40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.037+0000 7feccaffd640 1 --2- 172.21.15.31:0/3811529739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f60d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.037+0000 7fecc8ff9640 1 -- 172.21.15.31:0/479244430 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fecc41010c0 con 0x7feccc10a420 2024-04-25T16:05:41.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.040+0000 7feccb7fe640 1 --2- 172.21.15.31:0/3811529739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f60810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.043+0000 7feccbfff640 1 -- 172.21.15.31:0/3811529739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 msgr2=0x55c558f60d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.043+0000 7feccbfff640 1 --2- 172.21.15.31:0/3811529739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f60d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.043+0000 7feccbfff640 1 -- 172.21.15.31:0/3811529739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 msgr2=0x55c558f60810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.043+0000 7feccbfff640 1 --2- 172.21.15.31:0/3811529739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f60810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.043+0000 7feccbfff640 1 -- 172.21.15.31:0/3811529739 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c558e89c30 con 0x55c558b9d6b0 2024-04-25T16:05:41.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.045+0000 7feccbfff640 1 --2- 172.21.15.31:0/3811529739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f61a20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fecbc02c8f0 tx=0x7fecbc02cdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17a259c968e06aa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.046+0000 7fecb9ffb640 1 -- 172.21.15.31:0/3811529739 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc02ecd0 con 0x55c558b9d6b0 2024-04-25T16:05:41.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.046+0000 7fecb9ffb640 1 -- 172.21.15.31:0/3811529739 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecbc031910 con 0x55c558b9d6b0 2024-04-25T16:05:41.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.046+0000 7fecb9ffb640 1 -- 172.21.15.31:0/3811529739 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc03a8c0 con 0x55c558b9d6b0 2024-04-25T16:05:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.048+0000 7fecd8e47740 1 -- 172.21.15.31:0/3811529739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 msgr2=0x55c558f61a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.048+0000 7fecd8e47740 1 --2- 172.21.15.31:0/3811529739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f61a20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fecbc02c8f0 tx=0x7fecbc02cdb0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.049+0000 7fecd8e47740 1 -- 172.21.15.31:0/3811529739 shutdown_connections 2024-04-25T16:05:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.049+0000 7fecd8e47740 1 --2- 172.21.15.31:0/3811529739 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f61a20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.049+0000 7fecd8e47740 1 --2- 172.21.15.31:0/3811529739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f60d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.049+0000 7fecd8e47740 1 --2- 172.21.15.31:0/3811529739 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f60810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.049+0000 7fecd8e47740 1 -- 172.21.15.31:0/3811529739 >> 172.21.15.31:0/3811529739 conn(0x55c558bb24e0 msgr2=0x55c558f5c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.049+0000 7feccaffd640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecb40afcf0 0x7fecb40b2200 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7feccc11aec0 tx=0x7fecc000c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.057+0000 7fecd8e47740 1 -- 172.21.15.31:0/3811529739 shutdown_connections 2024-04-25T16:05:41.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.060+0000 7fecd8e47740 1 -- 172.21.15.31:0/3811529739 wait complete. 2024-04-25T16:05:41.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.061+0000 7fecd8e47740 1 -- start start 2024-04-25T16:05:41.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.062+0000 7fecd8e47740 1 Processor -- start 2024-04-25T16:05:41.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.063+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f6a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.063+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f6aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.063+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f71a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.064+0000 7fecd8e47740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c558f62580 con 0x55c558bbe390 2024-04-25T16:05:41.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.064+0000 7fecd8e47740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c558e4bdc0 con 0x55c558b9d6b0 2024-04-25T16:05:41.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.064+0000 7fecd8e47740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c558f62700 con 0x55c558b954d0 2024-04-25T16:05:41.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.064+0000 7feccbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f71a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.064+0000 7feccbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f71a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38568/0 (socket says 172.21.15.31:38568) 2024-04-25T16:05:41.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.064+0000 7feccbfff640 1 -- 172.21.15.31:0/1797616885 learned_addr learned my addr 172.21.15.31:0/1797616885 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.064+0000 7fecd8e47740 10 client.0 _ll_register_callbacks cb 0x55c558e67600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:41.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.064+0000 7feccb7fe640 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f6a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.064+0000 7feccaffd640 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f6aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.065+0000 7feccb7fe640 1 -- 172.21.15.31:0/1797616885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 msgr2=0x55c558f6aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.065+0000 7feccb7fe640 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f6aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.065+0000 7feccb7fe640 1 -- 172.21.15.31:0/1797616885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 msgr2=0x55c558f71a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.065+0000 7feccb7fe640 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f71a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.065+0000 7feccb7fe640 1 -- 172.21.15.31:0/1797616885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c558bd63f0 con 0x55c558b954d0 2024-04-25T16:05:41.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.066+0000 7feccb7fe640 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f6a5a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fecc40c9250 tx=0x7fecc40c8cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5719b3d754a0f2c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.066+0000 7fecaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:41.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.066+0000 7fecaeffd640 1 -- 172.21.15.31:0/1797616885 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc40e2020 con 0x55c558b954d0 2024-04-25T16:05:41.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.066+0000 7fecaeffd640 1 -- 172.21.15.31:0/1797616885 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fecc40c97b0 con 0x55c558b954d0 2024-04-25T16:05:41.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.068+0000 7fecd8e47740 1 -- 172.21.15.31:0/1797616885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecbc031910 con 0x55c558b954d0 2024-04-25T16:05:41.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.071+0000 7fecaeffd640 1 -- 172.21.15.31:0/1797616885 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fecc4030bc0 con 0x55c558b954d0 2024-04-25T16:05:41.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.071+0000 7fecd8e47740 10 client.26119 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:41.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.071+0000 7fecd8e47740 1 -- 172.21.15.31:0/1797616885 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c558ca07d0 con 0x55c558b954d0 2024-04-25T16:05:41.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.071+0000 7fecaeffd640 1 -- 172.21.15.31:0/1797616885 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecbc031910 con 0x55c558b954d0 2024-04-25T16:05:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.072+0000 7fecaeffd640 1 -- 172.21.15.31:0/1797616885 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fecc40d1f20 con 0x55c558b954d0 2024-04-25T16:05:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.072+0000 7fecd8e47740 10 client.26119 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.072+0000 7fecd8e47740 10 client.26119 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:05:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.072+0000 7fecd8e47740 20 client.26119 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.072+0000 7fecd8e47740 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55c558b7cf60 0x55c558f785c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.072+0000 7fecd8e47740 4 client.26119 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:41.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.072+0000 7fecd8e47740 1 -- 172.21.15.31:0/1797616885 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c558ed2990 con 0x55c558b7cf60 2024-04-25T16:05:41.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.074+0000 7feccaffd640 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55c558b7cf60 0x55c558f785c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.077+0000 7fecaeffd640 10 client.26119 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:05:41.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.077+0000 7feccaffd640 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55c558b7cf60 0x55c558f785c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fcac607f3f38f8ac server_cookie=9dff9ad0781ddf61 in_seq=0 out_seq=0 2024-04-25T16:05:41.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.078+0000 7fecaeffd640 1 -- 172.21.15.31:0/1797616885 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c558ed2990 con 0x55c558b7cf60 2024-04-25T16:05:41.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.078+0000 7fecaeffd640 10 client.26119 handle_command_reply: tid=0 2024-04-25T16:05:41.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.079+0000 7fecd8e47740 1 client.26119 shutdown 2024-04-25T16:05:41.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.079+0000 7fecd8e47740 1 -- 172.21.15.31:0/1797616885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 msgr2=0x55c558f6a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.079+0000 7fecd8e47740 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f6a5a0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fecc40c9250 tx=0x7fecc40c8cc0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.081+0000 7fecd8e47740 1 -- 172.21.15.31:0/1797616885 shutdown_connections 2024-04-25T16:05:41.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.081+0000 7fecd8e47740 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f71a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.081+0000 7fecd8e47740 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f6aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.081+0000 7fecd8e47740 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55c558b7cf60 0x55c558f785c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.081+0000 7fecd8e47740 1 --2- 172.21.15.31:0/1797616885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f6a5a0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.081+0000 7fecaeffd640 0 client.26119 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:05:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.081+0000 7fecd8e47740 1 -- 172.21.15.31:0/1797616885 >> 172.21.15.31:0/1797616885 conn(0x55c558bb24e0 msgr2=0x55c558f5b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.081+0000 7fecd8e47740 1 -- 172.21.15.31:0/1797616885 shutdown_connections 2024-04-25T16:05:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.081+0000 7fecd8e47740 1 -- 172.21.15.31:0/1797616885 wait complete. 2024-04-25T16:05:41.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.082+0000 7fecd8e47740 20 client.26119 trim_cache size 0 max 16384 2024-04-25T16:05:41.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.110+0000 7fecd8e47740 1 Processor -- start 2024-04-25T16:05:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.114+0000 7fecd8e47740 1 -- start start 2024-04-25T16:05:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.114+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f65860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.114+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f719d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.114+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f6a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.114+0000 7fecd8e47740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fecb000e4c0 con 0x55c558bbe390 2024-04-25T16:05:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.114+0000 7fecd8e47740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fecb0010000 con 0x55c558b9d6b0 2024-04-25T16:05:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.114+0000 7fecd8e47740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fecb000eb90 con 0x55c558b954d0 2024-04-25T16:05:41.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.116+0000 7feccbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f6a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.116+0000 7feccbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f6a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38582/0 (socket says 172.21.15.31:38582) 2024-04-25T16:05:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.116+0000 7feccbfff640 1 -- 172.21.15.31:0/2661971686 learned_addr learned my addr 172.21.15.31:0/2661971686 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.116+0000 7feccb7fe640 1 --2- 172.21.15.31:0/2661971686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f65860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.116+0000 7feccbfff640 1 -- 172.21.15.31:0/2661971686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 msgr2=0x55c558f65860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.116+0000 7feccbfff640 1 --2- 172.21.15.31:0/2661971686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f65860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.116+0000 7feccbfff640 1 -- 172.21.15.31:0/2661971686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 msgr2=0x55c558f719d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.116+0000 7feccbfff640 1 --2- 172.21.15.31:0/2661971686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f719d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.116+0000 7feccbfff640 1 -- 172.21.15.31:0/2661971686 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c558f5c1a0 con 0x55c558bbe390 2024-04-25T16:05:41.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.117+0000 7feccbfff640 1 --2- 172.21.15.31:0/2661971686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f6a880 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fecbc02c590 tx=0x7fecbc02c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bb620515559ca9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.118+0000 7fecb97fa640 1 -- 172.21.15.31:0/2661971686 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc02ee10 con 0x55c558bbe390 2024-04-25T16:05:41.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.118+0000 7fecb97fa640 1 -- 172.21.15.31:0/2661971686 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecbc048070 con 0x55c558bbe390 2024-04-25T16:05:41.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.118+0000 7fecb97fa640 1 -- 172.21.15.31:0/2661971686 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc004e50 con 0x55c558bbe390 2024-04-25T16:05:41.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:40 smithi063 ceph-mon[32413]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 793 MiB/s wr, 1.91k op/s 2024-04-25T16:05:41.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:05:41.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:05:41.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:05:41.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.119+0000 7fecd8e47740 1 -- 172.21.15.31:0/2661971686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 msgr2=0x55c558f6a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.119+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2661971686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f6a880 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fecbc02c590 tx=0x7fecbc02c5c0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.120+0000 7fecd8e47740 1 -- 172.21.15.31:0/2661971686 shutdown_connections 2024-04-25T16:05:41.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.120+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2661971686 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558f6a880 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.120+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2661971686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b9d6b0 0x55c558f719d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.120+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2661971686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b954d0 0x55c558f65860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.120+0000 7fecd8e47740 1 -- 172.21.15.31:0/2661971686 >> 172.21.15.31:0/2661971686 conn(0x55c558bb24e0 msgr2=0x55c558f6a170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.122+0000 7fecd8e47740 1 -- 172.21.15.31:0/2661971686 shutdown_connections 2024-04-25T16:05:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.122+0000 7fecd8e47740 1 -- 172.21.15.31:0/2661971686 wait complete. 2024-04-25T16:05:41.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.123+0000 7fecd8e47740 1 -- start start 2024-04-25T16:05:41.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.124+0000 7fecd8e47740 1 Processor -- start 2024-04-25T16:05:41.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.125+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b954d0 0x55c558f6e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.125+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b9d6b0 0x55c558f6ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.125+0000 7fecd8e47740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558fde960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.126+0000 7feccb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b954d0 0x55c558f6e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.126+0000 7feccb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b954d0 0x55c558f6e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52494/0 (socket says 172.21.15.31:52494) 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.127+0000 7fecd8e47740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fecb000f930 con 0x55c558bbe390 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.127+0000 7fecd8e47740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fecb000f260 con 0x55c558b954d0 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.127+0000 7fecd8e47740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fecb000c4a0 con 0x55c558b9d6b0 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.127+0000 7fecd8e47740 10 client.0 _ll_register_callbacks cb 0x7fecc4009440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.127+0000 7feccb7fe640 1 -- 172.21.15.31:0/2544442920 learned_addr learned my addr 172.21.15.31:0/2544442920 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.130+0000 7feccaffd640 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b9d6b0 0x55c558f6ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.130+0000 7feccbfff640 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558fde960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.131+0000 7feccbfff640 1 -- 172.21.15.31:0/2544442920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b9d6b0 msgr2=0x55c558f6ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.131+0000 7feccbfff640 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b9d6b0 0x55c558f6ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.131+0000 7feccbfff640 1 -- 172.21.15.31:0/2544442920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b954d0 msgr2=0x55c558f6e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.131+0000 7feccbfff640 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b954d0 0x55c558f6e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.131+0000 7feccbfff640 1 -- 172.21.15.31:0/2544442920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c558fdeea0 con 0x55c558bbe390 2024-04-25T16:05:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.131+0000 7feccbfff640 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558fde960 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fecbc02c680 tx=0x7fecbc0048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80982c3a0aa0c637 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.131+0000 7fecaeffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:41.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.132+0000 7fecaeffd640 1 -- 172.21.15.31:0/2544442920 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc050030 con 0x55c558bbe390 2024-04-25T16:05:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.132+0000 7fecaeffd640 1 -- 172.21.15.31:0/2544442920 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fecbc004020 con 0x55c558bbe390 2024-04-25T16:05:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.132+0000 7fecd8e47740 1 -- 172.21.15.31:0/2544442920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecbc048070 con 0x55c558bbe390 2024-04-25T16:05:41.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.133+0000 7fecaeffd640 1 -- 172.21.15.31:0/2544442920 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fecbc004220 con 0x55c558bbe390 2024-04-25T16:05:41.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.133+0000 7fecd8e47740 10 client.16494 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:41.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.133+0000 7fecd8e47740 1 -- 172.21.15.31:0/2544442920 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c558fe0020 con 0x55c558bbe390 2024-04-25T16:05:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.135+0000 7fecaeffd640 1 -- 172.21.15.31:0/2544442920 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecbc0043c0 con 0x55c558bbe390 2024-04-25T16:05:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.135+0000 7fecaeffd640 1 -- 172.21.15.31:0/2544442920 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fecbc031710 con 0x55c558bbe390 2024-04-25T16:05:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.137+0000 7fecd8e47740 10 client.16494 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.137+0000 7fecd8e47740 10 client.16494 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:05:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.137+0000 7fecd8e47740 20 client.16494 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:41.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.138+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55c558b523a0 0x55c558fe26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.138+0000 7fecd8e47740 4 client.16494 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:41.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.138+0000 7fecd8e47740 1 -- 172.21.15.31:0/2544442920 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fecbc048070 con 0x55c558b523a0 2024-04-25T16:05:41.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.139+0000 7feccb7fe640 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55c558b523a0 0x55c558fe26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.142+0000 7feccb7fe640 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55c558b523a0 0x55c558fe26e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=af5ac6e7240874d5 server_cookie=ce9f1a98249cdc5e in_seq=0 out_seq=0 2024-04-25T16:05:41.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.143+0000 7fecaeffd640 10 client.16494 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:05:41.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.143+0000 7fecaeffd640 1 -- 172.21.15.31:0/2544442920 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fecbc048070 con 0x55c558b523a0 2024-04-25T16:05:41.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.143+0000 7fecaeffd640 10 client.16494 handle_command_reply: tid=0 2024-04-25T16:05:41.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.143+0000 7fecd8e47740 1 client.16494 shutdown 2024-04-25T16:05:41.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:40 smithi186 ceph-mon[33258]: pgmap v482: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 793 MiB/s wr, 1.91k op/s 2024-04-25T16:05:41.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:05:41.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:05:41.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:05:41.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.148+0000 7fecd8e47740 1 -- 172.21.15.31:0/2544442920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 msgr2=0x55c558fde960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.148+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558fde960 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fecbc02c680 tx=0x7fecbc0048b0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.148+0000 7fecd8e47740 1 -- 172.21.15.31:0/2544442920 shutdown_connections 2024-04-25T16:05:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.148+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c558bbe390 0x55c558fde960 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.148+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55c558b523a0 0x55c558fe26e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.149+0000 7fecaeffd640 0 client.16494 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:05:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.149+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c558b9d6b0 0x55c558f6ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.149+0000 7fecd8e47740 1 --2- 172.21.15.31:0/2544442920 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c558b954d0 0x55c558f6e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.149+0000 7fecd8e47740 1 -- 172.21.15.31:0/2544442920 >> 172.21.15.31:0/2544442920 conn(0x55c558bb24e0 msgr2=0x55c558f5a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.150+0000 7fecd8e47740 1 -- 172.21.15.31:0/2544442920 shutdown_connections 2024-04-25T16:05:41.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.154+0000 7fecd8e47740 1 -- 172.21.15.31:0/2544442920 wait complete. 2024-04-25T16:05:41.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.154+0000 7fecd8e47740 20 client.16494 trim_cache size 0 max 16384 2024-04-25T16:05:41.159 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:41.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.160+0000 7fecd1f96640 1 -- 172.21.15.31:0/479244430 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecb40afcf0 msgr2=0x7fecb40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.160+0000 7fecd1f96640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecb40afcf0 0x7fecb40b2200 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7feccc11aec0 tx=0x7fecc000c920 comp rx=0 tx=0).stop 2024-04-25T16:05:41.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.160+0000 7fecd1f96640 1 -- 172.21.15.31:0/479244430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feccc10a420 msgr2=0x7feccc119e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.160+0000 7fecd1f96640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feccc10a420 0x7feccc119e20 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fecc402aea0 tx=0x7fecc4004270 comp rx=0 tx=0).stop 2024-04-25T16:05:41.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.165+0000 7fecd1f96640 1 -- 172.21.15.31:0/479244430 shutdown_connections 2024-04-25T16:05:41.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.165+0000 7fecd1f96640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fecb40afcf0 0x7fecb40b2200 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.165+0000 7fecd1f96640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feccc110030 0x7feccc11b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.165+0000 7fecd1f96640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feccc10b310 0x7feccc11a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.165+0000 7fecd1f96640 1 --2- 172.21.15.31:0/479244430 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feccc10a420 0x7feccc119e20 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.165+0000 7fecd1f96640 1 -- 172.21.15.31:0/479244430 >> 172.21.15.31:0/479244430 conn(0x7feccc0704e0 msgr2=0x7feccc10e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.169+0000 7fecd1f96640 1 -- 172.21.15.31:0/479244430 shutdown_connections 2024-04-25T16:05:41.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.169+0000 7fecd1f96640 1 -- 172.21.15.31:0/479244430 wait complete. 2024-04-25T16:05:41.227 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:05:41.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.642+0000 7f633c8aa640 1 -- 172.21.15.31:0/1516869153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6338110030 msgr2=0x7f6338110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.642+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1516869153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6338110030 0x7f6338110440 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6330009be0 tx=0x7f633002a710 comp rx=0 tx=0).stop 2024-04-25T16:05:41.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.642+0000 7f633c8aa640 1 -- 172.21.15.31:0/1516869153 shutdown_connections 2024-04-25T16:05:41.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.642+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1516869153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6338110030 0x7f6338110440 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.642+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1516869153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f633810b310 0x7f633810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.642+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1516869153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f633810a420 0x7f633810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.642+0000 7f633c8aa640 1 -- 172.21.15.31:0/1516869153 >> 172.21.15.31:0/1516869153 conn(0x7f63380704e0 msgr2=0x7f6338072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.644+0000 7f633c8aa640 1 -- 172.21.15.31:0/1516869153 shutdown_connections 2024-04-25T16:05:41.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.645+0000 7f633c8aa640 1 -- 172.21.15.31:0/1516869153 wait complete. 2024-04-25T16:05:41.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.647+0000 7f633c8aa640 1 Processor -- start 2024-04-25T16:05:41.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.647+0000 7f633c8aa640 1 -- start start 2024-04-25T16:05:41.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.648+0000 7f633c8aa640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f633810a420 0x7f63381a8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.648+0000 7f633c8aa640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f633810b310 0x7f63381a9530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.648+0000 7f633c8aa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6338110030 0x7f63381b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.648+0000 7f633c8aa640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f633811e490 con 0x7f633810a420 2024-04-25T16:05:41.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.648+0000 7f633c8aa640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f633811e310 con 0x7f6338110030 2024-04-25T16:05:41.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.648+0000 7f633c8aa640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f633811e610 con 0x7f633810b310 2024-04-25T16:05:41.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.650+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f633810b310 0x7f63381a9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.650+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f633810b310 0x7f63381a9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56436/0 (socket says 172.21.15.31:56436) 2024-04-25T16:05:41.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.650+0000 7f6335d9b640 1 -- 172.21.15.31:0/1173900941 learned_addr learned my addr 172.21.15.31:0/1173900941 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.651+0000 7f6335d9b640 1 -- 172.21.15.31:0/1173900941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6338110030 msgr2=0x7f63381b04c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:05:41.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.651+0000 7f6335d9b640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6338110030 0x7f63381b04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.651+0000 7f6335d9b640 1 -- 172.21.15.31:0/1173900941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f633810a420 msgr2=0x7f63381a8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.651+0000 7f6335d9b640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f633810a420 0x7f63381a8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.651+0000 7f6335d9b640 1 -- 172.21.15.31:0/1173900941 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6338111c90 con 0x7f633810b310 2024-04-25T16:05:41.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.652+0000 7f6335d9b640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f633810b310 0x7f63381a9530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f632800e9f0 tx=0x7f632800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.653+0000 7f63277fe640 1 -- 172.21.15.31:0/1173900941 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632800cd80 con 0x7f633810b310 2024-04-25T16:05:41.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.654+0000 7f633c8aa640 1 -- 172.21.15.31:0/1173900941 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6338111f80 con 0x7f633810b310 2024-04-25T16:05:41.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.654+0000 7f63277fe640 1 -- 172.21.15.31:0/1173900941 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63280108e0 con 0x7f633810b310 2024-04-25T16:05:41.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.654+0000 7f633c8aa640 1 -- 172.21.15.31:0/1173900941 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6338112520 con 0x7f633810b310 2024-04-25T16:05:41.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.654+0000 7f63277fe640 1 -- 172.21.15.31:0/1173900941 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6328018880 con 0x7f633810b310 2024-04-25T16:05:41.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.662+0000 7f6343833740 1 Processor -- start 2024-04-25T16:05:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.687+0000 7f6343833740 1 -- start start 2024-04-25T16:05:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.688+0000 7f6343833740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e071b810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.688+0000 7f6343833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e071bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.688+0000 7f6343833740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03586b0 0x5626e071ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.688+0000 7f6343833740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5626e067fd80 con 0x5626e0379390 2024-04-25T16:05:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.688+0000 7f6343833740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5626e0639a20 con 0x5626e03586b0 2024-04-25T16:05:41.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.688+0000 7f6343833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5626e067bd40 con 0x5626e03504d0 2024-04-25T16:05:41.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.692+0000 7f63277fe640 1 -- 172.21.15.31:0/1173900941 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6328010430 con 0x7f633810b310 2024-04-25T16:05:41.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.693+0000 7f633659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e071bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.693+0000 7f633659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e071bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56438/0 (socket says 172.21.15.31:56438) 2024-04-25T16:05:41.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.693+0000 7f633659c640 1 -- 172.21.15.31:0/2734718981 learned_addr learned my addr 172.21.15.31:0/2734718981 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.693+0000 7f63277fe640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f63200afbf0 0x7f63200b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.694+0000 7f63277fe640 1 -- 172.21.15.31:0/1173900941 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f63280df150 con 0x7f633810b310 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.694+0000 7f633659c640 1 -- 172.21.15.31:0/2734718981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03586b0 msgr2=0x5626e071ca20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.694+0000 7f633659c640 1 --2- 172.21.15.31:0/2734718981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03586b0 0x5626e071ca20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.694+0000 7f633659c640 1 -- 172.21.15.31:0/2734718981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 msgr2=0x5626e071b810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.694+0000 7f633659c640 1 --2- 172.21.15.31:0/2734718981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e071b810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.694+0000 7f633659c640 1 -- 172.21.15.31:0/2734718981 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e0644c30 con 0x5626e03504d0 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.698+0000 7f633659c640 1 --2- 172.21.15.31:0/2734718981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e071bd90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f632c00bab0 tx=0x7f632c00bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ca3c9cdf79e14d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.699+0000 7f6324ff9640 1 -- 172.21.15.31:0/2734718981 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f632c016070 con 0x5626e03504d0 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.699+0000 7f6324ff9640 1 -- 172.21.15.31:0/2734718981 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f632c0054d0 con 0x5626e03504d0 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.701+0000 7f6343833740 1 -- 172.21.15.31:0/2734718981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 msgr2=0x5626e071bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.701+0000 7f6343833740 1 --2- 172.21.15.31:0/2734718981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e071bd90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f632c00bab0 tx=0x7f632c00bf70 comp rx=0 tx=0).stop 2024-04-25T16:05:41.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.718+0000 7f6336d9d640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f63200afbf0 0x7f63200b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.719+0000 7f6343833740 1 -- 172.21.15.31:0/2734718981 shutdown_connections 2024-04-25T16:05:41.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.719+0000 7f6343833740 1 --2- 172.21.15.31:0/2734718981 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03586b0 0x5626e071ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.719+0000 7f6343833740 1 --2- 172.21.15.31:0/2734718981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e071bd90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.719+0000 7f6343833740 1 --2- 172.21.15.31:0/2734718981 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e071b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.719+0000 7f6343833740 1 -- 172.21.15.31:0/2734718981 >> 172.21.15.31:0/2734718981 conn(0x5626e036d4e0 msgr2=0x5626e0717020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.728+0000 7f6343833740 1 -- 172.21.15.31:0/2734718981 shutdown_connections 2024-04-25T16:05:41.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.728+0000 7f6343833740 1 -- 172.21.15.31:0/2734718981 wait complete. 2024-04-25T16:05:41.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.729+0000 7f6343833740 1 -- start start 2024-04-25T16:05:41.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.729+0000 7f6343833740 1 Processor -- start 2024-04-25T16:05:41.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.730+0000 7f6343833740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03504d0 0x5626e0725470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.730+0000 7f6343833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e07259b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.730+0000 7f6343833740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e072c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.730+0000 7f6343833740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5626e071d580 con 0x5626e0379390 2024-04-25T16:05:41.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.730+0000 7f6343833740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5626e0606dc0 con 0x5626e03504d0 2024-04-25T16:05:41.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.730+0000 7f6343833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5626e071d700 con 0x5626e03586b0 2024-04-25T16:05:41.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.730+0000 7f6343833740 10 client.0 _ll_register_callbacks cb 0x5626e0622600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:41.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.731+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e07259b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.731+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e07259b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56452/0 (socket says 172.21.15.31:56452) 2024-04-25T16:05:41.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.731+0000 7f6335d9b640 1 -- 172.21.15.31:0/2663239023 learned_addr learned my addr 172.21.15.31:0/2663239023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.731+0000 7f6335d9b640 1 -- 172.21.15.31:0/2663239023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03504d0 msgr2=0x5626e0725470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:41.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.731+0000 7f6335d9b640 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03504d0 0x5626e0725470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.731+0000 7f6335d9b640 1 -- 172.21.15.31:0/2663239023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 msgr2=0x5626e072c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.731+0000 7f6335d9b640 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e072c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.731+0000 7f6335d9b640 1 -- 172.21.15.31:0/2663239023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626e03913f0 con 0x5626e03586b0 2024-04-25T16:05:41.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.732+0000 7f6335d9b640 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e07259b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f63280a7570 tx=0x7f63280a7020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2975b365b6d379aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.738+0000 7f631dffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:41.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.738+0000 7f631dffb640 1 -- 172.21.15.31:0/2663239023 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63280a7ce0 con 0x5626e03586b0 2024-04-25T16:05:41.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.738+0000 7f6343833740 1 -- 172.21.15.31:0/2663239023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f632c0054d0 con 0x5626e03586b0 2024-04-25T16:05:41.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.738+0000 7f631dffb640 1 -- 172.21.15.31:0/2663239023 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6328010a90 con 0x5626e03586b0 2024-04-25T16:05:41.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.740+0000 7f631dffb640 1 -- 172.21.15.31:0/2663239023 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f63280a75e0 con 0x5626e03586b0 2024-04-25T16:05:41.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.740+0000 7f6343833740 10 client.26134 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:41.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.740+0000 7f6343833740 1 -- 172.21.15.31:0/2663239023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5626e045b7d0 con 0x5626e03586b0 2024-04-25T16:05:41.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.742+0000 7f631dffb640 1 -- 172.21.15.31:0/2663239023 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f632c0054d0 con 0x5626e03586b0 2024-04-25T16:05:41.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.742+0000 7f631dffb640 1 -- 172.21.15.31:0/2663239023 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f63280a6380 con 0x5626e03586b0 2024-04-25T16:05:41.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.742+0000 7f6343833740 10 client.26134 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:41.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.742+0000 7f6343833740 10 client.26134 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:05:41.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.742+0000 7f6343833740 20 client.26134 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:41.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.742+0000 7f6343833740 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5626e0337f60 0x5626e07336b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.742+0000 7f6343833740 4 client.26134 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:41.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.742+0000 7f6343833740 1 -- 172.21.15.31:0/2663239023 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5626e068d990 con 0x5626e0337f60 2024-04-25T16:05:41.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.749+0000 7f633659c640 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5626e0337f60 0x5626e07336b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.768+0000 7f631dffb640 10 client.26134 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:05:41.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.768+0000 7f633659c640 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5626e0337f60 0x5626e07336b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=68006a22466fc6c0 server_cookie=f5a22462f980fbb6 in_seq=0 out_seq=0 2024-04-25T16:05:41.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.770+0000 7f6336d9d640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f63200afbf0 0x7f63200b2100 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6330009b30 tx=0x7f63300303a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.771+0000 7f631dffb640 1 -- 172.21.15.31:0/2663239023 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5626e068d990 con 0x5626e0337f60 2024-04-25T16:05:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.771+0000 7f631dffb640 10 client.26134 handle_command_reply: tid=0 2024-04-25T16:05:41.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.771+0000 7f6343833740 1 client.26134 shutdown 2024-04-25T16:05:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.771+0000 7f6343833740 1 -- 172.21.15.31:0/2663239023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 msgr2=0x5626e07259b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.771+0000 7f6343833740 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e07259b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f63280a7570 tx=0x7f63280a7020 comp rx=0 tx=0).stop 2024-04-25T16:05:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.772+0000 7f6343833740 1 -- 172.21.15.31:0/2663239023 shutdown_connections 2024-04-25T16:05:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.772+0000 7f6343833740 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e072c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.772+0000 7f6343833740 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e07259b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.772+0000 7f6343833740 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5626e0337f60 0x5626e07336b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.772+0000 7f6343833740 1 --2- 172.21.15.31:0/2663239023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03504d0 0x5626e0725470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.772+0000 7f6343833740 1 -- 172.21.15.31:0/2663239023 >> 172.21.15.31:0/2663239023 conn(0x5626e036d4e0 msgr2=0x5626e0719f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.772+0000 7f631dffb640 0 client.26134 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:05:41.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.780+0000 7f6343833740 1 -- 172.21.15.31:0/2663239023 shutdown_connections 2024-04-25T16:05:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.780+0000 7f6343833740 1 -- 172.21.15.31:0/2663239023 wait complete. 2024-04-25T16:05:41.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.780+0000 7f6343833740 20 client.26134 trim_cache size 0 max 16384 2024-04-25T16:05:41.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.863+0000 7f6343833740 1 Processor -- start 2024-04-25T16:05:41.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.863+0000 7f6343833740 1 -- start start 2024-04-25T16:05:41.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.864+0000 7f6343833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e0711d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.864+0000 7f6343833740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e03586b0 0x5626e072c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.864+0000 7f6343833740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e0379390 0x5626e0737ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.865+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e03586b0 0x5626e072c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.865+0000 7f6335d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e03586b0 0x5626e072c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38630/0 (socket says 172.21.15.31:38630) 2024-04-25T16:05:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.866+0000 7f6336d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e0379390 0x5626e0737ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.866+0000 7f6343833740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f631800e590 con 0x5626e03586b0 2024-04-25T16:05:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.866+0000 7f6343833740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f63180100d0 con 0x5626e0379390 2024-04-25T16:05:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.866+0000 7f6343833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f631800ec60 con 0x5626e03504d0 2024-04-25T16:05:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.866+0000 7f6336d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e0379390 0x5626e0737ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52536/0 (socket says 172.21.15.31:52536) 2024-04-25T16:05:41.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.866+0000 7f6336d9d640 1 -- 172.21.15.31:0/3173607164 learned_addr learned my addr 172.21.15.31:0/3173607164 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.867+0000 7f633659c640 1 --2- 172.21.15.31:0/3173607164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e0711d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.867+0000 7f6335d9b640 1 -- 172.21.15.31:0/3173607164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 msgr2=0x5626e0711d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.867+0000 7f6335d9b640 1 --2- 172.21.15.31:0/3173607164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e0711d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.867+0000 7f6335d9b640 1 -- 172.21.15.31:0/3173607164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e0379390 msgr2=0x5626e0737ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.867+0000 7f6335d9b640 1 --2- 172.21.15.31:0/3173607164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e0379390 0x5626e0737ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.868+0000 7f6335d9b640 1 -- 172.21.15.31:0/3173607164 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5626e072cd70 con 0x5626e03586b0 2024-04-25T16:05:41.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.868+0000 7f6335d9b640 1 --2- 172.21.15.31:0/3173607164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e03586b0 0x5626e072c830 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f632800e9c0 tx=0x7f6328009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523468b194ba72ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.873+0000 7f631ffff640 1 -- 172.21.15.31:0/3173607164 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63280b0440 con 0x5626e03586b0 2024-04-25T16:05:41.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.874+0000 7f631ffff640 1 -- 172.21.15.31:0/3173607164 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63280ab990 con 0x5626e03586b0 2024-04-25T16:05:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.879+0000 7f6343833740 1 -- 172.21.15.31:0/3173607164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e03586b0 msgr2=0x5626e072c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.879+0000 7f6343833740 1 --2- 172.21.15.31:0/3173607164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e03586b0 0x5626e072c830 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f632800e9c0 tx=0x7f6328009700 comp rx=0 tx=0).stop 2024-04-25T16:05:41.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.884+0000 7f6343833740 1 -- 172.21.15.31:0/3173607164 shutdown_connections 2024-04-25T16:05:41.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.884+0000 7f6343833740 1 --2- 172.21.15.31:0/3173607164 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e0379390 0x5626e0737ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.884+0000 7f6343833740 1 --2- 172.21.15.31:0/3173607164 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e03586b0 0x5626e072c830 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.884+0000 7f6343833740 1 --2- 172.21.15.31:0/3173607164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03504d0 0x5626e0711d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.884+0000 7f6343833740 1 -- 172.21.15.31:0/3173607164 >> 172.21.15.31:0/3173607164 conn(0x5626e036d4e0 msgr2=0x5626e0720080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.884+0000 7f6343833740 1 -- 172.21.15.31:0/3173607164 shutdown_connections 2024-04-25T16:05:41.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.885+0000 7f6343833740 1 -- 172.21.15.31:0/3173607164 wait complete. 2024-04-25T16:05:41.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.889+0000 7f6343833740 1 -- start start 2024-04-25T16:05:41.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.890+0000 7f6343833740 1 Processor -- start 2024-04-25T16:05:41.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.891+0000 7f6343833740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03504d0 0x5626e072b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.891+0000 7f6343833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e072bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.892+0000 7f6343833740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e0799a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.892+0000 7f6343833740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f631800fa00 con 0x5626e0379390 2024-04-25T16:05:41.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.892+0000 7f6343833740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f631800f330 con 0x5626e03504d0 2024-04-25T16:05:41.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.892+0000 7f6343833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f631800c570 con 0x5626e03586b0 2024-04-25T16:05:41.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.892+0000 7f6336d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e0799a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.893+0000 7f6343833740 10 client.0 _ll_register_callbacks cb 0x7f63280a7420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:41.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.896+0000 7f6336d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e0799a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38634/0 (socket says 172.21.15.31:38634) 2024-04-25T16:05:41.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.896+0000 7f6336d9d640 1 -- 172.21.15.31:0/875893496 learned_addr learned my addr 172.21.15.31:0/875893496 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:41.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.896+0000 7f6335d9b640 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e072bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.896+0000 7f633659c640 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03504d0 0x5626e072b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.903+0000 7f6336d9d640 1 -- 172.21.15.31:0/875893496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 msgr2=0x5626e072bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.903+0000 7f6336d9d640 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e072bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.903+0000 7f6336d9d640 1 -- 172.21.15.31:0/875893496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03504d0 msgr2=0x5626e072b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:41.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.903+0000 7f6336d9d640 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03504d0 0x5626e072b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.903+0000 7f6336d9d640 1 -- 172.21.15.31:0/875893496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5626e0799f90 con 0x5626e0379390 2024-04-25T16:05:41.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.917+0000 7f6336d9d640 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e0799a50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6330034040 tx=0x7f633002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77063338943127 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:41.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.918+0000 7f631dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:41.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.925+0000 7f631dffb640 1 -- 172.21.15.31:0/875893496 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633004d070 con 0x5626e0379390 2024-04-25T16:05:41.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.925+0000 7f631dffb640 1 -- 172.21.15.31:0/875893496 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f633003cbb0 con 0x5626e0379390 2024-04-25T16:05:41.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.925+0000 7f6343833740 1 -- 172.21.15.31:0/875893496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63280ab990 con 0x5626e0379390 2024-04-25T16:05:41.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.927+0000 7f631dffb640 1 -- 172.21.15.31:0/875893496 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6330043040 con 0x5626e0379390 2024-04-25T16:05:41.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.929+0000 7f6343833740 10 client.16503 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:41.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.929+0000 7f6343833740 1 -- 172.21.15.31:0/875893496 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5626e079b1e0 con 0x5626e0379390 2024-04-25T16:05:41.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.930+0000 7f631dffb640 1 -- 172.21.15.31:0/875893496 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63280ab990 con 0x5626e0379390 2024-04-25T16:05:41.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.932+0000 7f631dffb640 1 -- 172.21.15.31:0/875893496 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6330049440 con 0x5626e0379390 2024-04-25T16:05:41.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.932+0000 7f6343833740 10 client.16503 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:41.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.932+0000 7f6343833740 10 client.16503 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:05:41.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.932+0000 7f6343833740 20 client.16503 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:41.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.932+0000 7f6343833740 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5626e030d3a0 0x5626e079db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:41.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.932+0000 7f6343833740 4 client.16503 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:41.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.932+0000 7f6343833740 1 -- 172.21.15.31:0/875893496 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5626e0687800 con 0x5626e030d3a0 2024-04-25T16:05:41.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.933+0000 7f633659c640 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5626e030d3a0 0x5626e079db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:41.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.953+0000 7f631dffb640 10 client.16503 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:05:41.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.953+0000 7f633659c640 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5626e030d3a0 0x5626e079db80 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6c56aa812a4f03a9 server_cookie=5734a763a6e1eef5 in_seq=0 out_seq=0 2024-04-25T16:05:41.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.959+0000 7f631dffb640 1 -- 172.21.15.31:0/875893496 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5626e0687800 con 0x5626e030d3a0 2024-04-25T16:05:41.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.959+0000 7f631dffb640 10 client.16503 handle_command_reply: tid=0 2024-04-25T16:05:41.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.959+0000 7f6343833740 1 client.16503 shutdown 2024-04-25T16:05:41.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.962+0000 7f6343833740 1 -- 172.21.15.31:0/875893496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 msgr2=0x5626e0799a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.962+0000 7f6343833740 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e0799a50 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f6330034040 tx=0x7f633002a710 comp rx=0 tx=0).stop 2024-04-25T16:05:41.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.962+0000 7f6343833740 1 -- 172.21.15.31:0/875893496 shutdown_connections 2024-04-25T16:05:41.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.962+0000 7f6343833740 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5626e0379390 0x5626e0799a50 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.962+0000 7f6343833740 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5626e03586b0 0x5626e072bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.962+0000 7f6343833740 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5626e030d3a0 0x5626e079db80 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.962+0000 7f6343833740 1 --2- 172.21.15.31:0/875893496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5626e03504d0 0x5626e072b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.962+0000 7f6343833740 1 -- 172.21.15.31:0/875893496 >> 172.21.15.31:0/875893496 conn(0x5626e036d4e0 msgr2=0x5626e0719e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.962+0000 7f631dffb640 0 client.16503 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:05:41.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.966+0000 7f6343833740 1 -- 172.21.15.31:0/875893496 shutdown_connections 2024-04-25T16:05:41.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.966+0000 7f6343833740 1 -- 172.21.15.31:0/875893496 wait complete. 2024-04-25T16:05:41.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.966+0000 7f6343833740 20 client.16503 trim_cache size 0 max 16384 2024-04-25T16:05:41.968 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 -- 172.21.15.31:0/1173900941 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f63200afbf0 msgr2=0x7f63200b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f63200afbf0 0x7f63200b2100 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6330009b30 tx=0x7f63300303a0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 -- 172.21.15.31:0/1173900941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f633810b310 msgr2=0x7f63381a9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f633810b310 0x7f63381a9530 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f632800e9f0 tx=0x7f632800eeb0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 -- 172.21.15.31:0/1173900941 shutdown_connections 2024-04-25T16:05:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f63200afbf0 0x7f63200b2100 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6338110030 0x7f63381b04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f633810b310 0x7f63381a9530 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 --2- 172.21.15.31:0/1173900941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f633810a420 0x7f63381a8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:41.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.970+0000 7f633c8aa640 1 -- 172.21.15.31:0/1173900941 >> 172.21.15.31:0/1173900941 conn(0x7f63380704e0 msgr2=0x7f633810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:41.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.974+0000 7f633c8aa640 1 -- 172.21.15.31:0/1173900941 shutdown_connections 2024-04-25T16:05:41.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:41.974+0000 7f633c8aa640 1 -- 172.21.15.31:0/1173900941 wait complete. 2024-04-25T16:05:42.018 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:05:42.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.534+0000 7f5974bf3640 1 -- 172.21.15.31:0/818780769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5970109fc0 msgr2=0x7f597010a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:42.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.534+0000 7f5974bf3640 1 --2- 172.21.15.31:0/818780769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5970109fc0 0x7f597010a3d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5960009900 tx=0x7f596002a420 comp rx=0 tx=0).stop 2024-04-25T16:05:42.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.534+0000 7f5974bf3640 1 -- 172.21.15.31:0/818780769 shutdown_connections 2024-04-25T16:05:42.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.534+0000 7f5974bf3640 1 --2- 172.21.15.31:0/818780769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59701120c0 0x7f5970114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.534+0000 7f5974bf3640 1 --2- 172.21.15.31:0/818780769 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f597010f760 0x7f5970111b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.534+0000 7f5974bf3640 1 --2- 172.21.15.31:0/818780769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5970109fc0 0x7f597010a3d0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.534+0000 7f5974bf3640 1 -- 172.21.15.31:0/818780769 >> 172.21.15.31:0/818780769 conn(0x7f59700704e0 msgr2=0x7f5970072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:42.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.539+0000 7f5974bf3640 1 -- 172.21.15.31:0/818780769 shutdown_connections 2024-04-25T16:05:42.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.540+0000 7f5974bf3640 1 -- 172.21.15.31:0/818780769 wait complete. 2024-04-25T16:05:42.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.542+0000 7f5974bf3640 1 Processor -- start 2024-04-25T16:05:42.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.542+0000 7f5974bf3640 1 -- start start 2024-04-25T16:05:42.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.542+0000 7f5974bf3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5970109fc0 0x7f59701a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.542+0000 7f5974bf3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f597010f760 0x7f59701a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.542+0000 7f5974bf3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59701120c0 0x7f59701a7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.542+0000 7f5974bf3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f597011a120 con 0x7f597010f760 2024-04-25T16:05:42.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.542+0000 7f5974bf3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5970119fa0 con 0x7f5970109fc0 2024-04-25T16:05:42.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.542+0000 7f5974bf3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f597011a2a0 con 0x7f59701120c0 2024-04-25T16:05:42.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.543+0000 7f596ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59701120c0 0x7f59701a7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.543+0000 7f596ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59701120c0 0x7f59701a7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56504/0 (socket says 172.21.15.31:56504) 2024-04-25T16:05:42.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.543+0000 7f596ed9d640 1 -- 172.21.15.31:0/902732116 learned_addr learned my addr 172.21.15.31:0/902732116 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:42.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.543+0000 7f596e59c640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5970109fc0 0x7f59701a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.544+0000 7f596e59c640 1 -- 172.21.15.31:0/902732116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59701120c0 msgr2=0x7f59701a7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:42.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.544+0000 7f596e59c640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59701120c0 0x7f59701a7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.544+0000 7f596e59c640 1 -- 172.21.15.31:0/902732116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f597010f760 msgr2=0x7f59701a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:42.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.544+0000 7f596e59c640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f597010f760 0x7f59701a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.544+0000 7f596e59c640 1 -- 172.21.15.31:0/902732116 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59701a8470 con 0x7f5970109fc0 2024-04-25T16:05:42.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.547+0000 7f596e59c640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5970109fc0 0x7f59701a0900 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5960009850 tx=0x7f5960003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.548+0000 7f595f7fe640 1 -- 172.21.15.31:0/902732116 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596003f070 con 0x7f5970109fc0 2024-04-25T16:05:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.549+0000 7f5974bf3640 1 -- 172.21.15.31:0/902732116 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59701a8700 con 0x7f5970109fc0 2024-04-25T16:05:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.549+0000 7f595f7fe640 1 -- 172.21.15.31:0/902732116 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5960031c50 con 0x7f5970109fc0 2024-04-25T16:05:42.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.549+0000 7f5974bf3640 1 -- 172.21.15.31:0/902732116 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59701a8c70 con 0x7f5970109fc0 2024-04-25T16:05:42.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.549+0000 7f595f7fe640 1 -- 172.21.15.31:0/902732116 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5960035030 con 0x7f5970109fc0 2024-04-25T16:05:42.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.559+0000 7f597bb81740 1 Processor -- start 2024-04-25T16:05:42.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.559+0000 7f597bb81740 1 -- start start 2024-04-25T16:05:42.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.559+0000 7f597bb81740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 0x564129d02810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.561+0000 7f597bb81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641299374d0 0x564129d02d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.561+0000 7f597bb81740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 0x564129d03a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.561+0000 7f597bb81740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564129c66d80 con 0x56412993f6b0 2024-04-25T16:05:42.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.561+0000 7f597bb81740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564129c20a20 con 0x564129960390 2024-04-25T16:05:42.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.561+0000 7f597bb81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564129c62d40 con 0x5641299374d0 2024-04-25T16:05:42.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.561+0000 7f596ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 0x564129d02810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.562+0000 7f595f7fe640 1 -- 172.21.15.31:0/902732116 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5960042070 con 0x7f5970109fc0 2024-04-25T16:05:42.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.562+0000 7f596ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 0x564129d02810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52582/0 (socket says 172.21.15.31:52582) 2024-04-25T16:05:42.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.562+0000 7f596ed9d640 1 -- 172.21.15.31:0/2802277645 learned_addr learned my addr 172.21.15.31:0/2802277645 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:42.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.564+0000 7f596e59c640 1 --2- 172.21.15.31:0/2802277645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641299374d0 0x564129d02d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.564+0000 7f595f7fe640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59580afd20 0x7f59580b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.565+0000 7f595f7fe640 1 -- 172.21.15.31:0/902732116 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f59600ff3d0 con 0x7f5970109fc0 2024-04-25T16:05:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.566+0000 7f596e59c640 1 -- 172.21.15.31:0/2802277645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 msgr2=0x564129d02810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.566+0000 7f596e59c640 1 --2- 172.21.15.31:0/2802277645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 0x564129d02810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.566+0000 7f596e59c640 1 -- 172.21.15.31:0/2802277645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 msgr2=0x564129d03a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.566+0000 7f596e59c640 1 --2- 172.21.15.31:0/2802277645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 0x564129d03a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.566+0000 7f596e59c640 1 -- 172.21.15.31:0/2802277645 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564129c2bc30 con 0x5641299374d0 2024-04-25T16:05:42.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.567+0000 7f596e59c640 1 --2- 172.21.15.31:0/2802277645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641299374d0 0x564129d02d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5960006010 tx=0x7f59600c76f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63350a98d1c5582 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:42.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.568+0000 7f595cff9640 1 -- 172.21.15.31:0/2802277645 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5960049070 con 0x5641299374d0 2024-04-25T16:05:42.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.569+0000 7f595cff9640 1 -- 172.21.15.31:0/2802277645 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59600c7f60 con 0x5641299374d0 2024-04-25T16:05:42.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.569+0000 7f595cff9640 1 -- 172.21.15.31:0/2802277645 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59600ea0d0 con 0x5641299374d0 2024-04-25T16:05:42.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.571+0000 7f596ed9d640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59580afd20 0x7f59580b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.573+0000 7f597bb81740 1 -- 172.21.15.31:0/2802277645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641299374d0 msgr2=0x564129d02d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:42.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.573+0000 7f597bb81740 1 --2- 172.21.15.31:0/2802277645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641299374d0 0x564129d02d90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5960006010 tx=0x7f59600c76f0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.577+0000 7f597bb81740 1 -- 172.21.15.31:0/2802277645 shutdown_connections 2024-04-25T16:05:42.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.577+0000 7f597bb81740 1 --2- 172.21.15.31:0/2802277645 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 0x564129d03a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.577+0000 7f597bb81740 1 --2- 172.21.15.31:0/2802277645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5641299374d0 0x564129d02d90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.577+0000 7f597bb81740 1 --2- 172.21.15.31:0/2802277645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 0x564129d02810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.577+0000 7f597bb81740 1 -- 172.21.15.31:0/2802277645 >> 172.21.15.31:0/2802277645 conn(0x5641299544e0 msgr2=0x564129cfe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:42.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.577+0000 7f597bb81740 1 -- 172.21.15.31:0/2802277645 shutdown_connections 2024-04-25T16:05:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.581+0000 7f597bb81740 1 -- 172.21.15.31:0/2802277645 wait complete. 2024-04-25T16:05:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.582+0000 7f597bb81740 1 -- start start 2024-04-25T16:05:42.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.582+0000 7f597bb81740 1 Processor -- start 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.582+0000 7f597bb81740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641299374d0 0x564129d0c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.582+0000 7f597bb81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56412993f6b0 0x564129d0c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.583+0000 7f597bb81740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 0x564129d136f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.583+0000 7f597bb81740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564129c62d40 con 0x5641299374d0 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.583+0000 7f597bb81740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564129beddc0 con 0x564129960390 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.583+0000 7f597bb81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564129d042f0 con 0x56412993f6b0 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.583+0000 7f597bb81740 10 client.0 _ll_register_callbacks cb 0x564129c09600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.583+0000 7f596dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56412993f6b0 0x564129d0c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.583+0000 7f596dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56412993f6b0 0x564129d0c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56520/0 (socket says 172.21.15.31:56520) 2024-04-25T16:05:42.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.583+0000 7f596dd9b640 1 -- 172.21.15.31:0/3309114877 learned_addr learned my addr 172.21.15.31:0/3309114877 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:42.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.584+0000 7f596dd9b640 1 -- 172.21.15.31:0/3309114877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 msgr2=0x564129d136f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.584+0000 7f596dd9b640 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 0x564129d136f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.584+0000 7f596dd9b640 1 -- 172.21.15.31:0/3309114877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641299374d0 msgr2=0x564129d0c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.584+0000 7f596dd9b640 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641299374d0 0x564129d0c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.584+0000 7f596dd9b640 1 -- 172.21.15.31:0/3309114877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5641299783f0 con 0x56412993f6b0 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.584+0000 7f596ed9d640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59580afd20 0x7f59580b2230 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x564129d03860 tx=0x7f5968004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.585+0000 7f596dd9b640 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56412993f6b0 0x564129d0c790 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5964009e20 tx=0x7f596400b4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f959da0f0eabea7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.585+0000 7f5955ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.585+0000 7f5955ffb640 1 -- 172.21.15.31:0/3309114877 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964015020 con 0x56412993f6b0 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.586+0000 7f597bb81740 1 -- 172.21.15.31:0/3309114877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f59600c7f60 con 0x56412993f6b0 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.586+0000 7f5955ffb640 1 -- 172.21.15.31:0/3309114877 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5964011620 con 0x56412993f6b0 2024-04-25T16:05:42.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.590+0000 7f5955ffb640 1 -- 172.21.15.31:0/3309114877 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5964011860 con 0x56412993f6b0 2024-04-25T16:05:42.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.591+0000 7f597bb81740 10 client.26149 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:42.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.591+0000 7f597bb81740 1 -- 172.21.15.31:0/3309114877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564129a427d0 con 0x56412993f6b0 2024-04-25T16:05:42.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.592+0000 7f5955ffb640 1 -- 172.21.15.31:0/3309114877 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f59600c7f60 con 0x56412993f6b0 2024-04-25T16:05:42.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.592+0000 7f5955ffb640 1 -- 172.21.15.31:0/3309114877 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f596401cb20 con 0x56412993f6b0 2024-04-25T16:05:42.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.592+0000 7f597bb81740 10 client.26149 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:42.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.592+0000 7f597bb81740 10 client.26149 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:05:42.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.592+0000 7f597bb81740 20 client.26149 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:42.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.592+0000 7f597bb81740 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56412991ef60 0x564129d1a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.592+0000 7f597bb81740 4 client.26149 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:42.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.592+0000 7f597bb81740 1 -- 172.21.15.31:0/3309114877 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564129c74990 con 0x56412991ef60 2024-04-25T16:05:42.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.596+0000 7f596e59c640 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56412991ef60 0x564129d1a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.602+0000 7f596e59c640 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56412991ef60 0x564129d1a5a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=29e6fa554ede89d9 server_cookie=d4bb422fd1031c93 in_seq=0 out_seq=0 2024-04-25T16:05:42.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.603+0000 7f5955ffb640 10 client.26149 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:05:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.604+0000 7f5955ffb640 1 -- 172.21.15.31:0/3309114877 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564129c74990 con 0x56412991ef60 2024-04-25T16:05:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.604+0000 7f5955ffb640 10 client.26149 handle_command_reply: tid=0 2024-04-25T16:05:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.604+0000 7f597bb81740 1 client.26149 shutdown 2024-04-25T16:05:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.605+0000 7f597bb81740 1 -- 172.21.15.31:0/3309114877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56412993f6b0 msgr2=0x564129d0c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:42.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.605+0000 7f597bb81740 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56412993f6b0 0x564129d0c790 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5964009e20 tx=0x7f596400b4b0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.611+0000 7f597bb81740 1 -- 172.21.15.31:0/3309114877 shutdown_connections 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.611+0000 7f597bb81740 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564129960390 0x564129d136f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.611+0000 7f597bb81740 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56412993f6b0 0x564129d0c790 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.611+0000 7f597bb81740 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56412991ef60 0x564129d1a5a0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.611+0000 7f597bb81740 1 --2- 172.21.15.31:0/3309114877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641299374d0 0x564129d0c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.611+0000 7f597bb81740 1 -- 172.21.15.31:0/3309114877 >> 172.21.15.31:0/3309114877 conn(0x5641299544e0 msgr2=0x564129cfd040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.611+0000 7f5955ffb640 0 client.26149 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.612+0000 7f597bb81740 1 -- 172.21.15.31:0/3309114877 shutdown_connections 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.612+0000 7f597bb81740 1 -- 172.21.15.31:0/3309114877 wait complete. 2024-04-25T16:05:42.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.612+0000 7f597bb81740 20 client.26149 trim_cache size 0 max 16384 2024-04-25T16:05:42.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.643+0000 7f597bb81740 1 Processor -- start 2024-04-25T16:05:42.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.643+0000 7f597bb81740 1 -- start start 2024-04-25T16:05:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.647+0000 7f597bb81740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641299374d0 0x564129d0a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.647+0000 7f597bb81740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56412993f6b0 0x564129d0adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.647+0000 7f597bb81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d13610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.647+0000 7f597bb81740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f595000e500 con 0x5641299374d0 2024-04-25T16:05:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.647+0000 7f597bb81740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5950010040 con 0x56412993f6b0 2024-04-25T16:05:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.647+0000 7f597bb81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f595000ebd0 con 0x564129960390 2024-04-25T16:05:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.648+0000 7f596ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d13610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.649+0000 7f596ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d13610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56536/0 (socket says 172.21.15.31:56536) 2024-04-25T16:05:42.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.649+0000 7f596ed9d640 1 -- 172.21.15.31:0/1034835689 learned_addr learned my addr 172.21.15.31:0/1034835689 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:42.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.649+0000 7f596e59c640 1 --2- 172.21.15.31:0/1034835689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641299374d0 0x564129d0a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.650+0000 7f596ed9d640 1 -- 172.21.15.31:0/1034835689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56412993f6b0 msgr2=0x564129d0adb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:05:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.650+0000 7f596ed9d640 1 --2- 172.21.15.31:0/1034835689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56412993f6b0 0x564129d0adb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.650+0000 7f596ed9d640 1 -- 172.21.15.31:0/1034835689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641299374d0 msgr2=0x564129d0a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.650+0000 7f596ed9d640 1 --2- 172.21.15.31:0/1034835689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641299374d0 0x564129d0a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.650+0000 7f596ed9d640 1 -- 172.21.15.31:0/1034835689 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564129d13b50 con 0x564129960390 2024-04-25T16:05:42.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.652+0000 7f596ed9d640 1 --2- 172.21.15.31:0/1034835689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d13610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f596800f7f0 tx=0x7f5968016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=504b55ba42a6905f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:42.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.656+0000 7f5957fff640 1 -- 172.21.15.31:0/1034835689 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596802b070 con 0x564129960390 2024-04-25T16:05:42.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.656+0000 7f5957fff640 1 -- 172.21.15.31:0/1034835689 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f596800fdd0 con 0x564129960390 2024-04-25T16:05:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.658+0000 7f597bb81740 1 -- 172.21.15.31:0/1034835689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 msgr2=0x564129d13610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:42.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.658+0000 7f597bb81740 1 --2- 172.21.15.31:0/1034835689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d13610 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f596800f7f0 tx=0x7f5968016000 comp rx=0 tx=0).stop 2024-04-25T16:05:42.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.663+0000 7f597bb81740 1 -- 172.21.15.31:0/1034835689 shutdown_connections 2024-04-25T16:05:42.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.663+0000 7f597bb81740 1 --2- 172.21.15.31:0/1034835689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d13610 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.663+0000 7f597bb81740 1 --2- 172.21.15.31:0/1034835689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56412993f6b0 0x564129d0adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.663+0000 7f597bb81740 1 --2- 172.21.15.31:0/1034835689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5641299374d0 0x564129d0a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.663+0000 7f597bb81740 1 -- 172.21.15.31:0/1034835689 >> 172.21.15.31:0/1034835689 conn(0x5641299544e0 msgr2=0x564129d07130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.664+0000 7f597bb81740 1 -- 172.21.15.31:0/1034835689 shutdown_connections 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.665+0000 7f597bb81740 1 -- 172.21.15.31:0/1034835689 wait complete. 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.666+0000 7f597bb81740 1 -- start start 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.666+0000 7f597bb81740 1 Processor -- start 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.667+0000 7f597bb81740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641299374d0 0x564129d13350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.667+0000 7f597bb81740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 0x564129d0ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.667+0000 7f597bb81740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d10480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.667+0000 7f597bb81740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f595000f970 con 0x56412993f6b0 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.667+0000 7f597bb81740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f595000f2a0 con 0x5641299374d0 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.667+0000 7f597bb81740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f595000c4e0 con 0x564129960390 2024-04-25T16:05:42.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.667+0000 7f597bb81740 10 client.0 _ll_register_callbacks cb 0x7f5964006ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.668+0000 7f596dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 0x564129d0ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.668+0000 7f596dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 0x564129d0ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38674/0 (socket says 172.21.15.31:38674) 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.668+0000 7f596dd9b640 1 -- 172.21.15.31:0/3542727358 learned_addr learned my addr 172.21.15.31:0/3542727358 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.668+0000 7f596ed9d640 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d10480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.668+0000 7f596dd9b640 1 -- 172.21.15.31:0/3542727358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 msgr2=0x564129d10480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.668+0000 7f596dd9b640 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d10480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.668+0000 7f596dd9b640 1 -- 172.21.15.31:0/3542727358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641299374d0 msgr2=0x564129d13350 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.668+0000 7f596dd9b640 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641299374d0 0x564129d13350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.669+0000 7f596dd9b640 1 -- 172.21.15.31:0/3542727358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564129d10ac0 con 0x56412993f6b0 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.669+0000 7f596dd9b640 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 0x564129d0ff40 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5964001eb0 tx=0x7f5964019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84e4f249122bb7f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.669+0000 7f5955ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.670+0000 7f5955ffb640 1 -- 172.21.15.31:0/3542727358 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964029020 con 0x56412993f6b0 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.670+0000 7f597bb81740 1 -- 172.21.15.31:0/3542727358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f596800fdd0 con 0x56412993f6b0 2024-04-25T16:05:42.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.670+0000 7f5955ffb640 1 -- 172.21.15.31:0/3542727358 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5964011a00 con 0x56412993f6b0 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.671+0000 7f5955ffb640 1 -- 172.21.15.31:0/3542727358 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5964015040 con 0x56412993f6b0 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.671+0000 7f597bb81740 10 client.16509 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.671+0000 7f597bb81740 1 -- 172.21.15.31:0/3542727358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564129d81a40 con 0x56412993f6b0 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.671+0000 7f5955ffb640 1 -- 172.21.15.31:0/3542727358 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f596800fdd0 con 0x56412993f6b0 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.672+0000 7f5955ffb640 1 -- 172.21.15.31:0/3542727358 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f596401c420 con 0x56412993f6b0 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.672+0000 7f597bb81740 10 client.16509 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.672+0000 7f597bb81740 10 client.16509 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.672+0000 7f597bb81740 20 client.16509 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.672+0000 7f597bb81740 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5641298f43a0 0x564129d84460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.672+0000 7f597bb81740 4 client.16509 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:42.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.672+0000 7f597bb81740 1 -- 172.21.15.31:0/3542727358 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564129c6e800 con 0x5641298f43a0 2024-04-25T16:05:42.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.678+0000 7f596e59c640 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5641298f43a0 0x564129d84460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.684+0000 7f596e59c640 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5641298f43a0 0x564129d84460 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a1599202799ded45 server_cookie=94e69dbd8d63f58e in_seq=0 out_seq=0 2024-04-25T16:05:42.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.684+0000 7f5955ffb640 10 client.16509 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:05:42.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.686+0000 7f5955ffb640 1 -- 172.21.15.31:0/3542727358 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564129c6e800 con 0x5641298f43a0 2024-04-25T16:05:42.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.686+0000 7f5955ffb640 10 client.16509 handle_command_reply: tid=0 2024-04-25T16:05:42.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.686+0000 7f597bb81740 1 client.16509 shutdown 2024-04-25T16:05:42.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.687+0000 7f597bb81740 1 -- 172.21.15.31:0/3542727358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 msgr2=0x564129d0ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:42.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.687+0000 7f597bb81740 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 0x564129d0ff40 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f5964001eb0 tx=0x7f5964019740 comp rx=0 tx=0).stop 2024-04-25T16:05:42.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.687+0000 7f597bb81740 1 -- 172.21.15.31:0/3542727358 shutdown_connections 2024-04-25T16:05:42.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.687+0000 7f597bb81740 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564129960390 0x564129d10480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.687+0000 7f597bb81740 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5641298f43a0 0x564129d84460 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.687+0000 7f597bb81740 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56412993f6b0 0x564129d0ff40 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.687+0000 7f597bb81740 1 --2- 172.21.15.31:0/3542727358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5641299374d0 0x564129d13350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.687+0000 7f597bb81740 1 -- 172.21.15.31:0/3542727358 >> 172.21.15.31:0/3542727358 conn(0x5641299544e0 msgr2=0x564129d0a7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:42.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.688+0000 7f5955ffb640 0 client.16509 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:05:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.688+0000 7f597bb81740 1 -- 172.21.15.31:0/3542727358 shutdown_connections 2024-04-25T16:05:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.691+0000 7f597bb81740 1 -- 172.21.15.31:0/3542727358 wait complete. 2024-04-25T16:05:42.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.692+0000 7f597bb81740 20 client.16509 trim_cache size 0 max 16384 2024-04-25T16:05:42.693 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:42.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 -- 172.21.15.31:0/902732116 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59580afd20 msgr2=0x7f59580b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:42.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59580afd20 0x7f59580b2230 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x564129d03860 tx=0x7f5968004a40 comp rx=0 tx=0).stop 2024-04-25T16:05:42.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 -- 172.21.15.31:0/902732116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5970109fc0 msgr2=0x7f59701a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:42.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5970109fc0 0x7f59701a0900 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5960009850 tx=0x7f5960003b20 comp rx=0 tx=0).stop 2024-04-25T16:05:42.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 -- 172.21.15.31:0/902732116 shutdown_connections 2024-04-25T16:05:42.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f59580afd20 0x7f59580b2230 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59701120c0 0x7f59701a7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f597010f760 0x7f59701a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 --2- 172.21.15.31:0/902732116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5970109fc0 0x7f59701a0900 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:42.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 -- 172.21.15.31:0/902732116 >> 172.21.15.31:0/902732116 conn(0x7f59700704e0 msgr2=0x7f5970110740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:42.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 -- 172.21.15.31:0/902732116 shutdown_connections 2024-04-25T16:05:42.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:42.694+0000 7f5974bf3640 1 -- 172.21.15.31:0/902732116 wait complete. 2024-04-25T16:05:42.709 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:05:43.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.019+0000 7fe17cd14640 1 -- 172.21.15.31:0/1403024036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17810b1d0 msgr2=0x7fe17810b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:43.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.019+0000 7fe17cd14640 1 --2- 172.21.15.31:0/1403024036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17810b1d0 0x7fe17810b5e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe170009ca0 tx=0x7fe17002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.020+0000 7fe17cd14640 1 -- 172.21.15.31:0/1403024036 shutdown_connections 2024-04-25T16:05:43.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.020+0000 7fe17cd14640 1 --2- 172.21.15.31:0/1403024036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe17810fef0 0x7fe178110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.020+0000 7fe17cd14640 1 --2- 172.21.15.31:0/1403024036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17810b1d0 0x7fe17810b5e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.020+0000 7fe17cd14640 1 --2- 172.21.15.31:0/1403024036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe17810a3e0 0x7fe17810ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.020+0000 7fe17cd14640 1 -- 172.21.15.31:0/1403024036 >> 172.21.15.31:0/1403024036 conn(0x7fe178070400 msgr2=0x7fe178072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:43.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.020+0000 7fe17cd14640 1 -- 172.21.15.31:0/1403024036 shutdown_connections 2024-04-25T16:05:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.020+0000 7fe17cd14640 1 -- 172.21.15.31:0/1403024036 wait complete. 2024-04-25T16:05:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.022+0000 7fe17cd14640 1 Processor -- start 2024-04-25T16:05:43.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.022+0000 7fe17cd14640 1 -- start start 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.023+0000 7fe17cd14640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe17810a3e0 0x7fe178116fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.023+0000 7fe17cd14640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17810b1d0 0x7fe1781174f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.023+0000 7fe17cd14640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe17810fef0 0x7fe178118450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.023+0000 7fe17cd14640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe17811e2e0 con 0x7fe17810fef0 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.023+0000 7fe17cd14640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe17811e160 con 0x7fe17810a3e0 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.023+0000 7fe17cd14640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe17811e460 con 0x7fe17810b1d0 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.026+0000 7fe176d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe17810fef0 0x7fe178118450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.026+0000 7fe176d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe17810fef0 0x7fe178118450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38684/0 (socket says 172.21.15.31:38684) 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.026+0000 7fe176d9d640 1 -- 172.21.15.31:0/2701416862 learned_addr learned my addr 172.21.15.31:0/2701416862 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.027+0000 7fe176d9d640 1 -- 172.21.15.31:0/2701416862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17810b1d0 msgr2=0x7fe1781174f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.027+0000 7fe176d9d640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17810b1d0 0x7fe1781174f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.027+0000 7fe176d9d640 1 -- 172.21.15.31:0/2701416862 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe17810a3e0 msgr2=0x7fe178116fb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.027+0000 7fe176d9d640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe17810a3e0 0x7fe178116fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.027+0000 7fe176d9d640 1 -- 172.21.15.31:0/2701416862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe178118ac0 con 0x7fe17810fef0 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.028+0000 7fe176d9d640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe17810fef0 0x7fe178118450 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe16800c970 tx=0x7fe16800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:43.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.029+0000 7fe1677fe640 1 -- 172.21.15.31:0/2701416862 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168007bb0 con 0x7fe17810fef0 2024-04-25T16:05:43.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.029+0000 7fe1677fe640 1 -- 172.21.15.31:0/2701416862 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe168007d50 con 0x7fe17810fef0 2024-04-25T16:05:43.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.030+0000 7fe1677fe640 1 -- 172.21.15.31:0/2701416862 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16800f450 con 0x7fe17810fef0 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.032+0000 7fe17cd14640 1 -- 172.21.15.31:0/2701416862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe178118db0 con 0x7fe17810fef0 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.032+0000 7fe17cd14640 1 -- 172.21.15.31:0/2701416862 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe178119280 con 0x7fe17810fef0 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.045+0000 7fe183bc7740 1 Processor -- start 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.046+0000 7fe183bc7740 1 -- start start 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.047+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 0x560cd9d7c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.047+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 0x560cd9d7cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.047+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9d7dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.047+0000 7fe183bc7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560cd9ce0d80 con 0x560cd99da390 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.047+0000 7fe183bc7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560cd9c9aa20 con 0x560cd99b96b0 2024-04-25T16:05:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.047+0000 7fe183bc7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560cd9cdcd40 con 0x560cd99b14d0 2024-04-25T16:05:43.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.049+0000 7fe17659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 0x560cd9d7cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.049+0000 7fe17659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 0x560cd9d7cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56554/0 (socket says 172.21.15.31:56554) 2024-04-25T16:05:43.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.049+0000 7fe17659c640 1 -- 172.21.15.31:0/552798544 learned_addr learned my addr 172.21.15.31:0/552798544 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:43.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.055+0000 7fe175d9b640 1 --2- 172.21.15.31:0/552798544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9d7dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.055+0000 7fe175d9b640 1 --2- 172.21.15.31:0/552798544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 0x560cd9d7c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.055+0000 7fe1677fe640 1 -- 172.21.15.31:0/2701416862 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe168004e90 con 0x7fe17810fef0 2024-04-25T16:05:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.056+0000 7fe175d9b640 1 -- 172.21.15.31:0/552798544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 msgr2=0x560cd9d7cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.056+0000 7fe175d9b640 1 --2- 172.21.15.31:0/552798544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 0x560cd9d7cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.056+0000 7fe175d9b640 1 -- 172.21.15.31:0/552798544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 msgr2=0x560cd9d7dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.056+0000 7fe175d9b640 1 --2- 172.21.15.31:0/552798544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9d7dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.056+0000 7fe175d9b640 1 -- 172.21.15.31:0/552798544 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cd9ca5c30 con 0x560cd99da390 2024-04-25T16:05:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.058+0000 7fe1677fe640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1600d1be0 0x7fe1600d40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.058+0000 7fe175d9b640 1 --2- 172.21.15.31:0/552798544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 0x560cd9d7c9e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe170009bf0 tx=0x7fe170005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66d63e5d5bd94f9c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:43.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.059+0000 7fe1677fe640 1 -- 172.21.15.31:0/2701416862 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe1680deb90 con 0x7fe17810fef0 2024-04-25T16:05:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.061+0000 7fe164ff9640 1 -- 172.21.15.31:0/552798544 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170031680 con 0x560cd99da390 2024-04-25T16:05:43.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.061+0000 7fe164ff9640 1 -- 172.21.15.31:0/552798544 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe170036040 con 0x560cd99da390 2024-04-25T16:05:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.062+0000 7fe17659c640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1600d1be0 0x7fe1600d40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.063+0000 7fe183bc7740 1 -- 172.21.15.31:0/552798544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 msgr2=0x560cd9d7c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:43.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.063+0000 7fe183bc7740 1 --2- 172.21.15.31:0/552798544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 0x560cd9d7c9e0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fe170009bf0 tx=0x7fe170005fd0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.063+0000 7fe17659c640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1600d1be0 0x7fe1600d40f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x560cd9d7c700 tx=0x7fe16c0090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.063+0000 7fe183bc7740 1 -- 172.21.15.31:0/552798544 shutdown_connections 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.063+0000 7fe183bc7740 1 --2- 172.21.15.31:0/552798544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9d7dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.063+0000 7fe183bc7740 1 --2- 172.21.15.31:0/552798544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 0x560cd9d7cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.063+0000 7fe183bc7740 1 --2- 172.21.15.31:0/552798544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 0x560cd9d7c9e0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.063+0000 7fe183bc7740 1 -- 172.21.15.31:0/552798544 >> 172.21.15.31:0/552798544 conn(0x560cd99ce4e0 msgr2=0x560cd9d781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.064+0000 7fe183bc7740 1 -- 172.21.15.31:0/552798544 shutdown_connections 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.065+0000 7fe183bc7740 1 -- 172.21.15.31:0/552798544 wait complete. 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.066+0000 7fe183bc7740 1 -- start start 2024-04-25T16:05:43.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.066+0000 7fe183bc7740 1 Processor -- start 2024-04-25T16:05:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.067+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 0x560cd9da87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.067+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9da8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.067+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 0x560cd9dafc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.068+0000 7fe183bc7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560cd9d7e6f0 con 0x560cd99da390 2024-04-25T16:05:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.068+0000 7fe183bc7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560cd9c67dc0 con 0x560cd99b96b0 2024-04-25T16:05:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.068+0000 7fe183bc7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560cd9d7e870 con 0x560cd99b14d0 2024-04-25T16:05:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.068+0000 7fe183bc7740 10 client.0 _ll_register_callbacks cb 0x560cd9c83600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.068+0000 7fe175d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9da8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.068+0000 7fe175d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9da8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:52628/0 (socket says 172.21.15.31:52628) 2024-04-25T16:05:43.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.068+0000 7fe175d9b640 1 -- 172.21.15.31:0/2560788213 learned_addr learned my addr 172.21.15.31:0/2560788213 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.069+0000 7fe175d9b640 1 -- 172.21.15.31:0/2560788213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 msgr2=0x560cd9da87f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.069+0000 7fe175d9b640 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 0x560cd9da87f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.069+0000 7fe175d9b640 1 -- 172.21.15.31:0/2560788213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 msgr2=0x560cd9dafc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.069+0000 7fe175d9b640 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 0x560cd9dafc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.069+0000 7fe175d9b640 1 -- 172.21.15.31:0/2560788213 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cd99f23f0 con 0x560cd99b96b0 2024-04-25T16:05:43.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.071+0000 7fe175d9b640 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9da8d30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe170005c40 tx=0x7fe170032d60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=873f6d0d0d258308 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.072+0000 7fe15dffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:05:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.072+0000 7fe15dffb640 1 -- 172.21.15.31:0/2560788213 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170003970 con 0x560cd99b96b0 2024-04-25T16:05:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.072+0000 7fe15dffb640 1 -- 172.21.15.31:0/2560788213 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe170003b10 con 0x560cd99b96b0 2024-04-25T16:05:43.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.072+0000 7fe183bc7740 1 -- 172.21.15.31:0/2560788213 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe170036040 con 0x560cd99b96b0 2024-04-25T16:05:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.073+0000 7fe15dffb640 1 -- 172.21.15.31:0/2560788213 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe170003d10 con 0x560cd99b96b0 2024-04-25T16:05:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.074+0000 7fe183bc7740 10 client.26213 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.074+0000 7fe183bc7740 1 -- 172.21.15.31:0/2560788213 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560cd9abc7d0 con 0x560cd99b96b0 2024-04-25T16:05:43.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.077+0000 7fe15dffb640 1 -- 172.21.15.31:0/2560788213 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe170036040 con 0x560cd99b96b0 2024-04-25T16:05:43.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.079+0000 7fe15dffb640 1 -- 172.21.15.31:0/2560788213 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe170030720 con 0x560cd99b96b0 2024-04-25T16:05:43.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.079+0000 7fe183bc7740 10 client.26213 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.079+0000 7fe183bc7740 10 client.26213 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.080+0000 7fe183bc7740 20 client.26213 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.080+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560cd9998f60 0x560cd9db60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.080+0000 7fe183bc7740 4 client.26213 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:43.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.080+0000 7fe183bc7740 1 -- 172.21.15.31:0/2560788213 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560cd9cee990 con 0x560cd9998f60 2024-04-25T16:05:43.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.084+0000 7fe17659c640 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560cd9998f60 0x560cd9db60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.101+0000 7fe17659c640 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560cd9998f60 0x560cd9db60e0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9264d316692be9b6 server_cookie=77e66acb9142c48e in_seq=0 out_seq=0 2024-04-25T16:05:43.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.101+0000 7fe15dffb640 10 client.26213 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:43.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.108+0000 7fe15dffb640 1 -- 172.21.15.31:0/2560788213 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560cd9cee990 con 0x560cd9998f60 2024-04-25T16:05:43.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.108+0000 7fe15dffb640 10 client.26213 handle_command_reply: tid=0 2024-04-25T16:05:43.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.108+0000 7fe183bc7740 1 client.26213 shutdown 2024-04-25T16:05:43.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.109+0000 7fe183bc7740 1 -- 172.21.15.31:0/2560788213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 msgr2=0x560cd9da8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:43.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.109+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9da8d30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe170005c40 tx=0x7fe170032d60 comp rx=0 tx=0).stop 2024-04-25T16:05:43.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:42 smithi063 ceph-mon[32413]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 740 MiB/s wr, 1.65k op/s 2024-04-25T16:05:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.121+0000 7fe183bc7740 1 -- 172.21.15.31:0/2560788213 shutdown_connections 2024-04-25T16:05:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.121+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560cd9998f60 0x560cd9db60e0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.121+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99da390 0x560cd9dafc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.121+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9da8d30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.121+0000 7fe15dffb640 0 client.26213 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.121+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2560788213 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b14d0 0x560cd9da87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.121+0000 7fe183bc7740 1 -- 172.21.15.31:0/2560788213 >> 172.21.15.31:0/2560788213 conn(0x560cd99ce4e0 msgr2=0x560cd9d773f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.123+0000 7fe183bc7740 1 -- 172.21.15.31:0/2560788213 shutdown_connections 2024-04-25T16:05:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.123+0000 7fe183bc7740 1 -- 172.21.15.31:0/2560788213 wait complete. 2024-04-25T16:05:43.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.125+0000 7fe183bc7740 20 client.26213 trim_cache size 0 max 16384 2024-04-25T16:05:43.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:42 smithi186 ceph-mon[33258]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 740 MiB/s wr, 1.65k op/s 2024-04-25T16:05:43.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.182+0000 7fe183bc7740 1 Processor -- start 2024-04-25T16:05:43.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.183+0000 7fe183bc7740 1 -- start start 2024-04-25T16:05:43.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.183+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9da7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.184+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b96b0 0x560cd9dafbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.184+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99da390 0x560cd9dbb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.184+0000 7fe183bc7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1781ab380 con 0x560cd99b14d0 2024-04-25T16:05:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.184+0000 7fe183bc7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe1781acec0 con 0x560cd99da390 2024-04-25T16:05:43.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.184+0000 7fe183bc7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe1781aba50 con 0x560cd99b96b0 2024-04-25T16:05:43.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.184+0000 7fe175d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b96b0 0x560cd9dafbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.184+0000 7fe175d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b96b0 0x560cd9dafbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56570/0 (socket says 172.21.15.31:56570) 2024-04-25T16:05:43.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.184+0000 7fe175d9b640 1 -- 172.21.15.31:0/2961195782 learned_addr learned my addr 172.21.15.31:0/2961195782 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:43.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.184+0000 7fe176d9d640 1 --2- 172.21.15.31:0/2961195782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99da390 0x560cd9dbb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.186+0000 7fe17659c640 1 --2- 172.21.15.31:0/2961195782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9da7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.187+0000 7fe175d9b640 1 -- 172.21.15.31:0/2961195782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99da390 msgr2=0x560cd9dbb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.187+0000 7fe175d9b640 1 --2- 172.21.15.31:0/2961195782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99da390 0x560cd9dbb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.187+0000 7fe175d9b640 1 -- 172.21.15.31:0/2961195782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 msgr2=0x560cd9da7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:43.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.187+0000 7fe175d9b640 1 --2- 172.21.15.31:0/2961195782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9da7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.187+0000 7fe175d9b640 1 -- 172.21.15.31:0/2961195782 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560cd9da9930 con 0x560cd99b96b0 2024-04-25T16:05:43.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.192+0000 7fe175d9b640 1 --2- 172.21.15.31:0/2961195782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b96b0 0x560cd9dafbb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe170000f80 tx=0x7fe170005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6256ea2be6f5a030 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:43.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.194+0000 7fe15ffff640 1 -- 172.21.15.31:0/2961195782 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170062070 con 0x560cd99b96b0 2024-04-25T16:05:43.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.194+0000 7fe15ffff640 1 -- 172.21.15.31:0/2961195782 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe170036040 con 0x560cd99b96b0 2024-04-25T16:05:43.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.197+0000 7fe183bc7740 1 -- 172.21.15.31:0/2961195782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b96b0 msgr2=0x560cd9dafbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:43.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.197+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2961195782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b96b0 0x560cd9dafbb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fe170000f80 tx=0x7fe170005fd0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.204+0000 7fe183bc7740 1 -- 172.21.15.31:0/2961195782 shutdown_connections 2024-04-25T16:05:43.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.204+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2961195782 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99da390 0x560cd9dbb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.204+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2961195782 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99b96b0 0x560cd9dafbb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.204+0000 7fe183bc7740 1 --2- 172.21.15.31:0/2961195782 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9da7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.205+0000 7fe183bc7740 1 -- 172.21.15.31:0/2961195782 >> 172.21.15.31:0/2961195782 conn(0x560cd99ce4e0 msgr2=0x560cd9d7ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:43.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.205+0000 7fe183bc7740 1 -- 172.21.15.31:0/2961195782 shutdown_connections 2024-04-25T16:05:43.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.207+0000 7fe183bc7740 1 -- 172.21.15.31:0/2961195782 wait complete. 2024-04-25T16:05:43.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.208+0000 7fe183bc7740 1 -- start start 2024-04-25T16:05:43.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.208+0000 7fe183bc7740 1 Processor -- start 2024-04-25T16:05:43.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.209+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9dad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.209+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9dad610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.209+0000 7fe183bc7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99da390 0x560cd9dae540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.209+0000 7fe183bc7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe1781ac7f0 con 0x560cd99b14d0 2024-04-25T16:05:43.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.209+0000 7fe183bc7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe1781ac120 con 0x560cd99b96b0 2024-04-25T16:05:43.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.209+0000 7fe183bc7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe1781a9410 con 0x560cd99da390 2024-04-25T16:05:43.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.209+0000 7fe183bc7740 10 client.0 _ll_register_callbacks cb 0x7fe170009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:43.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.210+0000 7fe17659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9dad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.210+0000 7fe17659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9dad0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38738/0 (socket says 172.21.15.31:38738) 2024-04-25T16:05:43.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.210+0000 7fe17659c640 1 -- 172.21.15.31:0/1319427663 learned_addr learned my addr 172.21.15.31:0/1319427663 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:43.211 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:42 smithi031 ceph-mon[29140]: pgmap v483: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 740 MiB/s wr, 1.65k op/s 2024-04-25T16:05:43.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.211+0000 7fe17659c640 1 -- 172.21.15.31:0/1319427663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99da390 msgr2=0x560cd9dae540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:43.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.211+0000 7fe17659c640 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99da390 0x560cd9dae540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.211+0000 7fe17659c640 1 -- 172.21.15.31:0/1319427663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 msgr2=0x560cd9dad610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:43.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.211+0000 7fe17659c640 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9dad610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.211+0000 7fe17659c640 1 -- 172.21.15.31:0/1319427663 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560cd9daeb00 con 0x560cd99b14d0 2024-04-25T16:05:43.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.211+0000 7fe17659c640 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9dad0d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x560cd9db6650 tx=0x7fe16c0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1444e51a1d26ddf5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:43.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.211+0000 7fe15dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:43.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.214+0000 7fe15dffb640 1 -- 172.21.15.31:0/1319427663 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16c017a30 con 0x560cd99b14d0 2024-04-25T16:05:43.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.214+0000 7fe183bc7740 1 -- 172.21.15.31:0/1319427663 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe170036040 con 0x560cd99b14d0 2024-04-25T16:05:43.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.214+0000 7fe15dffb640 1 -- 172.21.15.31:0/1319427663 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe16c017bd0 con 0x560cd99b14d0 2024-04-25T16:05:43.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.218+0000 7fe15dffb640 1 -- 172.21.15.31:0/1319427663 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe16c017dd0 con 0x560cd99b14d0 2024-04-25T16:05:43.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.218+0000 7fe183bc7740 10 client.16521 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:43.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.218+0000 7fe183bc7740 1 -- 172.21.15.31:0/1319427663 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560cd9dafe80 con 0x560cd99b14d0 2024-04-25T16:05:43.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.219+0000 7fe15dffb640 1 -- 172.21.15.31:0/1319427663 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe170036040 con 0x560cd99b14d0 2024-04-25T16:05:43.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.219+0000 7fe15dffb640 1 -- 172.21.15.31:0/1319427663 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe16c006530 con 0x560cd99b14d0 2024-04-25T16:05:43.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.219+0000 7fe183bc7740 10 client.16521 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:43.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.219+0000 7fe183bc7740 10 client.16521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:43.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.219+0000 7fe183bc7740 20 client.16521 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:43.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.219+0000 7fe183bc7740 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560cd996e3a0 0x560cd9e20b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:43.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.220+0000 7fe183bc7740 4 client.16521 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:05:43.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.220+0000 7fe183bc7740 1 -- 172.21.15.31:0/1319427663 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560cd9ce8800 con 0x560cd996e3a0 2024-04-25T16:05:43.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.222+0000 7fe175d9b640 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560cd996e3a0 0x560cd9e20b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:43.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.243+0000 7fe175d9b640 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560cd996e3a0 0x560cd9e20b10 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90bc2c0220ec1e95 server_cookie=25edce0da72f229b in_seq=0 out_seq=0 2024-04-25T16:05:43.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.244+0000 7fe15dffb640 10 client.16521 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:43.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.255+0000 7fe15dffb640 1 -- 172.21.15.31:0/1319427663 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560cd9ce8800 con 0x560cd996e3a0 2024-04-25T16:05:43.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.255+0000 7fe15dffb640 10 client.16521 handle_command_reply: tid=0 2024-04-25T16:05:43.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.256+0000 7fe183bc7740 1 client.16521 shutdown 2024-04-25T16:05:43.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.270+0000 7fe183bc7740 1 -- 172.21.15.31:0/1319427663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 msgr2=0x560cd9dad0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:43.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.270+0000 7fe183bc7740 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9dad0d0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x560cd9db6650 tx=0x7fe16c0047d0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.272+0000 7fe183bc7740 1 -- 172.21.15.31:0/1319427663 shutdown_connections 2024-04-25T16:05:43.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.272+0000 7fe183bc7740 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560cd996e3a0 0x560cd9e20b10 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.272+0000 7fe183bc7740 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560cd99da390 0x560cd9dae540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.272+0000 7fe183bc7740 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560cd99b96b0 0x560cd9dad610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.272+0000 7fe183bc7740 1 --2- 172.21.15.31:0/1319427663 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560cd99b14d0 0x560cd9dad0d0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.272+0000 7fe183bc7740 1 -- 172.21.15.31:0/1319427663 >> 172.21.15.31:0/1319427663 conn(0x560cd99ce4e0 msgr2=0x560cd9d7ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:43.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.272+0000 7fe15dffb640 0 client.16521 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:43.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.280+0000 7fe183bc7740 1 -- 172.21.15.31:0/1319427663 shutdown_connections 2024-04-25T16:05:43.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.281+0000 7fe183bc7740 1 -- 172.21.15.31:0/1319427663 wait complete. 2024-04-25T16:05:43.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.282+0000 7fe183bc7740 20 client.16521 trim_cache size 0 max 16384 2024-04-25T16:05:43.284 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:05:43.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.287+0000 7fe17cd14640 1 -- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1600d1be0 msgr2=0x7fe1600d40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:43.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.287+0000 7fe17cd14640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1600d1be0 0x7fe1600d40f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x560cd9d7c700 tx=0x7fe16c0090e0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.287+0000 7fe17cd14640 1 -- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe17810fef0 msgr2=0x7fe178118450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:43.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.287+0000 7fe17cd14640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe17810fef0 0x7fe178118450 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fe16800c970 tx=0x7fe16800ce30 comp rx=0 tx=0).stop 2024-04-25T16:05:43.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.288+0000 7fe17cd14640 1 -- 172.21.15.31:0/2701416862 shutdown_connections 2024-04-25T16:05:43.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.289+0000 7fe17cd14640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe1600d1be0 0x7fe1600d40f0 secure :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0x560cd9d7c700 tx=0x7fe16c0090e0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.289+0000 7fe17cd14640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe17810fef0 0x7fe178118450 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.289+0000 7fe17cd14640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe17810b1d0 0x7fe1781174f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.289+0000 7fe17cd14640 1 --2- 172.21.15.31:0/2701416862 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe17810a3e0 0x7fe178116fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:43.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.289+0000 7fe17cd14640 1 -- 172.21.15.31:0/2701416862 >> 172.21.15.31:0/2701416862 conn(0x7fe178070400 msgr2=0x7fe17810bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.307+0000 7fe17cd14640 1 -- 172.21.15.31:0/2701416862 shutdown_connections 2024-04-25T16:05:43.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:43.309+0000 7fe17cd14640 1 -- 172.21.15.31:0/2701416862 wait complete. 2024-04-25T16:05:44.381 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:05:44.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:05:44.382 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:05:44.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.860+0000 7f2cd6222640 1 -- 172.21.15.31:0/2851113307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80a6a20 msgr2=0x7f2cc80a6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:44.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.860+0000 7f2cd6222640 1 --2- 172.21.15.31:0/2851113307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80a6a20 0x7f2cc80a6e30 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009900 tx=0x7f2ccc02a410 comp rx=0 tx=0).stop 2024-04-25T16:05:44.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.860+0000 7f2cd6222640 1 -- 172.21.15.31:0/2851113307 shutdown_connections 2024-04-25T16:05:44.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.860+0000 7f2cd6222640 1 --2- 172.21.15.31:0/2851113307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cc80ab740 0x7f2cc80abb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.860+0000 7f2cd6222640 1 --2- 172.21.15.31:0/2851113307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80a6a20 0x7f2cc80a6e30 secure :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009900 tx=0x7f2ccc02a410 comp rx=0 tx=0).stop 2024-04-25T16:05:44.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.860+0000 7f2cd6222640 1 --2- 172.21.15.31:0/2851113307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cc80a5920 0x7f2cc80a64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.860+0000 7f2cd6222640 1 -- 172.21.15.31:0/2851113307 >> 172.21.15.31:0/2851113307 conn(0x7f2cc80a0990 msgr2=0x7f2cc80a2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:44.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.866+0000 7f2cd6222640 1 -- 172.21.15.31:0/2851113307 shutdown_connections 2024-04-25T16:05:44.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.868+0000 7f2cd6222640 1 -- 172.21.15.31:0/2851113307 wait complete. 2024-04-25T16:05:44.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.871+0000 7f2cd6222640 1 Processor -- start 2024-04-25T16:05:44.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.874+0000 7f2cd6222640 1 -- start start 2024-04-25T16:05:44.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.876+0000 7f2cd6222640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cc80a5920 0x7f2cc80b6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.876+0000 7f2cd6222640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80ab740 0x7f2cc80b7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.876+0000 7f2cd6222640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cc80b54a0 0x7f2cc80b5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.876+0000 7f2cd6222640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc80ad750 con 0x7f2cc80ab740 2024-04-25T16:05:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.876+0000 7f2cd6222640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc80ad5d0 con 0x7f2cc80a5920 2024-04-25T16:05:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.876+0000 7f2cd6222640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc80ad8d0 con 0x7f2cc80b54a0 2024-04-25T16:05:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.877+0000 7f2cd5a21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cc80b54a0 0x7f2cc80b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.877+0000 7f2cd5a21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cc80b54a0 0x7f2cc80b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:55718/0 (socket says 172.21.15.31:55718) 2024-04-25T16:05:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.877+0000 7f2cd5a21640 1 -- 172.21.15.31:0/274782026 learned_addr learned my addr 172.21.15.31:0/274782026 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:44.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.877+0000 7f2cd4a1f640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80ab740 0x7f2cc80b7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.878+0000 7f2cd5a21640 1 -- 172.21.15.31:0/274782026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cc80a5920 msgr2=0x7f2cc80b6d60 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:05:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.878+0000 7f2cd5a21640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cc80a5920 0x7f2cc80b6d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.878+0000 7f2cd5a21640 1 -- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80ab740 msgr2=0x7f2cc80b7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.878+0000 7f2cd5a21640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80ab740 0x7f2cc80b7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.878+0000 7f2cd5a21640 1 -- 172.21.15.31:0/274782026 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc80b6180 con 0x7f2cc80b54a0 2024-04-25T16:05:44.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.884+0000 7f2cd5a21640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cc80b54a0 0x7f2cc80b5960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0068910 tx=0x7f2cd006fe70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.890+0000 7f2cc67fc640 1 -- 172.21.15.31:0/274782026 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0080070 con 0x7f2cc80b54a0 2024-04-25T16:05:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.890+0000 7f2cc67fc640 1 -- 172.21.15.31:0/274782026 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cd007b480 con 0x7f2cc80b54a0 2024-04-25T16:05:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.892+0000 7f2cc67fc640 1 -- 172.21.15.31:0/274782026 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd007a640 con 0x7f2cc80b54a0 2024-04-25T16:05:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.894+0000 7f2cd6222640 1 -- 172.21.15.31:0/274782026 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc8155290 con 0x7f2cc80b54a0 2024-04-25T16:05:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.895+0000 7f2cd6222640 1 -- 172.21.15.31:0/274782026 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc8155760 con 0x7f2cc80b54a0 2024-04-25T16:05:44.904 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:44 smithi031 ceph-mon[29140]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 926 MiB/s wr, 1.95k op/s 2024-04-25T16:05:44.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.908+0000 7f2cde408740 1 Processor -- start 2024-04-25T16:05:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.912+0000 7f2cc67fc640 1 -- 172.21.15.31:0/274782026 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2cd006eb30 con 0x7f2cc80b54a0 2024-04-25T16:05:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.913+0000 7f2cde408740 1 -- start start 2024-04-25T16:05:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.913+0000 7f2cde408740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 0x55d73bc226a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.913+0000 7f2cde408740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b858550 0x55d73bc22c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.913+0000 7f2cde408740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 0x55d73bc29b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.913+0000 7f2cde408740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d73bb87b00 con 0x55d73b860730 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.913+0000 7f2cde408740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d73bb23920 con 0x55d73b858550 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.913+0000 7f2cde408740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d73bb83ac0 con 0x55d73b881410 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.914+0000 7f2cd5220640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b858550 0x55d73bc22c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.914+0000 7f2cd5220640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b858550 0x55d73bc22c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40902/0 (socket says 172.21.15.31:40902) 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.914+0000 7f2cd5220640 1 -- 172.21.15.31:0/845757662 learned_addr learned my addr 172.21.15.31:0/845757662 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.915+0000 7f2cc67fc640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2cbc0afb90 0x7f2cbc0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.915+0000 7f2cd4a1f640 1 --2- 172.21.15.31:0/845757662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 0x55d73bc29b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.915+0000 7f2cd5220640 1 -- 172.21.15.31:0/845757662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 msgr2=0x55d73bc226a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.915+0000 7f2cd5220640 1 --2- 172.21.15.31:0/845757662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 0x55d73bc226a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.915+0000 7f2cd5220640 1 -- 172.21.15.31:0/845757662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 msgr2=0x55d73bc29b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.915+0000 7f2cd5220640 1 --2- 172.21.15.31:0/845757662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 0x55d73bc29b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.920+0000 7f2cd5220640 1 -- 172.21.15.31:0/845757662 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d73b9617e0 con 0x55d73b858550 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.916+0000 7f2cc67fc640 1 -- 172.21.15.31:0/274782026 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2cd0143650 con 0x7f2cc80b54a0 2024-04-25T16:05:44.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.920+0000 7f2cd5a21640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2cbc0afb90 0x7f2cbc0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:44.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.925+0000 7f2cd5220640 1 --2- 172.21.15.31:0/845757662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b858550 0x55d73bc22c20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009b10 tx=0x7f2cc000e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f75589b8444878d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:44.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.932+0000 7f2cbb7fe640 1 -- 172.21.15.31:0/845757662 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc000ed50 con 0x55d73b858550 2024-04-25T16:05:44.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.932+0000 7f2cbb7fe640 1 -- 172.21.15.31:0/845757662 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cc0005020 con 0x55d73b858550 2024-04-25T16:05:44.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.932+0000 7f2cbb7fe640 1 -- 172.21.15.31:0/845757662 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0004a90 con 0x55d73b858550 2024-04-25T16:05:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.937+0000 7f2cde408740 1 -- 172.21.15.31:0/845757662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b858550 msgr2=0x55d73bc22c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:44.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.937+0000 7f2cde408740 1 --2- 172.21.15.31:0/845757662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b858550 0x55d73bc22c20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009b10 tx=0x7f2cc000e350 comp rx=0 tx=0).stop 2024-04-25T16:05:44.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.939+0000 7f2cde408740 1 -- 172.21.15.31:0/845757662 shutdown_connections 2024-04-25T16:05:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.939+0000 7f2cde408740 1 --2- 172.21.15.31:0/845757662 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 0x55d73bc29b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.939+0000 7f2cde408740 1 --2- 172.21.15.31:0/845757662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b858550 0x55d73bc22c20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.939+0000 7f2cde408740 1 --2- 172.21.15.31:0/845757662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 0x55d73bc226a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.939+0000 7f2cde408740 1 -- 172.21.15.31:0/845757662 >> 172.21.15.31:0/845757662 conn(0x55d73b875560 msgr2=0x55d73bc1e030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.942+0000 7f2cde408740 1 -- 172.21.15.31:0/845757662 shutdown_connections 2024-04-25T16:05:44.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.942+0000 7f2cde408740 1 -- 172.21.15.31:0/845757662 wait complete. 2024-04-25T16:05:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.943+0000 7f2cde408740 1 -- start start 2024-04-25T16:05:44.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.943+0000 7f2cde408740 1 Processor -- start 2024-04-25T16:05:44.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.944+0000 7f2cde408740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc2c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.944+0000 7f2cde408740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 0x55d73bc2ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.944+0000 7f2cde408740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 0x55d73bc33970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.944+0000 7f2cde408740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d73bc2a570 con 0x55d73b858550 2024-04-25T16:05:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.944+0000 7f2cde408740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d73bb215d0 con 0x55d73b860730 2024-04-25T16:05:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.944+0000 7f2cde408740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d73bc2a6f0 con 0x55d73b881410 2024-04-25T16:05:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.944+0000 7f2cde408740 10 client.0 _ll_register_callbacks cb 0x55d73bb0f080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.946+0000 7f2cd5a21640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2cbc0afb90 0x7f2cbc0b20a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0107b60 tx=0x7f2cd0068d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:44.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.946+0000 7f2cd5220640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 0x55d73bc2ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.947+0000 7f2cd4a1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc2c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.947+0000 7f2cd4a1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc2c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46220/0 (socket says 172.21.15.31:46220) 2024-04-25T16:05:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.947+0000 7f2cd4a1f640 1 -- 172.21.15.31:0/1919807674 learned_addr learned my addr 172.21.15.31:0/1919807674 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.947+0000 7f2cd4a1f640 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 0x55d73bc33970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:44.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.947+0000 7f2cd5220640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 0x55d73bc2ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40916/0 (socket says 172.21.15.31:40916) 2024-04-25T16:05:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.949+0000 7f2cd4a1f640 1 -- 172.21.15.31:0/1919807674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 msgr2=0x55d73bc2ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.949+0000 7f2cd4a1f640 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 0x55d73bc2ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.949+0000 7f2cd4a1f640 1 -- 172.21.15.31:0/1919807674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 msgr2=0x55d73bc2c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.949+0000 7f2cd4a1f640 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc2c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:44.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.949+0000 7f2cd4a1f640 1 -- 172.21.15.31:0/1919807674 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d73b9c1f70 con 0x55d73b881410 2024-04-25T16:05:44.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.959+0000 7f2cd4a1f640 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 0x55d73bc33970 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc005ab0 tx=0x7f2ccc009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85aaf9d5a146ccde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:44.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.959+0000 7f2cb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.962+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/1919807674 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ccc042070 con 0x55d73b881410 2024-04-25T16:05:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.962+0000 7f2cde408740 1 -- 172.21.15.31:0/1919807674 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cc0005020 con 0x55d73b881410 2024-04-25T16:05:44.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.962+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/1919807674 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2ccc005590 con 0x55d73b881410 2024-04-25T16:05:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.964+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/1919807674 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2ccc053050 con 0x55d73b881410 2024-04-25T16:05:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.964+0000 7f2cde408740 10 client.26173 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:44.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.964+0000 7f2cde408740 1 -- 172.21.15.31:0/1919807674 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d73b859170 con 0x55d73b881410 2024-04-25T16:05:44.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.968+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/1919807674 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cc0005020 con 0x55d73b881410 2024-04-25T16:05:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.970+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/1919807674 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f2ccc003bc0 con 0x55d73b881410 2024-04-25T16:05:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.970+0000 7f2cde408740 10 client.26173 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.970+0000 7f2cde408740 10 client.26173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.970+0000 7f2cde408740 20 client.26173 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.970+0000 7f2cde408740 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d73b8401f0 0x55d73bc3a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.970+0000 7f2cde408740 4 client.26173 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:44.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.970+0000 7f2cde408740 1 -- 172.21.15.31:0/1919807674 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d73bb95710 con 0x55d73b8401f0 2024-04-25T16:05:44.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:44.987+0000 7f2cd5220640 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d73b8401f0 0x55d73bc3a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.001+0000 7f2cd5220640 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d73b8401f0 0x55d73bc3a290 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e03a46b7a30c0fae server_cookie=60bb009ba760719 in_seq=0 out_seq=0 2024-04-25T16:05:45.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.002+0000 7f2cb8ff9640 10 client.26173 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:45.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.002+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/1919807674 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d73bb95710 con 0x55d73b8401f0 2024-04-25T16:05:45.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.003+0000 7f2cb8ff9640 10 client.26173 handle_command_reply: tid=0 2024-04-25T16:05:45.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.003+0000 7f2cde408740 1 client.26173 shutdown 2024-04-25T16:05:45.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.004+0000 7f2cde408740 1 -- 172.21.15.31:0/1919807674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 msgr2=0x55d73bc33970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.004+0000 7f2cde408740 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 0x55d73bc33970 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc005ab0 tx=0x7f2ccc009260 comp rx=0 tx=0).stop 2024-04-25T16:05:45.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.004+0000 7f2cde408740 1 -- 172.21.15.31:0/1919807674 shutdown_connections 2024-04-25T16:05:45.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.004+0000 7f2cde408740 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d73b8401f0 0x55d73bc3a290 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.004+0000 7f2cde408740 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b881410 0x55d73bc33970 secure :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc005ab0 tx=0x7f2ccc009260 comp rx=0 tx=0).stop 2024-04-25T16:05:45.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.004+0000 7f2cde408740 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 0x55d73bc2ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.004+0000 7f2cde408740 1 --2- 172.21.15.31:0/1919807674 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc2c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.004+0000 7f2cde408740 1 -- 172.21.15.31:0/1919807674 >> 172.21.15.31:0/1919807674 conn(0x55d73b875560 msgr2=0x55d73bc1d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.004+0000 7f2cb8ff9640 0 client.26173 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:45.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.020+0000 7f2cde408740 1 -- 172.21.15.31:0/1919807674 shutdown_connections 2024-04-25T16:05:45.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.021+0000 7f2cde408740 1 -- 172.21.15.31:0/1919807674 wait complete. 2024-04-25T16:05:45.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.021+0000 7f2cde408740 20 client.26173 trim_cache size 0 max 16384 2024-04-25T16:05:45.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.131+0000 7f2cde408740 1 Processor -- start 2024-04-25T16:05:45.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.131+0000 7f2cde408740 1 -- start start 2024-04-25T16:05:45.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.131+0000 7f2cde408740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b858550 0x55d73bc2be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.131+0000 7f2cde408740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 0x55d73bc33890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.131+0000 7f2cde408740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b815630 0x55d73bc1d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.131+0000 7f2cde408740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8010860 con 0x55d73b860730 2024-04-25T16:05:45.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.131+0000 7f2cde408740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc800e650 con 0x55d73b815630 2024-04-25T16:05:45.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.131+0000 7f2cde408740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8010190 con 0x55d73b858550 2024-04-25T16:05:45.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.132+0000 7f2cd4a1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b815630 0x55d73bc1d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.132+0000 7f2cd4a1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b815630 0x55d73bc1d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40920/0 (socket says 172.21.15.31:40920) 2024-04-25T16:05:45.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.133+0000 7f2cd4a1f640 1 -- 172.21.15.31:0/1141678296 learned_addr learned my addr 172.21.15.31:0/1141678296 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:45.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.133+0000 7f2cd4a1f640 1 --2- 172.21.15.31:0/1141678296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b858550 0x55d73bc2be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.137+0000 7f2cd5220640 1 --2- 172.21.15.31:0/1141678296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 0x55d73bc33890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.138+0000 7f2cd5220640 1 -- 172.21.15.31:0/1141678296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b858550 msgr2=0x55d73bc2be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:45.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.138+0000 7f2cd5220640 1 --2- 172.21.15.31:0/1141678296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b858550 0x55d73bc2be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.138+0000 7f2cd5220640 1 -- 172.21.15.31:0/1141678296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b815630 msgr2=0x55d73bc1d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:45.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.138+0000 7f2cd5220640 1 --2- 172.21.15.31:0/1141678296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b815630 0x55d73bc1d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.138+0000 7f2cd5220640 1 -- 172.21.15.31:0/1141678296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d73bc2b1f0 con 0x55d73b860730 2024-04-25T16:05:45.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.140+0000 7f2cd5220640 1 --2- 172.21.15.31:0/1141678296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 0x55d73bc33890 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0000c00 tx=0x7f2cc000e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39835ccad89e79c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:45.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.141+0000 7f2cbaffd640 1 -- 172.21.15.31:0/1141678296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0049030 con 0x55d73b860730 2024-04-25T16:05:45.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.142+0000 7f2cbaffd640 1 -- 172.21.15.31:0/1141678296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cc000ee70 con 0x55d73b860730 2024-04-25T16:05:45.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.144+0000 7f2cde408740 1 -- 172.21.15.31:0/1141678296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 msgr2=0x55d73bc33890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.144+0000 7f2cde408740 1 --2- 172.21.15.31:0/1141678296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 0x55d73bc33890 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0000c00 tx=0x7f2cc000e950 comp rx=0 tx=0).stop 2024-04-25T16:05:45.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:44 smithi186 ceph-mon[33258]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 926 MiB/s wr, 1.95k op/s 2024-04-25T16:05:45.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.149+0000 7f2cde408740 1 -- 172.21.15.31:0/1141678296 shutdown_connections 2024-04-25T16:05:45.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.149+0000 7f2cde408740 1 --2- 172.21.15.31:0/1141678296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b815630 0x55d73bc1d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.149+0000 7f2cde408740 1 --2- 172.21.15.31:0/1141678296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b860730 0x55d73bc33890 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.149+0000 7f2cde408740 1 --2- 172.21.15.31:0/1141678296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b858550 0x55d73bc2be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.149+0000 7f2cde408740 1 -- 172.21.15.31:0/1141678296 >> 172.21.15.31:0/1141678296 conn(0x55d73b875560 msgr2=0x55d73bc2b780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.167+0000 7f2cde408740 1 -- 172.21.15.31:0/1141678296 shutdown_connections 2024-04-25T16:05:45.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.167+0000 7f2cde408740 1 -- 172.21.15.31:0/1141678296 wait complete. 2024-04-25T16:05:45.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.168+0000 7f2cde408740 1 -- start start 2024-04-25T16:05:45.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.168+0000 7f2cde408740 1 Processor -- start 2024-04-25T16:05:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.170+0000 7f2cde408740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b815630 0x55d73bc30880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.170+0000 7f2cde408740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc30dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.170+0000 7f2cde408740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 0x55d73bca0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.170+0000 7f2cde408740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc800f3f0 con 0x55d73b858550 2024-04-25T16:05:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.170+0000 7f2cde408740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc800ed20 con 0x55d73b860730 2024-04-25T16:05:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.170+0000 7f2cde408740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc800fac0 con 0x55d73b815630 2024-04-25T16:05:45.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.170+0000 7f2cde408740 10 client.0 _ll_register_callbacks cb 0x7f2ccc004a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:45.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.171+0000 7f2cd4a1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b815630 0x55d73bc30880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.172+0000 7f2cd5220640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc30dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.172+0000 7f2cd5220640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc30dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46226/0 (socket says 172.21.15.31:46226) 2024-04-25T16:05:45.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.172+0000 7f2cd5220640 1 -- 172.21.15.31:0/4062413317 learned_addr learned my addr 172.21.15.31:0/4062413317 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:45.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.173+0000 7f2cd5220640 1 -- 172.21.15.31:0/4062413317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b815630 msgr2=0x55d73bc30880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.173+0000 7f2cd5220640 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b815630 0x55d73bc30880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.173+0000 7f2cd5220640 1 -- 172.21.15.31:0/4062413317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 msgr2=0x55d73bca0600 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.173+0000 7f2cd5220640 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 0x55d73bca0600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.173+0000 7f2cd5220640 1 -- 172.21.15.31:0/4062413317 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d73bca0b40 con 0x55d73b858550 2024-04-25T16:05:45.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.174+0000 7f2cd5220640 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc30dc0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009250 tx=0x7f2cc0003a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e172a664564cfe4b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:45.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.175+0000 7f2cb8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.175+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/4062413317 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0049040 con 0x55d73b858550 2024-04-25T16:05:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.176+0000 7f2cde408740 1 -- 172.21.15.31:0/4062413317 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cc000ee70 con 0x55d73b858550 2024-04-25T16:05:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.176+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/4062413317 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2cc004a450 con 0x55d73b858550 2024-04-25T16:05:45.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.177+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/4062413317 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2cc0003bb0 con 0x55d73b858550 2024-04-25T16:05:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.179+0000 7f2cde408740 10 client.16539 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:45.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.179+0000 7f2cde408740 1 -- 172.21.15.31:0/4062413317 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d73bca1d90 con 0x55d73b858550 2024-04-25T16:05:45.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.180+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/4062413317 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2cc000ee70 con 0x55d73b858550 2024-04-25T16:05:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.182+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/4062413317 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f2cc0032a60 con 0x55d73b858550 2024-04-25T16:05:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.182+0000 7f2cde408740 10 client.16539 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.182+0000 7f2cde408740 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.182+0000 7f2cde408740 20 client.16539 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.182+0000 7f2cde408740 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d73bca2620 0x55d73bca4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.182+0000 7f2cde408740 4 client.16539 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:05:45.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.182+0000 7f2cde408740 1 -- 172.21.15.31:0/4062413317 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d73bb8f580 con 0x55d73bca2620 2024-04-25T16:05:45.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.184+0000 7f2cd4a1f640 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d73bca2620 0x55d73bca4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.197+0000 7f2cd4a1f640 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d73bca2620 0x55d73bca4a40 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ab08aa7cdfdd2a8 server_cookie=95e6615c386cdf8b in_seq=0 out_seq=0 2024-04-25T16:05:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.197+0000 7f2cb8ff9640 10 client.16539 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.199+0000 7f2cb8ff9640 1 -- 172.21.15.31:0/4062413317 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d73bb8f580 con 0x55d73bca2620 2024-04-25T16:05:45.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.199+0000 7f2cb8ff9640 10 client.16539 handle_command_reply: tid=0 2024-04-25T16:05:45.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.202+0000 7f2cde408740 1 client.16539 shutdown 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.205+0000 7f2cde408740 1 -- 172.21.15.31:0/4062413317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 msgr2=0x55d73bc30dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.205+0000 7f2cde408740 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc30dc0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009250 tx=0x7f2cc0003a40 comp rx=0 tx=0).stop 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.205+0000 7f2cde408740 1 -- 172.21.15.31:0/4062413317 shutdown_connections 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.205+0000 7f2cde408740 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d73bca2620 0x55d73bca4a40 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.205+0000 7f2cde408740 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d73b860730 0x55d73bca0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.205+0000 7f2cde408740 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d73b858550 0x55d73bc30dc0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.205+0000 7f2cde408740 1 --2- 172.21.15.31:0/4062413317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d73b815630 0x55d73bc30880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.205+0000 7f2cde408740 1 -- 172.21.15.31:0/4062413317 >> 172.21.15.31:0/4062413317 conn(0x55d73b875560 msgr2=0x55d73bc1d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.206+0000 7f2cb8ff9640 0 client.16539 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.206+0000 7f2cde408740 1 -- 172.21.15.31:0/4062413317 shutdown_connections 2024-04-25T16:05:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.208+0000 7f2cde408740 1 -- 172.21.15.31:0/4062413317 wait complete. 2024-04-25T16:05:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.208+0000 7f2cde408740 20 client.16539 trim_cache size 0 max 16384 2024-04-25T16:05:45.211 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:05:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:05:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "0f22f229-3662-4c2e-ba27-e0b941acb830", 2024-04-25T16:05:45.211 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:05:45.211 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:05:45.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.215+0000 7f2cd6222640 1 -- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2cbc0afb90 msgr2=0x7f2cbc0b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.215+0000 7f2cd6222640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2cbc0afb90 0x7f2cbc0b20a0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0107b60 tx=0x7f2cd0068d50 comp rx=0 tx=0).stop 2024-04-25T16:05:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.215+0000 7f2cd6222640 1 -- 172.21.15.31:0/274782026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cc80b54a0 msgr2=0x7f2cc80b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.216+0000 7f2cd6222640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cc80b54a0 0x7f2cc80b5960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0068910 tx=0x7f2cd006fe70 comp rx=0 tx=0).stop 2024-04-25T16:05:45.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.216+0000 7f2cd6222640 1 -- 172.21.15.31:0/274782026 shutdown_connections 2024-04-25T16:05:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.216+0000 7f2cd6222640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2cbc0afb90 0x7f2cbc0b20a0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.216+0000 7f2cd6222640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cc80b54a0 0x7f2cc80b5960 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.216+0000 7f2cd6222640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2cc80ab740 0x7f2cc80b7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.216+0000 7f2cd6222640 1 --2- 172.21.15.31:0/274782026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2cc80a5920 0x7f2cc80b6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.217+0000 7f2cd6222640 1 -- 172.21.15.31:0/274782026 >> 172.21.15.31:0/274782026 conn(0x7f2cc80a0990 msgr2=0x7f2cc80a1060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.217+0000 7f2cd6222640 1 -- 172.21.15.31:0/274782026 shutdown_connections 2024-04-25T16:05:45.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.217+0000 7f2cd6222640 1 -- 172.21.15.31:0/274782026 wait complete. 2024-04-25T16:05:45.267 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:05:45.267 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:05:45.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:44 smithi063 ceph-mon[32413]: pgmap v484: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 926 MiB/s wr, 1.95k op/s 2024-04-25T16:05:45.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.644+0000 7f452ac92640 1 -- 172.21.15.31:0/258202937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410b310 msgr2=0x7f452410b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.644+0000 7f452ac92640 1 --2- 172.21.15.31:0/258202937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410b310 0x7f452410b720 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f451c009ca0 tx=0x7f451c02a8f0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.658+0000 7f452ac92640 1 -- 172.21.15.31:0/258202937 shutdown_connections 2024-04-25T16:05:45.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.658+0000 7f452ac92640 1 --2- 172.21.15.31:0/258202937 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4524110030 0x7f4524110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.658+0000 7f452ac92640 1 --2- 172.21.15.31:0/258202937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410b310 0x7f452410b720 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.658+0000 7f452ac92640 1 --2- 172.21.15.31:0/258202937 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410a420 0x7f452410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.658+0000 7f452ac92640 1 -- 172.21.15.31:0/258202937 >> 172.21.15.31:0/258202937 conn(0x7f45240704e0 msgr2=0x7f4524072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.670+0000 7f452ac92640 1 -- 172.21.15.31:0/258202937 shutdown_connections 2024-04-25T16:05:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.670+0000 7f452ac92640 1 -- 172.21.15.31:0/258202937 wait complete. 2024-04-25T16:05:45.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.671+0000 7f452ac92640 1 Processor -- start 2024-04-25T16:05:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.672+0000 7f452ac92640 1 -- start start 2024-04-25T16:05:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.672+0000 7f452ac92640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410a420 0x7f45241a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.673+0000 7f452ac92640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410b310 0x7f45241a9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.673+0000 7f452ac92640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4524110030 0x7f45241b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.673+0000 7f452ac92640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f452411e4e0 con 0x7f452410a420 2024-04-25T16:05:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.673+0000 7f452ac92640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f452411e360 con 0x7f4524110030 2024-04-25T16:05:45.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.673+0000 7f452ac92640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f452411e660 con 0x7f452410b310 2024-04-25T16:05:45.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.675+0000 7f4528a2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410a420 0x7f45241a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.675+0000 7f4528a2e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410a420 0x7f45241a90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46242/0 (socket says 172.21.15.31:46242) 2024-04-25T16:05:45.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.675+0000 7f4528a2e640 1 -- 172.21.15.31:0/4063851300 learned_addr learned my addr 172.21.15.31:0/4063851300 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:45.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.675+0000 7f452922f640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4524110030 0x7f45241b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.677+0000 7f4523fff640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410b310 0x7f45241a9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.681+0000 7f4528a2e640 1 -- 172.21.15.31:0/4063851300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410b310 msgr2=0x7f45241a9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.681+0000 7f4528a2e640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410b310 0x7f45241a9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.681+0000 7f4528a2e640 1 -- 172.21.15.31:0/4063851300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4524110030 msgr2=0x7f45241b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.681+0000 7f4528a2e640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4524110030 0x7f45241b05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.681+0000 7f4528a2e640 1 -- 172.21.15.31:0/4063851300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4524111be0 con 0x7f452410a420 2024-04-25T16:05:45.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.681+0000 7f4523fff640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410b310 0x7f45241a9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:05:45.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.681+0000 7f452922f640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4524110030 0x7f45241b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:05:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.684+0000 7f4528a2e640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410a420 0x7f45241a90f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f451800c970 tx=0x7f451800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:45.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.696+0000 7f4521ffb640 1 -- 172.21.15.31:0/4063851300 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518007bb0 con 0x7f452410a420 2024-04-25T16:05:45.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.696+0000 7f4521ffb640 1 -- 172.21.15.31:0/4063851300 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4518007d50 con 0x7f452410a420 2024-04-25T16:05:45.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.696+0000 7f4521ffb640 1 -- 172.21.15.31:0/4063851300 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451800f450 con 0x7f452410a420 2024-04-25T16:05:45.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.699+0000 7f452ac92640 1 -- 172.21.15.31:0/4063851300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4524111ed0 con 0x7f452410a420 2024-04-25T16:05:45.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.699+0000 7f452ac92640 1 -- 172.21.15.31:0/4063851300 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45241123d0 con 0x7f452410a420 2024-04-25T16:05:45.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.712+0000 7f4531c1c740 1 Processor -- start 2024-04-25T16:05:45.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.712+0000 7f4521ffb640 1 -- 172.21.15.31:0/4063851300 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f45180046c0 con 0x7f452410a420 2024-04-25T16:05:45.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.712+0000 7f4531c1c740 1 -- start start 2024-04-25T16:05:45.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.713+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 0x562e5f0c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.714+0000 7f452922f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 0x562e5f0c9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.714+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecff4d0 0x562e5f0c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.714+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0cac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.714+0000 7f4531c1c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562e5f02ed80 con 0x562e5ed28390 2024-04-25T16:05:45.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.714+0000 7f4531c1c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562e5efe88b0 con 0x562e5ecff4d0 2024-04-25T16:05:45.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.714+0000 7f4531c1c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562e5f02ad40 con 0x562e5ed076b0 2024-04-25T16:05:45.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.715+0000 7f452922f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 0x562e5f0c9a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46258/0 (socket says 172.21.15.31:46258) 2024-04-25T16:05:45.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.715+0000 7f452922f640 1 -- 172.21.15.31:0/1337411687 learned_addr learned my addr 172.21.15.31:0/1337411687 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.716+0000 7f452922f640 1 -- 172.21.15.31:0/1337411687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 msgr2=0x562e5f0cac70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.716+0000 7f452922f640 1 --2- 172.21.15.31:0/1337411687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0cac70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.716+0000 7f452922f640 1 -- 172.21.15.31:0/1337411687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecff4d0 msgr2=0x562e5f0c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.716+0000 7f452922f640 1 --2- 172.21.15.31:0/1337411687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecff4d0 0x562e5f0c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.716+0000 7f4521ffb640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f450c0afd00 0x7f450c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.716+0000 7f452922f640 1 -- 172.21.15.31:0/1337411687 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e5efb9a40 con 0x562e5ed28390 2024-04-25T16:05:45.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.718+0000 7f452922f640 1 --2- 172.21.15.31:0/1337411687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 0x562e5f0c9a60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f451400f3f0 tx=0x7f451400f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e34f370d3ec993 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:45.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.720+0000 7f4521ffb640 1 -- 172.21.15.31:0/4063851300 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f45180deb30 con 0x7f452410a420 2024-04-25T16:05:45.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.724+0000 7f4528a2e640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f450c0afd00 0x7f450c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.725+0000 7f4528a2e640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f450c0afd00 0x7f450c0b2210 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f45180a30f0 tx=0x7f4518005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:45.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.729+0000 7f450affd640 1 -- 172.21.15.31:0/1337411687 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4514005220 con 0x562e5ed28390 2024-04-25T16:05:45.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.730+0000 7f450affd640 1 -- 172.21.15.31:0/1337411687 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45140047a0 con 0x562e5ed28390 2024-04-25T16:05:45.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.731+0000 7f4531c1c740 1 -- 172.21.15.31:0/1337411687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 msgr2=0x562e5f0c9a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.731+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1337411687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 0x562e5f0c9a60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f451400f3f0 tx=0x7f451400f8b0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.741+0000 7f4531c1c740 1 -- 172.21.15.31:0/1337411687 shutdown_connections 2024-04-25T16:05:45.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.741+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1337411687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.741+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1337411687 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecff4d0 0x562e5f0c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.741+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1337411687 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 0x562e5f0c9a60 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.741+0000 7f4531c1c740 1 -- 172.21.15.31:0/1337411687 >> 172.21.15.31:0/1337411687 conn(0x562e5ed1c4e0 msgr2=0x562e5f0c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.745+0000 7f4531c1c740 1 -- 172.21.15.31:0/1337411687 shutdown_connections 2024-04-25T16:05:45.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.745+0000 7f4531c1c740 1 -- 172.21.15.31:0/1337411687 wait complete. 2024-04-25T16:05:45.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.746+0000 7f4531c1c740 1 -- start start 2024-04-25T16:05:45.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.746+0000 7f4531c1c740 1 Processor -- start 2024-04-25T16:05:45.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.748+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0d36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.748+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0d3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.748+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ed28390 0x562e5f0dab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.748+0000 7f4531c1c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562e5f0cb7d0 con 0x562e5ecff4d0 2024-04-25T16:05:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.748+0000 7f452922f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0d36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.748+0000 7f452922f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0d36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46266/0 (socket says 172.21.15.31:46266) 2024-04-25T16:05:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.748+0000 7f452922f640 1 -- 172.21.15.31:0/940740388 learned_addr learned my addr 172.21.15.31:0/940740388 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.748+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562e5efb5d10 con 0x562e5ed28390 2024-04-25T16:05:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.749+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562e5f0cb950 con 0x562e5ed076b0 2024-04-25T16:05:45.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.749+0000 7f4531c1c740 10 client.0 _ll_register_callbacks cb 0x562e5efe2210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:45.753 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.751+0000 7f4523fff640 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0d3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.752+0000 7f4523fff640 1 -- 172.21.15.31:0/940740388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ed28390 msgr2=0x562e5f0dab80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:45.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.752+0000 7f4523fff640 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ed28390 0x562e5f0dab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.752+0000 7f4523fff640 1 -- 172.21.15.31:0/940740388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 msgr2=0x562e5f0d36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:45.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.752+0000 7f4523fff640 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0d36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.752+0000 7f4523fff640 1 -- 172.21.15.31:0/940740388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e5ee0a7d0 con 0x562e5ed076b0 2024-04-25T16:05:45.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.753+0000 7f4523fff640 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0d3c20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f451c031af0 tx=0x7f451c0099c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd8a72414280b3c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:45.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.754+0000 7f4503fff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:05:45.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.761+0000 7f4503fff640 1 -- 172.21.15.31:0/940740388 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451c004040 con 0x562e5ed076b0 2024-04-25T16:05:45.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.761+0000 7f4503fff640 1 -- 172.21.15.31:0/940740388 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f451c0041e0 con 0x562e5ed076b0 2024-04-25T16:05:45.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.761+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f45140047a0 con 0x562e5ed076b0 2024-04-25T16:05:45.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.762+0000 7f4503fff640 1 -- 172.21.15.31:0/940740388 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f451c0043e0 con 0x562e5ed076b0 2024-04-25T16:05:45.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.762+0000 7f4531c1c740 10 client.26185 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:45.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.762+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e5ed403f0 con 0x562e5ed076b0 2024-04-25T16:05:45.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.768+0000 7f4503fff640 1 -- 172.21.15.31:0/940740388 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45140047a0 con 0x562e5ed076b0 2024-04-25T16:05:45.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.770+0000 7f4503fff640 1 -- 172.21.15.31:0/940740388 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f451c044070 con 0x562e5ed076b0 2024-04-25T16:05:45.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.770+0000 7f4531c1c740 10 client.26185 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:45.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.770+0000 7f4531c1c740 10 client.26185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:45.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.770+0000 7f4531c1c740 20 client.26185 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:45.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.770+0000 7f4531c1c740 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562e5ece6f60 0x562e5f0e1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.770+0000 7f4531c1c740 4 client.26185 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:45.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.770+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e5f03c990 con 0x562e5ece6f60 2024-04-25T16:05:45.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.775+0000 7f452922f640 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562e5ece6f60 0x562e5f0e1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.777+0000 7f452922f640 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562e5ece6f60 0x562e5f0e1050 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c8d126ed1377635 server_cookie=2698ed32699612fd in_seq=0 out_seq=0 2024-04-25T16:05:45.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.778+0000 7f4503fff640 10 client.26185 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:45.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.781+0000 7f4503fff640 1 -- 172.21.15.31:0/940740388 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562e5f03c990 con 0x562e5ece6f60 2024-04-25T16:05:45.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.781+0000 7f4503fff640 10 client.26185 handle_command_reply: tid=0 2024-04-25T16:05:45.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.781+0000 7f4531c1c740 1 client.26185 shutdown 2024-04-25T16:05:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.783+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 msgr2=0x562e5f0d3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.783+0000 7f4531c1c740 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0d3c20 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f451c031af0 tx=0x7f451c0099c0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.785+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 shutdown_connections 2024-04-25T16:05:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.785+0000 7f4531c1c740 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562e5ece6f60 0x562e5f0e1050 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.785+0000 7f4531c1c740 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ed28390 0x562e5f0dab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.785+0000 7f4531c1c740 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0d3c20 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.785+0000 7f4531c1c740 1 --2- 172.21.15.31:0/940740388 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0d36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.785+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 >> 172.21.15.31:0/940740388 conn(0x562e5ed1c4e0 msgr2=0x562e5f0c43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.786+0000 7f4503fff640 0 client.26185 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.786+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 shutdown_connections 2024-04-25T16:05:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.786+0000 7f4531c1c740 1 -- 172.21.15.31:0/940740388 wait complete. 2024-04-25T16:05:45.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.787+0000 7f4531c1c740 20 client.26185 trim_cache size 0 max 16384 2024-04-25T16:05:45.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.840+0000 7f4531c1c740 1 Processor -- start 2024-04-25T16:05:45.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.841+0000 7f4531c1c740 1 -- start start 2024-04-25T16:05:45.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.842+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0c40d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.843+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0daaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.843+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ed28390 0x562e5f0d1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.843+0000 7f4531c1c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f45241c26e0 con 0x562e5ecff4d0 2024-04-25T16:05:45.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.843+0000 7f4531c1c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f45241c4220 con 0x562e5ed28390 2024-04-25T16:05:45.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.843+0000 7f4531c1c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f45241c2db0 con 0x562e5ed076b0 2024-04-25T16:05:45.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.845+0000 7f4523fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0daaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.845+0000 7f452922f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0c40d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.845+0000 7f452922f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0c40d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46278/0 (socket says 172.21.15.31:46278) 2024-04-25T16:05:45.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.845+0000 7f452922f640 1 -- 172.21.15.31:0/3098794510 learned_addr learned my addr 172.21.15.31:0/3098794510 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:45.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.845+0000 7f452922f640 1 --2- 172.21.15.31:0/3098794510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ed28390 0x562e5f0d1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.847+0000 7f452922f640 1 -- 172.21.15.31:0/3098794510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 msgr2=0x562e5f0daaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.847+0000 7f452922f640 1 --2- 172.21.15.31:0/3098794510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0daaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.847+0000 7f452922f640 1 -- 172.21.15.31:0/3098794510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ed28390 msgr2=0x562e5f0d1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.847+0000 7f452922f640 1 --2- 172.21.15.31:0/3098794510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ed28390 0x562e5f0d1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.847+0000 7f452922f640 1 -- 172.21.15.31:0/3098794510 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e5ed000f0 con 0x562e5ecff4d0 2024-04-25T16:05:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.848+0000 7f4523fff640 1 --2- 172.21.15.31:0/3098794510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0daaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:05:45.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.848+0000 7f452922f640 1 --2- 172.21.15.31:0/3098794510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0c40d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f451401a040 tx=0x7f4514000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5397bf0cd3fb43 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:45.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.855+0000 7f450a7fc640 1 -- 172.21.15.31:0/3098794510 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451400ac40 con 0x562e5ecff4d0 2024-04-25T16:05:45.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.855+0000 7f450a7fc640 1 -- 172.21.15.31:0/3098794510 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4514004fd0 con 0x562e5ecff4d0 2024-04-25T16:05:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.856+0000 7f4531c1c740 1 -- 172.21.15.31:0/3098794510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 msgr2=0x562e5f0c40d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.856+0000 7f4531c1c740 1 --2- 172.21.15.31:0/3098794510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0c40d0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f451401a040 tx=0x7f4514000bc0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.857+0000 7f4531c1c740 1 -- 172.21.15.31:0/3098794510 shutdown_connections 2024-04-25T16:05:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.857+0000 7f4531c1c740 1 --2- 172.21.15.31:0/3098794510 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ed28390 0x562e5f0d1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.857+0000 7f4531c1c740 1 --2- 172.21.15.31:0/3098794510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0daaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.857+0000 7f4531c1c740 1 --2- 172.21.15.31:0/3098794510 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ecff4d0 0x562e5f0c40d0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.857+0000 7f4531c1c740 1 -- 172.21.15.31:0/3098794510 >> 172.21.15.31:0/3098794510 conn(0x562e5ed1c4e0 msgr2=0x562e5f0c39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.857+0000 7f4531c1c740 1 -- 172.21.15.31:0/3098794510 shutdown_connections 2024-04-25T16:05:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.857+0000 7f4531c1c740 1 -- 172.21.15.31:0/3098794510 wait complete. 2024-04-25T16:05:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.857+0000 7f4531c1c740 1 -- start start 2024-04-25T16:05:45.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.858+0000 7f4531c1c740 1 Processor -- start 2024-04-25T16:05:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.859+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0d7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.859+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 0x562e5f0d8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.859+0000 7f4531c1c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecbc3a0 0x562e5f0d9170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.859+0000 7f4531c1c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f45241c3b50 con 0x562e5ed28390 2024-04-25T16:05:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.859+0000 7f4531c1c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f45241c3480 con 0x562e5ecbc3a0 2024-04-25T16:05:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.859+0000 7f4531c1c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4524118fd0 con 0x562e5ed076b0 2024-04-25T16:05:45.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.859+0000 7f4531c1c740 10 client.0 _ll_register_callbacks cb 0x7f451c009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.860+0000 7f452922f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecbc3a0 0x562e5f0d9170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.860+0000 7f452922f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecbc3a0 0x562e5f0d9170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:41008/0 (socket says 172.21.15.31:41008) 2024-04-25T16:05:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.860+0000 7f452922f640 1 -- 172.21.15.31:0/1264216950 learned_addr learned my addr 172.21.15.31:0/1264216950 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.860+0000 7f452922f640 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0d7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.861+0000 7f452922f640 1 -- 172.21.15.31:0/1264216950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 msgr2=0x562e5f0d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.861+0000 7f452922f640 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0d7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.861+0000 7f452922f640 1 -- 172.21.15.31:0/1264216950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 msgr2=0x562e5f0d8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.861+0000 7f452922f640 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 0x562e5f0d8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.861+0000 7f452922f640 1 -- 172.21.15.31:0/1264216950 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e5efef4b0 con 0x562e5ecbc3a0 2024-04-25T16:05:45.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.862+0000 7f452922f640 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecbc3a0 0x562e5f0d9170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f451400af40 tx=0x7f451402c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40e82a07909f45d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.878+0000 7f4503fff640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:05:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.878+0000 7f4503fff640 1 -- 172.21.15.31:0/1264216950 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4514005520 con 0x562e5ecbc3a0 2024-04-25T16:05:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.878+0000 7f4503fff640 1 -- 172.21.15.31:0/1264216950 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f45140056c0 con 0x562e5ecbc3a0 2024-04-25T16:05:45.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.878+0000 7f4531c1c740 1 -- 172.21.15.31:0/1264216950 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562e5f036800 con 0x562e5ecbc3a0 2024-04-25T16:05:45.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.879+0000 7f4503fff640 1 -- 172.21.15.31:0/1264216950 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f451402a1f0 con 0x562e5ecbc3a0 2024-04-25T16:05:45.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.896+0000 7f4531c1c740 10 client.26237 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:45.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.896+0000 7f4531c1c740 1 -- 172.21.15.31:0/1264216950 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e5f0da7f0 con 0x562e5ecbc3a0 2024-04-25T16:05:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.901+0000 7f4503fff640 1 -- 172.21.15.31:0/1264216950 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562e5f036800 con 0x562e5ecbc3a0 2024-04-25T16:05:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.901+0000 7f4503fff640 1 -- 172.21.15.31:0/1264216950 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f451402ace0 con 0x562e5ecbc3a0 2024-04-25T16:05:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.901+0000 7f4531c1c740 10 client.26237 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.901+0000 7f4531c1c740 10 client.26237 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.901+0000 7f4531c1c740 20 client.26237 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.901+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562e5f149390 0x562e5f14b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.901+0000 7f4531c1c740 4 client.26237 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:05:45.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.901+0000 7f4531c1c740 1 -- 172.21.15.31:0/1264216950 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562e5efb14a0 con 0x562e5f149390 2024-04-25T16:05:45.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.902+0000 7f4523fff640 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562e5f149390 0x562e5f14b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:45.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.922+0000 7f4523fff640 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562e5f149390 0x562e5f14b7b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8bea05cd32d711c server_cookie=33fa3f4f14bc7273 in_seq=0 out_seq=0 2024-04-25T16:05:45.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.922+0000 7f4503fff640 10 client.26237 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:45.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.929+0000 7f4503fff640 1 -- 172.21.15.31:0/1264216950 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562e5efb14a0 con 0x562e5f149390 2024-04-25T16:05:45.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.929+0000 7f4503fff640 10 client.26237 handle_command_reply: tid=0 2024-04-25T16:05:45.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.929+0000 7f4531c1c740 1 client.26237 shutdown 2024-04-25T16:05:45.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.936+0000 7f4531c1c740 1 -- 172.21.15.31:0/1264216950 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecbc3a0 msgr2=0x562e5f0d9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.936+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecbc3a0 0x562e5f0d9170 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f451400af40 tx=0x7f451402c040 comp rx=0 tx=0).stop 2024-04-25T16:05:45.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.936+0000 7f4531c1c740 1 -- 172.21.15.31:0/1264216950 shutdown_connections 2024-04-25T16:05:45.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.936+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562e5f149390 0x562e5f14b7b0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.936+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562e5ecbc3a0 0x562e5f0d9170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.936+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562e5ed28390 0x562e5f0d8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.936+0000 7f4531c1c740 1 --2- 172.21.15.31:0/1264216950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562e5ed076b0 0x562e5f0d7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.936+0000 7f4531c1c740 1 -- 172.21.15.31:0/1264216950 >> 172.21.15.31:0/1264216950 conn(0x562e5ed1c4e0 msgr2=0x562e5f0d1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.936+0000 7f4503fff640 0 client.26237 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:45.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.937+0000 7f4531c1c740 1 -- 172.21.15.31:0/1264216950 shutdown_connections 2024-04-25T16:05:45.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.937+0000 7f4531c1c740 1 -- 172.21.15.31:0/1264216950 wait complete. 2024-04-25T16:05:45.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.937+0000 7f4531c1c740 20 client.26237 trim_cache size 0 max 16384 2024-04-25T16:05:45.941 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:05:45.942 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T16:05:45.942 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:05:45.942 INFO:teuthology.orchestra.run.smithi031.stdout: "0f22f229-3662-4c2e-ba27-e0b941acb830": { 2024-04-25T16:05:45.942 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:05:45.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "0f22f229-3662-4c2e-ba27-e0b941acb830", 2024-04-25T16:05:45.942 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:05:45.942 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:05:45.942 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:05:45.942 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:05:45.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.954+0000 7f452ac92640 1 -- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f450c0afd00 msgr2=0x7f450c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.954+0000 7f452ac92640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f450c0afd00 0x7f450c0b2210 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f45180a30f0 tx=0x7f4518005d90 comp rx=0 tx=0).stop 2024-04-25T16:05:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.954+0000 7f452ac92640 1 -- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410a420 msgr2=0x7f45241a90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:45.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.954+0000 7f452ac92640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410a420 0x7f45241a90f0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f451800c970 tx=0x7f451800ce30 comp rx=0 tx=0).stop 2024-04-25T16:05:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.959+0000 7f452ac92640 1 -- 172.21.15.31:0/4063851300 shutdown_connections 2024-04-25T16:05:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.959+0000 7f452ac92640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f450c0afd00 0x7f450c0b2210 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.959+0000 7f452ac92640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4524110030 0x7f45241b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.959+0000 7f452ac92640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f452410b310 0x7f45241a9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.959+0000 7f452ac92640 1 --2- 172.21.15.31:0/4063851300 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f452410a420 0x7f45241a90f0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.959+0000 7f452ac92640 1 -- 172.21.15.31:0/4063851300 >> 172.21.15.31:0/4063851300 conn(0x7f45240704e0 msgr2=0x7f452410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.960+0000 7f452ac92640 1 -- 172.21.15.31:0/4063851300 shutdown_connections 2024-04-25T16:05:45.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:05:45.960+0000 7f452ac92640 1 -- 172.21.15.31:0/4063851300 wait complete. 2024-04-25T16:05:46.026 INFO:tasks.cephfs.filesystem:scrub status for tag:0f22f229-3662-4c2e-ba27-e0b941acb830 - {'path': '/', 'tag': '0f22f229-3662-4c2e-ba27-e0b941acb830', 'options': 'recursive,force'} 2024-04-25T16:05:46.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:45 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:05:46.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:45 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:46.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:45 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:05:46.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:45 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:46.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:45 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:05:46.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:45 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:46.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:45 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:05:46.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:45 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:46.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:45 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:05:46.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:45 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:46.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:45 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:05:46.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:45 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:47.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:46 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:05:47.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:46 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:47.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:46 smithi063 ceph-mon[32413]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 726 MiB/s wr, 1.67k op/s 2024-04-25T16:05:47.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:46 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:05:47.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:46 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:47.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:46 smithi186 ceph-mon[33258]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 726 MiB/s wr, 1.67k op/s 2024-04-25T16:05:47.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:46 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:05:47.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:46 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:05:47.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:46 smithi031 ceph-mon[29140]: pgmap v485: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 726 MiB/s wr, 1.67k op/s 2024-04-25T16:05:49.261 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:48 smithi031 ceph-mon[29140]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 853 MiB/s wr, 1.87k op/s 2024-04-25T16:05:49.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:48 smithi063 ceph-mon[32413]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 853 MiB/s wr, 1.87k op/s 2024-04-25T16:05:49.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:48 smithi186 ceph-mon[33258]: pgmap v486: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 853 MiB/s wr, 1.87k op/s 2024-04-25T16:05:49.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.558+0000 7f6c5c184640 1 -- 172.21.15.31:0/4216410765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c5410b1d0 msgr2=0x7f6c5410b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:49.560 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.558+0000 7f6c5c184640 1 --2- 172.21.15.31:0/4216410765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c5410b1d0 0x7f6c5410b5e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6c44009900 tx=0x7f6c4402a400 comp rx=0 tx=0).stop 2024-04-25T16:05:49.562 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.560+0000 7f6c5c184640 1 -- 172.21.15.31:0/4216410765 shutdown_connections 2024-04-25T16:05:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.560+0000 7f6c5c184640 1 --2- 172.21.15.31:0/4216410765 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c5410fef0 0x7f6c54110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.560+0000 7f6c5c184640 1 --2- 172.21.15.31:0/4216410765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c5410b1d0 0x7f6c5410b5e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.560+0000 7f6c5c184640 1 --2- 172.21.15.31:0/4216410765 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c5410a350 0x7f6c5410ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.560+0000 7f6c5c184640 1 -- 172.21.15.31:0/4216410765 >> 172.21.15.31:0/4216410765 conn(0x7f6c54070390 msgr2=0x7f6c54072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:49.563 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.561+0000 7f6c5c184640 1 -- 172.21.15.31:0/4216410765 shutdown_connections 2024-04-25T16:05:49.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.561+0000 7f6c5c184640 1 -- 172.21.15.31:0/4216410765 wait complete. 2024-04-25T16:05:49.565 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.563+0000 7f6c5c184640 1 Processor -- start 2024-04-25T16:05:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.563+0000 7f6c5c184640 1 -- start start 2024-04-25T16:05:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.564+0000 7f6c5c184640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c5410a350 0x7f6c54117070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.564+0000 7f6c5c184640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c5410b1d0 0x7f6c541175b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.564+0000 7f6c5c184640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c5410fef0 0x7f6c541184c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.564+0000 7f6c5c184640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5411e2e0 con 0x7f6c5410fef0 2024-04-25T16:05:49.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.564+0000 7f6c5c184640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5411e160 con 0x7f6c5410a350 2024-04-25T16:05:49.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.564+0000 7f6c5c184640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6c5411e460 con 0x7f6c5410b1d0 2024-04-25T16:05:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.565+0000 7f6c59f20640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c5410a350 0x7f6c54117070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.565+0000 7f6c5971f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c5410b1d0 0x7f6c541175b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.565+0000 7f6c59f20640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c5410a350 0x7f6c54117070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:41020/0 (socket says 172.21.15.31:41020) 2024-04-25T16:05:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.565+0000 7f6c59f20640 1 -- 172.21.15.31:0/2584585558 learned_addr learned my addr 172.21.15.31:0/2584585558 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:49.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.566+0000 7f6c5a721640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c5410fef0 0x7f6c541184c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.567+0000 7f6c5971f640 1 -- 172.21.15.31:0/2584585558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c5410a350 msgr2=0x7f6c54117070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:49.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.567+0000 7f6c5971f640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c5410a350 0x7f6c54117070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.568+0000 7f6c5971f640 1 -- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c5410fef0 msgr2=0x7f6c541184c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:49.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.568+0000 7f6c5971f640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c5410fef0 0x7f6c541184c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.568+0000 7f6c5971f640 1 -- 172.21.15.31:0/2584585558 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c54118b30 con 0x7f6c5410b1d0 2024-04-25T16:05:49.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.571+0000 7f6c5971f640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c5410b1d0 0x7f6c541175b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6c4402a9f0 tx=0x7f6c44003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:49.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.573+0000 7f6c4affd640 1 -- 172.21.15.31:0/2584585558 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4403f070 con 0x7f6c5410b1d0 2024-04-25T16:05:49.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.573+0000 7f6c5c184640 1 -- 172.21.15.31:0/2584585558 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c54118dc0 con 0x7f6c5410b1d0 2024-04-25T16:05:49.575 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.573+0000 7f6c5c184640 1 -- 172.21.15.31:0/2584585558 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c54119280 con 0x7f6c5410b1d0 2024-04-25T16:05:49.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.574+0000 7f6c4affd640 1 -- 172.21.15.31:0/2584585558 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c44031c50 con 0x7f6c5410b1d0 2024-04-25T16:05:49.576 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.574+0000 7f6c4affd640 1 -- 172.21.15.31:0/2584585558 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c44035030 con 0x7f6c5410b1d0 2024-04-25T16:05:49.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.583+0000 7f6c6303c740 1 Processor -- start 2024-04-25T16:05:49.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.584+0000 7f6c6303c740 1 -- start start 2024-04-25T16:05:49.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.584+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c9221b0 0x55ba2ccc2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.584+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.584+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9014d0 0x55ba2ccc9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.584+0000 7f6c6303c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ba2cc28460 con 0x55ba2c8f92f0 2024-04-25T16:05:49.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.584+0000 7f6c6303c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ba2cbe2590 con 0x55ba2c9221b0 2024-04-25T16:05:49.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.584+0000 7f6c6303c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ba2cc24420 con 0x55ba2c9014d0 2024-04-25T16:05:49.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.585+0000 7f6c59f20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.585+0000 7f6c59f20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46306/0 (socket says 172.21.15.31:46306) 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.585+0000 7f6c59f20640 1 -- 172.21.15.31:0/3922413363 learned_addr learned my addr 172.21.15.31:0/3922413363 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.586+0000 7f6c59f20640 1 -- 172.21.15.31:0/3922413363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9014d0 msgr2=0x55ba2ccc9fa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.586+0000 7f6c59f20640 1 --2- 172.21.15.31:0/3922413363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9014d0 0x55ba2ccc9fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.586+0000 7f6c59f20640 1 -- 172.21.15.31:0/3922413363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c9221b0 msgr2=0x55ba2ccc2b20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.586+0000 7f6c59f20640 1 --2- 172.21.15.31:0/3922413363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c9221b0 0x55ba2ccc2b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.586+0000 7f6c59f20640 1 -- 172.21.15.31:0/3922413363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba2cbef0d0 con 0x55ba2c8f92f0 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.587+0000 7f6c59f20640 1 --2- 172.21.15.31:0/3922413363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc30a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c0114e0 tx=0x7f6c4c0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3429a3f662f1418f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.587+0000 7f6c4affd640 1 -- 172.21.15.31:0/2584585558 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6c44042070 con 0x7f6c5410b1d0 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.588+0000 7f6c3ffff640 1 -- 172.21.15.31:0/3922413363 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c01f070 con 0x55ba2c8f92f0 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.588+0000 7f6c3ffff640 1 -- 172.21.15.31:0/3922413363 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c4c012c20 con 0x55ba2c8f92f0 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.588+0000 7f6c4affd640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c400afcf0 0x7f6c400b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.589+0000 7f6c4affd640 1 -- 172.21.15.31:0/2584585558 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6c440ff560 con 0x7f6c5410b1d0 2024-04-25T16:05:49.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.590+0000 7f6c6303c740 1 -- 172.21.15.31:0/3922413363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 msgr2=0x55ba2ccc30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:49.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.590+0000 7f6c6303c740 1 --2- 172.21.15.31:0/3922413363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc30a0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c0114e0 tx=0x7f6c4c0119a0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.592+0000 7f6c6303c740 1 -- 172.21.15.31:0/3922413363 shutdown_connections 2024-04-25T16:05:49.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.592+0000 7f6c6303c740 1 --2- 172.21.15.31:0/3922413363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9014d0 0x55ba2ccc9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.592+0000 7f6c6303c740 1 --2- 172.21.15.31:0/3922413363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc30a0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.592+0000 7f6c6303c740 1 --2- 172.21.15.31:0/3922413363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c9221b0 0x55ba2ccc2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.592+0000 7f6c6303c740 1 -- 172.21.15.31:0/3922413363 >> 172.21.15.31:0/3922413363 conn(0x55ba2c916300 msgr2=0x55ba2ccbe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:49.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.593+0000 7f6c6303c740 1 -- 172.21.15.31:0/3922413363 shutdown_connections 2024-04-25T16:05:49.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.596+0000 7f6c5a721640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c400afcf0 0x7f6c400b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.596+0000 7f6c6303c740 1 -- 172.21.15.31:0/3922413363 wait complete. 2024-04-25T16:05:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.596+0000 7f6c6303c740 1 -- start start 2024-04-25T16:05:49.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.597+0000 7f6c6303c740 1 Processor -- start 2024-04-25T16:05:49.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.597+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c8f92f0 0x55ba2cccc870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.597+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c9014d0 0x55ba2ccccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.597+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.597+0000 7f6c6303c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ba2cccaa40 con 0x55ba2c9221b0 2024-04-25T16:05:49.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.597+0000 7f6c6303c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ba2cbafb60 con 0x55ba2c9014d0 2024-04-25T16:05:49.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.597+0000 7f6c6303c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ba2cccabc0 con 0x55ba2c8f92f0 2024-04-25T16:05:49.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.598+0000 7f6c6303c740 10 client.0 _ll_register_callbacks cb 0x55ba2cbd42b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.598+0000 7f6c5a721640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.598+0000 7f6c5a721640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46322/0 (socket says 172.21.15.31:46322) 2024-04-25T16:05:49.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.598+0000 7f6c5a721640 1 -- 172.21.15.31:0/1952701589 learned_addr learned my addr 172.21.15.31:0/1952701589 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:49.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.598+0000 7f6c5a721640 1 -- 172.21.15.31:0/1952701589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c8f92f0 msgr2=0x55ba2cccc870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:49.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.598+0000 7f6c5a721640 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c8f92f0 0x55ba2cccc870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.598+0000 7f6c5a721640 1 -- 172.21.15.31:0/1952701589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c9014d0 msgr2=0x55ba2ccccdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:05:49.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.598+0000 7f6c5a721640 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c9014d0 0x55ba2ccccdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.599+0000 7f6c5a721640 1 -- 172.21.15.31:0/1952701589 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba2c93a210 con 0x55ba2c9221b0 2024-04-25T16:05:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.606+0000 7f6c5a721640 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd3d10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6c50013db0 tx=0x7f6c5000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9026931b686ff7db server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.606+0000 7f6c3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.607+0000 7f6c5a721640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c400afcf0 0x7f6c400b2200 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6c5000fc90 tx=0x7f6c50016000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.607+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/1952701589 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c50024060 con 0x55ba2c9221b0 2024-04-25T16:05:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.607+0000 7f6c6303c740 1 -- 172.21.15.31:0/1952701589 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6c4c012c20 con 0x55ba2c9221b0 2024-04-25T16:05:49.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.607+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/1952701589 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6c50005050 con 0x55ba2c9221b0 2024-04-25T16:05:49.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.615+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/1952701589 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6c50005250 con 0x55ba2c9221b0 2024-04-25T16:05:49.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.615+0000 7f6c6303c740 10 client.16563 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:49.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.615+0000 7f6c6303c740 1 -- 172.21.15.31:0/1952701589 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ba2ca06620 con 0x55ba2c9221b0 2024-04-25T16:05:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.618+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/1952701589 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c4c012c20 con 0x55ba2c9221b0 2024-04-25T16:05:49.620 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.618+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/1952701589 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6c50028440 con 0x55ba2c9221b0 2024-04-25T16:05:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.618+0000 7f6c6303c740 10 client.16563 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.618+0000 7f6c6303c740 10 client.16563 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.618+0000 7f6c6303c740 20 client.16563 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:49.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.618+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ba2c8e0d80 0x55ba2ccdab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.618+0000 7f6c6303c740 4 client.16563 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:05:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.618+0000 7f6c6303c740 1 -- 172.21.15.31:0/1952701589 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba2cc36070 con 0x55ba2c8e0d80 2024-04-25T16:05:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.619+0000 7f6c59f20640 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ba2c8e0d80 0x55ba2ccdab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.620+0000 7f6c59f20640 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ba2c8e0d80 0x55ba2ccdab20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eccdbb93ba9de8e server_cookie=5b1fb0408e5b5836 in_seq=0 out_seq=0 2024-04-25T16:05:49.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.621+0000 7f6c3d7fa640 10 client.16563 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:49.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.624+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/1952701589 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ba2cc36070 con 0x55ba2c8e0d80 2024-04-25T16:05:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.625+0000 7f6c3d7fa640 10 client.16563 handle_command_reply: tid=0 2024-04-25T16:05:49.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.625+0000 7f6c6303c740 1 client.16563 shutdown 2024-04-25T16:05:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c6303c740 1 -- 172.21.15.31:0/1952701589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9221b0 msgr2=0x55ba2ccd3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd3d10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6c50013db0 tx=0x7f6c5000f790 comp rx=0 tx=0).stop 2024-04-25T16:05:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c6303c740 1 -- 172.21.15.31:0/1952701589 shutdown_connections 2024-04-25T16:05:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ba2c8e0d80 0x55ba2ccdab20 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd3d10 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c9014d0 0x55ba2ccccdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1952701589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c8f92f0 0x55ba2cccc870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c6303c740 1 -- 172.21.15.31:0/1952701589 >> 172.21.15.31:0/1952701589 conn(0x55ba2c916300 msgr2=0x55ba2ccbcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c3d7fa640 0 client.16563 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:49.639 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.636+0000 7f6c6303c740 1 -- 172.21.15.31:0/1952701589 shutdown_connections 2024-04-25T16:05:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.637+0000 7f6c6303c740 1 -- 172.21.15.31:0/1952701589 wait complete. 2024-04-25T16:05:49.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.637+0000 7f6c6303c740 20 client.16563 trim_cache size 0 max 16384 2024-04-25T16:05:49.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.728+0000 7f6c6303c740 1 Processor -- start 2024-04-25T16:05:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.728+0000 7f6c6303c740 1 -- start start 2024-04-25T16:05:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.728+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.729+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9014d0 0x55ba2ccd3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.736 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.731+0000 7f6c5971f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9014d0 0x55ba2ccd3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.731+0000 7f6c5971f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9014d0 0x55ba2ccd3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46330/0 (socket says 172.21.15.31:46330) 2024-04-25T16:05:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.734+0000 7f6c5971f640 1 -- 172.21.15.31:0/1136172919 learned_addr learned my addr 172.21.15.31:0/1136172919 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.734+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 0x55ba2ccdf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.734+0000 7f6c6303c740 1 -- 172.21.15.31:0/1136172919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3800e590 con 0x55ba2c9014d0 2024-04-25T16:05:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.734+0000 7f6c6303c740 1 -- 172.21.15.31:0/1136172919 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6c380100d0 con 0x55ba2c8f92f0 2024-04-25T16:05:49.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.734+0000 7f6c6303c740 1 -- 172.21.15.31:0/1136172919 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3800ec60 con 0x55ba2c9221b0 2024-04-25T16:05:49.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.736+0000 7f6c59f20640 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.737+0000 7f6c5a721640 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 0x55ba2ccdf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.737+0000 7f6c5971f640 1 -- 172.21.15.31:0/1136172919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 msgr2=0x55ba2ccdf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.737+0000 7f6c5971f640 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 0x55ba2ccdf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.737+0000 7f6c5971f640 1 -- 172.21.15.31:0/1136172919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8f92f0 msgr2=0x55ba2ccc2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.737+0000 7f6c5971f640 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.737+0000 7f6c5971f640 1 -- 172.21.15.31:0/1136172919 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba2ccd4170 con 0x55ba2c9014d0 2024-04-25T16:05:49.743 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.740+0000 7f6c5971f640 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9014d0 0x55ba2ccd3c30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6c440c4bd0 tx=0x7f6c44031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c188b16ee45ba97 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.743+0000 7f6c3f7fe640 1 -- 172.21.15.31:0/1136172919 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c44049070 con 0x55ba2c9014d0 2024-04-25T16:05:49.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.743+0000 7f6c3f7fe640 1 -- 172.21.15.31:0/1136172919 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c440c7500 con 0x55ba2c9014d0 2024-04-25T16:05:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.745+0000 7f6c6303c740 1 -- 172.21.15.31:0/1136172919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9014d0 msgr2=0x55ba2ccd3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:49.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.745+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9014d0 0x55ba2ccd3c30 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f6c440c4bd0 tx=0x7f6c44031400 comp rx=0 tx=0).stop 2024-04-25T16:05:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.746+0000 7f6c6303c740 1 -- 172.21.15.31:0/1136172919 shutdown_connections 2024-04-25T16:05:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.746+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 0x55ba2ccdf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.746+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c9014d0 0x55ba2ccd3c30 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.746+0000 7f6c6303c740 1 --2- 172.21.15.31:0/1136172919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccc2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.746+0000 7f6c6303c740 1 -- 172.21.15.31:0/1136172919 >> 172.21.15.31:0/1136172919 conn(0x55ba2c916300 msgr2=0x55ba2ccc6bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.749+0000 7f6c6303c740 1 -- 172.21.15.31:0/1136172919 shutdown_connections 2024-04-25T16:05:49.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.749+0000 7f6c6303c740 1 -- 172.21.15.31:0/1136172919 wait complete. 2024-04-25T16:05:49.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.750+0000 7f6c6303c740 1 -- start start 2024-04-25T16:05:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.751+0000 7f6c6303c740 1 Processor -- start 2024-04-25T16:05:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.752+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccd0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.753+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.753+0000 7f6c6303c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8b61c0 0x55ba2ccd2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.754+0000 7f6c6303c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3800fa00 con 0x55ba2c8f92f0 2024-04-25T16:05:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.754+0000 7f6c6303c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3800f330 con 0x55ba2c8b61c0 2024-04-25T16:05:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.754+0000 7f6c6303c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6c3800c570 con 0x55ba2c9221b0 2024-04-25T16:05:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.754+0000 7f6c6303c740 10 client.0 _ll_register_callbacks cb 0x7f6c50009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:05:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.755+0000 7f6c59f20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccd0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.755+0000 7f6c59f20640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccd0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:46334/0 (socket says 172.21.15.31:46334) 2024-04-25T16:05:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.755+0000 7f6c59f20640 1 -- 172.21.15.31:0/2092106860 learned_addr learned my addr 172.21.15.31:0/2092106860 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:05:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.756+0000 7f6c5a721640 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8b61c0 0x55ba2ccd2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.756+0000 7f6c5971f640 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.756+0000 7f6c59f20640 1 -- 172.21.15.31:0/2092106860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 msgr2=0x55ba2ccd1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.756+0000 7f6c59f20640 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.757+0000 7f6c59f20640 1 -- 172.21.15.31:0/2092106860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8b61c0 msgr2=0x55ba2ccd2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.757+0000 7f6c59f20640 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8b61c0 0x55ba2ccd2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.757+0000 7f6c59f20640 1 -- 172.21.15.31:0/2092106860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba2ccd2910 con 0x55ba2c8f92f0 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.758+0000 7f6c59f20640 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccd0f30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c0119e0 tx=0x7f6c4c01a440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcda750ae10d4516 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.758+0000 7f6c3d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.759+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/2092106860 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c01f040 con 0x55ba2c8f92f0 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.759+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/2092106860 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6c4c0098e0 con 0x55ba2c8f92f0 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.759+0000 7f6c6303c740 1 -- 172.21.15.31:0/2092106860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ba2cc2fee0 con 0x55ba2c8f92f0 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.760+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/2092106860 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6c4c035050 con 0x55ba2c8f92f0 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.760+0000 7f6c6303c740 10 client.16569 fetch_fsmap learned FSMap version 29 2024-04-25T16:05:49.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.760+0000 7f6c6303c740 1 -- 172.21.15.31:0/2092106860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ba2ccd3c90 con 0x55ba2c8f92f0 2024-04-25T16:05:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.761+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/2092106860 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c4c02a070 con 0x55ba2c8f92f0 2024-04-25T16:05:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.762+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/2092106860 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6c4c012a80 con 0x55ba2c8f92f0 2024-04-25T16:05:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.763+0000 7f6c6303c740 10 client.16569 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:05:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.763+0000 7f6c6303c740 10 client.16569 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:05:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.764+0000 7f6c6303c740 20 client.16569 populate_metadata read hostname 'smithi031' 2024-04-25T16:05:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.764+0000 7f6c6303c740 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ba2cd433d0 0x55ba2cd457f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:05:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.764+0000 7f6c6303c740 4 client.16569 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:05:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.764+0000 7f6c6303c740 1 -- 172.21.15.31:0/2092106860 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ba2cc2fee0 con 0x55ba2cd433d0 2024-04-25T16:05:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.766+0000 7f6c5971f640 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ba2cd433d0 0x55ba2cd457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:05:49.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.788+0000 7f6c5971f640 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ba2cd433d0 0x55ba2cd457f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9d2abe1a4dd7331 server_cookie=3cad1cea1fd71cd3 in_seq=0 out_seq=0 2024-04-25T16:05:49.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:49.790+0000 7f6c3d7fa640 10 client.16569 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:51.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:50 smithi063 ceph-mon[32413]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 792 MiB/s wr, 1.53k op/s 2024-04-25T16:05:51.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:50 smithi186 ceph-mon[33258]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 792 MiB/s wr, 1.53k op/s 2024-04-25T16:05:51.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:50 smithi031 ceph-mon[29140]: pgmap v487: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 792 MiB/s wr, 1.53k op/s 2024-04-25T16:05:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:53 smithi063 ceph-mon[32413]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 743 MiB/s wr, 1.42k op/s 2024-04-25T16:05:53.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:53 smithi186 ceph-mon[33258]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 743 MiB/s wr, 1.42k op/s 2024-04-25T16:05:53.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:53 smithi031 ceph-mon[29140]: pgmap v488: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 743 MiB/s wr, 1.42k op/s 2024-04-25T16:05:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.046+0000 7f6c3d7fa640 1 -- 172.21.15.31:0/2092106860 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ba2cc2fee0 con 0x55ba2cd433d0 2024-04-25T16:05:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.046+0000 7f6c3d7fa640 10 client.16569 handle_command_reply: tid=0 2024-04-25T16:05:54.049 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.047+0000 7f6c6303c740 1 client.16569 shutdown 2024-04-25T16:05:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.047+0000 7f6c6303c740 1 -- 172.21.15.31:0/2092106860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 msgr2=0x55ba2ccd0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.047+0000 7f6c6303c740 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccd0f30 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c0119e0 tx=0x7f6c4c01a440 comp rx=0 tx=0).stop 2024-04-25T16:05:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.048+0000 7f6c6303c740 1 -- 172.21.15.31:0/2092106860 shutdown_connections 2024-04-25T16:05:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.048+0000 7f6c6303c740 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55ba2cd433d0 0x55ba2cd457f0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.048+0000 7f6c6303c740 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ba2c8b61c0 0x55ba2ccd2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.048+0000 7f6c6303c740 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ba2c9221b0 0x55ba2ccd1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.048+0000 7f6c6303c740 1 --2- 172.21.15.31:0/2092106860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ba2c8f92f0 0x55ba2ccd0f30 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.048+0000 7f6c6303c740 1 -- 172.21.15.31:0/2092106860 >> 172.21.15.31:0/2092106860 conn(0x55ba2c916300 msgr2=0x55ba2ccc12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.048+0000 7f6c3d7fa640 0 client.16569 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:05:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.049+0000 7f6c6303c740 1 -- 172.21.15.31:0/2092106860 shutdown_connections 2024-04-25T16:05:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.049+0000 7f6c6303c740 1 -- 172.21.15.31:0/2092106860 wait complete. 2024-04-25T16:05:54.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.049+0000 7f6c6303c740 20 client.16569 trim_cache size 0 max 16384 2024-04-25T16:05:54.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:05:54.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:05:54.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:05:54.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:05:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.065+0000 7f6c5c184640 1 -- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c400afcf0 msgr2=0x7f6c400b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.065+0000 7f6c5c184640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c400afcf0 0x7f6c400b2200 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6c5000fc90 tx=0x7f6c50016000 comp rx=0 tx=0).stop 2024-04-25T16:05:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.065+0000 7f6c5c184640 1 -- 172.21.15.31:0/2584585558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c5410b1d0 msgr2=0x7f6c541175b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:05:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.065+0000 7f6c5c184640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c5410b1d0 0x7f6c541175b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6c4402a9f0 tx=0x7f6c44003b90 comp rx=0 tx=0).stop 2024-04-25T16:05:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.065+0000 7f6c5c184640 1 -- 172.21.15.31:0/2584585558 shutdown_connections 2024-04-25T16:05:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.066+0000 7f6c5c184640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6c400afcf0 0x7f6c400b2200 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.066+0000 7f6c5c184640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6c5410fef0 0x7f6c541184c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.066+0000 7f6c5c184640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c5410b1d0 0x7f6c541175b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:54.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.066+0000 7f6c5c184640 1 --2- 172.21.15.31:0/2584585558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c5410a350 0x7f6c54117070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:05:54.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.066+0000 7f6c5c184640 1 -- 172.21.15.31:0/2584585558 >> 172.21.15.31:0/2584585558 conn(0x7f6c54070390 msgr2=0x7f6c5410bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:05:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.070+0000 7f6c5c184640 1 -- 172.21.15.31:0/2584585558 shutdown_connections 2024-04-25T16:05:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:05:54.070+0000 7f6c5c184640 1 -- 172.21.15.31:0/2584585558 wait complete. 2024-04-25T16:05:54.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:54 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:05:54.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:54 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:05:54.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:54 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:05:55.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:55 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:05:55.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:55 smithi063 ceph-mon[32413]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 957 MiB/s wr, 1.89k op/s 2024-04-25T16:05:55.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:55 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:05:55.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:55 smithi186 ceph-mon[33258]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 957 MiB/s wr, 1.89k op/s 2024-04-25T16:05:55.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:55 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:05:55.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:55 smithi031 ceph-mon[29140]: pgmap v489: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 957 MiB/s wr, 1.89k op/s 2024-04-25T16:05:57.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:57 smithi063 ceph-mon[32413]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 803 MiB/s wr, 1.67k op/s 2024-04-25T16:05:57.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:57 smithi186 ceph-mon[33258]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 803 MiB/s wr, 1.67k op/s 2024-04-25T16:05:57.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:57 smithi031 ceph-mon[29140]: pgmap v490: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 803 MiB/s wr, 1.67k op/s 2024-04-25T16:05:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:05:59 smithi063 ceph-mon[32413]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 887 MiB/s wr, 2.13k op/s 2024-04-25T16:05:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:05:59 smithi186 ceph-mon[33258]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 887 MiB/s wr, 2.13k op/s 2024-04-25T16:05:59.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:05:59 smithi031 ceph-mon[29140]: pgmap v491: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 887 MiB/s wr, 2.13k op/s 2024-04-25T16:06:01.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:01 smithi063 ceph-mon[32413]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 806 MiB/s wr, 2.33k op/s 2024-04-25T16:06:01.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:01 smithi186 ceph-mon[33258]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 806 MiB/s wr, 2.33k op/s 2024-04-25T16:06:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:01 smithi031 ceph-mon[29140]: pgmap v492: 97 pgs: 97 active+clean; 10 GiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 806 MiB/s wr, 2.33k op/s 2024-04-25T16:06:03.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:03 smithi063 ceph-mon[32413]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 2.37k op/s 2024-04-25T16:06:03.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:03 smithi186 ceph-mon[33258]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 2.37k op/s 2024-04-25T16:06:03.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:03 smithi031 ceph-mon[29140]: pgmap v493: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 2.37k op/s 2024-04-25T16:06:05.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:05 smithi063 ceph-mon[32413]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 924 MiB/s wr, 3.47k op/s 2024-04-25T16:06:05.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:05 smithi186 ceph-mon[33258]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 924 MiB/s wr, 3.47k op/s 2024-04-25T16:06:05.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:05 smithi031 ceph-mon[29140]: pgmap v494: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 924 MiB/s wr, 3.47k op/s 2024-04-25T16:06:07.579 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.578+0000 7f35627fc640 1 -- 172.21.15.31:0/364302577 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554035020 con 0x7f356410b240 2024-04-25T16:06:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.580+0000 7f356a0e0640 1 -- 172.21.15.31:0/364302577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410b240 msgr2=0x7f356410b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:07.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.580+0000 7f356a0e0640 1 --2- 172.21.15.31:0/364302577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410b240 0x7f356410b650 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3554009900 tx=0x7f355402a440 comp rx=0 tx=0).stop 2024-04-25T16:06:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.581+0000 7f356a0e0640 1 -- 172.21.15.31:0/364302577 shutdown_connections 2024-04-25T16:06:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.581+0000 7f356a0e0640 1 --2- 172.21.15.31:0/364302577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f356410bb90 0x7f356410c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.581+0000 7f356a0e0640 1 --2- 172.21.15.31:0/364302577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410b240 0x7f356410b650 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.581+0000 7f356a0e0640 1 --2- 172.21.15.31:0/364302577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f356410a120 0x7f356410a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.581+0000 7f356a0e0640 1 -- 172.21.15.31:0/364302577 >> 172.21.15.31:0/364302577 conn(0x7f3564070960 msgr2=0x7f3564072e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.588+0000 7f356a0e0640 1 -- 172.21.15.31:0/364302577 shutdown_connections 2024-04-25T16:06:07.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.589+0000 7f356a0e0640 1 -- 172.21.15.31:0/364302577 wait complete. 2024-04-25T16:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.591+0000 7f356a0e0640 1 Processor -- start 2024-04-25T16:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.591+0000 7f356a0e0640 1 -- start start 2024-04-25T16:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.592+0000 7f356a0e0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410a120 0x7f356411bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.592+0000 7f356a0e0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f356410b240 0x7f356411c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.592+0000 7f356a0e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f356410bb90 0x7f3564121190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.592+0000 7f356a0e0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3564115d20 con 0x7f356410a120 2024-04-25T16:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.592+0000 7f356a0e0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3564115ba0 con 0x7f356410b240 2024-04-25T16:06:07.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.592+0000 7f356a0e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3564115ea0 con 0x7f356410bb90 2024-04-25T16:06:07.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.594+0000 7f35637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410a120 0x7f356411bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.594+0000 7f35637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410a120 0x7f356411bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59432/0 (socket says 172.21.15.31:59432) 2024-04-25T16:06:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.594+0000 7f35637fe640 1 -- 172.21.15.31:0/3204241872 learned_addr learned my addr 172.21.15.31:0/3204241872 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.594+0000 7f3563fff640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f356410bb90 0x7f3564121190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.594+0000 7f3562ffd640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f356410b240 0x7f356411c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.595+0000 7f35637fe640 1 -- 172.21.15.31:0/3204241872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f356410bb90 msgr2=0x7f3564121190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.595+0000 7f35637fe640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f356410bb90 0x7f3564121190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.595+0000 7f35637fe640 1 -- 172.21.15.31:0/3204241872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f356410b240 msgr2=0x7f356411c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.595+0000 7f35637fe640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f356410b240 0x7f356411c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.595+0000 7f35637fe640 1 -- 172.21.15.31:0/3204241872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3564121800 con 0x7f356410a120 2024-04-25T16:06:07.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.601+0000 7f35637fe640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410a120 0x7f356411bc50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f355c00b990 tx=0x7f355c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:07.610 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.609+0000 7f3560ff9640 1 -- 172.21.15.31:0/3204241872 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c019070 con 0x7f356410a120 2024-04-25T16:06:07.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.609+0000 7f3560ff9640 1 -- 172.21.15.31:0/3204241872 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f355c00cba0 con 0x7f356410a120 2024-04-25T16:06:07.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.609+0000 7f3560ff9640 1 -- 172.21.15.31:0/3204241872 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355c0125a0 con 0x7f356410a120 2024-04-25T16:06:07.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.610+0000 7f356a0e0640 1 -- 172.21.15.31:0/3204241872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3564121af0 con 0x7f356410a120 2024-04-25T16:06:07.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.610+0000 7f356a0e0640 1 -- 172.21.15.31:0/3204241872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3564122070 con 0x7f356410a120 2024-04-25T16:06:07.621 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:07 smithi063 ceph-mon[32413]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 734 MiB/s wr, 3.40k op/s 2024-04-25T16:06:07.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.634+0000 7f3570f94740 1 Processor -- start 2024-04-25T16:06:07.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.634+0000 7f3570f94740 1 -- start start 2024-04-25T16:06:07.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.640+0000 7f3560ff9640 1 -- 172.21.15.31:0/3204241872 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f355c01c070 con 0x7f356410a120 2024-04-25T16:06:07.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.642+0000 7f3570f94740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd1fc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.643 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.642+0000 7f3570f94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 0x5559cd1fcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.643+0000 7f3560ff9640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f354c0afce0 0x7f354c0b21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.643+0000 7f3560ff9640 1 -- 172.21.15.31:0/3204241872 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f355c0de1c0 con 0x7f356410a120 2024-04-25T16:06:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.644+0000 7f3563fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd1fc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.644+0000 7f3563fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd1fc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49472/0 (socket says 172.21.15.31:49472) 2024-04-25T16:06:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.644+0000 7f3563fff640 1 -- 172.21.15.31:0/4055043696 learned_addr learned my addr 172.21.15.31:0/4055043696 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.645+0000 7f3570f94740 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce3b4d0 0x5559cd203df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.645+0000 7f3570f94740 1 -- 172.21.15.31:0/4055043696 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5559cd162460 con 0x5559cce332f0 2024-04-25T16:06:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.645+0000 7f3570f94740 1 -- 172.21.15.31:0/4055043696 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5559cd11c590 con 0x5559cce5c1b0 2024-04-25T16:06:07.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.645+0000 7f3570f94740 1 -- 172.21.15.31:0/4055043696 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5559cd15e420 con 0x5559cce3b4d0 2024-04-25T16:06:07.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:07 smithi186 ceph-mon[33258]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 734 MiB/s wr, 3.40k op/s 2024-04-25T16:06:07.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.650+0000 7f3563fff640 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce3b4d0 0x5559cd203df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.650+0000 7f35637fe640 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 0x5559cd1fcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.651 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.650+0000 7f3562ffd640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f354c0afce0 0x7f354c0b21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.655+0000 7f3563fff640 1 -- 172.21.15.31:0/4055043696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce3b4d0 msgr2=0x5559cd203df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.655+0000 7f3563fff640 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce3b4d0 0x5559cd203df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.655+0000 7f3563fff640 1 -- 172.21.15.31:0/4055043696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 msgr2=0x5559cd1fcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.655+0000 7f3563fff640 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 0x5559cd1fcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.657 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.655+0000 7f3563fff640 1 -- 172.21.15.31:0/4055043696 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559cd1290d0 con 0x5559cce5c1b0 2024-04-25T16:06:07.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.656+0000 7f3563fff640 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd1fc970 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f355800aeb0 tx=0x7f355800a510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a070b67f0dfa6f79 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.659+0000 7f3549ffb640 1 -- 172.21.15.31:0/4055043696 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558004220 con 0x5559cce5c1b0 2024-04-25T16:06:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.659+0000 7f3549ffb640 1 -- 172.21.15.31:0/4055043696 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f355800a2a0 con 0x5559cce5c1b0 2024-04-25T16:06:07.660 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.659+0000 7f3549ffb640 1 -- 172.21.15.31:0/4055043696 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35580137c0 con 0x5559cce5c1b0 2024-04-25T16:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.661+0000 7f3570f94740 1 -- 172.21.15.31:0/4055043696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 msgr2=0x5559cd1fc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:07.662 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.661+0000 7f3570f94740 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd1fc970 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f355800aeb0 tx=0x7f355800a510 comp rx=0 tx=0).stop 2024-04-25T16:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.662+0000 7f3570f94740 1 -- 172.21.15.31:0/4055043696 shutdown_connections 2024-04-25T16:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.662+0000 7f3570f94740 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce3b4d0 0x5559cd203df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.662+0000 7f3570f94740 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 0x5559cd1fcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.662+0000 7f3570f94740 1 --2- 172.21.15.31:0/4055043696 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd1fc970 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.664 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.662+0000 7f3570f94740 1 -- 172.21.15.31:0/4055043696 >> 172.21.15.31:0/4055043696 conn(0x5559cce50300 msgr2=0x5559cd1f8300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:07.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.667+0000 7f3570f94740 1 -- 172.21.15.31:0/4055043696 shutdown_connections 2024-04-25T16:06:07.669 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.668+0000 7f3570f94740 1 -- 172.21.15.31:0/4055043696 wait complete. 2024-04-25T16:06:07.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.669+0000 7f3570f94740 1 -- start start 2024-04-25T16:06:07.670 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.670+0000 7f3570f94740 1 Processor -- start 2024-04-25T16:06:07.671 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.670+0000 7f3570f94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 0x5559cd206770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.671+0000 7f3570f94740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.671+0000 7f3570f94740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 0x5559cd20dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.671+0000 7f3570f94740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5559cd204830 con 0x5559cce332f0 2024-04-25T16:06:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.671+0000 7f3570f94740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5559cd0e9b60 con 0x5559cce3b4d0 2024-04-25T16:06:07.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.671+0000 7f3570f94740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5559cd2049b0 con 0x5559cce5c1b0 2024-04-25T16:06:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.672+0000 7f3570f94740 10 client.0 _ll_register_callbacks cb 0x5559cd10e2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.672+0000 7f3563fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 0x5559cd20dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.672+0000 7f3563fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 0x5559cd20dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:40384/0 (socket says 172.21.15.31:40384) 2024-04-25T16:06:07.674 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.672+0000 7f3563fff640 1 -- 172.21.15.31:0/877455557 learned_addr learned my addr 172.21.15.31:0/877455557 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.674+0000 7f3562ffd640 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.677 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.676+0000 7f35637fe640 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 0x5559cd206770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.677+0000 7f3563fff640 1 -- 172.21.15.31:0/877455557 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce3b4d0 msgr2=0x5559cd206cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.677+0000 7f3563fff640 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.677+0000 7f3563fff640 1 -- 172.21.15.31:0/877455557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 msgr2=0x5559cd206770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.677+0000 7f3563fff640 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 0x5559cd206770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.677+0000 7f3563fff640 1 -- 172.21.15.31:0/877455557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559cce74210 con 0x5559cce5c1b0 2024-04-25T16:06:07.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.679+0000 7f3562ffd640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f354c0afce0 0x7f354c0b21f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f35540057c0 tx=0x7f3554031500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:07.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.680+0000 7f3563fff640 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 0x5559cd20dc10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3558004280 tx=0x7f35580042b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91ad63c3562e41fc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:07.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.686+0000 7f3542ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:07.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.686+0000 7f3542ffd640 1 -- 172.21.15.31:0/877455557 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3558028020 con 0x5559cce5c1b0 2024-04-25T16:06:07.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.687+0000 7f3570f94740 1 -- 172.21.15.31:0/877455557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f355800a2a0 con 0x5559cce5c1b0 2024-04-25T16:06:07.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.687+0000 7f3542ffd640 1 -- 172.21.15.31:0/877455557 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3558006d50 con 0x5559cce5c1b0 2024-04-25T16:06:07.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.690+0000 7f3542ffd640 1 -- 172.21.15.31:0/877455557 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3558013780 con 0x5559cce5c1b0 2024-04-25T16:06:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.693+0000 7f3570f94740 10 client.26203 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:07.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.693+0000 7f3570f94740 1 -- 172.21.15.31:0/877455557 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5559ccf40620 con 0x5559cce5c1b0 2024-04-25T16:06:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.697+0000 7f3542ffd640 1 -- 172.21.15.31:0/877455557 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f355800a2a0 con 0x5559cce5c1b0 2024-04-25T16:06:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.697+0000 7f3542ffd640 1 -- 172.21.15.31:0/877455557 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3558015040 con 0x5559cce5c1b0 2024-04-25T16:06:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.699+0000 7f3570f94740 10 client.26203 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:07.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.699+0000 7f3570f94740 10 client.26203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.699+0000 7f3570f94740 20 client.26203 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.699+0000 7f3570f94740 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5559cce1ad80 0x5559cd213ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.699+0000 7f3570f94740 4 client.26203 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.699+0000 7f3570f94740 1 -- 172.21.15.31:0/877455557 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559cd170070 con 0x5559cce1ad80 2024-04-25T16:06:07.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:07 smithi031 ceph-mon[29140]: pgmap v495: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 734 MiB/s wr, 3.40k op/s 2024-04-25T16:06:07.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.700+0000 7f35637fe640 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5559cce1ad80 0x5559cd213ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.701+0000 7f35637fe640 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5559cce1ad80 0x5559cd213ef0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4fbdf5314407cf server_cookie=285d24be54ba859f in_seq=0 out_seq=0 2024-04-25T16:06:07.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.701+0000 7f3542ffd640 10 client.26203 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:07.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.705+0000 7f3542ffd640 1 -- 172.21.15.31:0/877455557 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5559cd170070 con 0x5559cce1ad80 2024-04-25T16:06:07.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.705+0000 7f3542ffd640 10 client.26203 handle_command_reply: tid=0 2024-04-25T16:06:07.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.705+0000 7f3570f94740 1 client.26203 shutdown 2024-04-25T16:06:07.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.713+0000 7f3570f94740 1 -- 172.21.15.31:0/877455557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 msgr2=0x5559cd20dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:07.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.713+0000 7f3570f94740 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 0x5559cd20dc10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3558004280 tx=0x7f35580042b0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.717+0000 7f3570f94740 1 -- 172.21.15.31:0/877455557 shutdown_connections 2024-04-25T16:06:07.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.717+0000 7f3570f94740 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5559cce1ad80 0x5559cd213ef0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.717+0000 7f3570f94740 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 0x5559cd20dc10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.717+0000 7f3570f94740 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.717+0000 7f3570f94740 1 --2- 172.21.15.31:0/877455557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce332f0 0x5559cd206770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.717+0000 7f3570f94740 1 -- 172.21.15.31:0/877455557 >> 172.21.15.31:0/877455557 conn(0x5559cce50300 msgr2=0x5559cd1f7250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.717+0000 7f3542ffd640 0 client.26203 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.717+0000 7f3570f94740 1 -- 172.21.15.31:0/877455557 shutdown_connections 2024-04-25T16:06:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.717+0000 7f3570f94740 1 -- 172.21.15.31:0/877455557 wait complete. 2024-04-25T16:06:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.724+0000 7f3570f94740 20 client.26203 trim_cache size 0 max 16384 2024-04-25T16:06:07.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.824+0000 7f3570f94740 1 Processor -- start 2024-04-25T16:06:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.824+0000 7f3570f94740 1 -- start start 2024-04-25T16:06:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.824+0000 7f3570f94740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce332f0 0x5559cd206770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.824+0000 7f3570f94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.825+0000 7f3570f94740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd20db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.825+0000 7f3570f94740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f355000e550 con 0x5559cce3b4d0 2024-04-25T16:06:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.825+0000 7f3570f94740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3550010090 con 0x5559cce5c1b0 2024-04-25T16:06:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.825+0000 7f3570f94740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f355000ec20 con 0x5559cce332f0 2024-04-25T16:06:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.825+0000 7f3562ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.825+0000 7f3562ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59464/0 (socket says 172.21.15.31:59464) 2024-04-25T16:06:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.826+0000 7f3562ffd640 1 -- 172.21.15.31:0/2845792538 learned_addr learned my addr 172.21.15.31:0/2845792538 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:07.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.826+0000 7f35637fe640 1 --2- 172.21.15.31:0/2845792538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce332f0 0x5559cd206770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.826+0000 7f3563fff640 1 --2- 172.21.15.31:0/2845792538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd20db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.827+0000 7f3562ffd640 1 -- 172.21.15.31:0/2845792538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce332f0 msgr2=0x5559cd206770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.827+0000 7f3562ffd640 1 --2- 172.21.15.31:0/2845792538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce332f0 0x5559cd206770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.827+0000 7f3562ffd640 1 -- 172.21.15.31:0/2845792538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 msgr2=0x5559cd20db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.827+0000 7f3562ffd640 1 --2- 172.21.15.31:0/2845792538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd20db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.827+0000 7f3562ffd640 1 -- 172.21.15.31:0/2845792538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559cd20e070 con 0x5559cce3b4d0 2024-04-25T16:06:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.828+0000 7f3562ffd640 1 --2- 172.21.15.31:0/2845792538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3554031650 tx=0x7f3554043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160b324d19fa4985 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.829+0000 7f35497fa640 1 -- 172.21.15.31:0/2845792538 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554035050 con 0x5559cce3b4d0 2024-04-25T16:06:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.829+0000 7f35497fa640 1 -- 172.21.15.31:0/2845792538 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3554031e20 con 0x5559cce3b4d0 2024-04-25T16:06:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.829+0000 7f35497fa640 1 -- 172.21.15.31:0/2845792538 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355403b850 con 0x5559cce3b4d0 2024-04-25T16:06:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.831+0000 7f3570f94740 1 -- 172.21.15.31:0/2845792538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 msgr2=0x5559cd206cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.831+0000 7f3570f94740 1 --2- 172.21.15.31:0/2845792538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f3554031650 tx=0x7f3554043000 comp rx=0 tx=0).stop 2024-04-25T16:06:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.841+0000 7f3570f94740 1 -- 172.21.15.31:0/2845792538 shutdown_connections 2024-04-25T16:06:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.842+0000 7f3570f94740 1 --2- 172.21.15.31:0/2845792538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce5c1b0 0x5559cd20db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.842+0000 7f3570f94740 1 --2- 172.21.15.31:0/2845792538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd206cb0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.842+0000 7f3570f94740 1 --2- 172.21.15.31:0/2845792538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce332f0 0x5559cd206770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.842+0000 7f3570f94740 1 -- 172.21.15.31:0/2845792538 >> 172.21.15.31:0/2845792538 conn(0x5559cce50300 msgr2=0x5559cd200b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.843+0000 7f3570f94740 1 -- 172.21.15.31:0/2845792538 shutdown_connections 2024-04-25T16:06:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.843+0000 7f3570f94740 1 -- 172.21.15.31:0/2845792538 wait complete. 2024-04-25T16:06:07.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.845+0000 7f3570f94740 1 -- start start 2024-04-25T16:06:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.846+0000 7f3570f94740 1 Processor -- start 2024-04-25T16:06:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.848+0000 7f3570f94740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce332f0 0x5559cd20a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.848+0000 7f3570f94740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd20ac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.848+0000 7f3570f94740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 0x5559cd27a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.849+0000 7f3570f94740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f355000f9c0 con 0x5559cce3b4d0 2024-04-25T16:06:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.849+0000 7f3570f94740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f355000f2f0 con 0x5559cce332f0 2024-04-25T16:06:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.849+0000 7f3570f94740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f355000c530 con 0x5559cce5c1b0 2024-04-25T16:06:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.849+0000 7f3570f94740 10 client.0 _ll_register_callbacks cb 0x7f355c0a2510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.849+0000 7f3562ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd20ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.849+0000 7f3562ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd20ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:59466/0 (socket says 172.21.15.31:59466) 2024-04-25T16:06:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.850+0000 7f3562ffd640 1 -- 172.21.15.31:0/1043733290 learned_addr learned my addr 172.21.15.31:0/1043733290 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.851+0000 7f3562ffd640 1 -- 172.21.15.31:0/1043733290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 msgr2=0x5559cd27a160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.851+0000 7f3562ffd640 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 0x5559cd27a160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.851+0000 7f3562ffd640 1 -- 172.21.15.31:0/1043733290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce332f0 msgr2=0x5559cd20a6e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.851+0000 7f3562ffd640 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce332f0 0x5559cd20a6e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.851+0000 7f3562ffd640 1 -- 172.21.15.31:0/1043733290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559cd20dbe0 con 0x5559cce3b4d0 2024-04-25T16:06:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.852+0000 7f3562ffd640 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd20ac20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3554031fd0 tx=0x7f355403bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2b19ece8bc3037 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.852+0000 7f3542ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.853+0000 7f3542ffd640 1 -- 172.21.15.31:0/1043733290 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3554035070 con 0x5559cce3b4d0 2024-04-25T16:06:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.853+0000 7f3542ffd640 1 -- 172.21.15.31:0/1043733290 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3554003a00 con 0x5559cce3b4d0 2024-04-25T16:06:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.853+0000 7f3570f94740 1 -- 172.21.15.31:0/1043733290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3554031e20 con 0x5559cce3b4d0 2024-04-25T16:06:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.854+0000 7f3542ffd640 1 -- 172.21.15.31:0/1043733290 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f355405e050 con 0x5559cce3b4d0 2024-04-25T16:06:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.854+0000 7f3570f94740 10 client.16587 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.855+0000 7f3570f94740 1 -- 172.21.15.31:0/1043733290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5559cd27b4d0 con 0x5559cce3b4d0 2024-04-25T16:06:07.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.856+0000 7f3542ffd640 1 -- 172.21.15.31:0/1043733290 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3554003c40 con 0x5559cce3b4d0 2024-04-25T16:06:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.856+0000 7f3542ffd640 1 -- 172.21.15.31:0/1043733290 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3554048400 con 0x5559cce3b4d0 2024-04-25T16:06:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.856+0000 7f3570f94740 10 client.16587 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.856+0000 7f3570f94740 10 client.16587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.856+0000 7f3570f94740 20 client.16587 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.856+0000 7f3570f94740 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5559ccdf01c0 0x5559cd27de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.856+0000 7f3570f94740 4 client.16587 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:06:07.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.856+0000 7f3570f94740 1 -- 172.21.15.31:0/1043733290 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f3554031e20 con 0x5559ccdf01c0 2024-04-25T16:06:07.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.862+0000 7f35637fe640 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5559ccdf01c0 0x5559cd27de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.867+0000 7f35637fe640 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5559ccdf01c0 0x5559cd27de00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f043ecaa37fccb server_cookie=ead4121e6698a89 in_seq=0 out_seq=0 2024-04-25T16:06:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:07.868+0000 7f3542ffd640 10 client.16587 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:09.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:09 smithi063 ceph-mon[32413]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 821 MiB/s wr, 3.83k op/s 2024-04-25T16:06:09.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:09 smithi186 ceph-mon[33258]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 821 MiB/s wr, 3.83k op/s 2024-04-25T16:06:09.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:09 smithi031 ceph-mon[29140]: pgmap v496: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 821 MiB/s wr, 3.83k op/s 2024-04-25T16:06:11.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:11 smithi063 ceph-mon[32413]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 753 MiB/s wr, 3.39k op/s 2024-04-25T16:06:11.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:11 smithi186 ceph-mon[33258]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 753 MiB/s wr, 3.39k op/s 2024-04-25T16:06:11.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:11 smithi031 ceph-mon[29140]: pgmap v497: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 753 MiB/s wr, 3.39k op/s 2024-04-25T16:06:13.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:13 smithi063 ceph-mon[32413]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 710 MiB/s wr, 2.89k op/s 2024-04-25T16:06:13.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:13 smithi186 ceph-mon[33258]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 710 MiB/s wr, 2.89k op/s 2024-04-25T16:06:13.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:13 smithi031 ceph-mon[29140]: pgmap v498: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 710 MiB/s wr, 2.89k op/s 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.751+0000 7f3542ffd640 1 -- 172.21.15.31:0/1043733290 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f3554031e20 con 0x5559ccdf01c0 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.751+0000 7f3542ffd640 10 client.16587 handle_command_reply: tid=0 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.751+0000 7f3570f94740 1 client.16587 shutdown 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.752+0000 7f3570f94740 1 -- 172.21.15.31:0/1043733290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 msgr2=0x5559cd20ac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.752+0000 7f3570f94740 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd20ac20 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3554031fd0 tx=0x7f355403bb00 comp rx=0 tx=0).stop 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.752+0000 7f3570f94740 1 -- 172.21.15.31:0/1043733290 shutdown_connections 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.752+0000 7f3570f94740 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5559ccdf01c0 0x5559cd27de00 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.752+0000 7f3570f94740 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5559cce5c1b0 0x5559cd27a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.752+0000 7f3570f94740 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5559cce3b4d0 0x5559cd20ac20 secure :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f3554031fd0 tx=0x7f355403bb00 comp rx=0 tx=0).stop 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.752+0000 7f3570f94740 1 --2- 172.21.15.31:0/1043733290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5559cce332f0 0x5559cd20a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:13.754 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.752+0000 7f3570f94740 1 -- 172.21.15.31:0/1043733290 >> 172.21.15.31:0/1043733290 conn(0x5559cce50300 msgr2=0x5559cd200b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:13.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.752+0000 7f3542ffd640 0 client.16587 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.756+0000 7f3570f94740 1 -- 172.21.15.31:0/1043733290 shutdown_connections 2024-04-25T16:06:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.756+0000 7f3570f94740 1 -- 172.21.15.31:0/1043733290 wait complete. 2024-04-25T16:06:13.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.756+0000 7f3570f94740 20 client.16587 trim_cache size 0 max 16384 2024-04-25T16:06:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:06:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:06:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:06:13.758 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:06:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.759+0000 7f356a0e0640 1 -- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f354c0afce0 msgr2=0x7f354c0b21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.759+0000 7f356a0e0640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f354c0afce0 0x7f354c0b21f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f35540057c0 tx=0x7f3554031500 comp rx=0 tx=0).stop 2024-04-25T16:06:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.759+0000 7f356a0e0640 1 -- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410a120 msgr2=0x7f356411bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:13.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.759+0000 7f356a0e0640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410a120 0x7f356411bc50 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f355c00b990 tx=0x7f355c00be50 comp rx=0 tx=0).stop 2024-04-25T16:06:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.762+0000 7f356a0e0640 1 -- 172.21.15.31:0/3204241872 shutdown_connections 2024-04-25T16:06:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.762+0000 7f356a0e0640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f354c0afce0 0x7f354c0b21f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.762+0000 7f356a0e0640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f356410bb90 0x7f3564121190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:13.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.762+0000 7f356a0e0640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f356410b240 0x7f356411c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.762+0000 7f356a0e0640 1 --2- 172.21.15.31:0/3204241872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f356410a120 0x7f356411bc50 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.762+0000 7f356a0e0640 1 -- 172.21.15.31:0/3204241872 >> 172.21.15.31:0/3204241872 conn(0x7f3564070960 msgr2=0x7f3564111790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.763+0000 7f356a0e0640 1 -- 172.21.15.31:0/3204241872 shutdown_connections 2024-04-25T16:06:13.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:13.763+0000 7f356a0e0640 1 -- 172.21.15.31:0/3204241872 wait complete. 2024-04-25T16:06:15.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:15 smithi063 ceph-mon[32413]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 896 MiB/s wr, 3.40k op/s 2024-04-25T16:06:15.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:15 smithi186 ceph-mon[33258]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 896 MiB/s wr, 3.40k op/s 2024-04-25T16:06:15.715 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:15 smithi031 ceph-mon[29140]: pgmap v499: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 896 MiB/s wr, 3.40k op/s 2024-04-25T16:06:16.028 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:06:16.028 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:06:16.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.494+0000 7fbd03fff640 1 -- 172.21.15.31:0/4128183110 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd0410a340 msgr2=0x7fbd0410ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:16.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.494+0000 7fbd03fff640 1 --2- 172.21.15.31:0/4128183110 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd0410a340 0x7fbd0410ac50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009900 tx=0x7fbcf402a3f0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.497+0000 7fbd03fff640 1 -- 172.21.15.31:0/4128183110 shutdown_connections 2024-04-25T16:06:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.497+0000 7fbd03fff640 1 --2- 172.21.15.31:0/4128183110 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd0410fe90 0x7fbd041102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.497+0000 7fbd03fff640 1 --2- 172.21.15.31:0/4128183110 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd0410b190 0x7fbd0410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.497+0000 7fbd03fff640 1 --2- 172.21.15.31:0/4128183110 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd0410a340 0x7fbd0410ac50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.497+0000 7fbd03fff640 1 -- 172.21.15.31:0/4128183110 >> 172.21.15.31:0/4128183110 conn(0x7fbd04070320 msgr2=0x7fbd040727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:16.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.514+0000 7fbd03fff640 1 -- 172.21.15.31:0/4128183110 shutdown_connections 2024-04-25T16:06:16.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.524+0000 7fbd03fff640 1 -- 172.21.15.31:0/4128183110 wait complete. 2024-04-25T16:06:16.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.525+0000 7fbd03fff640 1 Processor -- start 2024-04-25T16:06:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.528+0000 7fbd03fff640 1 -- start start 2024-04-25T16:06:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.529+0000 7fbd03fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd0410a340 0x7fbd041a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.529+0000 7fbd03fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd0410b190 0x7fbd041a9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.529+0000 7fbd03fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd0410fe90 0x7fbd041b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.529+0000 7fbd03fff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0411e240 con 0x7fbd0410a340 2024-04-25T16:06:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.529+0000 7fbd03fff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0411e0c0 con 0x7fbd0410fe90 2024-04-25T16:06:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.529+0000 7fbd03fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0411e3c0 con 0x7fbd0410b190 2024-04-25T16:06:16.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.530+0000 7fbd027fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd0410b190 0x7fbd041a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.530+0000 7fbd027fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd0410b190 0x7fbd041a9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52536/0 (socket says 172.21.15.31:52536) 2024-04-25T16:06:16.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.530+0000 7fbd027fc640 1 -- 172.21.15.31:0/3484092601 learned_addr learned my addr 172.21.15.31:0/3484092601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:16.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.533+0000 7fbd037fe640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd0410fe90 0x7fbd041b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.533+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd0410a340 0x7fbd041a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.534+0000 7fbd027fc640 1 -- 172.21.15.31:0/3484092601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd0410fe90 msgr2=0x7fbd041b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:16.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.534+0000 7fbd027fc640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd0410fe90 0x7fbd041b02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.534+0000 7fbd027fc640 1 -- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd0410a340 msgr2=0x7fbd041a8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:16.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.534+0000 7fbd027fc640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd0410a340 0x7fbd041a8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.534+0000 7fbd027fc640 1 -- 172.21.15.31:0/3484092601 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd04111c90 con 0x7fbd0410b190 2024-04-25T16:06:16.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.543+0000 7fbd027fc640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd0410b190 0x7fbd041a9330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800d950 tx=0x7fbcf800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:16.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.545+0000 7fbcebfff640 1 -- 172.21.15.31:0/3484092601 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8017070 con 0x7fbd0410b190 2024-04-25T16:06:16.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.545+0000 7fbcebfff640 1 -- 172.21.15.31:0/3484092601 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcf8009c30 con 0x7fbd0410b190 2024-04-25T16:06:16.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.545+0000 7fbcebfff640 1 -- 172.21.15.31:0/3484092601 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf800f620 con 0x7fbd0410b190 2024-04-25T16:06:16.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.547+0000 7fbd03fff640 1 -- 172.21.15.31:0/3484092601 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd04111f80 con 0x7fbd0410b190 2024-04-25T16:06:16.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.548+0000 7fbd03fff640 1 -- 172.21.15.31:0/3484092601 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd04112480 con 0x7fbd0410b190 2024-04-25T16:06:16.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.555+0000 7fbd1041d740 1 Processor -- start 2024-04-25T16:06:16.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.555+0000 7fbd1041d740 1 -- start start 2024-04-25T16:06:16.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.556+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61118c390 0x55d61152df70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.556+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 0x55d61152e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.556+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61152f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.556+0000 7fbd1041d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d611492d80 con 0x55d61116b6b0 2024-04-25T16:06:16.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.556+0000 7fbd1041d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d61144c8b0 con 0x55d61118c390 2024-04-25T16:06:16.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.556+0000 7fbd1041d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d61148ed40 con 0x55d6111634d0 2024-04-25T16:06:16.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.560+0000 7fbd02ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 0x55d61152e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.561+0000 7fbd027fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61152f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.561+0000 7fbd027fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61152f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58480/0 (socket says 172.21.15.31:58480) 2024-04-25T16:06:16.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.561+0000 7fbd027fc640 1 -- 172.21.15.31:0/242674005 learned_addr learned my addr 172.21.15.31:0/242674005 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:16.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.562+0000 7fbd037fe640 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61118c390 0x55d61152df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.566+0000 7fbd02ffd640 1 -- 172.21.15.31:0/242674005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61118c390 msgr2=0x55d61152df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:16.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.566+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61118c390 0x55d61152df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.566+0000 7fbd02ffd640 1 -- 172.21.15.31:0/242674005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 msgr2=0x55d61152f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:16.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.566+0000 7fbcebfff640 1 -- 172.21.15.31:0/3484092601 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbcf8005040 con 0x7fbd0410b190 2024-04-25T16:06:16.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.568+0000 7fbcebfff640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcec0afc20 0x7fbcec0b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.568+0000 7fbcebfff640 1 -- 172.21.15.31:0/3484092601 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbcf80e3fe0 con 0x7fbd0410b190 2024-04-25T16:06:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.569+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61152f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.569+0000 7fbd02ffd640 1 -- 172.21.15.31:0/242674005 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d61141da40 con 0x55d6111634d0 2024-04-25T16:06:16.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.569+0000 7fbd037fe640 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61118c390 0x55d61152df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:06:16.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.569+0000 7fbd027fc640 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61152f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:06:16.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.577+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 0x55d61152e4f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009850 tx=0x7fbcf4005d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44c20017262118c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:16.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.578+0000 7fbd037fe640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcec0afc20 0x7fbcec0b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.580+0000 7fbd037fe640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcec0afc20 0x7fbcec0b2130 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x55d61152efc0 tx=0x7fbcfc009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:16.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.586+0000 7fbce8ff9640 1 -- 172.21.15.31:0/242674005 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf403f070 con 0x55d6111634d0 2024-04-25T16:06:16.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.586+0000 7fbce8ff9640 1 -- 172.21.15.31:0/242674005 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcf4004030 con 0x55d6111634d0 2024-04-25T16:06:16.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.586+0000 7fbce8ff9640 1 -- 172.21.15.31:0/242674005 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4035070 con 0x55d6111634d0 2024-04-25T16:06:16.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.587+0000 7fbd1041d740 1 -- 172.21.15.31:0/242674005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 msgr2=0x55d61152e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:16.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.587+0000 7fbd1041d740 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 0x55d61152e4f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009850 tx=0x7fbcf4005d70 comp rx=0 tx=0).stop 2024-04-25T16:06:16.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.589+0000 7fbd1041d740 1 -- 172.21.15.31:0/242674005 shutdown_connections 2024-04-25T16:06:16.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.589+0000 7fbd1041d740 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61152f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.589+0000 7fbd1041d740 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 0x55d61152e4f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.589+0000 7fbd1041d740 1 --2- 172.21.15.31:0/242674005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61118c390 0x55d61152df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.589+0000 7fbd1041d740 1 -- 172.21.15.31:0/242674005 >> 172.21.15.31:0/242674005 conn(0x55d6111804e0 msgr2=0x55d611529780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:16.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.589+0000 7fbd1041d740 1 -- 172.21.15.31:0/242674005 shutdown_connections 2024-04-25T16:06:16.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.590+0000 7fbd1041d740 1 -- 172.21.15.31:0/242674005 wait complete. 2024-04-25T16:06:16.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.590+0000 7fbd1041d740 1 -- start start 2024-04-25T16:06:16.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.591+0000 7fbd1041d740 1 Processor -- start 2024-04-25T16:06:16.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.591+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 0x55d611537b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.591+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d611538080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.591+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61118c390 0x55d61153efe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.591+0000 7fbd1041d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d61152fba0 con 0x55d61118c390 2024-04-25T16:06:16.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.591+0000 7fbd1041d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d611419d10 con 0x55d61116b6b0 2024-04-25T16:06:16.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.591+0000 7fbd1041d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d61152fd20 con 0x55d6111634d0 2024-04-25T16:06:16.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.591+0000 7fbd1041d740 10 client.0 _ll_register_callbacks cb 0x55d611446210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:16.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.592+0000 7fbd037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61118c390 0x55d61153efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.592+0000 7fbd037fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61118c390 0x55d61153efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58492/0 (socket says 172.21.15.31:58492) 2024-04-25T16:06:16.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.592+0000 7fbd037fe640 1 -- 172.21.15.31:0/2534741271 learned_addr learned my addr 172.21.15.31:0/2534741271 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:16.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.597+0000 7fbd027fc640 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d611538080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.597+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 0x55d611537b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.598+0000 7fbd037fe640 1 -- 172.21.15.31:0/2534741271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 msgr2=0x55d611537b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:16.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.598+0000 7fbd037fe640 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 0x55d611537b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.598+0000 7fbd037fe640 1 -- 172.21.15.31:0/2534741271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 msgr2=0x55d611538080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:16.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.598+0000 7fbd037fe640 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d611538080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.598+0000 7fbd037fe640 1 -- 172.21.15.31:0/2534741271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d61126e7d0 con 0x55d61118c390 2024-04-25T16:06:16.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.601+0000 7fbd037fe640 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61118c390 0x55d61153efe0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc012850 tx=0x7fbcfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bde4427cb24e5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:16.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.606+0000 7fbce6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.606+0000 7fbce6ff5640 1 -- 172.21.15.31:0/2534741271 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00fb50 con 0x55d61118c390 2024-04-25T16:06:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.606+0000 7fbce6ff5640 1 -- 172.21.15.31:0/2534741271 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbcfc00fcf0 con 0x55d61118c390 2024-04-25T16:06:16.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.608+0000 7fbd1041d740 1 -- 172.21.15.31:0/2534741271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcf4004030 con 0x55d61118c390 2024-04-25T16:06:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.623+0000 7fbce6ff5640 1 -- 172.21.15.31:0/2534741271 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbcfc01f420 con 0x55d61118c390 2024-04-25T16:06:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.625+0000 7fbd1041d740 10 client.16599 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:16.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.625+0000 7fbd1041d740 1 -- 172.21.15.31:0/2534741271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6111a43f0 con 0x55d61118c390 2024-04-25T16:06:16.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.630+0000 7fbce6ff5640 1 -- 172.21.15.31:0/2534741271 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcf4004030 con 0x55d61118c390 2024-04-25T16:06:16.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.630+0000 7fbce6ff5640 1 -- 172.21.15.31:0/2534741271 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fbcfc0316f0 con 0x55d61118c390 2024-04-25T16:06:16.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.630+0000 7fbd1041d740 10 client.16599 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:16.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.630+0000 7fbd1041d740 10 client.16599 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:16.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.630+0000 7fbd1041d740 20 client.16599 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:16.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.630+0000 7fbd1041d740 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d61114af60 0x55d611545640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.630+0000 7fbd1041d740 4 client.16599 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:16.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.630+0000 7fbd1041d740 1 -- 172.21.15.31:0/2534741271 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d6114a0990 con 0x55d61114af60 2024-04-25T16:06:16.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.631+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d61114af60 0x55d611545640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.650+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d61114af60 0x55d611545640 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86560b755cbe8428 server_cookie=e35fe4964bb490b9 in_seq=0 out_seq=0 2024-04-25T16:06:16.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.651+0000 7fbce6ff5640 10 client.16599 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:16.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.653+0000 7fbce6ff5640 1 -- 172.21.15.31:0/2534741271 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d6114a0990 con 0x55d61114af60 2024-04-25T16:06:16.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.654+0000 7fbce6ff5640 10 client.16599 handle_command_reply: tid=0 2024-04-25T16:06:16.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.654+0000 7fbd1041d740 1 client.16599 shutdown 2024-04-25T16:06:16.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.654+0000 7fbd1041d740 1 -- 172.21.15.31:0/2534741271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61118c390 msgr2=0x55d61153efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:16.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.654+0000 7fbd1041d740 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61118c390 0x55d61153efe0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc012850 tx=0x7fbcfc004a40 comp rx=0 tx=0).stop 2024-04-25T16:06:16.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.656+0000 7fbd1041d740 1 -- 172.21.15.31:0/2534741271 shutdown_connections 2024-04-25T16:06:16.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.656+0000 7fbd1041d740 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d61114af60 0x55d611545640 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.656+0000 7fbd1041d740 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61118c390 0x55d61153efe0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.656+0000 7fbd1041d740 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d611538080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.656+0000 7fbd1041d740 1 --2- 172.21.15.31:0/2534741271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d6111634d0 0x55d611537b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.656+0000 7fbd1041d740 1 -- 172.21.15.31:0/2534741271 >> 172.21.15.31:0/2534741271 conn(0x55d6111804e0 msgr2=0x55d61152c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:16.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.656+0000 7fbce6ff5640 0 client.16599 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:16.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.664+0000 7fbd1041d740 1 -- 172.21.15.31:0/2534741271 shutdown_connections 2024-04-25T16:06:16.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.665+0000 7fbd1041d740 1 -- 172.21.15.31:0/2534741271 wait complete. 2024-04-25T16:06:16.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.668+0000 7fbd1041d740 20 client.16599 trim_cache size 0 max 16384 2024-04-25T16:06:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.738+0000 7fbd1041d740 1 Processor -- start 2024-04-25T16:06:16.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.738+0000 7fbd1041d740 1 -- start start 2024-04-25T16:06:16.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.739+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6111634d0 0x55d6115365d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.740+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d6115291b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.740+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d61118c390 0x55d6115296f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.740+0000 7fbd1041d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbd041c2820 con 0x55d6111634d0 2024-04-25T16:06:16.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.740+0000 7fbd1041d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbd041c4360 con 0x55d61116b6b0 2024-04-25T16:06:16.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.740+0000 7fbd1041d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd041c2ef0 con 0x55d61118c390 2024-04-25T16:06:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.741+0000 7fbd027fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d6115291b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.741+0000 7fbd027fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d6115291b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34466/0 (socket says 172.21.15.31:34466) 2024-04-25T16:06:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.741+0000 7fbd027fc640 1 -- 172.21.15.31:0/3195857402 learned_addr learned my addr 172.21.15.31:0/3195857402 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:16.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.742+0000 7fbd037fe640 1 --2- 172.21.15.31:0/3195857402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d61118c390 0x55d6115296f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.742+0000 7fbd027fc640 1 -- 172.21.15.31:0/3195857402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d61118c390 msgr2=0x55d6115296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.742+0000 7fbd027fc640 1 --2- 172.21.15.31:0/3195857402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d61118c390 0x55d6115296f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.742+0000 7fbd027fc640 1 -- 172.21.15.31:0/3195857402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6111634d0 msgr2=0x55d6115365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.742+0000 7fbd027fc640 1 --2- 172.21.15.31:0/3195857402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6111634d0 0x55d6115365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.742+0000 7fbd027fc640 1 -- 172.21.15.31:0/3195857402 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d611524920 con 0x55d61116b6b0 2024-04-25T16:06:16.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.743+0000 7fbd027fc640 1 --2- 172.21.15.31:0/3195857402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d6115291b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80a8460 tx=0x7fbcf800bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fba38f096f941de server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.745+0000 7fbce87f8640 1 -- 172.21.15.31:0/3195857402 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8020070 con 0x55d61116b6b0 2024-04-25T16:06:16.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.745+0000 7fbce87f8640 1 -- 172.21.15.31:0/3195857402 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcf80acb70 con 0x55d61116b6b0 2024-04-25T16:06:16.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.747+0000 7fbd1041d740 1 -- 172.21.15.31:0/3195857402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 msgr2=0x55d6115291b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:16.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.747+0000 7fbd1041d740 1 --2- 172.21.15.31:0/3195857402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d6115291b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80a8460 tx=0x7fbcf800bf60 comp rx=0 tx=0).stop 2024-04-25T16:06:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.755+0000 7fbd1041d740 1 -- 172.21.15.31:0/3195857402 shutdown_connections 2024-04-25T16:06:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.755+0000 7fbd1041d740 1 --2- 172.21.15.31:0/3195857402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d61118c390 0x55d6115296f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.755+0000 7fbd1041d740 1 --2- 172.21.15.31:0/3195857402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d61116b6b0 0x55d6115291b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.755+0000 7fbd1041d740 1 --2- 172.21.15.31:0/3195857402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d6111634d0 0x55d6115365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.755+0000 7fbd1041d740 1 -- 172.21.15.31:0/3195857402 >> 172.21.15.31:0/3195857402 conn(0x55d6111804e0 msgr2=0x55d611532b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:16.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.756+0000 7fbd1041d740 1 -- 172.21.15.31:0/3195857402 shutdown_connections 2024-04-25T16:06:16.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.759+0000 7fbd1041d740 1 -- 172.21.15.31:0/3195857402 wait complete. 2024-04-25T16:06:16.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.759+0000 7fbd1041d740 1 -- start start 2024-04-25T16:06:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.759+0000 7fbd1041d740 1 Processor -- start 2024-04-25T16:06:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.760+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d6111634d0 0x55d61153a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.760+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61153ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.760+0000 7fbd1041d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d61118c390 0x55d6115ab9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.760+0000 7fbd1041d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbd041c3c90 con 0x55d61116b6b0 2024-04-25T16:06:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.760+0000 7fbd1041d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbd041c35c0 con 0x55d6111634d0 2024-04-25T16:06:16.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.760+0000 7fbd1041d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd04119350 con 0x55d61118c390 2024-04-25T16:06:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.760+0000 7fbd1041d740 10 client.0 _ll_register_callbacks cb 0x7fbcfc011f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:16.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.762+0000 7fbd02ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d6111634d0 0x55d61153a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.762+0000 7fbd02ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d6111634d0 0x55d61153a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34468/0 (socket says 172.21.15.31:34468) 2024-04-25T16:06:16.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.762+0000 7fbd02ffd640 1 -- 172.21.15.31:0/1274557315 learned_addr learned my addr 172.21.15.31:0/1274557315 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:16.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.765+0000 7fbd027fc640 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61153ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.766+0000 7fbd02ffd640 1 -- 172.21.15.31:0/1274557315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d61118c390 msgr2=0x55d6115ab9a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:16.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.766+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d61118c390 0x55d6115ab9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.766+0000 7fbd02ffd640 1 -- 172.21.15.31:0/1274557315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 msgr2=0x55d61153ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:16.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.766+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61153ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.766+0000 7fbd02ffd640 1 -- 172.21.15.31:0/1274557315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d6111640f0 con 0x55d6111634d0 2024-04-25T16:06:16.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.768+0000 7fbd02ffd640 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d6111634d0 0x55d61153a850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4005ab0 tx=0x7fbcf4004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8c34e2cb97818a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.768+0000 7fbce6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:06:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.768+0000 7fbce6ff5640 1 -- 172.21.15.31:0/1274557315 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf403f040 con 0x55d6111634d0 2024-04-25T16:06:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.768+0000 7fbce6ff5640 1 -- 172.21.15.31:0/1274557315 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbcf4005040 con 0x55d6111634d0 2024-04-25T16:06:16.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.768+0000 7fbd1041d740 1 -- 172.21.15.31:0/1274557315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcf80acb70 con 0x55d6111634d0 2024-04-25T16:06:16.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.769+0000 7fbce6ff5640 1 -- 172.21.15.31:0/1274557315 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbcf4035040 con 0x55d6111634d0 2024-04-25T16:06:16.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.775+0000 7fbd1041d740 10 client.26264 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:16.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.775+0000 7fbd1041d740 1 -- 172.21.15.31:0/1274557315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d6114534b0 con 0x55d6111634d0 2024-04-25T16:06:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.779+0000 7fbce6ff5640 1 -- 172.21.15.31:0/1274557315 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbcf80acb70 con 0x55d6111634d0 2024-04-25T16:06:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.779+0000 7fbce6ff5640 1 -- 172.21.15.31:0/1274557315 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fbcf404d220 con 0x55d6111634d0 2024-04-25T16:06:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.779+0000 7fbd1041d740 10 client.26264 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.779+0000 7fbd1041d740 10 client.26264 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.779+0000 7fbd1041d740 20 client.26264 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:16.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.779+0000 7fbd1041d740 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d6111203a0 0x55d6115af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.779+0000 7fbd1041d740 4 client.26264 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:06:16.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.779+0000 7fbd1041d740 1 -- 172.21.15.31:0/1274557315 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fbcf80a8800 con 0x55d6111203a0 2024-04-25T16:06:16.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.785+0000 7fbd027fc640 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d6111203a0 0x55d6115af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:16.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.801+0000 7fbd027fc640 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d6111203a0 0x55d6115af820 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=494ff41c9b087789 server_cookie=4c04019785fa50dd in_seq=0 out_seq=0 2024-04-25T16:06:16.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.801+0000 7fbce6ff5640 10 client.26264 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.802+0000 7fbce6ff5640 1 -- 172.21.15.31:0/1274557315 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fbcf80a8800 con 0x55d6111203a0 2024-04-25T16:06:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.802+0000 7fbce6ff5640 10 client.26264 handle_command_reply: tid=0 2024-04-25T16:06:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.802+0000 7fbd1041d740 1 client.26264 shutdown 2024-04-25T16:06:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.802+0000 7fbd1041d740 1 -- 172.21.15.31:0/1274557315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d6111634d0 msgr2=0x55d61153a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.802+0000 7fbd1041d740 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d6111634d0 0x55d61153a850 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4005ab0 tx=0x7fbcf4004600 comp rx=0 tx=0).stop 2024-04-25T16:06:16.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.805+0000 7fbd1041d740 1 -- 172.21.15.31:0/1274557315 shutdown_connections 2024-04-25T16:06:16.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.805+0000 7fbd1041d740 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d6111203a0 0x55d6115af820 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.805+0000 7fbd1041d740 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d61118c390 0x55d6115ab9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.805+0000 7fbd1041d740 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d61116b6b0 0x55d61153ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.805+0000 7fbd1041d740 1 --2- 172.21.15.31:0/1274557315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d6111634d0 0x55d61153a850 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.805+0000 7fbd1041d740 1 -- 172.21.15.31:0/1274557315 >> 172.21.15.31:0/1274557315 conn(0x55d6111804e0 msgr2=0x55d61152c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:16.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.805+0000 7fbce6ff5640 0 client.26264 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:16.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.811+0000 7fbd1041d740 1 -- 172.21.15.31:0/1274557315 shutdown_connections 2024-04-25T16:06:16.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.812+0000 7fbd1041d740 1 -- 172.21.15.31:0/1274557315 wait complete. 2024-04-25T16:06:16.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.816+0000 7fbd1041d740 20 client.26264 trim_cache size 0 max 16384 2024-04-25T16:06:16.822 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:06:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:06:16.822 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:06:16.823 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:06:16.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.820+0000 7fbce9ffb640 1 -- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcec0afc20 msgr2=0x7fbcec0b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:16.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.820+0000 7fbce9ffb640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcec0afc20 0x7fbcec0b2130 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x55d61152efc0 tx=0x7fbcfc009500 comp rx=0 tx=0).stop 2024-04-25T16:06:16.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.820+0000 7fbce9ffb640 1 -- 172.21.15.31:0/3484092601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd0410b190 msgr2=0x7fbd041a9330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:16.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.820+0000 7fbce9ffb640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd0410b190 0x7fbd041a9330 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbcf800d950 tx=0x7fbcf800de10 comp rx=0 tx=0).stop 2024-04-25T16:06:16.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.823+0000 7fbce9ffb640 1 -- 172.21.15.31:0/3484092601 shutdown_connections 2024-04-25T16:06:16.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.823+0000 7fbce9ffb640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbcec0afc20 0x7fbcec0b2130 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.824+0000 7fbce9ffb640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd0410fe90 0x7fbd041b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.824+0000 7fbce9ffb640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd0410b190 0x7fbd041a9330 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.824+0000 7fbce9ffb640 1 --2- 172.21.15.31:0/3484092601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbd0410a340 0x7fbd041a8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:16.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.824+0000 7fbce9ffb640 1 -- 172.21.15.31:0/3484092601 >> 172.21.15.31:0/3484092601 conn(0x7fbd04070320 msgr2=0x7fbd0410be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:16.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.829+0000 7fbce9ffb640 1 -- 172.21.15.31:0/3484092601 shutdown_connections 2024-04-25T16:06:16.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:16.831+0000 7fbce9ffb640 1 -- 172.21.15.31:0/3484092601 wait complete. 2024-04-25T16:06:16.868 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:06:16.868 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:06:17.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.220+0000 7f3a0ce22640 1 -- 172.21.15.31:0/214155233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 msgr2=0x7f3a0810b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:17.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.220+0000 7f3a0ce22640 1 --2- 172.21.15.31:0/214155233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 0x7f3a0810b5b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00af30 tx=0x7f39fc0327e0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.224+0000 7f3a0ce22640 1 -- 172.21.15.31:0/214155233 shutdown_connections 2024-04-25T16:06:17.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.224+0000 7f3a0ce22640 1 --2- 172.21.15.31:0/214155233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a081102c0 0x7f3a081126e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.224+0000 7f3a0ce22640 1 --2- 172.21.15.31:0/214155233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 0x7f3a0810b5b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.224+0000 7f3a0ce22640 1 --2- 172.21.15.31:0/214155233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a0810a240 0x7f3a0810aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.224+0000 7f3a0ce22640 1 -- 172.21.15.31:0/214155233 >> 172.21.15.31:0/214155233 conn(0x7f3a08070780 msgr2=0x7f3a08072c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.225+0000 7f3a0ce22640 1 -- 172.21.15.31:0/214155233 shutdown_connections 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.226+0000 7f3a0ce22640 1 -- 172.21.15.31:0/214155233 wait complete. 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.227+0000 7f3a0ce22640 1 Processor -- start 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.227+0000 7f3a0ce22640 1 -- start start 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.227+0000 7f3a0ce22640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a0810a240 0x7f3a081a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.227+0000 7f3a0ce22640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 0x7f3a081a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.227+0000 7f3a0ce22640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a081102c0 0x7f3a081a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:17.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.227+0000 7f3a0ce22640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3a08115a60 con 0x7f3a0810a760 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.227+0000 7f3a0ce22640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3a081158e0 con 0x7f3a0810a240 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.227+0000 7f3a0ce22640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3a08115be0 con 0x7f3a081102c0 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.228+0000 7f3a06ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 0x7f3a081a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.228+0000 7f3a06ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 0x7f3a081a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58546/0 (socket says 172.21.15.31:58546) 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.228+0000 7f3a06ffd640 1 -- 172.21.15.31:0/637986214 learned_addr learned my addr 172.21.15.31:0/637986214 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.229+0000 7f3a06ffd640 1 -- 172.21.15.31:0/637986214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a081102c0 msgr2=0x7f3a081a7af0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.229+0000 7f3a06ffd640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a081102c0 0x7f3a081a7af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.229+0000 7f3a06ffd640 1 -- 172.21.15.31:0/637986214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a0810a240 msgr2=0x7f3a081a0690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.229+0000 7f3a06ffd640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a0810a240 0x7f3a081a0690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.229+0000 7f3a06ffd640 1 -- 172.21.15.31:0/637986214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a081a8160 con 0x7f3a0810a760 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.230+0000 7f3a06ffd640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 0x7f3a081a0bd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00af00 tx=0x7f39fc00a550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:17.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.231+0000 7f3a04ff9640 1 -- 172.21.15.31:0/637986214 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc048070 con 0x7f3a0810a760 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.231+0000 7f3a04ff9640 1 -- 172.21.15.31:0/637986214 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39fc004310 con 0x7f3a0810a760 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.231+0000 7f3a04ff9640 1 -- 172.21.15.31:0/637986214 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39fc044850 con 0x7f3a0810a760 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.233+0000 7f3a0ce22640 1 -- 172.21.15.31:0/637986214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a081a83f0 con 0x7f3a0810a760 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.234+0000 7f3a0ce22640 1 -- 172.21.15.31:0/637986214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a081a88b0 con 0x7f3a0810a760 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.237+0000 7f3a0ce22640 1 -- 172.21.15.31:0/637986214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a0810f0d0 con 0x7f3a0810a760 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.240+0000 7f3a04ff9640 1 -- 172.21.15.31:0/637986214 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f39fc04b070 con 0x7f3a0810a760 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.242+0000 7f3a04ff9640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39f00afcf0 0x7f39f00b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.243+0000 7f3a04ff9640 1 -- 172.21.15.31:0/637986214 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f39fc107110 con 0x7f3a0810a760 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.243+0000 7f3a077fe640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39f00afcf0 0x7f39f00b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.253+0000 7f3a04ff9640 1 -- 172.21.15.31:0/637986214 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39fc0cfca0 con 0x7f3a0810a760 2024-04-25T16:06:17.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.254+0000 7f3a077fe640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39f00afcf0 0x7f39f00b2200 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3a0810ae10 tx=0x7f39f8006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:17.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:17 smithi063 ceph-mon[32413]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 758 MiB/s wr, 2.50k op/s 2024-04-25T16:06:17.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:17 smithi186 ceph-mon[33258]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 758 MiB/s wr, 2.50k op/s 2024-04-25T16:06:17.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:17 smithi031 ceph-mon[29140]: pgmap v500: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 758 MiB/s wr, 2.50k op/s 2024-04-25T16:06:17.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.917+0000 7f3a0ce22640 1 -- 172.21.15.31:0/637986214 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3a0802db50 con 0x7f3a0810a760 2024-04-25T16:06:17.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.928+0000 7f3a04ff9640 1 -- 172.21.15.31:0/637986214 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7f39fc0cf5a0 con 0x7f3a0810a760 2024-04-25T16:06:17.929 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:06:17.929 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:06:17.930 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:06:17.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 -- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39f00afcf0 msgr2=0x7f39f00b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:17.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39f00afcf0 0x7f39f00b2200 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f3a0810ae10 tx=0x7f39f8006bd0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 -- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 msgr2=0x7f3a081a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 0x7f3a081a0bd0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00af00 tx=0x7f39fc00a550 comp rx=0 tx=0).stop 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 -- 172.21.15.31:0/637986214 shutdown_connections 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f39f00afcf0 0x7f39f00b2200 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a081102c0 0x7f3a081a7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3a0810a760 0x7f3a081a0bd0 secure :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f39fc00af00 tx=0x7f39fc00a550 comp rx=0 tx=0).stop 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 --2- 172.21.15.31:0/637986214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3a0810a240 0x7f3a081a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 -- 172.21.15.31:0/637986214 >> 172.21.15.31:0/637986214 conn(0x7f3a08070780 msgr2=0x7f3a0810c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.953+0000 7f39ee7fc640 1 -- 172.21.15.31:0/637986214 shutdown_connections 2024-04-25T16:06:17.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:17.954+0000 7f39ee7fc640 1 -- 172.21.15.31:0/637986214 wait complete. 2024-04-25T16:06:17.999 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:06:18.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.518+0000 7fbfbd150640 1 -- 172.21.15.31:0/517035601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 msgr2=0x7fbfb810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:18.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.518+0000 7fbfbd150640 1 --2- 172.21.15.31:0/517035601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 0x7fbfb810ac50 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009900 tx=0x7fbfa802a440 comp rx=0 tx=0).stop 2024-04-25T16:06:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.530+0000 7fbfbd150640 1 -- 172.21.15.31:0/517035601 shutdown_connections 2024-04-25T16:06:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.530+0000 7fbfbd150640 1 --2- 172.21.15.31:0/517035601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfb810fe90 0x7fbfb81102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.530+0000 7fbfbd150640 1 --2- 172.21.15.31:0/517035601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb810b190 0x7fbfb810b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.530+0000 7fbfbd150640 1 --2- 172.21.15.31:0/517035601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 0x7fbfb810ac50 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.530+0000 7fbfbd150640 1 -- 172.21.15.31:0/517035601 >> 172.21.15.31:0/517035601 conn(0x7fbfb8070320 msgr2=0x7fbfb80727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:18.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.534+0000 7fbfbd150640 1 -- 172.21.15.31:0/517035601 shutdown_connections 2024-04-25T16:06:18.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.538+0000 7fbfbd150640 1 -- 172.21.15.31:0/517035601 wait complete. 2024-04-25T16:06:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.539+0000 7fbfbd150640 1 Processor -- start 2024-04-25T16:06:18.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.540+0000 7fbfbd150640 1 -- start start 2024-04-25T16:06:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.540+0000 7fbfbd150640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 0x7fbfb8117090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.540+0000 7fbfbd150640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb810b190 0x7fbfb81175d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.540+0000 7fbfbd150640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfb810fe90 0x7fbfb8118530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.540+0000 7fbfbd150640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb811e280 con 0x7fbfb810a340 2024-04-25T16:06:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.540+0000 7fbfbd150640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb811e100 con 0x7fbfb810b190 2024-04-25T16:06:18.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.540+0000 7fbfbd150640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfb811e400 con 0x7fbfb810fe90 2024-04-25T16:06:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.542+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 0x7fbfb8117090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.542+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 0x7fbfb8117090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58564/0 (socket says 172.21.15.31:58564) 2024-04-25T16:06:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.542+0000 7fbfb7fff640 1 -- 172.21.15.31:0/4283949922 learned_addr learned my addr 172.21.15.31:0/4283949922 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.543+0000 7fbfb77fe640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb810b190 0x7fbfb81175d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.543+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfb810fe90 0x7fbfb8118530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.546+0000 7fbfb7fff640 1 -- 172.21.15.31:0/4283949922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfb810fe90 msgr2=0x7fbfb8118530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:18.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.546+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfb810fe90 0x7fbfb8118530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.546+0000 7fbfb7fff640 1 -- 172.21.15.31:0/4283949922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb810b190 msgr2=0x7fbfb81175d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:18.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.546+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb810b190 0x7fbfb81175d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.546+0000 7fbfb7fff640 1 -- 172.21.15.31:0/4283949922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfb8118ba0 con 0x7fbfb810a340 2024-04-25T16:06:18.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.547+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 0x7fbfb8117090 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004e40 tx=0x7fbfa8004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:18.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.557+0000 7fbfb57fa640 1 -- 172.21.15.31:0/4283949922 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8040020 con 0x7fbfb810a340 2024-04-25T16:06:18.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.557+0000 7fbfb57fa640 1 -- 172.21.15.31:0/4283949922 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfa80054b0 con 0x7fbfb810a340 2024-04-25T16:06:18.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.557+0000 7fbfb57fa640 1 -- 172.21.15.31:0/4283949922 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa8035070 con 0x7fbfb810a340 2024-04-25T16:06:18.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.559+0000 7fbfbd150640 1 -- 172.21.15.31:0/4283949922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfb8118e30 con 0x7fbfb810a340 2024-04-25T16:06:18.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.559+0000 7fbfbd150640 1 -- 172.21.15.31:0/4283949922 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfb81192d0 con 0x7fbfb810a340 2024-04-25T16:06:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.570+0000 7fbfc5268740 1 Processor -- start 2024-04-25T16:06:18.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.570+0000 7fbfc5268740 1 -- start start 2024-04-25T16:06:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.572+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6ccc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.572+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69014d0 0x55e2e6cccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.572+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69096b0 0x55e2e6ccdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.572+0000 7fbfc5268740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e2e6c30d80 con 0x55e2e69014d0 2024-04-25T16:06:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.572+0000 7fbfc5268740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e2e6beaa20 con 0x55e2e69096b0 2024-04-25T16:06:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.572+0000 7fbfc5268740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e2e6c2cd40 con 0x55e2e692a390 2024-04-25T16:06:18.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.574+0000 7fbfb57fa640 1 -- 172.21.15.31:0/4283949922 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbfa8043020 con 0x7fbfb810a340 2024-04-25T16:06:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.575+0000 7fbfbc94f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6ccc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.575+0000 7fbfbc94f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6ccc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52594/0 (socket says 172.21.15.31:52594) 2024-04-25T16:06:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.575+0000 7fbfbc94f640 1 -- 172.21.15.31:0/1781227798 learned_addr learned my addr 172.21.15.31:0/1781227798 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.575+0000 7fbfb57fa640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfa00afcf0 0x7fbfa00b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.576+0000 7fbfb77fe640 1 --2- 172.21.15.31:0/1781227798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69096b0 0x55e2e6ccdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.576+0000 7fbfb57fa640 1 -- 172.21.15.31:0/4283949922 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbfa8100790 con 0x7fbfb810a340 2024-04-25T16:06:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.576+0000 7fbfbc94f640 1 -- 172.21.15.31:0/1781227798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69096b0 msgr2=0x55e2e6ccdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.576+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/1781227798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69096b0 0x55e2e6ccdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.576+0000 7fbfbc94f640 1 -- 172.21.15.31:0/1781227798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69014d0 msgr2=0x55e2e6cccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:18.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.576+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/1781227798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69014d0 0x55e2e6cccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.576+0000 7fbfbc94f640 1 -- 172.21.15.31:0/1781227798 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2e6bf5c30 con 0x55e2e692a390 2024-04-25T16:06:18.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.577+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfa00afcf0 0x7fbfa00b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.578+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/1781227798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6ccc9e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009d30 tx=0x7fbfac0137d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5c293129b84c912 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.579+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfa00afcf0 0x7fbfa00b2200 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbfac022000 tx=0x7fbfac01f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.580+0000 7fbf9e7fc640 1 -- 172.21.15.31:0/1781227798 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac01b550 con 0x55e2e692a390 2024-04-25T16:06:18.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.580+0000 7fbf9e7fc640 1 -- 172.21.15.31:0/1781227798 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfac010880 con 0x55e2e692a390 2024-04-25T16:06:18.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.581+0000 7fbfc5268740 1 -- 172.21.15.31:0/1781227798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 msgr2=0x55e2e6ccc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:18.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.581+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1781227798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6ccc9e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009d30 tx=0x7fbfac0137d0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.590+0000 7fbfc5268740 1 -- 172.21.15.31:0/1781227798 shutdown_connections 2024-04-25T16:06:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.590+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1781227798 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69096b0 0x55e2e6ccdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.590+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1781227798 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69014d0 0x55e2e6cccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.590+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1781227798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6ccc9e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.590+0000 7fbfc5268740 1 -- 172.21.15.31:0/1781227798 >> 172.21.15.31:0/1781227798 conn(0x55e2e691e4e0 msgr2=0x55e2e6cc81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:18.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.591+0000 7fbfc5268740 1 -- 172.21.15.31:0/1781227798 shutdown_connections 2024-04-25T16:06:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.593+0000 7fbfc5268740 1 -- 172.21.15.31:0/1781227798 wait complete. 2024-04-25T16:06:18.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.593+0000 7fbfc5268740 1 -- start start 2024-04-25T16:06:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.593+0000 7fbfc5268740 1 Processor -- start 2024-04-25T16:06:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.594+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e69014d0 0x55e2e6cd6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.594+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69096b0 0x55e2e6cd6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.594+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e692a390 0x55e2e6cddcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.594+0000 7fbfc5268740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e2e6cce750 con 0x55e2e692a390 2024-04-25T16:06:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.594+0000 7fbfc5268740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e2e6bb7dc0 con 0x55e2e69096b0 2024-04-25T16:06:18.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.594+0000 7fbfc5268740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e2e6cce8d0 con 0x55e2e69014d0 2024-04-25T16:06:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.594+0000 7fbfc5268740 10 client.0 _ll_register_callbacks cb 0x55e2e6bd3600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.595+0000 7fbfbc94f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e692a390 0x55e2e6cddcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.595+0000 7fbfbc94f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e692a390 0x55e2e6cddcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58574/0 (socket says 172.21.15.31:58574) 2024-04-25T16:06:18.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.595+0000 7fbfbc94f640 1 -- 172.21.15.31:0/3504833015 learned_addr learned my addr 172.21.15.31:0/3504833015 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:18.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.596+0000 7fbfbc94f640 1 -- 172.21.15.31:0/3504833015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e69014d0 msgr2=0x55e2e6cd6830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:18.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.596+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e69014d0 0x55e2e6cd6830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.596+0000 7fbfbc94f640 1 -- 172.21.15.31:0/3504833015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69096b0 msgr2=0x55e2e6cd6d70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:18.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.596+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69096b0 0x55e2e6cd6d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.596+0000 7fbfbc94f640 1 -- 172.21.15.31:0/3504833015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2e69423f0 con 0x55e2e692a390 2024-04-25T16:06:18.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.597+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e692a390 0x55e2e6cddcd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009d00 tx=0x7fbfac010e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a5141423021da3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.604+0000 7fbf9bff7640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.604+0000 7fbf9bff7640 1 -- 172.21.15.31:0/3504833015 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfac0105e0 con 0x55e2e692a390 2024-04-25T16:06:18.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.604+0000 7fbfc5268740 1 -- 172.21.15.31:0/3504833015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfac010880 con 0x55e2e692a390 2024-04-25T16:06:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.604+0000 7fbf9bff7640 1 -- 172.21.15.31:0/3504833015 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbfac00e8e0 con 0x55e2e692a390 2024-04-25T16:06:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.605+0000 7fbf9bff7640 1 -- 172.21.15.31:0/3504833015 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbfac00eae0 con 0x55e2e692a390 2024-04-25T16:06:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.605+0000 7fbfc5268740 10 client.16617 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:18.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.605+0000 7fbfc5268740 1 -- 172.21.15.31:0/3504833015 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e2e6a0c7d0 con 0x55e2e692a390 2024-04-25T16:06:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.607+0000 7fbf9bff7640 1 -- 172.21.15.31:0/3504833015 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfac010880 con 0x55e2e692a390 2024-04-25T16:06:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.607+0000 7fbf9bff7640 1 -- 172.21.15.31:0/3504833015 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fbfac036480 con 0x55e2e692a390 2024-04-25T16:06:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.607+0000 7fbfc5268740 10 client.16617 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.607+0000 7fbfc5268740 10 client.16617 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:06:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.607+0000 7fbfc5268740 20 client.16617 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.607+0000 7fbfc5268740 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55e2e68e8f60 0x55e2e6ce47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.607+0000 7fbfc5268740 4 client.16617 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:18.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.607+0000 7fbfc5268740 1 -- 172.21.15.31:0/3504833015 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2e6c3e990 con 0x55e2e68e8f60 2024-04-25T16:06:18.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.616+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55e2e68e8f60 0x55e2e6ce47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.619+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55e2e68e8f60 0x55e2e6ce47f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b9c77814461dff2e server_cookie=6f0a0b483caed4f4 in_seq=0 out_seq=0 2024-04-25T16:06:18.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.620+0000 7fbf9bff7640 10 client.16617 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:06:18.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.621+0000 7fbf9bff7640 1 -- 172.21.15.31:0/3504833015 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e2e6c3e990 con 0x55e2e68e8f60 2024-04-25T16:06:18.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.621+0000 7fbf9bff7640 10 client.16617 handle_command_reply: tid=0 2024-04-25T16:06:18.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.621+0000 7fbfc5268740 1 client.16617 shutdown 2024-04-25T16:06:18.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.626+0000 7fbfc5268740 1 -- 172.21.15.31:0/3504833015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e692a390 msgr2=0x55e2e6cddcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:18.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.626+0000 7fbfc5268740 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e692a390 0x55e2e6cddcd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fbfac009d00 tx=0x7fbfac010e10 comp rx=0 tx=0).stop 2024-04-25T16:06:18.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.630+0000 7fbfc5268740 1 -- 172.21.15.31:0/3504833015 shutdown_connections 2024-04-25T16:06:18.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.630+0000 7fbfc5268740 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55e2e68e8f60 0x55e2e6ce47f0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.630+0000 7fbfc5268740 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e692a390 0x55e2e6cddcd0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.630+0000 7fbfc5268740 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69096b0 0x55e2e6cd6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.630+0000 7fbfc5268740 1 --2- 172.21.15.31:0/3504833015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e69014d0 0x55e2e6cd6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.630+0000 7fbfc5268740 1 -- 172.21.15.31:0/3504833015 >> 172.21.15.31:0/3504833015 conn(0x55e2e691e4e0 msgr2=0x55e2e6cc73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:18.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.630+0000 7fbf9bff7640 0 client.16617 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:06:18.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.633+0000 7fbfc5268740 1 -- 172.21.15.31:0/3504833015 shutdown_connections 2024-04-25T16:06:18.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.633+0000 7fbfc5268740 1 -- 172.21.15.31:0/3504833015 wait complete. 2024-04-25T16:06:18.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.633+0000 7fbfc5268740 20 client.16617 trim_cache size 0 max 16384 2024-04-25T16:06:18.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:18 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/637986214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:06:18.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.688+0000 7fbfc5268740 1 Processor -- start 2024-04-25T16:06:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.689+0000 7fbfc5268740 1 -- start start 2024-04-25T16:06:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.689+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cd61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.689+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6ce8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.689+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6ce9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.689+0000 7fbfc5268740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400e570 con 0x55e2e69096b0 2024-04-25T16:06:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.689+0000 7fbfc5268740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa40100b0 con 0x55e2e69014d0 2024-04-25T16:06:18.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.690+0000 7fbfc5268740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400ec40 con 0x55e2e692a390 2024-04-25T16:06:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.691+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cd61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.691+0000 7fbfb7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cd61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34528/0 (socket says 172.21.15.31:34528) 2024-04-25T16:06:18.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.691+0000 7fbfb7fff640 1 -- 172.21.15.31:0/536070919 learned_addr learned my addr 172.21.15.31:0/536070919 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.691+0000 7fbfb77fe640 1 --2- 172.21.15.31:0/536070919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6ce8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.692+0000 7fbfb7fff640 1 -- 172.21.15.31:0/536070919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 msgr2=0x55e2e6ce9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.692+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/536070919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6ce9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.692+0000 7fbfb7fff640 1 -- 172.21.15.31:0/536070919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 msgr2=0x55e2e6ce8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.692+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/536070919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6ce8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.692+0000 7fbfb7fff640 1 -- 172.21.15.31:0/536070919 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2e6ccc3f0 con 0x55e2e69014d0 2024-04-25T16:06:18.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.696+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/536070919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cd61d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80c5e90 tx=0x7fbfa8005660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=675b454a14d897a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:18.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.697+0000 7fbf9dffb640 1 -- 172.21.15.31:0/536070919 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa80d13c0 con 0x55e2e69014d0 2024-04-25T16:06:18.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.697+0000 7fbf9dffb640 1 -- 172.21.15.31:0/536070919 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfa80c8730 con 0x55e2e69014d0 2024-04-25T16:06:18.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.697+0000 7fbf9dffb640 1 -- 172.21.15.31:0/536070919 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfa80d1a80 con 0x55e2e69014d0 2024-04-25T16:06:18.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.699+0000 7fbfc5268740 1 -- 172.21.15.31:0/536070919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 msgr2=0x55e2e6cd61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.700+0000 7fbfc5268740 1 --2- 172.21.15.31:0/536070919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cd61d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbfa80c5e90 tx=0x7fbfa8005660 comp rx=0 tx=0).stop 2024-04-25T16:06:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.700+0000 7fbfc5268740 1 -- 172.21.15.31:0/536070919 shutdown_connections 2024-04-25T16:06:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.700+0000 7fbfc5268740 1 --2- 172.21.15.31:0/536070919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6ce9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.700+0000 7fbfc5268740 1 --2- 172.21.15.31:0/536070919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6ce8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.700+0000 7fbfc5268740 1 --2- 172.21.15.31:0/536070919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cd61d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.700+0000 7fbfc5268740 1 -- 172.21.15.31:0/536070919 >> 172.21.15.31:0/536070919 conn(0x55e2e691e4e0 msgr2=0x55e2e6cd5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:18.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.703+0000 7fbfc5268740 1 -- 172.21.15.31:0/536070919 shutdown_connections 2024-04-25T16:06:18.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.703+0000 7fbfc5268740 1 -- 172.21.15.31:0/536070919 wait complete. 2024-04-25T16:06:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.704+0000 7fbfc5268740 1 -- start start 2024-04-25T16:06:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.705+0000 7fbfc5268740 1 Processor -- start 2024-04-25T16:06:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.705+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cda170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.705+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6cda6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.705+0000 7fbfc5268740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6d4ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.705+0000 7fbfc5268740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400f9e0 con 0x55e2e69096b0 2024-04-25T16:06:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.705+0000 7fbfc5268740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400f310 con 0x55e2e69014d0 2024-04-25T16:06:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.705+0000 7fbfc5268740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbfa400c550 con 0x55e2e692a390 2024-04-25T16:06:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.705+0000 7fbfc5268740 10 client.0 _ll_register_callbacks cb 0x7fbfac009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:18.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:18 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/637986214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:06:18.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.706+0000 7fbfb77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6cda6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.706+0000 7fbfb77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6cda6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58592/0 (socket says 172.21.15.31:58592) 2024-04-25T16:06:18.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.706+0000 7fbfb77fe640 1 -- 172.21.15.31:0/1390524382 learned_addr learned my addr 172.21.15.31:0/1390524382 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.708+0000 7fbfbc94f640 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6d4ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.708+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cda170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.709+0000 7fbfb77fe640 1 -- 172.21.15.31:0/1390524382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 msgr2=0x55e2e6d4ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.709+0000 7fbfb77fe640 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6d4ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.709+0000 7fbfb77fe640 1 -- 172.21.15.31:0/1390524382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 msgr2=0x55e2e6cda170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.709+0000 7fbfb77fe640 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cda170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.709+0000 7fbfb77fe640 1 -- 172.21.15.31:0/1390524382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2e6cdaca0 con 0x55e2e69096b0 2024-04-25T16:06:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.709+0000 7fbfb77fe640 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6cda6b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0005fd0 tx=0x7fbfb0003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea5934fd6425c77 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.710+0000 7fbf9bff7640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.710+0000 7fbf9bff7640 1 -- 172.21.15.31:0/1390524382 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfb0004810 con 0x55e2e69096b0 2024-04-25T16:06:18.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.710+0000 7fbfc5268740 1 -- 172.21.15.31:0/1390524382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfa80c8730 con 0x55e2e69096b0 2024-04-25T16:06:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.710+0000 7fbf9bff7640 1 -- 172.21.15.31:0/1390524382 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbfb0012ae0 con 0x55e2e69096b0 2024-04-25T16:06:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.711+0000 7fbf9bff7640 1 -- 172.21.15.31:0/1390524382 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbfb0012ce0 con 0x55e2e69096b0 2024-04-25T16:06:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.711+0000 7fbfc5268740 10 client.16623 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:18.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.711+0000 7fbfc5268740 1 -- 172.21.15.31:0/1390524382 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e2e6d4c210 con 0x55e2e69096b0 2024-04-25T16:06:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.712+0000 7fbf9bff7640 1 -- 172.21.15.31:0/1390524382 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbfa80c8730 con 0x55e2e69096b0 2024-04-25T16:06:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.712+0000 7fbf9bff7640 1 -- 172.21.15.31:0/1390524382 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fbfb001c550 con 0x55e2e69096b0 2024-04-25T16:06:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.712+0000 7fbfc5268740 10 client.16623 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.712+0000 7fbfc5268740 10 client.16623 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:06:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.712+0000 7fbfc5268740 20 client.16623 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:18.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.712+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55e2e68be3a0 0x55e2e6d4ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:18.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.712+0000 7fbfc5268740 4 client.16623 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:06:18.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.712+0000 7fbfc5268740 1 -- 172.21.15.31:0/1390524382 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e2e6c38800 con 0x55e2e68be3a0 2024-04-25T16:06:18.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.717+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55e2e68be3a0 0x55e2e6d4ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:18.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.718+0000 7fbfb7fff640 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55e2e68be3a0 0x55e2e6d4ec60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3fce2bb271f5bdd8 server_cookie=7f6aa4aee71af287 in_seq=0 out_seq=0 2024-04-25T16:06:18.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.718+0000 7fbf9bff7640 10 client.16623 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:06:18.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.720+0000 7fbf9bff7640 1 -- 172.21.15.31:0/1390524382 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e2e6c38800 con 0x55e2e68be3a0 2024-04-25T16:06:18.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.720+0000 7fbf9bff7640 10 client.16623 handle_command_reply: tid=0 2024-04-25T16:06:18.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.720+0000 7fbfc5268740 1 client.16623 shutdown 2024-04-25T16:06:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 -- 172.21.15.31:0/1390524382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 msgr2=0x55e2e6cda6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:18.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6cda6b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fbfb0005fd0 tx=0x7fbfb0003f90 comp rx=0 tx=0).stop 2024-04-25T16:06:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 -- 172.21.15.31:0/1390524382 shutdown_connections 2024-04-25T16:06:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55e2e68be3a0 0x55e2e6d4ec60 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2e692a390 0x55e2e6d4ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2e69096b0 0x55e2e6cda6b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 --2- 172.21.15.31:0/1390524382 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2e69014d0 0x55e2e6cda170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 -- 172.21.15.31:0/1390524382 >> 172.21.15.31:0/1390524382 conn(0x55e2e691e4e0 msgr2=0x55e2e6ccaff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:18.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbf9bff7640 0 client.16623 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:06:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 -- 172.21.15.31:0/1390524382 shutdown_connections 2024-04-25T16:06:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.723+0000 7fbfc5268740 1 -- 172.21.15.31:0/1390524382 wait complete. 2024-04-25T16:06:18.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.724+0000 7fbfc5268740 20 client.16623 trim_cache size 0 max 16384 2024-04-25T16:06:18.729 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:06:18.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.728+0000 7fbfbd150640 1 -- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfa00afcf0 msgr2=0x7fbfa00b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:18.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.728+0000 7fbfbd150640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfa00afcf0 0x7fbfa00b2200 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbfac022000 tx=0x7fbfac01f000 comp rx=0 tx=0).stop 2024-04-25T16:06:18.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.728+0000 7fbfbd150640 1 -- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 msgr2=0x7fbfb8117090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:18.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.728+0000 7fbfbd150640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 0x7fbfb8117090 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004e40 tx=0x7fbfa8004e70 comp rx=0 tx=0).stop 2024-04-25T16:06:18.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.729+0000 7fbfbd150640 1 -- 172.21.15.31:0/4283949922 shutdown_connections 2024-04-25T16:06:18.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.729+0000 7fbfbd150640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbfa00afcf0 0x7fbfa00b2200 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.729+0000 7fbfbd150640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfb810fe90 0x7fbfb8118530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.729+0000 7fbfbd150640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb810b190 0x7fbfb81175d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.729+0000 7fbfbd150640 1 --2- 172.21.15.31:0/4283949922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbfb810a340 0x7fbfb8117090 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:18.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.729+0000 7fbfbd150640 1 -- 172.21.15.31:0/4283949922 >> 172.21.15.31:0/4283949922 conn(0x7fbfb8070320 msgr2=0x7fbfb810be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:18.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.729+0000 7fbfbd150640 1 -- 172.21.15.31:0/4283949922 shutdown_connections 2024-04-25T16:06:18.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:18.729+0000 7fbfbd150640 1 -- 172.21.15.31:0/4283949922 wait complete. 2024-04-25T16:06:18.777 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:06:18.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:18 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/637986214' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:06:19.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.317+0000 7fe270814640 1 -- 172.21.15.31:0/640478684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10a350 msgr2=0x7fe26c10ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:19.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.317+0000 7fe270814640 1 --2- 172.21.15.31:0/640478684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10a350 0x7fe26c10ac90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fe26401acb0 tx=0x7fe26403b920 comp rx=0 tx=0).stop 2024-04-25T16:06:19.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.317+0000 7fe270814640 1 -- 172.21.15.31:0/640478684 shutdown_connections 2024-04-25T16:06:19.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.317+0000 7fe270814640 1 --2- 172.21.15.31:0/640478684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe26c10fef0 0x7fe26c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.317+0000 7fe270814640 1 --2- 172.21.15.31:0/640478684 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe26c10b1d0 0x7fe26c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.317+0000 7fe270814640 1 --2- 172.21.15.31:0/640478684 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10a350 0x7fe26c10ac90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.317+0000 7fe270814640 1 -- 172.21.15.31:0/640478684 >> 172.21.15.31:0/640478684 conn(0x7fe26c070400 msgr2=0x7fe26c072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:19.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.317+0000 7fe270814640 1 -- 172.21.15.31:0/640478684 shutdown_connections 2024-04-25T16:06:19.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.317+0000 7fe270814640 1 -- 172.21.15.31:0/640478684 wait complete. 2024-04-25T16:06:19.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.319+0000 7fe270814640 1 Processor -- start 2024-04-25T16:06:19.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.319+0000 7fe270814640 1 -- start start 2024-04-25T16:06:19.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.320+0000 7fe270814640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe26c10a350 0x7fe26c1a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.320+0000 7fe270814640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10b1d0 0x7fe26c1a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.320+0000 7fe270814640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe26c10fef0 0x7fe26c1a7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.320+0000 7fe270814640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe26c113ba0 con 0x7fe26c10b1d0 2024-04-25T16:06:19.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.320+0000 7fe270814640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe26c113a20 con 0x7fe26c10a350 2024-04-25T16:06:19.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.320+0000 7fe270814640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe26c113d20 con 0x7fe26c10fef0 2024-04-25T16:06:19.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.320+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10b1d0 0x7fe26c1a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.321+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10b1d0 0x7fe26c1a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58616/0 (socket says 172.21.15.31:58616) 2024-04-25T16:06:19.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.321+0000 7fe26a59c640 1 -- 172.21.15.31:0/1871876100 learned_addr learned my addr 172.21.15.31:0/1871876100 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:19.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.321+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe26c10a350 0x7fe26c1a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.321+0000 7fe26a59c640 1 -- 172.21.15.31:0/1871876100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe26c10fef0 msgr2=0x7fe26c1a7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:19.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.321+0000 7fe26b59e640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe26c10fef0 0x7fe26c1a7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.321+0000 7fe26a59c640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe26c10fef0 0x7fe26c1a7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.321+0000 7fe26a59c640 1 -- 172.21.15.31:0/1871876100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe26c10a350 msgr2=0x7fe26c1a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:19.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.321+0000 7fe26a59c640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe26c10a350 0x7fe26c1a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.321+0000 7fe26a59c640 1 -- 172.21.15.31:0/1871876100 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26c1a82b0 con 0x7fe26c10b1d0 2024-04-25T16:06:19.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.322+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe26c10a350 0x7fe26c1a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:06:19.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.322+0000 7fe26b59e640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe26c10fef0 0x7fe26c1a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:06:19.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.322+0000 7fe26a59c640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10b1d0 0x7fe26c1a0cb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00b450 tx=0x7fe25c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:19.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.323+0000 7fe253fff640 1 -- 172.21.15.31:0/1871876100 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c015020 con 0x7fe26c10b1d0 2024-04-25T16:06:19.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.323+0000 7fe253fff640 1 -- 172.21.15.31:0/1871876100 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe25c011d40 con 0x7fe26c10b1d0 2024-04-25T16:06:19.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.323+0000 7fe253fff640 1 -- 172.21.15.31:0/1871876100 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c00f340 con 0x7fe26c10b1d0 2024-04-25T16:06:19.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.323+0000 7fe270814640 1 -- 172.21.15.31:0/1871876100 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe26c1a85a0 con 0x7fe26c10b1d0 2024-04-25T16:06:19.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.323+0000 7fe270814640 1 -- 172.21.15.31:0/1871876100 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe26c1a8b20 con 0x7fe26c10b1d0 2024-04-25T16:06:19.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.328+0000 7fe277fa4740 1 Processor -- start 2024-04-25T16:06:19.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.332+0000 7fe253fff640 1 -- 172.21.15.31:0/1871876100 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe25c018020 con 0x7fe26c10b1d0 2024-04-25T16:06:19.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.333+0000 7fe253fff640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2540afcc0 0x7fe2540b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.334+0000 7fe253fff640 1 -- 172.21.15.31:0/1871876100 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe25c0de610 con 0x7fe26c10b1d0 2024-04-25T16:06:19.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.335+0000 7fe277fa4740 1 -- start start 2024-04-25T16:06:19.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.335+0000 7fe26b59e640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2540afcc0 0x7fe2540b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.335+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 0x556f50684a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.335+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f50684f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.335+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502c16b0 0x556f50685c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.335+0000 7fe277fa4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556f505e8d80 con 0x556f502b94d0 2024-04-25T16:06:19.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.335+0000 7fe277fa4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556f505a2a20 con 0x556f502c16b0 2024-04-25T16:06:19.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.335+0000 7fe277fa4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556f505e4d40 con 0x556f502e2390 2024-04-25T16:06:19.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.336+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f50684f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.336+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f50684f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58624/0 (socket says 172.21.15.31:58624) 2024-04-25T16:06:19.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.336+0000 7fe26a59c640 1 -- 172.21.15.31:0/507972007 learned_addr learned my addr 172.21.15.31:0/507972007 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:19.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.336+0000 7fe26b59e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502c16b0 0x556f50685c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.339+0000 7fe26b59e640 1 -- 172.21.15.31:0/507972007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 msgr2=0x556f50684a00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:19.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.339+0000 7fe26b59e640 1 --2- 172.21.15.31:0/507972007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 0x556f50684a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.339+0000 7fe26b59e640 1 -- 172.21.15.31:0/507972007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 msgr2=0x556f50684f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:19.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.339+0000 7fe26b59e640 1 --2- 172.21.15.31:0/507972007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f50684f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.339+0000 7fe26b59e640 1 -- 172.21.15.31:0/507972007 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f505adc30 con 0x556f502c16b0 2024-04-25T16:06:19.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.340+0000 7fe26b59e640 1 --2- 172.21.15.31:0/507972007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502c16b0 0x556f50685c10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe26000c520 tx=0x7fe260004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e3547179ec94c95 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:19.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.341+0000 7fe2517fa640 1 -- 172.21.15.31:0/507972007 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe260015cf0 con 0x556f502c16b0 2024-04-25T16:06:19.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.341+0000 7fe26b59e640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2540afcc0 0x7fe2540b21d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe26c06cab0 tx=0x7fe260008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:19.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.342+0000 7fe2517fa640 1 -- 172.21.15.31:0/507972007 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe260022070 con 0x556f502c16b0 2024-04-25T16:06:19.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.343+0000 7fe277fa4740 1 -- 172.21.15.31:0/507972007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502c16b0 msgr2=0x556f50685c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:19.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.343+0000 7fe2517fa640 1 -- 172.21.15.31:0/507972007 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe260013790 con 0x556f502c16b0 2024-04-25T16:06:19.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.343+0000 7fe277fa4740 1 --2- 172.21.15.31:0/507972007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502c16b0 0x556f50685c10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe26000c520 tx=0x7fe260004770 comp rx=0 tx=0).stop 2024-04-25T16:06:19.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.344+0000 7fe277fa4740 1 -- 172.21.15.31:0/507972007 shutdown_connections 2024-04-25T16:06:19.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.344+0000 7fe277fa4740 1 --2- 172.21.15.31:0/507972007 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502c16b0 0x556f50685c10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.344+0000 7fe277fa4740 1 --2- 172.21.15.31:0/507972007 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f50684f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.344+0000 7fe277fa4740 1 --2- 172.21.15.31:0/507972007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 0x556f50684a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.344+0000 7fe277fa4740 1 -- 172.21.15.31:0/507972007 >> 172.21.15.31:0/507972007 conn(0x556f502d64e0 msgr2=0x556f50680210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:19.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.344+0000 7fe277fa4740 1 -- 172.21.15.31:0/507972007 shutdown_connections 2024-04-25T16:06:19.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.344+0000 7fe277fa4740 1 -- 172.21.15.31:0/507972007 wait complete. 2024-04-25T16:06:19.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.345+0000 7fe277fa4740 1 -- start start 2024-04-25T16:06:19.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.346+0000 7fe277fa4740 1 Processor -- start 2024-04-25T16:06:19.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.346+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502b94d0 0x556f5068e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.346+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502e2390 0x556f5068e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.346+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502a0f60 0x556f50695900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.347+0000 7fe277fa4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556f506867c0 con 0x556f502e2390 2024-04-25T16:06:19.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.347+0000 7fe277fa4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556f5056fdc0 con 0x556f502b94d0 2024-04-25T16:06:19.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.347+0000 7fe277fa4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556f50686940 con 0x556f502a0f60 2024-04-25T16:06:19.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.347+0000 7fe277fa4740 10 client.0 _ll_register_callbacks cb 0x556f5058b600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:19.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.347+0000 7fe26b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502a0f60 0x556f50695900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.348+0000 7fe26b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502a0f60 0x556f50695900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52648/0 (socket says 172.21.15.31:52648) 2024-04-25T16:06:19.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.348+0000 7fe26b59e640 1 -- 172.21.15.31:0/3273694321 learned_addr learned my addr 172.21.15.31:0/3273694321 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:19.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.348+0000 7fe26b59e640 1 -- 172.21.15.31:0/3273694321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502b94d0 msgr2=0x556f5068e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:19.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.348+0000 7fe26b59e640 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502b94d0 0x556f5068e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.348+0000 7fe26b59e640 1 -- 172.21.15.31:0/3273694321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502e2390 msgr2=0x556f5068e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:19.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.348+0000 7fe26b59e640 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502e2390 0x556f5068e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.348+0000 7fe26b59e640 1 -- 172.21.15.31:0/3273694321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f502fa3f0 con 0x556f502a0f60 2024-04-25T16:06:19.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.349+0000 7fe26b59e640 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502a0f60 0x556f50695900 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe260015d20 tx=0x7fe260015d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12e93ebb528669c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:19.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.349+0000 7fe24eff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:19.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.349+0000 7fe24eff5640 1 -- 172.21.15.31:0/3273694321 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26002b070 con 0x556f502a0f60 2024-04-25T16:06:19.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.349+0000 7fe277fa4740 1 -- 172.21.15.31:0/3273694321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe260022070 con 0x556f502a0f60 2024-04-25T16:06:19.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.350+0000 7fe24eff5640 1 -- 172.21.15.31:0/3273694321 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe260005da0 con 0x556f502a0f60 2024-04-25T16:06:19.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.350+0000 7fe24eff5640 1 -- 172.21.15.31:0/3273694321 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe26001d430 con 0x556f502a0f60 2024-04-25T16:06:19.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.350+0000 7fe277fa4740 10 client.26227 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:19.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.350+0000 7fe277fa4740 1 -- 172.21.15.31:0/3273694321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f503c47d0 con 0x556f502a0f60 2024-04-25T16:06:19.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.351+0000 7fe24eff5640 1 -- 172.21.15.31:0/3273694321 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe260022070 con 0x556f502a0f60 2024-04-25T16:06:19.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.353+0000 7fe24eff5640 1 -- 172.21.15.31:0/3273694321 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe26001c070 con 0x556f502a0f60 2024-04-25T16:06:19.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.353+0000 7fe277fa4740 10 client.26227 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:19.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.353+0000 7fe277fa4740 10 client.26227 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:06:19.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.353+0000 7fe277fa4740 20 client.26227 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:19.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.353+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x556f502763a0 0x556f5069c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.353+0000 7fe277fa4740 4 client.26227 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:19.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.353+0000 7fe277fa4740 1 -- 172.21.15.31:0/3273694321 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f505f6990 con 0x556f502763a0 2024-04-25T16:06:19.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.355+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x556f502763a0 0x556f5069c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.366+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x556f502763a0 0x556f5069c460 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=45a0068a039bcfe8 server_cookie=a2f385fab7f69e3f in_seq=0 out_seq=0 2024-04-25T16:06:19.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.367+0000 7fe24eff5640 10 client.26227 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:06:19.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.372+0000 7fe24eff5640 1 -- 172.21.15.31:0/3273694321 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556f505f6990 con 0x556f502763a0 2024-04-25T16:06:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.372+0000 7fe24eff5640 10 client.26227 handle_command_reply: tid=0 2024-04-25T16:06:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.372+0000 7fe277fa4740 1 client.26227 shutdown 2024-04-25T16:06:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.372+0000 7fe277fa4740 1 -- 172.21.15.31:0/3273694321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502a0f60 msgr2=0x556f50695900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.372+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502a0f60 0x556f50695900 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fe260015d20 tx=0x7fe260015d50 comp rx=0 tx=0).stop 2024-04-25T16:06:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.373+0000 7fe277fa4740 1 -- 172.21.15.31:0/3273694321 shutdown_connections 2024-04-25T16:06:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.373+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x556f502763a0 0x556f5069c460 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.373+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502a0f60 0x556f50695900 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.373+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502e2390 0x556f5068e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.373+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3273694321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502b94d0 0x556f5068e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.373+0000 7fe277fa4740 1 -- 172.21.15.31:0/3273694321 >> 172.21.15.31:0/3273694321 conn(0x556f502d64e0 msgr2=0x556f5067f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:19.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.373+0000 7fe24eff5640 0 client.26227 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:06:19.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.375+0000 7fe277fa4740 1 -- 172.21.15.31:0/3273694321 shutdown_connections 2024-04-25T16:06:19.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.380+0000 7fe277fa4740 1 -- 172.21.15.31:0/3273694321 wait complete. 2024-04-25T16:06:19.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.380+0000 7fe277fa4740 20 client.26227 trim_cache size 0 max 16384 2024-04-25T16:06:19.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.446+0000 7fe277fa4740 1 Processor -- start 2024-04-25T16:06:19.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.446+0000 7fe277fa4740 1 -- start start 2024-04-25T16:06:19.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.447+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f506896f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.448+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502b94d0 0x556f50683720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.448+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502e2390 0x556f50683c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.448+0000 7fe26ad9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f506896f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.448+0000 7fe26ad9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f506896f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34560/0 (socket says 172.21.15.31:34560) 2024-04-25T16:06:19.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.449+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502b94d0 0x556f50683720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.449+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502b94d0 0x556f50683720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52662/0 (socket says 172.21.15.31:52662) 2024-04-25T16:06:19.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.449+0000 7fe26ad9d640 1 -- 172.21.15.31:0/1976628917 learned_addr learned my addr 172.21.15.31:0/1976628917 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:19.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.449+0000 7fe277fa4740 1 -- 172.21.15.31:0/1976628917 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe25800e2d0 con 0x556f502e2390 2024-04-25T16:06:19.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.449+0000 7fe277fa4740 1 -- 172.21.15.31:0/1976628917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe25800fe10 con 0x556f502a0f60 2024-04-25T16:06:19.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.449+0000 7fe277fa4740 1 -- 172.21.15.31:0/1976628917 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe25800e9a0 con 0x556f502b94d0 2024-04-25T16:06:19.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.450+0000 7fe26ad9d640 1 -- 172.21.15.31:0/1976628917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502b94d0 msgr2=0x556f50683720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:19.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.450+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/1976628917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502b94d0 0x556f50683720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.450+0000 7fe26ad9d640 1 -- 172.21.15.31:0/1976628917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502e2390 msgr2=0x556f50683c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:19.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.450+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/1976628917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502e2390 0x556f50683c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.450+0000 7fe26ad9d640 1 -- 172.21.15.31:0/1976628917 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f502c18c0 con 0x556f502a0f60 2024-04-25T16:06:19.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.451+0000 7fe26a59c640 1 --2- 172.21.15.31:0/1976628917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502b94d0 0x556f50683720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:06:19.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.451+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/1976628917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f506896f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe26401aa70 tx=0x7fe264000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=711f114d18d72389 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:19.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.465+0000 7fe250ff9640 1 -- 172.21.15.31:0/1976628917 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264045070 con 0x556f502a0f60 2024-04-25T16:06:19.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.465+0000 7fe250ff9640 1 -- 172.21.15.31:0/1976628917 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe264003e20 con 0x556f502a0f60 2024-04-25T16:06:19.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.465+0000 7fe250ff9640 1 -- 172.21.15.31:0/1976628917 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe264053650 con 0x556f502a0f60 2024-04-25T16:06:19.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.467+0000 7fe277fa4740 1 -- 172.21.15.31:0/1976628917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 msgr2=0x556f506896f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:19.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.467+0000 7fe277fa4740 1 --2- 172.21.15.31:0/1976628917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f506896f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fe26401aa70 tx=0x7fe264000f40 comp rx=0 tx=0).stop 2024-04-25T16:06:19.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.478+0000 7fe277fa4740 1 -- 172.21.15.31:0/1976628917 shutdown_connections 2024-04-25T16:06:19.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.478+0000 7fe277fa4740 1 --2- 172.21.15.31:0/1976628917 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502e2390 0x556f50683c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.478+0000 7fe277fa4740 1 --2- 172.21.15.31:0/1976628917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502b94d0 0x556f50683720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.478+0000 7fe277fa4740 1 --2- 172.21.15.31:0/1976628917 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f506896f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.478+0000 7fe277fa4740 1 -- 172.21.15.31:0/1976628917 >> 172.21.15.31:0/1976628917 conn(0x556f502d64e0 msgr2=0x556f50683010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:19.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.479+0000 7fe277fa4740 1 -- 172.21.15.31:0/1976628917 shutdown_connections 2024-04-25T16:06:19.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.479+0000 7fe277fa4740 1 -- 172.21.15.31:0/1976628917 wait complete. 2024-04-25T16:06:19.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.480+0000 7fe277fa4740 1 -- start start 2024-04-25T16:06:19.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.481+0000 7fe277fa4740 1 Processor -- start 2024-04-25T16:06:19.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.482+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f5068fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.482+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f506900e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.482+0000 7fe277fa4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 0x556f50702800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.482+0000 7fe277fa4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe25800f740 con 0x556f502b94d0 2024-04-25T16:06:19.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.483+0000 7fe277fa4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe25800f070 con 0x556f502a0f60 2024-04-25T16:06:19.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.483+0000 7fe277fa4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe25800c360 con 0x556f502e2390 2024-04-25T16:06:19.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.483+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f506900e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.483+0000 7fe26a59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f506900e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58648/0 (socket says 172.21.15.31:58648) 2024-04-25T16:06:19.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.483+0000 7fe26a59c640 1 -- 172.21.15.31:0/3256557290 learned_addr learned my addr 172.21.15.31:0/3256557290 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:19.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.484+0000 7fe277fa4740 10 client.0 _ll_register_callbacks cb 0x7fe25c0a2320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:19.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.486+0000 7fe26b59e640 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 0x556f50702800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.486+0000 7fe26a59c640 1 -- 172.21.15.31:0/3256557290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 msgr2=0x556f50702800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:19.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.486+0000 7fe26a59c640 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 0x556f50702800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.486+0000 7fe26a59c640 1 -- 172.21.15.31:0/3256557290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 msgr2=0x556f5068fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:19.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.486+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f5068fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.486+0000 7fe26a59c640 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f5068fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.486+0000 7fe26a59c640 1 -- 172.21.15.31:0/3256557290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f506950a0 con 0x556f502b94d0 2024-04-25T16:06:19.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.486+0000 7fe26b59e640 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 0x556f50702800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:06:19.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.487+0000 7fe26a59c640 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f506900e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009fd0 tx=0x7fe25c0a7020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f41230c7de176d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:19.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.493+0000 7fe24eff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:19.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.493+0000 7fe24eff5640 1 -- 172.21.15.31:0/3256557290 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0c0070 con 0x556f502b94d0 2024-04-25T16:06:19.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.493+0000 7fe24eff5640 1 -- 172.21.15.31:0/3256557290 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe25c0a3090 con 0x556f502b94d0 2024-04-25T16:06:19.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.494+0000 7fe277fa4740 1 -- 172.21.15.31:0/3256557290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe264003e20 con 0x556f502b94d0 2024-04-25T16:06:19.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.495+0000 7fe24eff5640 1 -- 172.21.15.31:0/3256557290 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe25c0aae50 con 0x556f502b94d0 2024-04-25T16:06:19.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.496+0000 7fe277fa4740 10 client.16635 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:19.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.496+0000 7fe277fa4740 1 -- 172.21.15.31:0/3256557290 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556f50703bb0 con 0x556f502b94d0 2024-04-25T16:06:19.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.497+0000 7fe24eff5640 1 -- 172.21.15.31:0/3256557290 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe25c0af900 con 0x556f502b94d0 2024-04-25T16:06:19.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.497+0000 7fe24eff5640 1 -- 172.21.15.31:0/3256557290 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe25c0c90b0 con 0x556f502b94d0 2024-04-25T16:06:19.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.498+0000 7fe277fa4740 10 client.16635 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:19.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.498+0000 7fe277fa4740 10 client.16635 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:06:19.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.498+0000 7fe277fa4740 20 client.16635 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:19.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.498+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x556f507045e0 0x556f50706a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:19.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.498+0000 7fe277fa4740 4 client.16635 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:06:19.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.498+0000 7fe277fa4740 1 -- 172.21.15.31:0/3256557290 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fe264003e20 con 0x556f507045e0 2024-04-25T16:06:19.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.517+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x556f507045e0 0x556f50706a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:19.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.540+0000 7fe26ad9d640 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x556f507045e0 0x556f50706a00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=845368bceeb70f5 server_cookie=72818b6be3cc4eeb in_seq=0 out_seq=0 2024-04-25T16:06:19.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.540+0000 7fe24eff5640 10 client.16635 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:06:19.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.541+0000 7fe24eff5640 1 -- 172.21.15.31:0/3256557290 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fe264003e20 con 0x556f507045e0 2024-04-25T16:06:19.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.541+0000 7fe24eff5640 10 client.16635 handle_command_reply: tid=0 2024-04-25T16:06:19.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.541+0000 7fe277fa4740 1 client.16635 shutdown 2024-04-25T16:06:19.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.553+0000 7fe277fa4740 1 -- 172.21.15.31:0/3256557290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 msgr2=0x556f506900e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.554+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f506900e0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009fd0 tx=0x7fe25c0a7020 comp rx=0 tx=0).stop 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.554+0000 7fe277fa4740 1 -- 172.21.15.31:0/3256557290 shutdown_connections 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.554+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x556f507045e0 0x556f50706a00 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.554+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556f502e2390 0x556f50702800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.554+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556f502b94d0 0x556f506900e0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.554+0000 7fe277fa4740 1 --2- 172.21.15.31:0/3256557290 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556f502a0f60 0x556f5068fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.554+0000 7fe277fa4740 1 -- 172.21.15.31:0/3256557290 >> 172.21.15.31:0/3256557290 conn(0x556f502d64e0 msgr2=0x556f50683a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.554+0000 7fe24eff5640 0 client.16635 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.555+0000 7fe277fa4740 1 -- 172.21.15.31:0/3256557290 shutdown_connections 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.555+0000 7fe277fa4740 1 -- 172.21.15.31:0/3256557290 wait complete. 2024-04-25T16:06:19.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.555+0000 7fe277fa4740 20 client.16635 trim_cache size 0 max 16384 2024-04-25T16:06:19.561 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:06:19.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.585+0000 7fe270814640 1 -- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2540afcc0 msgr2=0x7fe2540b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:19.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.585+0000 7fe270814640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2540afcc0 0x7fe2540b21d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe26c06cab0 tx=0x7fe260008040 comp rx=0 tx=0).stop 2024-04-25T16:06:19.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.585+0000 7fe270814640 1 -- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10b1d0 msgr2=0x7fe26c1a0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:19.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.585+0000 7fe270814640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10b1d0 0x7fe26c1a0cb0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00b450 tx=0x7fe25c00b910 comp rx=0 tx=0).stop 2024-04-25T16:06:19.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.588+0000 7fe270814640 1 -- 172.21.15.31:0/1871876100 shutdown_connections 2024-04-25T16:06:19.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.588+0000 7fe270814640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe2540afcc0 0x7fe2540b21d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.588+0000 7fe270814640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe26c10fef0 0x7fe26c1a7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.588+0000 7fe270814640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe26c10b1d0 0x7fe26c1a0cb0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.588+0000 7fe270814640 1 --2- 172.21.15.31:0/1871876100 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe26c10a350 0x7fe26c1a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:19.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.588+0000 7fe270814640 1 -- 172.21.15.31:0/1871876100 >> 172.21.15.31:0/1871876100 conn(0x7fe26c070400 msgr2=0x7fe26c10dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:19.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.589+0000 7fe270814640 1 -- 172.21.15.31:0/1871876100 shutdown_connections 2024-04-25T16:06:19.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:19.590+0000 7fe270814640 1 -- 172.21.15.31:0/1871876100 wait complete. 2024-04-25T16:06:19.641 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:19 smithi031 ceph-mon[29140]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 861 MiB/s wr, 2.52k op/s 2024-04-25T16:06:19.643 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:06:19.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:19 smithi186 ceph-mon[33258]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 861 MiB/s wr, 2.52k op/s 2024-04-25T16:06:19.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:19 smithi063 ceph-mon[32413]: pgmap v501: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 861 MiB/s wr, 2.52k op/s 2024-04-25T16:06:20.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.115+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/2637645661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4810a0f0 msgr2=0x7f5c4810a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.115+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/2637645661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4810a0f0 0x7f5c4810a500 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5c38009860 tx=0x7f5c3802a360 comp rx=0 tx=0).stop 2024-04-25T16:06:20.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.117+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/2637645661 shutdown_connections 2024-04-25T16:06:20.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.117+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/2637645661 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c4810b990 0x7f5c4810c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.117+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/2637645661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c4810b040 0x7f5c4810b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.117+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/2637645661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4810a0f0 0x7f5c4810a500 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.117+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/2637645661 >> 172.21.15.31:0/2637645661 conn(0x7f5c480707b0 msgr2=0x7f5c48072c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.120+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/2637645661 shutdown_connections 2024-04-25T16:06:20.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.121+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/2637645661 wait complete. 2024-04-25T16:06:20.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.122+0000 7f5c4d4eb640 1 Processor -- start 2024-04-25T16:06:20.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.122+0000 7f5c4d4eb640 1 -- start start 2024-04-25T16:06:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c4d4eb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c4810a0f0 0x7f5c481a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c4d4eb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c4810b040 0x7f5c481a9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c4d4eb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4810b990 0x7f5c481b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c4d4eb640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c48119e90 con 0x7f5c4810b040 2024-04-25T16:06:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c4d4eb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5c48119d10 con 0x7f5c4810a0f0 2024-04-25T16:06:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c4d4eb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c4811a010 con 0x7f5c4810b990 2024-04-25T16:06:20.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c4810a0f0 0x7f5c481a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c4810a0f0 0x7f5c481a8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34592/0 (socket says 172.21.15.31:34592) 2024-04-25T16:06:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c47fff640 1 -- 172.21.15.31:0/3159794157 learned_addr learned my addr 172.21.15.31:0/3159794157 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4810b990 0x7f5c481b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.123+0000 7f5c477fe640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c4810b040 0x7f5c481a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.124+0000 7f5c477fe640 1 -- 172.21.15.31:0/3159794157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4810b990 msgr2=0x7f5c481b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.124+0000 7f5c477fe640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4810b990 0x7f5c481b02f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.124+0000 7f5c477fe640 1 -- 172.21.15.31:0/3159794157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c4810a0f0 msgr2=0x7f5c481a8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.124+0000 7f5c477fe640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c4810a0f0 0x7f5c481a8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.124+0000 7f5c477fe640 1 -- 172.21.15.31:0/3159794157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c481b0960 con 0x7f5c4810b040 2024-04-25T16:06:20.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.133+0000 7f5c477fe640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c4810b040 0x7f5c481a9360 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00e790 tx=0x7f5c3c00ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.139+0000 7f5c457fa640 1 -- 172.21.15.31:0/3159794157 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c00eea0 con 0x7f5c4810b040 2024-04-25T16:06:20.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.139+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/3159794157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c481b0c50 con 0x7f5c4810b040 2024-04-25T16:06:20.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.139+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/3159794157 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c481b11a0 con 0x7f5c4810b040 2024-04-25T16:06:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.145+0000 7f5c55604740 1 Processor -- start 2024-04-25T16:06:20.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.146+0000 7f5c457fa640 1 -- 172.21.15.31:0/3159794157 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c3c00ccc0 con 0x7f5c4810b040 2024-04-25T16:06:20.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.147+0000 7f5c55604740 1 -- start start 2024-04-25T16:06:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.147+0000 7f5c55604740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.147+0000 7f5c55604740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.148+0000 7f5c55604740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69c9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.148+0000 7f5c55604740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f7e692cd80 con 0x55f7e66056b0 2024-04-25T16:06:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.148+0000 7f5c55604740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f7e68e6a20 con 0x55f7e65fd4d0 2024-04-25T16:06:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.148+0000 7f5c55604740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f7e6928d40 con 0x55f7e6626390 2024-04-25T16:06:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.148+0000 7f5c457fa640 1 -- 172.21.15.31:0/3159794157 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c018680 con 0x7f5c4810b040 2024-04-25T16:06:20.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.148+0000 7f5c457fa640 1 -- 172.21.15.31:0/3159794157 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5c3c018900 con 0x7f5c4810b040 2024-04-25T16:06:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.149+0000 7f5c457fa640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5c300afc10 0x7f5c300b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.150+0000 7f5c457fa640 1 -- 172.21.15.31:0/3159794157 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5c3c0dfdb0 con 0x7f5c4810b040 2024-04-25T16:06:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.152+0000 7f5c4ccea640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.152+0000 7f5c4ccea640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52680/0 (socket says 172.21.15.31:52680) 2024-04-25T16:06:20.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.152+0000 7f5c4ccea640 1 -- 172.21.15.31:0/3331571384 learned_addr learned my addr 172.21.15.31:0/3331571384 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.153+0000 7f5c47fff640 1 --2- 172.21.15.31:0/3331571384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.153+0000 7f5c4ccea640 1 -- 172.21.15.31:0/3331571384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 msgr2=0x55f7e69c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.153+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/3331571384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.153+0000 7f5c4ccea640 1 -- 172.21.15.31:0/3331571384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 msgr2=0x55f7e69c9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.153+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/3331571384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69c9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.153+0000 7f5c4ccea640 1 -- 172.21.15.31:0/3331571384 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7e68f1c30 con 0x55f7e6626390 2024-04-25T16:06:20.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.155+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/3331571384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c8a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000e440 tx=0x7f5c4000e900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c380734f3e2e4fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.162+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5c300afc10 0x7f5c300b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.163+0000 7f5c2e7fc640 1 -- 172.21.15.31:0/3331571384 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4001c070 con 0x55f7e6626390 2024-04-25T16:06:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.164+0000 7f5c2e7fc640 1 -- 172.21.15.31:0/3331571384 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c400078e0 con 0x55f7e6626390 2024-04-25T16:06:20.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.164+0000 7f5c2e7fc640 1 -- 172.21.15.31:0/3331571384 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4000c960 con 0x55f7e6626390 2024-04-25T16:06:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.166+0000 7f5c55604740 1 -- 172.21.15.31:0/3331571384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 msgr2=0x55f7e69c8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.166+0000 7f5c55604740 1 --2- 172.21.15.31:0/3331571384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c8a00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000e440 tx=0x7f5c4000e900 comp rx=0 tx=0).stop 2024-04-25T16:06:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.166+0000 7f5c55604740 1 -- 172.21.15.31:0/3331571384 shutdown_connections 2024-04-25T16:06:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.166+0000 7f5c55604740 1 --2- 172.21.15.31:0/3331571384 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69c9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.166+0000 7f5c55604740 1 --2- 172.21.15.31:0/3331571384 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.166+0000 7f5c55604740 1 --2- 172.21.15.31:0/3331571384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c8a00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.166+0000 7f5c55604740 1 -- 172.21.15.31:0/3331571384 >> 172.21.15.31:0/3331571384 conn(0x55f7e661a4e0 msgr2=0x55f7e69c4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.172+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5c300afc10 0x7f5c300b2120 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5c40019000 tx=0x7f5c4000ccd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.178+0000 7f5c55604740 1 -- 172.21.15.31:0/3331571384 shutdown_connections 2024-04-25T16:06:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.180+0000 7f5c55604740 1 -- 172.21.15.31:0/3331571384 wait complete. 2024-04-25T16:06:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.180+0000 7f5c55604740 1 -- start start 2024-04-25T16:06:20.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.181+0000 7f5c55604740 1 Processor -- start 2024-04-25T16:06:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c55604740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69d2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c55604740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69d2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c55604740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c55604740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f7e69ca720 con 0x55f7e66056b0 2024-04-25T16:06:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c55604740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f7e68b3dc0 con 0x55f7e65fd4d0 2024-04-25T16:06:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c55604740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f7e69ca8a0 con 0x55f7e6626390 2024-04-25T16:06:20.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c55604740 10 client.0 _ll_register_callbacks cb 0x55f7e68cf600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c4ccea640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c4ccea640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52684/0 (socket says 172.21.15.31:52684) 2024-04-25T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.184+0000 7f5c4ccea640 1 -- 172.21.15.31:0/3627084707 learned_addr learned my addr 172.21.15.31:0/3627084707 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.185+0000 7f5c4ccea640 1 -- 172.21.15.31:0/3627084707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 msgr2=0x55f7e69d2790 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.185+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69d2790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.185+0000 7f5c4ccea640 1 -- 172.21.15.31:0/3627084707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 msgr2=0x55f7e69d2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.185+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69d2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.185+0000 7f5c4ccea640 1 -- 172.21.15.31:0/3627084707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7e663e3f0 con 0x55f7e6626390 2024-04-25T16:06:20.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.185+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d9c30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000e040 tx=0x7f5c4000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d2ad21ab7e5a0e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.195+0000 7f5c277fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:20.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.196+0000 7f5c277fe640 1 -- 172.21.15.31:0/3627084707 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4001c040 con 0x55f7e6626390 2024-04-25T16:06:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.196+0000 7f5c55604740 1 -- 172.21.15.31:0/3627084707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c400078e0 con 0x55f7e6626390 2024-04-25T16:06:20.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.196+0000 7f5c277fe640 1 -- 172.21.15.31:0/3627084707 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5c4000c890 con 0x55f7e6626390 2024-04-25T16:06:20.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.202+0000 7f5c277fe640 1 -- 172.21.15.31:0/3627084707 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5c40017530 con 0x55f7e6626390 2024-04-25T16:06:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.204+0000 7f5c55604740 10 client.26242 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.204+0000 7f5c55604740 1 -- 172.21.15.31:0/3627084707 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f7e67087d0 con 0x55f7e6626390 2024-04-25T16:06:20.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.204+0000 7f5c277fe640 1 -- 172.21.15.31:0/3627084707 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c400078e0 con 0x55f7e6626390 2024-04-25T16:06:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.206+0000 7f5c277fe640 1 -- 172.21.15.31:0/3627084707 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5c4001f040 con 0x55f7e6626390 2024-04-25T16:06:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.206+0000 7f5c55604740 10 client.26242 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.206+0000 7f5c55604740 10 client.26242 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:06:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.206+0000 7f5c55604740 20 client.26242 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.206+0000 7f5c55604740 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55f7e65e4f60 0x55f7e69f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.206+0000 7f5c55604740 4 client.26242 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:20.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.206+0000 7f5c55604740 1 -- 172.21.15.31:0/3627084707 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f7e693a990 con 0x55f7e65e4f60 2024-04-25T16:06:20.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.207+0000 7f5c47fff640 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55f7e65e4f60 0x55f7e69f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.208+0000 7f5c277fe640 10 client.26242 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:06:20.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.208+0000 7f5c47fff640 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55f7e65e4f60 0x55f7e69f1a00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=85ce2205a900fae2 server_cookie=b7e90e076b8eae81 in_seq=0 out_seq=0 2024-04-25T16:06:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.209+0000 7f5c277fe640 1 -- 172.21.15.31:0/3627084707 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f7e693a990 con 0x55f7e65e4f60 2024-04-25T16:06:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.209+0000 7f5c277fe640 10 client.26242 handle_command_reply: tid=0 2024-04-25T16:06:20.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.209+0000 7f5c55604740 1 client.26242 shutdown 2024-04-25T16:06:20.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.214+0000 7f5c55604740 1 -- 172.21.15.31:0/3627084707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 msgr2=0x55f7e69d9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.214+0000 7f5c55604740 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d9c30 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5c4000e040 tx=0x7f5c4000cbc0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.215+0000 7f5c55604740 1 -- 172.21.15.31:0/3627084707 shutdown_connections 2024-04-25T16:06:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.215+0000 7f5c55604740 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d9c30 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.215+0000 7f5c55604740 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69d2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.215+0000 7f5c55604740 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55f7e65e4f60 0x55f7e69f1a00 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.215+0000 7f5c55604740 1 --2- 172.21.15.31:0/3627084707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69d2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.215+0000 7f5c277fe640 0 client.26242 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:06:20.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.215+0000 7f5c55604740 1 -- 172.21.15.31:0/3627084707 >> 172.21.15.31:0/3627084707 conn(0x55f7e661a4e0 msgr2=0x55f7e69c33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.216+0000 7f5c55604740 1 -- 172.21.15.31:0/3627084707 shutdown_connections 2024-04-25T16:06:20.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.216+0000 7f5c55604740 1 -- 172.21.15.31:0/3627084707 wait complete. 2024-04-25T16:06:20.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.217+0000 7f5c55604740 20 client.26242 trim_cache size 0 max 16384 2024-04-25T16:06:20.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.251+0000 7f5c55604740 1 Processor -- start 2024-04-25T16:06:20.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.251+0000 7f5c55604740 1 -- start start 2024-04-25T16:06:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.252+0000 7f5c55604740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.252+0000 7f5c55604740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69c8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.252+0000 7f5c55604740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.252+0000 7f5c55604740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3400e360 con 0x55f7e66056b0 2024-04-25T16:06:20.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.252+0000 7f5c55604740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3400fea0 con 0x55f7e65fd4d0 2024-04-25T16:06:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.252+0000 7f5c55604740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3400ea30 con 0x55f7e6626390 2024-04-25T16:06:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.252+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.252+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34608/0 (socket says 172.21.15.31:34608) 2024-04-25T16:06:20.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.252+0000 7f5c47fff640 1 -- 172.21.15.31:0/2454107302 learned_addr learned my addr 172.21.15.31:0/2454107302 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.255+0000 7f5c477fe640 1 --2- 172.21.15.31:0/2454107302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69c8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.256+0000 7f5c4ccea640 1 --2- 172.21.15.31:0/2454107302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.258+0000 7f5c47fff640 1 -- 172.21.15.31:0/2454107302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 msgr2=0x55f7e69c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.258+0000 7f5c47fff640 1 --2- 172.21.15.31:0/2454107302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.258+0000 7f5c47fff640 1 -- 172.21.15.31:0/2454107302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 msgr2=0x55f7e69c8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.258+0000 7f5c47fff640 1 --2- 172.21.15.31:0/2454107302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69c8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.258+0000 7f5c47fff640 1 -- 172.21.15.31:0/2454107302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7e69c8480 con 0x55f7e65fd4d0 2024-04-25T16:06:20.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.263+0000 7f5c47fff640 1 --2- 172.21.15.31:0/2454107302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5c38005eb0 tx=0x7f5c38005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d14791461e0a3764 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.266+0000 7f5c2dffb640 1 -- 172.21.15.31:0/2454107302 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c38037070 con 0x55f7e65fd4d0 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.266+0000 7f5c2dffb640 1 -- 172.21.15.31:0/2454107302 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c38005020 con 0x55f7e65fd4d0 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.266+0000 7f5c2dffb640 1 -- 172.21.15.31:0/2454107302 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c380307b0 con 0x55f7e65fd4d0 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.267+0000 7f5c55604740 1 -- 172.21.15.31:0/2454107302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 msgr2=0x55f7e69c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.267+0000 7f5c55604740 1 --2- 172.21.15.31:0/2454107302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8980 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5c38005eb0 tx=0x7f5c38005f10 comp rx=0 tx=0).stop 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.268+0000 7f5c55604740 1 -- 172.21.15.31:0/2454107302 shutdown_connections 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.268+0000 7f5c55604740 1 --2- 172.21.15.31:0/2454107302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69c29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.268+0000 7f5c55604740 1 --2- 172.21.15.31:0/2454107302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69c8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.268+0000 7f5c55604740 1 --2- 172.21.15.31:0/2454107302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65fd4d0 0x55f7e69c8980 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5c38005eb0 tx=0x7f5c38005f10 comp rx=0 tx=0).stop 2024-04-25T16:06:20.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.268+0000 7f5c55604740 1 -- 172.21.15.31:0/2454107302 >> 172.21.15.31:0/2454107302 conn(0x55f7e661a4e0 msgr2=0x55f7e69cd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.273+0000 7f5c55604740 1 -- 172.21.15.31:0/2454107302 shutdown_connections 2024-04-25T16:06:20.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.276+0000 7f5c55604740 1 -- 172.21.15.31:0/2454107302 wait complete. 2024-04-25T16:06:20.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.276+0000 7f5c55604740 1 -- start start 2024-04-25T16:06:20.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.277+0000 7f5c55604740 1 Processor -- start 2024-04-25T16:06:20.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.280+0000 7f5c55604740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69d9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.280+0000 7f5c55604740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.280+0000 7f5c55604740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65ba3a0 0x55f7e69d61c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.280+0000 7f5c55604740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3400f7d0 con 0x55f7e66056b0 2024-04-25T16:06:20.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.280+0000 7f5c55604740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3400f100 con 0x55f7e65ba3a0 2024-04-25T16:06:20.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.280+0000 7f5c55604740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5c3400c3f0 con 0x55f7e6626390 2024-04-25T16:06:20.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.280+0000 7f5c55604740 10 client.0 _ll_register_callbacks cb 0x7f5c4001a070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.281+0000 7f5c4ccea640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65ba3a0 0x55f7e69d61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.281+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69d9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.281+0000 7f5c47fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69d9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58692/0 (socket says 172.21.15.31:58692) 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.281+0000 7f5c4ccea640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65ba3a0 0x55f7e69d61c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34614/0 (socket says 172.21.15.31:34614) 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.281+0000 7f5c47fff640 1 -- 172.21.15.31:0/1587274877 learned_addr learned my addr 172.21.15.31:0/1587274877 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.281+0000 7f5c477fe640 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.282+0000 7f5c477fe640 1 -- 172.21.15.31:0/1587274877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65ba3a0 msgr2=0x55f7e69d61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.282+0000 7f5c477fe640 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65ba3a0 0x55f7e69d61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.282+0000 7f5c477fe640 1 -- 172.21.15.31:0/1587274877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 msgr2=0x55f7e69d9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.282+0000 7f5c477fe640 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69d9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.282+0000 7f5c477fe640 1 -- 172.21.15.31:0/1587274877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7e6a57da0 con 0x55f7e6626390 2024-04-25T16:06:20.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.282+0000 7f5c477fe640 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d5c80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55f7e69d3800 tx=0x7f5c3c0a8860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b26f3acaff3fa3e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.282+0000 7f5c277fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:20.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.282+0000 7f5c277fe640 1 -- 172.21.15.31:0/1587274877 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c0a7dd0 con 0x55f7e6626390 2024-04-25T16:06:20.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.283+0000 7f5c277fe640 1 -- 172.21.15.31:0/1587274877 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5c3c0a3e40 con 0x55f7e6626390 2024-04-25T16:06:20.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.285+0000 7f5c55604740 1 -- 172.21.15.31:0/1587274877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c38005020 con 0x55f7e6626390 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.285+0000 7f5c277fe640 1 -- 172.21.15.31:0/1587274877 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5c3c0a89c0 con 0x55f7e6626390 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.285+0000 7f5c55604740 10 client.26248 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.285+0000 7f5c55604740 1 -- 172.21.15.31:0/1587274877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f7e6a58f90 con 0x55f7e6626390 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.286+0000 7f5c277fe640 1 -- 172.21.15.31:0/1587274877 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c38005020 con 0x55f7e6626390 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.286+0000 7f5c277fe640 1 -- 172.21.15.31:0/1587274877 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5c3c0c9190 con 0x55f7e6626390 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.286+0000 7f5c55604740 10 client.26248 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.286+0000 7f5c55604740 10 client.26248 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.286+0000 7f5c55604740 20 client.26248 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.286+0000 7f5c55604740 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55f7e6a599c0 0x55f7e6a5bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.286+0000 7f5c55604740 4 client.26248 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:06:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.286+0000 7f5c55604740 1 -- 172.21.15.31:0/1587274877 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f7e6934800 con 0x55f7e6a599c0 2024-04-25T16:06:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.287+0000 7f5c47fff640 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55f7e6a599c0 0x55f7e6a5bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.289+0000 7f5c277fe640 10 client.26248 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:06:20.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.289+0000 7f5c47fff640 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55f7e6a599c0 0x55f7e6a5bde0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1a0eb3b48e35ac0e server_cookie=864ea3af9080748a in_seq=0 out_seq=0 2024-04-25T16:06:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.291+0000 7f5c277fe640 1 -- 172.21.15.31:0/1587274877 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f7e6934800 con 0x55f7e6a599c0 2024-04-25T16:06:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.291+0000 7f5c277fe640 10 client.26248 handle_command_reply: tid=0 2024-04-25T16:06:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.291+0000 7f5c55604740 1 client.26248 shutdown 2024-04-25T16:06:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.291+0000 7f5c55604740 1 -- 172.21.15.31:0/1587274877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 msgr2=0x55f7e69d5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.291+0000 7f5c55604740 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d5c80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x55f7e69d3800 tx=0x7f5c3c0a8860 comp rx=0 tx=0).stop 2024-04-25T16:06:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.292+0000 7f5c55604740 1 -- 172.21.15.31:0/1587274877 shutdown_connections 2024-04-25T16:06:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.292+0000 7f5c55604740 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55f7e6a599c0 0x55f7e6a5bde0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.292+0000 7f5c55604740 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7e65ba3a0 0x55f7e69d61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.292+0000 7f5c55604740 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f7e6626390 0x55f7e69d5c80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.292+0000 7f5c55604740 1 --2- 172.21.15.31:0/1587274877 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7e66056b0 0x55f7e69d9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.292+0000 7f5c55604740 1 -- 172.21.15.31:0/1587274877 >> 172.21.15.31:0/1587274877 conn(0x55f7e661a4e0 msgr2=0x55f7e69d2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.292+0000 7f5c277fe640 0 client.26248 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:06:20.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.292+0000 7f5c55604740 1 -- 172.21.15.31:0/1587274877 shutdown_connections 2024-04-25T16:06:20.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.292+0000 7f5c55604740 1 -- 172.21.15.31:0/1587274877 wait complete. 2024-04-25T16:06:20.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.294+0000 7f5c55604740 20 client.26248 trim_cache size 0 max 16384 2024-04-25T16:06:20.298 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:06:20.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.298+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5c300afc10 msgr2=0x7f5c300b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.298+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5c300afc10 0x7f5c300b2120 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5c40019000 tx=0x7f5c4000ccd0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.298+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c4810b040 msgr2=0x7f5c481a9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.298+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c4810b040 0x7f5c481a9360 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00e790 tx=0x7f5c3c00ec50 comp rx=0 tx=0).stop 2024-04-25T16:06:20.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.299+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/3159794157 shutdown_connections 2024-04-25T16:06:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.299+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5c300afc10 0x7f5c300b2120 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.299+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4810b990 0x7f5c481b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.299+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5c4810b040 0x7f5c481a9360 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.299+0000 7f5c4d4eb640 1 --2- 172.21.15.31:0/3159794157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5c4810a0f0 0x7f5c481a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.299+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/3159794157 >> 172.21.15.31:0/3159794157 conn(0x7f5c480707b0 msgr2=0x7f5c48111620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.299+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/3159794157 shutdown_connections 2024-04-25T16:06:20.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.303+0000 7f5c4d4eb640 1 -- 172.21.15.31:0/3159794157 wait complete. 2024-04-25T16:06:20.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:06:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.710+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/3170930273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2b810a420 msgr2=0x7fb2b810add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.710+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/3170930273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2b810a420 0x7fb2b810add0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009ca0 tx=0x7fb2b002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.713+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/3170930273 shutdown_connections 2024-04-25T16:06:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.713+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/3170930273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2b8110030 0x7fb2b8110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.713+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/3170930273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2b810b310 0x7fb2b810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.713+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/3170930273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2b810a420 0x7fb2b810add0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.713+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/3170930273 >> 172.21.15.31:0/3170930273 conn(0x7fb2b80704e0 msgr2=0x7fb2b8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.715+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/3170930273 shutdown_connections 2024-04-25T16:06:20.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.717+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/3170930273 wait complete. 2024-04-25T16:06:20.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.719+0000 7fb2bd6e0640 1 Processor -- start 2024-04-25T16:06:20.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.719+0000 7fb2bd6e0640 1 -- start start 2024-04-25T16:06:20.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2bd6e0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2b810a420 0x7fb2b81170f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2bd6e0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2b810b310 0x7fb2b8117630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2bd6e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2b8110030 0x7fb2b8118540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2bd6e0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b811e540 con 0x7fb2b810a420 2024-04-25T16:06:20.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2bd6e0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b811e3c0 con 0x7fb2b810b310 2024-04-25T16:06:20.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2bd6e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b811e6c0 con 0x7fb2b8110030 2024-04-25T16:06:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2b6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2b810a420 0x7fb2b81170f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2b6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2b810a420 0x7fb2b81170f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58708/0 (socket says 172.21.15.31:58708) 2024-04-25T16:06:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2b6ffd640 1 -- 172.21.15.31:0/2392477682 learned_addr learned my addr 172.21.15.31:0/2392477682 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.720+0000 7fb2b67fc640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2b810b310 0x7fb2b8117630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.721+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2b8110030 0x7fb2b8118540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.721+0000 7fb2b6ffd640 1 -- 172.21.15.31:0/2392477682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2b8110030 msgr2=0x7fb2b8118540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.721+0000 7fb2b6ffd640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2b8110030 0x7fb2b8118540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.721+0000 7fb2b6ffd640 1 -- 172.21.15.31:0/2392477682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2b810b310 msgr2=0x7fb2b8117630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.721+0000 7fb2b6ffd640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2b810b310 0x7fb2b8117630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.721+0000 7fb2b6ffd640 1 -- 172.21.15.31:0/2392477682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b8118bb0 con 0x7fb2b810a420 2024-04-25T16:06:20.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.721+0000 7fb2b6ffd640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2b810a420 0x7fb2b81170f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0031af0 tx=0x7fb2b0003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.757+0000 7fb29ffff640 1 -- 172.21.15.31:0/2392477682 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b002f3b0 con 0x7fb2b810a420 2024-04-25T16:06:20.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.758+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/2392477682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b8118e40 con 0x7fb2b810a420 2024-04-25T16:06:20.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.758+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/2392477682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b8119410 con 0x7fb2b810a420 2024-04-25T16:06:20.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.758+0000 7fb29ffff640 1 -- 172.21.15.31:0/2392477682 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2b002d070 con 0x7fb2b810a420 2024-04-25T16:06:20.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.758+0000 7fb29ffff640 1 -- 172.21.15.31:0/2392477682 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b002fa20 con 0x7fb2b810a420 2024-04-25T16:06:20.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.763+0000 7fb2c4590740 1 Processor -- start 2024-04-25T16:06:20.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.763+0000 7fb2c4590740 1 -- start start 2024-04-25T16:06:20.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.764+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.764+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 0x55c6b82c5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.766+0000 7fb2b77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.766+0000 7fb2b77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52720/0 (socket says 172.21.15.31:52720) 2024-04-25T16:06:20.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.766+0000 7fb2b77fe640 1 -- 172.21.15.31:0/546965190 learned_addr learned my addr 172.21.15.31:0/546965190 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.767+0000 7fb2c4590740 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82c6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.767+0000 7fb2c4590740 1 -- 172.21.15.31:0/546965190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c6b8229d80 con 0x55c6b7efa4d0 2024-04-25T16:06:20.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.767+0000 7fb2c4590740 1 -- 172.21.15.31:0/546965190 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c6b81e3a20 con 0x55c6b7f026b0 2024-04-25T16:06:20.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.767+0000 7fb29ffff640 1 -- 172.21.15.31:0/2392477682 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb2b002fca0 con 0x7fb2b810a420 2024-04-25T16:06:20.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.767+0000 7fb2c4590740 1 -- 172.21.15.31:0/546965190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c6b8225d40 con 0x55c6b7f23390 2024-04-25T16:06:20.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.768+0000 7fb2b6ffd640 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 0x55c6b82c5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.768+0000 7fb2b67fc640 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82c6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.769+0000 7fb2b77fe640 1 -- 172.21.15.31:0/546965190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 msgr2=0x55c6b82c6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.769+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82c6a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.769+0000 7fb2b77fe640 1 -- 172.21.15.31:0/546965190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 msgr2=0x55c6b82c5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.769+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 0x55c6b82c5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.769+0000 7fb2b77fe640 1 -- 172.21.15.31:0/546965190 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6b81eec30 con 0x55c6b7f23390 2024-04-25T16:06:20.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.770+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c5810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0099c0 tx=0x7fb2ac009e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f6467a0740f84ef server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.771+0000 7fb29ffff640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2a00afd00 0x7fb2a00b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.772+0000 7fb29ffff640 1 -- 172.21.15.31:0/2392477682 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb2b0100160 con 0x7fb2b810a420 2024-04-25T16:06:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.773+0000 7fb29d7fa640 1 -- 172.21.15.31:0/546965190 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac017070 con 0x55c6b7f23390 2024-04-25T16:06:20.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.773+0000 7fb29d7fa640 1 -- 172.21.15.31:0/546965190 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2ac013470 con 0x55c6b7f23390 2024-04-25T16:06:20.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.775+0000 7fb2c4590740 1 -- 172.21.15.31:0/546965190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 msgr2=0x55c6b82c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.775+0000 7fb2c4590740 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c5810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0099c0 tx=0x7fb2ac009e80 comp rx=0 tx=0).stop 2024-04-25T16:06:20.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.778+0000 7fb2b6ffd640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2a00afd00 0x7fb2a00b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.782+0000 7fb2c4590740 1 -- 172.21.15.31:0/546965190 shutdown_connections 2024-04-25T16:06:20.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.782+0000 7fb2c4590740 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82c6a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.782+0000 7fb2c4590740 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 0x55c6b82c5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.782+0000 7fb2c4590740 1 --2- 172.21.15.31:0/546965190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c5810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.782+0000 7fb2c4590740 1 -- 172.21.15.31:0/546965190 >> 172.21.15.31:0/546965190 conn(0x55c6b7f174e0 msgr2=0x55c6b82c1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.783+0000 7fb2c4590740 1 -- 172.21.15.31:0/546965190 shutdown_connections 2024-04-25T16:06:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.783+0000 7fb2c4590740 1 -- 172.21.15.31:0/546965190 wait complete. 2024-04-25T16:06:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.784+0000 7fb2c4590740 1 -- start start 2024-04-25T16:06:20.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.784+0000 7fb2c4590740 1 Processor -- start 2024-04-25T16:06:20.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.785+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7efa4d0 0x55c6b82cf3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.785+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82cf900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.785+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f23390 0x55c6b82d6860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.785+0000 7fb2c4590740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c6b82c7580 con 0x55c6b7f23390 2024-04-25T16:06:20.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.785+0000 7fb2c4590740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c6b81b0dc0 con 0x55c6b7f026b0 2024-04-25T16:06:20.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.785+0000 7fb2c4590740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c6b82c7700 con 0x55c6b7efa4d0 2024-04-25T16:06:20.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.785+0000 7fb2c4590740 10 client.0 _ll_register_callbacks cb 0x55c6b81cc600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:20.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.786+0000 7fb2b77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7efa4d0 0x55c6b82cf3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.786+0000 7fb2b77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7efa4d0 0x55c6b82cf3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52732/0 (socket says 172.21.15.31:52732) 2024-04-25T16:06:20.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.786+0000 7fb2b77fe640 1 -- 172.21.15.31:0/2331706902 learned_addr learned my addr 172.21.15.31:0/2331706902 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.786+0000 7fb2b67fc640 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82cf900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.787+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f23390 0x55c6b82d6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.787+0000 7fb2b77fe640 1 -- 172.21.15.31:0/2331706902 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 msgr2=0x55c6b82cf900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.787+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82cf900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.787+0000 7fb2b77fe640 1 -- 172.21.15.31:0/2331706902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f23390 msgr2=0x55c6b82d6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.787+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f23390 0x55c6b82d6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.787+0000 7fb2b77fe640 1 -- 172.21.15.31:0/2331706902 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6b7f3b3f0 con 0x55c6b7efa4d0 2024-04-25T16:06:20.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.788+0000 7fb2b6ffd640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2a00afd00 0x7fb2a00b2210 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb2b00c57d0 tx=0x7fb2b0000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.789+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7efa4d0 0x55c6b82cf3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009fd0 tx=0x7fb2ac01e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7711fea8a0b965a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.789+0000 7fb29aff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.789+0000 7fb29aff5640 1 -- 172.21.15.31:0/2331706902 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac017070 con 0x55c6b7efa4d0 2024-04-25T16:06:20.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.790+0000 7fb2c4590740 1 -- 172.21.15.31:0/2331706902 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2ac013470 con 0x55c6b7efa4d0 2024-04-25T16:06:20.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.790+0000 7fb29aff5640 1 -- 172.21.15.31:0/2331706902 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb2ac012a80 con 0x55c6b7efa4d0 2024-04-25T16:06:20.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.792+0000 7fb29aff5640 1 -- 172.21.15.31:0/2331706902 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb2ac012c80 con 0x55c6b7efa4d0 2024-04-25T16:06:20.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.792+0000 7fb2c4590740 10 client.26260 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:20.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.792+0000 7fb2c4590740 1 -- 172.21.15.31:0/2331706902 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c6b80057d0 con 0x55c6b7efa4d0 2024-04-25T16:06:20.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.793+0000 7fb29aff5640 1 -- 172.21.15.31:0/2331706902 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2ac013470 con 0x55c6b7efa4d0 2024-04-25T16:06:20.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.793+0000 7fb29aff5640 1 -- 172.21.15.31:0/2331706902 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb2ac003ac0 con 0x55c6b7efa4d0 2024-04-25T16:06:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.795+0000 7fb2c4590740 10 client.26260 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.795+0000 7fb2c4590740 10 client.26260 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:06:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.795+0000 7fb2c4590740 20 client.26260 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.795+0000 7fb2c4590740 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c6b7ee1f60 0x55c6b82dd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.795+0000 7fb2c4590740 4 client.26260 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:20.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.795+0000 7fb2c4590740 1 -- 172.21.15.31:0/2331706902 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6b8237990 con 0x55c6b7ee1f60 2024-04-25T16:06:20.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.796+0000 7fb2b67fc640 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c6b7ee1f60 0x55c6b82dd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.797+0000 7fb2b67fc640 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c6b7ee1f60 0x55c6b82dd2a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75e1ac580e6e9a99 server_cookie=fef305aa8e29fc15 in_seq=0 out_seq=0 2024-04-25T16:06:20.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.797+0000 7fb29aff5640 10 client.26260 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:06:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.798+0000 7fb29aff5640 1 -- 172.21.15.31:0/2331706902 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c6b8237990 con 0x55c6b7ee1f60 2024-04-25T16:06:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.799+0000 7fb29aff5640 10 client.26260 handle_command_reply: tid=0 2024-04-25T16:06:20.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.799+0000 7fb2c4590740 1 client.26260 shutdown 2024-04-25T16:06:20.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.809+0000 7fb2c4590740 1 -- 172.21.15.31:0/2331706902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7efa4d0 msgr2=0x55c6b82cf3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.809+0000 7fb2c4590740 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7efa4d0 0x55c6b82cf3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009fd0 tx=0x7fb2ac01e040 comp rx=0 tx=0).stop 2024-04-25T16:06:20.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.810+0000 7fb2c4590740 1 -- 172.21.15.31:0/2331706902 shutdown_connections 2024-04-25T16:06:20.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.810+0000 7fb2c4590740 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c6b7ee1f60 0x55c6b82dd2a0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.810+0000 7fb2c4590740 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f23390 0x55c6b82d6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.810+0000 7fb2c4590740 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82cf900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.810+0000 7fb2c4590740 1 --2- 172.21.15.31:0/2331706902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7efa4d0 0x55c6b82cf3c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.810+0000 7fb2c4590740 1 -- 172.21.15.31:0/2331706902 >> 172.21.15.31:0/2331706902 conn(0x55c6b7f174e0 msgr2=0x55c6b82c0040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.810+0000 7fb29aff5640 0 client.26260 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:06:20.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.813+0000 7fb2c4590740 1 -- 172.21.15.31:0/2331706902 shutdown_connections 2024-04-25T16:06:20.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.813+0000 7fb2c4590740 1 -- 172.21.15.31:0/2331706902 wait complete. 2024-04-25T16:06:20.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.813+0000 7fb2c4590740 20 client.26260 trim_cache size 0 max 16384 2024-04-25T16:06:20.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.865+0000 7fb2c4590740 1 Processor -- start 2024-04-25T16:06:20.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.866+0000 7fb2c4590740 1 -- start start 2024-04-25T16:06:20.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.866+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 0x55c6b82c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.867+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82ca500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.867+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.867+0000 7fb2c4590740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a400e540 con 0x55c6b7efa4d0 2024-04-25T16:06:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.867+0000 7fb2c4590740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a4010080 con 0x55c6b7f026b0 2024-04-25T16:06:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.867+0000 7fb2c4590740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a400ec10 con 0x55c6b7f23390 2024-04-25T16:06:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.868+0000 7fb2b67fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.868+0000 7fb2b67fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82ca500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34672/0 (socket says 172.21.15.31:34672) 2024-04-25T16:06:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.868+0000 7fb2b67fc640 1 -- 172.21.15.31:0/1371027060 learned_addr learned my addr 172.21.15.31:0/1371027060 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.868+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1371027060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.872+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1371027060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 0x55c6b82c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.873+0000 7fb2b77fe640 1 -- 172.21.15.31:0/1371027060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 msgr2=0x55c6b82ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.873+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1371027060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.873+0000 7fb2b77fe640 1 -- 172.21.15.31:0/1371027060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 msgr2=0x55c6b82c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.873+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1371027060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 0x55c6b82c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.873+0000 7fb2b77fe640 1 -- 172.21.15.31:0/1371027060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6b82c5360 con 0x55c6b7f23390 2024-04-25T16:06:20.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.876+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1371027060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c7b80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac003f10 tx=0x7fb2ac001f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2db534a0f0535f9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.881+0000 7fb29cff9640 1 -- 172.21.15.31:0/1371027060 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac017040 con 0x55c6b7f23390 2024-04-25T16:06:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.881+0000 7fb29cff9640 1 -- 172.21.15.31:0/1371027060 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2ac02aec0 con 0x55c6b7f23390 2024-04-25T16:06:20.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.881+0000 7fb29cff9640 1 -- 172.21.15.31:0/1371027060 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac0093f0 con 0x55c6b7f23390 2024-04-25T16:06:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.884+0000 7fb2c4590740 1 -- 172.21.15.31:0/1371027060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 msgr2=0x55c6b82c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.884+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1371027060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c7b80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac003f10 tx=0x7fb2ac001f30 comp rx=0 tx=0).stop 2024-04-25T16:06:20.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.884+0000 7fb2c4590740 1 -- 172.21.15.31:0/1371027060 shutdown_connections 2024-04-25T16:06:20.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.884+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1371027060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82c7b80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.884+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1371027060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7f026b0 0x55c6b82ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.884+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1371027060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7efa4d0 0x55c6b82c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.884+0000 7fb2c4590740 1 -- 172.21.15.31:0/1371027060 >> 172.21.15.31:0/1371027060 conn(0x55c6b7f174e0 msgr2=0x55c6b82c9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.895+0000 7fb2c4590740 1 -- 172.21.15.31:0/1371027060 shutdown_connections 2024-04-25T16:06:20.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.897+0000 7fb2c4590740 1 -- 172.21.15.31:0/1371027060 wait complete. 2024-04-25T16:06:20.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.898+0000 7fb2c4590740 1 -- start start 2024-04-25T16:06:20.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.898+0000 7fb2c4590740 1 Processor -- start 2024-04-25T16:06:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7efa4d0 0x55c6b82d20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f026b0 0x55c6b82d2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2c4590740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82d5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2c4590740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a400f9b0 con 0x55c6b7f026b0 2024-04-25T16:06:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2c4590740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a400f2e0 con 0x55c6b7efa4d0 2024-04-25T16:06:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2c4590740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb2a400c520 con 0x55c6b7f23390 2024-04-25T16:06:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2c4590740 10 client.0 _ll_register_callbacks cb 0x7fb2ac009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2b67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f026b0 0x55c6b82d2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2b67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f026b0 0x55c6b82d2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58738/0 (socket says 172.21.15.31:58738) 2024-04-25T16:06:20.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.901+0000 7fb2b67fc640 1 -- 172.21.15.31:0/1307734342 learned_addr learned my addr 172.21.15.31:0/1307734342 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.903+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7efa4d0 0x55c6b82d20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.903+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82d5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.906+0000 7fb2b77fe640 1 -- 172.21.15.31:0/1307734342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 msgr2=0x55c6b82d5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.906+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82d5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.906+0000 7fb2b77fe640 1 -- 172.21.15.31:0/1307734342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f026b0 msgr2=0x55c6b82d2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:20.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.906+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f026b0 0x55c6b82d2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.906+0000 7fb2b77fe640 1 -- 172.21.15.31:0/1307734342 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6b82d5b00 con 0x55c6b7efa4d0 2024-04-25T16:06:20.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.906+0000 7fb2b67fc640 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f026b0 0x55c6b82d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:06:20.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.909+0000 7fb2b77fe640 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7efa4d0 0x55c6b82d20c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0135d0 tx=0x7fb2ac012e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17ad2737f31f5820 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:20.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.911+0000 7fb29aff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:06:20.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.911+0000 7fb29aff5640 1 -- 172.21.15.31:0/1307734342 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac012a80 con 0x55c6b7efa4d0 2024-04-25T16:06:20.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.911+0000 7fb29aff5640 1 -- 172.21.15.31:0/1307734342 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb2ac039040 con 0x55c6b7efa4d0 2024-04-25T16:06:20.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.913+0000 7fb2c4590740 1 -- 172.21.15.31:0/1307734342 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2ac02aec0 con 0x55c6b7efa4d0 2024-04-25T16:06:20.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.918+0000 7fb29aff5640 1 -- 172.21.15.31:0/1307734342 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb2ac047050 con 0x55c6b7efa4d0 2024-04-25T16:06:20.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.918+0000 7fb2c4590740 10 client.26315 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:20.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.918+0000 7fb2c4590740 1 -- 172.21.15.31:0/1307734342 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c6b8343f60 con 0x55c6b7efa4d0 2024-04-25T16:06:20.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.922+0000 7fb29aff5640 1 -- 172.21.15.31:0/1307734342 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb2ac02aec0 con 0x55c6b7efa4d0 2024-04-25T16:06:20.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.922+0000 7fb29aff5640 1 -- 172.21.15.31:0/1307734342 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb2ac02aa40 con 0x55c6b7efa4d0 2024-04-25T16:06:20.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.922+0000 7fb2c4590740 10 client.26315 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:20.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.922+0000 7fb2c4590740 10 client.26315 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:06:20.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.922+0000 7fb2c4590740 20 client.26315 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:20.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.922+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c6b7eb73a0 0x55c6b83469c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:20.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.922+0000 7fb2c4590740 4 client.26315 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:06:20.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.922+0000 7fb2c4590740 1 -- 172.21.15.31:0/1307734342 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c6b8231800 con 0x55c6b7eb73a0 2024-04-25T16:06:20.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.922+0000 7fb2b67fc640 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c6b7eb73a0 0x55c6b83469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:20.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.924+0000 7fb29aff5640 10 client.26315 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:06:20.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.925+0000 7fb2b67fc640 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c6b7eb73a0 0x55c6b83469c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8bd04503fd875a1d server_cookie=f116b0aaccfa5bdd in_seq=0 out_seq=0 2024-04-25T16:06:20.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.926+0000 7fb29aff5640 1 -- 172.21.15.31:0/1307734342 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c6b8231800 con 0x55c6b7eb73a0 2024-04-25T16:06:20.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.926+0000 7fb29aff5640 10 client.26315 handle_command_reply: tid=0 2024-04-25T16:06:20.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.926+0000 7fb2c4590740 1 client.26315 shutdown 2024-04-25T16:06:20.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.927+0000 7fb2c4590740 1 -- 172.21.15.31:0/1307734342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7efa4d0 msgr2=0x55c6b82d20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.927+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7efa4d0 0x55c6b82d20c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0135d0 tx=0x7fb2ac012e20 comp rx=0 tx=0).stop 2024-04-25T16:06:20.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.933+0000 7fb2c4590740 1 -- 172.21.15.31:0/1307734342 shutdown_connections 2024-04-25T16:06:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.933+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c6b7f23390 0x55c6b82d5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.933+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c6b7eb73a0 0x55c6b83469c0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.933+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c6b7f026b0 0x55c6b82d2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.933+0000 7fb2c4590740 1 --2- 172.21.15.31:0/1307734342 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c6b7efa4d0 0x55c6b82d20c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.933+0000 7fb2c4590740 1 -- 172.21.15.31:0/1307734342 >> 172.21.15.31:0/1307734342 conn(0x55c6b7f174e0 msgr2=0x55c6b82c3e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.933+0000 7fb29aff5640 0 client.26315 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:06:20.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.940+0000 7fb2c4590740 1 -- 172.21.15.31:0/1307734342 shutdown_connections 2024-04-25T16:06:20.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.940+0000 7fb2c4590740 1 -- 172.21.15.31:0/1307734342 wait complete. 2024-04-25T16:06:20.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.944+0000 7fb2c4590740 20 client.26315 trim_cache size 0 max 16384 2024-04-25T16:06:20.953 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:06:20.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.962+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2a00afd00 msgr2=0x7fb2a00b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.962+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2a00afd00 0x7fb2a00b2210 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fb2b00c57d0 tx=0x7fb2b0000f40 comp rx=0 tx=0).stop 2024-04-25T16:06:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.962+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2b810a420 msgr2=0x7fb2b81170f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:20.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.962+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2b810a420 0x7fb2b81170f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0031af0 tx=0x7fb2b0003ba0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.966+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/2392477682 shutdown_connections 2024-04-25T16:06:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.966+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb2a00afd00 0x7fb2a00b2210 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.966+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2b8110030 0x7fb2b8118540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.966+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb2b810b310 0x7fb2b8117630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.966+0000 7fb2bd6e0640 1 --2- 172.21.15.31:0/2392477682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb2b810a420 0x7fb2b81170f0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:20.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.966+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/2392477682 >> 172.21.15.31:0/2392477682 conn(0x7fb2b80704e0 msgr2=0x7fb2b810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:20.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.968+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/2392477682 shutdown_connections 2024-04-25T16:06:20.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:20.968+0000 7fb2bd6e0640 1 -- 172.21.15.31:0/2392477682 wait complete. 2024-04-25T16:06:21.009 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:06:21.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.376+0000 7faab9763640 1 -- 172.21.15.31:0/402223855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab410b190 msgr2=0x7faab410b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:21.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.376+0000 7faab9763640 1 --2- 172.21.15.31:0/402223855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab410b190 0x7faab410b5a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7faaa4009860 tx=0x7faaa402a380 comp rx=0 tx=0).stop 2024-04-25T16:06:21.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.378+0000 7faab9763640 1 -- 172.21.15.31:0/402223855 shutdown_connections 2024-04-25T16:06:21.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.378+0000 7faab9763640 1 --2- 172.21.15.31:0/402223855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab410fe90 0x7faab41102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.378+0000 7faab9763640 1 --2- 172.21.15.31:0/402223855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab410b190 0x7faab410b5a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.378+0000 7faab9763640 1 --2- 172.21.15.31:0/402223855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab410a340 0x7faab410ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.378+0000 7faab9763640 1 -- 172.21.15.31:0/402223855 >> 172.21.15.31:0/402223855 conn(0x7faab4070320 msgr2=0x7faab40727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:21.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.379+0000 7faab9763640 1 -- 172.21.15.31:0/402223855 shutdown_connections 2024-04-25T16:06:21.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.382+0000 7faab9763640 1 -- 172.21.15.31:0/402223855 wait complete. 2024-04-25T16:06:21.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.383+0000 7faab9763640 1 Processor -- start 2024-04-25T16:06:21.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.389+0000 7faab9763640 1 -- start start 2024-04-25T16:06:21.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.389+0000 7faab9763640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab410a340 0x7faab41170f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.389+0000 7faab9763640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab410b190 0x7faab4117630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.389+0000 7faab9763640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab410fe90 0x7faab4118590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.389+0000 7faab9763640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faab411e280 con 0x7faab410fe90 2024-04-25T16:06:21.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.389+0000 7faab9763640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faab411e100 con 0x7faab410a340 2024-04-25T16:06:21.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.389+0000 7faab9763640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faab411e400 con 0x7faab410b190 2024-04-25T16:06:21.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.392+0000 7faab37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab410b190 0x7faab4117630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.392+0000 7faab8f62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab410fe90 0x7faab4118590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.392+0000 7faab8f62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab410fe90 0x7faab4118590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58756/0 (socket says 172.21.15.31:58756) 2024-04-25T16:06:21.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.393+0000 7faab8f62640 1 -- 172.21.15.31:0/4285840721 learned_addr learned my addr 172.21.15.31:0/4285840721 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.396+0000 7faab8f62640 1 -- 172.21.15.31:0/4285840721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab410b190 msgr2=0x7faab4117630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.396+0000 7faab8f62640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab410b190 0x7faab4117630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.396+0000 7faab8f62640 1 -- 172.21.15.31:0/4285840721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab410a340 msgr2=0x7faab41170f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:06:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.396+0000 7faab8f62640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab410a340 0x7faab41170f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.396+0000 7faab8f62640 1 -- 172.21.15.31:0/4285840721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab4118c00 con 0x7faab410fe90 2024-04-25T16:06:21.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.398+0000 7faab8f62640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab410fe90 0x7faab4118590 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7faaa800e970 tx=0x7faaa800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:21.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.403+0000 7faab17fa640 1 -- 172.21.15.31:0/4285840721 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa800ccc0 con 0x7faab410fe90 2024-04-25T16:06:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.403+0000 7faab9763640 1 -- 172.21.15.31:0/4285840721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab4118ef0 con 0x7faab410fe90 2024-04-25T16:06:21.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.403+0000 7faab9763640 1 -- 172.21.15.31:0/4285840721 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab4119440 con 0x7faab410fe90 2024-04-25T16:06:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.412+0000 7faac1877740 1 Processor -- start 2024-04-25T16:06:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.412+0000 7faab17fa640 1 -- 172.21.15.31:0/4285840721 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faaa800ce60 con 0x7faab410fe90 2024-04-25T16:06:21.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.412+0000 7faab17fa640 1 -- 172.21.15.31:0/4285840721 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa8010430 con 0x7faab410fe90 2024-04-25T16:06:21.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.418+0000 7faab17fa640 1 -- 172.21.15.31:0/4285840721 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7faaa8010650 con 0x7faab410fe90 2024-04-25T16:06:21.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.425+0000 7faab17fa640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa9c0afcf0 0x7faa9c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.426+0000 7faab17fa640 1 -- 172.21.15.31:0/4285840721 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faaa80def00 con 0x7faab410fe90 2024-04-25T16:06:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.426+0000 7faac1877740 1 -- start start 2024-04-25T16:06:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.427+0000 7faac1877740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afa2e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.427+0000 7faac1877740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af6634d0 0x5621afa2ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.427+0000 7faac1877740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af66b6b0 0x5621afa2fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.427+0000 7faac1877740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5621af992d80 con 0x5621af68c390 2024-04-25T16:06:21.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.427+0000 7faac1877740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5621af94ca20 con 0x5621af66b6b0 2024-04-25T16:06:21.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.427+0000 7faac1877740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5621af98ed40 con 0x5621af6634d0 2024-04-25T16:06:21.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.429+0000 7faab8f62640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af66b6b0 0x5621afa2fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.434+0000 7faab3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afa2e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.434+0000 7faab3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afa2e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58766/0 (socket says 172.21.15.31:58766) 2024-04-25T16:06:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.434+0000 7faab3fff640 1 -- 172.21.15.31:0/1297835878 learned_addr learned my addr 172.21.15.31:0/1297835878 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.434+0000 7faab37fe640 1 --2- 172.21.15.31:0/1297835878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af6634d0 0x5621afa2ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.434+0000 7faab37fe640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa9c0afcf0 0x7faa9c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.437+0000 7faab37fe640 1 -- 172.21.15.31:0/1297835878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af66b6b0 msgr2=0x5621afa2fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.437+0000 7faab37fe640 1 --2- 172.21.15.31:0/1297835878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af66b6b0 0x5621afa2fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.437+0000 7faab37fe640 1 -- 172.21.15.31:0/1297835878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 msgr2=0x5621afa2e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.437+0000 7faab37fe640 1 --2- 172.21.15.31:0/1297835878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afa2e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.437+0000 7faab37fe640 1 -- 172.21.15.31:0/1297835878 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621af957c30 con 0x5621af6634d0 2024-04-25T16:06:21.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.442+0000 7faab37fe640 1 --2- 172.21.15.31:0/1297835878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af6634d0 0x5621afa2ef40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faaa4030380 tx=0x7faaa4004290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67002dd0dd80a1aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:21.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.444+0000 7faab9763640 1 -- 172.21.15.31:0/1297835878 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4037030 con 0x5621af6634d0 2024-04-25T16:06:21.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.444+0000 7faab37fe640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa9c0afcf0 0x7faa9c0b2200 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7faaa4047000 tx=0x7faaa4045000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:21.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.445+0000 7faab9763640 1 -- 172.21.15.31:0/1297835878 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faaa4045070 con 0x5621af6634d0 2024-04-25T16:06:21.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.446+0000 7faac1877740 1 -- 172.21.15.31:0/1297835878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af6634d0 msgr2=0x5621afa2ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:21.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.446+0000 7faac1877740 1 --2- 172.21.15.31:0/1297835878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af6634d0 0x5621afa2ef40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7faaa4030380 tx=0x7faaa4004290 comp rx=0 tx=0).stop 2024-04-25T16:06:21.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.448+0000 7faac1877740 1 -- 172.21.15.31:0/1297835878 shutdown_connections 2024-04-25T16:06:21.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.448+0000 7faac1877740 1 --2- 172.21.15.31:0/1297835878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af66b6b0 0x5621afa2fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.448+0000 7faac1877740 1 --2- 172.21.15.31:0/1297835878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af6634d0 0x5621afa2ef40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.448+0000 7faac1877740 1 --2- 172.21.15.31:0/1297835878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afa2e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.448+0000 7faac1877740 1 -- 172.21.15.31:0/1297835878 >> 172.21.15.31:0/1297835878 conn(0x5621af6804e0 msgr2=0x5621afa2a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:21.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.455+0000 7faac1877740 1 -- 172.21.15.31:0/1297835878 shutdown_connections 2024-04-25T16:06:21.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.457+0000 7faac1877740 1 -- 172.21.15.31:0/1297835878 wait complete. 2024-04-25T16:06:21.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.457+0000 7faac1877740 1 -- start start 2024-04-25T16:06:21.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.458+0000 7faac1877740 1 Processor -- start 2024-04-25T16:06:21.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.459+0000 7faac1877740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa386f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.459+0000 7faac1877740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa38c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.459+0000 7faac1877740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afa3fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.459+0000 7faac1877740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5621afa30720 con 0x5621af68c390 2024-04-25T16:06:21.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.459+0000 7faac1877740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5621af919dc0 con 0x5621af6634d0 2024-04-25T16:06:21.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.459+0000 7faac1877740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5621afa308a0 con 0x5621af66b6b0 2024-04-25T16:06:21.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.459+0000 7faac1877740 10 client.0 _ll_register_callbacks cb 0x5621af935600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:21.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.462+0000 7faab37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa38c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.462+0000 7faab3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.465+0000 7faab3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa386f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34712/0 (socket says 172.21.15.31:34712) 2024-04-25T16:06:21.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.465+0000 7faab3fff640 1 -- 172.21.15.31:0/18865036 learned_addr learned my addr 172.21.15.31:0/18865036 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:21.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.469+0000 7faab8f62640 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afa3fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.471+0000 7faab37fe640 1 -- 172.21.15.31:0/18865036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 msgr2=0x5621afa386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.471+0000 7faab37fe640 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa386f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.471+0000 7faab37fe640 1 -- 172.21.15.31:0/18865036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 msgr2=0x5621afa3fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.471+0000 7faab37fe640 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afa3fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.471+0000 7faab37fe640 1 -- 172.21.15.31:0/18865036 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621af6a43f0 con 0x5621af66b6b0 2024-04-25T16:06:21.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.474+0000 7faab37fe640 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa38c30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faaa402a970 tx=0x7faaa4032d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99754f7be397d6cc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:21.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.475+0000 7faa937fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:21.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.484+0000 7faa937fe640 1 -- 172.21.15.31:0/18865036 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa4056070 con 0x5621af66b6b0 2024-04-25T16:06:21.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.484+0000 7faac1877740 1 -- 172.21.15.31:0/18865036 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faaa4045070 con 0x5621af66b6b0 2024-04-25T16:06:21.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.484+0000 7faa937fe640 1 -- 172.21.15.31:0/18865036 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faaa4030a00 con 0x5621af66b6b0 2024-04-25T16:06:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.487+0000 7faa937fe640 1 -- 172.21.15.31:0/18865036 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faaa4060050 con 0x5621af66b6b0 2024-04-25T16:06:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.487+0000 7faac1877740 10 client.26278 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:21.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.487+0000 7faac1877740 1 -- 172.21.15.31:0/18865036 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5621af76e7d0 con 0x5621af66b6b0 2024-04-25T16:06:21.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.492+0000 7faa937fe640 1 -- 172.21.15.31:0/18865036 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faaa4045070 con 0x5621af66b6b0 2024-04-25T16:06:21.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.494+0000 7faa937fe640 1 -- 172.21.15.31:0/18865036 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7faaa4048070 con 0x5621af66b6b0 2024-04-25T16:06:21.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.494+0000 7faac1877740 10 client.26278 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:21.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.494+0000 7faac1877740 10 client.26278 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:21.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.494+0000 7faac1877740 20 client.26278 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:21.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.494+0000 7faac1877740 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621af64af60 0x5621afa46600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.494+0000 7faac1877740 4 client.26278 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:21.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.494+0000 7faac1877740 1 -- 172.21.15.31:0/18865036 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621af9a0990 con 0x5621af64af60 2024-04-25T16:06:21.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.505+0000 7faab3fff640 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621af64af60 0x5621afa46600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.507+0000 7faab3fff640 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621af64af60 0x5621afa46600 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=148dbc9f787e7b57 server_cookie=cea312ab2db44ae0 in_seq=0 out_seq=0 2024-04-25T16:06:21.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.508+0000 7faa937fe640 10 client.26278 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:21.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.513+0000 7faa937fe640 1 -- 172.21.15.31:0/18865036 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5621af9a0990 con 0x5621af64af60 2024-04-25T16:06:21.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.514+0000 7faa937fe640 10 client.26278 handle_command_reply: tid=0 2024-04-25T16:06:21.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.514+0000 7faac1877740 1 client.26278 shutdown 2024-04-25T16:06:21.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.514+0000 7faac1877740 1 -- 172.21.15.31:0/18865036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 msgr2=0x5621afa38c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:21.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.514+0000 7faac1877740 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa38c30 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faaa402a970 tx=0x7faaa4032d20 comp rx=0 tx=0).stop 2024-04-25T16:06:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.519+0000 7faac1877740 1 -- 172.21.15.31:0/18865036 shutdown_connections 2024-04-25T16:06:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.519+0000 7faac1877740 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621af64af60 0x5621afa46600 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.519+0000 7faac1877740 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afa3fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.519+0000 7faac1877740 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa38c30 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.519+0000 7faac1877740 1 --2- 172.21.15.31:0/18865036 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa386f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.519+0000 7faac1877740 1 -- 172.21.15.31:0/18865036 >> 172.21.15.31:0/18865036 conn(0x5621af6804e0 msgr2=0x5621afa2d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:21.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.519+0000 7faa937fe640 0 client.26278 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:21.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.525+0000 7faac1877740 1 -- 172.21.15.31:0/18865036 shutdown_connections 2024-04-25T16:06:21.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.527+0000 7faac1877740 1 -- 172.21.15.31:0/18865036 wait complete. 2024-04-25T16:06:21.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.527+0000 7faac1877740 20 client.26278 trim_cache size 0 max 16384 2024-04-25T16:06:21.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.566+0000 7faac1877740 1 Processor -- start 2024-04-25T16:06:21.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.567+0000 7faac1877740 1 -- start start 2024-04-25T16:06:21.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.568+0000 7faac1877740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa37f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.569+0000 7faac1877740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af66b6b0 0x5621afa2d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.569+0000 7faac1877740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af68c390 0x5621afa4adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.569+0000 7faac1877740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000e3b0 con 0x5621af66b6b0 2024-04-25T16:06:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.569+0000 7faac1877740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000fef0 con 0x5621af6634d0 2024-04-25T16:06:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.570+0000 7faac1877740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000ea80 con 0x5621af68c390 2024-04-25T16:06:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.570+0000 7faab8f62640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af68c390 0x5621afa4adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.570+0000 7faab8f62640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af68c390 0x5621afa4adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52800/0 (socket says 172.21.15.31:52800) 2024-04-25T16:06:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.570+0000 7faab8f62640 1 -- 172.21.15.31:0/2300747346 learned_addr learned my addr 172.21.15.31:0/2300747346 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.571+0000 7faab8f62640 1 -- 172.21.15.31:0/2300747346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 msgr2=0x5621afa37f70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.571+0000 7faab8f62640 1 --2- 172.21.15.31:0/2300747346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa37f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.571+0000 7faab8f62640 1 -- 172.21.15.31:0/2300747346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af66b6b0 msgr2=0x5621afa2d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.571+0000 7faab8f62640 1 --2- 172.21.15.31:0/2300747346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af66b6b0 0x5621afa2d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.571+0000 7faab8f62640 1 -- 172.21.15.31:0/2300747346 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621afa33d70 con 0x5621af68c390 2024-04-25T16:06:21.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.572+0000 7faab8f62640 1 --2- 172.21.15.31:0/2300747346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af68c390 0x5621afa4adf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faaa800c6a0 tx=0x7faaa8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9374ac7e5400fadd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:21.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.574+0000 7faa99ffb640 1 -- 172.21.15.31:0/2300747346 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa80c0070 con 0x5621af68c390 2024-04-25T16:06:21.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.574+0000 7faa99ffb640 1 -- 172.21.15.31:0/2300747346 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faaa80ab740 con 0x5621af68c390 2024-04-25T16:06:21.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.574+0000 7faa99ffb640 1 -- 172.21.15.31:0/2300747346 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa80afd60 con 0x5621af68c390 2024-04-25T16:06:21.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.576+0000 7faac1877740 1 -- 172.21.15.31:0/2300747346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af68c390 msgr2=0x5621afa4adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:21.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.576+0000 7faac1877740 1 --2- 172.21.15.31:0/2300747346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af68c390 0x5621afa4adf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faaa800c6a0 tx=0x7faaa8014000 comp rx=0 tx=0).stop 2024-04-25T16:06:21.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.577+0000 7faac1877740 1 -- 172.21.15.31:0/2300747346 shutdown_connections 2024-04-25T16:06:21.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.577+0000 7faac1877740 1 --2- 172.21.15.31:0/2300747346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af68c390 0x5621afa4adf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.577+0000 7faac1877740 1 --2- 172.21.15.31:0/2300747346 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af66b6b0 0x5621afa2d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.577+0000 7faac1877740 1 --2- 172.21.15.31:0/2300747346 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa37f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.577+0000 7faac1877740 1 -- 172.21.15.31:0/2300747346 >> 172.21.15.31:0/2300747346 conn(0x5621af6804e0 msgr2=0x5621afa2d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:21.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.598+0000 7faac1877740 1 -- 172.21.15.31:0/2300747346 shutdown_connections 2024-04-25T16:06:21.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.599+0000 7faac1877740 1 -- 172.21.15.31:0/2300747346 wait complete. 2024-04-25T16:06:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.599+0000 7faac1877740 1 -- start start 2024-04-25T16:06:21.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.600+0000 7faac1877740 1 Processor -- start 2024-04-25T16:06:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.602+0000 7faac1877740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa3e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.602+0000 7faac1877740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa3e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.602+0000 7faac1877740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afaac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.602+0000 7faac1877740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000f820 con 0x5621af68c390 2024-04-25T16:06:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.602+0000 7faac1877740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000f150 con 0x5621af6634d0 2024-04-25T16:06:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.602+0000 7faac1877740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faaa000c440 con 0x5621af66b6b0 2024-04-25T16:06:21.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.602+0000 7faac1877740 10 client.0 _ll_register_callbacks cb 0x7faaa8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:21.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.603+0000 7faab8f62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afaac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.603+0000 7faab8f62640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afaac9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58790/0 (socket says 172.21.15.31:58790) 2024-04-25T16:06:21.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.603+0000 7faab8f62640 1 -- 172.21.15.31:0/1328981032 learned_addr learned my addr 172.21.15.31:0/1328981032 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:21.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.606+0000 7faab37fe640 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa3e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.606+0000 7faab8f62640 1 -- 172.21.15.31:0/1328981032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 msgr2=0x5621afa3e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.606+0000 7faab8f62640 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa3e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.606+0000 7faab8f62640 1 -- 172.21.15.31:0/1328981032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 msgr2=0x5621afa3e180 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.606+0000 7faab8f62640 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa3e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.606+0000 7faab8f62640 1 -- 172.21.15.31:0/1328981032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621afaacee0 con 0x5621af68c390 2024-04-25T16:06:21.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.606+0000 7faab37fe640 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa3e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:06:21.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.612+0000 7faab8f62640 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afaac9a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7faaa80a4600 tx=0x7faaa80aff90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b162e4c861ab7e9a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:21.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.612+0000 7faa937fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:21.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.616+0000 7faa937fe640 1 -- 172.21.15.31:0/1328981032 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaa80c0030 con 0x5621af68c390 2024-04-25T16:06:21.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.616+0000 7faa937fe640 1 -- 172.21.15.31:0/1328981032 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faaa80a1d50 con 0x5621af68c390 2024-04-25T16:06:21.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.616+0000 7faac1877740 1 -- 172.21.15.31:0/1328981032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faaa80ab740 con 0x5621af68c390 2024-04-25T16:06:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.618+0000 7faa937fe640 1 -- 172.21.15.31:0/1328981032 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faaa80b01f0 con 0x5621af68c390 2024-04-25T16:06:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.618+0000 7faac1877740 10 client.16674 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:21.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.618+0000 7faac1877740 1 -- 172.21.15.31:0/1328981032 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5621afaae060 con 0x5621af68c390 2024-04-25T16:06:21.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.620+0000 7faa937fe640 1 -- 172.21.15.31:0/1328981032 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faaa80ab740 con 0x5621af68c390 2024-04-25T16:06:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.624+0000 7faa937fe640 1 -- 172.21.15.31:0/1328981032 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7faaa80a65f0 con 0x5621af68c390 2024-04-25T16:06:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.624+0000 7faac1877740 10 client.16674 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.624+0000 7faac1877740 10 client.16674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.624+0000 7faac1877740 20 client.16674 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.624+0000 7faac1877740 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621af6203a0 0x5621afab09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:21.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.624+0000 7faac1877740 4 client.16674 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:06:21.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.624+0000 7faac1877740 1 -- 172.21.15.31:0/1328981032 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621af99a800 con 0x5621af6203a0 2024-04-25T16:06:21.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.629+0000 7faab3fff640 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621af6203a0 0x5621afab09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:21.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.659+0000 7faab3fff640 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621af6203a0 0x5621afab09b0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccc82f76a22abe4c server_cookie=2f231fa4995a4f0 in_seq=0 out_seq=0 2024-04-25T16:06:21.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.659+0000 7faa937fe640 10 client.16674 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:21.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.667+0000 7faa937fe640 1 -- 172.21.15.31:0/1328981032 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621af99a800 con 0x5621af6203a0 2024-04-25T16:06:21.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.667+0000 7faa937fe640 10 client.16674 handle_command_reply: tid=0 2024-04-25T16:06:21.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.668+0000 7faac1877740 1 client.16674 shutdown 2024-04-25T16:06:21.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.673+0000 7faac1877740 1 -- 172.21.15.31:0/1328981032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 msgr2=0x5621afaac9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:21.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.673+0000 7faac1877740 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afaac9a0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7faaa80a4600 tx=0x7faaa80aff90 comp rx=0 tx=0).stop 2024-04-25T16:06:21.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.676+0000 7faac1877740 1 -- 172.21.15.31:0/1328981032 shutdown_connections 2024-04-25T16:06:21.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.676+0000 7faac1877740 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621af6203a0 0x5621afab09b0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.676+0000 7faac1877740 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621af68c390 0x5621afaac9a0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.676+0000 7faac1877740 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621af66b6b0 0x5621afa3e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.676+0000 7faac1877740 1 --2- 172.21.15.31:0/1328981032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621af6634d0 0x5621afa3e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.676+0000 7faac1877740 1 -- 172.21.15.31:0/1328981032 >> 172.21.15.31:0/1328981032 conn(0x5621af6804e0 msgr2=0x5621afa36f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:21.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.676+0000 7faa937fe640 0 client.16674 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:21.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.680+0000 7faac1877740 1 -- 172.21.15.31:0/1328981032 shutdown_connections 2024-04-25T16:06:21.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.684+0000 7faac1877740 1 -- 172.21.15.31:0/1328981032 wait complete. 2024-04-25T16:06:21.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.684+0000 7faac1877740 20 client.16674 trim_cache size 0 max 16384 2024-04-25T16:06:21.686 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:06:21.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.687+0000 7faa9affd640 1 -- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa9c0afcf0 msgr2=0x7faa9c0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:21.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.687+0000 7faa9affd640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa9c0afcf0 0x7faa9c0b2200 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7faaa4047000 tx=0x7faaa4045000 comp rx=0 tx=0).stop 2024-04-25T16:06:21.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.687+0000 7faa9affd640 1 -- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab410fe90 msgr2=0x7faab4118590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:21.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.687+0000 7faa9affd640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab410fe90 0x7faab4118590 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7faaa800e970 tx=0x7faaa800ed70 comp rx=0 tx=0).stop 2024-04-25T16:06:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:21 smithi031 ceph-mon[29140]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 788 MiB/s wr, 2.15k op/s 2024-04-25T16:06:21.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.704+0000 7faa9affd640 1 -- 172.21.15.31:0/4285840721 shutdown_connections 2024-04-25T16:06:21.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.704+0000 7faa9affd640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa9c0afcf0 0x7faa9c0b2200 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.704+0000 7faa9affd640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faab410fe90 0x7faab4118590 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.704+0000 7faa9affd640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faab410b190 0x7faab4117630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.704+0000 7faa9affd640 1 --2- 172.21.15.31:0/4285840721 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faab410a340 0x7faab41170f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:21.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.704+0000 7faa9affd640 1 -- 172.21.15.31:0/4285840721 >> 172.21.15.31:0/4285840721 conn(0x7faab4070320 msgr2=0x7faab410be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:21.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.721+0000 7faa9affd640 1 -- 172.21.15.31:0/4285840721 shutdown_connections 2024-04-25T16:06:21.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:21.723+0000 7faa9affd640 1 -- 172.21.15.31:0/4285840721 wait complete. 2024-04-25T16:06:21.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:21 smithi063 ceph-mon[32413]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 788 MiB/s wr, 2.15k op/s 2024-04-25T16:06:21.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:21 smithi186 ceph-mon[33258]: pgmap v502: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 788 MiB/s wr, 2.15k op/s 2024-04-25T16:06:22.810 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:06:22.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:06:22.810 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:06:23.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.322+0000 7f31af984640 1 -- 172.21.15.31:0/2632383787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31a81120c0 msgr2=0x7f31a8114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:23.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.322+0000 7f31af984640 1 --2- 172.21.15.31:0/2632383787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31a81120c0 0x7f31a8114520 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009ca0 tx=0x7f31a002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.332+0000 7f31af984640 1 -- 172.21.15.31:0/2632383787 shutdown_connections 2024-04-25T16:06:23.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.332+0000 7f31af984640 1 --2- 172.21.15.31:0/2632383787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31a81120c0 0x7f31a8114520 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.332+0000 7f31af984640 1 --2- 172.21.15.31:0/2632383787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a810f760 0x7f31a8111b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.332+0000 7f31af984640 1 --2- 172.21.15.31:0/2632383787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a8109fc0 0x7f31a810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.332+0000 7f31af984640 1 -- 172.21.15.31:0/2632383787 >> 172.21.15.31:0/2632383787 conn(0x7f31a80704e0 msgr2=0x7f31a8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:23.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.337+0000 7f31af984640 1 -- 172.21.15.31:0/2632383787 shutdown_connections 2024-04-25T16:06:23.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.342+0000 7f31af984640 1 -- 172.21.15.31:0/2632383787 wait complete. 2024-04-25T16:06:23.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.344+0000 7f31af984640 1 Processor -- start 2024-04-25T16:06:23.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.344+0000 7f31af984640 1 -- start start 2024-04-25T16:06:23.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.344+0000 7f31af984640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a8109fc0 0x7f31a81a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.344+0000 7f31af984640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31a810f760 0x7f31a81a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.344+0000 7f31af984640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a81120c0 0x7f31a81b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.344+0000 7f31af984640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f31a8117f90 con 0x7f31a810f760 2024-04-25T16:06:23.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.344+0000 7f31af984640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f31a8117e10 con 0x7f31a8109fc0 2024-04-25T16:06:23.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.344+0000 7f31af984640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f31a8118110 con 0x7f31a81120c0 2024-04-25T16:06:23.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.345+0000 7f31adf21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a81120c0 0x7f31a81b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.345+0000 7f31adf21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a81120c0 0x7f31a81b06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:52816/0 (socket says 172.21.15.31:52816) 2024-04-25T16:06:23.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.345+0000 7f31adf21640 1 -- 172.21.15.31:0/2747068451 learned_addr learned my addr 172.21.15.31:0/2747068451 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:23.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.346+0000 7f31ad720640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a8109fc0 0x7f31a81a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.347+0000 7f31ad720640 1 -- 172.21.15.31:0/2747068451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a81120c0 msgr2=0x7f31a81b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:23.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.347+0000 7f31ad720640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a81120c0 0x7f31a81b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.347+0000 7f31ad720640 1 -- 172.21.15.31:0/2747068451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31a810f760 msgr2=0x7f31a81a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:23.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.347+0000 7f31ad720640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31a810f760 0x7f31a81a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.347+0000 7f31ad720640 1 -- 172.21.15.31:0/2747068451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31a81a1bb0 con 0x7f31a8109fc0 2024-04-25T16:06:23.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.348+0000 7f31ad720640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a8109fc0 0x7f31a81a0a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009fd0 tx=0x7f31a400b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:23.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.351+0000 7f319e7fc640 1 -- 172.21.15.31:0/2747068451 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4015020 con 0x7f31a8109fc0 2024-04-25T16:06:23.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.351+0000 7f319e7fc640 1 -- 172.21.15.31:0/2747068451 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a4011ce0 con 0x7f31a8109fc0 2024-04-25T16:06:23.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.351+0000 7f31af984640 1 -- 172.21.15.31:0/2747068451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31a81a1ea0 con 0x7f31a8109fc0 2024-04-25T16:06:23.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.351+0000 7f31af984640 1 -- 172.21.15.31:0/2747068451 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31a81a2470 con 0x7f31a8109fc0 2024-04-25T16:06:23.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.352+0000 7f319e7fc640 1 -- 172.21.15.31:0/2747068451 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4010d40 con 0x7f31a8109fc0 2024-04-25T16:06:23.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.355+0000 7f31b6834740 1 Processor -- start 2024-04-25T16:06:23.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.356+0000 7f31b6834740 1 -- start start 2024-04-25T16:06:23.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.356+0000 7f31b6834740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edf2410 0x56442f1934a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.356+0000 7f31b6834740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edc9550 0x56442f193a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.356+0000 7f31b6834740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f19a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.357+0000 7f31acf1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f19a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.357+0000 7f31acf1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f19a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58808/0 (socket says 172.21.15.31:58808) 2024-04-25T16:06:23.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.357+0000 7f31adf21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edf2410 0x56442f1934a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.357+0000 7f31b6834740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56442f0f8b00 con 0x56442edd1730 2024-04-25T16:06:23.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.357+0000 7f31b6834740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56442f094920 con 0x56442edf2410 2024-04-25T16:06:23.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.357+0000 7f31b6834740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56442f0f4ac0 con 0x56442edc9550 2024-04-25T16:06:23.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.357+0000 7f31adf21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edf2410 0x56442f1934a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34766/0 (socket says 172.21.15.31:34766) 2024-04-25T16:06:23.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.357+0000 7f31adf21640 1 -- 172.21.15.31:0/3470878680 learned_addr learned my addr 172.21.15.31:0/3470878680 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:23.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.358+0000 7f31ad720640 1 --2- 172.21.15.31:0/3470878680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edc9550 0x56442f193a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.358+0000 7f31adf21640 1 -- 172.21.15.31:0/3470878680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edc9550 msgr2=0x56442f193a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:23.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.358+0000 7f31adf21640 1 --2- 172.21.15.31:0/3470878680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edc9550 0x56442f193a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.358+0000 7f31adf21640 1 -- 172.21.15.31:0/3470878680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 msgr2=0x56442f19a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:23.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.358+0000 7f31adf21640 1 --2- 172.21.15.31:0/3470878680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f19a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.358+0000 7f31adf21640 1 -- 172.21.15.31:0/3470878680 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56442eed27e0 con 0x56442edf2410 2024-04-25T16:06:23.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.358+0000 7f31acf1f640 1 --2- 172.21.15.31:0/3470878680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f19a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:06:23.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.359+0000 7f31adf21640 1 --2- 172.21.15.31:0/3470878680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edf2410 0x56442f1934a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009a10 tx=0x7f31a0031480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87eedbd3058adaf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:23.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.359+0000 7f31937fe640 1 -- 172.21.15.31:0/3470878680 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a0031820 con 0x56442edf2410 2024-04-25T16:06:23.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.362+0000 7f31937fe640 1 -- 172.21.15.31:0/3470878680 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a0036030 con 0x56442edf2410 2024-04-25T16:06:23.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.362+0000 7f319e7fc640 1 -- 172.21.15.31:0/2747068451 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f31a4018020 con 0x7f31a8109fc0 2024-04-25T16:06:23.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.363+0000 7f31b6834740 1 -- 172.21.15.31:0/3470878680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edf2410 msgr2=0x56442f1934a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:23.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.363+0000 7f31b6834740 1 --2- 172.21.15.31:0/3470878680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edf2410 0x56442f1934a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009a10 tx=0x7f31a0031480 comp rx=0 tx=0).stop 2024-04-25T16:06:23.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.364+0000 7f31b6834740 1 -- 172.21.15.31:0/3470878680 shutdown_connections 2024-04-25T16:06:23.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.364+0000 7f31b6834740 1 --2- 172.21.15.31:0/3470878680 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f19a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.364+0000 7f31b6834740 1 --2- 172.21.15.31:0/3470878680 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edc9550 0x56442f193a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.364+0000 7f31b6834740 1 --2- 172.21.15.31:0/3470878680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edf2410 0x56442f1934a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.364+0000 7f31b6834740 1 -- 172.21.15.31:0/3470878680 >> 172.21.15.31:0/3470878680 conn(0x56442ede6560 msgr2=0x56442f18ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:23.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.366+0000 7f319e7fc640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31940afd00 0x7f31940b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.366+0000 7f319e7fc640 1 -- 172.21.15.31:0/2747068451 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f31a40de3f0 con 0x7f31a8109fc0 2024-04-25T16:06:23.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.367+0000 7f31b6834740 1 -- 172.21.15.31:0/3470878680 shutdown_connections 2024-04-25T16:06:23.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.367+0000 7f31b6834740 1 -- 172.21.15.31:0/3470878680 wait complete. 2024-04-25T16:06:23.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.367+0000 7f31b6834740 1 -- start start 2024-04-25T16:06:23.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.368+0000 7f31b6834740 1 Processor -- start 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.369+0000 7f31acf1f640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31940afd00 0x7f31940b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.369+0000 7f31b6834740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 0x56442f19d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.369+0000 7f31b6834740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 0x56442f19d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.369+0000 7f31b6834740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.369+0000 7f31b6834740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56442f19b360 con 0x56442edc9550 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.369+0000 7f31b6834740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56442f0925d0 con 0x56442edb11f0 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.369+0000 7f31b6834740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56442f19b4e0 con 0x56442edd1730 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.369+0000 7f31b6834740 10 client.0 _ll_register_callbacks cb 0x56442f080080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.370+0000 7f31acf1f640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31940afd00 0x7f31940b2210 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x56442f18fbc0 tx=0x7f31980090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:23.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.371+0000 7f31acf1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 0x56442f19d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.372+0000 7f31ad720640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 0x56442f19d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.372+0000 7f31ad720640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 0x56442f19d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:58818/0 (socket says 172.21.15.31:58818) 2024-04-25T16:06:23.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.372+0000 7f31ad720640 1 -- 172.21.15.31:0/3747827995 learned_addr learned my addr 172.21.15.31:0/3747827995 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:23.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.375+0000 7f31adf21640 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.377+0000 7f31adf21640 1 -- 172.21.15.31:0/3747827995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 msgr2=0x56442f19d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:23.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.377+0000 7f31adf21640 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 0x56442f19d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.377+0000 7f31adf21640 1 -- 172.21.15.31:0/3747827995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 msgr2=0x56442f19d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:23.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.377+0000 7f31adf21640 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 0x56442f19d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.377+0000 7f31adf21640 1 -- 172.21.15.31:0/3747827995 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56442ef32f70 con 0x56442edb11f0 2024-04-25T16:06:23.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.377+0000 7f31acf1f640 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 0x56442f19d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:06:23.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.377+0000 7f31adf21640 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a4620 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f31a00314f0 tx=0x7f31a002fcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=949af02fbdb461da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:23.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.378+0000 7f3190ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:06:23.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.379+0000 7f3190ff9640 1 -- 172.21.15.31:0/3747827995 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a0003d80 con 0x56442edb11f0 2024-04-25T16:06:23.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.379+0000 7f3190ff9640 1 -- 172.21.15.31:0/3747827995 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f31a0004030 con 0x56442edb11f0 2024-04-25T16:06:23.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.381+0000 7f31b6834740 1 -- 172.21.15.31:0/3747827995 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56442f106710 con 0x56442edb11f0 2024-04-25T16:06:23.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.385+0000 7f3190ff9640 1 -- 172.21.15.31:0/3747827995 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f31a0004b40 con 0x56442edb11f0 2024-04-25T16:06:23.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.385+0000 7f31b6834740 10 client.26330 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:23.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.385+0000 7f31b6834740 1 -- 172.21.15.31:0/3747827995 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56442edca170 con 0x56442edb11f0 2024-04-25T16:06:23.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.385+0000 7f3190ff9640 1 -- 172.21.15.31:0/3747827995 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56442f106710 con 0x56442edb11f0 2024-04-25T16:06:23.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.386+0000 7f3190ff9640 1 -- 172.21.15.31:0/3747827995 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f31a0043070 con 0x56442edb11f0 2024-04-25T16:06:23.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.386+0000 7f31b6834740 10 client.26330 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:23.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.386+0000 7f31b6834740 10 client.26330 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:23.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.386+0000 7f31b6834740 20 client.26330 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:23.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.386+0000 7f31b6834740 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56442ed86630 0x56442f1aaa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.386+0000 7f31b6834740 4 client.26330 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:23.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.386+0000 7f31b6834740 1 -- 172.21.15.31:0/3747827995 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56442f100580 con 0x56442ed86630 2024-04-25T16:06:23.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.388+0000 7f31ad720640 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56442ed86630 0x56442f1aaa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.392+0000 7f31ad720640 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56442ed86630 0x56442f1aaa90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c47ed3d333118e09 server_cookie=652c0b0d486488c7 in_seq=0 out_seq=0 2024-04-25T16:06:23.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.392+0000 7f3190ff9640 10 client.26330 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.399+0000 7f3190ff9640 1 -- 172.21.15.31:0/3747827995 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56442f100580 con 0x56442ed86630 2024-04-25T16:06:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.399+0000 7f3190ff9640 10 client.26330 handle_command_reply: tid=0 2024-04-25T16:06:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.399+0000 7f31b6834740 1 client.26330 shutdown 2024-04-25T16:06:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.399+0000 7f31b6834740 1 -- 172.21.15.31:0/3747827995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 msgr2=0x56442f1a4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:23.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.399+0000 7f31b6834740 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a4620 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f31a00314f0 tx=0x7f31a002fcf0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.400+0000 7f31b6834740 1 -- 172.21.15.31:0/3747827995 shutdown_connections 2024-04-25T16:06:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.400+0000 7f31b6834740 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56442ed86630 0x56442f1aaa90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.400+0000 7f31b6834740 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a4620 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.400+0000 7f31b6834740 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 0x56442f19d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.400+0000 7f31b6834740 1 --2- 172.21.15.31:0/3747827995 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 0x56442f19d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.400+0000 7f31b6834740 1 -- 172.21.15.31:0/3747827995 >> 172.21.15.31:0/3747827995 conn(0x56442ede6560 msgr2=0x56442f191db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.401+0000 7f3190ff9640 0 client.26330 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.401+0000 7f31b6834740 1 -- 172.21.15.31:0/3747827995 shutdown_connections 2024-04-25T16:06:23.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.401+0000 7f31b6834740 1 -- 172.21.15.31:0/3747827995 wait complete. 2024-04-25T16:06:23.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.401+0000 7f31b6834740 20 client.26330 trim_cache size 0 max 16384 2024-04-25T16:06:23.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.506+0000 7f31b6834740 1 Processor -- start 2024-04-25T16:06:23.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.506+0000 7f31b6834740 1 -- start start 2024-04-25T16:06:23.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.507+0000 7f31b6834740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f19d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.507+0000 7f31b6834740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 0x56442f19d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.507+0000 7f31b6834740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 0x56442f19dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.507+0000 7f31b6834740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f318c00e520 con 0x56442edc9550 2024-04-25T16:06:23.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.507+0000 7f31b6834740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f318c010060 con 0x56442edb11f0 2024-04-25T16:06:23.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.507+0000 7f31b6834740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f318c00ebf0 con 0x56442edd1730 2024-04-25T16:06:23.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.508+0000 7f31ad720640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f19d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.513+0000 7f31ad720640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f19d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34794/0 (socket says 172.21.15.31:34794) 2024-04-25T16:06:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.513+0000 7f31ad720640 1 -- 172.21.15.31:0/916383702 learned_addr learned my addr 172.21.15.31:0/916383702 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:23.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.515+0000 7f31adf21640 1 --2- 172.21.15.31:0/916383702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 0x56442f19dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.516+0000 7f31acf1f640 1 --2- 172.21.15.31:0/916383702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 0x56442f19d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.519+0000 7f31ad720640 1 -- 172.21.15.31:0/916383702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 msgr2=0x56442f19dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.519+0000 7f31ad720640 1 --2- 172.21.15.31:0/916383702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 0x56442f19dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.519+0000 7f31ad720640 1 -- 172.21.15.31:0/916383702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 msgr2=0x56442f19d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.519+0000 7f31ad720640 1 --2- 172.21.15.31:0/916383702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 0x56442f19d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.519+0000 7f31ad720640 1 -- 172.21.15.31:0/916383702 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56442f18f9c0 con 0x56442edb11f0 2024-04-25T16:06:23.521 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.520+0000 7f31ad720640 1 --2- 172.21.15.31:0/916383702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f19d180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f31a40a3af0 tx=0x7f31a401c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=591b4e42baf3d45 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.527+0000 7f3192ffd640 1 -- 172.21.15.31:0/916383702 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a40c2030 con 0x56442edb11f0 2024-04-25T16:06:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.527+0000 7f3192ffd640 1 -- 172.21.15.31:0/916383702 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a40af6e0 con 0x56442edb11f0 2024-04-25T16:06:23.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.527+0000 7f3192ffd640 1 -- 172.21.15.31:0/916383702 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a40c90b0 con 0x56442edb11f0 2024-04-25T16:06:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.529+0000 7f31b6834740 1 -- 172.21.15.31:0/916383702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 msgr2=0x56442f19d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.529+0000 7f31b6834740 1 --2- 172.21.15.31:0/916383702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f19d180 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f31a40a3af0 tx=0x7f31a401c040 comp rx=0 tx=0).stop 2024-04-25T16:06:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.529+0000 7f31b6834740 1 -- 172.21.15.31:0/916383702 shutdown_connections 2024-04-25T16:06:23.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.529+0000 7f31b6834740 1 --2- 172.21.15.31:0/916383702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edd1730 0x56442f19dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.529+0000 7f31b6834740 1 --2- 172.21.15.31:0/916383702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edc9550 0x56442f19d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.529+0000 7f31b6834740 1 --2- 172.21.15.31:0/916383702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f19d180 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.529+0000 7f31b6834740 1 -- 172.21.15.31:0/916383702 >> 172.21.15.31:0/916383702 conn(0x56442ede6560 msgr2=0x56442f197560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:23.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.536+0000 7f31b6834740 1 -- 172.21.15.31:0/916383702 shutdown_connections 2024-04-25T16:06:23.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.536+0000 7f31b6834740 1 -- 172.21.15.31:0/916383702 wait complete. 2024-04-25T16:06:23.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.537+0000 7f31b6834740 1 -- start start 2024-04-25T16:06:23.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.537+0000 7f31b6834740 1 Processor -- start 2024-04-25T16:06:23.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.538+0000 7f31b6834740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.539+0000 7f31b6834740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edc9550 0x56442f1a2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.539+0000 7f31b6834740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f1a2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.539+0000 7f31b6834740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f318c00f990 con 0x56442edd1730 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.539+0000 7f31ad720640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.539+0000 7f31b6834740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f318c00f2c0 con 0x56442edb11f0 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.539+0000 7f31b6834740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f318c00c500 con 0x56442edc9550 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.539+0000 7f31b6834740 10 client.0 _ll_register_callbacks cb 0x7f31a0009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.539+0000 7f31ad720640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34806/0 (socket says 172.21.15.31:34806) 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.539+0000 7f31ad720640 1 -- 172.21.15.31:0/3491724028 learned_addr learned my addr 172.21.15.31:0/3491724028 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:23.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.540+0000 7f31adf21640 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f1a2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.540+0000 7f31adf21640 1 -- 172.21.15.31:0/3491724028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edc9550 msgr2=0x56442f1a2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:23.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.540+0000 7f31adf21640 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edc9550 0x56442f1a2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.540+0000 7f31adf21640 1 -- 172.21.15.31:0/3491724028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 msgr2=0x56442f1a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:23.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.540+0000 7f31adf21640 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:23.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.540+0000 7f31adf21640 1 -- 172.21.15.31:0/3491724028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56442f1a2eb0 con 0x56442edd1730 2024-04-25T16:06:23.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.541+0000 7f31adf21640 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f1a2750 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f31a0002230 tx=0x7f31a0031b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed565d034c937319 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:23.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.542+0000 7f3190ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:23.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.542+0000 7f3190ff9640 1 -- 172.21.15.31:0/3491724028 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a00098f0 con 0x56442edd1730 2024-04-25T16:06:23.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.543+0000 7f31b6834740 1 -- 172.21.15.31:0/3491724028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31a40af6e0 con 0x56442edd1730 2024-04-25T16:06:23.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.543+0000 7f3190ff9640 1 -- 172.21.15.31:0/3491724028 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f31a0045640 con 0x56442edd1730 2024-04-25T16:06:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.547+0000 7f3190ff9640 1 -- 172.21.15.31:0/3491724028 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f31a0030e90 con 0x56442edd1730 2024-04-25T16:06:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.547+0000 7f31b6834740 10 client.16689 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:23.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.547+0000 7f31b6834740 1 -- 172.21.15.31:0/3491724028 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56442f211d80 con 0x56442edd1730 2024-04-25T16:06:23.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.552+0000 7f3190ff9640 1 -- 172.21.15.31:0/3491724028 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f31a40af6e0 con 0x56442edd1730 2024-04-25T16:06:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.552+0000 7f3190ff9640 1 -- 172.21.15.31:0/3491724028 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f31a0043040 con 0x56442edd1730 2024-04-25T16:06:23.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.554+0000 7f31b6834740 10 client.16689 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.554+0000 7f31b6834740 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.554+0000 7f31b6834740 20 client.16689 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.554+0000 7f31b6834740 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56442f212680 0x56442f214aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.555+0000 7f31b6834740 4 client.16689 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:06:23.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.555+0000 7f31b6834740 1 -- 172.21.15.31:0/3491724028 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56442f07bac0 con 0x56442f212680 2024-04-25T16:06:23.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.560+0000 7f31ad720640 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56442f212680 0x56442f214aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:23.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.569+0000 7f31ad720640 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56442f212680 0x56442f214aa0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a73136c283596a7 server_cookie=ce23e420f420387a in_seq=0 out_seq=0 2024-04-25T16:06:23.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:23.570+0000 7f3190ff9640 10 client.16689 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:23.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:23 smithi031 ceph-mon[29140]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 726 MiB/s wr, 2.05k op/s 2024-04-25T16:06:23.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:23 smithi063 ceph-mon[32413]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 726 MiB/s wr, 2.05k op/s 2024-04-25T16:06:23.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:23 smithi186 ceph-mon[33258]: pgmap v503: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 726 MiB/s wr, 2.05k op/s 2024-04-25T16:06:25.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f3190ff9640 1 -- 172.21.15.31:0/3491724028 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56442f07bac0 con 0x56442f212680 2024-04-25T16:06:25.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f3190ff9640 10 client.16689 handle_command_reply: tid=0 2024-04-25T16:06:25.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 client.16689 shutdown 2024-04-25T16:06:25.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 -- 172.21.15.31:0/3491724028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 msgr2=0x56442f1a2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:25.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f1a2750 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f31a0002230 tx=0x7f31a0031b10 comp rx=0 tx=0).stop 2024-04-25T16:06:25.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 -- 172.21.15.31:0/3491724028 shutdown_connections 2024-04-25T16:06:25.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56442f212680 0x56442f214aa0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56442edd1730 0x56442f1a2750 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56442edc9550 0x56442f1a2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f3190ff9640 0 client.16689 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 --2- 172.21.15.31:0/3491724028 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56442edb11f0 0x56442f1a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 -- 172.21.15.31:0/3491724028 >> 172.21.15.31:0/3491724028 conn(0x56442ede6560 msgr2=0x56442f191c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:25.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 -- 172.21.15.31:0/3491724028 shutdown_connections 2024-04-25T16:06:25.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.211+0000 7f31b6834740 1 -- 172.21.15.31:0/3491724028 wait complete. 2024-04-25T16:06:25.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31b6834740 20 client.16689 trim_cache size 0 max 16384 2024-04-25T16:06:25.217 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:06:25.218 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:06:25.218 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "c1daa02d-8486-46e3-bedb-e1d6943ac561", 2024-04-25T16:06:25.218 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:06:25.218 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:06:25.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31af984640 1 -- 172.21.15.31:0/2747068451 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31940afd00 msgr2=0x7f31940b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:25.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31af984640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31940afd00 0x7f31940b2210 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x56442f18fbc0 tx=0x7f31980090e0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31af984640 1 -- 172.21.15.31:0/2747068451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a8109fc0 msgr2=0x7f31a81a0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:25.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31af984640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a8109fc0 0x7f31a81a0a10 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f31a4009fd0 tx=0x7f31a400b870 comp rx=0 tx=0).stop 2024-04-25T16:06:25.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31af984640 1 -- 172.21.15.31:0/2747068451 shutdown_connections 2024-04-25T16:06:25.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31af984640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31940afd00 0x7f31940b2210 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31af984640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31a81120c0 0x7f31a81b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31af984640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f31a810f760 0x7f31a81a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.212+0000 7f31af984640 1 --2- 172.21.15.31:0/2747068451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f31a8109fc0 0x7f31a81a0a10 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.213+0000 7f31af984640 1 -- 172.21.15.31:0/2747068451 >> 172.21.15.31:0/2747068451 conn(0x7f31a80704e0 msgr2=0x7f31a810c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:25.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.213+0000 7f31af984640 1 -- 172.21.15.31:0/2747068451 shutdown_connections 2024-04-25T16:06:25.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.213+0000 7f31af984640 1 -- 172.21.15.31:0/2747068451 wait complete. 2024-04-25T16:06:25.271 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:06:25.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:06:25.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:25 smithi031 ceph-mon[29140]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 916 MiB/s wr, 2.80k op/s 2024-04-25T16:06:25.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:25 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:06:25.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:25 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:06:25.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:25 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:06:25.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.786+0000 7f1a78a49640 1 -- 172.21.15.31:0/2065420794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 msgr2=0x7f1a7410adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:25.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.786+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2065420794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 0x7f1a7410adf0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c009ca0 tx=0x7f1a6c02a8f0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.805+0000 7f1a78a49640 1 -- 172.21.15.31:0/2065420794 shutdown_connections 2024-04-25T16:06:25.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.805+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2065420794 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a74110030 0x7f1a74112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.805+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2065420794 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a7410b330 0x7f1a7410b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.805+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2065420794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 0x7f1a7410adf0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.805+0000 7f1a78a49640 1 -- 172.21.15.31:0/2065420794 >> 172.21.15.31:0/2065420794 conn(0x7f1a74078de0 msgr2=0x7f1a741079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:25.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.818+0000 7f1a78a49640 1 -- 172.21.15.31:0/2065420794 shutdown_connections 2024-04-25T16:06:25.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.824+0000 7f1a78a49640 1 -- 172.21.15.31:0/2065420794 wait complete. 2024-04-25T16:06:25.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.825+0000 7f1a78a49640 1 Processor -- start 2024-04-25T16:06:25.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.826+0000 7f1a78a49640 1 -- start start 2024-04-25T16:06:25.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.826+0000 7f1a78a49640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 0x7f1a741a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.826+0000 7f1a78a49640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a7410b330 0x7f1a741a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.826+0000 7f1a78a49640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a74110030 0x7f1a741a7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.826+0000 7f1a78a49640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a74115d60 con 0x7f1a74110030 2024-04-25T16:06:25.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.826+0000 7f1a78a49640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1a74115be0 con 0x7f1a7410b330 2024-04-25T16:06:25.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.826+0000 7f1a78a49640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1a74115ee0 con 0x7f1a7410a0f0 2024-04-25T16:06:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.827+0000 7f1a7259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 0x7f1a741a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.827+0000 7f1a7259c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 0x7f1a741a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:50898/0 (socket says 172.21.15.31:50898) 2024-04-25T16:06:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.827+0000 7f1a7259c640 1 -- 172.21.15.31:0/2592083222 learned_addr learned my addr 172.21.15.31:0/2592083222 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.828+0000 7f1a7259c640 1 -- 172.21.15.31:0/2592083222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a7410b330 msgr2=0x7f1a741a0e20 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:06:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.828+0000 7f1a7259c640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a7410b330 0x7f1a741a0e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.828+0000 7f1a7259c640 1 -- 172.21.15.31:0/2592083222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a74110030 msgr2=0x7f1a741a7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.828+0000 7f1a7259c640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a74110030 0x7f1a741a7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.828+0000 7f1a7259c640 1 -- 172.21.15.31:0/2592083222 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a741a8420 con 0x7f1a7410a0f0 2024-04-25T16:06:25.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.829+0000 7f1a7259c640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 0x7f1a741a08e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c031a90 tx=0x7f1a6c003f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:25.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.829+0000 7f1a637fe640 1 -- 172.21.15.31:0/2592083222 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0042c0 con 0x7f1a7410a0f0 2024-04-25T16:06:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.831+0000 7f1a78a49640 1 -- 172.21.15.31:0/2592083222 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a741a86b0 con 0x7f1a7410a0f0 2024-04-25T16:06:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.831+0000 7f1a78a49640 1 -- 172.21.15.31:0/2592083222 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a741a8cd0 con 0x7f1a7410a0f0 2024-04-25T16:06:25.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.833+0000 7f1a637fe640 1 -- 172.21.15.31:0/2592083222 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a6c02d070 con 0x7f1a7410a0f0 2024-04-25T16:06:25.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.833+0000 7f1a637fe640 1 -- 172.21.15.31:0/2592083222 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0305f0 con 0x7f1a7410a0f0 2024-04-25T16:06:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.837+0000 7f1a7f9d3740 1 Processor -- start 2024-04-25T16:06:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.838+0000 7f1a7f9d3740 1 -- start start 2024-04-25T16:06:25.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.844+0000 7f1a637fe640 1 -- 172.21.15.31:0/2592083222 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1a6c02f7f0 con 0x7f1a7410a0f0 2024-04-25T16:06:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.846+0000 7f1a637fe640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a5c0afba0 0x7f1a5c0b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.847+0000 7f1a637fe640 1 -- 172.21.15.31:0/2592083222 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1a6c0c5730 con 0x7f1a7410a0f0 2024-04-25T16:06:25.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.848+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fdf390 0x558c43380a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.848+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fb64d0 0x558c43380fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.848+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c43381c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.849+0000 7f1a7f9d3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558c432e5d80 con 0x558c42fbe6b0 2024-04-25T16:06:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.850+0000 7f1a72d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c43381c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.850+0000 7f1a72d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c43381c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53752/0 (socket says 172.21.15.31:53752) 2024-04-25T16:06:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.850+0000 7f1a72d9d640 1 -- 172.21.15.31:0/928882386 learned_addr learned my addr 172.21.15.31:0/928882386 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.851+0000 7f1a7259c640 1 --2- 172.21.15.31:0/928882386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fdf390 0x558c43380a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.851+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/928882386 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558c4329f8b0 con 0x558c42fdf390 2024-04-25T16:06:25.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.851+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/928882386 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558c432e1d40 con 0x558c42fb64d0 2024-04-25T16:06:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.851+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a5c0afba0 0x7f1a5c0b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.852+0000 7f1a71d9b640 1 --2- 172.21.15.31:0/928882386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fb64d0 0x558c43380fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.854+0000 7f1a7259c640 1 -- 172.21.15.31:0/928882386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fb64d0 msgr2=0x558c43380fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:25.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.854+0000 7f1a7259c640 1 --2- 172.21.15.31:0/928882386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fb64d0 0x558c43380fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.854+0000 7f1a7259c640 1 -- 172.21.15.31:0/928882386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 msgr2=0x558c43381c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:25.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.854+0000 7f1a7259c640 1 --2- 172.21.15.31:0/928882386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c43381c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.854+0000 7f1a7259c640 1 -- 172.21.15.31:0/928882386 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c43270a40 con 0x558c42fdf390 2024-04-25T16:06:25.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.856+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a5c0afba0 0x7f1a5c0b20b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800bfd0 tx=0x7f1a6800b730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:25.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.860+0000 7f1a7259c640 1 --2- 172.21.15.31:0/928882386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fdf390 0x558c43380a60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c000f80 tx=0x7f1a6c0094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=519c2dda829d6af8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:25.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.863+0000 7f1a60ff9640 1 -- 172.21.15.31:0/928882386 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0c98d0 con 0x558c42fdf390 2024-04-25T16:06:25.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.863+0000 7f1a60ff9640 1 -- 172.21.15.31:0/928882386 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a6c036070 con 0x558c42fdf390 2024-04-25T16:06:25.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.863+0000 7f1a60ff9640 1 -- 172.21.15.31:0/928882386 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6c0cd170 con 0x558c42fdf390 2024-04-25T16:06:25.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.865+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/928882386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fdf390 msgr2=0x558c43380a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:25.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.865+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/928882386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fdf390 0x558c43380a60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c000f80 tx=0x7f1a6c0094a0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.866+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/928882386 shutdown_connections 2024-04-25T16:06:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.866+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/928882386 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c43381c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.866+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/928882386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fb64d0 0x558c43380fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.866+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/928882386 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fdf390 0x558c43380a60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.866+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/928882386 >> 172.21.15.31:0/928882386 conn(0x558c42fd34e0 msgr2=0x558c4337c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.868+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/928882386 shutdown_connections 2024-04-25T16:06:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.868+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/928882386 wait complete. 2024-04-25T16:06:25.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:25 smithi063 ceph-mon[32413]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 916 MiB/s wr, 2.80k op/s 2024-04-25T16:06:25.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:25 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:06:25.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:25 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:06:25.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:25 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:06:25.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.869+0000 7f1a7f9d3740 1 -- start start 2024-04-25T16:06:25.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.869+0000 7f1a7f9d3740 1 Processor -- start 2024-04-25T16:06:25.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.870+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4338a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.870+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c4338aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.871+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.871+0000 7f1a7f9d3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558c43382540 con 0x558c42fbe6b0 2024-04-25T16:06:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.871+0000 7f1a7f9d3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558c4326cd10 con 0x558c42fb64d0 2024-04-25T16:06:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.871+0000 7f1a7259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4338a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.871+0000 7f1a7259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4338a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:38834/0 (socket says 172.21.15.31:38834) 2024-04-25T16:06:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.871+0000 7f1a7259c640 1 -- 172.21.15.31:0/162131724 learned_addr learned my addr 172.21.15.31:0/162131724 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.871+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/162131724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558c433826c0 con 0x558c42fdf390 2024-04-25T16:06:25.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.871+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.872+0000 7f1a7f9d3740 10 client.0 _ll_register_callbacks cb 0x558c43299210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.872+0000 7f1a72d9d640 1 -- 172.21.15.31:0/162131724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 msgr2=0x558c4338a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.872+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4338a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.872+0000 7f1a72d9d640 1 -- 172.21.15.31:0/162131724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 msgr2=0x558c4338aba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:06:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.873+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c4338aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.873+0000 7f1a72d9d640 1 -- 172.21.15.31:0/162131724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c430c17d0 con 0x558c42fdf390 2024-04-25T16:06:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.875+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1a68012bd0 tx=0x7f1a6800b530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21edfb83967fa49d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.875+0000 7f1a59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.875+0000 7f1a59ffb640 1 -- 172.21.15.31:0/162131724 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a68019960 con 0x558c42fdf390 2024-04-25T16:06:25.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.875+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/162131724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1a6c036070 con 0x558c42fdf390 2024-04-25T16:06:25.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.876+0000 7f1a59ffb640 1 -- 172.21.15.31:0/162131724 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1a68019b00 con 0x558c42fdf390 2024-04-25T16:06:25.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.876+0000 7f1a59ffb640 1 -- 172.21.15.31:0/162131724 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1a68019d00 con 0x558c42fdf390 2024-04-25T16:06:25.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.883+0000 7f1a7f9d3740 10 client.26302 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:25.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.883+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/162131724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558c42ff73f0 con 0x558c42fdf390 2024-04-25T16:06:25.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.885+0000 7f1a59ffb640 1 -- 172.21.15.31:0/162131724 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a6c036070 con 0x558c42fdf390 2024-04-25T16:06:25.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.885+0000 7f1a59ffb640 1 -- 172.21.15.31:0/162131724 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f1a6801a450 con 0x558c42fdf390 2024-04-25T16:06:25.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.885+0000 7f1a7f9d3740 10 client.26302 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:25.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.885+0000 7f1a7f9d3740 10 client.26302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:25.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.885+0000 7f1a7f9d3740 20 client.26302 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.885+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558c42f9df60 0x558c43398530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.885+0000 7f1a7f9d3740 4 client.26302 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:25.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.885+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/162131724 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558c432f3990 con 0x558c42f9df60 2024-04-25T16:06:25.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.887+0000 7f1a7259c640 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558c42f9df60 0x558c43398530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.892+0000 7f1a59ffb640 10 client.26302 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:25.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.893+0000 7f1a7259c640 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558c42f9df60 0x558c43398530 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=841b2217924c3458 server_cookie=8ca2a4fce17383ef in_seq=0 out_seq=0 2024-04-25T16:06:25.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:25 smithi186 ceph-mon[33258]: pgmap v504: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 916 MiB/s wr, 2.80k op/s 2024-04-25T16:06:25.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:25 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:06:25.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:25 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:06:25.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:25 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:06:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.906+0000 7f1a59ffb640 1 -- 172.21.15.31:0/162131724 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558c432f3990 con 0x558c42f9df60 2024-04-25T16:06:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.906+0000 7f1a59ffb640 10 client.26302 handle_command_reply: tid=0 2024-04-25T16:06:25.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.906+0000 7f1a7f9d3740 1 client.26302 shutdown 2024-04-25T16:06:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.909+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/162131724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 msgr2=0x558c43391b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:25.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.909+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391b00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1a68012bd0 tx=0x7f1a6800b530 comp rx=0 tx=0).stop 2024-04-25T16:06:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.912+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/162131724 shutdown_connections 2024-04-25T16:06:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.912+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558c42f9df60 0x558c43398530 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.912+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391b00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.912+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c4338aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.912+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/162131724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4338a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.912+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/162131724 >> 172.21.15.31:0/162131724 conn(0x558c42fd34e0 msgr2=0x558c4337b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.912+0000 7f1a59ffb640 0 client.26302 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.912+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/162131724 shutdown_connections 2024-04-25T16:06:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.913+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/162131724 wait complete. 2024-04-25T16:06:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.913+0000 7f1a7f9d3740 20 client.26302 trim_cache size 0 max 16384 2024-04-25T16:06:25.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.982+0000 7f1a7f9d3740 1 Processor -- start 2024-04-25T16:06:25.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.982+0000 7f1a7f9d3740 1 -- start start 2024-04-25T16:06:25.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.983+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4339cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.984+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c43391a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.984+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.984+0000 7f1a7259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4339cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.984+0000 7f1a7259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4339cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:38850/0 (socket says 172.21.15.31:38850) 2024-04-25T16:06:25.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.984+0000 7f1a7259c640 1 -- 172.21.15.31:0/1611919068 learned_addr learned my addr 172.21.15.31:0/1611919068 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:25.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.985+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1611919068 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5400e550 con 0x558c42fbe6b0 2024-04-25T16:06:25.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.985+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/1611919068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.985+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1611919068 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1a54010090 con 0x558c42fb64d0 2024-04-25T16:06:25.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.985+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1611919068 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5400ec20 con 0x558c42fdf390 2024-04-25T16:06:25.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.986+0000 7f1a71d9b640 1 --2- 172.21.15.31:0/1611919068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c43391a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.986+0000 7f1a72d9d640 1 -- 172.21.15.31:0/1611919068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 msgr2=0x558c4339cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.986+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/1611919068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4339cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.986+0000 7f1a72d9d640 1 -- 172.21.15.31:0/1611919068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 msgr2=0x558c43391a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.986+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/1611919068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c43391a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.986+0000 7f1a72d9d640 1 -- 172.21.15.31:0/1611919068 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558c42fb70f0 con 0x558c42fdf390 2024-04-25T16:06:25.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.987+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/1611919068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391f60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1a6801a8a0 tx=0x7f1a68019ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14a434ae7e6cdcc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:25.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.988+0000 7f1a5bfff640 1 -- 172.21.15.31:0/1611919068 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6801ae60 con 0x558c42fdf390 2024-04-25T16:06:25.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.988+0000 7f1a5bfff640 1 -- 172.21.15.31:0/1611919068 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1a68034070 con 0x558c42fdf390 2024-04-25T16:06:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.989+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1611919068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 msgr2=0x558c43391f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.989+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1611919068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391f60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1a6801a8a0 tx=0x7f1a68019ea0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.989+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1611919068 shutdown_connections 2024-04-25T16:06:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.989+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1611919068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42fdf390 0x558c43391f60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.989+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1611919068 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fbe6b0 0x558c43391a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.989+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1611919068 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fb64d0 0x558c4339cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.989+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1611919068 >> 172.21.15.31:0/1611919068 conn(0x558c42fd34e0 msgr2=0x558c43385540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.989+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1611919068 shutdown_connections 2024-04-25T16:06:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.989+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1611919068 wait complete. 2024-04-25T16:06:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.990+0000 7f1a7f9d3740 1 -- start start 2024-04-25T16:06:25.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.990+0000 7f1a7f9d3740 1 Processor -- start 2024-04-25T16:06:25.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.990+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fb64d0 0x558c43391f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.991+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fbe6b0 0x558c4338ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.991+0000 7f1a7f9d3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42f733a0 0x558c4338f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:25.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.991+0000 7f1a7f9d3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5400f9c0 con 0x558c42fb64d0 2024-04-25T16:06:25.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.991+0000 7f1a7f9d3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5400f2f0 con 0x558c42fbe6b0 2024-04-25T16:06:25.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.991+0000 7f1a7f9d3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5400c530 con 0x558c42f733a0 2024-04-25T16:06:25.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.991+0000 7f1a7f9d3740 10 client.0 _ll_register_callbacks cb 0x7f1a68009190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:25.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.992+0000 7f1a7259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fb64d0 0x558c43391f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.992+0000 7f1a7259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fb64d0 0x558c43391f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53780/0 (socket says 172.21.15.31:53780) 2024-04-25T16:06:25.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.992+0000 7f1a7259c640 1 -- 172.21.15.31:0/1715517285 learned_addr learned my addr 172.21.15.31:0/1715517285 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:25.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.992+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42f733a0 0x558c4338f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.992+0000 7f1a71d9b640 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fbe6b0 0x558c4338ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:25.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.993+0000 7f1a72d9d640 1 -- 172.21.15.31:0/1715517285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fbe6b0 msgr2=0x558c4338ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:25.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.993+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fbe6b0 0x558c4338ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.993+0000 7f1a72d9d640 1 -- 172.21.15.31:0/1715517285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fb64d0 msgr2=0x558c43391f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:25.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.993+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fb64d0 0x558c43391f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:25.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.993+0000 7f1a72d9d640 1 -- 172.21.15.31:0/1715517285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558c432a64b0 con 0x558c42f733a0 2024-04-25T16:06:25.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:25.996+0000 7f1a72d9d640 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42f733a0 0x558c4338f010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1a6801a010 tx=0x7f1a68006930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b954af64e83e2d7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:26.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.001+0000 7f1a59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:26.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.001+0000 7f1a59ffb640 1 -- 172.21.15.31:0/1715517285 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6803d020 con 0x558c42f733a0 2024-04-25T16:06:26.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.001+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1715517285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558c432ed800 con 0x558c42f733a0 2024-04-25T16:06:26.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.001+0000 7f1a59ffb640 1 -- 172.21.15.31:0/1715517285 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1a68004220 con 0x558c42f733a0 2024-04-25T16:06:26.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.002+0000 7f1a59ffb640 1 -- 172.21.15.31:0/1715517285 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1a6800b900 con 0x558c42f733a0 2024-04-25T16:06:26.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.002+0000 7f1a7f9d3740 10 client.26308 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:26.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.002+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1715517285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558c433ff750 con 0x558c42f733a0 2024-04-25T16:06:26.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.004+0000 7f1a59ffb640 1 -- 172.21.15.31:0/1715517285 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x558c432ed800 con 0x558c42f733a0 2024-04-25T16:06:26.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.005+0000 7f1a59ffb640 1 -- 172.21.15.31:0/1715517285 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f1a68025480 con 0x558c42f733a0 2024-04-25T16:06:26.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.005+0000 7f1a7f9d3740 10 client.26308 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:26.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.005+0000 7f1a7f9d3740 10 client.26308 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:26.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.005+0000 7f1a7f9d3740 20 client.26308 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:26.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.005+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558c433fffb0 0x558c434023d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:26.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.005+0000 7f1a7f9d3740 4 client.26308 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:06:26.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.005+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1715517285 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558c432684a0 con 0x558c433fffb0 2024-04-25T16:06:26.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.005+0000 7f1a7259c640 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558c433fffb0 0x558c434023d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:26.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.006+0000 7f1a59ffb640 10 client.26308 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:26.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.007+0000 7f1a7259c640 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558c433fffb0 0x558c434023d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1857fd3d1005c541 server_cookie=a597d9e492675f0 in_seq=0 out_seq=0 2024-04-25T16:06:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.010+0000 7f1a59ffb640 1 -- 172.21.15.31:0/1715517285 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x558c432684a0 con 0x558c433fffb0 2024-04-25T16:06:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.010+0000 7f1a59ffb640 10 client.26308 handle_command_reply: tid=0 2024-04-25T16:06:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.010+0000 7f1a7f9d3740 1 client.26308 shutdown 2024-04-25T16:06:26.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.014+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1715517285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42f733a0 msgr2=0x558c4338f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:26.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.014+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42f733a0 0x558c4338f010 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1a6801a010 tx=0x7f1a68006930 comp rx=0 tx=0).stop 2024-04-25T16:06:26.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.017+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1715517285 shutdown_connections 2024-04-25T16:06:26.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.017+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558c433fffb0 0x558c434023d0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:26.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.017+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558c42f733a0 0x558c4338f010 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:26.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.017+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558c42fbe6b0 0x558c4338ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:26.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.017+0000 7f1a7f9d3740 1 --2- 172.21.15.31:0/1715517285 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558c42fb64d0 0x558c43391f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:26.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.017+0000 7f1a59ffb640 0 client.26308 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:26.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.018+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1715517285 >> 172.21.15.31:0/1715517285 conn(0x558c42fd34e0 msgr2=0x558c4337f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:26.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.019+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1715517285 shutdown_connections 2024-04-25T16:06:26.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.019+0000 7f1a7f9d3740 1 -- 172.21.15.31:0/1715517285 wait complete. 2024-04-25T16:06:26.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.020+0000 7f1a7f9d3740 20 client.26308 trim_cache size 0 max 16384 2024-04-25T16:06:26.030 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:06:26.030 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (3 inodes in the stack)", 2024-04-25T16:06:26.030 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:06:26.030 INFO:teuthology.orchestra.run.smithi031.stdout: "c1daa02d-8486-46e3-bedb-e1d6943ac561": { 2024-04-25T16:06:26.030 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:06:26.030 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "c1daa02d-8486-46e3-bedb-e1d6943ac561", 2024-04-25T16:06:26.030 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:06:26.030 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:06:26.031 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:06:26.031 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:06:26.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.046+0000 7f1a78a49640 1 -- 172.21.15.31:0/2592083222 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a5c0afba0 msgr2=0x7f1a5c0b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.046+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a5c0afba0 0x7f1a5c0b20b0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f1a6800bfd0 tx=0x7f1a6800b730 comp rx=0 tx=0).stop 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.046+0000 7f1a78a49640 1 -- 172.21.15.31:0/2592083222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 msgr2=0x7f1a741a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.046+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 0x7f1a741a08e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c031a90 tx=0x7f1a6c003f10 comp rx=0 tx=0).stop 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.048+0000 7f1a78a49640 1 -- 172.21.15.31:0/2592083222 shutdown_connections 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.048+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a5c0afba0 0x7f1a5c0b20b0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.048+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1a74110030 0x7f1a741a7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.048+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a7410b330 0x7f1a741a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.048+0000 7f1a78a49640 1 --2- 172.21.15.31:0/2592083222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a7410a0f0 0x7f1a741a08e0 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1a6c031a90 tx=0x7f1a6c003f10 comp rx=0 tx=0).stop 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.048+0000 7f1a78a49640 1 -- 172.21.15.31:0/2592083222 >> 172.21.15.31:0/2592083222 conn(0x7f1a74078de0 msgr2=0x7f1a7410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:26.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.049+0000 7f1a78a49640 1 -- 172.21.15.31:0/2592083222 shutdown_connections 2024-04-25T16:06:26.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:26.050+0000 7f1a78a49640 1 -- 172.21.15.31:0/2592083222 wait complete. 2024-04-25T16:06:26.091 INFO:tasks.cephfs.filesystem:scrub status for tag:c1daa02d-8486-46e3-bedb-e1d6943ac561 - {'path': '/', 'tag': 'c1daa02d-8486-46e3-bedb-e1d6943ac561', 'options': 'recursive,force'} 2024-04-25T16:06:27.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.114+0000 7fed4e5ce640 1 -- 172.21.15.31:0/3042731747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4810b350 msgr2=0x7fed4810b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:27.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.114+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/3042731747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4810b350 0x7fed4810b760 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fed40009900 tx=0x7fed4002a420 comp rx=0 tx=0).stop 2024-04-25T16:06:27.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.126+0000 7fed4e5ce640 1 -- 172.21.15.31:0/3042731747 shutdown_connections 2024-04-25T16:06:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.126+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/3042731747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed48110070 0x7fed48110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.126+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/3042731747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4810b350 0x7fed4810b760 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.126+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/3042731747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed4810a420 0x7fed4810ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.128 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.126+0000 7fed4e5ce640 1 -- 172.21.15.31:0/3042731747 >> 172.21.15.31:0/3042731747 conn(0x7fed48070540 msgr2=0x7fed480729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:27.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.129+0000 7fed4e5ce640 1 -- 172.21.15.31:0/3042731747 shutdown_connections 2024-04-25T16:06:27.131 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.131+0000 7fed4e5ce640 1 -- 172.21.15.31:0/3042731747 wait complete. 2024-04-25T16:06:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.132+0000 7fed4e5ce640 1 Processor -- start 2024-04-25T16:06:27.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.132+0000 7fed4e5ce640 1 -- start start 2024-04-25T16:06:27.134 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.133+0000 7fed4e5ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4810a420 0x7fed481a9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.134+0000 7fed4e5ce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed4810b350 0x7fed481a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.134+0000 7fed4e5ce640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed48110070 0x7fed481b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.134+0000 7fed4e5ce640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed4811e5a0 con 0x7fed4810a420 2024-04-25T16:06:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.134+0000 7fed4e5ce640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fed4811e420 con 0x7fed48110070 2024-04-25T16:06:27.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.134+0000 7fed4e5ce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed4811e720 con 0x7fed4810b350 2024-04-25T16:06:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.134+0000 7fed477fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed4810b350 0x7fed481a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.134+0000 7fed477fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed4810b350 0x7fed481a9680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:50954/0 (socket says 172.21.15.31:50954) 2024-04-25T16:06:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.134+0000 7fed477fe640 1 -- 172.21.15.31:0/1001837239 learned_addr learned my addr 172.21.15.31:0/1001837239 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.135+0000 7fed4cb6b640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed48110070 0x7fed481b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.136 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.135+0000 7fed477fe640 1 -- 172.21.15.31:0/1001837239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed48110070 msgr2=0x7fed481b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.135+0000 7fed477fe640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed48110070 0x7fed481b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.135+0000 7fed477fe640 1 -- 172.21.15.31:0/1001837239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4810a420 msgr2=0x7fed481a9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.135+0000 7fed477fe640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4810a420 0x7fed481a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.135+0000 7fed477fe640 1 -- 172.21.15.31:0/1001837239 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed481b0c80 con 0x7fed4810b350 2024-04-25T16:06:27.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.135+0000 7fed4cb6b640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed48110070 0x7fed481b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:06:27.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.136+0000 7fed477fe640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed4810b350 0x7fed481a9680 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fed40031440 tx=0x7fed40003dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.147+0000 7fed457fa640 1 -- 172.21.15.31:0/1001837239 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4003f070 con 0x7fed4810b350 2024-04-25T16:06:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.147+0000 7fed457fa640 1 -- 172.21.15.31:0/1001837239 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed4003b460 con 0x7fed4810b350 2024-04-25T16:06:27.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.147+0000 7fed457fa640 1 -- 172.21.15.31:0/1001837239 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40035030 con 0x7fed4810b350 2024-04-25T16:06:27.169 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.161+0000 7fed4e5ce640 1 -- 172.21.15.31:0/1001837239 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed481b0f10 con 0x7fed4810b350 2024-04-25T16:06:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.162+0000 7fed4e5ce640 1 -- 172.21.15.31:0/1001837239 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed481b12c0 con 0x7fed4810b350 2024-04-25T16:06:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.167+0000 7fed55480740 1 Processor -- start 2024-04-25T16:06:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.167+0000 7fed55480740 1 -- start start 2024-04-25T16:06:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.168+0000 7fed55480740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154914970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.168+0000 7fed55480740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x562154914ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.168+0000 7fed55480740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545534d0 0x56215491bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.168+0000 7fed55480740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56215487a460 con 0x5621545741b0 2024-04-25T16:06:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.168+0000 7fed55480740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562154834590 con 0x56215454b2f0 2024-04-25T16:06:27.170 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.168+0000 7fed55480740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562154876420 con 0x5621545534d0 2024-04-25T16:06:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.177+0000 7fed457fa640 1 -- 172.21.15.31:0/1001837239 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fed40042070 con 0x7fed4810b350 2024-04-25T16:06:27.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.178+0000 7fed477fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545534d0 0x56215491bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.179+0000 7fed4cb6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154914970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.179+0000 7fed457fa640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fed300afc70 0x7fed300b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.179+0000 7fed477fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545534d0 0x56215491bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:50958/0 (socket says 172.21.15.31:50958) 2024-04-25T16:06:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.182+0000 7fed457fa640 1 -- 172.21.15.31:0/1001837239 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fed400ff6f0 con 0x7fed4810b350 2024-04-25T16:06:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.179+0000 7fed477fe640 1 -- 172.21.15.31:0/3084307206 learned_addr learned my addr 172.21.15.31:0/3084307206 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.179+0000 7fed47fff640 1 --2- 172.21.15.31:0/3084307206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x562154914ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.183+0000 7fed4cb6b640 1 -- 172.21.15.31:0/3084307206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545534d0 msgr2=0x56215491bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.183+0000 7fed4cb6b640 1 --2- 172.21.15.31:0/3084307206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545534d0 0x56215491bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.183+0000 7fed4cb6b640 1 -- 172.21.15.31:0/3084307206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 msgr2=0x562154914ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.183+0000 7fed4cb6b640 1 --2- 172.21.15.31:0/3084307206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x562154914ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.183+0000 7fed4cb6b640 1 -- 172.21.15.31:0/3084307206 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621548410d0 con 0x5621545741b0 2024-04-25T16:06:27.184 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.183+0000 7fed47fff640 1 --2- 172.21.15.31:0/3084307206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x562154914ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:06:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.187+0000 7fed4cb6b640 1 --2- 172.21.15.31:0/3084307206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154914970 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00a9d0 tx=0x7fed3c00ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3d9feaf8a5a32e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:27.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.187+0000 7fed4cb6b640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fed300afc70 0x7fed300b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.206+0000 7fed4cb6b640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fed300afc70 0x7fed300b2180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fed3c01c000 tx=0x7fed3c01d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.206+0000 7fed2e7fc640 1 -- 172.21.15.31:0/3084307206 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c00f040 con 0x5621545741b0 2024-04-25T16:06:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.206+0000 7fed2e7fc640 1 -- 172.21.15.31:0/3084307206 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed3c017040 con 0x5621545741b0 2024-04-25T16:06:27.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.206+0000 7fed2e7fc640 1 -- 172.21.15.31:0/3084307206 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3c01c610 con 0x5621545741b0 2024-04-25T16:06:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.208+0000 7fed55480740 1 -- 172.21.15.31:0/3084307206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 msgr2=0x562154914970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.208+0000 7fed55480740 1 --2- 172.21.15.31:0/3084307206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154914970 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fed3c00a9d0 tx=0x7fed3c00ae90 comp rx=0 tx=0).stop 2024-04-25T16:06:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.209+0000 7fed55480740 1 -- 172.21.15.31:0/3084307206 shutdown_connections 2024-04-25T16:06:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.209+0000 7fed55480740 1 --2- 172.21.15.31:0/3084307206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545534d0 0x56215491bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.209+0000 7fed55480740 1 --2- 172.21.15.31:0/3084307206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x562154914ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.209+0000 7fed55480740 1 --2- 172.21.15.31:0/3084307206 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154914970 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.209+0000 7fed55480740 1 -- 172.21.15.31:0/3084307206 >> 172.21.15.31:0/3084307206 conn(0x562154568300 msgr2=0x562154910300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:27.220 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.219+0000 7fed55480740 1 -- 172.21.15.31:0/3084307206 shutdown_connections 2024-04-25T16:06:27.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.227+0000 7fed55480740 1 -- 172.21.15.31:0/3084307206 wait complete. 2024-04-25T16:06:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.229+0000 7fed55480740 1 -- start start 2024-04-25T16:06:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.229+0000 7fed55480740 1 Processor -- start 2024-04-25T16:06:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.230+0000 7fed55480740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x56215491e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.230+0000 7fed55480740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x56215491ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.230+0000 7fed55480740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 0x562154925ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.230+0000 7fed55480740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56215491c840 con 0x5621545534d0 2024-04-25T16:06:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.230+0000 7fed55480740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562154801b60 con 0x56215454b2f0 2024-04-25T16:06:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.230+0000 7fed55480740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56215491c9c0 con 0x5621545741b0 2024-04-25T16:06:27.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.230+0000 7fed55480740 10 client.0 _ll_register_callbacks cb 0x5621548262b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.233+0000 7fed4cb6b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 0x562154925ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.234+0000 7fed4cb6b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 0x562154925ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:50972/0 (socket says 172.21.15.31:50972) 2024-04-25T16:06:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.234+0000 7fed4cb6b640 1 -- 172.21.15.31:0/1726604538 learned_addr learned my addr 172.21.15.31:0/1726604538 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.234+0000 7fed477fe640 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x56215491ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.234+0000 7fed47fff640 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x56215491e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.235+0000 7fed477fe640 1 -- 172.21.15.31:0/1726604538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 msgr2=0x562154925ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.235+0000 7fed477fe640 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 0x562154925ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.236 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.235+0000 7fed477fe640 1 -- 172.21.15.31:0/1726604538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 msgr2=0x56215491e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.235+0000 7fed477fe640 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x56215491e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.237 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.235+0000 7fed477fe640 1 -- 172.21.15.31:0/1726604538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56215458c210 con 0x5621545534d0 2024-04-25T16:06:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.236+0000 7fed477fe640 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x56215491ec40 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fed400fd570 tx=0x7fed400c75b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31820360d750531f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.237+0000 7fed277fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.237+0000 7fed277fe640 1 -- 172.21.15.31:0/1726604538 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed400c7770 con 0x5621545534d0 2024-04-25T16:06:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.237+0000 7fed55480740 1 -- 172.21.15.31:0/1726604538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed3c017040 con 0x5621545534d0 2024-04-25T16:06:27.238 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.237+0000 7fed277fe640 1 -- 172.21.15.31:0/1726604538 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fed4003b910 con 0x5621545534d0 2024-04-25T16:06:27.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.239+0000 7fed277fe640 1 -- 172.21.15.31:0/1726604538 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fed400c8360 con 0x5621545534d0 2024-04-25T16:06:27.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.239+0000 7fed55480740 10 client.16707 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:27.241 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.239+0000 7fed55480740 1 -- 172.21.15.31:0/1726604538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562154658620 con 0x5621545534d0 2024-04-25T16:06:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.245+0000 7fed277fe640 1 -- 172.21.15.31:0/1726604538 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed3c017040 con 0x5621545534d0 2024-04-25T16:06:27.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.246+0000 7fed277fe640 1 -- 172.21.15.31:0/1726604538 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fed40102ce0 con 0x5621545534d0 2024-04-25T16:06:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.247+0000 7fed55480740 10 client.16707 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.247+0000 7fed55480740 10 client.16707 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.247+0000 7fed55480740 20 client.16707 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.247+0000 7fed55480740 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562154532d80 0x56215493d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.247+0000 7fed55480740 4 client.16707 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:27.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.247+0000 7fed55480740 1 -- 172.21.15.31:0/1726604538 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562154888070 con 0x562154532d80 2024-04-25T16:06:27.249 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.248+0000 7fed47fff640 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562154532d80 0x56215493d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.250+0000 7fed47fff640 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562154532d80 0x56215493d540 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df1dd1b4c0f1b2ac server_cookie=ea6171759d3daf04 in_seq=0 out_seq=0 2024-04-25T16:06:27.251 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.250+0000 7fed277fe640 10 client.16707 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.253+0000 7fed277fe640 1 -- 172.21.15.31:0/1726604538 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562154888070 con 0x562154532d80 2024-04-25T16:06:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.254+0000 7fed277fe640 10 client.16707 handle_command_reply: tid=0 2024-04-25T16:06:27.255 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.254+0000 7fed55480740 1 client.16707 shutdown 2024-04-25T16:06:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.255+0000 7fed55480740 1 -- 172.21.15.31:0/1726604538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 msgr2=0x56215491ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.255+0000 7fed55480740 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x56215491ec40 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fed400fd570 tx=0x7fed400c75b0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.255+0000 7fed55480740 1 -- 172.21.15.31:0/1726604538 shutdown_connections 2024-04-25T16:06:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.255+0000 7fed55480740 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562154532d80 0x56215493d540 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.255+0000 7fed55480740 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 0x562154925ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.255+0000 7fed55480740 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x56215491ec40 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.255+0000 7fed55480740 1 --2- 172.21.15.31:0/1726604538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x56215491e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.255+0000 7fed55480740 1 -- 172.21.15.31:0/1726604538 >> 172.21.15.31:0/1726604538 conn(0x562154568300 msgr2=0x56215490f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:27.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.255+0000 7fed277fe640 0 client.16707 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:27.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.257+0000 7fed55480740 1 -- 172.21.15.31:0/1726604538 shutdown_connections 2024-04-25T16:06:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.257+0000 7fed55480740 1 -- 172.21.15.31:0/1726604538 wait complete. 2024-04-25T16:06:27.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.258+0000 7fed55480740 20 client.16707 trim_cache size 0 max 16384 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.306+0000 7fed55480740 1 Processor -- start 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.306+0000 7fed55480740 1 -- start start 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.307+0000 7fed55480740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x5621549102b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.307+0000 7fed55480740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x562154925ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.307+0000 7fed55480740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 0x56215491cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.307+0000 7fed55480740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed481b1800 con 0x5621545534d0 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.307+0000 7fed55480740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fed481b3340 con 0x56215454b2f0 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.307+0000 7fed55480740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed481b1ed0 con 0x5621545741b0 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.307+0000 7fed477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x562154925ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.307+0000 7fed477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x562154925ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53826/0 (socket says 172.21.15.31:53826) 2024-04-25T16:06:27.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.307+0000 7fed477fe640 1 -- 172.21.15.31:0/2046842144 learned_addr learned my addr 172.21.15.31:0/2046842144 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:27.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.312+0000 7fed4cb6b640 1 --2- 172.21.15.31:0/2046842144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 0x56215491cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.312+0000 7fed47fff640 1 --2- 172.21.15.31:0/2046842144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x5621549102b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.314+0000 7fed477fe640 1 -- 172.21.15.31:0/2046842144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 msgr2=0x56215491cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.314+0000 7fed477fe640 1 --2- 172.21.15.31:0/2046842144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 0x56215491cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.314+0000 7fed477fe640 1 -- 172.21.15.31:0/2046842144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 msgr2=0x5621549102b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.314+0000 7fed477fe640 1 --2- 172.21.15.31:0/2046842144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x5621549102b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.314+0000 7fed477fe640 1 -- 172.21.15.31:0/2046842144 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621549195a0 con 0x5621545534d0 2024-04-25T16:06:27.318 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.315+0000 7fed477fe640 1 --2- 172.21.15.31:0/2046842144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x562154925ac0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fed40000c00 tx=0x7fed400ffa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb972ffe97462a53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:27.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.339+0000 7fed2dffb640 1 -- 172.21.15.31:0/2046842144 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed400c3a90 con 0x5621545534d0 2024-04-25T16:06:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.349+0000 7fed2dffb640 1 -- 172.21.15.31:0/2046842144 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed400c7770 con 0x5621545534d0 2024-04-25T16:06:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.349+0000 7fed2dffb640 1 -- 172.21.15.31:0/2046842144 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed400e2350 con 0x5621545534d0 2024-04-25T16:06:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.351+0000 7fed55480740 1 -- 172.21.15.31:0/2046842144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 msgr2=0x562154925ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:27.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.351+0000 7fed55480740 1 --2- 172.21.15.31:0/2046842144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x562154925ac0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fed40000c00 tx=0x7fed400ffa80 comp rx=0 tx=0).stop 2024-04-25T16:06:27.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.352+0000 7fed55480740 1 -- 172.21.15.31:0/2046842144 shutdown_connections 2024-04-25T16:06:27.354 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.352+0000 7fed55480740 1 --2- 172.21.15.31:0/2046842144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5621545741b0 0x56215491cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.352+0000 7fed55480740 1 --2- 172.21.15.31:0/2046842144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545534d0 0x562154925ac0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.352+0000 7fed55480740 1 --2- 172.21.15.31:0/2046842144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56215454b2f0 0x5621549102b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.352+0000 7fed55480740 1 -- 172.21.15.31:0/2046842144 >> 172.21.15.31:0/2046842144 conn(0x562154568300 msgr2=0x56215490fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:27.359 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.358+0000 7fed55480740 1 -- 172.21.15.31:0/2046842144 shutdown_connections 2024-04-25T16:06:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.364+0000 7fed55480740 1 -- 172.21.15.31:0/2046842144 wait complete. 2024-04-25T16:06:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.364+0000 7fed55480740 1 -- start start 2024-04-25T16:06:27.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.365+0000 7fed55480740 1 Processor -- start 2024-04-25T16:06:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.366+0000 7fed55480740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56215454b2f0 0x562154922fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.366+0000 7fed55480740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621545534d0 0x562154923510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.366+0000 7fed55480740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154924440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.366+0000 7fed55480740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fed481b2c70 con 0x5621545741b0 2024-04-25T16:06:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.366+0000 7fed55480740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fed481b25a0 con 0x5621545534d0 2024-04-25T16:06:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.366+0000 7fed55480740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fed481186a0 con 0x56215454b2f0 2024-04-25T16:06:27.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.366+0000 7fed55480740 10 client.0 _ll_register_callbacks cb 0x7fed400c33d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.370+0000 7fed477fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621545534d0 0x562154923510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.370+0000 7fed47fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56215454b2f0 0x562154922fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.370+0000 7fed4cb6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154924440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.370+0000 7fed4cb6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154924440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53836/0 (socket says 172.21.15.31:53836) 2024-04-25T16:06:27.371 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.370+0000 7fed4cb6b640 1 -- 172.21.15.31:0/784717949 learned_addr learned my addr 172.21.15.31:0/784717949 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.377+0000 7fed47fff640 1 -- 172.21.15.31:0/784717949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621545534d0 msgr2=0x562154923510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.377+0000 7fed47fff640 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621545534d0 0x562154923510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.378+0000 7fed47fff640 1 -- 172.21.15.31:0/784717949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 msgr2=0x562154924440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.378+0000 7fed47fff640 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154924440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.378+0000 7fed47fff640 1 -- 172.21.15.31:0/784717949 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562154924a00 con 0x56215454b2f0 2024-04-25T16:06:27.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.381+0000 7fed477fe640 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621545534d0 0x562154923510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:06:27.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.385+0000 7fed47fff640 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56215454b2f0 0x562154922fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x56215493dcb0 tx=0x7fed38010b40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a71ab5ecf8d3961b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:27.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.387+0000 7fed277fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.391+0000 7fed277fe640 1 -- 172.21.15.31:0/784717949 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed3800faf0 con 0x56215454b2f0 2024-04-25T16:06:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.391+0000 7fed55480740 1 -- 172.21.15.31:0/784717949 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed400c7770 con 0x56215454b2f0 2024-04-25T16:06:27.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.391+0000 7fed277fe640 1 -- 172.21.15.31:0/784717949 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fed3800fc90 con 0x56215454b2f0 2024-04-25T16:06:27.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.396+0000 7fed277fe640 1 -- 172.21.15.31:0/784717949 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fed38027050 con 0x56215454b2f0 2024-04-25T16:06:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.396+0000 7fed55480740 10 client.26323 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.396+0000 7fed55480740 1 -- 172.21.15.31:0/784717949 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562154925d60 con 0x56215454b2f0 2024-04-25T16:06:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.398+0000 7fed277fe640 1 -- 172.21.15.31:0/784717949 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fed400c7770 con 0x56215454b2f0 2024-04-25T16:06:27.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.398+0000 7fed277fe640 1 -- 172.21.15.31:0/784717949 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fed38021670 con 0x56215454b2f0 2024-04-25T16:06:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.398+0000 7fed55480740 10 client.26323 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.398+0000 7fed55480740 10 client.26323 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.398+0000 7fed55480740 20 client.26323 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.398+0000 7fed55480740 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621545081c0 0x5621549a7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.398+0000 7fed55480740 4 client.26323 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:06:27.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.398+0000 7fed55480740 1 -- 172.21.15.31:0/784717949 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562154881ee0 con 0x5621545081c0 2024-04-25T16:06:27.404 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.403+0000 7fed477fe640 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621545081c0 0x5621549a7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:27.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.420+0000 7fed477fe640 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621545081c0 0x5621549a7fe0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e4b9fd56ca6a3b41 server_cookie=94de3e2c4211229c in_seq=0 out_seq=0 2024-04-25T16:06:27.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:27.422+0000 7fed277fe640 10 client.26323 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:27.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:27 smithi063 ceph-mon[32413]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 717 MiB/s wr, 2.26k op/s 2024-04-25T16:06:27.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:27 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:06:27.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:27 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:06:27.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:27 smithi186 ceph-mon[33258]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 717 MiB/s wr, 2.26k op/s 2024-04-25T16:06:27.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:27 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:06:27.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:27 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:06:27.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:27 smithi031 ceph-mon[29140]: pgmap v505: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 717 MiB/s wr, 2.26k op/s 2024-04-25T16:06:27.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:27 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:06:27.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:27 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:06:29.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:29 smithi063 ceph-mon[32413]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 838 MiB/s wr, 2.56k op/s 2024-04-25T16:06:29.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:29 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:06:29.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:29 smithi186 ceph-mon[33258]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 838 MiB/s wr, 2.56k op/s 2024-04-25T16:06:29.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:29 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:06:29.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:29 smithi031 ceph-mon[29140]: pgmap v506: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 838 MiB/s wr, 2.56k op/s 2024-04-25T16:06:29.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:29 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:06:30.335 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.334+0000 7fed277fe640 1 -- 172.21.15.31:0/784717949 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562154881ee0 con 0x5621545081c0 2024-04-25T16:06:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.334+0000 7fed277fe640 10 client.26323 handle_command_reply: tid=0 2024-04-25T16:06:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.334+0000 7fed55480740 1 client.26323 shutdown 2024-04-25T16:06:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.334+0000 7fed55480740 1 -- 172.21.15.31:0/784717949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56215454b2f0 msgr2=0x562154922fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.334+0000 7fed55480740 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56215454b2f0 0x562154922fd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x56215493dcb0 tx=0x7fed38010b40 comp rx=0 tx=0).stop 2024-04-25T16:06:30.336 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.335+0000 7fed55480740 1 -- 172.21.15.31:0/784717949 shutdown_connections 2024-04-25T16:06:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.335+0000 7fed55480740 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5621545081c0 0x5621549a7fe0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.335+0000 7fed55480740 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5621545741b0 0x562154924440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.335+0000 7fed55480740 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5621545534d0 0x562154923510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.335+0000 7fed55480740 1 --2- 172.21.15.31:0/784717949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56215454b2f0 0x562154922fd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.335+0000 7fed55480740 1 -- 172.21.15.31:0/784717949 >> 172.21.15.31:0/784717949 conn(0x562154568300 msgr2=0x56215491cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.335+0000 7fed277fe640 0 client.26323 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.336+0000 7fed55480740 1 -- 172.21.15.31:0/784717949 shutdown_connections 2024-04-25T16:06:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.336+0000 7fed55480740 1 -- 172.21.15.31:0/784717949 wait complete. 2024-04-25T16:06:30.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.336+0000 7fed55480740 20 client.26323 trim_cache size 0 max 16384 2024-04-25T16:06:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:06:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:06:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:06:30.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:06:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.342+0000 7fed4e5ce640 1 -- 172.21.15.31:0/1001837239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fed300afc70 msgr2=0x7fed300b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.342+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fed300afc70 0x7fed300b2180 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fed3c01c000 tx=0x7fed3c01d000 comp rx=0 tx=0).stop 2024-04-25T16:06:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.342+0000 7fed4e5ce640 1 -- 172.21.15.31:0/1001837239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed4810b350 msgr2=0x7fed481a9680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:30.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.342+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed4810b350 0x7fed481a9680 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fed40031440 tx=0x7fed40003dc0 comp rx=0 tx=0).stop 2024-04-25T16:06:30.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.348+0000 7fed4e5ce640 1 -- 172.21.15.31:0/1001837239 shutdown_connections 2024-04-25T16:06:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.348+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fed300afc70 0x7fed300b2180 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.348+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fed48110070 0x7fed481b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.348+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed4810b350 0x7fed481a9680 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.348+0000 7fed4e5ce640 1 --2- 172.21.15.31:0/1001837239 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fed4810a420 0x7fed481a9140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:30.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.348+0000 7fed4e5ce640 1 -- 172.21.15.31:0/1001837239 >> 172.21.15.31:0/1001837239 conn(0x7fed48070540 msgr2=0x7fed4810c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:30.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.351+0000 7fed4e5ce640 1 -- 172.21.15.31:0/1001837239 shutdown_connections 2024-04-25T16:06:30.353 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:30.351+0000 7fed4e5ce640 1 -- 172.21.15.31:0/1001837239 wait complete. 2024-04-25T16:06:31.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:31 smithi063 ceph-mon[32413]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 758 MiB/s wr, 2.25k op/s 2024-04-25T16:06:31.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:31 smithi186 ceph-mon[33258]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 758 MiB/s wr, 2.25k op/s 2024-04-25T16:06:31.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:31 smithi031 ceph-mon[29140]: pgmap v507: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 758 MiB/s wr, 2.25k op/s 2024-04-25T16:06:33.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:33 smithi063 ceph-mon[32413]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 708 MiB/s wr, 2.10k op/s 2024-04-25T16:06:33.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:33 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:06:33.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:33 smithi186 ceph-mon[33258]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 708 MiB/s wr, 2.10k op/s 2024-04-25T16:06:33.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:33 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:06:33.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:33 smithi031 ceph-mon[29140]: pgmap v508: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 708 MiB/s wr, 2.10k op/s 2024-04-25T16:06:33.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:33 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:06:34.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:34 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:06:34.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:34 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:06:34.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:34 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:06:35.851 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:35 smithi031 ceph-mon[29140]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 918 MiB/s wr, 2.61k op/s 2024-04-25T16:06:35.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:35 smithi063 ceph-mon[32413]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 918 MiB/s wr, 2.61k op/s 2024-04-25T16:06:35.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:35 smithi186 ceph-mon[33258]: pgmap v509: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 918 MiB/s wr, 2.61k op/s 2024-04-25T16:06:37.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:37 smithi063 ceph-mon[32413]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 724 MiB/s wr, 1.94k op/s 2024-04-25T16:06:37.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:37 smithi186 ceph-mon[33258]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 724 MiB/s wr, 1.94k op/s 2024-04-25T16:06:37.953 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:37 smithi031 ceph-mon[29140]: pgmap v510: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 724 MiB/s wr, 1.94k op/s 2024-04-25T16:06:39.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:39 smithi063 ceph-mon[32413]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 904 MiB/s wr, 2.29k op/s 2024-04-25T16:06:39.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:39 smithi186 ceph-mon[33258]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 904 MiB/s wr, 2.29k op/s 2024-04-25T16:06:39.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:39 smithi031 ceph-mon[29140]: pgmap v511: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 904 MiB/s wr, 2.29k op/s 2024-04-25T16:06:40.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:40 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:06:40.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:40 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:06:41.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:40 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:06:41.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:41 smithi186 ceph-mon[33258]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 798 MiB/s wr, 2.10k op/s 2024-04-25T16:06:41.958 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:41 smithi031 ceph-mon[29140]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 798 MiB/s wr, 2.10k op/s 2024-04-25T16:06:42.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:41 smithi063 ceph-mon[32413]: pgmap v512: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 798 MiB/s wr, 2.10k op/s 2024-04-25T16:06:42.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:42 smithi186 ceph-mon[33258]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 749 MiB/s wr, 2.03k op/s 2024-04-25T16:06:42.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:42 smithi031 ceph-mon[29140]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 749 MiB/s wr, 2.03k op/s 2024-04-25T16:06:43.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:42 smithi063 ceph-mon[32413]: pgmap v513: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 749 MiB/s wr, 2.03k op/s 2024-04-25T16:06:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.709+0000 7f6b63fff640 1 -- 172.21.15.31:0/294009692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b6410feb0 msgr2=0x7f6b641102c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:43.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.709+0000 7f6b63fff640 1 --2- 172.21.15.31:0/294009692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b6410feb0 0x7f6b641102c0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009900 tx=0x7f6b5402a420 comp rx=0 tx=0).stop 2024-04-25T16:06:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.717+0000 7f6b63fff640 1 -- 172.21.15.31:0/294009692 shutdown_connections 2024-04-25T16:06:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.717+0000 7f6b63fff640 1 --2- 172.21.15.31:0/294009692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b6410feb0 0x7f6b641102c0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.717+0000 7f6b63fff640 1 --2- 172.21.15.31:0/294009692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b6410b190 0x7f6b6410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.717+0000 7f6b63fff640 1 --2- 172.21.15.31:0/294009692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b6410a340 0x7f6b6410ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.717+0000 7f6b63fff640 1 -- 172.21.15.31:0/294009692 >> 172.21.15.31:0/294009692 conn(0x7f6b64070330 msgr2=0x7f6b640727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:43.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.728+0000 7f6b63fff640 1 -- 172.21.15.31:0/294009692 shutdown_connections 2024-04-25T16:06:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.730+0000 7f6b63fff640 1 -- 172.21.15.31:0/294009692 wait complete. 2024-04-25T16:06:43.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.732+0000 7f6b63fff640 1 Processor -- start 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.733+0000 7f6b63fff640 1 -- start start 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.734+0000 7f6b63fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b6410a340 0x7f6b64119cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.734+0000 7f6b63fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b6410b190 0x7f6b6411a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.734+0000 7f6b63fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b6410feb0 0x7f6b6411b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.734+0000 7f6b63fff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64111f90 con 0x7f6b6410feb0 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.734+0000 7f6b63fff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64111e10 con 0x7f6b6410b190 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.734+0000 7f6b63fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b64112110 con 0x7f6b6410a340 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.735+0000 7f6b627fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b6410b190 0x7f6b6411a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.736+0000 7f6b627fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b6410b190 0x7f6b6411a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:33346/0 (socket says 172.21.15.31:33346) 2024-04-25T16:06:43.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.736+0000 7f6b627fc640 1 -- 172.21.15.31:0/1976761772 learned_addr learned my addr 172.21.15.31:0/1976761772 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.737+0000 7f6b627fc640 1 -- 172.21.15.31:0/1976761772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b6410a340 msgr2=0x7f6b64119cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.737+0000 7f6b627fc640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b6410a340 0x7f6b64119cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.737+0000 7f6b627fc640 1 -- 172.21.15.31:0/1976761772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b6410feb0 msgr2=0x7f6b6411b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.737+0000 7f6b627fc640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b6410feb0 0x7f6b6411b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.740 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.737+0000 7f6b627fc640 1 -- 172.21.15.31:0/1976761772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6411b7d0 con 0x7f6b6410b190 2024-04-25T16:06:43.741 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.739+0000 7f6b627fc640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b6410b190 0x7f6b6411a200 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00ba50 tx=0x7f6b5c00bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:43.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.741+0000 7f6b688de640 1 -- 172.21.15.31:0/1976761772 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c019070 con 0x7f6b6410b190 2024-04-25T16:06:43.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.742+0000 7f6b63fff640 1 -- 172.21.15.31:0/1976761772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6411bac0 con 0x7f6b6410b190 2024-04-25T16:06:43.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.742+0000 7f6b63fff640 1 -- 172.21.15.31:0/1976761772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6411c010 con 0x7f6b6410b190 2024-04-25T16:06:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.748+0000 7f6b70791740 1 Processor -- start 2024-04-25T16:06:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.748+0000 7f6b70791740 1 -- start start 2024-04-25T16:06:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.749+0000 7f6b70791740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992901b0 0x556599630b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.751 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.749+0000 7f6b70791740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x556599631090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.749+0000 7f6b70791740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 0x556599637f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.749+0000 7f6b70791740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556599596460 con 0x55659926f4d0 2024-04-25T16:06:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.749+0000 7f6b70791740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556599550590 con 0x5565992672f0 2024-04-25T16:06:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.749+0000 7f6b70791740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556599592420 con 0x5565992901b0 2024-04-25T16:06:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.749+0000 7f6b688de640 1 -- 172.21.15.31:0/1976761772 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b5c00cbe0 con 0x7f6b6410b190 2024-04-25T16:06:43.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.749+0000 7f6b688de640 1 -- 172.21.15.31:0/1976761772 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c0075f0 con 0x7f6b6410b190 2024-04-25T16:06:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.749+0000 7f6b62ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x556599631090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.750+0000 7f6b62ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x556599631090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:33360/0 (socket says 172.21.15.31:33360) 2024-04-25T16:06:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.750+0000 7f6b62ffd640 1 -- 172.21.15.31:0/2157718296 learned_addr learned my addr 172.21.15.31:0/2157718296 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.751+0000 7f6b62ffd640 1 -- 172.21.15.31:0/2157718296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992901b0 msgr2=0x556599630b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.751+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/2157718296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992901b0 0x556599630b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.751+0000 7f6b62ffd640 1 -- 172.21.15.31:0/2157718296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 msgr2=0x556599637f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:43.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.751+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/2157718296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 0x556599637f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.759+0000 7f6b688de640 1 -- 172.21.15.31:0/1976761772 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6b5c01c070 con 0x7f6b6410b190 2024-04-25T16:06:43.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.760+0000 7f6b688de640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b4c0afcf0 0x7f6b4c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.761+0000 7f6b688de640 1 -- 172.21.15.31:0/1976761772 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6b5c0e12b0 con 0x7f6b6410b190 2024-04-25T16:06:43.764 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.762+0000 7f6b62ffd640 1 -- 172.21.15.31:0/2157718296 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55659955d0d0 con 0x5565992672f0 2024-04-25T16:06:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.766+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/2157718296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x556599631090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800a9c0 tx=0x7f6b5800ae80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a075467de4fb4a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:43.769 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.767+0000 7f6b637fe640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b4c0afcf0 0x7f6b4c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.768+0000 7f6b49ffb640 1 -- 172.21.15.31:0/2157718296 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5800cd60 con 0x5565992672f0 2024-04-25T16:06:43.770 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.768+0000 7f6b49ffb640 1 -- 172.21.15.31:0/2157718296 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b5800f900 con 0x5565992672f0 2024-04-25T16:06:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.779+0000 7f6b70791740 1 -- 172.21.15.31:0/2157718296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 msgr2=0x556599631090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:43.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.779+0000 7f6b70791740 1 --2- 172.21.15.31:0/2157718296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x556599631090 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6b5800a9c0 tx=0x7f6b5800ae80 comp rx=0 tx=0).stop 2024-04-25T16:06:43.782 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.780+0000 7f6b637fe640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b4c0afcf0 0x7f6b4c0b2200 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6b540096c0 tx=0x7f6b540056d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.783+0000 7f6b70791740 1 -- 172.21.15.31:0/2157718296 shutdown_connections 2024-04-25T16:06:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.783+0000 7f6b70791740 1 --2- 172.21.15.31:0/2157718296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 0x556599637f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.783+0000 7f6b70791740 1 --2- 172.21.15.31:0/2157718296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x556599631090 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.783+0000 7f6b70791740 1 --2- 172.21.15.31:0/2157718296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992901b0 0x556599630b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.783+0000 7f6b70791740 1 -- 172.21.15.31:0/2157718296 >> 172.21.15.31:0/2157718296 conn(0x556599284300 msgr2=0x55659962c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:43.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.791+0000 7f6b70791740 1 -- 172.21.15.31:0/2157718296 shutdown_connections 2024-04-25T16:06:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.791+0000 7f6b70791740 1 -- 172.21.15.31:0/2157718296 wait complete. 2024-04-25T16:06:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.792+0000 7f6b70791740 1 -- start start 2024-04-25T16:06:43.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.792+0000 7f6b70791740 1 Processor -- start 2024-04-25T16:06:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.802+0000 7f6b70791740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.802+0000 7f6b70791740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 0x55659963ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.802+0000 7f6b70791740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992901b0 0x556599641d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.802+0000 7f6b70791740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556599638a80 con 0x55659926f4d0 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.802+0000 7f6b70791740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55659951db60 con 0x5565992672f0 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.802+0000 7f6b70791740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556599638c00 con 0x5565992901b0 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.802+0000 7f6b70791740 10 client.0 _ll_register_callbacks cb 0x5565995422b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.805+0000 7f6b627fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 0x55659963ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.805+0000 7f6b627fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 0x55659963ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33840/0 (socket says 172.21.15.31:33840) 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.805+0000 7f6b627fc640 1 -- 172.21.15.31:0/3648885788 learned_addr learned my addr 172.21.15.31:0/3648885788 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.807+0000 7f6b627fc640 1 -- 172.21.15.31:0/3648885788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992901b0 msgr2=0x556599641d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.807+0000 7f6b627fc640 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992901b0 0x556599641d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.807+0000 7f6b627fc640 1 -- 172.21.15.31:0/3648885788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 msgr2=0x55659963a8d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:06:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.809+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.809+0000 7f6b627fc640 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.809+0000 7f6b627fc640 1 -- 172.21.15.31:0/3648885788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565992a8210 con 0x55659926f4d0 2024-04-25T16:06:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.809+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:06:43.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.810+0000 7f6b627fc640 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 0x55659963ae10 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00ba20 tx=0x7f6b5c0a9310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b110e5e45489245e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:43.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.811+0000 7f6b42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:43.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.812+0000 7f6b42ffd640 1 -- 172.21.15.31:0/3648885788 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c0a9740 con 0x55659926f4d0 2024-04-25T16:06:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.813+0000 7f6b70791740 1 -- 172.21.15.31:0/3648885788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b5800f900 con 0x55659926f4d0 2024-04-25T16:06:43.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.813+0000 7f6b42ffd640 1 -- 172.21.15.31:0/3648885788 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6b5c0b2110 con 0x55659926f4d0 2024-04-25T16:06:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.814+0000 7f6b42ffd640 1 -- 172.21.15.31:0/3648885788 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6b5c0adaf0 con 0x55659926f4d0 2024-04-25T16:06:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.814+0000 7f6b70791740 10 client.16719 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:43.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.814+0000 7f6b70791740 1 -- 172.21.15.31:0/3648885788 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556599374620 con 0x55659926f4d0 2024-04-25T16:06:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.817+0000 7f6b42ffd640 1 -- 172.21.15.31:0/3648885788 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b5800f900 con 0x55659926f4d0 2024-04-25T16:06:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.817+0000 7f6b42ffd640 1 -- 172.21.15.31:0/3648885788 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6b5c014440 con 0x55659926f4d0 2024-04-25T16:06:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.817+0000 7f6b70791740 10 client.16719 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.817+0000 7f6b70791740 10 client.16719 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.817+0000 7f6b70791740 20 client.16719 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.817+0000 7f6b70791740 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55659924ed80 0x556599648ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.817+0000 7f6b70791740 4 client.16719 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:43.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.817+0000 7f6b70791740 1 -- 172.21.15.31:0/3648885788 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5565995a4070 con 0x55659924ed80 2024-04-25T16:06:43.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.821+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55659924ed80 0x556599648ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.824+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55659924ed80 0x556599648ac0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3a80a7aebf3416 server_cookie=66f74415cb526093 in_seq=0 out_seq=0 2024-04-25T16:06:43.827 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.825+0000 7f6b42ffd640 10 client.16719 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.831+0000 7f6b42ffd640 1 -- 172.21.15.31:0/3648885788 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5565995a4070 con 0x55659924ed80 2024-04-25T16:06:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.831+0000 7f6b42ffd640 10 client.16719 handle_command_reply: tid=0 2024-04-25T16:06:43.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.831+0000 7f6b70791740 1 client.16719 shutdown 2024-04-25T16:06:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.836+0000 7f6b70791740 1 -- 172.21.15.31:0/3648885788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 msgr2=0x55659963ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:43.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.836+0000 7f6b70791740 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 0x55659963ae10 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00ba20 tx=0x7f6b5c0a9310 comp rx=0 tx=0).stop 2024-04-25T16:06:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.840+0000 7f6b70791740 1 -- 172.21.15.31:0/3648885788 shutdown_connections 2024-04-25T16:06:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.840+0000 7f6b70791740 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55659924ed80 0x556599648ac0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.840+0000 7f6b70791740 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992901b0 0x556599641d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.840+0000 7f6b70791740 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55659926f4d0 0x55659963ae10 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.840+0000 7f6b70791740 1 --2- 172.21.15.31:0/3648885788 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.840+0000 7f6b70791740 1 -- 172.21.15.31:0/3648885788 >> 172.21.15.31:0/3648885788 conn(0x556599284300 msgr2=0x55659962f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:43.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.840+0000 7f6b42ffd640 0 client.16719 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.842+0000 7f6b70791740 1 -- 172.21.15.31:0/3648885788 shutdown_connections 2024-04-25T16:06:43.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.844+0000 7f6b70791740 1 -- 172.21.15.31:0/3648885788 wait complete. 2024-04-25T16:06:43.847 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.845+0000 7f6b70791740 20 client.16719 trim_cache size 0 max 16384 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.897+0000 7f6b70791740 1 Processor -- start 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.897+0000 7f6b70791740 1 -- start start 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.897+0000 7f6b70791740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992672f0 0x556599634bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.897+0000 7f6b70791740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55659926f4d0 0x556599641c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.898+0000 7f6b70791740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x556599635110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.898+0000 7f6b70791740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5000e4f0 con 0x5565992901b0 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.898+0000 7f6b637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x556599635110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.898+0000 7f6b637fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x556599635110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33850/0 (socket says 172.21.15.31:33850) 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.898+0000 7f6b637fe640 1 -- 172.21.15.31:0/2636684170 learned_addr learned my addr 172.21.15.31:0/2636684170 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.898+0000 7f6b627fc640 1 --2- 172.21.15.31:0/2636684170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55659926f4d0 0x556599641c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.898+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/2636684170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992672f0 0x556599634bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.898+0000 7f6b70791740 1 -- 172.21.15.31:0/2636684170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6b50010030 con 0x55659926f4d0 2024-04-25T16:06:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.898+0000 7f6b70791740 1 -- 172.21.15.31:0/2636684170 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5000ebc0 con 0x5565992672f0 2024-04-25T16:06:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.899+0000 7f6b62ffd640 1 -- 172.21.15.31:0/2636684170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55659926f4d0 msgr2=0x556599641c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.899+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/2636684170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55659926f4d0 0x556599641c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.899+0000 7f6b62ffd640 1 -- 172.21.15.31:0/2636684170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 msgr2=0x556599635110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.899+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/2636684170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x556599635110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.899+0000 7f6b62ffd640 1 -- 172.21.15.31:0/2636684170 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556599631800 con 0x5565992672f0 2024-04-25T16:06:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.900+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/2636684170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992672f0 0x556599634bd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6b58018510 tx=0x7f6b58018400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3da8f9524a6e0be5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.900+0000 7f6b497fa640 1 -- 172.21.15.31:0/2636684170 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5800cd60 con 0x5565992672f0 2024-04-25T16:06:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.900+0000 7f6b497fa640 1 -- 172.21.15.31:0/2636684170 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b5801e040 con 0x5565992672f0 2024-04-25T16:06:43.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.900+0000 7f6b497fa640 1 -- 172.21.15.31:0/2636684170 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b58003b00 con 0x5565992672f0 2024-04-25T16:06:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.903+0000 7f6b70791740 1 -- 172.21.15.31:0/2636684170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992672f0 msgr2=0x556599634bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.903+0000 7f6b70791740 1 --2- 172.21.15.31:0/2636684170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992672f0 0x556599634bd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6b58018510 tx=0x7f6b58018400 comp rx=0 tx=0).stop 2024-04-25T16:06:43.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.903+0000 7f6b70791740 1 -- 172.21.15.31:0/2636684170 shutdown_connections 2024-04-25T16:06:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.903+0000 7f6b70791740 1 --2- 172.21.15.31:0/2636684170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x556599635110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.903+0000 7f6b70791740 1 --2- 172.21.15.31:0/2636684170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55659926f4d0 0x556599641c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.903+0000 7f6b70791740 1 --2- 172.21.15.31:0/2636684170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5565992672f0 0x556599634bd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.903+0000 7f6b70791740 1 -- 172.21.15.31:0/2636684170 >> 172.21.15.31:0/2636684170 conn(0x556599284300 msgr2=0x556599630a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.905+0000 7f6b70791740 1 -- 172.21.15.31:0/2636684170 shutdown_connections 2024-04-25T16:06:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.905+0000 7f6b70791740 1 -- 172.21.15.31:0/2636684170 wait complete. 2024-04-25T16:06:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.906+0000 7f6b70791740 1 -- start start 2024-04-25T16:06:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.906+0000 7f6b70791740 1 Processor -- start 2024-04-25T16:06:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b70791740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b70791740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55659926f4d0 0x55659963f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b70791740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x5565996aee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b70791740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5000f960 con 0x5565992901b0 2024-04-25T16:06:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b70791740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5000f290 con 0x5565992672f0 2024-04-25T16:06:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b70791740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5000c4d0 con 0x55659926f4d0 2024-04-25T16:06:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b70791740 10 client.0 _ll_register_callbacks cb 0x7f6b5c0a8eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b62ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b62ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:33392/0 (socket says 172.21.15.31:33392) 2024-04-25T16:06:43.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.908+0000 7f6b637fe640 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x5565996aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.907+0000 7f6b62ffd640 1 -- 172.21.15.31:0/2832227582 learned_addr learned my addr 172.21.15.31:0/2832227582 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:43.911 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.908+0000 7f6b627fc640 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55659926f4d0 0x55659963f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.911+0000 7f6b637fe640 1 -- 172.21.15.31:0/2832227582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55659926f4d0 msgr2=0x55659963f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.911+0000 7f6b637fe640 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55659926f4d0 0x55659963f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.911+0000 7f6b637fe640 1 -- 172.21.15.31:0/2832227582 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 msgr2=0x55659963edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.911+0000 7f6b637fe640 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.911+0000 7f6b637fe640 1 -- 172.21.15.31:0/2832227582 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556599638e20 con 0x5565992901b0 2024-04-25T16:06:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.914+0000 7f6b637fe640 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x5565996aee60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009850 tx=0x7f6b54005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd06c2dd99b3195 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.915+0000 7f6b42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.916+0000 7f6b42ffd640 1 -- 172.21.15.31:0/2832227582 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b5404d070 con 0x5565992901b0 2024-04-25T16:06:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.916+0000 7f6b42ffd640 1 -- 172.21.15.31:0/2832227582 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f6b54003a00 con 0x5565992901b0 2024-04-25T16:06:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.916+0000 7f6b70791740 1 -- 172.21.15.31:0/2832227582 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b5801e040 con 0x5565992901b0 2024-04-25T16:06:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.919+0000 7f6b42ffd640 1 -- 172.21.15.31:0/2832227582 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6b54057050 con 0x5565992901b0 2024-04-25T16:06:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.919+0000 7f6b70791740 10 client.16725 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.920+0000 7f6b70791740 1 -- 172.21.15.31:0/2832227582 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565996d2400 con 0x5565992901b0 2024-04-25T16:06:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.921+0000 7f6b42ffd640 1 -- 172.21.15.31:0/2832227582 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b54003c00 con 0x5565992901b0 2024-04-25T16:06:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.921+0000 7f6b42ffd640 1 -- 172.21.15.31:0/2832227582 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6b54049420 con 0x5565992901b0 2024-04-25T16:06:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.921+0000 7f6b70791740 10 client.16725 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.921+0000 7f6b70791740 10 client.16725 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.921+0000 7f6b70791740 20 client.16725 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.921+0000 7f6b70791740 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5565992241c0 0x5565996d4e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.922+0000 7f6b70791740 4 client.16725 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:06:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.922+0000 7f6b70791740 1 -- 172.21.15.31:0/2832227582 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f6b5801e040 con 0x5565992241c0 2024-04-25T16:06:43.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.924+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5565992241c0 0x5565996d4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:43.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.935+0000 7f6b62ffd640 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5565992241c0 0x5565996d4e20 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97c928114fa8f0cc server_cookie=e44e0bf1eeafc4a0 in_seq=0 out_seq=0 2024-04-25T16:06:43.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:43.935+0000 7f6b42ffd640 10 client.16725 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:43.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:06:43.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:06:43.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:06:44.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:06:44.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:06:44.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:06:44.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:06:44.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:06:44.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:06:45.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:44 smithi063 ceph-mon[32413]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 939 MiB/s wr, 2.87k op/s 2024-04-25T16:06:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:44 smithi186 ceph-mon[33258]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 939 MiB/s wr, 2.87k op/s 2024-04-25T16:06:45.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:44 smithi031 ceph-mon[29140]: pgmap v514: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 939 MiB/s wr, 2.87k op/s 2024-04-25T16:06:47.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:46 smithi063 ceph-mon[32413]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 731 MiB/s wr, 2.32k op/s 2024-04-25T16:06:47.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:46 smithi186 ceph-mon[33258]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 731 MiB/s wr, 2.32k op/s 2024-04-25T16:06:47.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:46 smithi031 ceph-mon[29140]: pgmap v515: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 731 MiB/s wr, 2.32k op/s 2024-04-25T16:06:49.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:49 smithi063 ceph-mon[32413]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 909 MiB/s wr, 2.47k op/s 2024-04-25T16:06:49.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:49 smithi186 ceph-mon[33258]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 909 MiB/s wr, 2.47k op/s 2024-04-25T16:06:49.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:49 smithi031 ceph-mon[29140]: pgmap v516: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 909 MiB/s wr, 2.47k op/s 2024-04-25T16:06:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.497+0000 7f6b42ffd640 1 -- 172.21.15.31:0/2832227582 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f6b5801e040 con 0x5565992241c0 2024-04-25T16:06:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.498+0000 7f6b42ffd640 10 client.16725 handle_command_reply: tid=0 2024-04-25T16:06:49.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.498+0000 7f6b70791740 1 client.16725 shutdown 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.498+0000 7f6b70791740 1 -- 172.21.15.31:0/2832227582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 msgr2=0x5565996aee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.498+0000 7f6b70791740 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x5565996aee60 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6b54009850 tx=0x7f6b54005fd0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.498+0000 7f6b70791740 1 -- 172.21.15.31:0/2832227582 shutdown_connections 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.499+0000 7f6b70791740 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5565992241c0 0x5565996d4e20 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.499+0000 7f6b70791740 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5565992901b0 0x5565996aee60 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.499+0000 7f6b70791740 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55659926f4d0 0x55659963f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.499+0000 7f6b70791740 1 --2- 172.21.15.31:0/2832227582 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5565992672f0 0x55659963edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.499+0000 7f6b70791740 1 -- 172.21.15.31:0/2832227582 >> 172.21.15.31:0/2832227582 conn(0x556599284300 msgr2=0x556599630a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.499+0000 7f6b42ffd640 0 client.16725 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.499+0000 7f6b70791740 1 -- 172.21.15.31:0/2832227582 shutdown_connections 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.499+0000 7f6b70791740 1 -- 172.21.15.31:0/2832227582 wait complete. 2024-04-25T16:06:49.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.499+0000 7f6b70791740 20 client.16725 trim_cache size 0 max 16384 2024-04-25T16:06:49.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:06:49.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:06:49.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:06:49.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:06:49.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.508+0000 7f6b63fff640 1 -- 172.21.15.31:0/1976761772 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b4c0afcf0 msgr2=0x7f6b4c0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.508+0000 7f6b63fff640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b4c0afcf0 0x7f6b4c0b2200 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6b540096c0 tx=0x7f6b540056d0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.508+0000 7f6b63fff640 1 -- 172.21.15.31:0/1976761772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b6410b190 msgr2=0x7f6b6411a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.508+0000 7f6b63fff640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b6410b190 0x7f6b6411a200 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c00ba50 tx=0x7f6b5c00bf10 comp rx=0 tx=0).stop 2024-04-25T16:06:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.509+0000 7f6b63fff640 1 -- 172.21.15.31:0/1976761772 shutdown_connections 2024-04-25T16:06:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.509+0000 7f6b63fff640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f6b4c0afcf0 0x7f6b4c0b2200 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.509+0000 7f6b63fff640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6b6410feb0 0x7f6b6411b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.509+0000 7f6b63fff640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6b6410b190 0x7f6b6411a200 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.509+0000 7f6b63fff640 1 --2- 172.21.15.31:0/1976761772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b6410a340 0x7f6b64119cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:49.512 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.509+0000 7f6b63fff640 1 -- 172.21.15.31:0/1976761772 >> 172.21.15.31:0/1976761772 conn(0x7f6b64070330 msgr2=0x7f6b640727a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.513+0000 7f6b63fff640 1 -- 172.21.15.31:0/1976761772 shutdown_connections 2024-04-25T16:06:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:06:49.513+0000 7f6b63fff640 1 -- 172.21.15.31:0/1976761772 wait complete. 2024-04-25T16:06:51.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:51 smithi063 ceph-mon[32413]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 797 MiB/s wr, 2.05k op/s 2024-04-25T16:06:51.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:51 smithi186 ceph-mon[33258]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 797 MiB/s wr, 2.05k op/s 2024-04-25T16:06:51.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:51 smithi031 ceph-mon[29140]: pgmap v517: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 797 MiB/s wr, 2.05k op/s 2024-04-25T16:06:53.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:53 smithi063 ceph-mon[32413]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 747 MiB/s wr, 1.75k op/s 2024-04-25T16:06:53.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:53 smithi186 ceph-mon[33258]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 747 MiB/s wr, 1.75k op/s 2024-04-25T16:06:53.467 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:53 smithi031 ceph-mon[29140]: pgmap v518: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 747 MiB/s wr, 1.75k op/s 2024-04-25T16:06:55.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:55 smithi063 ceph-mon[32413]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 950 MiB/s wr, 2.17k op/s 2024-04-25T16:06:55.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:55 smithi186 ceph-mon[33258]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 950 MiB/s wr, 2.17k op/s 2024-04-25T16:06:55.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:55 smithi031 ceph-mon[29140]: pgmap v519: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 950 MiB/s wr, 2.17k op/s 2024-04-25T16:06:56.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:06:56.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:06:56.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.634+0000 7fd1effff640 1 -- 172.21.15.31:0/1822746136 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f40318a0 con 0x7fd1f8109fc0 2024-04-25T16:06:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.635+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/1822746136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1f8109fc0 msgr2=0x7fd1f810a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.635+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/1822746136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1f8109fc0 0x7fd1f810a3d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4009930 tx=0x7fd1f402a440 comp rx=0 tx=0).stop 2024-04-25T16:06:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.635+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/1822746136 shutdown_connections 2024-04-25T16:06:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.635+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/1822746136 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1f81120c0 0x7fd1f81144e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.635+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/1822746136 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1f810f760 0x7fd1f8111b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.635+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/1822746136 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1f8109fc0 0x7fd1f810a3d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.636+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/1822746136 >> 172.21.15.31:0/1822746136 conn(0x7fd1f8070400 msgr2=0x7fd1f8072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.637+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/1822746136 shutdown_connections 2024-04-25T16:06:56.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.637+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/1822746136 wait complete. 2024-04-25T16:06:56.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.639+0000 7fd1fe5d2640 1 Processor -- start 2024-04-25T16:06:56.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.639+0000 7fd1fe5d2640 1 -- start start 2024-04-25T16:06:56.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.639+0000 7fd1fe5d2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1f8109fc0 0x7fd1f81a8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.639+0000 7fd1fe5d2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1f810f760 0x7fd1f81a9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.639+0000 7fd1fe5d2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1f81120c0 0x7fd1f81b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.639+0000 7fd1fe5d2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f810d010 con 0x7fd1f8109fc0 2024-04-25T16:06:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.639+0000 7fd1fe5d2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f810ce90 con 0x7fd1f81120c0 2024-04-25T16:06:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.639+0000 7fd1fe5d2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f810d190 con 0x7fd1f810f760 2024-04-25T16:06:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.640+0000 7fd1fd5d0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1f8109fc0 0x7fd1f81a8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.640+0000 7fd1fd5d0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1f8109fc0 0x7fd1f81a8df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36816/0 (socket says 172.21.15.31:36816) 2024-04-25T16:06:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.640+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/225949406 learned_addr learned my addr 172.21.15.31:0/225949406 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:56.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.640+0000 7fd1fddd1640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1f81120c0 0x7fd1f81b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.642+0000 7fd1fcdcf640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1f810f760 0x7fd1f81a9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.642+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/225949406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1f810f760 msgr2=0x7fd1f81a9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:56.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.642+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1f810f760 0x7fd1f81a9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.642+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/225949406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1f81120c0 msgr2=0x7fd1f81b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:56.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.642+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1f81120c0 0x7fd1f81b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.642+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/225949406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1f81b0930 con 0x7fd1f8109fc0 2024-04-25T16:06:56.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.650+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1f8109fc0 0x7fd1f81a8df0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4031440 tx=0x7fd1f4003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.652+0000 7fd1ee7fc640 1 -- 172.21.15.31:0/225949406 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f403f070 con 0x7fd1f8109fc0 2024-04-25T16:06:56.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.652+0000 7fd1ee7fc640 1 -- 172.21.15.31:0/225949406 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f403b460 con 0x7fd1f8109fc0 2024-04-25T16:06:56.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.652+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/225949406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1f81b0bc0 con 0x7fd1f8109fc0 2024-04-25T16:06:56.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.652+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/225949406 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1f81b10b0 con 0x7fd1f8109fc0 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.660+0000 7fd2067be740 1 Processor -- start 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.661+0000 7fd2067be740 1 -- start start 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.661+0000 7fd2067be740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4da390 0x561dae87bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.661+0000 7fd2067be740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae87c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.661+0000 7fd2067be740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae87ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.661+0000 7fd2067be740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561dae7e0d80 con 0x561dae4b96b0 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.661+0000 7fd2067be740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561dae79a8b0 con 0x561dae4da390 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.661+0000 7fd2067be740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561dae7dcd40 con 0x561dae4b14d0 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.662+0000 7fd1fd5d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae87c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.662+0000 7fd1fd5d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae87c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39078/0 (socket says 172.21.15.31:39078) 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.662+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/3051492982 learned_addr learned my addr 172.21.15.31:0/3051492982 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.662+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/3051492982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4da390 msgr2=0x561dae87bca0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:56.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.662+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/3051492982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4da390 0x561dae87bca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.663+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/3051492982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 msgr2=0x561dae87ceb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:06:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.663+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/3051492982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae87ceb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.663+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/3051492982 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dae76ba40 con 0x561dae4b14d0 2024-04-25T16:06:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.664+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/3051492982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae87c220 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd1f41065f0 tx=0x7fd1f41066e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=489d9c7524a069e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.664+0000 7fd1ee7fc640 1 -- 172.21.15.31:0/225949406 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4035030 con 0x7fd1f8109fc0 2024-04-25T16:06:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.664+0000 7fd1ee7fc640 1 -- 172.21.15.31:0/225949406 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd1f4042070 con 0x7fd1f8109fc0 2024-04-25T16:06:56.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.665+0000 7fd1ee7fc640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd1e40afcf0 0x7fd1e40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.666+0000 7fd1e37fe640 1 -- 172.21.15.31:0/3051492982 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4100ce0 con 0x561dae4b14d0 2024-04-25T16:06:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.666+0000 7fd1e37fe640 1 -- 172.21.15.31:0/3051492982 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f410d030 con 0x561dae4b14d0 2024-04-25T16:06:56.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.666+0000 7fd1e37fe640 1 -- 172.21.15.31:0/3051492982 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f41118d0 con 0x561dae4b14d0 2024-04-25T16:06:56.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.668+0000 7fd2067be740 1 -- 172.21.15.31:0/3051492982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 msgr2=0x561dae87c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.668+0000 7fd2067be740 1 --2- 172.21.15.31:0/3051492982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae87c220 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd1f41065f0 tx=0x7fd1f41066e0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.669+0000 7fd2067be740 1 -- 172.21.15.31:0/3051492982 shutdown_connections 2024-04-25T16:06:56.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.669+0000 7fd2067be740 1 --2- 172.21.15.31:0/3051492982 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae87ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.669+0000 7fd2067be740 1 --2- 172.21.15.31:0/3051492982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae87c220 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.669+0000 7fd2067be740 1 --2- 172.21.15.31:0/3051492982 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4da390 0x561dae87bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.669+0000 7fd2067be740 1 -- 172.21.15.31:0/3051492982 >> 172.21.15.31:0/3051492982 conn(0x561dae4ce4e0 msgr2=0x561dae8774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.670+0000 7fd2067be740 1 -- 172.21.15.31:0/3051492982 shutdown_connections 2024-04-25T16:06:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.670+0000 7fd2067be740 1 -- 172.21.15.31:0/3051492982 wait complete. 2024-04-25T16:06:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.670+0000 7fd1ee7fc640 1 -- 172.21.15.31:0/225949406 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd1f41004f0 con 0x7fd1f8109fc0 2024-04-25T16:06:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.670+0000 7fd2067be740 1 -- start start 2024-04-25T16:06:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.670+0000 7fd2067be740 1 Processor -- start 2024-04-25T16:06:56.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.671+0000 7fd1fddd1640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd1e40afcf0 0x7fd1e40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd2067be740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 0x561dae885920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd2067be740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae885e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd2067be740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4da390 0x561dae88cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd2067be740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561dae87d780 con 0x561dae4b96b0 2024-04-25T16:06:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd2067be740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561dae767d10 con 0x561dae4b14d0 2024-04-25T16:06:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd2067be740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561dae87d900 con 0x561dae4da390 2024-04-25T16:06:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd1fddd1640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd1e40afcf0 0x7fd1e40b2200 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x561dae87ccf0 tx=0x7fd1f0009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:56.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd2067be740 10 client.0 _ll_register_callbacks cb 0x561dae794210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:56.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd1fcdcf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae885e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd1fcdcf640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae885e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36818/0 (socket says 172.21.15.31:36818) 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.672+0000 7fd1fcdcf640 1 -- 172.21.15.31:0/2805388455 learned_addr learned my addr 172.21.15.31:0/2805388455 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.673+0000 7fd1fcdcf640 1 -- 172.21.15.31:0/2805388455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4da390 msgr2=0x561dae88cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.673+0000 7fd1fcdcf640 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4da390 0x561dae88cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.673+0000 7fd1fcdcf640 1 -- 172.21.15.31:0/2805388455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 msgr2=0x561dae885920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.673+0000 7fd1fcdcf640 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 0x561dae885920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.673+0000 7fd1fcdcf640 1 -- 172.21.15.31:0/2805388455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dae5bc7d0 con 0x561dae4b96b0 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.673+0000 7fd1fcdcf640 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae885e60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8009fa0 tx=0x7fd1e800e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada5e92d984a6ef6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.674+0000 7fd1e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.674+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/2805388455 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8016070 con 0x561dae4b96b0 2024-04-25T16:06:56.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.675+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/2805388455 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd1e8005220 con 0x561dae4b96b0 2024-04-25T16:06:56.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.678+0000 7fd2067be740 1 -- 172.21.15.31:0/2805388455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1f410d030 con 0x561dae4b96b0 2024-04-25T16:06:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.681+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/2805388455 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd1e801f050 con 0x561dae4b96b0 2024-04-25T16:06:56.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.681+0000 7fd2067be740 10 client.16734 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:56.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.681+0000 7fd2067be740 1 -- 172.21.15.31:0/2805388455 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561dae4f23f0 con 0x561dae4b96b0 2024-04-25T16:06:56.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.685+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/2805388455 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f410d030 con 0x561dae4b96b0 2024-04-25T16:06:56.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.688+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/2805388455 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd1e800f040 con 0x561dae4b96b0 2024-04-25T16:06:56.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.689+0000 7fd2067be740 10 client.16734 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:56.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.689+0000 7fd2067be740 10 client.16734 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:56.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.689+0000 7fd2067be740 20 client.16734 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:56.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.689+0000 7fd2067be740 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561dae498f60 0x561dae8933b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.689+0000 7fd2067be740 4 client.16734 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:56.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.689+0000 7fd2067be740 1 -- 172.21.15.31:0/2805388455 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dae7ee990 con 0x561dae498f60 2024-04-25T16:06:56.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.696+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561dae498f60 0x561dae8933b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.717+0000 7fd1e0ff9640 10 client.16734 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:56.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.717+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561dae498f60 0x561dae8933b0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62a8fd99fea0c9f6 server_cookie=d0453d342258b327 in_seq=0 out_seq=0 2024-04-25T16:06:56.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.727+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/2805388455 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561dae7ee990 con 0x561dae498f60 2024-04-25T16:06:56.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.727+0000 7fd1e0ff9640 10 client.16734 handle_command_reply: tid=0 2024-04-25T16:06:56.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.732+0000 7fd2067be740 1 client.16734 shutdown 2024-04-25T16:06:56.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.735+0000 7fd2067be740 1 -- 172.21.15.31:0/2805388455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 msgr2=0x561dae885e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:56.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.735+0000 7fd2067be740 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae885e60 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8009fa0 tx=0x7fd1e800e780 comp rx=0 tx=0).stop 2024-04-25T16:06:56.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.736+0000 7fd2067be740 1 -- 172.21.15.31:0/2805388455 shutdown_connections 2024-04-25T16:06:56.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.736+0000 7fd2067be740 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561dae498f60 0x561dae8933b0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.736+0000 7fd2067be740 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4da390 0x561dae88cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.736+0000 7fd2067be740 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae885e60 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.736+0000 7fd2067be740 1 --2- 172.21.15.31:0/2805388455 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 0x561dae885920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.736+0000 7fd2067be740 1 -- 172.21.15.31:0/2805388455 >> 172.21.15.31:0/2805388455 conn(0x561dae4ce4e0 msgr2=0x561dae87a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:56.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.736+0000 7fd1e0ff9640 0 client.16734 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:56.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.743+0000 7fd2067be740 1 -- 172.21.15.31:0/2805388455 shutdown_connections 2024-04-25T16:06:56.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.744+0000 7fd2067be740 1 -- 172.21.15.31:0/2805388455 wait complete. 2024-04-25T16:06:56.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.744+0000 7fd2067be740 20 client.16734 trim_cache size 0 max 16384 2024-04-25T16:06:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.839+0000 7fd2067be740 1 Processor -- start 2024-04-25T16:06:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.839+0000 7fd2067be740 1 -- start start 2024-04-25T16:06:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.839+0000 7fd2067be740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 0x561dae884990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.839+0000 7fd2067be740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b96b0 0x561dae87a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.839+0000 7fd2067be740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4da390 0x561dae88cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.840+0000 7fd2067be740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81b9d70 con 0x561dae4da390 2024-04-25T16:06:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.840+0000 7fd2067be740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81bb8b0 con 0x561dae4b14d0 2024-04-25T16:06:56.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.840+0000 7fd2067be740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81ba440 con 0x561dae4b96b0 2024-04-25T16:06:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.840+0000 7fd1fd5d0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 0x561dae884990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.840+0000 7fd1fd5d0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 0x561dae884990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39988/0 (socket says 172.21.15.31:39988) 2024-04-25T16:06:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.840+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/2385517040 learned_addr learned my addr 172.21.15.31:0/2385517040 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.841+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/2385517040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b96b0 msgr2=0x561dae87a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:06:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.841+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/2385517040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b96b0 0x561dae87a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.841+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/2385517040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4da390 msgr2=0x561dae88cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.841+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/2385517040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4da390 0x561dae88cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.841+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/2385517040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dae4b20f0 con 0x561dae4b14d0 2024-04-25T16:06:56.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.843+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/2385517040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 0x561dae884990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4000c00 tx=0x7fd1f41066e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=878eb631b67e5566 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:56.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.854+0000 7fd1e2ffd640 1 -- 172.21.15.31:0/2385517040 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f4049040 con 0x561dae4b14d0 2024-04-25T16:06:56.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.857+0000 7fd1e2ffd640 1 -- 172.21.15.31:0/2385517040 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f410d070 con 0x561dae4b14d0 2024-04-25T16:06:56.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.863+0000 7fd2067be740 1 -- 172.21.15.31:0/2385517040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 msgr2=0x561dae884990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:56.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.863+0000 7fd2067be740 1 --2- 172.21.15.31:0/2385517040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 0x561dae884990 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4000c00 tx=0x7fd1f41066e0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.869+0000 7fd2067be740 1 -- 172.21.15.31:0/2385517040 shutdown_connections 2024-04-25T16:06:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.869+0000 7fd2067be740 1 --2- 172.21.15.31:0/2385517040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4da390 0x561dae88cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.869+0000 7fd2067be740 1 --2- 172.21.15.31:0/2385517040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b96b0 0x561dae87a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.869+0000 7fd2067be740 1 --2- 172.21.15.31:0/2385517040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4b14d0 0x561dae884990 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.869+0000 7fd2067be740 1 -- 172.21.15.31:0/2385517040 >> 172.21.15.31:0/2385517040 conn(0x561dae4ce4e0 msgr2=0x561dae87a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:56.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.874+0000 7fd2067be740 1 -- 172.21.15.31:0/2385517040 shutdown_connections 2024-04-25T16:06:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.874+0000 7fd2067be740 1 -- 172.21.15.31:0/2385517040 wait complete. 2024-04-25T16:06:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.875+0000 7fd2067be740 1 -- start start 2024-04-25T16:06:56.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.876+0000 7fd2067be740 1 Processor -- start 2024-04-25T16:06:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.877+0000 7fd2067be740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae88afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.877+0000 7fd2067be740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae88b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.878+0000 7fd2067be740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4da390 0x561dae88c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.878+0000 7fd2067be740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81bb1e0 con 0x561dae4b96b0 2024-04-25T16:06:56.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.878+0000 7fd2067be740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81bab10 con 0x561dae4da390 2024-04-25T16:06:56.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.878+0000 7fd2067be740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1f81b7e00 con 0x561dae4b14d0 2024-04-25T16:06:56.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.878+0000 7fd2067be740 10 client.0 _ll_register_callbacks cb 0x7fd1e80092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.880+0000 7fd1fd5d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae88afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.880+0000 7fd1fd5d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae88afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39100/0 (socket says 172.21.15.31:39100) 2024-04-25T16:06:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.880+0000 7fd1fd5d0640 1 -- 172.21.15.31:0/69723345 learned_addr learned my addr 172.21.15.31:0/69723345 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:56.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.880+0000 7fd1fcdcf640 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae88b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.883+0000 7fd1fcdcf640 1 -- 172.21.15.31:0/69723345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 msgr2=0x561dae88afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:56.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.883+0000 7fd1fcdcf640 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae88afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.883+0000 7fd1fcdcf640 1 -- 172.21.15.31:0/69723345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4da390 msgr2=0x561dae88c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:56.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.883+0000 7fd1fcdcf640 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4da390 0x561dae88c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.883+0000 7fd1fcdcf640 1 -- 172.21.15.31:0/69723345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dae7a14b0 con 0x561dae4b96b0 2024-04-25T16:06:56.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.886+0000 7fd1fcdcf640 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae88b520 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80118f0 tx=0x7fd1e8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f960ed3c2f6e5aed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:56.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.886+0000 7fd1e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:56.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.887+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/69723345 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1e8016030 con 0x561dae4b96b0 2024-04-25T16:06:56.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.888+0000 7fd2067be740 1 -- 172.21.15.31:0/69723345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1f410d070 con 0x561dae4b96b0 2024-04-25T16:06:56.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.888+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/69723345 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd1e80054d0 con 0x561dae4b96b0 2024-04-25T16:06:56.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.890+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/69723345 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd1e8004620 con 0x561dae4b96b0 2024-04-25T16:06:56.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.894+0000 7fd2067be740 10 client.16737 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:56.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.894+0000 7fd2067be740 1 -- 172.21.15.31:0/69723345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561dae8fb7c0 con 0x561dae4b96b0 2024-04-25T16:06:56.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.897+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/69723345 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd1f410d070 con 0x561dae4b96b0 2024-04-25T16:06:56.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.897+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/69723345 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd1e8024890 con 0x561dae4b96b0 2024-04-25T16:06:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.897+0000 7fd2067be740 10 client.16737 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.897+0000 7fd2067be740 10 client.16737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:06:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.897+0000 7fd2067be740 20 client.16737 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.897+0000 7fd2067be740 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561dae46e3a0 0x561dae8fe1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.897+0000 7fd2067be740 4 client.16737 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:06:56.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.897+0000 7fd2067be740 1 -- 172.21.15.31:0/69723345 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fd1f4103d70 con 0x561dae46e3a0 2024-04-25T16:06:56.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.900+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561dae46e3a0 0x561dae8fe1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:56.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.918+0000 7fd1fd5d0640 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561dae46e3a0 0x561dae8fe1e0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90b49ca6c0611d40 server_cookie=90f0fb7436576bbf in_seq=0 out_seq=0 2024-04-25T16:06:56.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.921+0000 7fd1e0ff9640 10 client.16737 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.930+0000 7fd1e0ff9640 1 -- 172.21.15.31:0/69723345 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fd1f4103d70 con 0x561dae46e3a0 2024-04-25T16:06:56.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.930+0000 7fd1e0ff9640 10 client.16737 handle_command_reply: tid=0 2024-04-25T16:06:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.930+0000 7fd2067be740 1 client.16737 shutdown 2024-04-25T16:06:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 -- 172.21.15.31:0/69723345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 msgr2=0x561dae88b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae88b520 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fd1e80118f0 tx=0x7fd1e8011e90 comp rx=0 tx=0).stop 2024-04-25T16:06:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 -- 172.21.15.31:0/69723345 shutdown_connections 2024-04-25T16:06:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561dae46e3a0 0x561dae8fe1e0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561dae4da390 0x561dae88c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561dae4b96b0 0x561dae88b520 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 --2- 172.21.15.31:0/69723345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561dae4b14d0 0x561dae88afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 -- 172.21.15.31:0/69723345 >> 172.21.15.31:0/69723345 conn(0x561dae4ce4e0 msgr2=0x561dae8846f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:56.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd1e0ff9640 0 client.16737 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:06:56.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 -- 172.21.15.31:0/69723345 shutdown_connections 2024-04-25T16:06:56.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.931+0000 7fd2067be740 1 -- 172.21.15.31:0/69723345 wait complete. 2024-04-25T16:06:56.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.932+0000 7fd2067be740 20 client.16737 trim_cache size 0 max 16384 2024-04-25T16:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:06:56.934 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:06:56.935 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:06:56.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.938+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd1e40afcf0 msgr2=0x7fd1e40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.938+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd1e40afcf0 0x7fd1e40b2200 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x561dae87ccf0 tx=0x7fd1f0009600 comp rx=0 tx=0).stop 2024-04-25T16:06:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.939+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1f8109fc0 msgr2=0x7fd1f81a8df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:56.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.939+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1f8109fc0 0x7fd1f81a8df0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fd1f4031440 tx=0x7fd1f4003d50 comp rx=0 tx=0).stop 2024-04-25T16:06:56.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.939+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/225949406 shutdown_connections 2024-04-25T16:06:56.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.939+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd1e40afcf0 0x7fd1e40b2200 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.939+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1f81120c0 0x7fd1f81b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.939+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1f810f760 0x7fd1f81a9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.939+0000 7fd1fe5d2640 1 --2- 172.21.15.31:0/225949406 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd1f8109fc0 0x7fd1f81a8df0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:56.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.939+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/225949406 >> 172.21.15.31:0/225949406 conn(0x7fd1f8070400 msgr2=0x7fd1f81105a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:56.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.945+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/225949406 shutdown_connections 2024-04-25T16:06:56.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:56.945+0000 7fd1fe5d2640 1 -- 172.21.15.31:0/225949406 wait complete. 2024-04-25T16:06:56.993 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:06:56.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:06:57.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.440+0000 7f41687eb640 1 -- 172.21.15.31:0/3292734823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f416010b760 msgr2=0x7f416010bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:57.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.440+0000 7f41687eb640 1 --2- 172.21.15.31:0/3292734823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f416010b760 0x7f416010bb70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4150009900 tx=0x7f415002a440 comp rx=0 tx=0).stop 2024-04-25T16:06:57.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.441+0000 7f41687eb640 1 -- 172.21.15.31:0/3292734823 shutdown_connections 2024-04-25T16:06:57.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.441+0000 7f41687eb640 1 --2- 172.21.15.31:0/3292734823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4160110460 0x7f4160112880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:57.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.441+0000 7f41687eb640 1 --2- 172.21.15.31:0/3292734823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f416010b760 0x7f416010bb70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:57.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.441+0000 7f41687eb640 1 --2- 172.21.15.31:0/3292734823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416010a520 0x7f416010b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:57.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.441+0000 7f41687eb640 1 -- 172.21.15.31:0/3292734823 >> 172.21.15.31:0/3292734823 conn(0x7f41600704e0 msgr2=0x7f4160072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:57.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.452+0000 7f41687eb640 1 -- 172.21.15.31:0/3292734823 shutdown_connections 2024-04-25T16:06:57.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.460+0000 7f41687eb640 1 -- 172.21.15.31:0/3292734823 wait complete. 2024-04-25T16:06:57.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.463+0000 7f41687eb640 1 Processor -- start 2024-04-25T16:06:57.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.464+0000 7f41687eb640 1 -- start start 2024-04-25T16:06:57.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.465+0000 7f41687eb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416010a520 0x7f41601a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:57.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.466+0000 7f41687eb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f416010b760 0x7f41601a9900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:57.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.466+0000 7f41687eb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4160110460 0x7f41601b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:57.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.466+0000 7f41687eb640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f4160115d50 con 0x7f416010a520 2024-04-25T16:06:57.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.466+0000 7f41687eb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f4160115bd0 con 0x7f416010b760 2024-04-25T16:06:57.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.466+0000 7f41687eb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4160115ed0 con 0x7f4160110460 2024-04-25T16:06:57.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.474+0000 7f4166d88640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4160110460 0x7f41601b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:57.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.474+0000 7f4165d86640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f416010b760 0x7f41601a9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:57.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.474+0000 7f4166587640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416010a520 0x7f41601a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:57.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.474+0000 7f4166d88640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4160110460 0x7f41601b0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39122/0 (socket says 172.21.15.31:39122) 2024-04-25T16:06:57.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.474+0000 7f4166d88640 1 -- 172.21.15.31:0/3116401072 learned_addr learned my addr 172.21.15.31:0/3116401072 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:57.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.478+0000 7f4165d86640 1 -- 172.21.15.31:0/3116401072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4160110460 msgr2=0x7f41601b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:57.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.478+0000 7f4165d86640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4160110460 0x7f41601b0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:57.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.478+0000 7f4165d86640 1 -- 172.21.15.31:0/3116401072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416010a520 msgr2=0x7f41601a93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.478+0000 7f4165d86640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416010a520 0x7f41601a93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.478+0000 7f4165d86640 1 -- 172.21.15.31:0/3116401072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41601b0da0 con 0x7f416010b760 2024-04-25T16:06:57.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.479+0000 7f4165d86640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f416010b760 0x7f41601a9900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f415002aa30 tx=0x7f4150003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.480+0000 7f41577fe640 1 -- 172.21.15.31:0/3116401072 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f415003f070 con 0x7f416010b760 2024-04-25T16:06:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.480+0000 7f41577fe640 1 -- 172.21.15.31:0/3116401072 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f415003b460 con 0x7f416010b760 2024-04-25T16:06:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.480+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41601b1030 con 0x7f416010b760 2024-04-25T16:06:57.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.480+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41601b1520 con 0x7f416010b760 2024-04-25T16:06:57.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.480+0000 7f41577fe640 1 -- 172.21.15.31:0/3116401072 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4150035030 con 0x7f416010b760 2024-04-25T16:06:57.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.487+0000 7f41577fe640 1 -- 172.21.15.31:0/3116401072 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4150042070 con 0x7f416010b760 2024-04-25T16:06:57.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.489+0000 7f41577fe640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f414c0afca0 0x7f414c0b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:57.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.489+0000 7f41577fe640 1 -- 172.21.15.31:0/3116401072 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f41501003d0 con 0x7f416010b760 2024-04-25T16:06:57.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.494+0000 7f4166587640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f414c0afca0 0x7f414c0b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:57.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.497+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4144005290 con 0x7f416010b760 2024-04-25T16:06:57.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.511+0000 7f41577fe640 1 -- 172.21.15.31:0/3116401072 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41500c8910 con 0x7f416010b760 2024-04-25T16:06:57.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:57.514+0000 7f4166587640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f414c0afca0 0x7f414c0b21b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f41601aa660 tx=0x7f415800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:57.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:57 smithi063 ceph-mon[32413]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 757 MiB/s wr, 1.37k op/s 2024-04-25T16:06:57.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:57 smithi186 ceph-mon[33258]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 757 MiB/s wr, 1.37k op/s 2024-04-25T16:06:57.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:57 smithi031 ceph-mon[29140]: pgmap v520: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 757 MiB/s wr, 1.37k op/s 2024-04-25T16:06:58.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.168+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4144005580 con 0x7f416010b760 2024-04-25T16:06:58.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.173+0000 7f41577fe640 1 -- 172.21.15.31:0/3116401072 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7f41500ccc10 con 0x7f416010b760 2024-04-25T16:06:58.181 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:06:58.181 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:06:58.182 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:06:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.193+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f414c0afca0 msgr2=0x7f414c0b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.193+0000 7f41687eb640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f414c0afca0 0x7f414c0b21b0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f41601aa660 tx=0x7f415800d040 comp rx=0 tx=0).stop 2024-04-25T16:06:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.194+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f416010b760 msgr2=0x7f41601a9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.194+0000 7f41687eb640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f416010b760 0x7f41601a9900 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f415002aa30 tx=0x7f4150003d50 comp rx=0 tx=0).stop 2024-04-25T16:06:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.194+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 shutdown_connections 2024-04-25T16:06:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.194+0000 7f41687eb640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f414c0afca0 0x7f414c0b21b0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.194+0000 7f41687eb640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4160110460 0x7f41601b0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.194+0000 7f41687eb640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f416010b760 0x7f41601a9900 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.194+0000 7f41687eb640 1 --2- 172.21.15.31:0/3116401072 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f416010a520 0x7f41601a93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.194+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 >> 172.21.15.31:0/3116401072 conn(0x7f41600704e0 msgr2=0x7f4160072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:58.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.195+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 shutdown_connections 2024-04-25T16:06:58.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.200+0000 7f41687eb640 1 -- 172.21.15.31:0/3116401072 wait complete. 2024-04-25T16:06:58.252 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:06:58.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.769+0000 7f2fe02b1640 1 -- 172.21.15.31:0/1754312130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fd8110030 msgr2=0x7f2fd8110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.769+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/1754312130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fd8110030 0x7f2fd8110440 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009ca0 tx=0x7f2fd002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.776+0000 7f2fe02b1640 1 -- 172.21.15.31:0/1754312130 shutdown_connections 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.776+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/1754312130 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fd8110030 0x7f2fd8110440 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.776+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/1754312130 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fd810b310 0x7f2fd810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.776+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/1754312130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd810a420 0x7f2fd810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.776+0000 7f2fe02b1640 1 -- 172.21.15.31:0/1754312130 >> 172.21.15.31:0/1754312130 conn(0x7f2fd80704e0 msgr2=0x7f2fd8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.777+0000 7f2fe02b1640 1 -- 172.21.15.31:0/1754312130 shutdown_connections 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.777+0000 7f2fe02b1640 1 -- 172.21.15.31:0/1754312130 wait complete. 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.778+0000 7f2fe02b1640 1 Processor -- start 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.778+0000 7f2fe02b1640 1 -- start start 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.778+0000 7f2fe02b1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fd810a420 0x7f2fd8075150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.778+0000 7f2fe02b1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fd810b310 0x7f2fd8075690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.778+0000 7f2fe02b1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd8110030 0x7f2fd807c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.778+0000 7f2fe02b1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd811e590 con 0x7f2fd810b310 2024-04-25T16:06:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.778+0000 7f2fe02b1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd811e410 con 0x7f2fd810a420 2024-04-25T16:06:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.778+0000 7f2fe02b1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fd811e710 con 0x7f2fd8110030 2024-04-25T16:06:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.779+0000 7f2fde84e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd8110030 0x7f2fd807c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.779+0000 7f2fde84e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd8110030 0x7f2fd807c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39130/0 (socket says 172.21.15.31:39130) 2024-04-25T16:06:58.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.779+0000 7f2fde84e640 1 -- 172.21.15.31:0/3619955391 learned_addr learned my addr 172.21.15.31:0/3619955391 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.779+0000 7f2fde84e640 1 -- 172.21.15.31:0/3619955391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fd810a420 msgr2=0x7f2fd8075150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.779+0000 7f2fde84e640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fd810a420 0x7f2fd8075150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.779+0000 7f2fde84e640 1 -- 172.21.15.31:0/3619955391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fd810b310 msgr2=0x7f2fd8075690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.779+0000 7f2fde84e640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fd810b310 0x7f2fd8075690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.779+0000 7f2fde84e640 1 -- 172.21.15.31:0/3619955391 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fd807cc40 con 0x7f2fd8110030 2024-04-25T16:06:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.780+0000 7f2fde84e640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd8110030 0x7f2fd807c5d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00098d0 tx=0x7f2fd0003ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:58.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.780+0000 7f2fcf7fe640 1 -- 172.21.15.31:0/3619955391 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0004240 con 0x7f2fd8110030 2024-04-25T16:06:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.780+0000 7f2fe02b1640 1 -- 172.21.15.31:0/3619955391 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fd807ced0 con 0x7f2fd8110030 2024-04-25T16:06:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.780+0000 7f2fe02b1640 1 -- 172.21.15.31:0/3619955391 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fd807d4f0 con 0x7f2fd8110030 2024-04-25T16:06:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.781+0000 7f2fcf7fe640 1 -- 172.21.15.31:0/3619955391 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd002d070 con 0x7f2fd8110030 2024-04-25T16:06:58.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.781+0000 7f2fcf7fe640 1 -- 172.21.15.31:0/3619955391 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0030ca0 con 0x7f2fd8110030 2024-04-25T16:06:58.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.785+0000 7f2fcf7fe640 1 -- 172.21.15.31:0/3619955391 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2fd0045480 con 0x7f2fd8110030 2024-04-25T16:06:58.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.797+0000 7f2fcf7fe640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fc40afc30 0x7f2fc40b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.797+0000 7f2fcf7fe640 1 -- 172.21.15.31:0/3619955391 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2fd0100790 con 0x7f2fd8110030 2024-04-25T16:06:58.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.799+0000 7f2fde04d640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fc40afc30 0x7f2fc40b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.801+0000 7f2fde04d640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fc40afc30 0x7f2fc40b2140 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8109c60 tx=0x7f2fd4006c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:58.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.801+0000 7f2fe6dfd740 1 Processor -- start 2024-04-25T16:06:58.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.802+0000 7f2fe6dfd740 1 -- start start 2024-04-25T16:06:58.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.802+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c724390 0x55ab3cac6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.802+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.802+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c7036b0 0x55ab3cac7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.803+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ab3ca2ad80 con 0x55ab3c7036b0 2024-04-25T16:06:58.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.803+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ab3c9e4a20 con 0x55ab3c724390 2024-04-25T16:06:58.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.803+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ab3ca26d40 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.806+0000 7f2fde84e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c7036b0 0x55ab3cac7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.806+0000 7f2fde84e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c7036b0 0x55ab3cac7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36864/0 (socket says 172.21.15.31:36864) 2024-04-25T16:06:58.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.806+0000 7f2fde84e640 1 -- 172.21.15.31:0/691583522 learned_addr learned my addr 172.21.15.31:0/691583522 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:58.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.809+0000 7f2fdd84c640 1 --2- 172.21.15.31:0/691583522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.809+0000 7f2fde84e640 1 -- 172.21.15.31:0/691583522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c6fb4d0 msgr2=0x55ab3cac6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:58.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.809+0000 7f2fde84e640 1 --2- 172.21.15.31:0/691583522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.809+0000 7f2fde04d640 1 --2- 172.21.15.31:0/691583522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c724390 0x55ab3cac6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.809+0000 7f2fde84e640 1 -- 172.21.15.31:0/691583522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c724390 msgr2=0x55ab3cac6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:58.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.809+0000 7f2fde84e640 1 --2- 172.21.15.31:0/691583522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c724390 0x55ab3cac6810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.809+0000 7f2fde84e640 1 -- 172.21.15.31:0/691583522 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab3c9efc30 con 0x55ab3c7036b0 2024-04-25T16:06:58.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.813+0000 7f2fde84e640 1 --2- 172.21.15.31:0/691583522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c7036b0 0x55ab3cac7a20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00c5e90 tx=0x7f2fd00c8570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32adb64e8a819579 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:58.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.818+0000 7f2fccbfe640 1 -- 172.21.15.31:0/691583522 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd00e1070 con 0x55ab3c7036b0 2024-04-25T16:06:58.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.818+0000 7f2fccbfe640 1 -- 172.21.15.31:0/691583522 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd0036030 con 0x55ab3c7036b0 2024-04-25T16:06:58.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.820+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/691583522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c7036b0 msgr2=0x55ab3cac7a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.820+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/691583522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c7036b0 0x55ab3cac7a20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00c5e90 tx=0x7f2fd00c8570 comp rx=0 tx=0).stop 2024-04-25T16:06:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.822+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/691583522 shutdown_connections 2024-04-25T16:06:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.822+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/691583522 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c7036b0 0x55ab3cac7a20 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.822+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/691583522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.822+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/691583522 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c724390 0x55ab3cac6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.822+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/691583522 >> 172.21.15.31:0/691583522 conn(0x55ab3c7184e0 msgr2=0x55ab3cac2020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:58.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.826+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/691583522 shutdown_connections 2024-04-25T16:06:58.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.826+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/691583522 wait complete. 2024-04-25T16:06:58.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.826+0000 7f2fe6dfd740 1 -- start start 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.827+0000 7f2fe6dfd740 1 Processor -- start 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.828+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cad02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.828+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 0x55ab3cad0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.828+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cad7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.828+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ab3cac82f0 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.828+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ab3c9b1dc0 con 0x55ab3c7036b0 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.828+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ab3cac8470 con 0x55ab3c724390 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.828+0000 7f2fe6dfd740 10 client.0 _ll_register_callbacks cb 0x55ab3c9cd600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.828+0000 7f2fde04d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cad02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.829+0000 7f2fde04d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cad02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36878/0 (socket says 172.21.15.31:36878) 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.829+0000 7f2fde04d640 1 -- 172.21.15.31:0/3551781854 learned_addr learned my addr 172.21.15.31:0/3551781854 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.831+0000 7f2fde04d640 1 -- 172.21.15.31:0/3551781854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 msgr2=0x55ab3cad7770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.831+0000 7f2fde04d640 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cad7770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.831+0000 7f2fde04d640 1 -- 172.21.15.31:0/3551781854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 msgr2=0x55ab3cad0810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:58.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.831+0000 7f2fde04d640 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 0x55ab3cad0810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.831+0000 7f2fde04d640 1 -- 172.21.15.31:0/3551781854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab3c73c3f0 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.834+0000 7f2fde04d640 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cad02d0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4013ee0 tx=0x7f2fd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ed8e4a7e7b67f0b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:58.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.840+0000 7f2fc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:58.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.840+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/3551781854 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4023060 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.840+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3551781854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fd0036030 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.840+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/3551781854 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2fd40105b0 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.841+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/3551781854 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2fd40107b0 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.845+0000 7f2fe6dfd740 10 client.16749 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:58.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.845+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3551781854 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ab3c8067d0 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.847+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/3551781854 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd0036030 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.847+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/3551781854 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f2fd4012070 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.847+0000 7f2fe6dfd740 10 client.16749 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:58.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.847+0000 7f2fe6dfd740 10 client.16749 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:06:58.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.847+0000 7f2fe6dfd740 20 client.16749 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:58.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.847+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55ab3c6e2f60 0x55ab3cade5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.847+0000 7f2fe6dfd740 4 client.16749 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:58.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.847+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3551781854 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ab3ca38990 con 0x55ab3c6e2f60 2024-04-25T16:06:58.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.856+0000 7f2fdd84c640 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55ab3c6e2f60 0x55ab3cade5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.877+0000 7f2fdd84c640 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55ab3c6e2f60 0x55ab3cade5b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e0eae4e13709c9e server_cookie=d3362e7b47760b57 in_seq=0 out_seq=0 2024-04-25T16:06:58.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.878+0000 7f2fc1ffb640 10 client.16749 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.882+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/3551781854 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ab3ca38990 con 0x55ab3c6e2f60 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.882+0000 7f2fc1ffb640 10 client.16749 handle_command_reply: tid=0 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.883+0000 7f2fe6dfd740 1 client.16749 shutdown 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.883+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3551781854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 msgr2=0x55ab3cad02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.883+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cad02d0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f2fd4013ee0 tx=0x7f2fd4004750 comp rx=0 tx=0).stop 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.885+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3551781854 shutdown_connections 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.885+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55ab3c6e2f60 0x55ab3cade5b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.885+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cad7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.885+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 0x55ab3cad0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.885+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3551781854 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cad02d0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.885+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3551781854 >> 172.21.15.31:0/3551781854 conn(0x55ab3c7184e0 msgr2=0x55ab3cac4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:58.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.885+0000 7f2fc1ffb640 0 client.16749 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:06:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.888+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3551781854 shutdown_connections 2024-04-25T16:06:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.888+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3551781854 wait complete. 2024-04-25T16:06:58.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.888+0000 7f2fe6dfd740 20 client.16749 trim_cache size 0 max 16384 2024-04-25T16:06:58.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:58 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3116401072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:06:58.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.938+0000 7f2fe6dfd740 1 Processor -- start 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.939+0000 7f2fe6dfd740 1 -- start start 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.940+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.940+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 0x55ab3cad7700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.940+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cac0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.940+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc00e520 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.940+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc010060 con 0x55ab3c7036b0 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.940+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc00ebf0 con 0x55ab3c724390 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.940+0000 7f2fde04d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.940+0000 7f2fde04d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36882/0 (socket says 172.21.15.31:36882) 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.940+0000 7f2fde04d640 1 -- 172.21.15.31:0/3965294029 learned_addr learned my addr 172.21.15.31:0/3965294029 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.941+0000 7f2fde04d640 1 -- 172.21.15.31:0/3965294029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 msgr2=0x55ab3cac0610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.941+0000 7f2fde04d640 1 --2- 172.21.15.31:0/3965294029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cac0610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.941+0000 7f2fde04d640 1 -- 172.21.15.31:0/3965294029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 msgr2=0x55ab3cad7700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:06:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.941+0000 7f2fde04d640 1 --2- 172.21.15.31:0/3965294029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 0x55ab3cad7700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.941+0000 7f2fde04d640 1 -- 172.21.15.31:0/3965294029 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ab3cacbbe0 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.941+0000 7f2fde04d640 1 --2- 172.21.15.31:0/3965294029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac2160 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2fd401ebe0 tx=0x7f2fd4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf9a4a97f6eb1a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.942+0000 7f2fc3fff640 1 -- 172.21.15.31:0/3965294029 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4037070 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.942+0000 7f2fc3fff640 1 -- 172.21.15.31:0/3965294029 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd401ee50 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.942+0000 7f2fc3fff640 1 -- 172.21.15.31:0/3965294029 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd402e070 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.943+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3965294029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 msgr2=0x55ab3cac2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.943+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3965294029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac2160 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2fd401ebe0 tx=0x7f2fd4010740 comp rx=0 tx=0).stop 2024-04-25T16:06:58.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.945+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3965294029 shutdown_connections 2024-04-25T16:06:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.945+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3965294029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cac0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.945+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3965294029 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 0x55ab3cad7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.945+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/3965294029 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cac2160 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.945+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3965294029 >> 172.21.15.31:0/3965294029 conn(0x55ab3c7184e0 msgr2=0x55ab3cac1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:58.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.949+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3965294029 shutdown_connections 2024-04-25T16:06:58.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:58 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3116401072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:06:58.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.949+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/3965294029 wait complete. 2024-04-25T16:06:58.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.953+0000 7f2fe6dfd740 1 -- start start 2024-04-25T16:06:58.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.954+0000 7f2fe6dfd740 1 Processor -- start 2024-04-25T16:06:58.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.956+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cad4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.956+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 0x55ab3cad4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.956+0000 7f2fe6dfd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cb44950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.957+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc00f990 con 0x55ab3c6fb4d0 2024-04-25T16:06:58.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.957+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc00f2c0 con 0x55ab3c7036b0 2024-04-25T16:06:58.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.957+0000 7f2fe6dfd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2fbc00c500 con 0x55ab3c724390 2024-04-25T16:06:58.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.957+0000 7f2fe6dfd740 10 client.0 _ll_register_callbacks cb 0x7f2fd4013b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:58.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.958+0000 7f2fde84e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cb44950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.958+0000 7f2fde84e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cb44950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39146/0 (socket says 172.21.15.31:39146) 2024-04-25T16:06:58.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.958+0000 7f2fde84e640 1 -- 172.21.15.31:0/769153791 learned_addr learned my addr 172.21.15.31:0/769153791 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:58.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.960+0000 7f2fde84e640 1 -- 172.21.15.31:0/769153791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 msgr2=0x55ab3cad4c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:58.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.960+0000 7f2fde84e640 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 0x55ab3cad4c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.960+0000 7f2fde84e640 1 -- 172.21.15.31:0/769153791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 msgr2=0x55ab3cad4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:58.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.960+0000 7f2fde84e640 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cad4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.960+0000 7f2fde84e640 1 -- 172.21.15.31:0/769153791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ab3cb44e90 con 0x55ab3c724390 2024-04-25T16:06:58.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.962+0000 7f2fde84e640 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cb44950 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009a20 tx=0x7f2fd0003ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63e583aa52b9bbda server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:58.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.962+0000 7f2fc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:58.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.964+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/769153791 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fd00e1070 con 0x55ab3c724390 2024-04-25T16:06:58.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.964+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/769153791 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2fd00d15f0 con 0x55ab3c724390 2024-04-25T16:06:58.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.965+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/769153791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2fd401ee50 con 0x55ab3c724390 2024-04-25T16:06:58.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.970+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/769153791 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f2fd00d13c0 con 0x55ab3c724390 2024-04-25T16:06:58.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.970+0000 7f2fe6dfd740 10 client.26347 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:58.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.971+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/769153791 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55ab3cb46070 con 0x55ab3c724390 2024-04-25T16:06:58.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.975+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/769153791 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fd401ee50 con 0x55ab3c724390 2024-04-25T16:06:58.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.975+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/769153791 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f2fd004d070 con 0x55ab3c724390 2024-04-25T16:06:58.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.975+0000 7f2fe6dfd740 10 client.26347 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:58.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.975+0000 7f2fe6dfd740 10 client.26347 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:06:58.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.975+0000 7f2fe6dfd740 20 client.26347 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:58.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.975+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55ab3c6b83a0 0x55ab3cb487d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:58.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.975+0000 7f2fe6dfd740 4 client.26347 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:06:58.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.975+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/769153791 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ab3ca32800 con 0x55ab3c6b83a0 2024-04-25T16:06:58.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.976+0000 7f2fde04d640 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55ab3c6b83a0 0x55ab3cb487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.978+0000 7f2fde04d640 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55ab3c6b83a0 0x55ab3cb487d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7c71596bf987e51c server_cookie=bd9ef875a91b03ce in_seq=0 out_seq=0 2024-04-25T16:06:58.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.978+0000 7f2fc1ffb640 10 client.26347 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:06:58.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.979+0000 7f2fc1ffb640 1 -- 172.21.15.31:0/769153791 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ab3ca32800 con 0x55ab3c6b83a0 2024-04-25T16:06:58.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.980+0000 7f2fc1ffb640 10 client.26347 handle_command_reply: tid=0 2024-04-25T16:06:58.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.980+0000 7f2fe6dfd740 1 client.26347 shutdown 2024-04-25T16:06:58.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.980+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/769153791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 msgr2=0x55ab3cb44950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.980+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cb44950 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009a20 tx=0x7f2fd0003ec0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.980+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/769153791 shutdown_connections 2024-04-25T16:06:58.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.980+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55ab3c6b83a0 0x55ab3cb487d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.981+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ab3c724390 0x55ab3cb44950 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.981+0000 7f2fc1ffb640 0 client.26347 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:06:58.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.981+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ab3c7036b0 0x55ab3cad4c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.981+0000 7f2fe6dfd740 1 --2- 172.21.15.31:0/769153791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ab3c6fb4d0 0x55ab3cad4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.981+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/769153791 >> 172.21.15.31:0/769153791 conn(0x55ab3c7184e0 msgr2=0x55ab3cacf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:58.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.982+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/769153791 shutdown_connections 2024-04-25T16:06:58.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.982+0000 7f2fe6dfd740 1 -- 172.21.15.31:0/769153791 wait complete. 2024-04-25T16:06:58.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.982+0000 7f2fe6dfd740 20 client.26347 trim_cache size 0 max 16384 2024-04-25T16:06:58.984 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:06:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.983+0000 7f2fe02b1640 1 -- 172.21.15.31:0/3619955391 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fc40afc30 msgr2=0x7f2fc40b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.983+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fc40afc30 0x7f2fc40b2140 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2fd8109c60 tx=0x7f2fd4006c40 comp rx=0 tx=0).stop 2024-04-25T16:06:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.983+0000 7f2fe02b1640 1 -- 172.21.15.31:0/3619955391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd8110030 msgr2=0x7f2fd807c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:58.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.983+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd8110030 0x7f2fd807c5d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f2fd00098d0 tx=0x7f2fd0003ce0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.986+0000 7f2fe02b1640 1 -- 172.21.15.31:0/3619955391 shutdown_connections 2024-04-25T16:06:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.987+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2fc40afc30 0x7f2fc40b2140 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.987+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fd8110030 0x7f2fd807c5d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.987+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2fd810b310 0x7f2fd8075690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.987+0000 7f2fe02b1640 1 --2- 172.21.15.31:0/3619955391 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2fd810a420 0x7f2fd8075150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:58.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.987+0000 7f2fe02b1640 1 -- 172.21.15.31:0/3619955391 >> 172.21.15.31:0/3619955391 conn(0x7f2fd80704e0 msgr2=0x7f2fd810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:58.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.988+0000 7f2fe02b1640 1 -- 172.21.15.31:0/3619955391 shutdown_connections 2024-04-25T16:06:58.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:58.993+0000 7f2fe02b1640 1 -- 172.21.15.31:0/3619955391 wait complete. 2024-04-25T16:06:59.033 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:06:59.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:58 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3116401072' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:06:59.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.541+0000 7f22bf3f3640 1 -- 172.21.15.31:0/3487724310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b81029f0 msgr2=0x7f22b81036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:59.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.541+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/3487724310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b81029f0 0x7f22b81036f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009860 tx=0x7f22ac02a360 comp rx=0 tx=0).stop 2024-04-25T16:06:59.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.544+0000 7f22bf3f3640 1 -- 172.21.15.31:0/3487724310 shutdown_connections 2024-04-25T16:06:59.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.544+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/3487724310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b8108930 0x7f22b810ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.544+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/3487724310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b8103c30 0x7f22b8104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.544+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/3487724310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b81029f0 0x7f22b81036f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.544+0000 7f22bf3f3640 1 -- 172.21.15.31:0/3487724310 >> 172.21.15.31:0/3487724310 conn(0x7f22b80fdc00 msgr2=0x7f22b8100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:59.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.545+0000 7f22bf3f3640 1 -- 172.21.15.31:0/3487724310 shutdown_connections 2024-04-25T16:06:59.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.545+0000 7f22bf3f3640 1 -- 172.21.15.31:0/3487724310 wait complete. 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.546+0000 7f22bf3f3640 1 Processor -- start 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.547+0000 7f22bf3f3640 1 -- start start 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.547+0000 7f22bf3f3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b81029f0 0x7f22b806fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.547+0000 7f22bf3f3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8103c30 0x7f22b806ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.547+0000 7f22bf3f3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b8108930 0x7f22b8112060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.547+0000 7f22bf3f3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f22b80723c0 con 0x7f22b8103c30 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.547+0000 7f22bf3f3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f22b8072240 con 0x7f22b81029f0 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.547+0000 7f22bf3f3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22b8072540 con 0x7f22b8108930 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.548+0000 7f22bc98e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8103c30 0x7f22b806ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.548+0000 7f22bc98e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8103c30 0x7f22b806ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36896/0 (socket says 172.21.15.31:36896) 2024-04-25T16:06:59.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.548+0000 7f22bc98e640 1 -- 172.21.15.31:0/1970823233 learned_addr learned my addr 172.21.15.31:0/1970823233 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:59.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.549+0000 7f22bd990640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b8108930 0x7f22b8112060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.549+0000 7f22bd18f640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b81029f0 0x7f22b806fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.549+0000 7f22bc98e640 1 -- 172.21.15.31:0/1970823233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b8108930 msgr2=0x7f22b8112060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.549+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b8108930 0x7f22b8112060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.549+0000 7f22bc98e640 1 -- 172.21.15.31:0/1970823233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b81029f0 msgr2=0x7f22b806fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.549+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b81029f0 0x7f22b806fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.549+0000 7f22bc98e640 1 -- 172.21.15.31:0/1970823233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b81126d0 con 0x7f22b8103c30 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.550+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8103c30 0x7f22b806ffe0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f22b401ea20 tx=0x7f22b401eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.551+0000 7f22aa7fc640 1 -- 172.21.15.31:0/1970823233 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b4028070 con 0x7f22b8103c30 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.551+0000 7f22aa7fc640 1 -- 172.21.15.31:0/1970823233 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22b401abc0 con 0x7f22b8103c30 2024-04-25T16:06:59.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.551+0000 7f22aa7fc640 1 -- 172.21.15.31:0/1970823233 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b4020630 con 0x7f22b8103c30 2024-04-25T16:06:59.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.559+0000 7f22bf3f3640 1 -- 172.21.15.31:0/1970823233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b81129c0 con 0x7f22b8103c30 2024-04-25T16:06:59.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.559+0000 7f22bf3f3640 1 -- 172.21.15.31:0/1970823233 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b8112e90 con 0x7f22b8103c30 2024-04-25T16:06:59.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.567+0000 7f22aa7fc640 1 -- 172.21.15.31:0/1970823233 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f22b4021470 con 0x7f22b8103c30 2024-04-25T16:06:59.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.569+0000 7f22aa7fc640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f22a40afd30 0x7f22a40b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.570+0000 7f22aa7fc640 1 -- 172.21.15.31:0/1970823233 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f22b40ef600 con 0x7f22b8103c30 2024-04-25T16:06:59.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.574+0000 7f22c6379740 1 Processor -- start 2024-04-25T16:06:59.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.574+0000 7f22bd18f640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f22a40afd30 0x7f22a40b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.574+0000 7f22c6379740 1 -- start start 2024-04-25T16:06:59.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.574+0000 7f22c6379740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d8c390 0x563c8a12e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.574+0000 7f22c6379740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a12ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.575+0000 7f22c6379740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d6b6b0 0x563c8a12fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.575+0000 7f22c6379740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563c8a092d80 con 0x563c89d8c390 2024-04-25T16:06:59.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.575+0000 7f22c6379740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563c8a04ca20 con 0x563c89d6b6b0 2024-04-25T16:06:59.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.575+0000 7f22c6379740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563c8a08ed40 con 0x563c89d634d0 2024-04-25T16:06:59.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.575+0000 7f22bd18f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d8c390 0x563c8a12e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.576+0000 7f22bd18f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d8c390 0x563c8a12e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36906/0 (socket says 172.21.15.31:36906) 2024-04-25T16:06:59.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.576+0000 7f22bd18f640 1 -- 172.21.15.31:0/702915418 learned_addr learned my addr 172.21.15.31:0/702915418 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:59.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.576+0000 7f22bd18f640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f22a40afd30 0x7f22a40b2240 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f22b810bfc0 tx=0x7f22ac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:59.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.576+0000 7f22bc98e640 1 --2- 172.21.15.31:0/702915418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a12ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.576+0000 7f22bd990640 1 --2- 172.21.15.31:0/702915418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d6b6b0 0x563c8a12fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.577+0000 7f22bd18f640 1 -- 172.21.15.31:0/702915418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 msgr2=0x563c8a12ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:59.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.577+0000 7f22bd18f640 1 --2- 172.21.15.31:0/702915418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a12ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.577+0000 7f22bd18f640 1 -- 172.21.15.31:0/702915418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d6b6b0 msgr2=0x563c8a12fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:59.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.577+0000 7f22bd18f640 1 --2- 172.21.15.31:0/702915418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d6b6b0 0x563c8a12fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.577+0000 7f22bd18f640 1 -- 172.21.15.31:0/702915418 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c8a057c30 con 0x563c89d8c390 2024-04-25T16:06:59.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.577+0000 7f22bd18f640 1 --2- 172.21.15.31:0/702915418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d8c390 0x563c8a12e810 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009780 tx=0x7f22ac03a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19724af2d78eaf0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:59.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.586+0000 7f22a37fe640 1 -- 172.21.15.31:0/702915418 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac044040 con 0x563c89d8c390 2024-04-25T16:06:59.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.586+0000 7f22a37fe640 1 -- 172.21.15.31:0/702915418 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22ac04d070 con 0x563c89d8c390 2024-04-25T16:06:59.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.586+0000 7f22a37fe640 1 -- 172.21.15.31:0/702915418 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac0327b0 con 0x563c89d8c390 2024-04-25T16:06:59.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.587+0000 7f22c6379740 1 -- 172.21.15.31:0/702915418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d8c390 msgr2=0x563c8a12e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:59.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.587+0000 7f22c6379740 1 --2- 172.21.15.31:0/702915418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d8c390 0x563c8a12e810 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009780 tx=0x7f22ac03a000 comp rx=0 tx=0).stop 2024-04-25T16:06:59.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.587+0000 7f22c6379740 1 -- 172.21.15.31:0/702915418 shutdown_connections 2024-04-25T16:06:59.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.587+0000 7f22c6379740 1 --2- 172.21.15.31:0/702915418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d6b6b0 0x563c8a12fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.587+0000 7f22c6379740 1 --2- 172.21.15.31:0/702915418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a12ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.587+0000 7f22c6379740 1 --2- 172.21.15.31:0/702915418 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d8c390 0x563c8a12e810 secure :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009780 tx=0x7f22ac03a000 comp rx=0 tx=0).stop 2024-04-25T16:06:59.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.587+0000 7f22c6379740 1 -- 172.21.15.31:0/702915418 >> 172.21.15.31:0/702915418 conn(0x563c89d804e0 msgr2=0x563c8a12a020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:59.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.591+0000 7f22c6379740 1 -- 172.21.15.31:0/702915418 shutdown_connections 2024-04-25T16:06:59.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.591+0000 7f22c6379740 1 -- 172.21.15.31:0/702915418 wait complete. 2024-04-25T16:06:59.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.592+0000 7f22c6379740 1 -- start start 2024-04-25T16:06:59.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.592+0000 7f22c6379740 1 Processor -- start 2024-04-25T16:06:59.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.594+0000 7f22c6379740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 0x563c8a1388e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.594+0000 7f22c6379740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 0x563c8a138e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.594+0000 7f22c6379740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.594+0000 7f22c6379740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563c8a1304d0 con 0x563c89d6b6b0 2024-04-25T16:06:59.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.594+0000 7f22c6379740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563c8a019dc0 con 0x563c89d634d0 2024-04-25T16:06:59.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.594+0000 7f22c6379740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563c8a130650 con 0x563c89d4af60 2024-04-25T16:06:59.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.594+0000 7f22c6379740 10 client.0 _ll_register_callbacks cb 0x563c8a035600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:59.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.595+0000 7f22bd18f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 0x563c8a1388e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.595+0000 7f22bc98e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 0x563c8a138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.595+0000 7f22bc98e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 0x563c8a138e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36910/0 (socket says 172.21.15.31:36910) 2024-04-25T16:06:59.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.595+0000 7f22bc98e640 1 -- 172.21.15.31:0/296655840 learned_addr learned my addr 172.21.15.31:0/296655840 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:59.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.595+0000 7f22bd990640 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.597+0000 7f22bc98e640 1 -- 172.21.15.31:0/296655840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 msgr2=0x563c8a13fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:59.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.597+0000 7f22bc98e640 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.597+0000 7f22bc98e640 1 -- 172.21.15.31:0/296655840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 msgr2=0x563c8a1388e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:59.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.597+0000 7f22bc98e640 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 0x563c8a1388e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.597+0000 7f22bc98e640 1 -- 172.21.15.31:0/296655840 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c89da43f0 con 0x563c89d6b6b0 2024-04-25T16:06:59.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.599+0000 7f22bc98e640 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 0x563c8a138e20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f22b40ed050 tx=0x7f22b40b7620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9c412970857da8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:59.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.600+0000 7f22a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:06:59.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.600+0000 7f22a0ff9640 1 -- 172.21.15.31:0/296655840 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b40bbec0 con 0x563c89d6b6b0 2024-04-25T16:06:59.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.601+0000 7f22c6379740 1 -- 172.21.15.31:0/296655840 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22ac04d070 con 0x563c89d6b6b0 2024-04-25T16:06:59.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.602+0000 7f22a0ff9640 1 -- 172.21.15.31:0/296655840 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f22b40da2b0 con 0x563c89d6b6b0 2024-04-25T16:06:59.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.602+0000 7f22a0ff9640 1 -- 172.21.15.31:0/296655840 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f22b40c0970 con 0x563c89d6b6b0 2024-04-25T16:06:59.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.603+0000 7f22c6379740 10 client.16764 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:59.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.603+0000 7f22c6379740 1 -- 172.21.15.31:0/296655840 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c89e6e7d0 con 0x563c89d6b6b0 2024-04-25T16:06:59.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.603+0000 7f22a0ff9640 1 -- 172.21.15.31:0/296655840 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22ac04d070 con 0x563c89d6b6b0 2024-04-25T16:06:59.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.605+0000 7f22a0ff9640 1 -- 172.21.15.31:0/296655840 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f22b401f040 con 0x563c89d6b6b0 2024-04-25T16:06:59.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.606+0000 7f22c6379740 10 client.16764 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:59.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.606+0000 7f22c6379740 10 client.16764 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:06:59.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.606+0000 7f22c6379740 20 client.16764 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:59.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.606+0000 7f22c6379740 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563c89d203a0 0x563c8a1469e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.606+0000 7f22c6379740 4 client.16764 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:06:59.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.606+0000 7f22c6379740 1 -- 172.21.15.31:0/296655840 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563c8a0a0990 con 0x563c89d203a0 2024-04-25T16:06:59.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.606+0000 7f22bd18f640 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563c89d203a0 0x563c8a1469e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.609+0000 7f22bd18f640 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563c89d203a0 0x563c8a1469e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ca69ac7995d167bc server_cookie=cc00670d2cdcd280 in_seq=0 out_seq=0 2024-04-25T16:06:59.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.609+0000 7f22a0ff9640 10 client.16764 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:06:59.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.610+0000 7f22a0ff9640 1 -- 172.21.15.31:0/296655840 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563c8a0a0990 con 0x563c89d203a0 2024-04-25T16:06:59.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.610+0000 7f22a0ff9640 10 client.16764 handle_command_reply: tid=0 2024-04-25T16:06:59.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.610+0000 7f22c6379740 1 client.16764 shutdown 2024-04-25T16:06:59.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.610+0000 7f22c6379740 1 -- 172.21.15.31:0/296655840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 msgr2=0x563c8a138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:59.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.610+0000 7f22c6379740 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 0x563c8a138e20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f22b40ed050 tx=0x7f22b40b7620 comp rx=0 tx=0).stop 2024-04-25T16:06:59.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.611+0000 7f22c6379740 1 -- 172.21.15.31:0/296655840 shutdown_connections 2024-04-25T16:06:59.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.611+0000 7f22c6379740 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563c89d203a0 0x563c8a1469e0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.611+0000 7f22c6379740 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.611+0000 7f22c6379740 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 0x563c8a138e20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.611+0000 7f22c6379740 1 --2- 172.21.15.31:0/296655840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 0x563c8a1388e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.611+0000 7f22c6379740 1 -- 172.21.15.31:0/296655840 >> 172.21.15.31:0/296655840 conn(0x563c89d804e0 msgr2=0x563c8a129040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:59.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.611+0000 7f22a0ff9640 0 client.16764 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:06:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.612+0000 7f22c6379740 1 -- 172.21.15.31:0/296655840 shutdown_connections 2024-04-25T16:06:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.612+0000 7f22c6379740 1 -- 172.21.15.31:0/296655840 wait complete. 2024-04-25T16:06:59.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.612+0000 7f22c6379740 20 client.16764 trim_cache size 0 max 16384 2024-04-25T16:06:59.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.644+0000 7f22c6379740 1 Processor -- start 2024-04-25T16:06:59.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.645+0000 7f22c6379740 1 -- start start 2024-04-25T16:06:59.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.646+0000 7f22c6379740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d4af60 0x563c8a12a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.646+0000 7f22c6379740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a13fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.646+0000 7f22c6379740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d6b6b0 0x563c8a136f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.647+0000 7f22c6379740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f22b81a0640 con 0x563c89d4af60 2024-04-25T16:06:59.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.647+0000 7f22bc98e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a13fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.647+0000 7f22bc98e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a13fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39208/0 (socket says 172.21.15.31:39208) 2024-04-25T16:06:59.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.647+0000 7f22bc98e640 1 -- 172.21.15.31:0/1168405803 learned_addr learned my addr 172.21.15.31:0/1168405803 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:59.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.647+0000 7f22c6379740 1 -- 172.21.15.31:0/1168405803 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f22b81a2180 con 0x563c89d6b6b0 2024-04-25T16:06:59.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.647+0000 7f22c6379740 1 -- 172.21.15.31:0/1168405803 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22b81a0d10 con 0x563c89d634d0 2024-04-25T16:06:59.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.648+0000 7f22bd18f640 1 --2- 172.21.15.31:0/1168405803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d4af60 0x563c8a12a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.649+0000 7f22bc98e640 1 -- 172.21.15.31:0/1168405803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d6b6b0 msgr2=0x563c8a136f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:06:59.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.649+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1168405803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d6b6b0 0x563c8a136f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.649+0000 7f22bc98e640 1 -- 172.21.15.31:0/1168405803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d4af60 msgr2=0x563c8a12a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:59.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.649+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1168405803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d4af60 0x563c8a12a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.649+0000 7f22bc98e640 1 -- 172.21.15.31:0/1168405803 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563c8a1242b0 con 0x563c89d634d0 2024-04-25T16:06:59.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.650+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1168405803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a13fd40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f22b40b40b0 tx=0x7f22b40da500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7837ad1fbdb12ecf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:59.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.652+0000 7f22a2ffd640 1 -- 172.21.15.31:0/1168405803 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b40bbec0 con 0x563c89d634d0 2024-04-25T16:06:59.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.653+0000 7f22a2ffd640 1 -- 172.21.15.31:0/1168405803 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22b40d1070 con 0x563c89d634d0 2024-04-25T16:06:59.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.654+0000 7f22c6379740 1 -- 172.21.15.31:0/1168405803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 msgr2=0x563c8a13fd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:59.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.654+0000 7f22c6379740 1 --2- 172.21.15.31:0/1168405803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a13fd40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f22b40b40b0 tx=0x7f22b40da500 comp rx=0 tx=0).stop 2024-04-25T16:06:59.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.667+0000 7f22c6379740 1 -- 172.21.15.31:0/1168405803 shutdown_connections 2024-04-25T16:06:59.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.667+0000 7f22c6379740 1 --2- 172.21.15.31:0/1168405803 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d6b6b0 0x563c8a136f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.667+0000 7f22c6379740 1 --2- 172.21.15.31:0/1168405803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d634d0 0x563c8a13fd40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.667+0000 7f22c6379740 1 --2- 172.21.15.31:0/1168405803 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d4af60 0x563c8a12a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.667+0000 7f22c6379740 1 -- 172.21.15.31:0/1168405803 >> 172.21.15.31:0/1168405803 conn(0x563c89d804e0 msgr2=0x563c8a1299e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:59.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.668+0000 7f22c6379740 1 -- 172.21.15.31:0/1168405803 shutdown_connections 2024-04-25T16:06:59.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.668+0000 7f22c6379740 1 -- 172.21.15.31:0/1168405803 wait complete. 2024-04-25T16:06:59.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.669+0000 7f22c6379740 1 -- start start 2024-04-25T16:06:59.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.669+0000 7f22c6379740 1 Processor -- start 2024-04-25T16:06:59.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.670+0000 7f22c6379740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.670+0000 7f22c6379740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 0x563c8a13d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.670+0000 7f22c6379740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 0x563c8a1acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.670+0000 7f22c6379740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f22b81a1ab0 con 0x563c89d6b6b0 2024-04-25T16:06:59.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.670+0000 7f22c6379740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f22b81a13e0 con 0x563c89d634d0 2024-04-25T16:06:59.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.670+0000 7f22c6379740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22b819e6d0 con 0x563c89d4af60 2024-04-25T16:06:59.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.671+0000 7f22c6379740 10 client.0 _ll_register_callbacks cb 0x7f22b40b3310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:06:59.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.672+0000 7f22bd18f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.672+0000 7f22bd18f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39214/0 (socket says 172.21.15.31:39214) 2024-04-25T16:06:59.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.672+0000 7f22bd18f640 1 -- 172.21.15.31:0/1664563434 learned_addr learned my addr 172.21.15.31:0/1664563434 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:06:59.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.672+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 0x563c8a13d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.673+0000 7f22bd18f640 1 -- 172.21.15.31:0/1664563434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 msgr2=0x563c8a13d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:59.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.673+0000 7f22bd18f640 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 0x563c8a13d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.673+0000 7f22bd18f640 1 -- 172.21.15.31:0/1664563434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 msgr2=0x563c8a1acd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:06:59.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.673+0000 7f22bd18f640 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 0x563c8a1acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.673+0000 7f22bd18f640 1 -- 172.21.15.31:0/1664563434 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563c8a134060 con 0x563c89d4af60 2024-04-25T16:06:59.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.673+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 0x563c8a13d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:06:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.681+0000 7f22bd18f640 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13cb20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f22ac02a950 tx=0x7f22ac030340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acbb7076d8be25c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:06:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.681+0000 7f22a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:06:59.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.681+0000 7f22a0ff9640 1 -- 172.21.15.31:0/1664563434 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac061060 con 0x563c89d4af60 2024-04-25T16:06:59.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.682+0000 7f22c6379740 1 -- 172.21.15.31:0/1664563434 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f22b40d1070 con 0x563c89d4af60 2024-04-25T16:06:59.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.682+0000 7f22a0ff9640 1 -- 172.21.15.31:0/1664563434 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f22ac05b5b0 con 0x563c89d4af60 2024-04-25T16:06:59.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.682+0000 7f22a0ff9640 1 -- 172.21.15.31:0/1664563434 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f22ac05b7b0 con 0x563c89d4af60 2024-04-25T16:06:59.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.683+0000 7f22c6379740 10 client.26362 fetch_fsmap learned FSMap version 29 2024-04-25T16:06:59.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.683+0000 7f22c6379740 1 -- 172.21.15.31:0/1664563434 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563c8a1ae150 con 0x563c89d4af60 2024-04-25T16:06:59.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.699+0000 7f22a0ff9640 1 -- 172.21.15.31:0/1664563434 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22b40d1070 con 0x563c89d4af60 2024-04-25T16:06:59.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.699+0000 7f22a0ff9640 1 -- 172.21.15.31:0/1664563434 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f22ac047030 con 0x563c89d4af60 2024-04-25T16:06:59.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.702+0000 7f22c6379740 10 client.26362 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:06:59.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.702+0000 7f22c6379740 10 client.26362 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:06:59.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.702+0000 7f22c6379740 20 client.26362 populate_metadata read hostname 'smithi031' 2024-04-25T16:06:59.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.702+0000 7f22c6379740 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563c8a1aea80 0x563c8a1b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:06:59.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.702+0000 7f22c6379740 4 client.26362 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:06:59.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.702+0000 7f22c6379740 1 -- 172.21.15.31:0/1664563434 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563c8a09a800 con 0x563c8a1aea80 2024-04-25T16:06:59.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.705+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563c8a1aea80 0x563c8a1b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:06:59.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.709+0000 7f22bc98e640 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563c8a1aea80 0x563c8a1b0ea0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=26b05aa86d5f1931 server_cookie=399a7f9cb5f6f8c5 in_seq=0 out_seq=0 2024-04-25T16:06:59.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.709+0000 7f22a0ff9640 10 client.26362 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:06:59.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.720+0000 7f22a0ff9640 1 -- 172.21.15.31:0/1664563434 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563c8a09a800 con 0x563c8a1aea80 2024-04-25T16:06:59.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.720+0000 7f22a0ff9640 10 client.26362 handle_command_reply: tid=0 2024-04-25T16:06:59.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.722+0000 7f22c6379740 1 client.26362 shutdown 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.742+0000 7f22c6379740 1 -- 172.21.15.31:0/1664563434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 msgr2=0x563c8a13cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.742+0000 7f22c6379740 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13cb20 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f22ac02a950 tx=0x7f22ac030340 comp rx=0 tx=0).stop 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.742+0000 7f22c6379740 1 -- 172.21.15.31:0/1664563434 shutdown_connections 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.742+0000 7f22c6379740 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563c89d6b6b0 0x563c8a1acd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.742+0000 7f22c6379740 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563c89d634d0 0x563c8a13d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.742+0000 7f22c6379740 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563c8a1aea80 0x563c8a1b0ea0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.742+0000 7f22c6379740 1 --2- 172.21.15.31:0/1664563434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563c89d4af60 0x563c8a13cb20 secure :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f22ac02a950 tx=0x7f22ac030340 comp rx=0 tx=0).stop 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.742+0000 7f22c6379740 1 -- 172.21.15.31:0/1664563434 >> 172.21.15.31:0/1664563434 conn(0x563c89d804e0 msgr2=0x563c8a12ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.744+0000 7f22a0ff9640 0 client.26362 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:06:59.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.744+0000 7f22c6379740 1 -- 172.21.15.31:0/1664563434 shutdown_connections 2024-04-25T16:06:59.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.744+0000 7f22c6379740 1 -- 172.21.15.31:0/1664563434 wait complete. 2024-04-25T16:06:59.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.750+0000 7f22c6379740 20 client.26362 trim_cache size 0 max 16384 2024-04-25T16:06:59.756 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:06:59.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.768+0000 7f22bf3f3640 1 -- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f22a40afd30 msgr2=0x7f22a40b2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:59.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.768+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f22a40afd30 0x7f22a40b2240 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f22b810bfc0 tx=0x7f22ac000bc0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.768+0000 7f22bf3f3640 1 -- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8103c30 msgr2=0x7f22b806ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:06:59.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.768+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8103c30 0x7f22b806ffe0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f22b401ea20 tx=0x7f22b401eee0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.769+0000 7f22bf3f3640 1 -- 172.21.15.31:0/1970823233 shutdown_connections 2024-04-25T16:06:59.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.769+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f22a40afd30 0x7f22a40b2240 secure :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f22b810bfc0 tx=0x7f22ac000bc0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.769+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22b8108930 0x7f22b8112060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.769+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f22b8103c30 0x7f22b806ffe0 secure :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f22b401ea20 tx=0x7f22b401eee0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.769+0000 7f22bf3f3640 1 --2- 172.21.15.31:0/1970823233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f22b81029f0 0x7f22b806fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:06:59.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.769+0000 7f22bf3f3640 1 -- 172.21.15.31:0/1970823233 >> 172.21.15.31:0/1970823233 conn(0x7f22b80fdc00 msgr2=0x7f22b8100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:06:59.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.775+0000 7f22bf3f3640 1 -- 172.21.15.31:0/1970823233 shutdown_connections 2024-04-25T16:06:59.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:06:59.780+0000 7f22bf3f3640 1 -- 172.21.15.31:0/1970823233 wait complete. 2024-04-25T16:06:59.896 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:06:59.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:06:59 smithi186 ceph-mon[33258]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 922 MiB/s wr, 1.86k op/s 2024-04-25T16:06:59.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:06:59 smithi031 ceph-mon[29140]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 922 MiB/s wr, 1.86k op/s 2024-04-25T16:07:00.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:06:59 smithi063 ceph-mon[32413]: pgmap v521: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 922 MiB/s wr, 1.86k op/s 2024-04-25T16:07:00.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.320+0000 7f3fd12e7640 1 -- 172.21.15.31:0/1451662595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc10b1d0 msgr2=0x7f3fcc10b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:00.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.320+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/1451662595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc10b1d0 0x7f3fcc10b5e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc009900 tx=0x7f3fbc02a440 comp rx=0 tx=0).stop 2024-04-25T16:07:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.337+0000 7f3fd12e7640 1 -- 172.21.15.31:0/1451662595 shutdown_connections 2024-04-25T16:07:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.337+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/1451662595 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc10fef0 0x7f3fcc110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.337+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/1451662595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc10b1d0 0x7f3fcc10b5e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.337+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/1451662595 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fcc10a3e0 0x7f3fcc10ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.337+0000 7f3fd12e7640 1 -- 172.21.15.31:0/1451662595 >> 172.21.15.31:0/1451662595 conn(0x7f3fcc070400 msgr2=0x7f3fcc072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:00.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.338+0000 7f3fd12e7640 1 -- 172.21.15.31:0/1451662595 shutdown_connections 2024-04-25T16:07:00.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.338+0000 7f3fd12e7640 1 -- 172.21.15.31:0/1451662595 wait complete. 2024-04-25T16:07:00.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.339+0000 7f3fd12e7640 1 Processor -- start 2024-04-25T16:07:00.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.340+0000 7f3fd12e7640 1 -- start start 2024-04-25T16:07:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.340+0000 7f3fd12e7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc10a3e0 0x7f3fcc117010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.340+0000 7f3fd12e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc10b1d0 0x7f3fcc117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.340+0000 7f3fd12e7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fcc10fef0 0x7f3fcc1184b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.340+0000 7f3fd12e7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc11e280 con 0x7f3fcc10a3e0 2024-04-25T16:07:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.340+0000 7f3fd12e7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc11e100 con 0x7f3fcc10fef0 2024-04-25T16:07:00.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.340+0000 7f3fd12e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc11e400 con 0x7f3fcc10b1d0 2024-04-25T16:07:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.341+0000 7f3fcb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fcc10fef0 0x7f3fcc1184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.341+0000 7f3fcb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fcc10fef0 0x7f3fcc1184b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40124/0 (socket says 172.21.15.31:40124) 2024-04-25T16:07:00.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.341+0000 7f3fcb7fe640 1 -- 172.21.15.31:0/2652096178 learned_addr learned my addr 172.21.15.31:0/2652096178 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.343+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc10a3e0 0x7f3fcc117010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.344+0000 7f3fca7fc640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc10b1d0 0x7f3fcc117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.344+0000 7f3fcaffd640 1 -- 172.21.15.31:0/2652096178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc10b1d0 msgr2=0x7f3fcc117550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:00.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.344+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc10b1d0 0x7f3fcc117550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.344+0000 7f3fcaffd640 1 -- 172.21.15.31:0/2652096178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fcc10fef0 msgr2=0x7f3fcc1184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:00.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.344+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fcc10fef0 0x7f3fcc1184b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.344+0000 7f3fcaffd640 1 -- 172.21.15.31:0/2652096178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fcc118b20 con 0x7f3fcc10a3e0 2024-04-25T16:07:00.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.345+0000 7f3fcb7fe640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fcc10fef0 0x7f3fcc1184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:07:00.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.345+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc10a3e0 0x7f3fcc117010 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc074760 tx=0x7f3fc000cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.352+0000 7f3fbbfff640 1 -- 172.21.15.31:0/2652096178 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0007bb0 con 0x7f3fcc10a3e0 2024-04-25T16:07:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.352+0000 7f3fbbfff640 1 -- 172.21.15.31:0/2652096178 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fc0007d50 con 0x7f3fcc10a3e0 2024-04-25T16:07:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.352+0000 7f3fbbfff640 1 -- 172.21.15.31:0/2652096178 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc0004de0 con 0x7f3fcc10a3e0 2024-04-25T16:07:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.354+0000 7f3fd12e7640 1 -- 172.21.15.31:0/2652096178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fcc118e10 con 0x7f3fcc10a3e0 2024-04-25T16:07:00.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.354+0000 7f3fd12e7640 1 -- 172.21.15.31:0/2652096178 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fcc1192e0 con 0x7f3fcc10a3e0 2024-04-25T16:07:00.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.359+0000 7f3fd819a740 1 Processor -- start 2024-04-25T16:07:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.365+0000 7f3fd819a740 1 -- start start 2024-04-25T16:07:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.366+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 0x55588f36b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.366+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f36bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.366+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa86b0 0x55588f36cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.366+0000 7f3fd819a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55588f2cfd80 con 0x55588efc9390 2024-04-25T16:07:00.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.366+0000 7f3fd819a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55588f289a20 con 0x55588efa86b0 2024-04-25T16:07:00.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.366+0000 7f3fd819a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55588f2cbd40 con 0x55588efa04d0 2024-04-25T16:07:00.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.369+0000 7f3fbbfff640 1 -- 172.21.15.31:0/2652096178 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3fc0004f80 con 0x7f3fcc10a3e0 2024-04-25T16:07:00.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.370+0000 7f3fcaffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f36bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.370+0000 7f3fca7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa86b0 0x55588f36cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.370+0000 7f3fca7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa86b0 0x55588f36cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40128/0 (socket says 172.21.15.31:40128) 2024-04-25T16:07:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.370+0000 7f3fca7fc640 1 -- 172.21.15.31:0/1720035211 learned_addr learned my addr 172.21.15.31:0/1720035211 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.379+0000 7f3fcb7fe640 1 --2- 172.21.15.31:0/1720035211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 0x55588f36b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.383+0000 7f3fbbfff640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fb40afcf0 0x7f3fb40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.384+0000 7f3fbbfff640 1 -- 172.21.15.31:0/2652096178 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3fc00debc0 con 0x7f3fcc10a3e0 2024-04-25T16:07:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.394+0000 7f3fca7fc640 1 -- 172.21.15.31:0/1720035211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 msgr2=0x55588f36bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.394+0000 7f3fca7fc640 1 --2- 172.21.15.31:0/1720035211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f36bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.394+0000 7f3fca7fc640 1 -- 172.21.15.31:0/1720035211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 msgr2=0x55588f36b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.394+0000 7f3fca7fc640 1 --2- 172.21.15.31:0/1720035211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 0x55588f36b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.394+0000 7f3fca7fc640 1 -- 172.21.15.31:0/1720035211 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55588f294c30 con 0x55588efa86b0 2024-04-25T16:07:00.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.394+0000 7f3fcb7fe640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fb40afcf0 0x7f3fb40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.400+0000 7f3fca7fc640 1 --2- 172.21.15.31:0/1720035211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa86b0 0x55588f36cbf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc031470 tx=0x7f3fbc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cf6e9a8d8f2bdde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:00.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.403+0000 7f3fb97fa640 1 -- 172.21.15.31:0/1720035211 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc035040 con 0x55588efa86b0 2024-04-25T16:07:00.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.403+0000 7f3fb97fa640 1 -- 172.21.15.31:0/1720035211 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fbc031c70 con 0x55588efa86b0 2024-04-25T16:07:00.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.403+0000 7f3fb97fa640 1 -- 172.21.15.31:0/1720035211 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc005650 con 0x55588efa86b0 2024-04-25T16:07:00.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.404+0000 7f3fd819a740 1 -- 172.21.15.31:0/1720035211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa86b0 msgr2=0x55588f36cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.404+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1720035211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa86b0 0x55588f36cbf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc031470 tx=0x7f3fbc000bc0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.406+0000 7f3fd819a740 1 -- 172.21.15.31:0/1720035211 shutdown_connections 2024-04-25T16:07:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.406+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1720035211 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa86b0 0x55588f36cbf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.406+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1720035211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f36bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.406+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1720035211 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 0x55588f36b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.406+0000 7f3fd819a740 1 -- 172.21.15.31:0/1720035211 >> 172.21.15.31:0/1720035211 conn(0x55588efbd4e0 msgr2=0x55588f3671f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:00.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.406+0000 7f3fd819a740 1 -- 172.21.15.31:0/1720035211 shutdown_connections 2024-04-25T16:07:00.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.406+0000 7f3fd819a740 1 -- 172.21.15.31:0/1720035211 wait complete. 2024-04-25T16:07:00.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.407+0000 7f3fd819a740 1 -- start start 2024-04-25T16:07:00.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.407+0000 7f3fd819a740 1 Processor -- start 2024-04-25T16:07:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efa86b0 0x55588f375c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efc9390 0x55588f37cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fd819a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55588f36d6a0 con 0x55588efa86b0 2024-04-25T16:07:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fd819a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55588f256dc0 con 0x55588efc9390 2024-04-25T16:07:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fd819a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55588f36d820 con 0x55588efa04d0 2024-04-25T16:07:00.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fd819a740 10 client.0 _ll_register_callbacks cb 0x55588f272600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fcaffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fcaffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39244/0 (socket says 172.21.15.31:39244) 2024-04-25T16:07:00.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.408+0000 7f3fcaffd640 1 -- 172.21.15.31:0/4282101857 learned_addr learned my addr 172.21.15.31:0/4282101857 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:00.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.413+0000 7f3fcaffd640 1 -- 172.21.15.31:0/4282101857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efc9390 msgr2=0x55588f37cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:00.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.413+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efc9390 0x55588f37cbe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.413+0000 7f3fcaffd640 1 -- 172.21.15.31:0/4282101857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efa86b0 msgr2=0x55588f375c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:00.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.413+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efa86b0 0x55588f375c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.413+0000 7f3fcaffd640 1 -- 172.21.15.31:0/4282101857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55588efe13f0 con 0x55588efa04d0 2024-04-25T16:07:00.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.413+0000 7f3fcb7fe640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fb40afcf0 0x7f3fb40b2200 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55588f36ca30 tx=0x7f3fc4004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:00.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.416+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375740 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3fc00a3150 tx=0x7f3fc00a76f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd8e13a1ff206274 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:00.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.421+0000 7f3fae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:00.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.421+0000 7f3fae7fc640 1 -- 172.21.15.31:0/4282101857 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc00a77d0 con 0x55588efa04d0 2024-04-25T16:07:00.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.421+0000 7f3fae7fc640 1 -- 172.21.15.31:0/4282101857 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3fc00a6330 con 0x55588efa04d0 2024-04-25T16:07:00.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.421+0000 7f3fd819a740 1 -- 172.21.15.31:0/4282101857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fbc031c70 con 0x55588efa04d0 2024-04-25T16:07:00.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.426+0000 7f3fae7fc640 1 -- 172.21.15.31:0/4282101857 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3fc00a70d0 con 0x55588efa04d0 2024-04-25T16:07:00.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.426+0000 7f3fd819a740 10 client.26374 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:00.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.426+0000 7f3fd819a740 1 -- 172.21.15.31:0/4282101857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55588f0ab7d0 con 0x55588efa04d0 2024-04-25T16:07:00.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.428+0000 7f3fae7fc640 1 -- 172.21.15.31:0/4282101857 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fbc031c70 con 0x55588efa04d0 2024-04-25T16:07:00.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.431+0000 7f3fae7fc640 1 -- 172.21.15.31:0/4282101857 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3fc00a2a40 con 0x55588efa04d0 2024-04-25T16:07:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.434+0000 7f3fd819a740 10 client.26374 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.434+0000 7f3fd819a740 10 client.26374 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:07:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.434+0000 7f3fd819a740 20 client.26374 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.434+0000 7f3fd819a740 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55588ef87f60 0x55588f3834e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.434+0000 7f3fd819a740 4 client.26374 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:00.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.434+0000 7f3fd819a740 1 -- 172.21.15.31:0/4282101857 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55588f2dd990 con 0x55588ef87f60 2024-04-25T16:07:00.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.438+0000 7f3fca7fc640 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55588ef87f60 0x55588f3834e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.452+0000 7f3fca7fc640 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55588ef87f60 0x55588f3834e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ae3957c2347e63fa server_cookie=b915bb479f20c5db in_seq=0 out_seq=0 2024-04-25T16:07:00.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.452+0000 7f3fae7fc640 10 client.26374 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:07:00.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.454+0000 7f3fae7fc640 1 -- 172.21.15.31:0/4282101857 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55588f2dd990 con 0x55588ef87f60 2024-04-25T16:07:00.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.454+0000 7f3fae7fc640 10 client.26374 handle_command_reply: tid=0 2024-04-25T16:07:00.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.454+0000 7f3fd819a740 1 client.26374 shutdown 2024-04-25T16:07:00.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.473+0000 7f3fd819a740 1 -- 172.21.15.31:0/4282101857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 msgr2=0x55588f375740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:00.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.473+0000 7f3fd819a740 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375740 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3fc00a3150 tx=0x7f3fc00a76f0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.473+0000 7f3fd819a740 1 -- 172.21.15.31:0/4282101857 shutdown_connections 2024-04-25T16:07:00.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.473+0000 7f3fd819a740 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55588ef87f60 0x55588f3834e0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.473+0000 7f3fd819a740 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efc9390 0x55588f37cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.473+0000 7f3fd819a740 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efa86b0 0x55588f375c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.473+0000 7f3fd819a740 1 --2- 172.21.15.31:0/4282101857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375740 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.473+0000 7f3fd819a740 1 -- 172.21.15.31:0/4282101857 >> 172.21.15.31:0/4282101857 conn(0x55588efbd4e0 msgr2=0x55588f3663f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.474+0000 7f3fae7fc640 0 client.26374 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:07:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.474+0000 7f3fd819a740 1 -- 172.21.15.31:0/4282101857 shutdown_connections 2024-04-25T16:07:00.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.474+0000 7f3fd819a740 1 -- 172.21.15.31:0/4282101857 wait complete. 2024-04-25T16:07:00.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.474+0000 7f3fd819a740 20 client.26374 trim_cache size 0 max 16384 2024-04-25T16:07:00.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.522+0000 7f3fd819a740 1 Processor -- start 2024-04-25T16:07:00.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.523+0000 7f3fd819a740 1 -- start start 2024-04-25T16:07:00.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.524+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.524+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efa86b0 0x55588f375850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.524+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efc9390 0x55588f387c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.524+0000 7f3fd819a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc1c2450 con 0x55588efa86b0 2024-04-25T16:07:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.524+0000 7f3fd819a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc1c3f90 con 0x55588efc9390 2024-04-25T16:07:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.524+0000 7f3fd819a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc1c2b20 con 0x55588efa04d0 2024-04-25T16:07:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.529+0000 7f3fcaffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.529+0000 7f3fcb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efc9390 0x55588f387c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.529+0000 7f3fcaffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39256/0 (socket says 172.21.15.31:39256) 2024-04-25T16:07:00.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.529+0000 7f3fcaffd640 1 -- 172.21.15.31:0/2760788926 learned_addr learned my addr 172.21.15.31:0/2760788926 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.530+0000 7f3fcaffd640 1 -- 172.21.15.31:0/2760788926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efc9390 msgr2=0x55588f387c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.530+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/2760788926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efc9390 0x55588f387c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.530+0000 7f3fcaffd640 1 -- 172.21.15.31:0/2760788926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efa86b0 msgr2=0x55588f375850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.530+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/2760788926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efa86b0 0x55588f375850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.530+0000 7f3fcaffd640 1 -- 172.21.15.31:0/2760788926 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55588f36d5e0 con 0x55588efa04d0 2024-04-25T16:07:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.530+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/2760788926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3fc00deef0 tx=0x7f3fc0004830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b1ff0a0af5f559c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.531+0000 7f3fb8ff9640 1 -- 172.21.15.31:0/2760788926 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc00a6ea0 con 0x55588efa04d0 2024-04-25T16:07:00.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.531+0000 7f3fb8ff9640 1 -- 172.21.15.31:0/2760788926 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fc00b1070 con 0x55588efa04d0 2024-04-25T16:07:00.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.533+0000 7f3fd819a740 1 -- 172.21.15.31:0/2760788926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 msgr2=0x55588f375310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:00.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.533+0000 7f3fd819a740 1 --2- 172.21.15.31:0/2760788926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3fc00deef0 tx=0x7f3fc0004830 comp rx=0 tx=0).stop 2024-04-25T16:07:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.539+0000 7f3fd819a740 1 -- 172.21.15.31:0/2760788926 shutdown_connections 2024-04-25T16:07:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.539+0000 7f3fd819a740 1 --2- 172.21.15.31:0/2760788926 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efc9390 0x55588f387c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.539+0000 7f3fd819a740 1 --2- 172.21.15.31:0/2760788926 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efa86b0 0x55588f375850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.539+0000 7f3fd819a740 1 --2- 172.21.15.31:0/2760788926 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa04d0 0x55588f375310 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.539+0000 7f3fd819a740 1 -- 172.21.15.31:0/2760788926 >> 172.21.15.31:0/2760788926 conn(0x55588efbd4e0 msgr2=0x55588f3704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:00.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.539+0000 7f3fd819a740 1 -- 172.21.15.31:0/2760788926 shutdown_connections 2024-04-25T16:07:00.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.541+0000 7f3fd819a740 1 -- 172.21.15.31:0/2760788926 wait complete. 2024-04-25T16:07:00.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.542+0000 7f3fd819a740 1 -- start start 2024-04-25T16:07:00.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.542+0000 7f3fd819a740 1 Processor -- start 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.543+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa04d0 0x55588f379560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.543+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa86b0 0x55588f379aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.543+0000 7f3fd819a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 0x55588f37a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.543+0000 7f3fd819a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc1c38c0 con 0x55588efc9390 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.543+0000 7f3fd819a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc1c31f0 con 0x55588efa04d0 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.543+0000 7f3fd819a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3fcc1c04e0 con 0x55588efa86b0 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.543+0000 7f3fd819a740 10 client.0 _ll_register_callbacks cb 0x7f3fc000cf00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.544+0000 7f3fca7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa86b0 0x55588f379aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.544+0000 7f3fca7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa86b0 0x55588f379aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39260/0 (socket says 172.21.15.31:39260) 2024-04-25T16:07:00.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.544+0000 7f3fca7fc640 1 -- 172.21.15.31:0/1355172138 learned_addr learned my addr 172.21.15.31:0/1355172138 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:00.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.548+0000 7f3fcb7fe640 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 0x55588f37a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.548+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa04d0 0x55588f379560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.549+0000 7f3fca7fc640 1 -- 172.21.15.31:0/1355172138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa04d0 msgr2=0x55588f379560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:00.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.549+0000 7f3fca7fc640 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa04d0 0x55588f379560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.549+0000 7f3fca7fc640 1 -- 172.21.15.31:0/1355172138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 msgr2=0x55588f37a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:00.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.549+0000 7f3fca7fc640 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 0x55588f37a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.549+0000 7f3fca7fc640 1 -- 172.21.15.31:0/1355172138 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55588f37af90 con 0x55588efa86b0 2024-04-25T16:07:00.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.549+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa04d0 0x55588f379560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:07:00.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.550+0000 7f3fca7fc640 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa86b0 0x55588f379aa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc000c00 tx=0x7f3fbc031690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df5f5397d958678d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:00.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.551+0000 7f3fae7fc640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:00.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.563+0000 7f3fae7fc640 1 -- 172.21.15.31:0/1355172138 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fbc035040 con 0x55588efa86b0 2024-04-25T16:07:00.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.563+0000 7f3fae7fc640 1 -- 172.21.15.31:0/1355172138 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3fbc031c80 con 0x55588efa86b0 2024-04-25T16:07:00.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.563+0000 7f3fd819a740 1 -- 172.21.15.31:0/1355172138 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3fc00b1070 con 0x55588efa86b0 2024-04-25T16:07:00.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.564+0000 7f3fae7fc640 1 -- 172.21.15.31:0/1355172138 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3fbc004040 con 0x55588efa86b0 2024-04-25T16:07:00.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.570+0000 7f3fd819a740 10 client.26380 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:00.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.570+0000 7f3fd819a740 1 -- 172.21.15.31:0/1355172138 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55588f37c240 con 0x55588efa86b0 2024-04-25T16:07:00.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.575+0000 7f3fae7fc640 1 -- 172.21.15.31:0/1355172138 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3fc00b1070 con 0x55588efa86b0 2024-04-25T16:07:00.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.575+0000 7f3fae7fc640 1 -- 172.21.15.31:0/1355172138 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3fbc03a040 con 0x55588efa86b0 2024-04-25T16:07:00.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.585+0000 7f3fd819a740 10 client.26380 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:00.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.585+0000 7f3fd819a740 10 client.26380 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:07:00.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.585+0000 7f3fd819a740 20 client.26380 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:00.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.585+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55588ef5d3a0 0x55588f3ed860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:00.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.585+0000 7f3fd819a740 4 client.26380 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:07:00.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.585+0000 7f3fd819a740 1 -- 172.21.15.31:0/1355172138 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55588f2d7800 con 0x55588ef5d3a0 2024-04-25T16:07:00.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.586+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55588ef5d3a0 0x55588f3ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:00.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.587+0000 7f3fae7fc640 10 client.26380 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:07:00.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.587+0000 7f3fcaffd640 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55588ef5d3a0 0x55588f3ed860 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd203556669e21d0 server_cookie=6c70b6e3862cffbd in_seq=0 out_seq=0 2024-04-25T16:07:00.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.588+0000 7f3fae7fc640 1 -- 172.21.15.31:0/1355172138 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55588f2d7800 con 0x55588ef5d3a0 2024-04-25T16:07:00.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.588+0000 7f3fae7fc640 10 client.26380 handle_command_reply: tid=0 2024-04-25T16:07:00.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.588+0000 7f3fd819a740 1 client.26380 shutdown 2024-04-25T16:07:00.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fd819a740 1 -- 172.21.15.31:0/1355172138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa86b0 msgr2=0x55588f379aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa86b0 0x55588f379aa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3fbc000c00 tx=0x7f3fbc031690 comp rx=0 tx=0).stop 2024-04-25T16:07:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fd819a740 1 -- 172.21.15.31:0/1355172138 shutdown_connections 2024-04-25T16:07:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55588efc9390 0x55588f37a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55588efa86b0 0x55588f379aa0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55588ef5d3a0 0x55588f3ed860 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fd819a740 1 --2- 172.21.15.31:0/1355172138 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55588efa04d0 0x55588f379560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fd819a740 1 -- 172.21.15.31:0/1355172138 >> 172.21.15.31:0/1355172138 conn(0x55588efbd4e0 msgr2=0x55588f375310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fae7fc640 0 client.26380 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:07:00.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.590+0000 7f3fd819a740 1 -- 172.21.15.31:0/1355172138 shutdown_connections 2024-04-25T16:07:00.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.591+0000 7f3fd819a740 1 -- 172.21.15.31:0/1355172138 wait complete. 2024-04-25T16:07:00.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.595+0000 7f3fd819a740 20 client.26380 trim_cache size 0 max 16384 2024-04-25T16:07:00.604 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:07:00.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.606+0000 7f3fd12e7640 1 -- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fb40afcf0 msgr2=0x7f3fb40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:00.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.606+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fb40afcf0 0x7f3fb40b2200 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x55588f36ca30 tx=0x7f3fc4004ab0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.606+0000 7f3fd12e7640 1 -- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc10a3e0 msgr2=0x7f3fcc117010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:00.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.606+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc10a3e0 0x7f3fcc117010 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f3fcc074760 tx=0x7f3fc000cdd0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.615+0000 7f3fd12e7640 1 -- 172.21.15.31:0/2652096178 shutdown_connections 2024-04-25T16:07:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.616+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fb40afcf0 0x7f3fb40b2200 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.616+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3fcc10fef0 0x7f3fcc1184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.616+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc10b1d0 0x7f3fcc117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.616+0000 7f3fd12e7640 1 --2- 172.21.15.31:0/2652096178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3fcc10a3e0 0x7f3fcc117010 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:00.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.616+0000 7f3fd12e7640 1 -- 172.21.15.31:0/2652096178 >> 172.21.15.31:0/2652096178 conn(0x7f3fcc070400 msgr2=0x7f3fcc10bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:00.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.621+0000 7f3fd12e7640 1 -- 172.21.15.31:0/2652096178 shutdown_connections 2024-04-25T16:07:00.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:00.621+0000 7f3fd12e7640 1 -- 172.21.15.31:0/2652096178 wait complete. 2024-04-25T16:07:00.724 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:07:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.236+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2671554889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 msgr2=0x7fbb1010a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.236+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2671554889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 0x7fbb1010a4f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbb04009980 tx=0x7fbb0402c620 comp rx=0 tx=0).stop 2024-04-25T16:07:01.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.240+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2671554889 shutdown_connections 2024-04-25T16:07:01.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.240+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2671554889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb10078da0 0x7fbb1007b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.240+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2671554889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 0x7fbb1010a4f0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.240+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2671554889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb1010b3f0 0x7fbb10109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.240+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2671554889 >> 172.21.15.31:0/2671554889 conn(0x7fbb10070320 msgr2=0x7fbb100727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.244+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2671554889 shutdown_connections 2024-04-25T16:07:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.247+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2671554889 wait complete. 2024-04-25T16:07:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.249+0000 7fbb0f59e640 1 Processor -- start 2024-04-25T16:07:01.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.253+0000 7fbb0f59e640 1 -- start start 2024-04-25T16:07:01.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.253+0000 7fbb0f59e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb10078da0 0x7fbb10084890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.253+0000 7fbb0f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 0x7fbb10084dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.253+0000 7fbb0f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb1010b3f0 0x7fbb101b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.253+0000 7fbb0f59e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1007ead0 con 0x7fbb1010a0e0 2024-04-25T16:07:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.253+0000 7fbb0f59e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1007e950 con 0x7fbb10078da0 2024-04-25T16:07:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.253+0000 7fbb0f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbb1007ec50 con 0x7fbb1010b3f0 2024-04-25T16:07:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.254+0000 7fbb0dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 0x7fbb10084dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.254+0000 7fbb0dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 0x7fbb10084dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36980/0 (socket says 172.21.15.31:36980) 2024-04-25T16:07:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.254+0000 7fbb0dd9b640 1 -- 172.21.15.31:0/2953485759 learned_addr learned my addr 172.21.15.31:0/2953485759 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.254+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb10078da0 0x7fbb10084890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.256+0000 7fbb0dd9b640 1 -- 172.21.15.31:0/2953485759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb1010b3f0 msgr2=0x7fbb101b8bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:07:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.256+0000 7fbb0dd9b640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb1010b3f0 0x7fbb101b8bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.256+0000 7fbb0dd9b640 1 -- 172.21.15.31:0/2953485759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb10078da0 msgr2=0x7fbb10084890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.256+0000 7fbb0dd9b640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb10078da0 0x7fbb10084890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.256+0000 7fbb0dd9b640 1 -- 172.21.15.31:0/2953485759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb101bd5e0 con 0x7fbb1010a0e0 2024-04-25T16:07:01.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.260+0000 7fbb0dd9b640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 0x7fbb10084dd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbb04033870 tx=0x7fbb04032940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.266+0000 7fbaff7fe640 1 -- 172.21.15.31:0/2953485759 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04040070 con 0x7fbb1010a0e0 2024-04-25T16:07:01.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.266+0000 7fbaff7fe640 1 -- 172.21.15.31:0/2953485759 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb040035f0 con 0x7fbb1010a0e0 2024-04-25T16:07:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.266+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2953485759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb101b9100 con 0x7fbb1010a0e0 2024-04-25T16:07:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.266+0000 7fbaff7fe640 1 -- 172.21.15.31:0/2953485759 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb04037030 con 0x7fbb1010a0e0 2024-04-25T16:07:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.266+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2953485759 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb101b95f0 con 0x7fbb1010a0e0 2024-04-25T16:07:01.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.271+0000 7fbb1bdf4740 1 Processor -- start 2024-04-25T16:07:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.272+0000 7fbb1bdf4740 1 -- start start 2024-04-25T16:07:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.272+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 0x5584841379e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.273+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484137f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.273+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d746b0 0x558484138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.273+0000 7fbb1bdf4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55848409bd80 con 0x558483d746b0 2024-04-25T16:07:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.273+0000 7fbb1bdf4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558484055a20 con 0x558483d6c4d0 2024-04-25T16:07:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.273+0000 7fbb1bdf4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558484097d40 con 0x558483d95390 2024-04-25T16:07:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.275+0000 7fbb0ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 0x5584841379e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.275+0000 7fbb0e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484137f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.275+0000 7fbb0ed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 0x5584841379e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39268/0 (socket says 172.21.15.31:39268) 2024-04-25T16:07:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.275+0000 7fbb0ed9d640 1 -- 172.21.15.31:0/4206736722 learned_addr learned my addr 172.21.15.31:0/4206736722 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.276+0000 7fbb0e59c640 1 -- 172.21.15.31:0/4206736722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 msgr2=0x5584841379e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.276+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/4206736722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 0x5584841379e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.276+0000 7fbb0e59c640 1 -- 172.21.15.31:0/4206736722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d746b0 msgr2=0x558484138bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:07:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.276+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/4206736722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d746b0 0x558484138bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.276+0000 7fbb0e59c640 1 -- 172.21.15.31:0/4206736722 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558484060c30 con 0x558483d6c4d0 2024-04-25T16:07:01.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.277+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/4206736722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484137f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009970 tx=0x7fbb00009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7c368ef7d32a399 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.280+0000 7fbaff7fe640 1 -- 172.21.15.31:0/2953485759 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fbb04003aa0 con 0x7fbb1010a0e0 2024-04-25T16:07:01.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.282+0000 7fbaff7fe640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbaf80afcf0 0x7fbaf80b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.282+0000 7fbafcff9640 1 -- 172.21.15.31:0/4206736722 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00017070 con 0x558483d6c4d0 2024-04-25T16:07:01.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.282+0000 7fbafcff9640 1 -- 172.21.15.31:0/4206736722 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb00013470 con 0x558483d6c4d0 2024-04-25T16:07:01.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.282+0000 7fbafcff9640 1 -- 172.21.15.31:0/4206736722 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00012630 con 0x558483d6c4d0 2024-04-25T16:07:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.283+0000 7fbaff7fe640 1 -- 172.21.15.31:0/2953485759 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbb040ff6f0 con 0x7fbb1010a0e0 2024-04-25T16:07:01.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.285+0000 7fbb0ed9d640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbaf80afcf0 0x7fbaf80b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.286+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/4206736722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 msgr2=0x558484137f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.286+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/4206736722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484137f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009970 tx=0x7fbb00009e30 comp rx=0 tx=0).stop 2024-04-25T16:07:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.287+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/4206736722 shutdown_connections 2024-04-25T16:07:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.287+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/4206736722 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d746b0 0x558484138bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.287+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/4206736722 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484137f60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.287+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/4206736722 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 0x5584841379e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.287+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/4206736722 >> 172.21.15.31:0/4206736722 conn(0x558483d894e0 msgr2=0x5584841331f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:01.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.289+0000 7fbb0ed9d640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbaf80afcf0 0x7fbaf80b2200 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x558484138a30 tx=0x7fbb0800c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.289+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/4206736722 shutdown_connections 2024-04-25T16:07:01.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.293+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/4206736722 wait complete. 2024-04-25T16:07:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.293+0000 7fbb1bdf4740 1 -- start start 2024-04-25T16:07:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.293+0000 7fbb1bdf4740 1 Processor -- start 2024-04-25T16:07:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.294+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x5584841417d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.294+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d746b0 0x558484141d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.294+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d95390 0x558484148c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.294+0000 7fbb1bdf4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558484139700 con 0x558483d6c4d0 2024-04-25T16:07:01.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.294+0000 7fbb1bdf4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x558484022dc0 con 0x558483d95390 2024-04-25T16:07:01.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.294+0000 7fbb1bdf4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558484139880 con 0x558483d746b0 2024-04-25T16:07:01.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.294+0000 7fbb1bdf4740 10 client.0 _ll_register_callbacks cb 0x55848403e600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:01.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.295+0000 7fbb0dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d746b0 0x558484141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.295+0000 7fbb0dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d746b0 0x558484141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39270/0 (socket says 172.21.15.31:39270) 2024-04-25T16:07:01.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.295+0000 7fbb0dd9b640 1 -- 172.21.15.31:0/272306880 learned_addr learned my addr 172.21.15.31:0/272306880 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:01.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.296+0000 7fbb0ed9d640 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d95390 0x558484148c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.296+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x5584841417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.296+0000 7fbb0dd9b640 1 -- 172.21.15.31:0/272306880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d95390 msgr2=0x558484148c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.296+0000 7fbb0dd9b640 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d95390 0x558484148c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.296+0000 7fbb0dd9b640 1 -- 172.21.15.31:0/272306880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 msgr2=0x5584841417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.296+0000 7fbb0dd9b640 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x5584841417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.296+0000 7fbb0dd9b640 1 -- 172.21.15.31:0/272306880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558483dad3f0 con 0x558483d746b0 2024-04-25T16:07:01.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.312+0000 7fbb0dd9b640 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d746b0 0x558484141d10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbb040c4df0 tx=0x7fbb040c7590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2ab74ba224e5459 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.314+0000 7fbaf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.314+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/272306880 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb040e1060 con 0x558483d746b0 2024-04-25T16:07:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.314+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/272306880 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbb040c3e10 con 0x558483d746b0 2024-04-25T16:07:01.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.314+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/272306880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb00013470 con 0x558483d746b0 2024-04-25T16:07:01.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.317+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/272306880 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb040e2250 con 0x558483d746b0 2024-04-25T16:07:01.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.326+0000 7fbb1bdf4740 10 client.26386 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:01.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.326+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/272306880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558483e777d0 con 0x558483d746b0 2024-04-25T16:07:01.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.328+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/272306880 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb00013470 con 0x558483d746b0 2024-04-25T16:07:01.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.328+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/272306880 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fbb040d0550 con 0x558483d746b0 2024-04-25T16:07:01.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.328+0000 7fbb1bdf4740 10 client.26386 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:01.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.328+0000 7fbb1bdf4740 10 client.26386 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:07:01.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.328+0000 7fbb1bdf4740 20 client.26386 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:01.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.328+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x558483d53f60 0x55848414f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.328+0000 7fbb1bdf4740 4 client.26386 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:01.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.328+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/272306880 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584840a9990 con 0x558483d53f60 2024-04-25T16:07:01.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.329+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x558483d53f60 0x55848414f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.330+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x558483d53f60 0x55848414f390 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=956ab4b8cb554d49 server_cookie=62f104206ec93efb in_seq=0 out_seq=0 2024-04-25T16:07:01.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.330+0000 7fbaf5ffb640 10 client.26386 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:07:01.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.339+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/272306880 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5584840a9990 con 0x558483d53f60 2024-04-25T16:07:01.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.339+0000 7fbaf5ffb640 10 client.26386 handle_command_reply: tid=0 2024-04-25T16:07:01.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.339+0000 7fbb1bdf4740 1 client.26386 shutdown 2024-04-25T16:07:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.342+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/272306880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d746b0 msgr2=0x558484141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.342+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d746b0 0x558484141d10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbb040c4df0 tx=0x7fbb040c7590 comp rx=0 tx=0).stop 2024-04-25T16:07:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.342+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/272306880 shutdown_connections 2024-04-25T16:07:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.342+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d95390 0x558484148c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.342+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d746b0 0x558484141d10 secure :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbb040c4df0 tx=0x7fbb040c7590 comp rx=0 tx=0).stop 2024-04-25T16:07:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.342+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x558483d53f60 0x55848414f390 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.342+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/272306880 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x5584841417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.342+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/272306880 >> 172.21.15.31:0/272306880 conn(0x558483d894e0 msgr2=0x558484136100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:01.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.342+0000 7fbaf5ffb640 0 client.26386 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:07:01.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.345+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/272306880 shutdown_connections 2024-04-25T16:07:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.349+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/272306880 wait complete. 2024-04-25T16:07:01.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.349+0000 7fbb1bdf4740 20 client.26386 trim_cache size 0 max 16384 2024-04-25T16:07:01.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.394+0000 7fbb1bdf4740 1 Processor -- start 2024-04-25T16:07:01.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.395+0000 7fbb1bdf4740 1 -- start start 2024-04-25T16:07:01.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.395+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x558484132c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.395+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 0x558484133160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.396+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d293a0 0x55848413fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.396+0000 7fbb0e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x558484132c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.396+0000 7fbb0e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x558484132c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:36990/0 (socket says 172.21.15.31:36990) 2024-04-25T16:07:01.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.396+0000 7fbb0e59c640 1 -- 172.21.15.31:0/2667756724 learned_addr learned my addr 172.21.15.31:0/2667756724 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:01.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.396+0000 7fbb0dd9b640 1 --2- 172.21.15.31:0/2667756724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 0x558484133160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.397+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/2667756724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf000e550 con 0x558483d6c4d0 2024-04-25T16:07:01.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.397+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/2667756724 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf0010090 con 0x558483d293a0 2024-04-25T16:07:01.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.397+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/2667756724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf000ec20 con 0x558483d95390 2024-04-25T16:07:01.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.397+0000 7fbb0ed9d640 1 --2- 172.21.15.31:0/2667756724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d293a0 0x55848413fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.398+0000 7fbb0e59c640 1 -- 172.21.15.31:0/2667756724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 msgr2=0x558484133160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.398+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/2667756724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 0x558484133160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.398+0000 7fbb0e59c640 1 -- 172.21.15.31:0/2667756724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d293a0 msgr2=0x55848413fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.398+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/2667756724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d293a0 0x55848413fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.398+0000 7fbb0e59c640 1 -- 172.21.15.31:0/2667756724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558484139970 con 0x558483d6c4d0 2024-04-25T16:07:01.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.398+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/2667756724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x558484132c20 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x558484137700 tx=0x7fbb000047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9468102ca65dd622 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.399+0000 7fbaf7fff640 1 -- 172.21.15.31:0/2667756724 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00017040 con 0x558483d6c4d0 2024-04-25T16:07:01.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.399+0000 7fbaf7fff640 1 -- 172.21.15.31:0/2667756724 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb0002b070 con 0x558483d6c4d0 2024-04-25T16:07:01.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.399+0000 7fbaf7fff640 1 -- 172.21.15.31:0/2667756724 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb00013af0 con 0x558483d6c4d0 2024-04-25T16:07:01.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.400+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/2667756724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 msgr2=0x558484132c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.400+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/2667756724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x558484132c20 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x558484137700 tx=0x7fbb000047d0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.403+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/2667756724 shutdown_connections 2024-04-25T16:07:01.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.403+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/2667756724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d293a0 0x55848413fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.403+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/2667756724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d95390 0x558484133160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.403+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/2667756724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d6c4d0 0x558484132c20 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.403+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/2667756724 >> 172.21.15.31:0/2667756724 conn(0x558483d894e0 msgr2=0x55848413c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:01.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.404+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/2667756724 shutdown_connections 2024-04-25T16:07:01.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.405+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/2667756724 wait complete. 2024-04-25T16:07:01.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.406+0000 7fbb1bdf4740 1 -- start start 2024-04-25T16:07:01.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.406+0000 7fbb1bdf4740 1 Processor -- start 2024-04-25T16:07:01.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.407+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d293a0 0x5584841454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.407+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484145a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.407+0000 7fbb1bdf4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d95390 0x558484146950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.407+0000 7fbb0dd9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484145a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.407+0000 7fbb0e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d293a0 0x5584841454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.407+0000 7fbb0dd9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484145a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40204/0 (socket says 172.21.15.31:40204) 2024-04-25T16:07:01.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.407+0000 7fbb0e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d293a0 0x5584841454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39290/0 (socket says 172.21.15.31:39290) 2024-04-25T16:07:01.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.408+0000 7fbb0dd9b640 1 -- 172.21.15.31:0/3622266071 learned_addr learned my addr 172.21.15.31:0/3622266071 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:01.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.408+0000 7fbb0ed9d640 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d95390 0x558484146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.408+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf000f9c0 con 0x558483d95390 2024-04-25T16:07:01.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.408+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf000f2f0 con 0x558483d6c4d0 2024-04-25T16:07:01.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.408+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbaf000c530 con 0x558483d293a0 2024-04-25T16:07:01.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.408+0000 7fbb1bdf4740 10 client.0 _ll_register_callbacks cb 0x7fbb04009260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.409+0000 7fbb0ed9d640 1 -- 172.21.15.31:0/3622266071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d293a0 msgr2=0x5584841454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.409+0000 7fbb0ed9d640 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d293a0 0x5584841454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.409+0000 7fbb0ed9d640 1 -- 172.21.15.31:0/3622266071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 msgr2=0x558484145a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.409+0000 7fbb0ed9d640 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484145a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.409+0000 7fbb0ed9d640 1 -- 172.21.15.31:0/3622266071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558484146f10 con 0x558483d95390 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.409+0000 7fbb0ed9d640 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d95390 0x558484146950 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005fd0 tx=0x7fbb08004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34ba04dc6e42f69f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.409+0000 7fbaf5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.409+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/3622266071 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb08016e20 con 0x558483d95390 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.410+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/3622266071 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fbb08003b80 con 0x558483d95390 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.410+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb0002b070 con 0x558483d95390 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.410+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/3622266071 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbb08003d80 con 0x558483d95390 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.410+0000 7fbb1bdf4740 10 client.16791 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.410+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558484148240 con 0x558483d95390 2024-04-25T16:07:01.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.410+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/3622266071 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb08029070 con 0x558483d95390 2024-04-25T16:07:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.411+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/3622266071 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fbb080045d0 con 0x558483d95390 2024-04-25T16:07:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.411+0000 7fbb1bdf4740 10 client.16791 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.411+0000 7fbb1bdf4740 10 client.16791 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:07:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.411+0000 7fbb1bdf4740 20 client.16791 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.411+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5584841b7b50 0x5584841b9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.411+0000 7fbb1bdf4740 4 client.16791 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:07:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.411+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fbb0002b070 con 0x5584841b7b50 2024-04-25T16:07:01.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.412+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5584841b7b50 0x5584841b9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.415+0000 7fbb0e59c640 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5584841b7b50 0x5584841b9f70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0b43b8e0e96479d server_cookie=fc7fec80f304356d in_seq=0 out_seq=0 2024-04-25T16:07:01.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.416+0000 7fbaf5ffb640 10 client.16791 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:07:01.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbaf5ffb640 1 -- 172.21.15.31:0/3622266071 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fbb0002b070 con 0x5584841b7b50 2024-04-25T16:07:01.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbaf5ffb640 10 client.16791 handle_command_reply: tid=0 2024-04-25T16:07:01.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbb1bdf4740 1 client.16791 shutdown 2024-04-25T16:07:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d95390 msgr2=0x558484146950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d95390 0x558484146950 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fbb08005fd0 tx=0x7fbb08004ab0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 shutdown_connections 2024-04-25T16:07:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5584841b7b50 0x5584841b9f70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558483d95390 0x558484146950 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558483d6c4d0 0x558484145a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbaf5ffb640 0 client.16791 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:07:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbb1bdf4740 1 --2- 172.21.15.31:0/3622266071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558483d293a0 0x5584841454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.417+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 >> 172.21.15.31:0/3622266071 conn(0x558483d894e0 msgr2=0x55848413fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:01.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.418+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 shutdown_connections 2024-04-25T16:07:01.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.418+0000 7fbb1bdf4740 1 -- 172.21.15.31:0/3622266071 wait complete. 2024-04-25T16:07:01.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.418+0000 7fbb1bdf4740 20 client.16791 trim_cache size 0 max 16384 2024-04-25T16:07:01.421 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:07:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbaf80afcf0 msgr2=0x7fbaf80b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbaf80afcf0 0x7fbaf80b2200 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x558484138a30 tx=0x7fbb0800c040 comp rx=0 tx=0).stop 2024-04-25T16:07:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 msgr2=0x7fbb10084dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 0x7fbb10084dd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fbb04033870 tx=0x7fbb04032940 comp rx=0 tx=0).stop 2024-04-25T16:07:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2953485759 shutdown_connections 2024-04-25T16:07:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fbaf80afcf0 0x7fbaf80b2200 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb1010b3f0 0x7fbb101b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fbb1010a0e0 0x7fbb10084dd0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 --2- 172.21.15.31:0/2953485759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbb10078da0 0x7fbb10084890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2953485759 >> 172.21.15.31:0/2953485759 conn(0x7fbb10070320 msgr2=0x7fbb10074d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2953485759 shutdown_connections 2024-04-25T16:07:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.420+0000 7fbb0f59e640 1 -- 172.21.15.31:0/2953485759 wait complete. 2024-04-25T16:07:01.457 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:07:01.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.858+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3828203161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4e410b310 msgr2=0x7fe4e410b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.858+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3828203161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4e410b310 0x7fe4e410b720 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fe4e0009900 tx=0x7fe4e002a420 comp rx=0 tx=0).stop 2024-04-25T16:07:01.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.864+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3828203161 shutdown_connections 2024-04-25T16:07:01.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.864+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3828203161 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4e4110030 0x7fe4e4110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.864+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3828203161 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4e410b310 0x7fe4e410b720 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.864+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3828203161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4e410a420 0x7fe4e410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.864+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3828203161 >> 172.21.15.31:0/3828203161 conn(0x7fe4e40704e0 msgr2=0x7fe4e4072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:01.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.864+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3828203161 shutdown_connections 2024-04-25T16:07:01.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.864+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3828203161 wait complete. 2024-04-25T16:07:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.866+0000 7fe4eb8e0640 1 Processor -- start 2024-04-25T16:07:01.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.866+0000 7fe4eb8e0640 1 -- start start 2024-04-25T16:07:01.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.867+0000 7fe4eb8e0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4e410a420 0x7fe4e41a8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.867+0000 7fe4eb8e0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4e410b310 0x7fe4e41a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.867+0000 7fe4eb8e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4e4110030 0x7fe4e41b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.867+0000 7fe4eb8e0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e411e540 con 0x7fe4e410b310 2024-04-25T16:07:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.867+0000 7fe4eb8e0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e411e3c0 con 0x7fe4e410a420 2024-04-25T16:07:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.868+0000 7fe4eb8e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e411e6c0 con 0x7fe4e4110030 2024-04-25T16:07:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.868+0000 7fe4e967c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4e410a420 0x7fe4e41a8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.868+0000 7fe4e967c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4e410a420 0x7fe4e41a8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40222/0 (socket says 172.21.15.31:40222) 2024-04-25T16:07:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.868+0000 7fe4e967c640 1 -- 172.21.15.31:0/3401165533 learned_addr learned my addr 172.21.15.31:0/3401165533 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:01.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.868+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4e410b310 0x7fe4e41a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.869+0000 7fe4e967c640 1 -- 172.21.15.31:0/3401165533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4e4110030 msgr2=0x7fe4e41b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.869+0000 7fe4e967c640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4e4110030 0x7fe4e41b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.869+0000 7fe4e967c640 1 -- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4e410b310 msgr2=0x7fe4e41a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.869+0000 7fe4e967c640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4e410b310 0x7fe4e41a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.869+0000 7fe4e967c640 1 -- 172.21.15.31:0/3401165533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e41b0a70 con 0x7fe4e410a420 2024-04-25T16:07:01.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.870+0000 7fe4e967c640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4e410a420 0x7fe4e41a8f30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00b9e0 tx=0x7fe4dc00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.878+0000 7fe4da7fc640 1 -- 172.21.15.31:0/3401165533 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc019070 con 0x7fe4e410a420 2024-04-25T16:07:01.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.878+0000 7fe4da7fc640 1 -- 172.21.15.31:0/3401165533 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4dc00cba0 con 0x7fe4e410a420 2024-04-25T16:07:01.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.879+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3401165533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e41b0d60 con 0x7fe4e410a420 2024-04-25T16:07:01.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.879+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3401165533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e41b1220 con 0x7fe4e410a420 2024-04-25T16:07:01.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.880+0000 7fe4da7fc640 1 -- 172.21.15.31:0/3401165533 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0125a0 con 0x7fe4e410a420 2024-04-25T16:07:01.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.897+0000 7fe4da7fc640 1 -- 172.21.15.31:0/3401165533 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe4dc01c070 con 0x7fe4e410a420 2024-04-25T16:07:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.917+0000 7fe4da7fc640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe4d00afd00 0x7fe4d00b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.924+0000 7fe4da7fc640 1 -- 172.21.15.31:0/3401165533 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe4dc0de0f0 con 0x7fe4e410a420 2024-04-25T16:07:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.927+0000 7fe4f2794740 1 Processor -- start 2024-04-25T16:07:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.929+0000 7fe4f2794740 1 -- start start 2024-04-25T16:07:01.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.931+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e73390 0x55de09215810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.931+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e4a4d0 0x55de09215d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.931+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de09216a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.931+0000 7fe4f2794740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55de09179d80 con 0x55de08e73390 2024-04-25T16:07:01.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.931+0000 7fe4f2794740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55de09133a20 con 0x55de08e526b0 2024-04-25T16:07:01.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.931+0000 7fe4f2794740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55de09175d40 con 0x55de08e4a4d0 2024-04-25T16:07:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.933+0000 7fe4e9e7d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de09216a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.933+0000 7fe4e9e7d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de09216a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40230/0 (socket says 172.21.15.31:40230) 2024-04-25T16:07:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.933+0000 7fe4e9e7d640 1 -- 172.21.15.31:0/3147919566 learned_addr learned my addr 172.21.15.31:0/3147919566 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:01.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.942+0000 7fe4e967c640 1 --2- 172.21.15.31:0/3147919566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e73390 0x55de09215810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.942+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe4d00afd00 0x7fe4d00b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.944+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/3147919566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e4a4d0 0x55de09215d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.946+0000 7fe4e9e7d640 1 -- 172.21.15.31:0/3147919566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e4a4d0 msgr2=0x55de09215d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.946+0000 7fe4e9e7d640 1 --2- 172.21.15.31:0/3147919566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e4a4d0 0x55de09215d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.946+0000 7fe4e9e7d640 1 -- 172.21.15.31:0/3147919566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e73390 msgr2=0x55de09215810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.946+0000 7fe4e9e7d640 1 --2- 172.21.15.31:0/3147919566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e73390 0x55de09215810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.946+0000 7fe4e9e7d640 1 -- 172.21.15.31:0/3147919566 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0913ec30 con 0x55de08e526b0 2024-04-25T16:07:01.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:01 smithi031 ceph-mon[29140]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 801 MiB/s wr, 1.77k op/s 2024-04-25T16:07:01.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.949+0000 7fe4e9e7d640 1 --2- 172.21.15.31:0/3147919566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de09216a20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400ba90 tx=0x7fe4d400bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7840f0ee7e117409 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.951+0000 7fe4cf7fe640 1 -- 172.21.15.31:0/3147919566 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4016070 con 0x55de08e526b0 2024-04-25T16:07:01.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.951+0000 7fe4cf7fe640 1 -- 172.21.15.31:0/3147919566 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4d40054d0 con 0x55de08e526b0 2024-04-25T16:07:01.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.951+0000 7fe4cf7fe640 1 -- 172.21.15.31:0/3147919566 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4010460 con 0x55de08e526b0 2024-04-25T16:07:01.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.953+0000 7fe4f2794740 1 -- 172.21.15.31:0/3147919566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 msgr2=0x55de09216a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:01.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.953+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3147919566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de09216a20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400ba90 tx=0x7fe4d400bf50 comp rx=0 tx=0).stop 2024-04-25T16:07:01.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.955+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe4d00afd00 0x7fe4d00b2210 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00096c0 tx=0x7fe4e0009560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.955+0000 7fe4f2794740 1 -- 172.21.15.31:0/3147919566 shutdown_connections 2024-04-25T16:07:01.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.955+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3147919566 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de09216a20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.955+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3147919566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e4a4d0 0x55de09215d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.955+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3147919566 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e73390 0x55de09215810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.955+0000 7fe4f2794740 1 -- 172.21.15.31:0/3147919566 >> 172.21.15.31:0/3147919566 conn(0x55de08e674e0 msgr2=0x55de09211020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:01.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.974+0000 7fe4f2794740 1 -- 172.21.15.31:0/3147919566 shutdown_connections 2024-04-25T16:07:01.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.974+0000 7fe4f2794740 1 -- 172.21.15.31:0/3147919566 wait complete. 2024-04-25T16:07:01.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.975+0000 7fe4f2794740 1 -- start start 2024-04-25T16:07:01.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.975+0000 7fe4f2794740 1 Processor -- start 2024-04-25T16:07:01.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.976+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de0921f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.976+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 0x55de0921fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.976+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e31f60 0x55de09226a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.976+0000 7fe4f2794740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55de09217530 con 0x55de08e4a4d0 2024-04-25T16:07:01.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.976+0000 7fe4f2794740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55de09100dc0 con 0x55de08e31f60 2024-04-25T16:07:01.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.976+0000 7fe4f2794740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55de092176b0 con 0x55de08e526b0 2024-04-25T16:07:01.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.976+0000 7fe4f2794740 10 client.0 _ll_register_callbacks cb 0x55de0911c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:01.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.978+0000 7fe4e967c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de0921f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.978+0000 7fe4e967c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de0921f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37046/0 (socket says 172.21.15.31:37046) 2024-04-25T16:07:01.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.978+0000 7fe4e967c640 1 -- 172.21.15.31:0/957482055 learned_addr learned my addr 172.21.15.31:0/957482055 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:01.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.978+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 0x55de0921fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.979+0000 7fe4e9e7d640 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e31f60 0x55de09226a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:01.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.980+0000 7fe4e967c640 1 -- 172.21.15.31:0/957482055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 msgr2=0x55de0921fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.980+0000 7fe4e967c640 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 0x55de0921fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.980+0000 7fe4e967c640 1 -- 172.21.15.31:0/957482055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e31f60 msgr2=0x55de09226a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:01.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.980+0000 7fe4e967c640 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e31f60 0x55de09226a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:01.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.980+0000 7fe4e967c640 1 -- 172.21.15.31:0/957482055 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de08e8b3f0 con 0x55de08e4a4d0 2024-04-25T16:07:01.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.983+0000 7fe4e967c640 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de0921f5e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0a37f0 tx=0x7fe4dc0a61f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c684ae174796540 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:01.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.988+0000 7fe4ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:07:01.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.988+0000 7fe4ccff9640 1 -- 172.21.15.31:0/957482055 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0bf070 con 0x55de08e4a4d0 2024-04-25T16:07:01.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.988+0000 7fe4f2794740 1 -- 172.21.15.31:0/957482055 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d40054d0 con 0x55de08e4a4d0 2024-04-25T16:07:01.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.988+0000 7fe4ccff9640 1 -- 172.21.15.31:0/957482055 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe4dc0aef50 con 0x55de08e4a4d0 2024-04-25T16:07:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.995+0000 7fe4ccff9640 1 -- 172.21.15.31:0/957482055 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe4dc0c6050 con 0x55de08e4a4d0 2024-04-25T16:07:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.995+0000 7fe4f2794740 10 client.16803 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:01.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.995+0000 7fe4f2794740 1 -- 172.21.15.31:0/957482055 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55de08f557d0 con 0x55de08e4a4d0 2024-04-25T16:07:01.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.996+0000 7fe4ccff9640 1 -- 172.21.15.31:0/957482055 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4d40054d0 con 0x55de08e4a4d0 2024-04-25T16:07:01.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.997+0000 7fe4ccff9640 1 -- 172.21.15.31:0/957482055 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe4dc0a5250 con 0x55de08e4a4d0 2024-04-25T16:07:01.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.998+0000 7fe4f2794740 10 client.16803 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:01.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.998+0000 7fe4f2794740 10 client.16803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:01.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.998+0000 7fe4f2794740 20 client.16803 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:01.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.998+0000 7fe4f2794740 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55de08e073a0 0x55de0923e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:01.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.998+0000 7fe4f2794740 4 client.16803 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:01.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:01.998+0000 7fe4f2794740 1 -- 172.21.15.31:0/957482055 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de09187990 con 0x55de08e073a0 2024-04-25T16:07:02.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.003+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55de08e073a0 0x55de0923e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.012+0000 7fe4ccff9640 10 client.16803 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:02.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.012+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55de08e073a0 0x55de0923e430 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f204a699569c7f42 server_cookie=52aab26d3ea7394 in_seq=0 out_seq=0 2024-04-25T16:07:02.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.015+0000 7fe4ccff9640 1 -- 172.21.15.31:0/957482055 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55de09187990 con 0x55de08e073a0 2024-04-25T16:07:02.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.015+0000 7fe4ccff9640 10 client.16803 handle_command_reply: tid=0 2024-04-25T16:07:02.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.016+0000 7fe4f2794740 1 client.16803 shutdown 2024-04-25T16:07:02.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.022+0000 7fe4f2794740 1 -- 172.21.15.31:0/957482055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 msgr2=0x55de0921f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.022+0000 7fe4f2794740 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de0921f5e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc0a37f0 tx=0x7fe4dc0a61f0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.023+0000 7fe4f2794740 1 -- 172.21.15.31:0/957482055 shutdown_connections 2024-04-25T16:07:02.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.023+0000 7fe4f2794740 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55de08e073a0 0x55de0923e430 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.023+0000 7fe4f2794740 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e31f60 0x55de09226a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.023+0000 7fe4f2794740 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 0x55de0921fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.023+0000 7fe4f2794740 1 --2- 172.21.15.31:0/957482055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de0921f5e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.023+0000 7fe4ccff9640 0 client.16803 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:02.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.023+0000 7fe4f2794740 1 -- 172.21.15.31:0/957482055 >> 172.21.15.31:0/957482055 conn(0x55de08e674e0 msgr2=0x55de09210040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.027+0000 7fe4f2794740 1 -- 172.21.15.31:0/957482055 shutdown_connections 2024-04-25T16:07:02.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.029+0000 7fe4f2794740 1 -- 172.21.15.31:0/957482055 wait complete. 2024-04-25T16:07:02.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.030+0000 7fe4f2794740 20 client.16803 trim_cache size 0 max 16384 2024-04-25T16:07:02.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.095+0000 7fe4f2794740 1 Processor -- start 2024-04-25T16:07:02.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.095+0000 7fe4f2794740 1 -- start start 2024-04-25T16:07:02.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.096+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e31f60 0x55de09238450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.096+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de09238990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.096+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de0921f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.096+0000 7fe4f2794740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c800e540 con 0x55de08e4a4d0 2024-04-25T16:07:02.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.096+0000 7fe4f2794740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c8010080 con 0x55de08e526b0 2024-04-25T16:07:02.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.096+0000 7fe4f2794740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c800ec10 con 0x55de08e31f60 2024-04-25T16:07:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.100+0000 7fe4e9e7d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de0921f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.101+0000 7fe4e9e7d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de0921f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40248/0 (socket says 172.21.15.31:40248) 2024-04-25T16:07:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.101+0000 7fe4e9e7d640 1 -- 172.21.15.31:0/2379807679 learned_addr learned my addr 172.21.15.31:0/2379807679 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.101+0000 7fe4e967c640 1 --2- 172.21.15.31:0/2379807679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e31f60 0x55de09238450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.102+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/2379807679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de09238990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.103+0000 7fe4e967c640 1 -- 172.21.15.31:0/2379807679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 msgr2=0x55de0921f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:02.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.103+0000 7fe4e967c640 1 --2- 172.21.15.31:0/2379807679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de0921f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.103+0000 7fe4e967c640 1 -- 172.21.15.31:0/2379807679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 msgr2=0x55de09238990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:02.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.103+0000 7fe4e967c640 1 --2- 172.21.15.31:0/2379807679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de09238990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.103+0000 7fe4e967c640 1 -- 172.21.15.31:0/2379807679 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de0921a9e0 con 0x55de08e31f60 2024-04-25T16:07:02.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.105+0000 7fe4e967c640 1 --2- 172.21.15.31:0/2379807679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e31f60 0x55de09238450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00bfd0 tx=0x7fe4dc009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d5dda27a3c4fe58 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:02.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.107+0000 7fe4ceffd640 1 -- 172.21.15.31:0/2379807679 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4dc0bf070 con 0x55de08e31f60 2024-04-25T16:07:02.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.107+0000 7fe4ceffd640 1 -- 172.21.15.31:0/2379807679 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4dc0af3e0 con 0x55de08e31f60 2024-04-25T16:07:02.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.108+0000 7fe4f2794740 1 -- 172.21.15.31:0/2379807679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e31f60 msgr2=0x55de09238450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.108+0000 7fe4f2794740 1 --2- 172.21.15.31:0/2379807679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e31f60 0x55de09238450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00bfd0 tx=0x7fe4dc009920 comp rx=0 tx=0).stop 2024-04-25T16:07:02.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.110+0000 7fe4f2794740 1 -- 172.21.15.31:0/2379807679 shutdown_connections 2024-04-25T16:07:02.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.110+0000 7fe4f2794740 1 --2- 172.21.15.31:0/2379807679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e526b0 0x55de0921f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.110+0000 7fe4f2794740 1 --2- 172.21.15.31:0/2379807679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e4a4d0 0x55de09238990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.110+0000 7fe4f2794740 1 --2- 172.21.15.31:0/2379807679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e31f60 0x55de09238450 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.110+0000 7fe4f2794740 1 -- 172.21.15.31:0/2379807679 >> 172.21.15.31:0/2379807679 conn(0x55de08e674e0 msgr2=0x55de0921a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:02.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:01 smithi063 ceph-mon[32413]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 801 MiB/s wr, 1.77k op/s 2024-04-25T16:07:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.118+0000 7fe4f2794740 1 -- 172.21.15.31:0/2379807679 shutdown_connections 2024-04-25T16:07:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.119+0000 7fe4f2794740 1 -- 172.21.15.31:0/2379807679 wait complete. 2024-04-25T16:07:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.120+0000 7fe4f2794740 1 -- start start 2024-04-25T16:07:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.120+0000 7fe4f2794740 1 Processor -- start 2024-04-25T16:07:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.121+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e31f60 0x55de09224d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.121+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e4a4d0 0x55de09225260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.121+0000 7fe4f2794740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 0x55de092a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.121+0000 7fe4f2794740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c800f9b0 con 0x55de08e31f60 2024-04-25T16:07:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.121+0000 7fe4f2794740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c800f2e0 con 0x55de08e4a4d0 2024-04-25T16:07:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.121+0000 7fe4f2794740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c800c520 con 0x55de08e526b0 2024-04-25T16:07:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.121+0000 7fe4f2794740 10 client.0 _ll_register_callbacks cb 0x7fe4dc0a2480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.122+0000 7fe4e8e7b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e4a4d0 0x55de09225260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.122+0000 7fe4e8e7b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e4a4d0 0x55de09225260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40264/0 (socket says 172.21.15.31:40264) 2024-04-25T16:07:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.122+0000 7fe4e8e7b640 1 -- 172.21.15.31:0/3214688506 learned_addr learned my addr 172.21.15.31:0/3214688506 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.122+0000 7fe4e9e7d640 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 0x55de092a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.123+0000 7fe4e8e7b640 1 -- 172.21.15.31:0/3214688506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 msgr2=0x55de092a47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.123+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 0x55de092a47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.123+0000 7fe4e8e7b640 1 -- 172.21.15.31:0/3214688506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e31f60 msgr2=0x55de09224d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.123+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e31f60 0x55de09224d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.123+0000 7fe4e8e7b640 1 -- 172.21.15.31:0/3214688506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de09225b20 con 0x55de08e4a4d0 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.123+0000 7fe4e8e7b640 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e4a4d0 0x55de09225260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe4e002aa10 tx=0x7fe4e0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=650017e9be4515d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.124+0000 7fe4ccff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.124+0000 7fe4ccff9640 1 -- 172.21.15.31:0/3214688506 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4e005a060 con 0x55de08e4a4d0 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.124+0000 7fe4ccff9640 1 -- 172.21.15.31:0/3214688506 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe4e0003a00 con 0x55de08e4a4d0 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.124+0000 7fe4f2794740 1 -- 172.21.15.31:0/3214688506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55de09181800 con 0x55de08e4a4d0 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.125+0000 7fe4ccff9640 1 -- 172.21.15.31:0/3214688506 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe4e0035030 con 0x55de08e4a4d0 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.125+0000 7fe4f2794740 10 client.26438 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.125+0000 7fe4f2794740 1 -- 172.21.15.31:0/3214688506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55de092a5f20 con 0x55de08e4a4d0 2024-04-25T16:07:02.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.126+0000 7fe4ccff9640 1 -- 172.21.15.31:0/3214688506 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55de09181800 con 0x55de08e4a4d0 2024-04-25T16:07:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.126+0000 7fe4ccff9640 1 -- 172.21.15.31:0/3214688506 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe4e0054070 con 0x55de08e4a4d0 2024-04-25T16:07:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.126+0000 7fe4f2794740 10 client.26438 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.126+0000 7fe4f2794740 10 client.26438 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.126+0000 7fe4f2794740 20 client.26438 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.126+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55de092a68b0 0x55de092a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.126+0000 7fe4f2794740 4 client.26438 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:07:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.126+0000 7fe4f2794740 1 -- 172.21.15.31:0/3214688506 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55de090fc610 con 0x55de092a68b0 2024-04-25T16:07:02.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.130+0000 7fe4e967c640 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55de092a68b0 0x55de092a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:01 smithi186 ceph-mon[33258]: pgmap v522: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 801 MiB/s wr, 1.77k op/s 2024-04-25T16:07:02.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.139+0000 7fe4e967c640 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55de092a68b0 0x55de092a8d60 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cb7916f3c81159 server_cookie=8f1b266c4284ff4 in_seq=0 out_seq=0 2024-04-25T16:07:02.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.139+0000 7fe4ccff9640 10 client.26438 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:02.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.145+0000 7fe4ccff9640 1 -- 172.21.15.31:0/3214688506 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55de090fc610 con 0x55de092a68b0 2024-04-25T16:07:02.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.145+0000 7fe4ccff9640 10 client.26438 handle_command_reply: tid=0 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.152+0000 7fe4f2794740 1 client.26438 shutdown 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4f2794740 1 -- 172.21.15.31:0/3214688506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e4a4d0 msgr2=0x55de09225260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e4a4d0 0x55de09225260 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe4e002aa10 tx=0x7fe4e0005fd0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4f2794740 1 -- 172.21.15.31:0/3214688506 shutdown_connections 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55de092a68b0 0x55de092a8d60 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55de08e526b0 0x55de092a47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55de08e4a4d0 0x55de09225260 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4f2794740 1 --2- 172.21.15.31:0/3214688506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55de08e31f60 0x55de09224d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4f2794740 1 -- 172.21.15.31:0/3214688506 >> 172.21.15.31:0/3214688506 conn(0x55de08e674e0 msgr2=0x55de0921a2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4ccff9640 0 client.26438 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:02.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.153+0000 7fe4f2794740 1 -- 172.21.15.31:0/3214688506 shutdown_connections 2024-04-25T16:07:02.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.154+0000 7fe4f2794740 1 -- 172.21.15.31:0/3214688506 wait complete. 2024-04-25T16:07:02.159 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:07:02.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.156+0000 7fe4f2794740 20 client.26438 trim_cache size 0 max 16384 2024-04-25T16:07:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.164+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe4d00afd00 msgr2=0x7fe4d00b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.164+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe4d00afd00 0x7fe4d00b2210 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00096c0 tx=0x7fe4e0009560 comp rx=0 tx=0).stop 2024-04-25T16:07:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.164+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3401165533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4e410a420 msgr2=0x7fe4e41a8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.164+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4e410a420 0x7fe4e41a8f30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe4dc00b9e0 tx=0x7fe4dc00bea0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.171+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3401165533 shutdown_connections 2024-04-25T16:07:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.171+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe4d00afd00 0x7fe4d00b2210 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.171+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4e4110030 0x7fe4e41b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.171+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe4e410b310 0x7fe4e41a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.171+0000 7fe4eb8e0640 1 --2- 172.21.15.31:0/3401165533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe4e410a420 0x7fe4e41a8f30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.171+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3401165533 >> 172.21.15.31:0/3401165533 conn(0x7fe4e40704e0 msgr2=0x7fe4e4072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.173+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3401165533 shutdown_connections 2024-04-25T16:07:02.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:02.173+0000 7fe4eb8e0640 1 -- 172.21.15.31:0/3401165533 wait complete. 2024-04-25T16:07:02.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.852+0000 7f5896d9d640 1 -- 172.21.15.31:0/1345652044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f589010b240 msgr2=0x7f589010b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.855 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.852+0000 7f5896d9d640 1 --2- 172.21.15.31:0/1345652044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f589010b240 0x7f589010b650 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f5880009900 tx=0x7f588002a440 comp rx=0 tx=0).stop 2024-04-25T16:07:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.856+0000 7f5896d9d640 1 -- 172.21.15.31:0/1345652044 shutdown_connections 2024-04-25T16:07:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.856+0000 7f5896d9d640 1 --2- 172.21.15.31:0/1345652044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f589010ff40 0x7f5890112360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.856+0000 7f5896d9d640 1 --2- 172.21.15.31:0/1345652044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f589010b240 0x7f589010b650 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.856+0000 7f5896d9d640 1 --2- 172.21.15.31:0/1345652044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f589010a2d0 0x7f589010ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.856+0000 7f5896d9d640 1 -- 172.21.15.31:0/1345652044 >> 172.21.15.31:0/1345652044 conn(0x7f5890078c00 msgr2=0x7f589007b0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.857+0000 7f5896d9d640 1 -- 172.21.15.31:0/1345652044 shutdown_connections 2024-04-25T16:07:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.858+0000 7f5896d9d640 1 -- 172.21.15.31:0/1345652044 wait complete. 2024-04-25T16:07:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.859+0000 7f5896d9d640 1 Processor -- start 2024-04-25T16:07:02.860 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.859+0000 7f5896d9d640 1 -- start start 2024-04-25T16:07:02.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.859+0000 7f5896d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f589010a2d0 0x7f5890081a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.860+0000 7f5896d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f589010b240 0x7f5890081f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.860+0000 7f589559a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f589010b240 0x7f5890081f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.860+0000 7f5895d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f589010a2d0 0x7f5890081a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.860+0000 7f589559a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f589010b240 0x7f5890081f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39364/0 (socket says 172.21.15.31:39364) 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.861+0000 7f5895d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f589010a2d0 0x7f5890081a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40294/0 (socket says 172.21.15.31:40294) 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.860+0000 7f5896d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f589010ff40 0x7f5890082e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.861+0000 7f589559a640 1 -- 172.21.15.31:0/4743934 learned_addr learned my addr 172.21.15.31:0/4743934 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.861+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5890115bb0 con 0x7f589010ff40 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.861+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5890115a30 con 0x7f589010a2d0 2024-04-25T16:07:02.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.861+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5890115d30 con 0x7f589010b240 2024-04-25T16:07:02.864 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.862+0000 7f589659c640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f589010ff40 0x7f5890082e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.864+0000 7f589659c640 1 -- 172.21.15.31:0/4743934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f589010b240 msgr2=0x7f5890081f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.864+0000 7f589659c640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f589010b240 0x7f5890081f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.864+0000 7f589659c640 1 -- 172.21.15.31:0/4743934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f589010a2d0 msgr2=0x7f5890081a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.864+0000 7f589659c640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f589010a2d0 0x7f5890081a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.864+0000 7f589659c640 1 -- 172.21.15.31:0/4743934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58900834d0 con 0x7f589010ff40 2024-04-25T16:07:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.865+0000 7f589559a640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f589010b240 0x7f5890081f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:07:02.866 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.865+0000 7f5895d9b640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f589010a2d0 0x7f5890081a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:07:02.867 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.866+0000 7f589659c640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f589010ff40 0x7f5890082e60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f588c00b410 tx=0x7f588c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.876+0000 7f5886ffd640 1 -- 172.21.15.31:0/4743934 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c015020 con 0x7f589010ff40 2024-04-25T16:07:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.876+0000 7f5886ffd640 1 -- 172.21.15.31:0/4743934 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f588c011ce0 con 0x7f589010ff40 2024-04-25T16:07:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.876+0000 7f5886ffd640 1 -- 172.21.15.31:0/4743934 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c010da0 con 0x7f589010ff40 2024-04-25T16:07:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.877+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58900837c0 con 0x7f589010ff40 2024-04-25T16:07:02.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.877+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5890083d40 con 0x7f589010ff40 2024-04-25T16:07:02.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.881+0000 7f589ef8c740 1 Processor -- start 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.884+0000 7f5886ffd640 1 -- 172.21.15.31:0/4743934 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f588c018020 con 0x7f589010ff40 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589ef8c740 1 -- start start 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f52111b0 0x5640f55b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51e82f0 0x5640f55b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 0x5640f55b8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589ef8c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5640f5517460 con 0x5640f51e82f0 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589ef8c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5640f54d1590 con 0x5640f52111b0 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589ef8c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5640f5513420 con 0x5640f51f04d0 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589559a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f52111b0 0x5640f55b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589559a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f52111b0 0x5640f55b1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40304/0 (socket says 172.21.15.31:40304) 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.885+0000 7f589559a640 1 -- 172.21.15.31:0/3602737779 learned_addr learned my addr 172.21.15.31:0/3602737779 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:02.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.886+0000 7f5895d9b640 1 --2- 172.21.15.31:0/3602737779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51e82f0 0x5640f55b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.886+0000 7f589559a640 1 --2- 172.21.15.31:0/3602737779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 0x5640f55b8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.886+0000 7f589559a640 1 -- 172.21.15.31:0/3602737779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 msgr2=0x5640f55b8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.886+0000 7f589559a640 1 --2- 172.21.15.31:0/3602737779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 0x5640f55b8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.886+0000 7f589559a640 1 -- 172.21.15.31:0/3602737779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51e82f0 msgr2=0x5640f55b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.886+0000 7f589559a640 1 --2- 172.21.15.31:0/3602737779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51e82f0 0x5640f55b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.886+0000 7f589559a640 1 -- 172.21.15.31:0/3602737779 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640f54de0d0 con 0x5640f52111b0 2024-04-25T16:07:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.886+0000 7f589559a640 1 --2- 172.21.15.31:0/3602737779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f52111b0 0x5640f55b1b20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5880003aa0 tx=0x7f5880009460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2509670884a763d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.887+0000 7f5886ffd640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f587c0afc70 0x7f587c0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.888+0000 7f5886ffd640 1 -- 172.21.15.31:0/4743934 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f588c0de360 con 0x7f589010ff40 2024-04-25T16:07:02.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.888+0000 7f5895d9b640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f587c0afc70 0x7f587c0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.891+0000 7f5895d9b640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f587c0afc70 0x7f587c0b2180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x5640f55b2f00 tx=0x7f58880077a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:02.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.891+0000 7f587bfff640 1 -- 172.21.15.31:0/3602737779 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880035040 con 0x5640f52111b0 2024-04-25T16:07:02.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.891+0000 7f587bfff640 1 -- 172.21.15.31:0/3602737779 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5880003d00 con 0x5640f52111b0 2024-04-25T16:07:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.893+0000 7f589ef8c740 1 -- 172.21.15.31:0/3602737779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f52111b0 msgr2=0x5640f55b1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.893+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3602737779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f52111b0 0x5640f55b1b20 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5880003aa0 tx=0x7f5880009460 comp rx=0 tx=0).stop 2024-04-25T16:07:02.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.893+0000 7f589ef8c740 1 -- 172.21.15.31:0/3602737779 shutdown_connections 2024-04-25T16:07:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.893+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3602737779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 0x5640f55b8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.893+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3602737779 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51e82f0 0x5640f55b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.893+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3602737779 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f52111b0 0x5640f55b1b20 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5880003aa0 tx=0x7f5880009460 comp rx=0 tx=0).stop 2024-04-25T16:07:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.893+0000 7f589ef8c740 1 -- 172.21.15.31:0/3602737779 >> 172.21.15.31:0/3602737779 conn(0x5640f5205300 msgr2=0x5640f55ad4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:02.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.895+0000 7f589ef8c740 1 -- 172.21.15.31:0/3602737779 shutdown_connections 2024-04-25T16:07:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.895+0000 7f589ef8c740 1 -- 172.21.15.31:0/3602737779 wait complete. 2024-04-25T16:07:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.896+0000 7f589ef8c740 1 -- start start 2024-04-25T16:07:02.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.896+0000 7f589ef8c740 1 Processor -- start 2024-04-25T16:07:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.899+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51e82f0 0x5640f55bb980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.899+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51f04d0 0x5640f55bbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.899+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51cfd80 0x5640f55c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.899+0000 7f589ef8c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5640f55b99e0 con 0x5640f51e82f0 2024-04-25T16:07:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.899+0000 7f589ef8c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5640f549eb60 con 0x5640f51f04d0 2024-04-25T16:07:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.899+0000 7f589ef8c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5640f55b9b60 con 0x5640f51cfd80 2024-04-25T16:07:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.899+0000 7f589ef8c740 10 client.0 _ll_register_callbacks cb 0x5640f54c32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.900+0000 7f589559a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51f04d0 0x5640f55bbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.900+0000 7f589559a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51f04d0 0x5640f55bbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40320/0 (socket says 172.21.15.31:40320) 2024-04-25T16:07:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.900+0000 7f589559a640 1 -- 172.21.15.31:0/3377454420 learned_addr learned my addr 172.21.15.31:0/3377454420 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:02.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.900+0000 7f589659c640 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51cfd80 0x5640f55c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.901+0000 7f589559a640 1 -- 172.21.15.31:0/3377454420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51cfd80 msgr2=0x5640f55c2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:02.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.901+0000 7f589559a640 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51cfd80 0x5640f55c2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.901+0000 7f589559a640 1 -- 172.21.15.31:0/3377454420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51e82f0 msgr2=0x5640f55bb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:02.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.901+0000 7f589559a640 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51e82f0 0x5640f55bb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.901+0000 7f589559a640 1 -- 172.21.15.31:0/3377454420 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640f5229210 con 0x5640f51f04d0 2024-04-25T16:07:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.901+0000 7f589559a640 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51f04d0 0x5640f55bbec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5880005f70 tx=0x7f588003ce10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a02163c05fb4c6f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.901+0000 7f58797fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.902+0000 7f58797fa640 1 -- 172.21.15.31:0/3377454420 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588004f070 con 0x5640f51f04d0 2024-04-25T16:07:02.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.902+0000 7f58797fa640 1 -- 172.21.15.31:0/3377454420 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5880031720 con 0x5640f51f04d0 2024-04-25T16:07:02.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.902+0000 7f589ef8c740 1 -- 172.21.15.31:0/3377454420 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5880003d00 con 0x5640f51f04d0 2024-04-25T16:07:02.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.903+0000 7f58797fa640 1 -- 172.21.15.31:0/3377454420 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5880035040 con 0x5640f51f04d0 2024-04-25T16:07:02.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.903+0000 7f589ef8c740 10 client.26444 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:02.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.903+0000 7f589ef8c740 1 -- 172.21.15.31:0/3377454420 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5640f52f5620 con 0x5640f51f04d0 2024-04-25T16:07:02.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.904+0000 7f58797fa640 1 -- 172.21.15.31:0/3377454420 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f588005b070 con 0x5640f51f04d0 2024-04-25T16:07:02.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.904+0000 7f58797fa640 1 -- 172.21.15.31:0/3377454420 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f588003a040 con 0x5640f51f04d0 2024-04-25T16:07:02.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.905+0000 7f589ef8c740 10 client.26444 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:02.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.905+0000 7f589ef8c740 10 client.26444 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:02.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.905+0000 7f589ef8c740 20 client.26444 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:02.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.905+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5640f51a51c0 0x5640f55c9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:02.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.905+0000 7f589ef8c740 4 client.26444 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:02.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.905+0000 7f589ef8c740 1 -- 172.21.15.31:0/3377454420 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640f5525070 con 0x5640f51a51c0 2024-04-25T16:07:02.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.911+0000 7f5895d9b640 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5640f51a51c0 0x5640f55c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.914+0000 7f58797fa640 10 client.26444 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:02.915 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.914+0000 7f5895d9b640 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5640f51a51c0 0x5640f55c9200 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4c2a27d396ba553 server_cookie=272088bbf56065c7 in_seq=0 out_seq=0 2024-04-25T16:07:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.917+0000 7f58797fa640 1 -- 172.21.15.31:0/3377454420 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5640f5525070 con 0x5640f51a51c0 2024-04-25T16:07:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f58797fa640 10 client.26444 handle_command_reply: tid=0 2024-04-25T16:07:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f589ef8c740 1 client.26444 shutdown 2024-04-25T16:07:02.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f589ef8c740 1 -- 172.21.15.31:0/3377454420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51f04d0 msgr2=0x5640f55bbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51f04d0 0x5640f55bbec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5880005f70 tx=0x7f588003ce10 comp rx=0 tx=0).stop 2024-04-25T16:07:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f589ef8c740 1 -- 172.21.15.31:0/3377454420 shutdown_connections 2024-04-25T16:07:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5640f51a51c0 0x5640f55c9200 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51cfd80 0x5640f55c2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51f04d0 0x5640f55bbec0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3377454420 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51e82f0 0x5640f55bb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.918+0000 7f589ef8c740 1 -- 172.21.15.31:0/3377454420 >> 172.21.15.31:0/3377454420 conn(0x5640f5205300 msgr2=0x5640f55b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:02.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.919+0000 7f58797fa640 0 client.26444 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.919+0000 7f589ef8c740 1 -- 172.21.15.31:0/3377454420 shutdown_connections 2024-04-25T16:07:02.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.920+0000 7f589ef8c740 1 -- 172.21.15.31:0/3377454420 wait complete. 2024-04-25T16:07:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:02.923+0000 7f589ef8c740 20 client.26444 trim_cache size 0 max 16384 2024-04-25T16:07:03.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.012+0000 7f589ef8c740 1 Processor -- start 2024-04-25T16:07:03.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.013+0000 7f589ef8c740 1 -- start start 2024-04-25T16:07:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.013+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51cfd80 0x5640f55ba660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.013+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51e82f0 0x5640f55c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.014+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51f04d0 0x5640f55ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.014+0000 7f589ef8c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f58901ba080 con 0x5640f51f04d0 2024-04-25T16:07:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.014+0000 7f589ef8c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f58901bbbc0 con 0x5640f51cfd80 2024-04-25T16:07:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.014+0000 7f589ef8c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f58901ba750 con 0x5640f51e82f0 2024-04-25T16:07:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.015+0000 7f589559a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51e82f0 0x5640f55c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.015+0000 7f589559a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51e82f0 0x5640f55c2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39386/0 (socket says 172.21.15.31:39386) 2024-04-25T16:07:03.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.015+0000 7f589559a640 1 -- 172.21.15.31:0/4153022548 learned_addr learned my addr 172.21.15.31:0/4153022548 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:03.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.015+0000 7f589559a640 1 -- 172.21.15.31:0/4153022548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51cfd80 msgr2=0x5640f55ba660 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.015+0000 7f589559a640 1 --2- 172.21.15.31:0/4153022548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51cfd80 0x5640f55ba660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.015+0000 7f589559a640 1 -- 172.21.15.31:0/4153022548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51f04d0 msgr2=0x5640f55ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.015+0000 7f589559a640 1 --2- 172.21.15.31:0/4153022548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51f04d0 0x5640f55ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.015+0000 7f589559a640 1 -- 172.21.15.31:0/4153022548 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640f55b67f0 con 0x5640f51e82f0 2024-04-25T16:07:03.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.018+0000 7f589559a640 1 --2- 172.21.15.31:0/4153022548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51e82f0 0x5640f55c2d70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5880009850 tx=0x7f58800035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2b2ba2f9cf3eb50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.031+0000 7f587b7fe640 1 -- 172.21.15.31:0/4153022548 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5880058020 con 0x5640f51e82f0 2024-04-25T16:07:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.031+0000 7f587b7fe640 1 -- 172.21.15.31:0/4153022548 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5880042030 con 0x5640f51e82f0 2024-04-25T16:07:03.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.031+0000 7f587b7fe640 1 -- 172.21.15.31:0/4153022548 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588004f070 con 0x5640f51e82f0 2024-04-25T16:07:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.066+0000 7f589ef8c740 1 -- 172.21.15.31:0/4153022548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51e82f0 msgr2=0x5640f55c2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.066+0000 7f589ef8c740 1 --2- 172.21.15.31:0/4153022548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51e82f0 0x5640f55c2d70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5880009850 tx=0x7f58800035f0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.066+0000 7f589ef8c740 1 -- 172.21.15.31:0/4153022548 shutdown_connections 2024-04-25T16:07:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.066+0000 7f589ef8c740 1 --2- 172.21.15.31:0/4153022548 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51f04d0 0x5640f55ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.066+0000 7f589ef8c740 1 --2- 172.21.15.31:0/4153022548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51e82f0 0x5640f55c2d70 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5880009850 tx=0x7f58800035f0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.066+0000 7f589ef8c740 1 --2- 172.21.15.31:0/4153022548 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f51cfd80 0x5640f55ba660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.066+0000 7f589ef8c740 1 -- 172.21.15.31:0/4153022548 >> 172.21.15.31:0/4153022548 conn(0x5640f5205300 msgr2=0x5640f55b9f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.067+0000 7f589ef8c740 1 -- 172.21.15.31:0/4153022548 shutdown_connections 2024-04-25T16:07:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.067+0000 7f589ef8c740 1 -- 172.21.15.31:0/4153022548 wait complete. 2024-04-25T16:07:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.068+0000 7f589ef8c740 1 -- start start 2024-04-25T16:07:03.071 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.069+0000 7f589ef8c740 1 Processor -- start 2024-04-25T16:07:03.073 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.071+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51cfd80 0x5640f55c0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.073+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 0x5640f55c05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.073+0000 7f589ef8c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f55c1410 0x5640f55c1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.073+0000 7f589ef8c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f58901bb4f0 con 0x5640f51cfd80 2024-04-25T16:07:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.073+0000 7f589ef8c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f58901bae20 con 0x5640f55c1410 2024-04-25T16:07:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.073+0000 7f589ef8c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f58901b8060 con 0x5640f51f04d0 2024-04-25T16:07:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.074+0000 7f589ef8c740 10 client.0 _ll_register_callbacks cb 0x7f5888007640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.075+0000 7f589559a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 0x5640f55c05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.075+0000 7f589559a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 0x5640f55c05a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39402/0 (socket says 172.21.15.31:39402) 2024-04-25T16:07:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.076+0000 7f589559a640 1 -- 172.21.15.31:0/3901761717 learned_addr learned my addr 172.21.15.31:0/3901761717 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.076+0000 7f589659c640 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f55c1410 0x5640f55c1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.077+0000 7f589659c640 1 -- 172.21.15.31:0/3901761717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 msgr2=0x5640f55c05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.077+0000 7f589659c640 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 0x5640f55c05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.077+0000 7f589659c640 1 -- 172.21.15.31:0/3901761717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51cfd80 msgr2=0x5640f55c0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.077+0000 7f589659c640 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51cfd80 0x5640f55c0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.077+0000 7f589659c640 1 -- 172.21.15.31:0/3901761717 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640f55c1de0 con 0x5640f55c1410 2024-04-25T16:07:03.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.084+0000 7f589659c640 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f55c1410 0x5640f55c1820 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f588c00b3e0 tx=0x7f588c012f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b37f6ab41671b957 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.086+0000 7f58797fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:03.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.091+0000 7f58797fa640 1 -- 172.21.15.31:0/3901761717 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c0bf020 con 0x5640f55c1410 2024-04-25T16:07:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.092+0000 7f589ef8c740 1 -- 172.21.15.31:0/3901761717 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5880042030 con 0x5640f55c1410 2024-04-25T16:07:03.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.092+0000 7f58797fa640 1 -- 172.21.15.31:0/3901761717 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f588c0a2ae0 con 0x5640f55c1410 2024-04-25T16:07:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.093+0000 7f58797fa640 1 -- 172.21.15.31:0/3901761717 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f588c0a6f50 con 0x5640f55c1410 2024-04-25T16:07:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.093+0000 7f589ef8c740 10 client.26447 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:03.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.093+0000 7f589ef8c740 1 -- 172.21.15.31:0/3901761717 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5640f5631540 con 0x5640f55c1410 2024-04-25T16:07:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.097+0000 7f58797fa640 1 -- 172.21.15.31:0/3901761717 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5880042030 con 0x5640f55c1410 2024-04-25T16:07:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.099+0000 7f58797fa640 1 -- 172.21.15.31:0/3901761717 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f588c0c7020 con 0x5640f55c1410 2024-04-25T16:07:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.099+0000 7f589ef8c740 10 client.26447 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.099+0000 7f589ef8c740 10 client.26447 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.099+0000 7f589ef8c740 20 client.26447 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.099+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5640f5631d30 0x5640f5634150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.100+0000 7f589ef8c740 4 client.26447 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:07:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.100+0000 7f589ef8c740 1 -- 172.21.15.31:0/3901761717 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640f551eee0 con 0x5640f5631d30 2024-04-25T16:07:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.101+0000 7f5895d9b640 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5640f5631d30 0x5640f5634150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.102+0000 7f5895d9b640 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5640f5631d30 0x5640f5634150 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4dfd1f451269b7e server_cookie=431e7019fa2b00f6 in_seq=0 out_seq=0 2024-04-25T16:07:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:03.103+0000 7f58797fa640 10 client.26447 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:03.215 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:07:03.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:07:03.215 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:07:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.728+0000 7fe0fa138640 1 -- 172.21.15.31:0/626504170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f410a420 msgr2=0x7fe0f410add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.728+0000 7fe0fa138640 1 --2- 172.21.15.31:0/626504170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f410a420 0x7fe0f410add0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8009900 tx=0x7fe0e802a440 comp rx=0 tx=0).stop 2024-04-25T16:07:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.728+0000 7fe0fa138640 1 -- 172.21.15.31:0/626504170 shutdown_connections 2024-04-25T16:07:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.728+0000 7fe0fa138640 1 --2- 172.21.15.31:0/626504170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f4110030 0x7fe0f4110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.728+0000 7fe0fa138640 1 --2- 172.21.15.31:0/626504170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f410b310 0x7fe0f410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.728+0000 7fe0fa138640 1 --2- 172.21.15.31:0/626504170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f410a420 0x7fe0f410add0 secure :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8009900 tx=0x7fe0e802a440 comp rx=0 tx=0).stop 2024-04-25T16:07:03.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.728+0000 7fe0fa138640 1 -- 172.21.15.31:0/626504170 >> 172.21.15.31:0/626504170 conn(0x7fe0f40704e0 msgr2=0x7fe0f4072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:03.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.746+0000 7fe0fa138640 1 -- 172.21.15.31:0/626504170 shutdown_connections 2024-04-25T16:07:03.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.750+0000 7fe0fa138640 1 -- 172.21.15.31:0/626504170 wait complete. 2024-04-25T16:07:03.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.754+0000 7fe0fa138640 1 Processor -- start 2024-04-25T16:07:03.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.754+0000 7fe0fa138640 1 -- start start 2024-04-25T16:07:03.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.755+0000 7fe0fa138640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f410b310 0x7fe0f41a91e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.756+0000 7fe0fa138640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f4110030 0x7fe0f41a97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.756+0000 7fe0f37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f410b310 0x7fe0f41a91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.756+0000 7fe0f37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f410b310 0x7fe0f41a91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:51148/0 (socket says 172.21.15.31:51148) 2024-04-25T16:07:03.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.757+0000 7fe0f37fe640 1 -- 172.21.15.31:0/872979345 learned_addr learned my addr 172.21.15.31:0/872979345 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:03.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.757+0000 7fe0f2ffd640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f4110030 0x7fe0f41a97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.757+0000 7fe0fa138640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f41ae6a0 0x7fe0f41b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.758+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f411e520 con 0x7fe0f41ae6a0 2024-04-25T16:07:03.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.758+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f411e3a0 con 0x7fe0f410b310 2024-04-25T16:07:03.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.758+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f411e6a0 con 0x7fe0f4110030 2024-04-25T16:07:03.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.761+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f41ae6a0 0x7fe0f41b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.765+0000 7fe0f2ffd640 1 -- 172.21.15.31:0/872979345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f410b310 msgr2=0x7fe0f41a91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:03.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.765+0000 7fe0f2ffd640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f410b310 0x7fe0f41a91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.765+0000 7fe0f2ffd640 1 -- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f41ae6a0 msgr2=0x7fe0f41b0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:03.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.765+0000 7fe0f2ffd640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f41ae6a0 0x7fe0f41b0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.766+0000 7fe0f2ffd640 1 -- 172.21.15.31:0/872979345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0f41b11f0 con 0x7fe0f4110030 2024-04-25T16:07:03.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.769+0000 7fe0f2ffd640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f4110030 0x7fe0f41a97a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400d6d0 tx=0x7fe0e400db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:03.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.772+0000 7fe0f0ff9640 1 -- 172.21.15.31:0/872979345 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4017070 con 0x7fe0f4110030 2024-04-25T16:07:03.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.772+0000 7fe0f0ff9640 1 -- 172.21.15.31:0/872979345 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0e4009820 con 0x7fe0f4110030 2024-04-25T16:07:03.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.772+0000 7fe0f0ff9640 1 -- 172.21.15.31:0/872979345 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4004c40 con 0x7fe0f4110030 2024-04-25T16:07:03.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.773+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0f41b14e0 con 0x7fe0f4110030 2024-04-25T16:07:03.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.773+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0f41b19e0 con 0x7fe0f4110030 2024-04-25T16:07:03.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.780+0000 7fe0f0ff9640 1 -- 172.21.15.31:0/872979345 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe0e4010460 con 0x7fe0f4110030 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.783+0000 7fe0f0ff9640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0dc0afd00 0x7fe0dc0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.783+0000 7fe0f0ff9640 1 -- 172.21.15.31:0/872979345 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe0e40df600 con 0x7fe0f4110030 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.785+0000 7fe0f37fe640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0dc0afd00 0x7fe0dc0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.792+0000 7fe100fef740 1 Processor -- start 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.792+0000 7fe100fef740 1 -- start start 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.792+0000 7fe100fef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38f4410 0x5634d3c954a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.793+0000 7fe100fef740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38cb550 0x5634d3c95a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.793+0000 7fe100fef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38d3730 0x5634d3c9c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.793+0000 7fe100fef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5634d3bfab00 con 0x5634d38d3730 2024-04-25T16:07:03.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.793+0000 7fe100fef740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5634d3b96920 con 0x5634d38cb550 2024-04-25T16:07:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.793+0000 7fe100fef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5634d3bf6ac0 con 0x5634d38f4410 2024-04-25T16:07:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.796+0000 7fe0f2ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38cb550 0x5634d3c95a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.797+0000 7fe0f2ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38cb550 0x5634d3c95a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:51164/0 (socket says 172.21.15.31:51164) 2024-04-25T16:07:03.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.797+0000 7fe0f2ffd640 1 -- 172.21.15.31:0/4186226631 learned_addr learned my addr 172.21.15.31:0/4186226631 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:03.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.798+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/4186226631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38d3730 0x5634d3c9c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.799+0000 7fe0f3fff640 1 -- 172.21.15.31:0/4186226631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38f4410 msgr2=0x5634d3c954a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:03.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.799+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/4186226631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38f4410 0x5634d3c954a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.799+0000 7fe0f3fff640 1 -- 172.21.15.31:0/4186226631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38cb550 msgr2=0x5634d3c95a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.799+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/4186226631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38cb550 0x5634d3c95a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.799+0000 7fe0f3fff640 1 -- 172.21.15.31:0/4186226631 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634d39d47e0 con 0x5634d38d3730 2024-04-25T16:07:03.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.801+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/4186226631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38d3730 0x5634d3c9c920 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009e60 tx=0x7fe0ec00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d214e096e8f998a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:03.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.805+0000 7fe0d9ffb640 1 -- 172.21.15.31:0/4186226631 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec01a070 con 0x5634d38d3730 2024-04-25T16:07:03.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.805+0000 7fe0d9ffb640 1 -- 172.21.15.31:0/4186226631 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec0158c0 con 0x5634d38d3730 2024-04-25T16:07:03.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.805+0000 7fe0d9ffb640 1 -- 172.21.15.31:0/4186226631 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec00a820 con 0x5634d38d3730 2024-04-25T16:07:03.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.810+0000 7fe100fef740 1 -- 172.21.15.31:0/4186226631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38d3730 msgr2=0x5634d3c9c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:03.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.810+0000 7fe100fef740 1 --2- 172.21.15.31:0/4186226631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38d3730 0x5634d3c9c920 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009e60 tx=0x7fe0ec00c7f0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.810+0000 7fe100fef740 1 -- 172.21.15.31:0/4186226631 shutdown_connections 2024-04-25T16:07:03.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.810+0000 7fe100fef740 1 --2- 172.21.15.31:0/4186226631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38d3730 0x5634d3c9c920 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.810+0000 7fe100fef740 1 --2- 172.21.15.31:0/4186226631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38cb550 0x5634d3c95a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.811+0000 7fe100fef740 1 --2- 172.21.15.31:0/4186226631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38f4410 0x5634d3c954a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.811+0000 7fe100fef740 1 -- 172.21.15.31:0/4186226631 >> 172.21.15.31:0/4186226631 conn(0x5634d38e8560 msgr2=0x5634d3c90e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:03.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.813+0000 7fe0f37fe640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0dc0afd00 0x7fe0dc0b2210 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe0f41aa280 tx=0x7fe0e8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:03.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.818+0000 7fe100fef740 1 -- 172.21.15.31:0/4186226631 shutdown_connections 2024-04-25T16:07:03.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.831+0000 7fe100fef740 1 -- 172.21.15.31:0/4186226631 wait complete. 2024-04-25T16:07:03.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.832+0000 7fe100fef740 1 -- start start 2024-04-25T16:07:03.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.832+0000 7fe100fef740 1 Processor -- start 2024-04-25T16:07:03.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.834+0000 7fe100fef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3c9f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.834+0000 7fe100fef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3c9f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.834+0000 7fe100fef740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.834+0000 7fe100fef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5634d3c9d370 con 0x5634d38cb550 2024-04-25T16:07:03.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.834+0000 7fe100fef740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5634d3b945d0 con 0x5634d38f4410 2024-04-25T16:07:03.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.834+0000 7fe100fef740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5634d3c9d4f0 con 0x5634d38d3730 2024-04-25T16:07:03.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.834+0000 7fe100fef740 10 client.0 _ll_register_callbacks cb 0x5634d3b82080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:03.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.835+0000 7fe0f3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.835+0000 7fe0f3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:51174/0 (socket says 172.21.15.31:51174) 2024-04-25T16:07:03.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.835+0000 7fe0f3fff640 1 -- 172.21.15.31:0/592194784 learned_addr learned my addr 172.21.15.31:0/592194784 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:03.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.835+0000 7fe0f2ffd640 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3c9f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.836+0000 7fe0f37fe640 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3c9f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.836+0000 7fe0f3fff640 1 -- 172.21.15.31:0/592194784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 msgr2=0x5634d3c9f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.836+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3c9f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.836+0000 7fe0f3fff640 1 -- 172.21.15.31:0/592194784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 msgr2=0x5634d3c9f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.836+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3c9f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.836+0000 7fe0f3fff640 1 -- 172.21.15.31:0/592194784 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634d3a34f70 con 0x5634d38f4410 2024-04-25T16:07:03.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.836+0000 7fe0f37fe640 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3c9f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:07:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.838+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca65b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009e30 tx=0x7fe0ec015ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cec6476bbbe00c29 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:03.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.838+0000 7fe0d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.839+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/592194784 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec01a040 con 0x5634d38f4410 2024-04-25T16:07:03.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.839+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/592194784 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe0ec00a5a0 con 0x5634d38f4410 2024-04-25T16:07:03.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.840+0000 7fe100fef740 1 -- 172.21.15.31:0/592194784 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0ec0158c0 con 0x5634d38f4410 2024-04-25T16:07:03.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.841+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/592194784 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe0ec00a7c0 con 0x5634d38f4410 2024-04-25T16:07:03.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.845+0000 7fe100fef740 10 client.26459 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:03.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.845+0000 7fe100fef740 1 -- 172.21.15.31:0/592194784 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5634d38cc170 con 0x5634d38f4410 2024-04-25T16:07:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.849+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/592194784 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0ec0158c0 con 0x5634d38f4410 2024-04-25T16:07:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.849+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/592194784 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe0ec028030 con 0x5634d38f4410 2024-04-25T16:07:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.849+0000 7fe100fef740 10 client.26459 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.849+0000 7fe100fef740 10 client.26459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.849+0000 7fe100fef740 20 client.26459 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.849+0000 7fe100fef740 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5634d38b31f0 0x5634d3cac730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.849+0000 7fe100fef740 4 client.26459 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:03.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.849+0000 7fe100fef740 1 -- 172.21.15.31:0/592194784 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5634d3c08710 con 0x5634d38b31f0 2024-04-25T16:07:03.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.854+0000 7fe0f37fe640 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5634d38b31f0 0x5634d3cac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:03.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.855+0000 7fe0f37fe640 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5634d38b31f0 0x5634d3cac730 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eced1567ce802d3a server_cookie=675c9d4bc544cc97 in_seq=0 out_seq=0 2024-04-25T16:07:03.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:03.857+0000 7fe0d2ffd640 10 client.26459 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:03 smithi031 ceph-mon[29140]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 1.72k op/s 2024-04-25T16:07:04.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:03 smithi063 ceph-mon[32413]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 1.72k op/s 2024-04-25T16:07:04.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:03 smithi186 ceph-mon[33258]: pgmap v523: 97 pgs: 97 active+clean; 10 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 1.72k op/s 2024-04-25T16:07:05.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:04 smithi063 ceph-mon[32413]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 951 MiB/s wr, 2.09k op/s 2024-04-25T16:07:05.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:04 smithi186 ceph-mon[33258]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 951 MiB/s wr, 2.09k op/s 2024-04-25T16:07:05.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:04 smithi031 ceph-mon[29140]: pgmap v524: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 951 MiB/s wr, 2.09k op/s 2024-04-25T16:07:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.582+0000 7f58797fa640 1 -- 172.21.15.31:0/3901761717 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640f551eee0 con 0x5640f5631d30 2024-04-25T16:07:06.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.582+0000 7f58797fa640 10 client.26447 handle_command_reply: tid=0 2024-04-25T16:07:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.582+0000 7f589ef8c740 1 client.26447 shutdown 2024-04-25T16:07:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.582+0000 7f589ef8c740 1 -- 172.21.15.31:0/3901761717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f55c1410 msgr2=0x5640f55c1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.582+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f55c1410 0x5640f55c1820 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f588c00b3e0 tx=0x7f588c012f30 comp rx=0 tx=0).stop 2024-04-25T16:07:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.583+0000 7f589ef8c740 1 -- 172.21.15.31:0/3901761717 shutdown_connections 2024-04-25T16:07:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.583+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5640f5631d30 0x5640f5634150 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.583+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5640f55c1410 0x5640f55c1820 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.583+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5640f51f04d0 0x5640f55c05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.583+0000 7f589ef8c740 1 --2- 172.21.15.31:0/3901761717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5640f51cfd80 0x5640f55c0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.583+0000 7f589ef8c740 1 -- 172.21.15.31:0/3901761717 >> 172.21.15.31:0/3901761717 conn(0x5640f5205300 msgr2=0x5640f55b02b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:06.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.583+0000 7f58797fa640 0 client.26447 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.583+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/592194784 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5634d3c08710 con 0x5634d38b31f0 2024-04-25T16:07:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.583+0000 7fe0d2ffd640 10 client.26459 handle_command_reply: tid=0 2024-04-25T16:07:06.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.583+0000 7fe100fef740 1 client.26459 shutdown 2024-04-25T16:07:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.583+0000 7f589ef8c740 1 -- 172.21.15.31:0/3901761717 shutdown_connections 2024-04-25T16:07:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 -- 172.21.15.31:0/592194784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 msgr2=0x5634d3ca65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:06.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca65b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec009e30 tx=0x7fe0ec015ef0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.584+0000 7f589ef8c740 1 -- 172.21.15.31:0/3901761717 wait complete. 2024-04-25T16:07:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 -- 172.21.15.31:0/592194784 shutdown_connections 2024-04-25T16:07:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5634d38b31f0 0x5634d3cac730 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca65b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3c9f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 --2- 172.21.15.31:0/592194784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3c9f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 -- 172.21.15.31:0/592194784 >> 172.21.15.31:0/592194784 conn(0x5634d38e8560 msgr2=0x5634d3c8fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:06.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe0d2ffd640 0 client.26459 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:06.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 -- 172.21.15.31:0/592194784 shutdown_connections 2024-04-25T16:07:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.584+0000 7f589ef8c740 20 client.26447 trim_cache size 0 max 16384 2024-04-25T16:07:06.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 1 -- 172.21.15.31:0/592194784 wait complete. 2024-04-25T16:07:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:07:06.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:07:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:07:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:07:06.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.584+0000 7fe100fef740 20 client.26459 trim_cache size 0 max 16384 2024-04-25T16:07:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.588+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f587c0afc70 msgr2=0x7f587c0b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.588+0000 7f5896d9d640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f587c0afc70 0x7f587c0b2180 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x5640f55b2f00 tx=0x7f58880077a0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.588+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f589010ff40 msgr2=0x7f5890082e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:06.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.588+0000 7f5896d9d640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f589010ff40 0x7f5890082e60 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f588c00b410 tx=0x7f588c00b8d0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.592+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 shutdown_connections 2024-04-25T16:07:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.592+0000 7f5896d9d640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f587c0afc70 0x7f587c0b2180 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.592+0000 7f5896d9d640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f589010ff40 0x7f5890082e60 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.592+0000 7f5896d9d640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f589010b240 0x7f5890081f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.592+0000 7f5896d9d640 1 --2- 172.21.15.31:0/4743934 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f589010a2d0 0x7f5890081a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.592+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 >> 172.21.15.31:0/4743934 conn(0x7f5890078c00 msgr2=0x7f589010bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:06.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.594+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 shutdown_connections 2024-04-25T16:07:06.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:06.594+0000 7f5896d9d640 1 -- 172.21.15.31:0/4743934 wait complete. 2024-04-25T16:07:06.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.644+0000 7fe100fef740 1 Processor -- start 2024-04-25T16:07:06.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.644+0000 7fe100fef740 1 -- start start 2024-04-25T16:07:06.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.645+0000 7fe100fef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3c94340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:06.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.645+0000 7fe100fef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3ca64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:06.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.646+0000 7fe100fef740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:06.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.646+0000 7fe100fef740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f41ba860 con 0x5634d38cb550 2024-04-25T16:07:06.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.646+0000 7fe100fef740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f41bc3a0 con 0x5634d38f4410 2024-04-25T16:07:06.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.646+0000 7fe0f2ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3ca64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:06.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.646+0000 7fe0f2ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3ca64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:40152/0 (socket says 172.21.15.31:40152) 2024-04-25T16:07:06.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.646+0000 7fe0f2ffd640 1 -- 172.21.15.31:0/2218149126 learned_addr learned my addr 172.21.15.31:0/2218149126 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:06.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.646+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/2218149126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:06.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.646+0000 7fe100fef740 1 -- 172.21.15.31:0/2218149126 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f41baf30 con 0x5634d38d3730 2024-04-25T16:07:06.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.647+0000 7fe0f2ffd640 1 -- 172.21.15.31:0/2218149126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 msgr2=0x5634d3ca6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:06.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.647+0000 7fe0f2ffd640 1 --2- 172.21.15.31:0/2218149126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.647+0000 7fe0f2ffd640 1 -- 172.21.15.31:0/2218149126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 msgr2=0x5634d3c94340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:07:06.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.647+0000 7fe0f2ffd640 1 --2- 172.21.15.31:0/2218149126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3c94340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.647+0000 7fe0f2ffd640 1 -- 172.21.15.31:0/2218149126 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5634d3ca6f50 con 0x5634d38d3730 2024-04-25T16:07:06.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.648+0000 7fe0f2ffd640 1 --2- 172.21.15.31:0/2218149126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3ca64d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009610 tx=0x7fe0e40a3950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba5a534a96be4106 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:06.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.650+0000 7fe0d97fa640 1 -- 172.21.15.31:0/2218149126 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4020040 con 0x5634d38d3730 2024-04-25T16:07:06.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.650+0000 7fe0d97fa640 1 -- 172.21.15.31:0/2218149126 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0e40c1070 con 0x5634d38d3730 2024-04-25T16:07:06.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.652+0000 7fe100fef740 1 -- 172.21.15.31:0/2218149126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 msgr2=0x5634d3ca64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:06.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.652+0000 7fe100fef740 1 --2- 172.21.15.31:0/2218149126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3ca64d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009610 tx=0x7fe0e40a3950 comp rx=0 tx=0).stop 2024-04-25T16:07:06.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.652+0000 7fe100fef740 1 -- 172.21.15.31:0/2218149126 shutdown_connections 2024-04-25T16:07:06.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.653+0000 7fe100fef740 1 --2- 172.21.15.31:0/2218149126 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d38f4410 0x5634d3ca6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.653+0000 7fe100fef740 1 --2- 172.21.15.31:0/2218149126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38d3730 0x5634d3ca64d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.653+0000 7fe100fef740 1 --2- 172.21.15.31:0/2218149126 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3c94340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.653+0000 7fe100fef740 1 -- 172.21.15.31:0/2218149126 >> 172.21.15.31:0/2218149126 conn(0x5634d38e8560 msgr2=0x5634d3c93c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:06.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.653+0000 7fe100fef740 1 -- 172.21.15.31:0/2218149126 shutdown_connections 2024-04-25T16:07:06.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.654+0000 7fe100fef740 1 -- 172.21.15.31:0/2218149126 wait complete. 2024-04-25T16:07:06.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.656+0000 7fe100fef740 1 -- start start 2024-04-25T16:07:06.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.657+0000 7fe100fef740 1 Processor -- start 2024-04-25T16:07:06.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.658+0000 7fe100fef740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3ca3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:06.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.658+0000 7fe0f37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3ca3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:06.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.658+0000 7fe0f37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3ca3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34086/0 (socket says 172.21.15.31:34086) 2024-04-25T16:07:06.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.659+0000 7fe100fef740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38f4410 0x5634d3ca43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:06.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.659+0000 7fe0f37fe640 1 -- 172.21.15.31:0/1586183176 learned_addr learned my addr 172.21.15.31:0/1586183176 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:06.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.659+0000 7fe100fef740 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d3888630 0x5634d3ca5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:06.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.659+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f41bbcd0 con 0x5634d38cb550 2024-04-25T16:07:06.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.659+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f41bb600 con 0x5634d3888630 2024-04-25T16:07:06.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.659+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0f41b8840 con 0x5634d38f4410 2024-04-25T16:07:06.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.660+0000 7fe100fef740 10 client.0 _ll_register_callbacks cb 0x7fe0ec009d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:06.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.663+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d3888630 0x5634d3ca5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:06.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.663+0000 7fe0f2ffd640 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38f4410 0x5634d3ca43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:06.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.665+0000 7fe0f3fff640 1 -- 172.21.15.31:0/1586183176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38f4410 msgr2=0x5634d3ca43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:06.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.665+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38f4410 0x5634d3ca43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.665+0000 7fe0f3fff640 1 -- 172.21.15.31:0/1586183176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 msgr2=0x5634d3ca3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.665+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3ca3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.665+0000 7fe0f3fff640 1 -- 172.21.15.31:0/1586183176 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5634d3ca57c0 con 0x5634d3888630 2024-04-25T16:07:06.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.666+0000 7fe0f3fff640 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d3888630 0x5634d3ca5200 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0153e0 tx=0x7fe0ec01ebb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb9c9eb5607f8e99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:06.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.668+0000 7fe0d2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:06.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.671+0000 7fe0f37fe640 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3ca3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:07:06.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.672+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/1586183176 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec01a040 con 0x5634d3888630 2024-04-25T16:07:06.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.672+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5634d3c02580 con 0x5634d3888630 2024-04-25T16:07:06.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.672+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/1586183176 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe0ec00a400 con 0x5634d3888630 2024-04-25T16:07:06.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.676+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/1586183176 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe0ec00a620 con 0x5634d3888630 2024-04-25T16:07:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.676+0000 7fe100fef740 10 client.26465 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:06.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.676+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5634d3ca6b10 con 0x5634d3888630 2024-04-25T16:07:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.680+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/1586183176 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5634d3c02580 con 0x5634d3888630 2024-04-25T16:07:06.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.680+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/1586183176 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe0ec028070 con 0x5634d3888630 2024-04-25T16:07:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.681+0000 7fe100fef740 10 client.26465 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.681+0000 7fe100fef740 10 client.26465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.681+0000 7fe100fef740 20 client.26465 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.682+0000 7fe100fef740 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5634d3d14df0 0x5634d3d17210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.682+0000 7fe100fef740 4 client.26465 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:07:06.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.682+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5634d3b7dac0 con 0x5634d3d14df0 2024-04-25T16:07:06.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.693+0000 7fe0f37fe640 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5634d3d14df0 0x5634d3d17210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:06.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.721+0000 7fe0d2ffd640 10 client.26465 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:06.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.721+0000 7fe0f37fe640 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5634d3d14df0 0x5634d3d17210 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3a906c46d5547c7 server_cookie=6edafbf51a3cedc6 in_seq=0 out_seq=0 2024-04-25T16:07:06.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.727+0000 7fe0d2ffd640 1 -- 172.21.15.31:0/1586183176 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5634d3b7dac0 con 0x5634d3d14df0 2024-04-25T16:07:06.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.727+0000 7fe0d2ffd640 10 client.26465 handle_command_reply: tid=0 2024-04-25T16:07:06.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.727+0000 7fe100fef740 1 client.26465 shutdown 2024-04-25T16:07:06.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.737+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d3888630 msgr2=0x5634d3ca5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:06.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.737+0000 7fe100fef740 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d3888630 0x5634d3ca5200 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec0153e0 tx=0x7fe0ec01ebb0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.739+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 shutdown_connections 2024-04-25T16:07:06.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.739+0000 7fe100fef740 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5634d3d14df0 0x5634d3d17210 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.739+0000 7fe100fef740 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5634d3888630 0x5634d3ca5200 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.739+0000 7fe100fef740 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5634d38f4410 0x5634d3ca43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.739+0000 7fe100fef740 1 --2- 172.21.15.31:0/1586183176 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5634d38cb550 0x5634d3ca3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.739+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 >> 172.21.15.31:0/1586183176 conn(0x5634d38e8560 msgr2=0x5634d3c94660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:06.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.739+0000 7fe0d2ffd640 0 client.26465 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:06.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.740+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 shutdown_connections 2024-04-25T16:07:06.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.746+0000 7fe100fef740 1 -- 172.21.15.31:0/1586183176 wait complete. 2024-04-25T16:07:06.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.747+0000 7fe100fef740 20 client.26465 trim_cache size 0 max 16384 2024-04-25T16:07:06.753 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:07:06.753 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:07:06.753 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "b319b764-dc54-4483-b61a-5afbfacbd25e", 2024-04-25T16:07:06.753 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:07:06.753 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:07:06.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.758+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0dc0afd00 msgr2=0x7fe0dc0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:06.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.758+0000 7fe0fa138640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0dc0afd00 0x7fe0dc0b2210 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fe0f41aa280 tx=0x7fe0e8005fd0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.758+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f4110030 msgr2=0x7fe0f41a97a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:06.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.758+0000 7fe0fa138640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f4110030 0x7fe0f41a97a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400d6d0 tx=0x7fe0e400db90 comp rx=0 tx=0).stop 2024-04-25T16:07:06.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.759+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 shutdown_connections 2024-04-25T16:07:06.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.759+0000 7fe0fa138640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0dc0afd00 0x7fe0dc0b2210 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.759+0000 7fe0fa138640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0f41ae6a0 0x7fe0f41b0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.759+0000 7fe0fa138640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0f4110030 0x7fe0f41a97a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.759+0000 7fe0fa138640 1 --2- 172.21.15.31:0/872979345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0f410b310 0x7fe0f41a91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:06.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.760+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 >> 172.21.15.31:0/872979345 conn(0x7fe0f40704e0 msgr2=0x7fe0f410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:06.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.764+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 shutdown_connections 2024-04-25T16:07:06.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:06.764+0000 7fe0fa138640 1 -- 172.21.15.31:0/872979345 wait complete. 2024-04-25T16:07:06.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:07:06.811 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:07:07.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:06 smithi063 ceph-mon[32413]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 746 MiB/s wr, 1.66k op/s 2024-04-25T16:07:07.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:06 smithi186 ceph-mon[33258]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 746 MiB/s wr, 1.66k op/s 2024-04-25T16:07:07.205 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:06 smithi031 ceph-mon[29140]: pgmap v525: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 746 MiB/s wr, 1.66k op/s 2024-04-25T16:07:07.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.313+0000 7f0f826cc640 1 -- 172.21.15.31:0/2652048514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f7c10b190 msgr2=0x7f0f7c10b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:07.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.313+0000 7f0f826cc640 1 --2- 172.21.15.31:0/2652048514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f7c10b190 0x7f0f7c10b5a0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f0f74009ca0 tx=0x7f0f7402a8d0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.317+0000 7f0f826cc640 1 -- 172.21.15.31:0/2652048514 shutdown_connections 2024-04-25T16:07:07.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.317+0000 7f0f826cc640 1 --2- 172.21.15.31:0/2652048514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f7c10fe90 0x7f0f7c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.317+0000 7f0f826cc640 1 --2- 172.21.15.31:0/2652048514 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f7c10b190 0x7f0f7c10b5a0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.317+0000 7f0f826cc640 1 --2- 172.21.15.31:0/2652048514 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f7c10a340 0x7f0f7c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.317+0000 7f0f826cc640 1 -- 172.21.15.31:0/2652048514 >> 172.21.15.31:0/2652048514 conn(0x7f0f7c070320 msgr2=0x7f0f7c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:07.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.319+0000 7f0f826cc640 1 -- 172.21.15.31:0/2652048514 shutdown_connections 2024-04-25T16:07:07.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.323+0000 7f0f826cc640 1 -- 172.21.15.31:0/2652048514 wait complete. 2024-04-25T16:07:07.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.324+0000 7f0f826cc640 1 Processor -- start 2024-04-25T16:07:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.326+0000 7f0f826cc640 1 -- start start 2024-04-25T16:07:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.326+0000 7f0f826cc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f7c10a340 0x7f0f7c11baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f826cc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f7c10b190 0x7f0f7c11a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f826cc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f7c10fe90 0x7f0f7c11a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f826cc640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c112460 con 0x7f0f7c10b190 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f826cc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1122e0 con 0x7f0f7c10a340 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f826cc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0f7c1125e0 con 0x7f0f7c10fe90 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f816ca640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f7c10a340 0x7f0f7c11baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f816ca640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f7c10a340 0x7f0f7c11baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:51204/0 (socket says 172.21.15.31:51204) 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f816ca640 1 -- 172.21.15.31:0/830256800 learned_addr learned my addr 172.21.15.31:0/830256800 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f816ca640 1 -- 172.21.15.31:0/830256800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f7c10fe90 msgr2=0x7f0f7c11a680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f816ca640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f7c10fe90 0x7f0f7c11a680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f816ca640 1 -- 172.21.15.31:0/830256800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f7c10b190 msgr2=0x7f0f7c11a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f816ca640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f7c10b190 0x7f0f7c11a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.327+0000 7f0f816ca640 1 -- 172.21.15.31:0/830256800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f7c11ac70 con 0x7f0f7c10a340 2024-04-25T16:07:07.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.328+0000 7f0f816ca640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f7c10a340 0x7f0f7c11baa0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00b450 tx=0x7f0f6c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.329+0000 7f0f727fc640 1 -- 172.21.15.31:0/830256800 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c015020 con 0x7f0f7c10a340 2024-04-25T16:07:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.329+0000 7f0f727fc640 1 -- 172.21.15.31:0/830256800 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f6c011d40 con 0x7f0f7c10a340 2024-04-25T16:07:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.329+0000 7f0f727fc640 1 -- 172.21.15.31:0/830256800 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c00f340 con 0x7f0f7c10a340 2024-04-25T16:07:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.330+0000 7f0f826cc640 1 -- 172.21.15.31:0/830256800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f7c11af00 con 0x7f0f7c10a340 2024-04-25T16:07:07.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.330+0000 7f0f826cc640 1 -- 172.21.15.31:0/830256800 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f7c1b9a80 con 0x7f0f7c10a340 2024-04-25T16:07:07.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.335+0000 7f0f8a7dc740 1 Processor -- start 2024-04-25T16:07:07.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.336+0000 7f0f8a7dc740 1 -- start start 2024-04-25T16:07:07.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.337+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1c0390 0x55e7ce561c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.337+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce1974d0 0x55e7ce5621d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.337+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce562e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.337+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e7ce4c6d80 con 0x55e7ce1c0390 2024-04-25T16:07:07.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.337+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e7ce4808b0 con 0x55e7ce19f6b0 2024-04-25T16:07:07.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.337+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e7ce4c2d40 con 0x55e7ce1974d0 2024-04-25T16:07:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.339+0000 7f0f81ecb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1c0390 0x55e7ce561c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.339+0000 7f0f81ecb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1c0390 0x55e7ce561c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34116/0 (socket says 172.21.15.31:34116) 2024-04-25T16:07:07.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.339+0000 7f0f81ecb640 1 -- 172.21.15.31:0/1073192335 learned_addr learned my addr 172.21.15.31:0/1073192335 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:07.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.340+0000 7f0f727fc640 1 -- 172.21.15.31:0/830256800 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0f6c018020 con 0x7f0f7c10a340 2024-04-25T16:07:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.341+0000 7f0f816ca640 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce1974d0 0x55e7ce5621d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.341+0000 7f0f81ecb640 1 -- 172.21.15.31:0/1073192335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce1974d0 msgr2=0x55e7ce5621d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.342+0000 7f0f80ec9640 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce562e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.342+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce1974d0 0x55e7ce5621d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.342+0000 7f0f81ecb640 1 -- 172.21.15.31:0/1073192335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 msgr2=0x55e7ce562e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.342+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce562e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.342+0000 7f0f81ecb640 1 -- 172.21.15.31:0/1073192335 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7ce451a40 con 0x55e7ce1c0390 2024-04-25T16:07:07.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.343+0000 7f0f816ca640 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce1974d0 0x55e7ce5621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:07:07.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.343+0000 7f0f80ec9640 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce562e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:07:07.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.344+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1c0390 0x55e7ce561c50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009bd0 tx=0x7f0f7800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f93733794965ed3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:07.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.346+0000 7f0f677fe640 1 -- 172.21.15.31:0/1073192335 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f78005020 con 0x55e7ce1c0390 2024-04-25T16:07:07.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.346+0000 7f0f677fe640 1 -- 172.21.15.31:0/1073192335 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f780047a0 con 0x55e7ce1c0390 2024-04-25T16:07:07.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.348+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/1073192335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1c0390 msgr2=0x55e7ce561c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:07.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.348+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1c0390 0x55e7ce561c50 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f0f78009bd0 tx=0x7f0f7800e720 comp rx=0 tx=0).stop 2024-04-25T16:07:07.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.349+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/1073192335 shutdown_connections 2024-04-25T16:07:07.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.349+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce562e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.349+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce1974d0 0x55e7ce5621d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.349+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/1073192335 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1c0390 0x55e7ce561c50 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.349+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/1073192335 >> 172.21.15.31:0/1073192335 conn(0x55e7ce1b44e0 msgr2=0x55e7ce55d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:07.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.350+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/1073192335 shutdown_connections 2024-04-25T16:07:07.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.350+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/1073192335 wait complete. 2024-04-25T16:07:07.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.351+0000 7f0f8a7dc740 1 -- start start 2024-04-25T16:07:07.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.352+0000 7f0f8a7dc740 1 Processor -- start 2024-04-25T16:07:07.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.353+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.353+0000 7f0f727fc640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f680afbb0 0x7f0f680b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.353+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce56bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.353+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 0x55e7ce572f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.353+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e7ce5639c0 con 0x55e7ce1974d0 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.353+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e7ce44dd10 con 0x55e7ce19f6b0 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.353+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e7ce563b40 con 0x55e7ce17ef60 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.353+0000 7f0f80ec9640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f680afbb0 0x7f0f680b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.353+0000 7f0f8a7dc740 10 client.0 _ll_register_callbacks cb 0x55e7ce47a210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.354+0000 7f0f81ecb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 0x55e7ce572f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.354+0000 7f0f81ecb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.354+0000 7f0f81ecb640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34122/0 (socket says 172.21.15.31:34122) 2024-04-25T16:07:07.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.354+0000 7f0f81ecb640 1 -- 172.21.15.31:0/3336554784 learned_addr learned my addr 172.21.15.31:0/3336554784 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:07.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.355+0000 7f0f81ecb640 1 -- 172.21.15.31:0/3336554784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 msgr2=0x55e7ce56bfc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:07.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.355+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce56bfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.355+0000 7f0f81ecb640 1 -- 172.21.15.31:0/3336554784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 msgr2=0x55e7ce56ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:07.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.355+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.355+0000 7f0f81ecb640 1 -- 172.21.15.31:0/3336554784 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7ce2a27d0 con 0x55e7ce17ef60 2024-04-25T16:07:07.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.356+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 0x55e7ce572f20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800f010 tx=0x7f0f78005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72b974cd1062682e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:07.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.356+0000 7f0f64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:07.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.357+0000 7f0f64ff9640 1 -- 172.21.15.31:0/3336554784 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7802b060 con 0x55e7ce17ef60 2024-04-25T16:07:07.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.357+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/3336554784 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e7ce4d4990 con 0x55e7ce17ef60 2024-04-25T16:07:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.357+0000 7f0f64ff9640 1 -- 172.21.15.31:0/3336554784 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0f78005460 con 0x55e7ce17ef60 2024-04-25T16:07:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.358+0000 7f0f64ff9640 1 -- 172.21.15.31:0/3336554784 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0f780035f0 con 0x55e7ce17ef60 2024-04-25T16:07:07.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.358+0000 7f0f727fc640 1 -- 172.21.15.31:0/830256800 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0f6c0de330 con 0x7f0f7c10a340 2024-04-25T16:07:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.360+0000 7f0f8a7dc740 10 client.26431 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:07.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.360+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/3336554784 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e7ce1d83f0 con 0x55e7ce17ef60 2024-04-25T16:07:07.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.363+0000 7f0f64ff9640 1 -- 172.21.15.31:0/3336554784 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55e7ce4d4990 con 0x55e7ce17ef60 2024-04-25T16:07:07.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.363+0000 7f0f64ff9640 1 -- 172.21.15.31:0/3336554784 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0f78023040 con 0x55e7ce17ef60 2024-04-25T16:07:07.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.363+0000 7f0f8a7dc740 10 client.26431 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:07.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.363+0000 7f0f8a7dc740 10 client.26431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:07.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.363+0000 7f0f8a7dc740 20 client.26431 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:07.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.363+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7ce1543a0 0x55e7ce579040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.363+0000 7f0f8a7dc740 4 client.26431 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:07.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.363+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/3336554784 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7ce4ce800 con 0x55e7ce1543a0 2024-04-25T16:07:07.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.365+0000 7f0f816ca640 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7ce1543a0 0x55e7ce579040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.372+0000 7f0f80ec9640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f680afbb0 0x7f0f680b20c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0f74005e30 tx=0x7f0f74000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.375+0000 7f0f816ca640 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7ce1543a0 0x55e7ce579040 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c64a6a3d65e34884 server_cookie=9a99ae8754be33cb in_seq=0 out_seq=0 2024-04-25T16:07:07.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.376+0000 7f0f64ff9640 10 client.26431 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:07.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.384+0000 7f0f64ff9640 1 -- 172.21.15.31:0/3336554784 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e7ce4ce800 con 0x55e7ce1543a0 2024-04-25T16:07:07.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.384+0000 7f0f64ff9640 10 client.26431 handle_command_reply: tid=0 2024-04-25T16:07:07.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f8a7dc740 1 client.26431 shutdown 2024-04-25T16:07:07.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/3336554784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 msgr2=0x55e7ce572f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:07.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 0x55e7ce572f20 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0f7800f010 tx=0x7f0f78005970 comp rx=0 tx=0).stop 2024-04-25T16:07:07.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/3336554784 shutdown_connections 2024-04-25T16:07:07.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7ce1543a0 0x55e7ce579040 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 0x55e7ce572f20 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce56bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/3336554784 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/3336554784 >> 172.21.15.31:0/3336554784 conn(0x55e7ce1b44e0 msgr2=0x55e7ce55c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:07.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.386+0000 7f0f64ff9640 0 client.26431 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:07.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.395+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/3336554784 shutdown_connections 2024-04-25T16:07:07.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.403+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/3336554784 wait complete. 2024-04-25T16:07:07.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.404+0000 7f0f8a7dc740 20 client.26431 trim_cache size 0 max 16384 2024-04-25T16:07:07.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.476+0000 7f0f8a7dc740 1 Processor -- start 2024-04-25T16:07:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.479+0000 7f0f8a7dc740 1 -- start start 2024-04-25T16:07:07.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.479+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce17ef60 0x55e7ce56a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.479+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce55ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.480+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce19f6b0 0x55e7ce55d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.480+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6000e590 con 0x55e7ce1974d0 2024-04-25T16:07:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.480+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0f600100d0 con 0x55e7ce17ef60 2024-04-25T16:07:07.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.480+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6000ec60 con 0x55e7ce19f6b0 2024-04-25T16:07:07.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.485+0000 7f0f816ca640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce17ef60 0x55e7ce56a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.485+0000 7f0f81ecb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce19f6b0 0x55e7ce55d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.485+0000 7f0f80ec9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce55ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.485+0000 7f0f80ec9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce55ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34128/0 (socket says 172.21.15.31:34128) 2024-04-25T16:07:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.485+0000 7f0f80ec9640 1 -- 172.21.15.31:0/15515541 learned_addr learned my addr 172.21.15.31:0/15515541 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.487+0000 7f0f816ca640 1 -- 172.21.15.31:0/15515541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce19f6b0 msgr2=0x55e7ce55d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.487+0000 7f0f816ca640 1 --2- 172.21.15.31:0/15515541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce19f6b0 0x55e7ce55d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.487+0000 7f0f816ca640 1 -- 172.21.15.31:0/15515541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 msgr2=0x55e7ce55ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:07.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.487+0000 7f0f816ca640 1 --2- 172.21.15.31:0/15515541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce55ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.487+0000 7f0f816ca640 1 -- 172.21.15.31:0/15515541 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7ce5622f0 con 0x55e7ce17ef60 2024-04-25T16:07:07.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.487+0000 7f0f816ca640 1 --2- 172.21.15.31:0/15515541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce17ef60 0x55e7ce56a220 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55e7ce561970 tx=0x7f0f6c0dbcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9961d31565844e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:07.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.490+0000 7f0f66ffd640 1 -- 172.21.15.31:0/15515541 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c0a67c0 con 0x55e7ce17ef60 2024-04-25T16:07:07.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.490+0000 7f0f66ffd640 1 -- 172.21.15.31:0/15515541 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f6c0aab70 con 0x55e7ce17ef60 2024-04-25T16:07:07.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.490+0000 7f0f66ffd640 1 -- 172.21.15.31:0/15515541 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c020dc0 con 0x55e7ce17ef60 2024-04-25T16:07:07.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.491+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/15515541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce17ef60 msgr2=0x55e7ce56a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:07.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.491+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/15515541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce17ef60 0x55e7ce56a220 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x55e7ce561970 tx=0x7f0f6c0dbcb0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.492+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/15515541 shutdown_connections 2024-04-25T16:07:07.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.492+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/15515541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce19f6b0 0x55e7ce55d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.492+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/15515541 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce55ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.492+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/15515541 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce17ef60 0x55e7ce56a220 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.492+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/15515541 >> 172.21.15.31:0/15515541 conn(0x55e7ce1b44e0 msgr2=0x55e7ce5666b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:07.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.497+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/15515541 shutdown_connections 2024-04-25T16:07:07.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.497+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/15515541 wait complete. 2024-04-25T16:07:07.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.498+0000 7f0f8a7dc740 1 -- start start 2024-04-25T16:07:07.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.498+0000 7f0f8a7dc740 1 Processor -- start 2024-04-25T16:07:07.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.499+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 0x55e7ce56f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.500+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.500+0000 7f0f8a7dc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce5df2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.500+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6000fa00 con 0x55e7ce1974d0 2024-04-25T16:07:07.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.500+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6000f330 con 0x55e7ce19f6b0 2024-04-25T16:07:07.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.500+0000 7f0f8a7dc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0f6000c570 con 0x55e7ce17ef60 2024-04-25T16:07:07.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.501+0000 7f0f8a7dc740 10 client.0 _ll_register_callbacks cb 0x7f0f78009dc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:07.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.510+0000 7f0f80ec9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.511+0000 7f0f80ec9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34142/0 (socket says 172.21.15.31:34142) 2024-04-25T16:07:07.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.511+0000 7f0f80ec9640 1 -- 172.21.15.31:0/621414574 learned_addr learned my addr 172.21.15.31:0/621414574 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:07.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.511+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce5df2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.511+0000 7f0f816ca640 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 0x55e7ce56f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.517+0000 7f0f81ecb640 1 -- 172.21.15.31:0/621414574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 msgr2=0x55e7ce56f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:07.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.517+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 0x55e7ce56f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.517+0000 7f0f81ecb640 1 -- 172.21.15.31:0/621414574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 msgr2=0x55e7ce56fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:07.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.517+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.517+0000 7f0f81ecb640 1 -- 172.21.15.31:0/621414574 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7ce1980f0 con 0x55e7ce19f6b0 2024-04-25T16:07:07.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.522+0000 7f0f81ecb640 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce5df2e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55e7ce44bb20 tx=0x7f0f78009cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1657ab715212fbe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:07.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.522+0000 7f0f64ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:07.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.522+0000 7f0f64ff9640 1 -- 172.21.15.31:0/621414574 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f7802b040 con 0x55e7ce19f6b0 2024-04-25T16:07:07.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.522+0000 7f0f64ff9640 1 -- 172.21.15.31:0/621414574 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0f78005670 con 0x55e7ce19f6b0 2024-04-25T16:07:07.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.523+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/621414574 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0f6c0aab70 con 0x55e7ce19f6b0 2024-04-25T16:07:07.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.523+0000 7f0f64ff9640 1 -- 172.21.15.31:0/621414574 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0f78009e70 con 0x55e7ce19f6b0 2024-04-25T16:07:07.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.524+0000 7f0f8a7dc740 10 client.26477 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:07.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.524+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/621414574 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e7ce4874b0 con 0x55e7ce19f6b0 2024-04-25T16:07:07.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.524+0000 7f0f64ff9640 1 -- 172.21.15.31:0/621414574 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f6c0aab70 con 0x55e7ce19f6b0 2024-04-25T16:07:07.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.525+0000 7f0f64ff9640 1 -- 172.21.15.31:0/621414574 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0f78023070 con 0x55e7ce19f6b0 2024-04-25T16:07:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.525+0000 7f0f8a7dc740 10 client.26477 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.525+0000 7f0f8a7dc740 10 client.26477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.525+0000 7f0f8a7dc740 20 client.26477 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.525+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7ce5e10d0 0x55e7ce5e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:07.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.525+0000 7f0f8a7dc740 4 client.26477 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:07:07.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.525+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/621414574 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e7ce4494a0 con 0x55e7ce5e10d0 2024-04-25T16:07:07.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.527+0000 7f0f816ca640 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7ce5e10d0 0x55e7ce5e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:07.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.531+0000 7f0f816ca640 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7ce5e10d0 0x55e7ce5e34f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=815a82b93510c4b4 server_cookie=27b873375e471ef0 in_seq=0 out_seq=0 2024-04-25T16:07:07.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.531+0000 7f0f64ff9640 10 client.26477 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f64ff9640 1 -- 172.21.15.31:0/621414574 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55e7ce4494a0 con 0x55e7ce5e10d0 2024-04-25T16:07:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f64ff9640 10 client.26477 handle_command_reply: tid=0 2024-04-25T16:07:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f8a7dc740 1 client.26477 shutdown 2024-04-25T16:07:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/621414574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 msgr2=0x55e7ce5df2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce5df2e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x55e7ce44bb20 tx=0x7f0f78009cc0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/621414574 shutdown_connections 2024-04-25T16:07:07.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7ce5e10d0 0x55e7ce5e34f0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7ce19f6b0 0x55e7ce5df2e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7ce1974d0 0x55e7ce56fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f8a7dc740 1 --2- 172.21.15.31:0/621414574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7ce17ef60 0x55e7ce56f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/621414574 >> 172.21.15.31:0/621414574 conn(0x55e7ce1b44e0 msgr2=0x55e7ce55ce20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:07.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.534+0000 7f0f64ff9640 0 client.26477 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:07.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.537+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/621414574 shutdown_connections 2024-04-25T16:07:07.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.538+0000 7f0f8a7dc740 1 -- 172.21.15.31:0/621414574 wait complete. 2024-04-25T16:07:07.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.538+0000 7f0f8a7dc740 20 client.26477 trim_cache size 0 max 16384 2024-04-25T16:07:07.540 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:07:07.540 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T16:07:07.540 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:07:07.540 INFO:teuthology.orchestra.run.smithi031.stdout: "b319b764-dc54-4483-b61a-5afbfacbd25e": { 2024-04-25T16:07:07.540 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:07:07.541 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "b319b764-dc54-4483-b61a-5afbfacbd25e", 2024-04-25T16:07:07.541 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:07:07.541 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:07:07.541 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:07:07.541 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:07:07.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.544+0000 7f0f826cc640 1 -- 172.21.15.31:0/830256800 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f680afbb0 msgr2=0x7f0f680b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:07.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.544+0000 7f0f826cc640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f680afbb0 0x7f0f680b20c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f0f74005e30 tx=0x7f0f74000f40 comp rx=0 tx=0).stop 2024-04-25T16:07:07.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.544+0000 7f0f826cc640 1 -- 172.21.15.31:0/830256800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f7c10a340 msgr2=0x7f0f7c11baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:07.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.544+0000 7f0f826cc640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f7c10a340 0x7f0f7c11baa0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c00b450 tx=0x7f0f6c00b910 comp rx=0 tx=0).stop 2024-04-25T16:07:07.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.550+0000 7f0f826cc640 1 -- 172.21.15.31:0/830256800 shutdown_connections 2024-04-25T16:07:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.550+0000 7f0f826cc640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f680afbb0 0x7f0f680b20c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.550+0000 7f0f826cc640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f7c10fe90 0x7f0f7c11a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.550+0000 7f0f826cc640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0f7c10b190 0x7f0f7c11a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.550+0000 7f0f826cc640 1 --2- 172.21.15.31:0/830256800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f7c10a340 0x7f0f7c11baa0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:07.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.550+0000 7f0f826cc640 1 -- 172.21.15.31:0/830256800 >> 172.21.15.31:0/830256800 conn(0x7f0f7c070320 msgr2=0x7f0f7c06a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:07.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.551+0000 7f0f826cc640 1 -- 172.21.15.31:0/830256800 shutdown_connections 2024-04-25T16:07:07.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:07.552+0000 7f0f826cc640 1 -- 172.21.15.31:0/830256800 wait complete. 2024-04-25T16:07:07.605 INFO:tasks.cephfs.filesystem:scrub status for tag:b319b764-dc54-4483-b61a-5afbfacbd25e - {'path': '/', 'tag': 'b319b764-dc54-4483-b61a-5afbfacbd25e', 'options': 'recursive,force'} 2024-04-25T16:07:07.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:07 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:07:07.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:07 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:07.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:07 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:07:07.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:07 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:07:07.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:07 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:08.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:07 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:07:08.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:07 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:08.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:07 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:07:08.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:07 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:07:08.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:07 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:08.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:07 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:07:08.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:07 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:08.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:07 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:07:08.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:07 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:07:08.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:07 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:09.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:08 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:09.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:08 smithi063 ceph-mon[32413]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 873 MiB/s wr, 2.02k op/s 2024-04-25T16:07:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:08 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:09.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:08 smithi186 ceph-mon[33258]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 873 MiB/s wr, 2.02k op/s 2024-04-25T16:07:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:08 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:09.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:08 smithi031 ceph-mon[29140]: pgmap v526: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 873 MiB/s wr, 2.02k op/s 2024-04-25T16:07:11.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:10 smithi031 ceph-mon[29140]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 760 MiB/s wr, 1.73k op/s 2024-04-25T16:07:11.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:10 smithi063 ceph-mon[32413]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 760 MiB/s wr, 1.73k op/s 2024-04-25T16:07:11.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:10 smithi186 ceph-mon[33258]: pgmap v527: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 760 MiB/s wr, 1.73k op/s 2024-04-25T16:07:13.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:12 smithi031 ceph-mon[29140]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 746 MiB/s wr, 1.80k op/s 2024-04-25T16:07:13.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:12 smithi063 ceph-mon[32413]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 746 MiB/s wr, 1.80k op/s 2024-04-25T16:07:13.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:12 smithi186 ceph-mon[33258]: pgmap v528: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 746 MiB/s wr, 1.80k op/s 2024-04-25T16:07:14.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:13 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:07:14.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:13 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:07:14.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:13 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:07:15.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:14 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:07:15.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:14 smithi031 ceph-mon[29140]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 942 MiB/s wr, 2.30k op/s 2024-04-25T16:07:15.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:14 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:07:15.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:14 smithi063 ceph-mon[32413]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 942 MiB/s wr, 2.30k op/s 2024-04-25T16:07:15.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:14 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:07:15.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:14 smithi186 ceph-mon[33258]: pgmap v529: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 942 MiB/s wr, 2.30k op/s 2024-04-25T16:07:17.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:16 smithi031 ceph-mon[29140]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 730 MiB/s wr, 1.90k op/s 2024-04-25T16:07:17.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:16 smithi063 ceph-mon[32413]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 730 MiB/s wr, 1.90k op/s 2024-04-25T16:07:17.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:16 smithi186 ceph-mon[33258]: pgmap v530: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 730 MiB/s wr, 1.90k op/s 2024-04-25T16:07:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:18 smithi063 ceph-mon[32413]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 914 MiB/s wr, 2.25k op/s 2024-04-25T16:07:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:18 smithi186 ceph-mon[33258]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 914 MiB/s wr, 2.25k op/s 2024-04-25T16:07:19.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:18 smithi031 ceph-mon[29140]: pgmap v531: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 914 MiB/s wr, 2.25k op/s 2024-04-25T16:07:19.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.964+0000 7fda47fff640 1 -- 172.21.15.31:0/3255700205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810b450 msgr2=0x7fda48109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:19.966 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.964+0000 7fda47fff640 1 --2- 172.21.15.31:0/3255700205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810b450 0x7fda48109ba0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fda3c009900 tx=0x7fda3c02a410 comp rx=0 tx=0).stop 2024-04-25T16:07:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.982+0000 7fda47fff640 1 -- 172.21.15.31:0/3255700205 shutdown_connections 2024-04-25T16:07:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.982+0000 7fda47fff640 1 --2- 172.21.15.31:0/3255700205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda48078df0 0x7fda4807b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.982+0000 7fda47fff640 1 --2- 172.21.15.31:0/3255700205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda4810a0e0 0x7fda4810a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.982+0000 7fda47fff640 1 --2- 172.21.15.31:0/3255700205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810b450 0x7fda48109ba0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.982+0000 7fda47fff640 1 -- 172.21.15.31:0/3255700205 >> 172.21.15.31:0/3255700205 conn(0x7fda48070330 msgr2=0x7fda480727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:19.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.982+0000 7fda47fff640 1 -- 172.21.15.31:0/3255700205 shutdown_connections 2024-04-25T16:07:19.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.982+0000 7fda47fff640 1 -- 172.21.15.31:0/3255700205 wait complete. 2024-04-25T16:07:19.987 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.986+0000 7fda47fff640 1 Processor -- start 2024-04-25T16:07:19.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.988+0000 7fda47fff640 1 -- start start 2024-04-25T16:07:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.989+0000 7fda47fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda48078df0 0x7fda48084930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.989+0000 7fda47fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810a0e0 0x7fda48084e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.989+0000 7fda47fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda4810b450 0x7fda481c13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.989+0000 7fda47fff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda4807ea20 con 0x7fda4810a0e0 2024-04-25T16:07:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.989+0000 7fda47fff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fda4807e8a0 con 0x7fda48078df0 2024-04-25T16:07:19.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.989+0000 7fda47fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda4807eba0 con 0x7fda4810b450 2024-04-25T16:07:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.992+0000 7fda467fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810a0e0 0x7fda48084e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.992+0000 7fda467fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810a0e0 0x7fda48084e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39044/0 (socket says 172.21.15.31:39044) 2024-04-25T16:07:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.992+0000 7fda467fc640 1 -- 172.21.15.31:0/2748297118 learned_addr learned my addr 172.21.15.31:0/2748297118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.992+0000 7fda477fe640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda4810b450 0x7fda481c13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:19.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.992+0000 7fda46ffd640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda48078df0 0x7fda48084930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.998+0000 7fda467fc640 1 -- 172.21.15.31:0/2748297118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda4810b450 msgr2=0x7fda481c13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.998+0000 7fda467fc640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda4810b450 0x7fda481c13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.998+0000 7fda467fc640 1 -- 172.21.15.31:0/2748297118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda48078df0 msgr2=0x7fda48084930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.998+0000 7fda467fc640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda48078df0 0x7fda48084930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:19.998+0000 7fda467fc640 1 -- 172.21.15.31:0/2748297118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda481c19e0 con 0x7fda4810a0e0 2024-04-25T16:07:20.002 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.001+0000 7fda467fc640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810a0e0 0x7fda48084e70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fda3800d7a0 tx=0x7fda3800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:20.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.010+0000 7fda2ffff640 1 -- 172.21.15.31:0/2748297118 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38017070 con 0x7fda4810a0e0 2024-04-25T16:07:20.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.010+0000 7fda47fff640 1 -- 172.21.15.31:0/2748297118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda481c1cd0 con 0x7fda4810a0e0 2024-04-25T16:07:20.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.010+0000 7fda47fff640 1 -- 172.21.15.31:0/2748297118 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda481c21d0 con 0x7fda4810a0e0 2024-04-25T16:07:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.011+0000 7fda2ffff640 1 -- 172.21.15.31:0/2748297118 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda380054d0 con 0x7fda4810a0e0 2024-04-25T16:07:20.012 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.011+0000 7fda2ffff640 1 -- 172.21.15.31:0/2748297118 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38004c10 con 0x7fda4810a0e0 2024-04-25T16:07:20.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.019+0000 7fda2ffff640 1 -- 172.21.15.31:0/2748297118 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fda38010460 con 0x7fda4810a0e0 2024-04-25T16:07:20.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.020+0000 7fda2ffff640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda300afcf0 0x7fda300b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.021+0000 7fda2ffff640 1 -- 172.21.15.31:0/2748297118 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fda380df590 con 0x7fda4810a0e0 2024-04-25T16:07:20.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.023+0000 7fda46ffd640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda300afcf0 0x7fda300b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.043+0000 7fda54429740 1 Processor -- start 2024-04-25T16:07:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.043+0000 7fda54429740 1 -- start start 2024-04-25T16:07:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.044+0000 7fda54429740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722e011b0 0x5617231a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.044+0000 7fda54429740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722dd82f0 0x5617231a2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.044+0000 7fda54429740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722de04d0 0x5617231a8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.044+0000 7fda54429740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561723107460 con 0x561722de04d0 2024-04-25T16:07:20.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.044+0000 7fda54429740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5617230c1590 con 0x561722dd82f0 2024-04-25T16:07:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.044+0000 7fda54429740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561723103420 con 0x561722e011b0 2024-04-25T16:07:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.045+0000 7fda467fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722dd82f0 0x5617231a2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.045+0000 7fda467fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722dd82f0 0x5617231a2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:59522/0 (socket says 172.21.15.31:59522) 2024-04-25T16:07:20.046 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.045+0000 7fda467fc640 1 -- 172.21.15.31:0/4051745122 learned_addr learned my addr 172.21.15.31:0/4051745122 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:20.047 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.046+0000 7fda467fc640 1 -- 172.21.15.31:0/4051745122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722e011b0 msgr2=0x5617231a1b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:07:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.046+0000 7fda467fc640 1 --2- 172.21.15.31:0/4051745122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722e011b0 0x5617231a1b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.046+0000 7fda467fc640 1 -- 172.21.15.31:0/4051745122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722de04d0 msgr2=0x5617231a8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.046+0000 7fda467fc640 1 --2- 172.21.15.31:0/4051745122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722de04d0 0x5617231a8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.046+0000 7fda467fc640 1 -- 172.21.15.31:0/4051745122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617230ce0d0 con 0x561722dd82f0 2024-04-25T16:07:20.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.047+0000 7fda467fc640 1 --2- 172.21.15.31:0/4051745122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722dd82f0 0x5617231a2090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fda38005c20 tx=0x7fda380b0280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=874254cf8a9508c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:20.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.059+0000 7fda2d7fa640 1 -- 172.21.15.31:0/4051745122 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38020040 con 0x561722dd82f0 2024-04-25T16:07:20.060 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.059+0000 7fda2d7fa640 1 -- 172.21.15.31:0/4051745122 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda380a8120 con 0x561722dd82f0 2024-04-25T16:07:20.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.060+0000 7fda54429740 1 -- 172.21.15.31:0/4051745122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722dd82f0 msgr2=0x5617231a2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:20.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.060+0000 7fda54429740 1 --2- 172.21.15.31:0/4051745122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722dd82f0 0x5617231a2090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fda38005c20 tx=0x7fda380b0280 comp rx=0 tx=0).stop 2024-04-25T16:07:20.066 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.065+0000 7fda46ffd640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda300afcf0 0x7fda300b2200 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fda3c02aa00 tx=0x7fda3c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:20.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.081+0000 7fda54429740 1 -- 172.21.15.31:0/4051745122 shutdown_connections 2024-04-25T16:07:20.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.081+0000 7fda54429740 1 --2- 172.21.15.31:0/4051745122 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722de04d0 0x5617231a8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.082 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.081+0000 7fda54429740 1 --2- 172.21.15.31:0/4051745122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722dd82f0 0x5617231a2090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.081+0000 7fda54429740 1 --2- 172.21.15.31:0/4051745122 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722e011b0 0x5617231a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.081+0000 7fda54429740 1 -- 172.21.15.31:0/4051745122 >> 172.21.15.31:0/4051745122 conn(0x561722df5300 msgr2=0x56172319d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.081+0000 7fda54429740 1 -- 172.21.15.31:0/4051745122 shutdown_connections 2024-04-25T16:07:20.083 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.081+0000 7fda54429740 1 -- 172.21.15.31:0/4051745122 wait complete. 2024-04-25T16:07:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.082+0000 7fda54429740 1 -- start start 2024-04-25T16:07:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.082+0000 7fda54429740 1 Processor -- start 2024-04-25T16:07:20.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.083+0000 7fda54429740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722dd82f0 0x5617231ab850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.083+0000 7fda54429740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722de04d0 0x5617231abd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.083+0000 7fda54429740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722e011b0 0x5617231b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.083+0000 7fda54429740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5617231a99d0 con 0x561722e011b0 2024-04-25T16:07:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.083+0000 7fda54429740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56172308eb60 con 0x561722de04d0 2024-04-25T16:07:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.083+0000 7fda54429740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5617231a9b50 con 0x561722dd82f0 2024-04-25T16:07:20.085 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.083+0000 7fda54429740 10 client.0 _ll_register_callbacks cb 0x5617230b32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.084+0000 7fda477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722e011b0 0x5617231b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.084+0000 7fda477fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722e011b0 0x5617231b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39050/0 (socket says 172.21.15.31:39050) 2024-04-25T16:07:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.084+0000 7fda477fe640 1 -- 172.21.15.31:0/57733777 learned_addr learned my addr 172.21.15.31:0/57733777 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.085+0000 7fda477fe640 1 -- 172.21.15.31:0/57733777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722dd82f0 msgr2=0x5617231ab850 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.085+0000 7fda477fe640 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722dd82f0 0x5617231ab850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.085+0000 7fda477fe640 1 -- 172.21.15.31:0/57733777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722de04d0 msgr2=0x5617231abd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.085+0000 7fda477fe640 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722de04d0 0x5617231abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.085+0000 7fda477fe640 1 -- 172.21.15.31:0/57733777 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561722e19210 con 0x561722e011b0 2024-04-25T16:07:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.110+0000 7fda477fe640 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722e011b0 0x5617231b2cf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fda40010670 tx=0x7fda40010b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=868f33912cca707b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:20.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.110+0000 7fda2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:07:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.115+0000 7fda2aff5640 1 -- 172.21.15.31:0/57733777 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda4000ead0 con 0x561722e011b0 2024-04-25T16:07:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.116+0000 7fda2aff5640 1 -- 172.21.15.31:0/57733777 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fda4000ec70 con 0x561722e011b0 2024-04-25T16:07:20.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.116+0000 7fda54429740 1 -- 172.21.15.31:0/57733777 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda380a8120 con 0x561722e011b0 2024-04-25T16:07:20.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.117+0000 7fda2aff5640 1 -- 172.21.15.31:0/57733777 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fda4000ee70 con 0x561722e011b0 2024-04-25T16:07:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.118+0000 7fda54429740 10 client.16848 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:20.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.118+0000 7fda54429740 1 -- 172.21.15.31:0/57733777 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561722ee5620 con 0x561722e011b0 2024-04-25T16:07:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.119+0000 7fda2aff5640 1 -- 172.21.15.31:0/57733777 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda380a8120 con 0x561722e011b0 2024-04-25T16:07:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.119+0000 7fda2aff5640 1 -- 172.21.15.31:0/57733777 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fda4000f040 con 0x561722e011b0 2024-04-25T16:07:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.120+0000 7fda54429740 10 client.16848 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:20.121 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.120+0000 7fda54429740 10 client.16848 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.120+0000 7fda54429740 20 client.16848 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.120+0000 7fda54429740 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561722dbfd80 0x5617231b96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.120+0000 7fda54429740 4 client.16848 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:20.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.120+0000 7fda54429740 1 -- 172.21.15.31:0/57733777 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561723115070 con 0x561722dbfd80 2024-04-25T16:07:20.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.122+0000 7fda46ffd640 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561722dbfd80 0x5617231b96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.133+0000 7fda46ffd640 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561722dbfd80 0x5617231b96a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32612bce49824de server_cookie=13cea03975b81317 in_seq=0 out_seq=0 2024-04-25T16:07:20.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.137+0000 7fda2aff5640 10 client.16848 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.141+0000 7fda2aff5640 1 -- 172.21.15.31:0/57733777 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561723115070 con 0x561722dbfd80 2024-04-25T16:07:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.141+0000 7fda2aff5640 10 client.16848 handle_command_reply: tid=0 2024-04-25T16:07:20.142 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.141+0000 7fda54429740 1 client.16848 shutdown 2024-04-25T16:07:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.141+0000 7fda54429740 1 -- 172.21.15.31:0/57733777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722e011b0 msgr2=0x5617231b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.141+0000 7fda54429740 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722e011b0 0x5617231b2cf0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fda40010670 tx=0x7fda40010b30 comp rx=0 tx=0).stop 2024-04-25T16:07:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda54429740 1 -- 172.21.15.31:0/57733777 shutdown_connections 2024-04-25T16:07:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda54429740 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561722dbfd80 0x5617231b96a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda54429740 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722e011b0 0x5617231b2cf0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda54429740 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722de04d0 0x5617231abd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda54429740 1 --2- 172.21.15.31:0/57733777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722dd82f0 0x5617231ab850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda54429740 1 -- 172.21.15.31:0/57733777 >> 172.21.15.31:0/57733777 conn(0x561722df5300 msgr2=0x5617231a03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:20.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda2aff5640 0 client.16848 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda54429740 1 -- 172.21.15.31:0/57733777 shutdown_connections 2024-04-25T16:07:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda54429740 1 -- 172.21.15.31:0/57733777 wait complete. 2024-04-25T16:07:20.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.142+0000 7fda54429740 20 client.16848 trim_cache size 0 max 16384 2024-04-25T16:07:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.183+0000 7fda54429740 1 Processor -- start 2024-04-25T16:07:20.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.184+0000 7fda54429740 1 -- start start 2024-04-25T16:07:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.184+0000 7fda54429740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722dd82f0 0x5617231a09b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.184+0000 7fda54429740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231a0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.184+0000 7fda54429740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 0x5617231b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.184+0000 7fda54429740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda3400e570 con 0x561722dd82f0 2024-04-25T16:07:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.184+0000 7fda54429740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fda340100b0 con 0x561722e011b0 2024-04-25T16:07:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.184+0000 7fda54429740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda3400ec40 con 0x561722de04d0 2024-04-25T16:07:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.186+0000 7fda467fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231a0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.186+0000 7fda467fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231a0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54356/0 (socket says 172.21.15.31:54356) 2024-04-25T16:07:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.186+0000 7fda467fc640 1 -- 172.21.15.31:0/1902996231 learned_addr learned my addr 172.21.15.31:0/1902996231 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:20.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.186+0000 7fda477fe640 1 --2- 172.21.15.31:0/1902996231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 0x5617231b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.193 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.187+0000 7fda46ffd640 1 --2- 172.21.15.31:0/1902996231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722dd82f0 0x5617231a09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.194+0000 7fda467fc640 1 -- 172.21.15.31:0/1902996231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 msgr2=0x5617231b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.194+0000 7fda467fc640 1 --2- 172.21.15.31:0/1902996231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 0x5617231b2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.194+0000 7fda467fc640 1 -- 172.21.15.31:0/1902996231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722dd82f0 msgr2=0x5617231a09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.194+0000 7fda467fc640 1 --2- 172.21.15.31:0/1902996231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722dd82f0 0x5617231a09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.195 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.194+0000 7fda467fc640 1 -- 172.21.15.31:0/1902996231 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617231b3150 con 0x561722de04d0 2024-04-25T16:07:20.196 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.195+0000 7fda467fc640 1 --2- 172.21.15.31:0/1902996231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231a0ef0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fda3800b6d0 tx=0x7fda380a7260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d69db3abbc8a605a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.198+0000 7fda2cff9640 1 -- 172.21.15.31:0/1902996231 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda380c0070 con 0x561722de04d0 2024-04-25T16:07:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.198+0000 7fda2cff9640 1 -- 172.21.15.31:0/1902996231 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda380a8120 con 0x561722de04d0 2024-04-25T16:07:20.199 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.198+0000 7fda2cff9640 1 -- 172.21.15.31:0/1902996231 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda38020040 con 0x561722de04d0 2024-04-25T16:07:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.203+0000 7fda54429740 1 -- 172.21.15.31:0/1902996231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 msgr2=0x5617231a0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:20.204 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.203+0000 7fda54429740 1 --2- 172.21.15.31:0/1902996231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231a0ef0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fda3800b6d0 tx=0x7fda380a7260 comp rx=0 tx=0).stop 2024-04-25T16:07:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.209+0000 7fda54429740 1 -- 172.21.15.31:0/1902996231 shutdown_connections 2024-04-25T16:07:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.209+0000 7fda54429740 1 --2- 172.21.15.31:0/1902996231 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 0x5617231b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.209+0000 7fda54429740 1 --2- 172.21.15.31:0/1902996231 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231a0ef0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.210 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.209+0000 7fda54429740 1 --2- 172.21.15.31:0/1902996231 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722dd82f0 0x5617231a09b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.209+0000 7fda54429740 1 -- 172.21.15.31:0/1902996231 >> 172.21.15.31:0/1902996231 conn(0x561722df5300 msgr2=0x5617231a02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:20.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.220+0000 7fda54429740 1 -- 172.21.15.31:0/1902996231 shutdown_connections 2024-04-25T16:07:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.220+0000 7fda54429740 1 -- 172.21.15.31:0/1902996231 wait complete. 2024-04-25T16:07:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.221+0000 7fda54429740 1 -- start start 2024-04-25T16:07:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.221+0000 7fda54429740 1 Processor -- start 2024-04-25T16:07:20.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.221+0000 7fda54429740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722dd82f0 0x5617231afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.222+0000 7fda54429740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.223 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.222+0000 7fda54429740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 0x5617231b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.222+0000 7fda54429740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fda3400f9e0 con 0x561722dd82f0 2024-04-25T16:07:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.222+0000 7fda54429740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fda3400f310 con 0x561722e011b0 2024-04-25T16:07:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.222+0000 7fda54429740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fda3400c550 con 0x561722de04d0 2024-04-25T16:07:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.223+0000 7fda54429740 10 client.0 _ll_register_callbacks cb 0x7fda40009620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.223+0000 7fda467fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.223+0000 7fda467fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54366/0 (socket says 172.21.15.31:54366) 2024-04-25T16:07:20.224 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.223+0000 7fda467fc640 1 -- 172.21.15.31:0/2134325836 learned_addr learned my addr 172.21.15.31:0/2134325836 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.223+0000 7fda477fe640 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 0x5617231b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.224+0000 7fda467fc640 1 -- 172.21.15.31:0/2134325836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 msgr2=0x5617231b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.224+0000 7fda467fc640 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 0x5617231b11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.224+0000 7fda467fc640 1 -- 172.21.15.31:0/2134325836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722dd82f0 msgr2=0x5617231afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.224+0000 7fda467fc640 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722dd82f0 0x5617231afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:20.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.224+0000 7fda467fc640 1 -- 172.21.15.31:0/2134325836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617231b1790 con 0x561722de04d0 2024-04-25T16:07:20.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.225+0000 7fda467fc640 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231b0310 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fda3800d770 tx=0x7fda3800dfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=775d19cba51e9ba7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:20.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.225+0000 7fda2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:20.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.225+0000 7fda2aff5640 1 -- 172.21.15.31:0/2134325836 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda380c0070 con 0x561722de04d0 2024-04-25T16:07:20.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.226+0000 7fda54429740 1 -- 172.21.15.31:0/2134325836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fda380a8120 con 0x561722de04d0 2024-04-25T16:07:20.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.226+0000 7fda2aff5640 1 -- 172.21.15.31:0/2134325836 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fda380b0600 con 0x561722de04d0 2024-04-25T16:07:20.227 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.226+0000 7fda2aff5640 1 -- 172.21.15.31:0/2134325836 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fda38020040 con 0x561722de04d0 2024-04-25T16:07:20.228 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.227+0000 7fda54429740 10 client.26446 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:20.229 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.227+0000 7fda54429740 1 -- 172.21.15.31:0/2134325836 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5617231b2a20 con 0x561722de04d0 2024-04-25T16:07:20.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.228+0000 7fda2aff5640 1 -- 172.21.15.31:0/2134325836 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda380a8120 con 0x561722de04d0 2024-04-25T16:07:20.230 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.229+0000 7fda2aff5640 1 -- 172.21.15.31:0/2134325836 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fda380a3390 con 0x561722de04d0 2024-04-25T16:07:20.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.230+0000 7fda54429740 10 client.26446 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.230+0000 7fda54429740 10 client.26446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.230+0000 7fda54429740 20 client.26446 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.230+0000 7fda54429740 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561722d951c0 0x561723223a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.230+0000 7fda54429740 4 client.26446 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:07:20.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.230+0000 7fda54429740 1 -- 172.21.15.31:0/2134325836 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56172310eee0 con 0x561722d951c0 2024-04-25T16:07:20.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.242+0000 7fda46ffd640 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561722d951c0 0x561723223a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.247+0000 7fda2aff5640 10 client.26446 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:20.248 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:20.247+0000 7fda46ffd640 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561722d951c0 0x561723223a20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bed214e6296ecf0e server_cookie=363a2d7c72189669 in_seq=0 out_seq=0 2024-04-25T16:07:21.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:21 smithi063 ceph-mon[32413]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 805 MiB/s wr, 1.89k op/s 2024-04-25T16:07:21.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:21 smithi186 ceph-mon[33258]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 805 MiB/s wr, 1.89k op/s 2024-04-25T16:07:21.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:21 smithi031 ceph-mon[29140]: pgmap v532: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 805 MiB/s wr, 1.89k op/s 2024-04-25T16:07:23.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:23 smithi063 ceph-mon[32413]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 798 MiB/s wr, 1.77k op/s 2024-04-25T16:07:23.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:23 smithi186 ceph-mon[33258]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 798 MiB/s wr, 1.77k op/s 2024-04-25T16:07:23.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:23 smithi031 ceph-mon[29140]: pgmap v533: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 798 MiB/s wr, 1.77k op/s 2024-04-25T16:07:23.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.647+0000 7fda2aff5640 1 -- 172.21.15.31:0/2134325836 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56172310eee0 con 0x561722d951c0 2024-04-25T16:07:23.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.647+0000 7fda2aff5640 10 client.26446 handle_command_reply: tid=0 2024-04-25T16:07:23.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.647+0000 7fda54429740 1 client.26446 shutdown 2024-04-25T16:07:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.649+0000 7fda54429740 1 -- 172.21.15.31:0/2134325836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 msgr2=0x5617231b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:23.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.649+0000 7fda54429740 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231b0310 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fda3800d770 tx=0x7fda3800dfd0 comp rx=0 tx=0).stop 2024-04-25T16:07:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.650+0000 7fda54429740 1 -- 172.21.15.31:0/2134325836 shutdown_connections 2024-04-25T16:07:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.650+0000 7fda54429740 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561722d951c0 0x561723223a20 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.650+0000 7fda54429740 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561722e011b0 0x5617231b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.650+0000 7fda54429740 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561722de04d0 0x5617231b0310 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.650+0000 7fda54429740 1 --2- 172.21.15.31:0/2134325836 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561722dd82f0 0x5617231afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.650+0000 7fda54429740 1 -- 172.21.15.31:0/2134325836 >> 172.21.15.31:0/2134325836 conn(0x561722df5300 msgr2=0x5617231a0a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.651+0000 7fda2aff5640 0 client.26446 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.651+0000 7fda54429740 1 -- 172.21.15.31:0/2134325836 shutdown_connections 2024-04-25T16:07:23.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.651+0000 7fda54429740 1 -- 172.21.15.31:0/2134325836 wait complete. 2024-04-25T16:07:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.651+0000 7fda54429740 20 client.26446 trim_cache size 0 max 16384 2024-04-25T16:07:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:07:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:07:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:07:23.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:07:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.656+0000 7fda47fff640 1 -- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda300afcf0 msgr2=0x7fda300b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.656+0000 7fda47fff640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda300afcf0 0x7fda300b2200 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fda3c02aa00 tx=0x7fda3c031400 comp rx=0 tx=0).stop 2024-04-25T16:07:23.658 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.656+0000 7fda47fff640 1 -- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810a0e0 msgr2=0x7fda48084e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.656+0000 7fda47fff640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810a0e0 0x7fda48084e70 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fda3800d7a0 tx=0x7fda3800dc60 comp rx=0 tx=0).stop 2024-04-25T16:07:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.657+0000 7fda47fff640 1 -- 172.21.15.31:0/2748297118 shutdown_connections 2024-04-25T16:07:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.657+0000 7fda47fff640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fda300afcf0 0x7fda300b2200 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.657+0000 7fda47fff640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda4810b450 0x7fda481c13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.657+0000 7fda47fff640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fda4810a0e0 0x7fda48084e70 secure :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fda3800d7a0 tx=0x7fda3800dc60 comp rx=0 tx=0).stop 2024-04-25T16:07:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.657+0000 7fda47fff640 1 --2- 172.21.15.31:0/2748297118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda48078df0 0x7fda48084930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.657+0000 7fda47fff640 1 -- 172.21.15.31:0/2748297118 >> 172.21.15.31:0/2748297118 conn(0x7fda48070330 msgr2=0x7fda48072750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.657+0000 7fda47fff640 1 -- 172.21.15.31:0/2748297118 shutdown_connections 2024-04-25T16:07:23.659 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:23.657+0000 7fda47fff640 1 -- 172.21.15.31:0/2748297118 wait complete. 2024-04-25T16:07:25.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:25 smithi063 ceph-mon[32413]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 970 MiB/s wr, 2.00k op/s 2024-04-25T16:07:25.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:25 smithi186 ceph-mon[33258]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 970 MiB/s wr, 2.00k op/s 2024-04-25T16:07:25.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:25 smithi031 ceph-mon[29140]: pgmap v534: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 970 MiB/s wr, 2.00k op/s 2024-04-25T16:07:27.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:27 smithi063 ceph-mon[32413]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 765 MiB/s wr, 1.51k op/s 2024-04-25T16:07:27.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:27 smithi186 ceph-mon[33258]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 765 MiB/s wr, 1.51k op/s 2024-04-25T16:07:27.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:27 smithi031 ceph-mon[29140]: pgmap v535: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 765 MiB/s wr, 1.51k op/s 2024-04-25T16:07:29.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:29 smithi063 ceph-mon[32413]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 916 MiB/s wr, 1.86k op/s 2024-04-25T16:07:29.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:29 smithi186 ceph-mon[33258]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 916 MiB/s wr, 1.86k op/s 2024-04-25T16:07:29.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:29 smithi031 ceph-mon[29140]: pgmap v536: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 916 MiB/s wr, 1.86k op/s 2024-04-25T16:07:31.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:31 smithi063 ceph-mon[32413]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 784 MiB/s wr, 1.59k op/s 2024-04-25T16:07:31.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:31 smithi186 ceph-mon[33258]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 784 MiB/s wr, 1.59k op/s 2024-04-25T16:07:31.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:31 smithi031 ceph-mon[29140]: pgmap v537: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 784 MiB/s wr, 1.59k op/s 2024-04-25T16:07:33.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:33 smithi063 ceph-mon[32413]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 767 MiB/s wr, 1.58k op/s 2024-04-25T16:07:33.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:33 smithi186 ceph-mon[33258]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 767 MiB/s wr, 1.58k op/s 2024-04-25T16:07:33.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:33 smithi031 ceph-mon[29140]: pgmap v538: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 767 MiB/s wr, 1.58k op/s 2024-04-25T16:07:35.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:35 smithi063 ceph-mon[32413]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 958 MiB/s wr, 2.57k op/s 2024-04-25T16:07:35.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:35 smithi186 ceph-mon[33258]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 958 MiB/s wr, 2.57k op/s 2024-04-25T16:07:35.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:35 smithi031 ceph-mon[29140]: pgmap v539: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 958 MiB/s wr, 2.57k op/s 2024-04-25T16:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.487+0000 7f728a84d640 1 -- 172.21.15.31:0/3539723935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f728410a340 msgr2=0x7f728410ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.487+0000 7f728a84d640 1 --2- 172.21.15.31:0/3539723935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f728410a340 0x7f728410ac50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7274009860 tx=0x7f727402a360 comp rx=0 tx=0).stop 2024-04-25T16:07:37.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.487+0000 7f728a84d640 1 -- 172.21.15.31:0/3539723935 shutdown_connections 2024-04-25T16:07:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.487+0000 7f728a84d640 1 --2- 172.21.15.31:0/3539723935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f728410feb0 0x7f72841102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.487+0000 7f728a84d640 1 --2- 172.21.15.31:0/3539723935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f728410b190 0x7f728410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.487+0000 7f728a84d640 1 --2- 172.21.15.31:0/3539723935 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f728410a340 0x7f728410ac50 secure :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f7274009860 tx=0x7f727402a360 comp rx=0 tx=0).stop 2024-04-25T16:07:37.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.487+0000 7f728a84d640 1 -- 172.21.15.31:0/3539723935 >> 172.21.15.31:0/3539723935 conn(0x7f7284070330 msgr2=0x7f72840727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.498+0000 7f728a84d640 1 -- 172.21.15.31:0/3539723935 shutdown_connections 2024-04-25T16:07:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.498+0000 7f728a84d640 1 -- 172.21.15.31:0/3539723935 wait complete. 2024-04-25T16:07:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.499+0000 7f728a84d640 1 Processor -- start 2024-04-25T16:07:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.500+0000 7f728a84d640 1 -- start start 2024-04-25T16:07:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.500+0000 7f728a84d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f728410b190 0x7f7284116d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.500+0000 7f728a84d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f728410feb0 0x7f7284117280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.500+0000 7f728a84d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72841a59e0 0x7f72841a7e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.500+0000 7f728a84d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f728411e2f0 con 0x7f72841a59e0 2024-04-25T16:07:37.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.500+0000 7f728a84d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f728411e170 con 0x7f728410b190 2024-04-25T16:07:37.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.500+0000 7f728a84d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f728411e470 con 0x7f728410feb0 2024-04-25T16:07:37.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.502+0000 7f728a04c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72841a59e0 0x7f72841a7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.502+0000 7f728a04c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72841a59e0 0x7f72841a7e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43594/0 (socket says 172.21.15.31:43594) 2024-04-25T16:07:37.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.502+0000 7f728a04c640 1 -- 172.21.15.31:0/1881056538 learned_addr learned my addr 172.21.15.31:0/1881056538 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:37.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.503+0000 7f728904a640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f728410feb0 0x7f7284117280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.503+0000 7f728984b640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f728410b190 0x7f7284116d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.506 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.505+0000 7f728a04c640 1 -- 172.21.15.31:0/1881056538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f728410feb0 msgr2=0x7f7284117280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.505+0000 7f728a04c640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f728410feb0 0x7f7284117280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.505+0000 7f728a04c640 1 -- 172.21.15.31:0/1881056538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f728410b190 msgr2=0x7f7284116d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.505+0000 7f728a04c640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f728410b190 0x7f7284116d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.505+0000 7f728a04c640 1 -- 172.21.15.31:0/1881056538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72841a8440 con 0x7f72841a59e0 2024-04-25T16:07:37.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.506+0000 7f728a04c640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72841a59e0 0x7f72841a7e00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f728000b3f0 tx=0x7f728000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.507+0000 7f727affd640 1 -- 172.21.15.31:0/1881056538 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280015020 con 0x7f72841a59e0 2024-04-25T16:07:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.507+0000 7f727affd640 1 -- 172.21.15.31:0/1881056538 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7280011ce0 con 0x7f72841a59e0 2024-04-25T16:07:37.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.507+0000 7f727affd640 1 -- 172.21.15.31:0/1881056538 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7280010db0 con 0x7f72841a59e0 2024-04-25T16:07:37.510 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.508+0000 7f728a84d640 1 -- 172.21.15.31:0/1881056538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72841a8730 con 0x7f72841a59e0 2024-04-25T16:07:37.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.508+0000 7f728a84d640 1 -- 172.21.15.31:0/1881056538 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72841a8c30 con 0x7f72841a59e0 2024-04-25T16:07:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.516+0000 7f727affd640 1 -- 172.21.15.31:0/1881056538 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7280018020 con 0x7f72841a59e0 2024-04-25T16:07:37.519 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.517+0000 7f727affd640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72700afc70 0x7f72700b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.519+0000 7f728984b640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72700afc70 0x7f72700b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.520 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.519+0000 7f727affd640 1 -- 172.21.15.31:0/1881056538 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f72800a7850 con 0x7f72841a59e0 2024-04-25T16:07:37.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.529+0000 7f728984b640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72700afc70 0x7f72700b2180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f727402a950 tx=0x7f7274009280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:37.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.568+0000 7f7292963740 1 Processor -- start 2024-04-25T16:07:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.569+0000 7f7292963740 1 -- start start 2024-04-25T16:07:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.569+0000 7f7292963740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 0x5614a1353b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.569+0000 7f7292963740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f8a2f0 0x5614a13540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.569+0000 7f7292963740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 0x5614a135afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.569+0000 7f7292963740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5614a12b9460 con 0x5614a0f924d0 2024-04-25T16:07:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.569+0000 7f7292963740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5614a1273590 con 0x5614a0f8a2f0 2024-04-25T16:07:37.572 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.569+0000 7f7292963740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5614a12b5420 con 0x5614a0fb31b0 2024-04-25T16:07:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.572+0000 7f728a04c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 0x5614a135afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.572+0000 7f728a04c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 0x5614a135afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43610/0 (socket says 172.21.15.31:43610) 2024-04-25T16:07:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.572+0000 7f728a04c640 1 -- 172.21.15.31:0/2521946197 learned_addr learned my addr 172.21.15.31:0/2521946197 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.573+0000 7f728984b640 1 --2- 172.21.15.31:0/2521946197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 0x5614a1353b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.574+0000 7f728904a640 1 --2- 172.21.15.31:0/2521946197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f8a2f0 0x5614a13540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.576+0000 7f728904a640 1 -- 172.21.15.31:0/2521946197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 msgr2=0x5614a1353b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.576+0000 7f728904a640 1 --2- 172.21.15.31:0/2521946197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 0x5614a1353b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.576+0000 7f728904a640 1 -- 172.21.15.31:0/2521946197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 msgr2=0x5614a135afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.576+0000 7f728904a640 1 --2- 172.21.15.31:0/2521946197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 0x5614a135afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.577 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.576+0000 7f728904a640 1 -- 172.21.15.31:0/2521946197 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614a12800d0 con 0x5614a0f8a2f0 2024-04-25T16:07:37.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.578+0000 7f728904a640 1 --2- 172.21.15.31:0/2521946197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f8a2f0 0x5614a13540a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f727c009f70 tx=0x7f727c00f890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1335f1aea8447d4c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.588+0000 7f726ffff640 1 -- 172.21.15.31:0/2521946197 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c01d070 con 0x5614a0f8a2f0 2024-04-25T16:07:37.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.588+0000 7f726ffff640 1 -- 172.21.15.31:0/2521946197 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f727c00caf0 con 0x5614a0f8a2f0 2024-04-25T16:07:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.590+0000 7f7292963740 1 -- 172.21.15.31:0/2521946197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f8a2f0 msgr2=0x5614a13540a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.590+0000 7f7292963740 1 --2- 172.21.15.31:0/2521946197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f8a2f0 0x5614a13540a0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f727c009f70 tx=0x7f727c00f890 comp rx=0 tx=0).stop 2024-04-25T16:07:37.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.590+0000 7f7292963740 1 -- 172.21.15.31:0/2521946197 shutdown_connections 2024-04-25T16:07:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.590+0000 7f7292963740 1 --2- 172.21.15.31:0/2521946197 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 0x5614a135afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.590+0000 7f7292963740 1 --2- 172.21.15.31:0/2521946197 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f8a2f0 0x5614a13540a0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.590+0000 7f7292963740 1 --2- 172.21.15.31:0/2521946197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 0x5614a1353b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.590+0000 7f7292963740 1 -- 172.21.15.31:0/2521946197 >> 172.21.15.31:0/2521946197 conn(0x5614a0fa7300 msgr2=0x5614a134f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:37.594 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.593+0000 7f7292963740 1 -- 172.21.15.31:0/2521946197 shutdown_connections 2024-04-25T16:07:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.594+0000 7f7292963740 1 -- 172.21.15.31:0/2521946197 wait complete. 2024-04-25T16:07:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.594+0000 7f7292963740 1 -- start start 2024-04-25T16:07:37.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.595+0000 7f7292963740 1 Processor -- start 2024-04-25T16:07:37.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.596+0000 7f7292963740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f924d0 0x5614a135d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.596+0000 7f7292963740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 0x5614a135dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.596+0000 7f7292963740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f71d80 0x5614a1364bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.596+0000 7f7292963740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5614a135ba40 con 0x5614a0f71d80 2024-04-25T16:07:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.596+0000 7f7292963740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5614a1240b60 con 0x5614a0f924d0 2024-04-25T16:07:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.596+0000 7f7292963740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5614a135bbc0 con 0x5614a0fb31b0 2024-04-25T16:07:37.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.596+0000 7f7292963740 10 client.0 _ll_register_callbacks cb 0x5614a12652b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:37.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.598+0000 7f728a04c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f71d80 0x5614a1364bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.599+0000 7f728a04c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f71d80 0x5614a1364bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43626/0 (socket says 172.21.15.31:43626) 2024-04-25T16:07:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.599+0000 7f728a04c640 1 -- 172.21.15.31:0/33669088 learned_addr learned my addr 172.21.15.31:0/33669088 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.599+0000 7f728904a640 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 0x5614a135dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.599+0000 7f728984b640 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f924d0 0x5614a135d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.600+0000 7f728904a640 1 -- 172.21.15.31:0/33669088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f924d0 msgr2=0x5614a135d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.600+0000 7f728904a640 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f924d0 0x5614a135d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.600+0000 7f728904a640 1 -- 172.21.15.31:0/33669088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f71d80 msgr2=0x5614a1364bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.600+0000 7f728904a640 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f71d80 0x5614a1364bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.600+0000 7f728904a640 1 -- 172.21.15.31:0/33669088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614a0fcb210 con 0x5614a0fb31b0 2024-04-25T16:07:37.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.601+0000 7f728a04c640 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f71d80 0x5614a1364bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:07:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.602+0000 7f728904a640 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 0x5614a135dc90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f727c01b040 tx=0x7f727c018780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8442aaef58c0094b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:37.603 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.602+0000 7f726d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.603+0000 7f726d7fa640 1 -- 172.21.15.31:0/33669088 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c026020 con 0x5614a0fb31b0 2024-04-25T16:07:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.603+0000 7f7292963740 1 -- 172.21.15.31:0/33669088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5614a12c7070 con 0x5614a0fb31b0 2024-04-25T16:07:37.604 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.603+0000 7f726d7fa640 1 -- 172.21.15.31:0/33669088 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f727c004bd0 con 0x5614a0fb31b0 2024-04-25T16:07:37.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.604+0000 7f726d7fa640 1 -- 172.21.15.31:0/33669088 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f727c01d070 con 0x5614a0fb31b0 2024-04-25T16:07:37.605 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.604+0000 7f7292963740 10 client.26455 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:37.606 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:07:37.606 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:07:37.607 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.606+0000 7f7292963740 1 -- 172.21.15.31:0/33669088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614a1097620 con 0x5614a0fb31b0 2024-04-25T16:07:37.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:37 smithi063 ceph-mon[32413]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 2.19k op/s 2024-04-25T16:07:37.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.607+0000 7f726d7fa640 1 -- 172.21.15.31:0/33669088 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5614a12c7070 con 0x5614a0fb31b0 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.607+0000 7f726d7fa640 1 -- 172.21.15.31:0/33669088 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f727c00b040 con 0x5614a0fb31b0 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.607+0000 7f7292963740 10 client.26455 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.607+0000 7f7292963740 10 client.26455 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.607+0000 7f7292963740 20 client.26455 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.607+0000 7f7292963740 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5614a0f471c0 0x5614a136b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.607+0000 7f7292963740 4 client.26455 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.607+0000 7f7292963740 1 -- 172.21.15.31:0/33669088 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614a12c0ee0 con 0x5614a0f471c0 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.608+0000 7f728984b640 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5614a0f471c0 0x5614a136b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.609+0000 7f726d7fa640 10 client.26455 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:37.630 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.614+0000 7f728984b640 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5614a0f471c0 0x5614a136b240 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b77dc515e63803c4 server_cookie=8b90d3e2744c81ad in_seq=0 out_seq=0 2024-04-25T16:07:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.618+0000 7f726d7fa640 1 -- 172.21.15.31:0/33669088 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5614a12c0ee0 con 0x5614a0f471c0 2024-04-25T16:07:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.618+0000 7f726d7fa640 10 client.26455 handle_command_reply: tid=0 2024-04-25T16:07:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.618+0000 7f7292963740 1 client.26455 shutdown 2024-04-25T16:07:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.622+0000 7f7292963740 1 -- 172.21.15.31:0/33669088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 msgr2=0x5614a135dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.622+0000 7f7292963740 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 0x5614a135dc90 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f727c01b040 tx=0x7f727c018780 comp rx=0 tx=0).stop 2024-04-25T16:07:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.622+0000 7f7292963740 1 -- 172.21.15.31:0/33669088 shutdown_connections 2024-04-25T16:07:37.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.622+0000 7f7292963740 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5614a0f471c0 0x5614a136b240 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.622+0000 7f7292963740 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f71d80 0x5614a1364bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.622+0000 7f7292963740 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0fb31b0 0x5614a135dc90 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.622+0000 7f7292963740 1 --2- 172.21.15.31:0/33669088 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f924d0 0x5614a135d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.622+0000 7f7292963740 1 -- 172.21.15.31:0/33669088 >> 172.21.15.31:0/33669088 conn(0x5614a0fa7300 msgr2=0x5614a1352430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.624+0000 7f726d7fa640 0 client.26455 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:37.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.626+0000 7f7292963740 1 -- 172.21.15.31:0/33669088 shutdown_connections 2024-04-25T16:07:37.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.626+0000 7f7292963740 1 -- 172.21.15.31:0/33669088 wait complete. 2024-04-25T16:07:37.638 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.635+0000 7f7292963740 20 client.26455 trim_cache size 0 max 16384 2024-04-25T16:07:37.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:37 smithi186 ceph-mon[33258]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 2.19k op/s 2024-04-25T16:07:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.694+0000 7f7292963740 1 Processor -- start 2024-04-25T16:07:37.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.695+0000 7f7292963740 1 -- start start 2024-04-25T16:07:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.695+0000 7f7292963740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f71d80 0x5614a1357b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.695+0000 7f7292963740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 0x5614a1364bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.695+0000 7f7292963740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0fb31b0 0x5614a1358040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.695+0000 7f7292963740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b1ad0 con 0x5614a0f924d0 2024-04-25T16:07:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.695+0000 7f7292963740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b3610 con 0x5614a0fb31b0 2024-04-25T16:07:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.695+0000 7f7292963740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b21a0 con 0x5614a0f71d80 2024-04-25T16:07:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.696+0000 7f728984b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f71d80 0x5614a1357b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.696+0000 7f728984b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f71d80 0x5614a1357b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48172/0 (socket says 172.21.15.31:48172) 2024-04-25T16:07:37.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.696+0000 7f728984b640 1 -- 172.21.15.31:0/1186744549 learned_addr learned my addr 172.21.15.31:0/1186744549 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.696+0000 7f728904a640 1 --2- 172.21.15.31:0/1186744549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 0x5614a1364bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.696+0000 7f728984b640 1 -- 172.21.15.31:0/1186744549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0fb31b0 msgr2=0x5614a1358040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.696+0000 7f728984b640 1 --2- 172.21.15.31:0/1186744549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0fb31b0 0x5614a1358040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.696+0000 7f728984b640 1 -- 172.21.15.31:0/1186744549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 msgr2=0x5614a1364bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.696+0000 7f728984b640 1 --2- 172.21.15.31:0/1186744549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 0x5614a1364bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.696+0000 7f728984b640 1 -- 172.21.15.31:0/1186744549 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614a0f8a530 con 0x5614a0f71d80 2024-04-25T16:07:37.698 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.697+0000 7f728984b640 1 --2- 172.21.15.31:0/1186744549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f71d80 0x5614a1357b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7274009860 tx=0x7f727402c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=438a4f8dccb1b347 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.697+0000 7f726f7fe640 1 -- 172.21.15.31:0/1186744549 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7274037040 con 0x5614a0f71d80 2024-04-25T16:07:37.699 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.698+0000 7f726f7fe640 1 -- 172.21.15.31:0/1186744549 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7274057070 con 0x5614a0f71d80 2024-04-25T16:07:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.699+0000 7f7292963740 1 -- 172.21.15.31:0/1186744549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f71d80 msgr2=0x5614a1357b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:37.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.699+0000 7f7292963740 1 --2- 172.21.15.31:0/1186744549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f71d80 0x5614a1357b00 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7274009860 tx=0x7f727402c000 comp rx=0 tx=0).stop 2024-04-25T16:07:37.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:37 smithi031 ceph-mon[29140]: pgmap v540: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 2.19k op/s 2024-04-25T16:07:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.700+0000 7f7292963740 1 -- 172.21.15.31:0/1186744549 shutdown_connections 2024-04-25T16:07:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.700+0000 7f7292963740 1 --2- 172.21.15.31:0/1186744549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0fb31b0 0x5614a1358040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.700+0000 7f7292963740 1 --2- 172.21.15.31:0/1186744549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0f924d0 0x5614a1364bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.700+0000 7f7292963740 1 --2- 172.21.15.31:0/1186744549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f71d80 0x5614a1357b00 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.700+0000 7f7292963740 1 -- 172.21.15.31:0/1186744549 >> 172.21.15.31:0/1186744549 conn(0x5614a0fa7300 msgr2=0x5614a1353aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:37.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.703+0000 7f7292963740 1 -- 172.21.15.31:0/1186744549 shutdown_connections 2024-04-25T16:07:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.704+0000 7f7292963740 1 -- 172.21.15.31:0/1186744549 wait complete. 2024-04-25T16:07:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.705+0000 7f7292963740 1 -- start start 2024-04-25T16:07:37.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.705+0000 7f7292963740 1 Processor -- start 2024-04-25T16:07:37.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.706+0000 7f7292963740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f71d80 0x5614a1361de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.706+0000 7f7292963740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f924d0 0x5614a1362320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.706+0000 7f7292963740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0fb31b0 0x5614a1363250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.706+0000 7f7292963740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b2f40 con 0x5614a0fb31b0 2024-04-25T16:07:37.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.706+0000 7f7292963740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f72841b2870 con 0x5614a0f71d80 2024-04-25T16:07:37.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.706+0000 7f7292963740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72841afab0 con 0x5614a0f924d0 2024-04-25T16:07:37.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.706+0000 7f7292963740 10 client.0 _ll_register_callbacks cb 0x7f727c01b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:37.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.707+0000 7f728904a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f924d0 0x5614a1362320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.707+0000 7f728904a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f924d0 0x5614a1362320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48186/0 (socket says 172.21.15.31:48186) 2024-04-25T16:07:37.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.707+0000 7f728904a640 1 -- 172.21.15.31:0/1716126679 learned_addr learned my addr 172.21.15.31:0/1716126679 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.707+0000 7f728984b640 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f71d80 0x5614a1361de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.707+0000 7f728a04c640 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0fb31b0 0x5614a1363250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.707+0000 7f728904a640 1 -- 172.21.15.31:0/1716126679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f71d80 msgr2=0x5614a1361de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.707+0000 7f728904a640 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f71d80 0x5614a1361de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.707+0000 7f728904a640 1 -- 172.21.15.31:0/1716126679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0fb31b0 msgr2=0x5614a1363250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.707+0000 7f728904a640 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0fb31b0 0x5614a1363250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:37.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.708+0000 7f728904a640 1 -- 172.21.15.31:0/1716126679 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614a1363810 con 0x5614a0f924d0 2024-04-25T16:07:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.708+0000 7f728904a640 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f924d0 0x5614a1362320 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f727c000f80 tx=0x7f727c00cca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8dde178b28414c3e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.709+0000 7f726d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.709+0000 7f726d7fa640 1 -- 172.21.15.31:0/1716126679 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f727c026040 con 0x5614a0f924d0 2024-04-25T16:07:37.710 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.709+0000 7f7292963740 1 -- 172.21.15.31:0/1716126679 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7274057070 con 0x5614a0f924d0 2024-04-25T16:07:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.709+0000 7f726d7fa640 1 -- 172.21.15.31:0/1716126679 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f727c00ce10 con 0x5614a0f924d0 2024-04-25T16:07:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.710+0000 7f726d7fa640 1 -- 172.21.15.31:0/1716126679 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f727c004780 con 0x5614a0f924d0 2024-04-25T16:07:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.710+0000 7f7292963740 10 client.26461 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:37.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.710+0000 7f7292963740 1 -- 172.21.15.31:0/1716126679 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5614a1364aa0 con 0x5614a0f924d0 2024-04-25T16:07:37.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.712+0000 7f726d7fa640 1 -- 172.21.15.31:0/1716126679 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7274057070 con 0x5614a0f924d0 2024-04-25T16:07:37.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.715+0000 7f726d7fa640 1 -- 172.21.15.31:0/1716126679 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f727c00b040 con 0x5614a0f924d0 2024-04-25T16:07:37.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.715+0000 7f7292963740 10 client.26461 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:37.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.715+0000 7f7292963740 10 client.26461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:37.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.715+0000 7f7292963740 20 client.26461 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:37.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.715+0000 7f7292963740 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5614a13d36b0 0x5614a13d5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:37.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.715+0000 7f7292963740 4 client.26461 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:07:37.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.715+0000 7f7292963740 1 -- 172.21.15.31:0/1716126679 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5614a123c5f0 con 0x5614a13d36b0 2024-04-25T16:07:37.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.716+0000 7f728984b640 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5614a13d36b0 0x5614a13d5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:37.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.717+0000 7f728984b640 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5614a13d36b0 0x5614a13d5ad0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9f53a691dca5f78 server_cookie=3f010a67bea0fdfd in_seq=0 out_seq=0 2024-04-25T16:07:37.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:37.718+0000 7f726d7fa640 10 client.26461 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.209+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/4044969870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410b9f0 msgr2=0x7fcc3410de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:38.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.209+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/4044969870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410b9f0 0x7fcc3410de10 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009900 tx=0x7fcc2402a440 comp rx=0 tx=0).stop 2024-04-25T16:07:38.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.210+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/4044969870 shutdown_connections 2024-04-25T16:07:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.210+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/4044969870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc34110f50 0x7fcc34113430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.210+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/4044969870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc3410e3e0 0x7fcc341108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.211+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/4044969870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410b9f0 0x7fcc3410de10 secure :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009900 tx=0x7fcc2402a440 comp rx=0 tx=0).stop 2024-04-25T16:07:38.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.211+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/4044969870 >> 172.21.15.31:0/4044969870 conn(0x7fcc340704e0 msgr2=0x7fcc34072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:38.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.216+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/4044969870 shutdown_connections 2024-04-25T16:07:38.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.218+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/4044969870 wait complete. 2024-04-25T16:07:38.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.220+0000 7fcc3b9ff640 1 Processor -- start 2024-04-25T16:07:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.220+0000 7fcc3b9ff640 1 -- start start 2024-04-25T16:07:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.220+0000 7fcc3b9ff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410e3e0 0x7fcc341a3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.220+0000 7fcc3b9ff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc34110f50 0x7fcc341a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.220+0000 7fcc3b9ff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc341a09e0 0x7fcc341a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.220+0000 7fcc3b9ff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34115d20 con 0x7fcc3410e3e0 2024-04-25T16:07:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.220+0000 7fcc3b9ff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34115ba0 con 0x7fcc34110f50 2024-04-25T16:07:38.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.220+0000 7fcc3b9ff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcc34115ea0 con 0x7fcc341a09e0 2024-04-25T16:07:38.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.221+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410e3e0 0x7fcc341a3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:38.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.221+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410e3e0 0x7fcc341a3e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43664/0 (socket says 172.21.15.31:43664) 2024-04-25T16:07:38.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.221+0000 7fcc3979b640 1 -- 172.21.15.31:0/2847543160 learned_addr learned my addr 172.21.15.31:0/2847543160 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.222+0000 7fcc39f9c640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc341a09e0 0x7fcc341a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.222+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc34110f50 0x7fcc341a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.223+0000 7fcc3979b640 1 -- 172.21.15.31:0/2847543160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc341a09e0 msgr2=0x7fcc341a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.223+0000 7fcc3979b640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc341a09e0 0x7fcc341a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.223+0000 7fcc3979b640 1 -- 172.21.15.31:0/2847543160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc34110f50 msgr2=0x7fcc341a4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.223+0000 7fcc3979b640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc34110f50 0x7fcc341a4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.223+0000 7fcc3979b640 1 -- 172.21.15.31:0/2847543160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc341b12a0 con 0x7fcc3410e3e0 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.223+0000 7fcc3979b640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410e3e0 0x7fcc341a3e90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fcc24033040 tx=0x7fcc240048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.224+0000 7fcc2a7fc640 1 -- 172.21.15.31:0/2847543160 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24040030 con 0x7fcc3410e3e0 2024-04-25T16:07:38.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.224+0000 7fcc2a7fc640 1 -- 172.21.15.31:0/2847543160 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc2403cb00 con 0x7fcc3410e3e0 2024-04-25T16:07:38.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.225+0000 7fcc2a7fc640 1 -- 172.21.15.31:0/2847543160 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24031a80 con 0x7fcc3410e3e0 2024-04-25T16:07:38.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.225+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/2847543160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc341b14d0 con 0x7fcc3410e3e0 2024-04-25T16:07:38.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.225+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/2847543160 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc341b1940 con 0x7fcc3410e3e0 2024-04-25T16:07:38.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.231+0000 7fcc42985740 1 Processor -- start 2024-04-25T16:07:38.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.231+0000 7fcc42985740 1 -- start start 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.231+0000 7fcc42985740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cfb390 0x55706109ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.232+0000 7fcc42985740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x55706109cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.232+0000 7fcc42985740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 0x55706109dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.232+0000 7fcc42985740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557061001d80 con 0x557060cd24d0 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.232+0000 7fcc42985740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557060fbb8b0 con 0x557060cfb390 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.232+0000 7fcc42985740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557060ffdd40 con 0x557060cda6b0 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.233+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x55706109cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.233+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x55706109cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43670/0 (socket says 172.21.15.31:43670) 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.233+0000 7fcc3979b640 1 -- 172.21.15.31:0/2590687340 learned_addr learned my addr 172.21.15.31:0/2590687340 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.234+0000 7fcc3979b640 1 -- 172.21.15.31:0/2590687340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 msgr2=0x55706109dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.234+0000 7fcc3979b640 1 --2- 172.21.15.31:0/2590687340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 0x55706109dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.234+0000 7fcc3979b640 1 -- 172.21.15.31:0/2590687340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cfb390 msgr2=0x55706109ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:38.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.234+0000 7fcc3979b640 1 --2- 172.21.15.31:0/2590687340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cfb390 0x55706109ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.234+0000 7fcc3979b640 1 -- 172.21.15.31:0/2590687340 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557060f8ca40 con 0x557060cd24d0 2024-04-25T16:07:38.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.235+0000 7fcc3979b640 1 --2- 172.21.15.31:0/2590687340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x55706109cfe0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009850 tx=0x7fcc24005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18b3fd503ea7c5c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.238+0000 7fcc2a7fc640 1 -- 172.21.15.31:0/2847543160 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcc24043030 con 0x7fcc3410e3e0 2024-04-25T16:07:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.238+0000 7fcc1f7fe640 1 -- 172.21.15.31:0/2590687340 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc241016a0 con 0x557060cd24d0 2024-04-25T16:07:38.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.238+0000 7fcc1f7fe640 1 -- 172.21.15.31:0/2590687340 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc24101ee0 con 0x557060cd24d0 2024-04-25T16:07:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.240+0000 7fcc42985740 1 -- 172.21.15.31:0/2590687340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 msgr2=0x55706109cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:38.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.240+0000 7fcc42985740 1 --2- 172.21.15.31:0/2590687340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x55706109cfe0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009850 tx=0x7fcc24005a40 comp rx=0 tx=0).stop 2024-04-25T16:07:38.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.240+0000 7fcc2a7fc640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcc200afa20 0x7fcc200b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.240+0000 7fcc2a7fc640 1 -- 172.21.15.31:0/2847543160 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcc24056070 con 0x7fcc3410e3e0 2024-04-25T16:07:38.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.241+0000 7fcc39f9c640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcc200afa20 0x7fcc200b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.241+0000 7fcc42985740 1 -- 172.21.15.31:0/2590687340 shutdown_connections 2024-04-25T16:07:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.241+0000 7fcc42985740 1 --2- 172.21.15.31:0/2590687340 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 0x55706109dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.241+0000 7fcc42985740 1 --2- 172.21.15.31:0/2590687340 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x55706109cfe0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.241+0000 7fcc42985740 1 --2- 172.21.15.31:0/2590687340 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cfb390 0x55706109ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.241+0000 7fcc42985740 1 -- 172.21.15.31:0/2590687340 >> 172.21.15.31:0/2590687340 conn(0x557060cef4e0 msgr2=0x557061098270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.242+0000 7fcc39f9c640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcc200afa20 0x7fcc200b1f30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x55706109dab0 tx=0x7fcc2c00b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.243+0000 7fcc42985740 1 -- 172.21.15.31:0/2590687340 shutdown_connections 2024-04-25T16:07:38.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.243+0000 7fcc42985740 1 -- 172.21.15.31:0/2590687340 wait complete. 2024-04-25T16:07:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.243+0000 7fcc42985740 1 -- start start 2024-04-25T16:07:38.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.244+0000 7fcc42985740 1 Processor -- start 2024-04-25T16:07:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.245+0000 7fcc42985740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x5570610a6370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.245+0000 7fcc42985740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cda6b0 0x5570610a68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.245+0000 7fcc42985740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 0x5570610ad810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.245+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x5570610a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.245+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x5570610a6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:43674/0 (socket says 172.21.15.31:43674) 2024-04-25T16:07:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.246+0000 7fcc42985740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557061001d80 con 0x557060cd24d0 2024-04-25T16:07:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.246+0000 7fcc38f9a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cda6b0 0x5570610a68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.246+0000 7fcc38f9a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cda6b0 0x5570610a68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36874/0 (socket says 172.21.15.31:36874) 2024-04-25T16:07:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.246+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1484427399 learned_addr learned my addr 172.21.15.31:0/1484427399 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:38.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.246+0000 7fcc42985740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557060f88d10 con 0x557060cda6b0 2024-04-25T16:07:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.246+0000 7fcc42985740 1 -- 172.21.15.31:0/1484427399 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55706109e540 con 0x557060cfb390 2024-04-25T16:07:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.246+0000 7fcc42985740 10 client.0 _ll_register_callbacks cb 0x557060fb5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:38.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.246+0000 7fcc39f9c640 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 0x5570610ad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.248+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1484427399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 msgr2=0x5570610ad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.248+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 0x5570610ad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.248+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1484427399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 msgr2=0x5570610a6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.248+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x5570610a6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:38.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.248+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1484427399 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557060ddd7d0 con 0x557060cda6b0 2024-04-25T16:07:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.248+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cda6b0 0x5570610a68b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000d1e0 tx=0x7fcc3000bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e4f462453d05b63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.249+0000 7fcc1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:38.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.249+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1484427399 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30013480 con 0x557060cda6b0 2024-04-25T16:07:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.249+0000 7fcc42985740 1 -- 172.21.15.31:0/1484427399 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc24101ee0 con 0x557060cda6b0 2024-04-25T16:07:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.250+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1484427399 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcc30013620 con 0x557060cda6b0 2024-04-25T16:07:38.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.250+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1484427399 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc30013820 con 0x557060cda6b0 2024-04-25T16:07:38.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.250+0000 7fcc42985740 10 client.26513 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:38.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.250+0000 7fcc42985740 1 -- 172.21.15.31:0/1484427399 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557060d133f0 con 0x557060cda6b0 2024-04-25T16:07:38.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.257+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1484427399 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc24101ee0 con 0x557060cda6b0 2024-04-25T16:07:38.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.257+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1484427399 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcc3001cce0 con 0x557060cda6b0 2024-04-25T16:07:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.257+0000 7fcc42985740 10 client.26513 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.257+0000 7fcc42985740 10 client.26513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.257+0000 7fcc42985740 20 client.26513 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.257+0000 7fcc42985740 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557060cb9f60 0x5570610b4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.257+0000 7fcc42985740 4 client.26513 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.257+0000 7fcc42985740 1 -- 172.21.15.31:0/1484427399 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55706100f990 con 0x557060cb9f60 2024-04-25T16:07:38.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.259+0000 7fcc3979b640 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557060cb9f60 0x5570610b4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:38.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.261+0000 7fcc3979b640 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557060cb9f60 0x5570610b4260 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23a3935981b00c3a server_cookie=bc79fccbe7a2457d in_seq=0 out_seq=0 2024-04-25T16:07:38.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:38.262+0000 7fcc1cff9640 10 client.26513 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:39.621 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:39 smithi063 ceph-mon[32413]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 895 MiB/s wr, 2.34k op/s 2024-04-25T16:07:39.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:39 smithi186 ceph-mon[33258]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 895 MiB/s wr, 2.34k op/s 2024-04-25T16:07:39.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:39 smithi031 ceph-mon[29140]: pgmap v541: 97 pgs: 97 active+clean; 10 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 895 MiB/s wr, 2.34k op/s 2024-04-25T16:07:41.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:41 smithi063 ceph-mon[32413]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 799 MiB/s wr, 2.08k op/s 2024-04-25T16:07:41.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:41 smithi186 ceph-mon[33258]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 799 MiB/s wr, 2.08k op/s 2024-04-25T16:07:41.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:41 smithi031 ceph-mon[29140]: pgmap v542: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 799 MiB/s wr, 2.08k op/s 2024-04-25T16:07:43.231 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f726d7fa640 1 -- 172.21.15.31:0/1716126679 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5614a123c5f0 con 0x5614a13d36b0 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f726d7fa640 10 client.26461 handle_command_reply: tid=0 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f7292963740 1 client.26461 shutdown 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f7292963740 1 -- 172.21.15.31:0/1716126679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f924d0 msgr2=0x5614a1362320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f7292963740 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f924d0 0x5614a1362320 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f727c000f80 tx=0x7f727c00cca0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f7292963740 1 -- 172.21.15.31:0/1716126679 shutdown_connections 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f7292963740 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5614a13d36b0 0x5614a13d5ad0 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f7292963740 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5614a0fb31b0 0x5614a1363250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f7292963740 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5614a0f924d0 0x5614a1362320 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.228+0000 7f7292963740 1 --2- 172.21.15.31:0/1716126679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5614a0f71d80 0x5614a1361de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.229+0000 7f7292963740 1 -- 172.21.15.31:0/1716126679 >> 172.21.15.31:0/1716126679 conn(0x5614a0fa7300 msgr2=0x5614a135d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.229+0000 7f726d7fa640 0 client.26461 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.229+0000 7f7292963740 1 -- 172.21.15.31:0/1716126679 shutdown_connections 2024-04-25T16:07:43.232 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.229+0000 7f7292963740 1 -- 172.21.15.31:0/1716126679 wait complete. 2024-04-25T16:07:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.229+0000 7f7292963740 20 client.26461 trim_cache size 0 max 16384 2024-04-25T16:07:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:07:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:07:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:07:43.233 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.235+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1484427399 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55706100f990 con 0x557060cb9f60 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.235+0000 7fcc1cff9640 10 client.26513 handle_command_reply: tid=0 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.235+0000 7fcc42985740 1 client.26513 shutdown 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.235+0000 7fcc42985740 1 -- 172.21.15.31:0/1484427399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cda6b0 msgr2=0x5570610a68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.235+0000 7fcc42985740 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cda6b0 0x5570610a68b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000d1e0 tx=0x7fcc3000bf90 comp rx=0 tx=0).stop 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.236+0000 7fcc42985740 1 -- 172.21.15.31:0/1484427399 shutdown_connections 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.236+0000 7fcc42985740 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557060cb9f60 0x5570610b4260 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.236+0000 7fcc42985740 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 0x5570610ad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.236+0000 7fcc42985740 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cda6b0 0x5570610a68b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.236+0000 7fcc42985740 1 --2- 172.21.15.31:0/1484427399 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cd24d0 0x5570610a6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.236+0000 7fcc42985740 1 -- 172.21.15.31:0/1484427399 >> 172.21.15.31:0/1484427399 conn(0x557060cef4e0 msgr2=0x55706109b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:43.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.236+0000 7fcc1cff9640 0 client.26513 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.237+0000 7f728a84d640 1 -- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72700afc70 msgr2=0x7f72700b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:43.239 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.237+0000 7f728a84d640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72700afc70 0x7f72700b2180 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f727402a950 tx=0x7f7274009280 comp rx=0 tx=0).stop 2024-04-25T16:07:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.237+0000 7f728a84d640 1 -- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72841a59e0 msgr2=0x7f72841a7e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:43.240 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.237+0000 7f728a84d640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72841a59e0 0x7f72841a7e00 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f728000b3f0 tx=0x7f728000b8b0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.239+0000 7fcc42985740 1 -- 172.21.15.31:0/1484427399 shutdown_connections 2024-04-25T16:07:43.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.239+0000 7fcc42985740 1 -- 172.21.15.31:0/1484427399 wait complete. 2024-04-25T16:07:43.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.239+0000 7fcc42985740 20 client.26513 trim_cache size 0 max 16384 2024-04-25T16:07:43.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.240+0000 7f728a84d640 1 -- 172.21.15.31:0/1881056538 shutdown_connections 2024-04-25T16:07:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.240+0000 7f728a84d640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72700afc70 0x7f72700b2180 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.240+0000 7f728a84d640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72841a59e0 0x7f72841a7e00 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.240+0000 7f728a84d640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f728410feb0 0x7f7284117280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.240+0000 7f728a84d640 1 --2- 172.21.15.31:0/1881056538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f728410b190 0x7f7284116d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.240+0000 7f728a84d640 1 -- 172.21.15.31:0/1881056538 >> 172.21.15.31:0/1881056538 conn(0x7f7284070330 msgr2=0x7f7284072750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.244+0000 7f728a84d640 1 -- 172.21.15.31:0/1881056538 shutdown_connections 2024-04-25T16:07:43.247 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:43.245+0000 7f728a84d640 1 -- 172.21.15.31:0/1881056538 wait complete. 2024-04-25T16:07:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.303+0000 7fcc42985740 1 Processor -- start 2024-04-25T16:07:43.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.303+0000 7fcc42985740 1 -- start start 2024-04-25T16:07:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.304+0000 7fcc42985740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x55706109b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:43.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.304+0000 7fcc42985740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cda6b0 0x55706109b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.305+0000 7fcc42985740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 0x5570610ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.305+0000 7fcc42985740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341ba450 con 0x557060cda6b0 2024-04-25T16:07:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.305+0000 7fcc42985740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341bbf90 con 0x557060cd24d0 2024-04-25T16:07:43.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.305+0000 7fcc42985740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341bab20 con 0x557060cfb390 2024-04-25T16:07:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.306+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x55706109b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.306+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x55706109b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36876/0 (socket says 172.21.15.31:36876) 2024-04-25T16:07:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.306+0000 7fcc3979b640 1 -- 172.21.15.31:0/1733648280 learned_addr learned my addr 172.21.15.31:0/1733648280 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:43.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.306+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1733648280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cda6b0 0x55706109b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.306+0000 7fcc39f9c640 1 --2- 172.21.15.31:0/1733648280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 0x5570610ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:43.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.307+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1733648280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 msgr2=0x5570610ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:43.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.307+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1733648280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 0x5570610ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.307+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1733648280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 msgr2=0x55706109b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:43.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.307+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1733648280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x55706109b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.307+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1733648280 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557060cd30f0 con 0x557060cda6b0 2024-04-25T16:07:43.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.308+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1733648280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cda6b0 0x55706109b5b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fcc30001eb0 tx=0x7fcc30006d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df38fb4f2f15c8f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:43.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.315+0000 7fcc1effd640 1 -- 172.21.15.31:0/1733648280 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc30012770 con 0x557060cda6b0 2024-04-25T16:07:43.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.316+0000 7fcc1effd640 1 -- 172.21.15.31:0/1733648280 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc30018070 con 0x557060cda6b0 2024-04-25T16:07:43.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.316+0000 7fcc1effd640 1 -- 172.21.15.31:0/1733648280 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc300139c0 con 0x557060cda6b0 2024-04-25T16:07:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.318+0000 7fcc42985740 1 -- 172.21.15.31:0/1733648280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cda6b0 msgr2=0x55706109b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:43.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.318+0000 7fcc42985740 1 --2- 172.21.15.31:0/1733648280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cda6b0 0x55706109b5b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fcc30001eb0 tx=0x7fcc30006d60 comp rx=0 tx=0).stop 2024-04-25T16:07:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.319+0000 7fcc42985740 1 -- 172.21.15.31:0/1733648280 shutdown_connections 2024-04-25T16:07:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.319+0000 7fcc42985740 1 --2- 172.21.15.31:0/1733648280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cfb390 0x5570610ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.319+0000 7fcc42985740 1 --2- 172.21.15.31:0/1733648280 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cda6b0 0x55706109b5b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.319+0000 7fcc42985740 1 --2- 172.21.15.31:0/1733648280 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x55706109b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.319+0000 7fcc42985740 1 -- 172.21.15.31:0/1733648280 >> 172.21.15.31:0/1733648280 conn(0x557060cef4e0 msgr2=0x5570610a4990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.323+0000 7fcc42985740 1 -- 172.21.15.31:0/1733648280 shutdown_connections 2024-04-25T16:07:43.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.323+0000 7fcc42985740 1 -- 172.21.15.31:0/1733648280 wait complete. 2024-04-25T16:07:43.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.324+0000 7fcc42985740 1 -- start start 2024-04-25T16:07:43.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.324+0000 7fcc42985740 1 Processor -- start 2024-04-25T16:07:43.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.328+0000 7fcc42985740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x5570610addb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.328+0000 7fcc42985740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 0x5570610aa930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.332+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x5570610addb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.332+0000 7fcc3979b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x5570610addb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36882/0 (socket says 172.21.15.31:36882) 2024-04-25T16:07:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.332+0000 7fcc3979b640 1 -- 172.21.15.31:0/1965912227 learned_addr learned my addr 172.21.15.31:0/1965912227 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.332+0000 7fcc42985740 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cfb390 0x5570610aae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.332+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 0x5570610aa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.333+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341bb8c0 con 0x557060cfb390 2024-04-25T16:07:43.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.333+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341bb1f0 con 0x557060cd24d0 2024-04-25T16:07:43.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.333+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcc341b8430 con 0x557060cda6b0 2024-04-25T16:07:43.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.333+0000 7fcc42985740 10 client.0 _ll_register_callbacks cb 0x7fcc3000b1a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:43.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.334+0000 7fcc39f9c640 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cfb390 0x5570610aae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.335+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1965912227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 msgr2=0x5570610addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.335+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x5570610addb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.335+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1965912227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cfb390 msgr2=0x5570610aae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.335+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cfb390 0x5570610aae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.335+0000 7fcc38f9a640 1 -- 172.21.15.31:0/1965912227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557060fc24b0 con 0x557060cda6b0 2024-04-25T16:07:43.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.335+0000 7fcc38f9a640 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 0x5570610aa930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000da70 tx=0x7fcc30012ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd65ba2ecdb0ae0d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:43.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.336+0000 7fcc1cff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:43.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.336+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1965912227 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc300139c0 con 0x557060cda6b0 2024-04-25T16:07:43.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.336+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1965912227 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcc3002c040 con 0x557060cda6b0 2024-04-25T16:07:43.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.337+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcc30018070 con 0x557060cda6b0 2024-04-25T16:07:43.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.342+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1965912227 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcc30013b60 con 0x557060cda6b0 2024-04-25T16:07:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.342+0000 7fcc42985740 10 client.26476 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:43.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.342+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55706111b520 con 0x557060cda6b0 2024-04-25T16:07:43.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.347+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1965912227 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc30013dc0 con 0x557060cda6b0 2024-04-25T16:07:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.349+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1965912227 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcc3003c430 con 0x557060cda6b0 2024-04-25T16:07:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.349+0000 7fcc42985740 10 client.26476 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:43.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.349+0000 7fcc42985740 10 client.26476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.349+0000 7fcc42985740 20 client.26476 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.349+0000 7fcc42985740 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557060c8f3a0 0x55706111dde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.349+0000 7fcc42985740 4 client.26476 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:07:43.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.350+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fcc30018070 con 0x557060c8f3a0 2024-04-25T16:07:43.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.353+0000 7fcc3979b640 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557060c8f3a0 0x55706111dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:43.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.354+0000 7fcc3979b640 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557060c8f3a0 0x55706111dde0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=928f22ba1c2799a4 server_cookie=5b8a8f5e5d6c3150 in_seq=0 out_seq=0 2024-04-25T16:07:43.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.354+0000 7fcc1cff9640 10 client.26476 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:43.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.356+0000 7fcc1cff9640 1 -- 172.21.15.31:0/1965912227 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fcc30018070 con 0x557060c8f3a0 2024-04-25T16:07:43.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.357+0000 7fcc1cff9640 10 client.26476 handle_command_reply: tid=0 2024-04-25T16:07:43.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.357+0000 7fcc42985740 1 client.26476 shutdown 2024-04-25T16:07:43.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.366+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 msgr2=0x5570610aa930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:43.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.366+0000 7fcc42985740 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 0x5570610aa930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fcc3000da70 tx=0x7fcc30012ca0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.366+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 shutdown_connections 2024-04-25T16:07:43.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.366+0000 7fcc42985740 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557060c8f3a0 0x55706111dde0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.366+0000 7fcc42985740 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557060cfb390 0x5570610aae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.366+0000 7fcc42985740 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557060cda6b0 0x5570610aa930 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.366+0000 7fcc42985740 1 --2- 172.21.15.31:0/1965912227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557060cd24d0 0x5570610addb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.366+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 >> 172.21.15.31:0/1965912227 conn(0x557060cef4e0 msgr2=0x557061097c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:43.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.367+0000 7fcc1cff9640 0 client.26476 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:43.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.367+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 shutdown_connections 2024-04-25T16:07:43.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.368+0000 7fcc42985740 1 -- 172.21.15.31:0/1965912227 wait complete. 2024-04-25T16:07:43.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.375+0000 7fcc42985740 20 client.26476 trim_cache size 0 max 16384 2024-04-25T16:07:43.385 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:07:43.385 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:07:43.385 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:07:43.385 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:07:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.384+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcc200afa20 msgr2=0x7fcc200b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.385+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcc200afa20 0x7fcc200b1f30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x55706109dab0 tx=0x7fcc2c00b2c0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.385+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410e3e0 msgr2=0x7fcc341a3e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:43.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.385+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410e3e0 0x7fcc341a3e90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fcc24033040 tx=0x7fcc240048f0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.393+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/2847543160 shutdown_connections 2024-04-25T16:07:43.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.393+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcc200afa20 0x7fcc200b1f30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.393+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc341a09e0 0x7fcc341a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.393+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcc34110f50 0x7fcc341a4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.393+0000 7fcc3b9ff640 1 --2- 172.21.15.31:0/2847543160 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcc3410e3e0 0x7fcc341a3e90 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:43.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.393+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/2847543160 >> 172.21.15.31:0/2847543160 conn(0x7fcc340704e0 msgr2=0x7fcc3410da70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:43.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.401+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/2847543160 shutdown_connections 2024-04-25T16:07:43.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:43.401+0000 7fcc3b9ff640 1 -- 172.21.15.31:0/2847543160 wait complete. 2024-04-25T16:07:43.512 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:07:43.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:07:43.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:43 smithi063 ceph-mon[32413]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 2.03k op/s 2024-04-25T16:07:43.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:43 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:07:43.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:43 smithi186 ceph-mon[33258]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 2.03k op/s 2024-04-25T16:07:43.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:43 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:07:43.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:43 smithi031 ceph-mon[29140]: pgmap v543: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 748 MiB/s wr, 2.03k op/s 2024-04-25T16:07:43.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:43 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:07:44.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.014+0000 7f36b8e8c640 1 -- 172.21.15.31:0/2621711866 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36b00318a0 con 0x7f36b410b310 2024-04-25T16:07:44.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.014+0000 7f36bc0f2640 1 -- 172.21.15.31:0/2621711866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410b310 msgr2=0x7f36b410b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:44.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.014+0000 7f36bc0f2640 1 --2- 172.21.15.31:0/2621711866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410b310 0x7f36b410b720 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f36b0009900 tx=0x7f36b002a420 comp rx=0 tx=0).stop 2024-04-25T16:07:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.018+0000 7f36bc0f2640 1 -- 172.21.15.31:0/2621711866 shutdown_connections 2024-04-25T16:07:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.018+0000 7f36bc0f2640 1 --2- 172.21.15.31:0/2621711866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b4110030 0x7f36b4110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.018+0000 7f36bc0f2640 1 --2- 172.21.15.31:0/2621711866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410b310 0x7f36b410b720 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.018+0000 7f36bc0f2640 1 --2- 172.21.15.31:0/2621711866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36b410a420 0x7f36b410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.018+0000 7f36bc0f2640 1 -- 172.21.15.31:0/2621711866 >> 172.21.15.31:0/2621711866 conn(0x7f36b40704e0 msgr2=0x7f36b4072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:44.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.023+0000 7f36bc0f2640 1 -- 172.21.15.31:0/2621711866 shutdown_connections 2024-04-25T16:07:44.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.024+0000 7f36bc0f2640 1 -- 172.21.15.31:0/2621711866 wait complete. 2024-04-25T16:07:44.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.025+0000 7f36bc0f2640 1 Processor -- start 2024-04-25T16:07:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.026+0000 7f36bc0f2640 1 -- start start 2024-04-25T16:07:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.026+0000 7f36bc0f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410a420 0x7f36b41a9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.026+0000 7f36bc0f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b410b310 0x7f36b41a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.026+0000 7f36bc0f2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36b4110030 0x7f36b41b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.026+0000 7f36bc0f2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f36b411e560 con 0x7f36b410a420 2024-04-25T16:07:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.026+0000 7f36bc0f2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f36b411e3e0 con 0x7f36b4110030 2024-04-25T16:07:44.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.026+0000 7f36bc0f2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36b411e6e0 con 0x7f36b410b310 2024-04-25T16:07:44.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.028+0000 7f36b968d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b410b310 0x7f36b41a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:44.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.029+0000 7f36ba68f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36b4110030 0x7f36b41b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:44.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.030+0000 7f36b9e8e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410a420 0x7f36b41a9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:44.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.030+0000 7f36b9e8e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410a420 0x7f36b41a9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60340/0 (socket says 172.21.15.31:60340) 2024-04-25T16:07:44.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.030+0000 7f36b9e8e640 1 -- 172.21.15.31:0/1884184864 learned_addr learned my addr 172.21.15.31:0/1884184864 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.038+0000 7f36b9e8e640 1 -- 172.21.15.31:0/1884184864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b410b310 msgr2=0x7f36b41a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.038+0000 7f36b9e8e640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b410b310 0x7f36b41a9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.038+0000 7f36b9e8e640 1 -- 172.21.15.31:0/1884184864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36b4110030 msgr2=0x7f36b41b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.038+0000 7f36b9e8e640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36b4110030 0x7f36b41b0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.038+0000 7f36b9e8e640 1 -- 172.21.15.31:0/1884184864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36b41b0c80 con 0x7f36b410a420 2024-04-25T16:07:44.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.038+0000 7f36ba68f640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36b4110030 0x7f36b41b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:07:44.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.039+0000 7f36b9e8e640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410a420 0x7f36b41a9140 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f36a400c970 tx=0x7f36a400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:44.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.040+0000 7f36aaffd640 1 -- 172.21.15.31:0/1884184864 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a4007bb0 con 0x7f36b410a420 2024-04-25T16:07:44.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.040+0000 7f36aaffd640 1 -- 172.21.15.31:0/1884184864 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36a4007d50 con 0x7f36b410a420 2024-04-25T16:07:44.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.040+0000 7f36bc0f2640 1 -- 172.21.15.31:0/1884184864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36b41b0f70 con 0x7f36b410a420 2024-04-25T16:07:44.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.040+0000 7f36aaffd640 1 -- 172.21.15.31:0/1884184864 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36a400f450 con 0x7f36b410a420 2024-04-25T16:07:44.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.040+0000 7f36bc0f2640 1 -- 172.21.15.31:0/1884184864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36b41b13a0 con 0x7f36b410a420 2024-04-25T16:07:44.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.053+0000 7f36aaffd640 1 -- 172.21.15.31:0/1884184864 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f36a4004f20 con 0x7f36b410a420 2024-04-25T16:07:44.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.054+0000 7f36aaffd640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f36a00afd00 0x7f36a00b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:44.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.055+0000 7f36aaffd640 1 -- 172.21.15.31:0/1884184864 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f36a40deb10 con 0x7f36b410a420 2024-04-25T16:07:44.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.056+0000 7f36b968d640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f36a00afd00 0x7f36a00b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:44.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.056+0000 7f36bc0f2640 1 -- 172.21.15.31:0/1884184864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36b4069d40 con 0x7f36b410a420 2024-04-25T16:07:44.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.063+0000 7f36b968d640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f36a00afd00 0x7f36a00b2210 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f36b0031500 tx=0x7f36b0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:44.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.076+0000 7f36aaffd640 1 -- 172.21.15.31:0/1884184864 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f36a4014030 con 0x7f36b410a420 2024-04-25T16:07:44.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.862+0000 7f36bc0f2640 1 -- 172.21.15.31:0/1884184864 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f36b402de90 con 0x7f36b410a420 2024-04-25T16:07:44.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.869+0000 7f36aaffd640 1 -- 172.21.15.31:0/1884184864 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7f36a40a7050 con 0x7f36b410a420 2024-04-25T16:07:44.889 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:07:44.889 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:07:44.889 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:07:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.890+0000 7f36a8ff9640 1 -- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f36a00afd00 msgr2=0x7f36a00b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.890+0000 7f36a8ff9640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f36a00afd00 0x7f36a00b2210 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f36b0031500 tx=0x7f36b0000bc0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.890+0000 7f36a8ff9640 1 -- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410a420 msgr2=0x7f36b41a9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:44.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.890+0000 7f36a8ff9640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410a420 0x7f36b41a9140 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f36a400c970 tx=0x7f36a400ce30 comp rx=0 tx=0).stop 2024-04-25T16:07:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.891+0000 7f36a8ff9640 1 -- 172.21.15.31:0/1884184864 shutdown_connections 2024-04-25T16:07:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.891+0000 7f36a8ff9640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f36a00afd00 0x7f36a00b2210 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.891+0000 7f36a8ff9640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f36b4110030 0x7f36b41b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.891+0000 7f36a8ff9640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36b410b310 0x7f36b41a9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.891+0000 7f36a8ff9640 1 --2- 172.21.15.31:0/1884184864 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f36b410a420 0x7f36b41a9140 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:44.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.891+0000 7f36a8ff9640 1 -- 172.21.15.31:0/1884184864 >> 172.21.15.31:0/1884184864 conn(0x7f36b40704e0 msgr2=0x7f36b410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.895+0000 7f36a8ff9640 1 -- 172.21.15.31:0/1884184864 shutdown_connections 2024-04-25T16:07:44.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:44.896+0000 7f36a8ff9640 1 -- 172.21.15.31:0/1884184864 wait complete. 2024-04-25T16:07:44.994 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:07:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.527+0000 7f66af59e640 1 -- 172.21.15.31:0/2069177134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b01102c0 msgr2=0x7f66b01126e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.527+0000 7f66af59e640 1 --2- 172.21.15.31:0/2069177134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b01102c0 0x7f66b01126e0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f66a8009be0 tx=0x7f66a802a710 comp rx=0 tx=0).stop 2024-04-25T16:07:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.527+0000 7f66af59e640 1 -- 172.21.15.31:0/2069177134 shutdown_connections 2024-04-25T16:07:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.527+0000 7f66af59e640 1 --2- 172.21.15.31:0/2069177134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b01102c0 0x7f66b01126e0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.527+0000 7f66af59e640 1 --2- 172.21.15.31:0/2069177134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66b010a790 0x7f66b010b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.527+0000 7f66af59e640 1 --2- 172.21.15.31:0/2069177134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66b010a2d0 0x7f66b010aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.527+0000 7f66af59e640 1 -- 172.21.15.31:0/2069177134 >> 172.21.15.31:0/2069177134 conn(0x7f66b0070320 msgr2=0x7f66b00727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:45.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.529+0000 7f66af59e640 1 -- 172.21.15.31:0/2069177134 shutdown_connections 2024-04-25T16:07:45.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.533+0000 7f66af59e640 1 -- 172.21.15.31:0/2069177134 wait complete. 2024-04-25T16:07:45.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.534+0000 7f66af59e640 1 Processor -- start 2024-04-25T16:07:45.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.535+0000 7f66af59e640 1 -- start start 2024-04-25T16:07:45.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.535+0000 7f66af59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b010a2d0 0x7f66b01a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.536+0000 7f66af59e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66b01102c0 0x7f66b01a05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.536+0000 7f66ae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b010a2d0 0x7f66b01a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.536+0000 7f66ae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b010a2d0 0x7f66b01a0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60352/0 (socket says 172.21.15.31:60352) 2024-04-25T16:07:45.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.550+0000 7f66add9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66b01102c0 0x7f66b01a05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.550+0000 7f66ae59c640 1 -- 172.21.15.31:0/2860610174 learned_addr learned my addr 172.21.15.31:0/2860610174 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:45.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.550+0000 7f66af59e640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66b01a5520 0x7f66b01a7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.550+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f66b0115ac0 con 0x7f66b010a2d0 2024-04-25T16:07:45.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.550+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f66b0115940 con 0x7f66b01102c0 2024-04-25T16:07:45.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.550+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f66b0115c40 con 0x7f66b01a5520 2024-04-25T16:07:45.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.555+0000 7f66aed9d640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66b01a5520 0x7f66b01a7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.561+0000 7f66ae59c640 1 -- 172.21.15.31:0/2860610174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66b01a5520 msgr2=0x7f66b01a7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:45.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.561+0000 7f66ae59c640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66b01a5520 0x7f66b01a7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.561+0000 7f66ae59c640 1 -- 172.21.15.31:0/2860610174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66b01102c0 msgr2=0x7f66b01a05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:45.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.561+0000 7f66ae59c640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66b01102c0 0x7f66b01a05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.561+0000 7f66ae59c640 1 -- 172.21.15.31:0/2860610174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b01a7fb0 con 0x7f66b010a2d0 2024-04-25T16:07:45.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.561+0000 7f66add9b640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66b01102c0 0x7f66b01a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:07:45.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.570+0000 7f66ae59c640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b010a2d0 0x7f66b01a0060 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f66a000f7d0 tx=0x7f66a000fc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.575+0000 7f669f7fe640 1 -- 172.21.15.31:0/2860610174 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a000d6a0 con 0x7f66b010a2d0 2024-04-25T16:07:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.575+0000 7f669f7fe640 1 -- 172.21.15.31:0/2860610174 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a001a070 con 0x7f66b010a2d0 2024-04-25T16:07:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.575+0000 7f669f7fe640 1 -- 172.21.15.31:0/2860610174 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a000a690 con 0x7f66b010a2d0 2024-04-25T16:07:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.575+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b01a8240 con 0x7f66b010a2d0 2024-04-25T16:07:45.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.575+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b01a8680 con 0x7f66b010a2d0 2024-04-25T16:07:45.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.583+0000 7f669f7fe640 1 -- 172.21.15.31:0/2860610174 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f66a0013050 con 0x7f66b010a2d0 2024-04-25T16:07:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.586+0000 7f669f7fe640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f66980afc20 0x7f66980b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.587+0000 7f669f7fe640 1 -- 172.21.15.31:0/2860610174 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f66a00e0690 con 0x7f66b010a2d0 2024-04-25T16:07:45.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.589+0000 7f66bb9ac740 1 Processor -- start 2024-04-25T16:07:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.590+0000 7f66bb9ac740 1 -- start start 2024-04-25T16:07:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.590+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db8135390 0x564db84d7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.590+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 0x564db84d7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.590+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 0x564db84d8b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.591+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564db843bd80 con 0x564db81146b0 2024-04-25T16:07:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.591+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564db83f5a20 con 0x564db810c4d0 2024-04-25T16:07:45.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.591+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564db8437d40 con 0x564db8135390 2024-04-25T16:07:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.593+0000 7f66add9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 0x564db84d7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.593+0000 7f66add9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 0x564db84d7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48120/0 (socket says 172.21.15.31:48120) 2024-04-25T16:07:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.593+0000 7f66add9b640 1 -- 172.21.15.31:0/3753859471 learned_addr learned my addr 172.21.15.31:0/3753859471 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:45.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.594+0000 7f66ae59c640 1 --2- 172.21.15.31:0/3753859471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db8135390 0x564db84d7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.596+0000 7f66ae59c640 1 -- 172.21.15.31:0/3753859471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 msgr2=0x564db84d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.596+0000 7f66ae59c640 1 --2- 172.21.15.31:0/3753859471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 0x564db84d7eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.596+0000 7f66ae59c640 1 -- 172.21.15.31:0/3753859471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 msgr2=0x564db84d8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.596+0000 7f66ae59c640 1 --2- 172.21.15.31:0/3753859471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 0x564db84d8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.596+0000 7f66ae59c640 1 -- 172.21.15.31:0/3753859471 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db8400c30 con 0x564db8135390 2024-04-25T16:07:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.597+0000 7f66add9b640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f66980afc20 0x7f66980b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.601+0000 7f66ae59c640 1 --2- 172.21.15.31:0/3753859471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db8135390 0x564db84d7910 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f66a00a8ab0 tx=0x7f66a00a8a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=220c4d952342524b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.605+0000 7f669cff9640 1 -- 172.21.15.31:0/3753859471 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00a8630 con 0x564db8135390 2024-04-25T16:07:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.606+0000 7f669cff9640 1 -- 172.21.15.31:0/3753859471 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a00a9e50 con 0x564db8135390 2024-04-25T16:07:45.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.607+0000 7f66bb9ac740 1 -- 172.21.15.31:0/3753859471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db8135390 msgr2=0x564db84d7910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:45.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.607+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/3753859471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db8135390 0x564db84d7910 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f66a00a8ab0 tx=0x7f66a00a8a10 comp rx=0 tx=0).stop 2024-04-25T16:07:45.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.608+0000 7f66bb9ac740 1 -- 172.21.15.31:0/3753859471 shutdown_connections 2024-04-25T16:07:45.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.608+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/3753859471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 0x564db84d8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.608+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/3753859471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 0x564db84d7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.608+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/3753859471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db8135390 0x564db84d7910 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.608+0000 7f66bb9ac740 1 -- 172.21.15.31:0/3753859471 >> 172.21.15.31:0/3753859471 conn(0x564db81294e0 msgr2=0x564db84d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:45.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.608+0000 7f66bb9ac740 1 -- 172.21.15.31:0/3753859471 shutdown_connections 2024-04-25T16:07:45.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.608+0000 7f66bb9ac740 1 -- 172.21.15.31:0/3753859471 wait complete. 2024-04-25T16:07:45.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.609+0000 7f66bb9ac740 1 -- start start 2024-04-25T16:07:45.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.609+0000 7f66bb9ac740 1 Processor -- start 2024-04-25T16:07:45.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.609+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 0x564db84e1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.610+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 0x564db84e1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.610+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db80f3f60 0x564db84e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.610+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564db84d9640 con 0x564db81146b0 2024-04-25T16:07:45.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.610+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564db83c2dc0 con 0x564db810c4d0 2024-04-25T16:07:45.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.610+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564db84d97c0 con 0x564db80f3f60 2024-04-25T16:07:45.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.610+0000 7f66bb9ac740 10 client.0 _ll_register_callbacks cb 0x564db83de600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:45.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.610+0000 7f66add9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 0x564db84e1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.610+0000 7f66add9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 0x564db84e1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60376/0 (socket says 172.21.15.31:60376) 2024-04-25T16:07:45.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.610+0000 7f66add9b640 1 -- 172.21.15.31:0/1558540893 learned_addr learned my addr 172.21.15.31:0/1558540893 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.612+0000 7f66add9b640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f66980afc20 0x7f66980b2130 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f66a40058c0 tx=0x7f66a4004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.612+0000 7f66add9b640 1 -- 172.21.15.31:0/1558540893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db80f3f60 msgr2=0x564db84e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.612+0000 7f66add9b640 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db80f3f60 0x564db84e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.612+0000 7f66add9b640 1 -- 172.21.15.31:0/1558540893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 msgr2=0x564db84e1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.612+0000 7f66add9b640 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 0x564db84e1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.612+0000 7f66add9b640 1 -- 172.21.15.31:0/1558540893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db814d3f0 con 0x564db81146b0 2024-04-25T16:07:45.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.613+0000 7f66add9b640 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 0x564db84e1ad0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f66a4011ac0 tx=0x7f66a400d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84553266d96d5869 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:45.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.613+0000 7f6695ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:07:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.615+0000 7f6695ffb640 1 -- 172.21.15.31:0/1558540893 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a4024070 con 0x564db81146b0 2024-04-25T16:07:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.615+0000 7f66bb9ac740 1 -- 172.21.15.31:0/1558540893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564db8449990 con 0x564db81146b0 2024-04-25T16:07:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.615+0000 7f6695ffb640 1 -- 172.21.15.31:0/1558540893 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f66a4020420 con 0x564db81146b0 2024-04-25T16:07:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.616+0000 7f6695ffb640 1 -- 172.21.15.31:0/1558540893 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f66a4030050 con 0x564db81146b0 2024-04-25T16:07:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.616+0000 7f66bb9ac740 10 client.16902 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:45.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.616+0000 7f66bb9ac740 1 -- 172.21.15.31:0/1558540893 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564db82177d0 con 0x564db81146b0 2024-04-25T16:07:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:45 smithi063 ceph-mon[32413]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 976 MiB/s wr, 2.75k op/s 2024-04-25T16:07:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:45 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1884184864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:07:45.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.617+0000 7f6695ffb640 1 -- 172.21.15.31:0/1558540893 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564db8449990 con 0x564db81146b0 2024-04-25T16:07:45.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.619+0000 7f6695ffb640 1 -- 172.21.15.31:0/1558540893 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f66a400f6a0 con 0x564db81146b0 2024-04-25T16:07:45.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.619+0000 7f66bb9ac740 10 client.16902 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:45.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.619+0000 7f66bb9ac740 10 client.16902 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:07:45.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.619+0000 7f66bb9ac740 20 client.16902 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:45.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.620+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564db80c93a0 0x564db8500a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.620+0000 7f66bb9ac740 4 client.16902 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:45.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.620+0000 7f66bb9ac740 1 -- 172.21.15.31:0/1558540893 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564db8443800 con 0x564db80c93a0 2024-04-25T16:07:45.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.625+0000 7f66ae59c640 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564db80c93a0 0x564db8500a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.626+0000 7f66ae59c640 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564db80c93a0 0x564db8500a00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7082a1e45e9c7534 server_cookie=6b430264cb99d4d1 in_seq=0 out_seq=0 2024-04-25T16:07:45.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.626+0000 7f6695ffb640 10 client.16902 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:07:45.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.628+0000 7f6695ffb640 1 -- 172.21.15.31:0/1558540893 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564db8443800 con 0x564db80c93a0 2024-04-25T16:07:45.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.628+0000 7f6695ffb640 10 client.16902 handle_command_reply: tid=0 2024-04-25T16:07:45.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.628+0000 7f66bb9ac740 1 client.16902 shutdown 2024-04-25T16:07:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.637+0000 7f66bb9ac740 1 -- 172.21.15.31:0/1558540893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 msgr2=0x564db84e1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:45.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.637+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 0x564db84e1ad0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f66a4011ac0 tx=0x7f66a400d5c0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.638+0000 7f66bb9ac740 1 -- 172.21.15.31:0/1558540893 shutdown_connections 2024-04-25T16:07:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.638+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564db80c93a0 0x564db8500a00 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.638+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db80f3f60 0x564db84e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.638+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db81146b0 0x564db84e1ad0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.638+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/1558540893 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db810c4d0 0x564db84e1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.638+0000 7f66bb9ac740 1 -- 172.21.15.31:0/1558540893 >> 172.21.15.31:0/1558540893 conn(0x564db81294e0 msgr2=0x564db84d23f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:45.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.640+0000 7f6695ffb640 0 client.16902 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:07:45.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.640+0000 7f66bb9ac740 1 -- 172.21.15.31:0/1558540893 shutdown_connections 2024-04-25T16:07:45.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.640+0000 7f66bb9ac740 1 -- 172.21.15.31:0/1558540893 wait complete. 2024-04-25T16:07:45.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.641+0000 7f66bb9ac740 20 client.16902 trim_cache size 0 max 16384 2024-04-25T16:07:45.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:45 smithi186 ceph-mon[33258]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 976 MiB/s wr, 2.75k op/s 2024-04-25T16:07:45.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:45 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1884184864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:07:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.684+0000 7f66bb9ac740 1 Processor -- start 2024-04-25T16:07:45.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.684+0000 7f66bb9ac740 1 -- start start 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db80f3f60 0x564db84dfbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db810c4d0 0x564db84e89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db81146b0 0x564db84e00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f669000e400 con 0x564db80f3f60 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66ae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db80f3f60 0x564db84dfbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66ae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db80f3f60 0x564db84dfbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60378/0 (socket says 172.21.15.31:60378) 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f669000ff40 con 0x564db81146b0 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66ae59c640 1 -- 172.21.15.31:0/162710274 learned_addr learned my addr 172.21.15.31:0/162710274 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66bb9ac740 1 -- 172.21.15.31:0/162710274 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f669000ead0 con 0x564db810c4d0 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66aed9d640 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db81146b0 0x564db84e00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.685+0000 7f66add9b640 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db810c4d0 0x564db84e89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.686+0000 7f66aed9d640 1 -- 172.21.15.31:0/162710274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db810c4d0 msgr2=0x564db84e89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:45.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.686+0000 7f66aed9d640 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db810c4d0 0x564db84e89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.686+0000 7f66aed9d640 1 -- 172.21.15.31:0/162710274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db80f3f60 msgr2=0x564db84dfbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:45.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.686+0000 7f66aed9d640 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db80f3f60 0x564db84dfbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.686+0000 7f66aed9d640 1 -- 172.21.15.31:0/162710274 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564db84c4ba0 con 0x564db81146b0 2024-04-25T16:07:45.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.686+0000 7f66add9b640 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db810c4d0 0x564db84e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:07:45.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.686+0000 7f66aed9d640 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db81146b0 0x564db84e00f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f66a802ad00 tx=0x7f66a80303a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de6f92ff647cb102 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:45.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.688+0000 7f6697fff640 1 -- 172.21.15.31:0/162710274 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a8042070 con 0x564db81146b0 2024-04-25T16:07:45.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.688+0000 7f6697fff640 1 -- 172.21.15.31:0/162710274 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a8031440 con 0x564db81146b0 2024-04-25T16:07:45.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.690+0000 7f66bb9ac740 1 -- 172.21.15.31:0/162710274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db81146b0 msgr2=0x564db84e00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:45.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.690+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db81146b0 0x564db84e00f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f66a802ad00 tx=0x7f66a80303a0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:45 smithi031 ceph-mon[29140]: pgmap v544: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 976 MiB/s wr, 2.75k op/s 2024-04-25T16:07:45.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:45 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1884184864' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:07:45.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.699+0000 7f66bb9ac740 1 -- 172.21.15.31:0/162710274 shutdown_connections 2024-04-25T16:07:45.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.699+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db81146b0 0x564db84e00f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.699+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db810c4d0 0x564db84e89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.699+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/162710274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db80f3f60 0x564db84dfbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.699+0000 7f66bb9ac740 1 -- 172.21.15.31:0/162710274 >> 172.21.15.31:0/162710274 conn(0x564db81294e0 msgr2=0x564db84dc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:45.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.699+0000 7f66bb9ac740 1 -- 172.21.15.31:0/162710274 shutdown_connections 2024-04-25T16:07:45.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.703+0000 7f66bb9ac740 1 -- 172.21.15.31:0/162710274 wait complete. 2024-04-25T16:07:45.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.703+0000 7f66bb9ac740 1 -- start start 2024-04-25T16:07:45.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.703+0000 7f66bb9ac740 1 Processor -- start 2024-04-25T16:07:45.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.704+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db80f3f60 0x564db84e8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.704+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db810c4d0 0x564db84e89d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.704+0000 7f66bb9ac740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db81146b0 0x564db84e50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.704+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f669000f870 con 0x564db810c4d0 2024-04-25T16:07:45.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.705+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f669000f1a0 con 0x564db80f3f60 2024-04-25T16:07:45.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.705+0000 7f66bb9ac740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f669000c490 con 0x564db81146b0 2024-04-25T16:07:45.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.705+0000 7f66bb9ac740 10 client.0 _ll_register_callbacks cb 0x7f66a400dd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:45.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.705+0000 7f66ae59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db80f3f60 0x564db84e8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.706+0000 7f66aed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db81146b0 0x564db84e50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.706+0000 7f66aed9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db81146b0 0x564db84e50b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53818/0 (socket says 172.21.15.31:53818) 2024-04-25T16:07:45.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.706+0000 7f66ae59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db80f3f60 0x564db84e8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48152/0 (socket says 172.21.15.31:48152) 2024-04-25T16:07:45.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.706+0000 7f66aed9d640 1 -- 172.21.15.31:0/4254065649 learned_addr learned my addr 172.21.15.31:0/4254065649 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:45.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.706+0000 7f66ae59c640 1 -- 172.21.15.31:0/4254065649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db81146b0 msgr2=0x564db84e50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:45.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.706+0000 7f66ae59c640 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db81146b0 0x564db84e50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.707+0000 7f66ae59c640 1 -- 172.21.15.31:0/4254065649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db810c4d0 msgr2=0x564db84e89d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:07:45.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.707+0000 7f66ae59c640 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db810c4d0 0x564db84e89d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.707+0000 7f66ae59c640 1 -- 172.21.15.31:0/4254065649 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564db84e5810 con 0x564db80f3f60 2024-04-25T16:07:45.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.707+0000 7f66ae59c640 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db80f3f60 0x564db84e8490 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f66a000d490 tx=0x7f66a00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bf9cc4e49c24d13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:45.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.708+0000 7f6695ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:45.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.708+0000 7f6695ffb640 1 -- 172.21.15.31:0/4254065649 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a008f060 con 0x564db80f3f60 2024-04-25T16:07:45.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.708+0000 7f6695ffb640 1 -- 172.21.15.31:0/4254065649 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f66a00a5110 con 0x564db80f3f60 2024-04-25T16:07:45.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.709+0000 7f66bb9ac740 1 -- 172.21.15.31:0/4254065649 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66a8031440 con 0x564db80f3f60 2024-04-25T16:07:45.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.709+0000 7f6695ffb640 1 -- 172.21.15.31:0/4254065649 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f66a00c2050 con 0x564db80f3f60 2024-04-25T16:07:45.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.709+0000 7f66bb9ac740 10 client.26534 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:45.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.709+0000 7f66bb9ac740 1 -- 172.21.15.31:0/4254065649 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564db8567a70 con 0x564db80f3f60 2024-04-25T16:07:45.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.710+0000 7f6695ffb640 1 -- 172.21.15.31:0/4254065649 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66a8031440 con 0x564db80f3f60 2024-04-25T16:07:45.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.710+0000 7f6695ffb640 1 -- 172.21.15.31:0/4254065649 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f66a00a4490 con 0x564db80f3f60 2024-04-25T16:07:45.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.710+0000 7f66bb9ac740 10 client.26534 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:45.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.710+0000 7f66bb9ac740 10 client.26534 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:07:45.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.710+0000 7f66bb9ac740 20 client.26534 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:45.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.710+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564db8567f60 0x564db856a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:45.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.710+0000 7f66bb9ac740 4 client.26534 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:07:45.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.710+0000 7f66bb9ac740 1 -- 172.21.15.31:0/4254065649 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564db83be610 con 0x564db8567f60 2024-04-25T16:07:45.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.713+0000 7f66add9b640 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564db8567f60 0x564db856a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.714+0000 7f66add9b640 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564db8567f60 0x564db856a380 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d9f5c7c5c649814c server_cookie=da1769bb85e2f2e5 in_seq=0 out_seq=0 2024-04-25T16:07:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f6695ffb640 10 client.26534 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:07:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f6695ffb640 1 -- 172.21.15.31:0/4254065649 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564db83be610 con 0x564db8567f60 2024-04-25T16:07:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f6695ffb640 10 client.26534 handle_command_reply: tid=0 2024-04-25T16:07:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f66bb9ac740 1 client.26534 shutdown 2024-04-25T16:07:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f66bb9ac740 1 -- 172.21.15.31:0/4254065649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db80f3f60 msgr2=0x564db84e8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:45.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db80f3f60 0x564db84e8490 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f66a000d490 tx=0x7f66a00057f0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f66bb9ac740 1 -- 172.21.15.31:0/4254065649 shutdown_connections 2024-04-25T16:07:45.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x564db8567f60 0x564db856a380 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564db81146b0 0x564db84e50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564db810c4d0 0x564db84e89d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f66bb9ac740 1 --2- 172.21.15.31:0/4254065649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564db80f3f60 0x564db84e8490 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f66bb9ac740 1 -- 172.21.15.31:0/4254065649 >> 172.21.15.31:0/4254065649 conn(0x564db81294e0 msgr2=0x564db84dfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:45.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.715+0000 7f6695ffb640 0 client.26534 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:07:45.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.716+0000 7f66bb9ac740 1 -- 172.21.15.31:0/4254065649 shutdown_connections 2024-04-25T16:07:45.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.716+0000 7f66bb9ac740 1 -- 172.21.15.31:0/4254065649 wait complete. 2024-04-25T16:07:45.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.716+0000 7f66bb9ac740 20 client.26534 trim_cache size 0 max 16384 2024-04-25T16:07:45.720 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:07:45.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.724+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f66980afc20 msgr2=0x7f66980b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:45.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.724+0000 7f66af59e640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f66980afc20 0x7f66980b2130 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f66a40058c0 tx=0x7f66a4004730 comp rx=0 tx=0).stop 2024-04-25T16:07:45.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.724+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b010a2d0 msgr2=0x7f66b01a0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:45.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.724+0000 7f66af59e640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b010a2d0 0x7f66b01a0060 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f66a000f7d0 tx=0x7f66a000fc90 comp rx=0 tx=0).stop 2024-04-25T16:07:45.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.725+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 shutdown_connections 2024-04-25T16:07:45.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.725+0000 7f66af59e640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f66980afc20 0x7f66980b2130 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.725+0000 7f66af59e640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66b01a5520 0x7f66b01a7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.725+0000 7f66af59e640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f66b01102c0 0x7f66b01a05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.725+0000 7f66af59e640 1 --2- 172.21.15.31:0/2860610174 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f66b010a2d0 0x7f66b01a0060 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:45.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.725+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 >> 172.21.15.31:0/2860610174 conn(0x7f66b0070320 msgr2=0x7f66b0072740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:45.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.727+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 shutdown_connections 2024-04-25T16:07:45.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:45.728+0000 7f66af59e640 1 -- 172.21.15.31:0/2860610174 wait complete. 2024-04-25T16:07:45.804 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:07:46.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.245+0000 7fab47fff640 1 -- 172.21.15.31:0/2543559605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4810b190 msgr2=0x7fab4810b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:46.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.245+0000 7fab47fff640 1 --2- 172.21.15.31:0/2543559605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4810b190 0x7fab4810b5a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fab40009ca0 tx=0x7fab4002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.246+0000 7fab47fff640 1 -- 172.21.15.31:0/2543559605 shutdown_connections 2024-04-25T16:07:46.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.246+0000 7fab47fff640 1 --2- 172.21.15.31:0/2543559605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab4810fe90 0x7fab481102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.246+0000 7fab47fff640 1 --2- 172.21.15.31:0/2543559605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4810b190 0x7fab4810b5a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.246+0000 7fab47fff640 1 --2- 172.21.15.31:0/2543559605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4810a340 0x7fab4810ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.246+0000 7fab47fff640 1 -- 172.21.15.31:0/2543559605 >> 172.21.15.31:0/2543559605 conn(0x7fab48070320 msgr2=0x7fab480727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:46.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.246+0000 7fab47fff640 1 -- 172.21.15.31:0/2543559605 shutdown_connections 2024-04-25T16:07:46.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.246+0000 7fab47fff640 1 -- 172.21.15.31:0/2543559605 wait complete. 2024-04-25T16:07:46.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.248+0000 7fab47fff640 1 Processor -- start 2024-04-25T16:07:46.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.249+0000 7fab47fff640 1 -- start start 2024-04-25T16:07:46.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.249+0000 7fab47fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4810a340 0x7fab48117090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.249+0000 7fab47fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab4810b190 0x7fab481175d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.249+0000 7fab47fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4810fe90 0x7fab48118530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.250+0000 7fab47fff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab4811e280 con 0x7fab4810a340 2024-04-25T16:07:46.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.250+0000 7fab47fff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fab4811e100 con 0x7fab4810b190 2024-04-25T16:07:46.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.250+0000 7fab47fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab4811e400 con 0x7fab4810fe90 2024-04-25T16:07:46.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.250+0000 7fab467fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab4810b190 0x7fab481175d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.250+0000 7fab467fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab4810b190 0x7fab481175d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48160/0 (socket says 172.21.15.31:48160) 2024-04-25T16:07:46.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.251+0000 7fab467fc640 1 -- 172.21.15.31:0/3417678321 learned_addr learned my addr 172.21.15.31:0/3417678321 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:46.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.251+0000 7fab46ffd640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4810a340 0x7fab48117090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.252+0000 7fab467fc640 1 -- 172.21.15.31:0/3417678321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4810fe90 msgr2=0x7fab48118530 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:07:46.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.252+0000 7fab467fc640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4810fe90 0x7fab48118530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.252+0000 7fab467fc640 1 -- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4810a340 msgr2=0x7fab48117090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:46.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.252+0000 7fab467fc640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4810a340 0x7fab48117090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.252+0000 7fab467fc640 1 -- 172.21.15.31:0/3417678321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab48118ba0 con 0x7fab4810b190 2024-04-25T16:07:46.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.265+0000 7fab467fc640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab4810b190 0x7fab481175d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fab4002aee0 tx=0x7fab40031f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:46.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.265+0000 7fab2ffff640 1 -- 172.21.15.31:0/3417678321 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab40030d30 con 0x7fab4810b190 2024-04-25T16:07:46.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.265+0000 7fab2ffff640 1 -- 172.21.15.31:0/3417678321 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab4002d070 con 0x7fab4810b190 2024-04-25T16:07:46.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.265+0000 7fab2ffff640 1 -- 172.21.15.31:0/3417678321 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab4003a640 con 0x7fab4810b190 2024-04-25T16:07:46.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.267+0000 7fab47fff640 1 -- 172.21.15.31:0/3417678321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab48118e30 con 0x7fab4810b190 2024-04-25T16:07:46.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.267+0000 7fab47fff640 1 -- 172.21.15.31:0/3417678321 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab48119330 con 0x7fab4810b190 2024-04-25T16:07:46.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.276+0000 7fab2ffff640 1 -- 172.21.15.31:0/3417678321 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fab4003b9d0 con 0x7fab4810b190 2024-04-25T16:07:46.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.278+0000 7fab2ffff640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fab300afc30 0x7fab300b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.278+0000 7fab2ffff640 1 -- 172.21.15.31:0/3417678321 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fab400ffd60 con 0x7fab4810b190 2024-04-25T16:07:46.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.289+0000 7fab5437d740 1 Processor -- start 2024-04-25T16:07:46.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.296+0000 7fab46ffd640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fab300afc30 0x7fab300b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.298+0000 7fab5437d740 1 -- start start 2024-04-25T16:07:46.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.299+0000 7fab5437d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19d29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.299+0000 7fab5437d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de16074d0 0x563de19d2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.299+0000 7fab5437d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de160f6b0 0x563de19d3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.300+0000 7fab5437d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563de1936d80 con 0x563de160f6b0 2024-04-25T16:07:46.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.300+0000 7fab5437d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563de18f0a20 con 0x563de1630390 2024-04-25T16:07:46.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.300+0000 7fab5437d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563de1932d40 con 0x563de16074d0 2024-04-25T16:07:46.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.302+0000 7fab467fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de16074d0 0x563de19d2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.302+0000 7fab467fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de16074d0 0x563de19d2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53834/0 (socket says 172.21.15.31:53834) 2024-04-25T16:07:46.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.302+0000 7fab467fc640 1 -- 172.21.15.31:0/3858371999 learned_addr learned my addr 172.21.15.31:0/3858371999 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:46.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.302+0000 7fab46ffd640 1 --2- 172.21.15.31:0/3858371999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19d29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.311+0000 7fab467fc640 1 -- 172.21.15.31:0/3858371999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 msgr2=0x563de19d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:46.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.311+0000 7fab467fc640 1 --2- 172.21.15.31:0/3858371999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19d29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.311+0000 7fab467fc640 1 -- 172.21.15.31:0/3858371999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de160f6b0 msgr2=0x563de19d3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:46.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.311+0000 7fab467fc640 1 --2- 172.21.15.31:0/3858371999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de160f6b0 0x563de19d3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.311+0000 7fab467fc640 1 -- 172.21.15.31:0/3858371999 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563de18fbc30 con 0x563de16074d0 2024-04-25T16:07:46.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.312+0000 7fab46ffd640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fab300afc30 0x7fab300b2140 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fab48109c10 tx=0x7fab380113e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:46.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.320+0000 7fab467fc640 1 --2- 172.21.15.31:0/3858371999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de16074d0 0x563de19d2f60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab400c8180 tx=0x7fab4002a8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2bb1f4c2b1e589d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:46.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.324+0000 7fab2d7fa640 1 -- 172.21.15.31:0/3858371999 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab400c88f0 con 0x563de16074d0 2024-04-25T16:07:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.325+0000 7fab2d7fa640 1 -- 172.21.15.31:0/3858371999 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab40036040 con 0x563de16074d0 2024-04-25T16:07:46.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.325+0000 7fab2d7fa640 1 -- 172.21.15.31:0/3858371999 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab400d0bc0 con 0x563de16074d0 2024-04-25T16:07:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.330+0000 7fab5437d740 1 -- 172.21.15.31:0/3858371999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de16074d0 msgr2=0x563de19d2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:46.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.330+0000 7fab5437d740 1 --2- 172.21.15.31:0/3858371999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de16074d0 0x563de19d2f60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab400c8180 tx=0x7fab4002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.336+0000 7fab5437d740 1 -- 172.21.15.31:0/3858371999 shutdown_connections 2024-04-25T16:07:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.336+0000 7fab5437d740 1 --2- 172.21.15.31:0/3858371999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de160f6b0 0x563de19d3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.336+0000 7fab5437d740 1 --2- 172.21.15.31:0/3858371999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de16074d0 0x563de19d2f60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.336+0000 7fab5437d740 1 --2- 172.21.15.31:0/3858371999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19d29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.336+0000 7fab5437d740 1 -- 172.21.15.31:0/3858371999 >> 172.21.15.31:0/3858371999 conn(0x563de16244e0 msgr2=0x563de19ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:46.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.347+0000 7fab5437d740 1 -- 172.21.15.31:0/3858371999 shutdown_connections 2024-04-25T16:07:46.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.347+0000 7fab5437d740 1 -- 172.21.15.31:0/3858371999 wait complete. 2024-04-25T16:07:46.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.348+0000 7fab5437d740 1 -- start start 2024-04-25T16:07:46.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.348+0000 7fab5437d740 1 Processor -- start 2024-04-25T16:07:46.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.349+0000 7fab5437d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de16074d0 0x563de19dc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.349+0000 7fab5437d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 0x563de19dc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.349+0000 7fab5437d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19e38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.350+0000 7fab5437d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563de1932d40 con 0x563de16074d0 2024-04-25T16:07:46.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.350+0000 7fab5437d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563de18bddc0 con 0x563de1630390 2024-04-25T16:07:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.350+0000 7fab5437d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563de19d44c0 con 0x563de160f6b0 2024-04-25T16:07:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.350+0000 7fab5437d740 10 client.0 _ll_register_callbacks cb 0x563de18d9600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:46.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.351+0000 7fab477fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19e38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.352+0000 7fab477fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19e38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48176/0 (socket says 172.21.15.31:48176) 2024-04-25T16:07:46.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.352+0000 7fab477fe640 1 -- 172.21.15.31:0/351010865 learned_addr learned my addr 172.21.15.31:0/351010865 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:46.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.353+0000 7fab467fc640 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 0x563de19dc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.354+0000 7fab467fc640 1 -- 172.21.15.31:0/351010865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 msgr2=0x563de19e38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.354+0000 7fab467fc640 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19e38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.354+0000 7fab467fc640 1 -- 172.21.15.31:0/351010865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de16074d0 msgr2=0x563de19dc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:46.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.354+0000 7fab467fc640 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de16074d0 0x563de19dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.354+0000 7fab467fc640 1 -- 172.21.15.31:0/351010865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563de16483f0 con 0x563de160f6b0 2024-04-25T16:07:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.355+0000 7fab467fc640 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 0x563de19dc960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab400c7ef0 tx=0x7fab400d0dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b246b79d38697016 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.355+0000 7fab2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:46.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.355+0000 7fab2aff5640 1 -- 172.21.15.31:0/351010865 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab400e1070 con 0x563de160f6b0 2024-04-25T16:07:46.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.356+0000 7fab5437d740 1 -- 172.21.15.31:0/351010865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab40036040 con 0x563de160f6b0 2024-04-25T16:07:46.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.356+0000 7fab2aff5640 1 -- 172.21.15.31:0/351010865 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fab40044c50 con 0x563de160f6b0 2024-04-25T16:07:46.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.356+0000 7fab2aff5640 1 -- 172.21.15.31:0/351010865 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fab400c81f0 con 0x563de160f6b0 2024-04-25T16:07:46.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.356+0000 7fab5437d740 10 client.26500 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:46.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.356+0000 7fab5437d740 1 -- 172.21.15.31:0/351010865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563de17127d0 con 0x563de160f6b0 2024-04-25T16:07:46.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.361+0000 7fab2aff5640 1 -- 172.21.15.31:0/351010865 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab40036040 con 0x563de160f6b0 2024-04-25T16:07:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.362+0000 7fab2aff5640 1 -- 172.21.15.31:0/351010865 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fab400c3b60 con 0x563de160f6b0 2024-04-25T16:07:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.362+0000 7fab5437d740 10 client.26500 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.362+0000 7fab5437d740 10 client.26500 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:07:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.362+0000 7fab5437d740 20 client.26500 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.362+0000 7fab5437d740 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563de15eef60 0x563de19ea0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.362+0000 7fab5437d740 4 client.26500 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:46.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.362+0000 7fab5437d740 1 -- 172.21.15.31:0/351010865 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563de1944990 con 0x563de15eef60 2024-04-25T16:07:46.372 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.369+0000 7fab46ffd640 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563de15eef60 0x563de19ea0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.378+0000 7fab46ffd640 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563de15eef60 0x563de19ea0c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9c4cc3e3f670b7c6 server_cookie=472cdd4e08383227 in_seq=0 out_seq=0 2024-04-25T16:07:46.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.378+0000 7fab2aff5640 10 client.26500 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:07:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.382+0000 7fab2aff5640 1 -- 172.21.15.31:0/351010865 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563de1944990 con 0x563de15eef60 2024-04-25T16:07:46.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.386+0000 7fab2aff5640 10 client.26500 handle_command_reply: tid=0 2024-04-25T16:07:46.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.386+0000 7fab5437d740 1 client.26500 shutdown 2024-04-25T16:07:46.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.387+0000 7fab5437d740 1 -- 172.21.15.31:0/351010865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 msgr2=0x563de19dc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:46.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.387+0000 7fab5437d740 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 0x563de19dc960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fab400c7ef0 tx=0x7fab400d0dd0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.387+0000 7fab5437d740 1 -- 172.21.15.31:0/351010865 shutdown_connections 2024-04-25T16:07:46.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.387+0000 7fab5437d740 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19e38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.387+0000 7fab5437d740 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563de15eef60 0x563de19ea0c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.387+0000 7fab5437d740 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 0x563de19dc960 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.387+0000 7fab5437d740 1 --2- 172.21.15.31:0/351010865 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de16074d0 0x563de19dc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.387+0000 7fab5437d740 1 -- 172.21.15.31:0/351010865 >> 172.21.15.31:0/351010865 conn(0x563de16244e0 msgr2=0x563de19d1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.388+0000 7fab2aff5640 0 client.26500 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:07:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.388+0000 7fab5437d740 1 -- 172.21.15.31:0/351010865 shutdown_connections 2024-04-25T16:07:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.388+0000 7fab5437d740 1 -- 172.21.15.31:0/351010865 wait complete. 2024-04-25T16:07:46.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.389+0000 7fab5437d740 20 client.26500 trim_cache size 0 max 16384 2024-04-25T16:07:46.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.477+0000 7fab5437d740 1 Processor -- start 2024-04-25T16:07:46.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.477+0000 7fab5437d740 1 -- start start 2024-04-25T16:07:46.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.478+0000 7fab5437d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de16074d0 0x563de19c9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.478+0000 7fab5437d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de160f6b0 0x563de19d7300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.478+0000 7fab5437d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de1630390 0x563de19d7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.478+0000 7fab5437d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab3400e370 con 0x563de160f6b0 2024-04-25T16:07:46.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.479+0000 7fab5437d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fab3400feb0 con 0x563de16074d0 2024-04-25T16:07:46.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.479+0000 7fab5437d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab3400ea40 con 0x563de1630390 2024-04-25T16:07:46.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.479+0000 7fab46ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de16074d0 0x563de19c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.479+0000 7fab46ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de16074d0 0x563de19c9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48178/0 (socket says 172.21.15.31:48178) 2024-04-25T16:07:46.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.479+0000 7fab46ffd640 1 -- 172.21.15.31:0/4121908297 learned_addr learned my addr 172.21.15.31:0/4121908297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:46.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.480+0000 7fab46ffd640 1 -- 172.21.15.31:0/4121908297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de1630390 msgr2=0x563de19d7840 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:46.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.480+0000 7fab46ffd640 1 --2- 172.21.15.31:0/4121908297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de1630390 0x563de19d7840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.480+0000 7fab46ffd640 1 -- 172.21.15.31:0/4121908297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de160f6b0 msgr2=0x563de19d7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:46.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.480+0000 7fab46ffd640 1 --2- 172.21.15.31:0/4121908297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de160f6b0 0x563de19d7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.480+0000 7fab46ffd640 1 -- 172.21.15.31:0/4121908297 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563de19bfba0 con 0x563de16074d0 2024-04-25T16:07:46.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.482+0000 7fab46ffd640 1 --2- 172.21.15.31:0/4121908297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de16074d0 0x563de19c9360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab380127f0 tx=0x7fab3801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=46725974e0d5c808 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:46.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.487+0000 7fab2cff9640 1 -- 172.21.15.31:0/4121908297 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38004020 con 0x563de16074d0 2024-04-25T16:07:46.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.487+0000 7fab2cff9640 1 -- 172.21.15.31:0/4121908297 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab38020d60 con 0x563de16074d0 2024-04-25T16:07:46.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.488+0000 7fab5437d740 1 -- 172.21.15.31:0/4121908297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de16074d0 msgr2=0x563de19c9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:46.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.488+0000 7fab5437d740 1 --2- 172.21.15.31:0/4121908297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de16074d0 0x563de19c9360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab380127f0 tx=0x7fab3801f000 comp rx=0 tx=0).stop 2024-04-25T16:07:46.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.489+0000 7fab5437d740 1 -- 172.21.15.31:0/4121908297 shutdown_connections 2024-04-25T16:07:46.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.489+0000 7fab5437d740 1 --2- 172.21.15.31:0/4121908297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de1630390 0x563de19d7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.489+0000 7fab5437d740 1 --2- 172.21.15.31:0/4121908297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de160f6b0 0x563de19d7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.489+0000 7fab5437d740 1 --2- 172.21.15.31:0/4121908297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de16074d0 0x563de19c9360 secure :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fab380127f0 tx=0x7fab3801f000 comp rx=0 tx=0).stop 2024-04-25T16:07:46.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.489+0000 7fab5437d740 1 -- 172.21.15.31:0/4121908297 >> 172.21.15.31:0/4121908297 conn(0x563de16244e0 msgr2=0x563de19dc420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:46.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.499+0000 7fab5437d740 1 -- 172.21.15.31:0/4121908297 shutdown_connections 2024-04-25T16:07:46.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.499+0000 7fab5437d740 1 -- 172.21.15.31:0/4121908297 wait complete. 2024-04-25T16:07:46.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.500+0000 7fab5437d740 1 -- start start 2024-04-25T16:07:46.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.500+0000 7fab5437d740 1 Processor -- start 2024-04-25T16:07:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.506+0000 7fab5437d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 0x563de19e2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.506+0000 7fab5437d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19dfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.507+0000 7fab5437d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de15c43a0 0x563de19e0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.507+0000 7fab5437d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fab3400f7e0 con 0x563de15c43a0 2024-04-25T16:07:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.507+0000 7fab5437d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fab3400f110 con 0x563de1630390 2024-04-25T16:07:46.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.507+0000 7fab5437d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fab3400c400 con 0x563de160f6b0 2024-04-25T16:07:46.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.507+0000 7fab5437d740 10 client.0 _ll_register_callbacks cb 0x7fab400c41a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:46.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.507+0000 7fab467fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19dfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.507+0000 7fab467fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19dfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48192/0 (socket says 172.21.15.31:48192) 2024-04-25T16:07:46.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.508+0000 7fab467fc640 1 -- 172.21.15.31:0/3320546056 learned_addr learned my addr 172.21.15.31:0/3320546056 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:46.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.508+0000 7fab46ffd640 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 0x563de19e2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.508+0000 7fab467fc640 1 -- 172.21.15.31:0/3320546056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 msgr2=0x563de19e2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.508+0000 7fab467fc640 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 0x563de19e2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.508+0000 7fab467fc640 1 -- 172.21.15.31:0/3320546056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de15c43a0 msgr2=0x563de19e0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.508+0000 7fab467fc640 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de15c43a0 0x563de19e0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.508+0000 7fab467fc640 1 -- 172.21.15.31:0/3320546056 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563de1a504e0 con 0x563de1630390 2024-04-25T16:07:46.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.509+0000 7fab467fc640 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19dfab0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fab400c8180 tx=0x7fab40009940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e71015e205e21b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:46.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.510+0000 7fab2aff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:46.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.510+0000 7fab2aff5640 1 -- 172.21.15.31:0/3320546056 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab400cc5a0 con 0x563de1630390 2024-04-25T16:07:46.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.510+0000 7fab5437d740 1 -- 172.21.15.31:0/3320546056 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563de193e800 con 0x563de1630390 2024-04-25T16:07:46.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.510+0000 7fab2aff5640 1 -- 172.21.15.31:0/3320546056 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fab400c7120 con 0x563de1630390 2024-04-25T16:07:46.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.511+0000 7fab2aff5640 1 -- 172.21.15.31:0/3320546056 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fab400c81f0 con 0x563de1630390 2024-04-25T16:07:46.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.511+0000 7fab5437d740 10 client.26552 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:46.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.511+0000 7fab5437d740 1 -- 172.21.15.31:0/3320546056 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563de1a504e0 con 0x563de1630390 2024-04-25T16:07:46.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.512+0000 7fab2aff5640 1 -- 172.21.15.31:0/3320546056 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563de193e800 con 0x563de1630390 2024-04-25T16:07:46.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.513+0000 7fab2aff5640 1 -- 172.21.15.31:0/3320546056 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fab400c7450 con 0x563de1630390 2024-04-25T16:07:46.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.514+0000 7fab5437d740 10 client.26552 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:46.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.514+0000 7fab5437d740 10 client.26552 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:07:46.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.514+0000 7fab5437d740 20 client.26552 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:46.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.514+0000 7fab5437d740 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563de1a51da0 0x563de1a541c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:46.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.514+0000 7fab5437d740 4 client.26552 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:07:46.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.514+0000 7fab5437d740 1 -- 172.21.15.31:0/3320546056 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563de18b9610 con 0x563de1a51da0 2024-04-25T16:07:46.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.514+0000 7fab46ffd640 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563de1a51da0 0x563de1a541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:46.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.516+0000 7fab46ffd640 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563de1a51da0 0x563de1a541c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f06d503b6fab6c74 server_cookie=39380d68391117b3 in_seq=0 out_seq=0 2024-04-25T16:07:46.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.516+0000 7fab2aff5640 10 client.26552 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:07:46.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.516+0000 7fab2aff5640 1 -- 172.21.15.31:0/3320546056 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563de18b9610 con 0x563de1a51da0 2024-04-25T16:07:46.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.516+0000 7fab2aff5640 10 client.26552 handle_command_reply: tid=0 2024-04-25T16:07:46.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.516+0000 7fab5437d740 1 client.26552 shutdown 2024-04-25T16:07:46.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 -- 172.21.15.31:0/3320546056 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 msgr2=0x563de19dfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:46.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19dfab0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fab400c8180 tx=0x7fab40009940 comp rx=0 tx=0).stop 2024-04-25T16:07:46.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 -- 172.21.15.31:0/3320546056 shutdown_connections 2024-04-25T16:07:46.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563de15c43a0 0x563de19e0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563de1630390 0x563de19dfab0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x563de1a51da0 0x563de1a541c0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 --2- 172.21.15.31:0/3320546056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563de160f6b0 0x563de19e2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab2aff5640 0 client.26552 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:07:46.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 -- 172.21.15.31:0/3320546056 >> 172.21.15.31:0/3320546056 conn(0x563de16244e0 msgr2=0x563de19dc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:46.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 -- 172.21.15.31:0/3320546056 shutdown_connections 2024-04-25T16:07:46.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.522+0000 7fab5437d740 1 -- 172.21.15.31:0/3320546056 wait complete. 2024-04-25T16:07:46.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.523+0000 7fab5437d740 20 client.26552 trim_cache size 0 max 16384 2024-04-25T16:07:46.526 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:07:46.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.533+0000 7fab47fff640 1 -- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fab300afc30 msgr2=0x7fab300b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:46.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.533+0000 7fab47fff640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fab300afc30 0x7fab300b2140 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fab48109c10 tx=0x7fab380113e0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.533+0000 7fab47fff640 1 -- 172.21.15.31:0/3417678321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab4810b190 msgr2=0x7fab481175d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:46.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.533+0000 7fab47fff640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab4810b190 0x7fab481175d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fab4002aee0 tx=0x7fab40031f90 comp rx=0 tx=0).stop 2024-04-25T16:07:46.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.538+0000 7fab47fff640 1 -- 172.21.15.31:0/3417678321 shutdown_connections 2024-04-25T16:07:46.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.538+0000 7fab47fff640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fab300afc30 0x7fab300b2140 secure :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fab48109c10 tx=0x7fab380113e0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.538+0000 7fab47fff640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab4810fe90 0x7fab48118530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.538+0000 7fab47fff640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fab4810b190 0x7fab481175d0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.538+0000 7fab47fff640 1 --2- 172.21.15.31:0/3417678321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fab4810a340 0x7fab48117090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:46.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.538+0000 7fab47fff640 1 -- 172.21.15.31:0/3417678321 >> 172.21.15.31:0/3417678321 conn(0x7fab48070320 msgr2=0x7fab4810be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:46.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.543+0000 7fab47fff640 1 -- 172.21.15.31:0/3417678321 shutdown_connections 2024-04-25T16:07:46.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:46.543+0000 7fab47fff640 1 -- 172.21.15.31:0/3417678321 wait complete. 2024-04-25T16:07:46.619 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:07:46.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:46 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.689 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:46 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:46.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:46 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:07:47.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.067+0000 7fb8a23a8640 1 -- 172.21.15.31:0/184300326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb89c10a420 msgr2=0x7fb89c10add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.067+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/184300326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb89c10a420 0x7fb89c10add0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009900 tx=0x7fb88c02a410 comp rx=0 tx=0).stop 2024-04-25T16:07:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.072+0000 7fb8a23a8640 1 -- 172.21.15.31:0/184300326 shutdown_connections 2024-04-25T16:07:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.072+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/184300326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb89c110030 0x7fb89c110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.072+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/184300326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb89c10b310 0x7fb89c10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.072+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/184300326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb89c10a420 0x7fb89c10add0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.072+0000 7fb8a23a8640 1 -- 172.21.15.31:0/184300326 >> 172.21.15.31:0/184300326 conn(0x7fb89c0704e0 msgr2=0x7fb89c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:47.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.077+0000 7fb8a23a8640 1 -- 172.21.15.31:0/184300326 shutdown_connections 2024-04-25T16:07:47.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.077+0000 7fb8a23a8640 1 -- 172.21.15.31:0/184300326 wait complete. 2024-04-25T16:07:47.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.078+0000 7fb8a23a8640 1 Processor -- start 2024-04-25T16:07:47.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.078+0000 7fb8a23a8640 1 -- start start 2024-04-25T16:07:47.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.079+0000 7fb8a23a8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb89c10a420 0x7fb89c1a9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.079+0000 7fb8a23a8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb89c10b310 0x7fb89c1a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.079+0000 7fb8a23a8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb89c110030 0x7fb89c07ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.079+0000 7fb8a23a8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c11e560 con 0x7fb89c10a420 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.079+0000 7fb8a23a8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c11e3e0 con 0x7fb89c110030 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.079+0000 7fb8a23a8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb89c11e6e0 con 0x7fb89c10b310 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.080+0000 7fb8a0945640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb89c110030 0x7fb89c07ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.080+0000 7fb8a0945640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb89c110030 0x7fb89c07ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48208/0 (socket says 172.21.15.31:48208) 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.080+0000 7fb8a0945640 1 -- 172.21.15.31:0/923102405 learned_addr learned my addr 172.21.15.31:0/923102405 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.080+0000 7fb89b7fe640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb89c10b310 0x7fb89c1a9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.080+0000 7fb8a0945640 1 -- 172.21.15.31:0/923102405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb89c10b310 msgr2=0x7fb89c1a9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.080+0000 7fb8a0945640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb89c10b310 0x7fb89c1a9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.080+0000 7fb8a0945640 1 -- 172.21.15.31:0/923102405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb89c10a420 msgr2=0x7fb89c1a9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.080+0000 7fb8a0945640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb89c10a420 0x7fb89c1a9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.080+0000 7fb8a0945640 1 -- 172.21.15.31:0/923102405 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89c07b470 con 0x7fb89c110030 2024-04-25T16:07:47.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.081+0000 7fb8a0945640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb89c110030 0x7fb89c07ae00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb89400eb10 tx=0x7fb89400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.087+0000 7fb8997fa640 1 -- 172.21.15.31:0/923102405 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89400ce20 con 0x7fb89c110030 2024-04-25T16:07:47.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.087+0000 7fb8a23a8640 1 -- 172.21.15.31:0/923102405 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb89c07b760 con 0x7fb89c110030 2024-04-25T16:07:47.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.087+0000 7fb8a23a8640 1 -- 172.21.15.31:0/923102405 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb89c07bd30 con 0x7fb89c110030 2024-04-25T16:07:47.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.087+0000 7fb8997fa640 1 -- 172.21.15.31:0/923102405 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb894005020 con 0x7fb89c110030 2024-04-25T16:07:47.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.087+0000 7fb8997fa640 1 -- 172.21.15.31:0/923102405 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894010690 con 0x7fb89c110030 2024-04-25T16:07:47.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.095+0000 7fb8997fa640 1 -- 172.21.15.31:0/923102405 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb894010830 con 0x7fb89c110030 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.103+0000 7fb8a9259740 1 Processor -- start 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.103+0000 7fb8a9259740 1 -- start start 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.103+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896423d390 0x5589645df810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.103+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 0x5589645dfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.103+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 0x5589645e0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.103+0000 7fb8a9259740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x558964543d80 con 0x55896421c6b0 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.103+0000 7fb8a9259740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5589644fda20 con 0x55896423d390 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.103+0000 7fb8a9259740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55896453fd40 con 0x5589642144d0 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.103+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 0x5589645dfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.104+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 0x5589645dfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53902/0 (socket says 172.21.15.31:53902) 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.104+0000 7fb89bfff640 1 -- 172.21.15.31:0/1712399936 learned_addr learned my addr 172.21.15.31:0/1712399936 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:47.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.104+0000 7fb8997fa640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8840afd20 0x7fb8840b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.104+0000 7fb89bfff640 1 -- 172.21.15.31:0/1712399936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896423d390 msgr2=0x5589645df810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:07:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.104+0000 7fb89bfff640 1 --2- 172.21.15.31:0/1712399936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896423d390 0x5589645df810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.104+0000 7fb89bfff640 1 -- 172.21.15.31:0/1712399936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 msgr2=0x5589645e0a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:07:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.104+0000 7fb89bfff640 1 --2- 172.21.15.31:0/1712399936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 0x5589645e0a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.104+0000 7fb89bfff640 1 -- 172.21.15.31:0/1712399936 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558964508c30 con 0x5589642144d0 2024-04-25T16:07:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.105+0000 7fb89bfff640 1 --2- 172.21.15.31:0/1712399936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 0x5589645dfd90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb88c031500 tx=0x7fb88c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c11d275dbdaef35 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.105+0000 7fb8997fa640 1 -- 172.21.15.31:0/923102405 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb8940e00d0 con 0x7fb89c110030 2024-04-25T16:07:47.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.106+0000 7fb8a0945640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8840afd20 0x7fb8840b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.109+0000 7fb8a0945640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8840afd20 0x7fb8840b2230 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb8940059c0 tx=0x7fb894005b40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.112+0000 7fb8827fc640 1 -- 172.21.15.31:0/1712399936 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c03e070 con 0x5589642144d0 2024-04-25T16:07:47.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.112+0000 7fb8827fc640 1 -- 172.21.15.31:0/1712399936 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88c005570 con 0x5589642144d0 2024-04-25T16:07:47.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.116+0000 7fb8827fc640 1 -- 172.21.15.31:0/1712399936 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c035040 con 0x5589642144d0 2024-04-25T16:07:47.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.117+0000 7fb8a9259740 1 -- 172.21.15.31:0/1712399936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 msgr2=0x5589645dfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.117+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1712399936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 0x5589645dfd90 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb88c031500 tx=0x7fb88c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.118+0000 7fb8a9259740 1 -- 172.21.15.31:0/1712399936 shutdown_connections 2024-04-25T16:07:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.118+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1712399936 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 0x5589645e0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.118+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1712399936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 0x5589645dfd90 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.118+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1712399936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896423d390 0x5589645df810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.118+0000 7fb8a9259740 1 -- 172.21.15.31:0/1712399936 >> 172.21.15.31:0/1712399936 conn(0x5589642314e0 msgr2=0x5589645db020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.118+0000 7fb8a9259740 1 -- 172.21.15.31:0/1712399936 shutdown_connections 2024-04-25T16:07:47.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.118+0000 7fb8a9259740 1 -- 172.21.15.31:0/1712399936 wait complete. 2024-04-25T16:07:47.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.119+0000 7fb8a9259740 1 -- start start 2024-04-25T16:07:47.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.119+0000 7fb8a9259740 1 Processor -- start 2024-04-25T16:07:47.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.120+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 0x5589645e9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.120+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 0x5589645e9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.120+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896423d390 0x5589645f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.120+0000 7fb8a9259740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5589645e14d0 con 0x55896421c6b0 2024-04-25T16:07:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.120+0000 7fb8a9259740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5589644cadc0 con 0x55896423d390 2024-04-25T16:07:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.120+0000 7fb8a9259740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5589645e1650 con 0x5589642144d0 2024-04-25T16:07:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.120+0000 7fb8a9259740 10 client.0 _ll_register_callbacks cb 0x5589644e6600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.121+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 0x5589645e9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.121+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 0x5589645e9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60470/0 (socket says 172.21.15.31:60470) 2024-04-25T16:07:47.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.121+0000 7fb89bfff640 1 -- 172.21.15.31:0/217364673 learned_addr learned my addr 172.21.15.31:0/217364673 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:47.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.122+0000 7fb89bfff640 1 -- 172.21.15.31:0/217364673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 msgr2=0x5589645e9650 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:47.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.122+0000 7fb89bfff640 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 0x5589645e9650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.122+0000 7fb89bfff640 1 -- 172.21.15.31:0/217364673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896423d390 msgr2=0x5589645f0a10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:47.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.122+0000 7fb89bfff640 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896423d390 0x5589645f0a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.122+0000 7fb89bfff640 1 -- 172.21.15.31:0/217364673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589642553f0 con 0x55896421c6b0 2024-04-25T16:07:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.122+0000 7fb89bfff640 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 0x5589645e9b90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009900 tx=0x7fb88c004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa3bab6cde1fcc8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.122+0000 7fb87b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:07:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.123+0000 7fb87b7fe640 1 -- 172.21.15.31:0/217364673 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c03e070 con 0x55896421c6b0 2024-04-25T16:07:47.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.123+0000 7fb8a9259740 1 -- 172.21.15.31:0/217364673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb88c005570 con 0x55896421c6b0 2024-04-25T16:07:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.123+0000 7fb87b7fe640 1 -- 172.21.15.31:0/217364673 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb88c004050 con 0x55896421c6b0 2024-04-25T16:07:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.124+0000 7fb87b7fe640 1 -- 172.21.15.31:0/217364673 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb88c035040 con 0x55896421c6b0 2024-04-25T16:07:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.124+0000 7fb8a9259740 10 client.16914 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:47.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.124+0000 7fb8a9259740 1 -- 172.21.15.31:0/217364673 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55896431f7d0 con 0x55896421c6b0 2024-04-25T16:07:47.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.125+0000 7fb87b7fe640 1 -- 172.21.15.31:0/217364673 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88c005570 con 0x55896421c6b0 2024-04-25T16:07:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.127+0000 7fb87b7fe640 1 -- 172.21.15.31:0/217364673 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb88c0315a0 con 0x55896421c6b0 2024-04-25T16:07:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.127+0000 7fb8a9259740 10 client.16914 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.127+0000 7fb8a9259740 10 client.16914 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:07:47.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.127+0000 7fb8a9259740 20 client.16914 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:47.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.127+0000 7fb8a9259740 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5589641fbf60 0x5589645f7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.127+0000 7fb8a9259740 4 client.16914 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:47.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.127+0000 7fb8a9259740 1 -- 172.21.15.31:0/217364673 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558964551990 con 0x5589641fbf60 2024-04-25T16:07:47.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.129+0000 7fb89b7fe640 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5589641fbf60 0x5589645f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.131+0000 7fb89b7fe640 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5589641fbf60 0x5589645f7410 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d71915ac035ab4d8 server_cookie=f6c7b5b087b7a5ca in_seq=0 out_seq=0 2024-04-25T16:07:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.131+0000 7fb87b7fe640 10 client.16914 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:07:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.131+0000 7fb87b7fe640 1 -- 172.21.15.31:0/217364673 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558964551990 con 0x5589641fbf60 2024-04-25T16:07:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.132+0000 7fb87b7fe640 10 client.16914 handle_command_reply: tid=0 2024-04-25T16:07:47.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.132+0000 7fb8a9259740 1 client.16914 shutdown 2024-04-25T16:07:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.142+0000 7fb8a9259740 1 -- 172.21.15.31:0/217364673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 msgr2=0x5589645e9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.142+0000 7fb8a9259740 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 0x5589645e9b90 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009900 tx=0x7fb88c004dc0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.142+0000 7fb8a9259740 1 -- 172.21.15.31:0/217364673 shutdown_connections 2024-04-25T16:07:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.142+0000 7fb8a9259740 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5589641fbf60 0x5589645f7410 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.142+0000 7fb8a9259740 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896423d390 0x5589645f0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.142+0000 7fb8a9259740 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896421c6b0 0x5589645e9b90 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.142+0000 7fb8a9259740 1 --2- 172.21.15.31:0/217364673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5589642144d0 0x5589645e9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.142+0000 7fb8a9259740 1 -- 172.21.15.31:0/217364673 >> 172.21.15.31:0/217364673 conn(0x5589642314e0 msgr2=0x5589645ddf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.143+0000 7fb87b7fe640 0 client.16914 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:07:47.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.143+0000 7fb8a9259740 1 -- 172.21.15.31:0/217364673 shutdown_connections 2024-04-25T16:07:47.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.144+0000 7fb8a9259740 1 -- 172.21.15.31:0/217364673 wait complete. 2024-04-25T16:07:47.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.144+0000 7fb8a9259740 20 client.16914 trim_cache size 0 max 16384 2024-04-25T16:07:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.203+0000 7fb8a9259740 1 Processor -- start 2024-04-25T16:07:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.203+0000 7fb8a9259740 1 -- start start 2024-04-25T16:07:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.204+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589642144d0 0x5589645e7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.204+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896421c6b0 0x5589645e8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.204+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896423d390 0x5589645f0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.204+0000 7fb8a9259740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb88800e550 con 0x55896423d390 2024-04-25T16:07:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.204+0000 7fb8a9259740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb888010090 con 0x5589642144d0 2024-04-25T16:07:47.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.204+0000 7fb8a9259740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb88800ec20 con 0x55896421c6b0 2024-04-25T16:07:47.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.205+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896421c6b0 0x5589645e8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.207+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896421c6b0 0x5589645e8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53928/0 (socket says 172.21.15.31:53928) 2024-04-25T16:07:47.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.207+0000 7fb89bfff640 1 -- 172.21.15.31:0/3610721768 learned_addr learned my addr 172.21.15.31:0/3610721768 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:47.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.208+0000 7fb89bfff640 1 -- 172.21.15.31:0/3610721768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589642144d0 msgr2=0x5589645e7d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:07:47.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.208+0000 7fb89bfff640 1 --2- 172.21.15.31:0/3610721768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589642144d0 0x5589645e7d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.208+0000 7fb89bfff640 1 -- 172.21.15.31:0/3610721768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896423d390 msgr2=0x5589645f0720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:07:47.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.208+0000 7fb89bfff640 1 --2- 172.21.15.31:0/3610721768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896423d390 0x5589645f0720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.208+0000 7fb89bfff640 1 -- 172.21.15.31:0/3610721768 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589641fc190 con 0x55896421c6b0 2024-04-25T16:07:47.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.209+0000 7fb89bfff640 1 --2- 172.21.15.31:0/3610721768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896421c6b0 0x5589645e8260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009850 tx=0x7fb88c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4abf34ba734200ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.212+0000 7fb881ffb640 1 -- 172.21.15.31:0/3610721768 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c03e040 con 0x55896421c6b0 2024-04-25T16:07:47.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.212+0000 7fb881ffb640 1 -- 172.21.15.31:0/3610721768 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88c04c070 con 0x55896421c6b0 2024-04-25T16:07:47.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.212+0000 7fb881ffb640 1 -- 172.21.15.31:0/3610721768 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c053930 con 0x55896421c6b0 2024-04-25T16:07:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.214+0000 7fb8a9259740 1 -- 172.21.15.31:0/3610721768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896421c6b0 msgr2=0x5589645e8260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.214+0000 7fb8a9259740 1 --2- 172.21.15.31:0/3610721768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896421c6b0 0x5589645e8260 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb88c009850 tx=0x7fb88c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.218+0000 7fb8a9259740 1 -- 172.21.15.31:0/3610721768 shutdown_connections 2024-04-25T16:07:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.218+0000 7fb8a9259740 1 --2- 172.21.15.31:0/3610721768 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55896423d390 0x5589645f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.218+0000 7fb8a9259740 1 --2- 172.21.15.31:0/3610721768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896421c6b0 0x5589645e8260 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.218+0000 7fb8a9259740 1 --2- 172.21.15.31:0/3610721768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5589642144d0 0x5589645e7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.218+0000 7fb8a9259740 1 -- 172.21.15.31:0/3610721768 >> 172.21.15.31:0/3610721768 conn(0x5589642314e0 msgr2=0x5589645df790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:47.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.223+0000 7fb8a9259740 1 -- 172.21.15.31:0/3610721768 shutdown_connections 2024-04-25T16:07:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.223+0000 7fb8a9259740 1 -- 172.21.15.31:0/3610721768 wait complete. 2024-04-25T16:07:47.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.224+0000 7fb8a9259740 1 -- start start 2024-04-25T16:07:47.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.225+0000 7fb8a9259740 1 Processor -- start 2024-04-25T16:07:47.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.226+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589642144d0 0x5589645eca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.226+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896421c6b0 0x5589645ecf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.226+0000 7fb8a9259740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896423d390 0x5589645efec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.226+0000 7fb8a9259740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb88800f9c0 con 0x5589642144d0 2024-04-25T16:07:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.226+0000 7fb8a9259740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb88800f2f0 con 0x55896421c6b0 2024-04-25T16:07:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.226+0000 7fb8a9259740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb88800c530 con 0x55896423d390 2024-04-25T16:07:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.226+0000 7fb8a9259740 10 client.0 _ll_register_callbacks cb 0x7fb88c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:47.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.227+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896421c6b0 0x5589645ecf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.227+0000 7fb89bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896421c6b0 0x5589645ecf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48232/0 (socket says 172.21.15.31:48232) 2024-04-25T16:07:47.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.227+0000 7fb89bfff640 1 -- 172.21.15.31:0/1715831173 learned_addr learned my addr 172.21.15.31:0/1715831173 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:47.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.230+0000 7fb89bfff640 1 -- 172.21.15.31:0/1715831173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896423d390 msgr2=0x5589645efec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:07:47.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.230+0000 7fb89bfff640 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896423d390 0x5589645efec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.230+0000 7fb89bfff640 1 -- 172.21.15.31:0/1715831173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589642144d0 msgr2=0x5589645eca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:47.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.230+0000 7fb89bfff640 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589642144d0 0x5589645eca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.230+0000 7fb89bfff640 1 -- 172.21.15.31:0/1715831173 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5589645f0480 con 0x55896421c6b0 2024-04-25T16:07:47.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.232+0000 7fb89bfff640 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896421c6b0 0x5589645ecf80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb88c02aa00 tx=0x7fb88c005670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54edaaddab241a3d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.233+0000 7fb87b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:47.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.233+0000 7fb87b7fe640 1 -- 172.21.15.31:0/1715831173 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88c03e040 con 0x55896421c6b0 2024-04-25T16:07:47.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.234+0000 7fb8a9259740 1 -- 172.21.15.31:0/1715831173 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb88c04c070 con 0x55896421c6b0 2024-04-25T16:07:47.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.234+0000 7fb87b7fe640 1 -- 172.21.15.31:0/1715831173 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb88c004030 con 0x55896421c6b0 2024-04-25T16:07:47.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.234+0000 7fb87b7fe640 1 -- 172.21.15.31:0/1715831173 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fb88c004230 con 0x55896421c6b0 2024-04-25T16:07:47.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.236+0000 7fb8a9259740 10 client.26561 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:47.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.236+0000 7fb8a9259740 1 -- 172.21.15.31:0/1715831173 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55896466ef40 con 0x55896421c6b0 2024-04-25T16:07:47.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.240+0000 7fb87b7fe640 1 -- 172.21.15.31:0/1715831173 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb88c0043d0 con 0x55896421c6b0 2024-04-25T16:07:47.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.240+0000 7fb87b7fe640 1 -- 172.21.15.31:0/1715831173 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fb88c067510 con 0x55896421c6b0 2024-04-25T16:07:47.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.240+0000 7fb8a9259740 10 client.26561 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:47.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.240+0000 7fb8a9259740 10 client.26561 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:07:47.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.240+0000 7fb8a9259740 20 client.26561 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:47.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.240+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5589641d13a0 0x5589646718e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.241+0000 7fb8a9259740 4 client.26561 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:07:47.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.241+0000 7fb8a9259740 1 -- 172.21.15.31:0/1715831173 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fb88c04c070 con 0x5589641d13a0 2024-04-25T16:07:47.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.243+0000 7fb89b7fe640 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5589641d13a0 0x5589646718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.245+0000 7fb89b7fe640 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5589641d13a0 0x5589646718e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6b86a634fbfb229 server_cookie=379f53341ba26562 in_seq=0 out_seq=0 2024-04-25T16:07:47.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.245+0000 7fb87b7fe640 10 client.26561 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:07:47.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.246+0000 7fb87b7fe640 1 -- 172.21.15.31:0/1715831173 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fb88c04c070 con 0x5589641d13a0 2024-04-25T16:07:47.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.246+0000 7fb87b7fe640 10 client.26561 handle_command_reply: tid=0 2024-04-25T16:07:47.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.249+0000 7fb8a9259740 1 client.26561 shutdown 2024-04-25T16:07:47.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.250+0000 7fb8a9259740 1 -- 172.21.15.31:0/1715831173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896421c6b0 msgr2=0x5589645ecf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.250+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896421c6b0 0x5589645ecf80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb88c02aa00 tx=0x7fb88c005670 comp rx=0 tx=0).stop 2024-04-25T16:07:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.251+0000 7fb8a9259740 1 -- 172.21.15.31:0/1715831173 shutdown_connections 2024-04-25T16:07:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.251+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55896423d390 0x5589645efec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.251+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5589641d13a0 0x5589646718e0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.251+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55896421c6b0 0x5589645ecf80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.251+0000 7fb8a9259740 1 --2- 172.21.15.31:0/1715831173 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5589642144d0 0x5589645eca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.251+0000 7fb8a9259740 1 -- 172.21.15.31:0/1715831173 >> 172.21.15.31:0/1715831173 conn(0x5589642314e0 msgr2=0x5589645df790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:47.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.253+0000 7fb87b7fe640 0 client.26561 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:07:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.258+0000 7fb8a9259740 1 -- 172.21.15.31:0/1715831173 shutdown_connections 2024-04-25T16:07:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.258+0000 7fb8a9259740 1 -- 172.21.15.31:0/1715831173 wait complete. 2024-04-25T16:07:47.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.258+0000 7fb8a9259740 20 client.26561 trim_cache size 0 max 16384 2024-04-25T16:07:47.261 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:07:47.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.286+0000 7fb8a23a8640 1 -- 172.21.15.31:0/923102405 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8840afd20 msgr2=0x7fb8840b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.286+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8840afd20 0x7fb8840b2230 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb8940059c0 tx=0x7fb894005b40 comp rx=0 tx=0).stop 2024-04-25T16:07:47.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.286+0000 7fb8a23a8640 1 -- 172.21.15.31:0/923102405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb89c110030 msgr2=0x7fb89c07ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.286+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb89c110030 0x7fb89c07ae00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fb89400eb10 tx=0x7fb89400ef10 comp rx=0 tx=0).stop 2024-04-25T16:07:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.292+0000 7fb8a23a8640 1 -- 172.21.15.31:0/923102405 shutdown_connections 2024-04-25T16:07:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.292+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fb8840afd20 0x7fb8840b2230 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.292+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb89c110030 0x7fb89c07ae00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.292+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb89c10b310 0x7fb89c1a9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.292+0000 7fb8a23a8640 1 --2- 172.21.15.31:0/923102405 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb89c10a420 0x7fb89c1a9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.292+0000 7fb8a23a8640 1 -- 172.21.15.31:0/923102405 >> 172.21.15.31:0/923102405 conn(0x7fb89c0704e0 msgr2=0x7fb89c10c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:47.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.292+0000 7fb8a23a8640 1 -- 172.21.15.31:0/923102405 shutdown_connections 2024-04-25T16:07:47.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.292+0000 7fb8a23a8640 1 -- 172.21.15.31:0/923102405 wait complete. 2024-04-25T16:07:47.346 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:07:47.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:47 smithi031 ceph-mon[29140]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 740 MiB/s wr, 1.65k op/s 2024-04-25T16:07:47.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:47 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T16:07:47.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:47 smithi031 ceph-mon[29140]: Unable to set osd_memory_target on smithi031 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T16:07:47.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:07:47.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:07:47.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:07:47.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.867+0000 7f35a32de640 1 -- 172.21.15.31:0/1202595641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10a340 msgr2=0x7f359c10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.867+0000 7f35a32de640 1 --2- 172.21.15.31:0/1202595641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10a340 0x7f359c10ac50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f358c009900 tx=0x7f358c02a440 comp rx=0 tx=0).stop 2024-04-25T16:07:47.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:47 smithi063 ceph-mon[32413]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 740 MiB/s wr, 1.65k op/s 2024-04-25T16:07:47.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:47 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T16:07:47.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:47 smithi063 ceph-mon[32413]: Unable to set osd_memory_target on smithi031 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T16:07:47.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:07:47.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:07:47.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:07:47.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.869+0000 7f35a32de640 1 -- 172.21.15.31:0/1202595641 shutdown_connections 2024-04-25T16:07:47.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.869+0000 7f35a32de640 1 --2- 172.21.15.31:0/1202595641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359c10fe90 0x7f359c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.869+0000 7f35a32de640 1 --2- 172.21.15.31:0/1202595641 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f359c10b190 0x7f359c10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.869+0000 7f35a32de640 1 --2- 172.21.15.31:0/1202595641 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10a340 0x7f359c10ac50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.869+0000 7f35a32de640 1 -- 172.21.15.31:0/1202595641 >> 172.21.15.31:0/1202595641 conn(0x7f359c070320 msgr2=0x7f359c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:47 smithi186 ceph-mon[33258]: pgmap v545: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 740 MiB/s wr, 1.65k op/s 2024-04-25T16:07:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:47 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi031 to 177.2M 2024-04-25T16:07:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:47 smithi186 ceph-mon[33258]: Unable to set osd_memory_target on smithi031 to 185897062: error parsing value: Value '185897062' is below minimum 939524096 2024-04-25T16:07:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:07:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:07:47.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:07:47.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.902+0000 7f35a32de640 1 -- 172.21.15.31:0/1202595641 shutdown_connections 2024-04-25T16:07:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.902+0000 7f35a32de640 1 -- 172.21.15.31:0/1202595641 wait complete. 2024-04-25T16:07:47.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.903+0000 7f35a32de640 1 Processor -- start 2024-04-25T16:07:47.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.903+0000 7f35a32de640 1 -- start start 2024-04-25T16:07:47.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.904+0000 7f35a32de640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359c10a340 0x7f359c1a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.904+0000 7f35a32de640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f359c10b190 0x7f359c1a9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.904+0000 7f35a32de640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10fe90 0x7f359c1b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.904+0000 7f35a32de640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f359c11e2e0 con 0x7f359c10b190 2024-04-25T16:07:47.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.904+0000 7f35a32de640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f359c11e160 con 0x7f359c10fe90 2024-04-25T16:07:47.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.904+0000 7f35a32de640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f359c11e460 con 0x7f359c10a340 2024-04-25T16:07:47.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.905+0000 7f35a2add640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10fe90 0x7f359c1b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.905+0000 7f35a2add640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10fe90 0x7f359c1b02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48252/0 (socket says 172.21.15.31:48252) 2024-04-25T16:07:47.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.905+0000 7f35a2add640 1 -- 172.21.15.31:0/594717402 learned_addr learned my addr 172.21.15.31:0/594717402 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:47.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.906+0000 7f35a22dc640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359c10a340 0x7f359c1a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.906+0000 7f35a2add640 1 -- 172.21.15.31:0/594717402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359c10a340 msgr2=0x7f359c1a8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.906+0000 7f35a2add640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359c10a340 0x7f359c1a8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.906+0000 7f35a2add640 1 -- 172.21.15.31:0/594717402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f359c10b190 msgr2=0x7f359c1a9360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:07:47.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.906+0000 7f35a2add640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f359c10b190 0x7f359c1a9360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.906+0000 7f35a2add640 1 -- 172.21.15.31:0/594717402 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f359c111c90 con 0x7f359c10fe90 2024-04-25T16:07:47.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.907+0000 7f35a2add640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10fe90 0x7f359c1b02f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f359400cad0 tx=0x7f359400cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.910+0000 7f35937fe640 1 -- 172.21.15.31:0/594717402 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359400ae20 con 0x7f359c10fe90 2024-04-25T16:07:47.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.911+0000 7f35a32de640 1 -- 172.21.15.31:0/594717402 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f359c111f80 con 0x7f359c10fe90 2024-04-25T16:07:47.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.911+0000 7f35a32de640 1 -- 172.21.15.31:0/594717402 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f359c1124d0 con 0x7f359c10fe90 2024-04-25T16:07:47.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.926+0000 7f35ab3f1740 1 Processor -- start 2024-04-25T16:07:47.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.927+0000 7f35ab3f1740 1 -- start start 2024-04-25T16:07:47.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.928+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d669e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.928+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55932199b4d0 0x559321d66f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.929+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 0x559321d67bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.929+0000 7f35ab3f1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559321ccad80 con 0x55932199b4d0 2024-04-25T16:07:47.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.929+0000 7f35ab3f1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559321c84a20 con 0x5593219c4390 2024-04-25T16:07:47.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.929+0000 7f35ab3f1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559321cc6d40 con 0x5593219a36b0 2024-04-25T16:07:47.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.930+0000 7f35937fe640 1 -- 172.21.15.31:0/594717402 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3594005020 con 0x7f359c10fe90 2024-04-25T16:07:47.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.930+0000 7f35937fe640 1 -- 172.21.15.31:0/594717402 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359400f6b0 con 0x7f359c10fe90 2024-04-25T16:07:47.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.930+0000 7f35937fe640 1 -- 172.21.15.31:0/594717402 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f359400f910 con 0x7f359c10fe90 2024-04-25T16:07:47.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.932+0000 7f35a1adb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.932+0000 7f35a1adb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 0x559321d67bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.932+0000 7f35a1adb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d669e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48266/0 (socket says 172.21.15.31:48266) 2024-04-25T16:07:47.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.932+0000 7f35a1adb640 1 -- 172.21.15.31:0/4249620094 learned_addr learned my addr 172.21.15.31:0/4249620094 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:47.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.941+0000 7f35a22dc640 1 --2- 172.21.15.31:0/4249620094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55932199b4d0 0x559321d66f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.943+0000 7f35a1adb640 1 -- 172.21.15.31:0/4249620094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 msgr2=0x559321d669e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:47.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.943+0000 7f35a1adb640 1 --2- 172.21.15.31:0/4249620094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d669e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.944+0000 7f35a1adb640 1 -- 172.21.15.31:0/4249620094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55932199b4d0 msgr2=0x559321d66f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:47.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.944+0000 7f35a1adb640 1 --2- 172.21.15.31:0/4249620094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55932199b4d0 0x559321d66f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.944+0000 7f35a1adb640 1 -- 172.21.15.31:0/4249620094 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559321c8fc30 con 0x5593219a36b0 2024-04-25T16:07:47.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.944+0000 7f35937fe640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f35880afd20 0x7f35880b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.945+0000 7f35a1adb640 1 --2- 172.21.15.31:0/4249620094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 0x559321d67bf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3598014770 tx=0x7f35980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=291cb9cd9563506 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.945+0000 7f35937fe640 1 -- 172.21.15.31:0/594717402 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f35940dfc00 con 0x7f359c10fe90 2024-04-25T16:07:47.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.948+0000 7f3590bfe640 1 -- 172.21.15.31:0/4249620094 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3598005220 con 0x5593219a36b0 2024-04-25T16:07:47.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.949+0000 7f3590bfe640 1 -- 172.21.15.31:0/4249620094 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3598004c50 con 0x5593219a36b0 2024-04-25T16:07:47.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.951+0000 7f35ab3f1740 1 -- 172.21.15.31:0/4249620094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 msgr2=0x559321d67bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.951+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/4249620094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 0x559321d67bf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3598014770 tx=0x7f35980046c0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.951+0000 7f35ab3f1740 1 -- 172.21.15.31:0/4249620094 shutdown_connections 2024-04-25T16:07:47.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.951+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/4249620094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 0x559321d67bf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.951+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/4249620094 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55932199b4d0 0x559321d66f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.951+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/4249620094 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d669e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.951+0000 7f35ab3f1740 1 -- 172.21.15.31:0/4249620094 >> 172.21.15.31:0/4249620094 conn(0x5593219b84e0 msgr2=0x559321d621f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:47.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.954+0000 7f35a22dc640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f35880afd20 0x7f35880b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.966+0000 7f35ab3f1740 1 -- 172.21.15.31:0/4249620094 shutdown_connections 2024-04-25T16:07:47.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.966+0000 7f35a22dc640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f35880afd20 0x7f35880b2230 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f358c006010 tx=0x7f358c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.967+0000 7f35ab3f1740 1 -- 172.21.15.31:0/4249620094 wait complete. 2024-04-25T16:07:47.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.967+0000 7f35ab3f1740 1 -- start start 2024-04-25T16:07:47.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.967+0000 7f35ab3f1740 1 Processor -- start 2024-04-25T16:07:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.968+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d707d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.968+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 0x559321d70d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.968+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219c4390 0x559321d77c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.968+0000 7f35ab3f1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559321d68750 con 0x5593219c4390 2024-04-25T16:07:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.968+0000 7f35ab3f1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559321c51dc0 con 0x55932199b4d0 2024-04-25T16:07:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.968+0000 7f35ab3f1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559321d688d0 con 0x5593219a36b0 2024-04-25T16:07:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.968+0000 7f35ab3f1740 10 client.0 _ll_register_callbacks cb 0x559321c6d600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.969+0000 7f35a2add640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219c4390 0x559321d77c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.969+0000 7f35a2add640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219c4390 0x559321d77c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60514/0 (socket says 172.21.15.31:60514) 2024-04-25T16:07:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.969+0000 7f35a2add640 1 -- 172.21.15.31:0/2767064498 learned_addr learned my addr 172.21.15.31:0/2767064498 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.969+0000 7f35a22dc640 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d707d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.970+0000 7f35a22dc640 1 -- 172.21.15.31:0/2767064498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 msgr2=0x559321d70d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:47.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.970+0000 7f35a22dc640 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 0x559321d70d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.970+0000 7f35a22dc640 1 -- 172.21.15.31:0/2767064498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219c4390 msgr2=0x559321d77c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:47.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.970+0000 7f35a22dc640 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219c4390 0x559321d77c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:47.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.970+0000 7f35a22dc640 1 -- 172.21.15.31:0/2767064498 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5593219dc3f0 con 0x55932199b4d0 2024-04-25T16:07:47.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.971+0000 7f35a22dc640 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d707d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f358c009850 tx=0x7f358c02a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdb8e2bb737a4c7f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:47.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.975+0000 7f3585ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:47.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.976+0000 7f3585ffb640 1 -- 172.21.15.31:0/2767064498 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c04c020 con 0x55932199b4d0 2024-04-25T16:07:47.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.976+0000 7f35ab3f1740 1 -- 172.21.15.31:0/2767064498 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3598004c50 con 0x55932199b4d0 2024-04-25T16:07:47.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.976+0000 7f3585ffb640 1 -- 172.21.15.31:0/2767064498 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f358c004260 con 0x55932199b4d0 2024-04-25T16:07:47.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.982+0000 7f3585ffb640 1 -- 172.21.15.31:0/2767064498 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f358c035050 con 0x55932199b4d0 2024-04-25T16:07:47.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.982+0000 7f35ab3f1740 10 client.26573 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:47.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.982+0000 7f35ab3f1740 1 -- 172.21.15.31:0/2767064498 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559321aa67d0 con 0x55932199b4d0 2024-04-25T16:07:47.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.984+0000 7f3585ffb640 1 -- 172.21.15.31:0/2767064498 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3598004c50 con 0x55932199b4d0 2024-04-25T16:07:47.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.986+0000 7f3585ffb640 1 -- 172.21.15.31:0/2767064498 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f358c003700 con 0x55932199b4d0 2024-04-25T16:07:47.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.986+0000 7f35ab3f1740 10 client.26573 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:47.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.986+0000 7f35ab3f1740 10 client.26573 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:07:47.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.986+0000 7f35ab3f1740 20 client.26573 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:47.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.986+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x559321982f60 0x559321d7e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:47.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.986+0000 7f35ab3f1740 4 client.26573 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:47.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.986+0000 7f35ab3f1740 1 -- 172.21.15.31:0/2767064498 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559321cd8990 con 0x559321982f60 2024-04-25T16:07:47.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.987+0000 7f35a1adb640 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x559321982f60 0x559321d7e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:47.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.989+0000 7f35a1adb640 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x559321982f60 0x559321d7e960 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=35021462e309474a server_cookie=fe49a13d2eceeb40 in_seq=0 out_seq=0 2024-04-25T16:07:47.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.989+0000 7f3585ffb640 10 client.26573 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:07:47.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.990+0000 7f3585ffb640 1 -- 172.21.15.31:0/2767064498 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559321cd8990 con 0x559321982f60 2024-04-25T16:07:47.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.991+0000 7f3585ffb640 10 client.26573 handle_command_reply: tid=0 2024-04-25T16:07:47.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.991+0000 7f35ab3f1740 1 client.26573 shutdown 2024-04-25T16:07:47.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.994+0000 7f35ab3f1740 1 -- 172.21.15.31:0/2767064498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 msgr2=0x559321d707d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:47.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.994+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d707d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f358c009850 tx=0x7f358c02a440 comp rx=0 tx=0).stop 2024-04-25T16:07:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.999+0000 7f35ab3f1740 1 -- 172.21.15.31:0/2767064498 shutdown_connections 2024-04-25T16:07:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.999+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x559321982f60 0x559321d7e960 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.999+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219c4390 0x559321d77c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.999+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219a36b0 0x559321d70d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.999+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/2767064498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d707d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:47.999+0000 7f35ab3f1740 1 -- 172.21.15.31:0/2767064498 >> 172.21.15.31:0/2767064498 conn(0x5593219b84e0 msgr2=0x559321d65100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:48.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.000+0000 7f3585ffb640 0 client.26573 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:07:48.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.005+0000 7f35ab3f1740 1 -- 172.21.15.31:0/2767064498 shutdown_connections 2024-04-25T16:07:48.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.005+0000 7f35ab3f1740 1 -- 172.21.15.31:0/2767064498 wait complete. 2024-04-25T16:07:48.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.011+0000 7f35ab3f1740 20 client.26573 trim_cache size 0 max 16384 2024-04-25T16:07:48.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.068+0000 7f35ab3f1740 1 Processor -- start 2024-04-25T16:07:48.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.076+0000 7f35ab3f1740 1 -- start start 2024-04-25T16:07:48.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.076+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d70a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.076+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 0x559321d64ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.076+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219c4390 0x559321d65530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.076+0000 7f35ab3f1740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f358000e400 con 0x5593219a36b0 2024-04-25T16:07:48.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.076+0000 7f35ab3f1740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f358000ff40 con 0x55932199b4d0 2024-04-25T16:07:48.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.076+0000 7f35ab3f1740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f358000ead0 con 0x5593219c4390 2024-04-25T16:07:48.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.079+0000 7f35a22dc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d70a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.079+0000 7f35a22dc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d70a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48272/0 (socket says 172.21.15.31:48272) 2024-04-25T16:07:48.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.079+0000 7f35a22dc640 1 -- 172.21.15.31:0/502660061 learned_addr learned my addr 172.21.15.31:0/502660061 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:48.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.079+0000 7f35a2add640 1 --2- 172.21.15.31:0/502660061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219c4390 0x559321d65530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.087+0000 7f35a1adb640 1 --2- 172.21.15.31:0/502660061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 0x559321d64ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.090+0000 7f35a22dc640 1 -- 172.21.15.31:0/502660061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219c4390 msgr2=0x559321d65530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.090+0000 7f35a22dc640 1 --2- 172.21.15.31:0/502660061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219c4390 0x559321d65530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.090+0000 7f35a22dc640 1 -- 172.21.15.31:0/502660061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 msgr2=0x559321d64ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.090+0000 7f35a22dc640 1 --2- 172.21.15.31:0/502660061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 0x559321d64ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.090+0000 7f35a22dc640 1 -- 172.21.15.31:0/502660061 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559321d53ba0 con 0x55932199b4d0 2024-04-25T16:07:48.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.101+0000 7f35a22dc640 1 --2- 172.21.15.31:0/502660061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d70a70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f358c055040 tx=0x7f358c031650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d58fd32154a097bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.106+0000 7f3587fff640 1 -- 172.21.15.31:0/502660061 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c04c070 con 0x55932199b4d0 2024-04-25T16:07:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.106+0000 7f3587fff640 1 -- 172.21.15.31:0/502660061 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f358c003e80 con 0x55932199b4d0 2024-04-25T16:07:48.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.106+0000 7f3587fff640 1 -- 172.21.15.31:0/502660061 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f358c004b70 con 0x55932199b4d0 2024-04-25T16:07:48.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.109+0000 7f35ab3f1740 1 -- 172.21.15.31:0/502660061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 msgr2=0x559321d70a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.109+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/502660061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d70a70 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f358c055040 tx=0x7f358c031650 comp rx=0 tx=0).stop 2024-04-25T16:07:48.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.116+0000 7f35ab3f1740 1 -- 172.21.15.31:0/502660061 shutdown_connections 2024-04-25T16:07:48.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.116+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/502660061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5593219c4390 0x559321d65530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.116+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/502660061 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 0x559321d64ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.116+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/502660061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55932199b4d0 0x559321d70a70 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.116+0000 7f35ab3f1740 1 -- 172.21.15.31:0/502660061 >> 172.21.15.31:0/502660061 conn(0x5593219b84e0 msgr2=0x559321d6b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:48.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.120+0000 7f35ab3f1740 1 -- 172.21.15.31:0/502660061 shutdown_connections 2024-04-25T16:07:48.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.120+0000 7f35ab3f1740 1 -- 172.21.15.31:0/502660061 wait complete. 2024-04-25T16:07:48.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.121+0000 7f35ab3f1740 1 -- start start 2024-04-25T16:07:48.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.122+0000 7f35ab3f1740 1 Processor -- start 2024-04-25T16:07:48.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.122+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55932199b4d0 0x559321d77280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.123+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 0x559321d73e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.123+0000 7f35ab3f1740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d74400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.123+0000 7f35a22dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55932199b4d0 0x559321d77280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.123+0000 7f35a22dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55932199b4d0 0x559321d77280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53986/0 (socket says 172.21.15.31:53986) 2024-04-25T16:07:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.123+0000 7f35a22dc640 1 -- 172.21.15.31:0/1268893843 learned_addr learned my addr 172.21.15.31:0/1268893843 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.123+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f358000f870 con 0x5593219a36b0 2024-04-25T16:07:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.124+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f358000f1a0 con 0x5593219c4390 2024-04-25T16:07:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.124+0000 7f35a2add640 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d74400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.124+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f358000c490 con 0x55932199b4d0 2024-04-25T16:07:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.124+0000 7f35ab3f1740 10 client.0 _ll_register_callbacks cb 0x7f358c040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:48.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.132+0000 7f35a1adb640 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 0x559321d73e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.132+0000 7f35a2add640 1 -- 172.21.15.31:0/1268893843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55932199b4d0 msgr2=0x559321d77280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.132+0000 7f35a2add640 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55932199b4d0 0x559321d77280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.133+0000 7f35a2add640 1 -- 172.21.15.31:0/1268893843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 msgr2=0x559321d73e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.133+0000 7f35a2add640 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 0x559321d73e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.133+0000 7f35a2add640 1 -- 172.21.15.31:0/1268893843 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559321de4d30 con 0x5593219c4390 2024-04-25T16:07:48.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.134+0000 7f35a2add640 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d74400 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f359400caa0 tx=0x7f35940dda70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efbba5da522088b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:48.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.139+0000 7f3585ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:48.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.139+0000 7f3585ffb640 1 -- 172.21.15.31:0/1268893843 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359401d040 con 0x5593219c4390 2024-04-25T16:07:48.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.139+0000 7f3585ffb640 1 -- 172.21.15.31:0/1268893843 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f35940ddb70 con 0x5593219c4390 2024-04-25T16:07:48.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.141+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f358c003e80 con 0x5593219c4390 2024-04-25T16:07:48.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.145+0000 7f3585ffb640 1 -- 172.21.15.31:0/1268893843 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f35940ac4c0 con 0x5593219c4390 2024-04-25T16:07:48.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.148+0000 7f35ab3f1740 10 client.26579 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:48.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.148+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559321de5fe0 con 0x5593219c4390 2024-04-25T16:07:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.154+0000 7f3585ffb640 1 -- 172.21.15.31:0/1268893843 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35940c2070 con 0x5593219c4390 2024-04-25T16:07:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.154+0000 7f3585ffb640 1 -- 172.21.15.31:0/1268893843 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f35940a6f80 con 0x5593219c4390 2024-04-25T16:07:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.157+0000 7f35ab3f1740 10 client.26579 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:48.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.157+0000 7f35ab3f1740 10 client.26579 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:07:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.157+0000 7f35ab3f1740 20 client.26579 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.157+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5593219583a0 0x559321de8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.157+0000 7f35ab3f1740 4 client.26579 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:07:48.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.157+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f358c003e80 con 0x5593219583a0 2024-04-25T16:07:48.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.160+0000 7f35a22dc640 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5593219583a0 0x559321de8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.171+0000 7f3585ffb640 10 client.26579 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:07:48.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.171+0000 7f35a22dc640 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5593219583a0 0x559321de8950 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=69d5a505eb357a8a server_cookie=ebcbd35df617f72d in_seq=0 out_seq=0 2024-04-25T16:07:48.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.176+0000 7f3585ffb640 1 -- 172.21.15.31:0/1268893843 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f358c003e80 con 0x5593219583a0 2024-04-25T16:07:48.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.176+0000 7f3585ffb640 10 client.26579 handle_command_reply: tid=0 2024-04-25T16:07:48.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.179+0000 7f35ab3f1740 1 client.26579 shutdown 2024-04-25T16:07:48.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.186+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 msgr2=0x559321d74400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.186+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d74400 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f359400caa0 tx=0x7f35940dda70 comp rx=0 tx=0).stop 2024-04-25T16:07:48.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.189+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 shutdown_connections 2024-04-25T16:07:48.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.189+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5593219c4390 0x559321d74400 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.190+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5593219583a0 0x559321de8950 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.190+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5593219a36b0 0x559321d73e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.190+0000 7f3585ffb640 0 client.26579 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:07:48.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.190+0000 7f35ab3f1740 1 --2- 172.21.15.31:0/1268893843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55932199b4d0 0x559321d77280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.190+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 >> 172.21.15.31:0/1268893843 conn(0x5593219b84e0 msgr2=0x559321d64ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:48.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.190+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 shutdown_connections 2024-04-25T16:07:48.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.190+0000 7f35ab3f1740 1 -- 172.21.15.31:0/1268893843 wait complete. 2024-04-25T16:07:48.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.193+0000 7f35ab3f1740 20 client.26579 trim_cache size 0 max 16384 2024-04-25T16:07:48.199 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:07:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.201+0000 7f35a32de640 1 -- 172.21.15.31:0/594717402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f35880afd20 msgr2=0x7f35880b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.201+0000 7f35a32de640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f35880afd20 0x7f35880b2230 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f358c006010 tx=0x7f358c031400 comp rx=0 tx=0).stop 2024-04-25T16:07:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.201+0000 7f35a32de640 1 -- 172.21.15.31:0/594717402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10fe90 msgr2=0x7f359c1b02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.201+0000 7f35a32de640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10fe90 0x7f359c1b02f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f359400cad0 tx=0x7f359400cf90 comp rx=0 tx=0).stop 2024-04-25T16:07:48.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.201+0000 7f35a32de640 1 -- 172.21.15.31:0/594717402 shutdown_connections 2024-04-25T16:07:48.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.201+0000 7f35a32de640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f35880afd20 0x7f35880b2230 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.202+0000 7f35a32de640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f359c10fe90 0x7f359c1b02f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.202+0000 7f35a32de640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f359c10b190 0x7f359c1a9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.202+0000 7f35a32de640 1 --2- 172.21.15.31:0/594717402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359c10a340 0x7f359c1a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.202+0000 7f35a32de640 1 -- 172.21.15.31:0/594717402 >> 172.21.15.31:0/594717402 conn(0x7f359c070320 msgr2=0x7f359c10be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.202+0000 7f35a32de640 1 -- 172.21.15.31:0/594717402 shutdown_connections 2024-04-25T16:07:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.202+0000 7f35a32de640 1 -- 172.21.15.31:0/594717402 wait complete. 2024-04-25T16:07:48.243 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:07:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.741+0000 7f3132422640 1 -- 172.21.15.31:0/3988424159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312c10fe90 msgr2=0x7f312c1102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.741+0000 7f3132422640 1 --2- 172.21.15.31:0/3988424159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312c10fe90 0x7f312c1102a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f311c009900 tx=0x7f311c02a410 comp rx=0 tx=0).stop 2024-04-25T16:07:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.743+0000 7f3132422640 1 -- 172.21.15.31:0/3988424159 shutdown_connections 2024-04-25T16:07:48.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.743+0000 7f3132422640 1 --2- 172.21.15.31:0/3988424159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312c10fe90 0x7f312c1102a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.743+0000 7f3132422640 1 --2- 172.21.15.31:0/3988424159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f312c10b190 0x7f312c10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.743+0000 7f3132422640 1 --2- 172.21.15.31:0/3988424159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f312c10a340 0x7f312c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.743+0000 7f3132422640 1 -- 172.21.15.31:0/3988424159 >> 172.21.15.31:0/3988424159 conn(0x7f312c070320 msgr2=0x7f312c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:48.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.752+0000 7f3132422640 1 -- 172.21.15.31:0/3988424159 shutdown_connections 2024-04-25T16:07:48.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.758+0000 7f3132422640 1 -- 172.21.15.31:0/3988424159 wait complete. 2024-04-25T16:07:48.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.760+0000 7f3132422640 1 Processor -- start 2024-04-25T16:07:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.761+0000 7f3132422640 1 -- start start 2024-04-25T16:07:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.761+0000 7f3132422640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f312c10a340 0x7f312c1a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.761+0000 7f3132422640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f312c10b190 0x7f312c1a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.761+0000 7f3132422640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312c10fe90 0x7f312c1b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.761+0000 7f3132422640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f312c11e260 con 0x7f312c10b190 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.761+0000 7f3132422640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f312c11e0e0 con 0x7f312c10a340 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.761+0000 7f3132422640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f312c11e3e0 con 0x7f312c10fe90 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.762+0000 7f3131c21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312c10fe90 0x7f312c1b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.762+0000 7f3131420640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f312c10a340 0x7f312c1a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.762+0000 7f3131c21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312c10fe90 0x7f312c1b0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54006/0 (socket says 172.21.15.31:54006) 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.762+0000 7f3131c21640 1 -- 172.21.15.31:0/3627589823 learned_addr learned my addr 172.21.15.31:0/3627589823 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.763+0000 7f3130c1f640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f312c10b190 0x7f312c1a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.764+0000 7f3130c1f640 1 -- 172.21.15.31:0/3627589823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312c10fe90 msgr2=0x7f312c1b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.764+0000 7f3130c1f640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312c10fe90 0x7f312c1b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.764+0000 7f3130c1f640 1 -- 172.21.15.31:0/3627589823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f312c10a340 msgr2=0x7f312c1a8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.764+0000 7f3130c1f640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f312c10a340 0x7f312c1a8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.764+0000 7f3130c1f640 1 -- 172.21.15.31:0/3627589823 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312c111c90 con 0x7f312c10b190 2024-04-25T16:07:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.764+0000 7f3130c1f640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f312c10b190 0x7f312c1a93c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f312800d6d0 tx=0x7f312800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.767+0000 7f31227fc640 1 -- 172.21.15.31:0/3627589823 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128039070 con 0x7f312c10b190 2024-04-25T16:07:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.767+0000 7f3132422640 1 -- 172.21.15.31:0/3627589823 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f312c111f80 con 0x7f312c10b190 2024-04-25T16:07:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.767+0000 7f3132422640 1 -- 172.21.15.31:0/3627589823 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f312c1124d0 con 0x7f312c10b190 2024-04-25T16:07:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.767+0000 7f31227fc640 1 -- 172.21.15.31:0/3627589823 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3128009820 con 0x7f312c10b190 2024-04-25T16:07:48.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.769+0000 7f31227fc640 1 -- 172.21.15.31:0/3627589823 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128004b90 con 0x7f312c10b190 2024-04-25T16:07:48.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.776+0000 7f313a608740 1 Processor -- start 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.776+0000 7f313a608740 1 -- start start 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.776+0000 7f31227fc640 1 -- 172.21.15.31:0/3627589823 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3128005220 con 0x7f312c10b190 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.776+0000 7f313a608740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be58659e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.776+0000 7f313a608740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5865f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.776+0000 7f313a608740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be5866bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.776+0000 7f313a608740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562be57c9d80 con 0x562be54c3390 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.776+0000 7f313a608740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562be5783a20 con 0x562be54a26b0 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.776+0000 7f313a608740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562be57c5d40 con 0x562be549a4d0 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.777+0000 7f3131c21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be58659e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.777+0000 7f3131c21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be58659e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60554/0 (socket says 172.21.15.31:60554) 2024-04-25T16:07:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.777+0000 7f3131c21640 1 -- 172.21.15.31:0/1735934438 learned_addr learned my addr 172.21.15.31:0/1735934438 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.777+0000 7f3131420640 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5865f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.778+0000 7f3130c1f640 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be5866bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.778+0000 7f3131c21640 1 -- 172.21.15.31:0/1735934438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 msgr2=0x562be5865f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.778+0000 7f3131c21640 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5865f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.778+0000 7f3131c21640 1 -- 172.21.15.31:0/1735934438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 msgr2=0x562be5866bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.778+0000 7f3131c21640 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be5866bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.778+0000 7f3131c21640 1 -- 172.21.15.31:0/1735934438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be578ec30 con 0x562be54c3390 2024-04-25T16:07:48.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.778+0000 7f3130c1f640 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be5866bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:07:48.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.782+0000 7f3131c21640 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be58659e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f311c004160 tx=0x7f311c004190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7879ba8b2102a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:48.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.783+0000 7f31227fc640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31180afcf0 0x7f31180b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.785+0000 7f31227fc640 1 -- 172.21.15.31:0/3627589823 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f31281006f0 con 0x7f312c10b190 2024-04-25T16:07:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.786+0000 7f31177fe640 1 -- 172.21.15.31:0/1735934438 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c035030 con 0x562be54c3390 2024-04-25T16:07:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.786+0000 7f31177fe640 1 -- 172.21.15.31:0/1735934438 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f311c0054f0 con 0x562be54c3390 2024-04-25T16:07:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.786+0000 7f31177fe640 1 -- 172.21.15.31:0/1735934438 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c0035f0 con 0x562be54c3390 2024-04-25T16:07:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.788+0000 7f313a608740 1 -- 172.21.15.31:0/1735934438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 msgr2=0x562be58659e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.788+0000 7f313a608740 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be58659e0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f311c004160 tx=0x7f311c004190 comp rx=0 tx=0).stop 2024-04-25T16:07:48.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.790+0000 7f3131420640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31180afcf0 0x7f31180b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.793+0000 7f3131420640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31180afcf0 0x7f31180b2200 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x562be5865700 tx=0x7f3124009580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.795+0000 7f313a608740 1 -- 172.21.15.31:0/1735934438 shutdown_connections 2024-04-25T16:07:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.795+0000 7f313a608740 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be5866bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.795+0000 7f313a608740 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5865f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.795+0000 7f313a608740 1 --2- 172.21.15.31:0/1735934438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be58659e0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.795+0000 7f313a608740 1 -- 172.21.15.31:0/1735934438 >> 172.21.15.31:0/1735934438 conn(0x562be54b74e0 msgr2=0x562be58611f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:48.800 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.798+0000 7f313a608740 1 -- 172.21.15.31:0/1735934438 shutdown_connections 2024-04-25T16:07:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.798+0000 7f313a608740 1 -- 172.21.15.31:0/1735934438 wait complete. 2024-04-25T16:07:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.799+0000 7f313a608740 1 -- start start 2024-04-25T16:07:48.801 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.799+0000 7f313a608740 1 Processor -- start 2024-04-25T16:07:48.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.803+0000 7f313a608740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be586f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.803+0000 7f313a608740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be586fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.803+0000 7f313a608740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5876ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.803+0000 7f313a608740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562be58676a0 con 0x562be54a26b0 2024-04-25T16:07:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.803+0000 7f313a608740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x562be5750dc0 con 0x562be54c3390 2024-04-25T16:07:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.803+0000 7f313a608740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562be5867820 con 0x562be549a4d0 2024-04-25T16:07:48.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.803+0000 7f313a608740 10 client.0 _ll_register_callbacks cb 0x562be576c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:48.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.805+0000 7f3131420640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be586f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.805+0000 7f3130c1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be586fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.805+0000 7f3130c1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be586fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60562/0 (socket says 172.21.15.31:60562) 2024-04-25T16:07:48.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.805+0000 7f3130c1f640 1 -- 172.21.15.31:0/2148265546 learned_addr learned my addr 172.21.15.31:0/2148265546 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:48.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.805+0000 7f3131c21640 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5876ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.806+0000 7f3131c21640 1 -- 172.21.15.31:0/2148265546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 msgr2=0x562be586f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.806+0000 7f3131c21640 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be586f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.806+0000 7f3131c21640 1 -- 172.21.15.31:0/2148265546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 msgr2=0x562be586fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.806+0000 7f3131c21640 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be586fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.806+0000 7f3131c21640 1 -- 172.21.15.31:0/2148265546 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be54db3f0 con 0x562be54c3390 2024-04-25T16:07:48.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.807+0000 7f3131c21640 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5876ce0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f311c009850 tx=0x7f311c02a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57468c135413d842 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.808+0000 7f3114ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:48.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.808+0000 7f3114ff9640 1 -- 172.21.15.31:0/2148265546 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c051060 con 0x562be54c3390 2024-04-25T16:07:48.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.808+0000 7f313a608740 1 -- 172.21.15.31:0/2148265546 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f311c0054f0 con 0x562be54c3390 2024-04-25T16:07:48.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.808+0000 7f3114ff9640 1 -- 172.21.15.31:0/2148265546 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f311c005130 con 0x562be54c3390 2024-04-25T16:07:48.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.808+0000 7f3114ff9640 1 -- 172.21.15.31:0/2148265546 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f311c035040 con 0x562be54c3390 2024-04-25T16:07:48.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.808+0000 7f313a608740 10 client.26588 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:48.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.808+0000 7f313a608740 1 -- 172.21.15.31:0/2148265546 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562be55a57d0 con 0x562be54c3390 2024-04-25T16:07:48.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.810+0000 7f3114ff9640 1 -- 172.21.15.31:0/2148265546 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f311c0054f0 con 0x562be54c3390 2024-04-25T16:07:48.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.812+0000 7f3114ff9640 1 -- 172.21.15.31:0/2148265546 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f311c04b030 con 0x562be54c3390 2024-04-25T16:07:48.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.812+0000 7f313a608740 10 client.26588 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:48.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.812+0000 7f313a608740 10 client.26588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:48.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.812+0000 7f313a608740 20 client.26588 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:48.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.812+0000 7f313a608740 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562be5481f60 0x562be587d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.812+0000 7f313a608740 4 client.26588 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:48.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.812+0000 7f313a608740 1 -- 172.21.15.31:0/2148265546 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562be57d7990 con 0x562be5481f60 2024-04-25T16:07:48.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.818+0000 7f3131420640 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562be5481f60 0x562be587d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.824+0000 7f3131420640 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562be5481f60 0x562be587d0a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=751586b95c2b9e77 server_cookie=23ef260ac39906b1 in_seq=0 out_seq=0 2024-04-25T16:07:48.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.824+0000 7f3114ff9640 10 client.26588 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.828+0000 7f3114ff9640 1 -- 172.21.15.31:0/2148265546 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562be57d7990 con 0x562be5481f60 2024-04-25T16:07:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.828+0000 7f3114ff9640 10 client.26588 handle_command_reply: tid=0 2024-04-25T16:07:48.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.828+0000 7f313a608740 1 client.26588 shutdown 2024-04-25T16:07:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.830+0000 7f313a608740 1 -- 172.21.15.31:0/2148265546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 msgr2=0x562be5876ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.830+0000 7f313a608740 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5876ce0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f311c009850 tx=0x7f311c02a410 comp rx=0 tx=0).stop 2024-04-25T16:07:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.834+0000 7f313a608740 1 -- 172.21.15.31:0/2148265546 shutdown_connections 2024-04-25T16:07:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.835+0000 7f313a608740 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562be5481f60 0x562be587d0a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.835+0000 7f313a608740 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5876ce0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.835+0000 7f313a608740 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be586fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.835+0000 7f313a608740 1 --2- 172.21.15.31:0/2148265546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be586f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.835+0000 7f313a608740 1 -- 172.21.15.31:0/2148265546 >> 172.21.15.31:0/2148265546 conn(0x562be54b74e0 msgr2=0x562be5864100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:48.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.835+0000 7f3114ff9640 0 client.26588 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.836+0000 7f313a608740 1 -- 172.21.15.31:0/2148265546 shutdown_connections 2024-04-25T16:07:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.836+0000 7f313a608740 1 -- 172.21.15.31:0/2148265546 wait complete. 2024-04-25T16:07:48.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.836+0000 7f313a608740 20 client.26588 trim_cache size 0 max 16384 2024-04-25T16:07:48.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.889+0000 7f313a608740 1 Processor -- start 2024-04-25T16:07:48.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.889+0000 7f313a608740 1 -- start start 2024-04-25T16:07:48.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.890+0000 7f313a608740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be586ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.890+0000 7f313a608740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be586eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.890+0000 7f313a608740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be586f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.890+0000 7f3130c1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be586eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.890+0000 7f3130c1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be586eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48324/0 (socket says 172.21.15.31:48324) 2024-04-25T16:07:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.891+0000 7f313a608740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f311000e500 con 0x562be54c3390 2024-04-25T16:07:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.891+0000 7f313a608740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3110010040 con 0x562be54a26b0 2024-04-25T16:07:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.891+0000 7f313a608740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f311000ebd0 con 0x562be549a4d0 2024-04-25T16:07:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.891+0000 7f3130c1f640 1 -- 172.21.15.31:0/3820379870 learned_addr learned my addr 172.21.15.31:0/3820379870 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:48.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.891+0000 7f3131c21640 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be586f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.891+0000 7f3131420640 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be586ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.892+0000 7f3131420640 1 -- 172.21.15.31:0/3820379870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 msgr2=0x562be586eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.892+0000 7f3131420640 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be586eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.892+0000 7f3131420640 1 -- 172.21.15.31:0/3820379870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 msgr2=0x562be586f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.892+0000 7f3131420640 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be586f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.892+0000 7f3131420640 1 -- 172.21.15.31:0/3820379870 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562be58702c0 con 0x562be549a4d0 2024-04-25T16:07:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.892+0000 7f3131420640 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be586ac50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x562be587d640 tx=0x7f3124014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f98557815c6c6fd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.893+0000 7f3116ffd640 1 -- 172.21.15.31:0/3820379870 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312402f070 con 0x562be549a4d0 2024-04-25T16:07:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.893+0000 7f3116ffd640 1 -- 172.21.15.31:0/3820379870 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3124020e30 con 0x562be549a4d0 2024-04-25T16:07:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.893+0000 7f3116ffd640 1 -- 172.21.15.31:0/3820379870 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3124029870 con 0x562be549a4d0 2024-04-25T16:07:48.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.893+0000 7f3130c1f640 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be586eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:07:48.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.894+0000 7f313a608740 1 -- 172.21.15.31:0/3820379870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 msgr2=0x562be586ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:48.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.898+0000 7f313a608740 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be586ac50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x562be587d640 tx=0x7f3124014000 comp rx=0 tx=0).stop 2024-04-25T16:07:48.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.898+0000 7f313a608740 1 -- 172.21.15.31:0/3820379870 shutdown_connections 2024-04-25T16:07:48.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.898+0000 7f313a608740 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54c3390 0x562be586f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.898+0000 7f313a608740 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54a26b0 0x562be586eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.898+0000 7f313a608740 1 --2- 172.21.15.31:0/3820379870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be586ac50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.898+0000 7f313a608740 1 -- 172.21.15.31:0/3820379870 >> 172.21.15.31:0/3820379870 conn(0x562be54b74e0 msgr2=0x562be586def0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.923+0000 7f313a608740 1 -- 172.21.15.31:0/3820379870 shutdown_connections 2024-04-25T16:07:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.924+0000 7f313a608740 1 -- 172.21.15.31:0/3820379870 wait complete. 2024-04-25T16:07:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.924+0000 7f313a608740 1 -- start start 2024-04-25T16:07:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.925+0000 7f313a608740 1 Processor -- start 2024-04-25T16:07:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.925+0000 7f313a608740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5876070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.925+0000 7f313a608740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be5872bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.925+0000 7f313a608740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5873130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.925+0000 7f313a608740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f311000f970 con 0x562be54a26b0 2024-04-25T16:07:48.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.925+0000 7f313a608740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f311000f2a0 con 0x562be54c3390 2024-04-25T16:07:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.925+0000 7f313a608740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f311000c4e0 con 0x562be549a4d0 2024-04-25T16:07:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.926+0000 7f313a608740 10 client.0 _ll_register_callbacks cb 0x7f311c0040b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.927+0000 7f3131420640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5876070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.927+0000 7f3131c21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5873130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.931+0000 7f3131420640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5876070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54038/0 (socket says 172.21.15.31:54038) 2024-04-25T16:07:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.931+0000 7f3131420640 1 -- 172.21.15.31:0/1302793443 learned_addr learned my addr 172.21.15.31:0/1302793443 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:48.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.931+0000 7f3130c1f640 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be5872bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.938+0000 7f3130c1f640 1 -- 172.21.15.31:0/1302793443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 msgr2=0x562be5876070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.938+0000 7f3130c1f640 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5876070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.938+0000 7f3130c1f640 1 -- 172.21.15.31:0/1302793443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 msgr2=0x562be5873130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:48.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.938+0000 7f3130c1f640 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5873130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:48.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.938+0000 7f3130c1f640 1 -- 172.21.15.31:0/1302793443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562be5873770 con 0x562be54a26b0 2024-04-25T16:07:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.939+0000 7f3131c21640 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5873130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:07:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.940+0000 7f3131420640 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5876070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:07:48.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.951+0000 7f3130c1f640 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be5872bf0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f31280c4c00 tx=0x7f3128009740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38873a538a2ea48 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:48.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.956+0000 7f3114ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:07:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.956+0000 7f3114ff9640 1 -- 172.21.15.31:0/1302793443 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128042040 con 0x562be54a26b0 2024-04-25T16:07:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.956+0000 7f3114ff9640 1 -- 172.21.15.31:0/1302793443 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f31280e31e0 con 0x562be54a26b0 2024-04-25T16:07:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.956+0000 7f313a608740 1 -- 172.21.15.31:0/1302793443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3124020e30 con 0x562be54a26b0 2024-04-25T16:07:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.957+0000 7f3114ff9640 1 -- 172.21.15.31:0/1302793443 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f31280c8690 con 0x562be54a26b0 2024-04-25T16:07:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.957+0000 7f313a608740 10 client.16947 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.957+0000 7f313a608740 1 -- 172.21.15.31:0/1302793443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562be58e4260 con 0x562be54a26b0 2024-04-25T16:07:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.957+0000 7f3114ff9640 1 -- 172.21.15.31:0/1302793443 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3124020e30 con 0x562be54a26b0 2024-04-25T16:07:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.960+0000 7f3114ff9640 1 -- 172.21.15.31:0/1302793443 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f31280d1550 con 0x562be54a26b0 2024-04-25T16:07:48.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.960+0000 7f313a608740 10 client.16947 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:48.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.960+0000 7f313a608740 10 client.16947 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:48.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.960+0000 7f313a608740 20 client.16947 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:48.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.960+0000 7f313a608740 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562be54573a0 0x562be58e6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:48.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.960+0000 7f313a608740 4 client.16947 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:07:48.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.960+0000 7f313a608740 1 -- 172.21.15.31:0/1302793443 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562be57d1800 con 0x562be54573a0 2024-04-25T16:07:48.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.965+0000 7f3131420640 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562be54573a0 0x562be58e6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:48.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.995+0000 7f3114ff9640 10 client.16947 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:48.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:48.995+0000 7f3131420640 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562be54573a0 0x562be58e6cf0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=239a89eedf419242 server_cookie=40a666b4d6a95696 in_seq=0 out_seq=0 2024-04-25T16:07:49.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f3114ff9640 1 -- 172.21.15.31:0/1302793443 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562be57d1800 con 0x562be54573a0 2024-04-25T16:07:49.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f3114ff9640 10 client.16947 handle_command_reply: tid=0 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f313a608740 1 client.16947 shutdown 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f313a608740 1 -- 172.21.15.31:0/1302793443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 msgr2=0x562be5872bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f313a608740 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be5872bf0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f31280c4c00 tx=0x7f3128009740 comp rx=0 tx=0).stop 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f313a608740 1 -- 172.21.15.31:0/1302793443 shutdown_connections 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f313a608740 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x562be54573a0 0x562be58e6cf0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f313a608740 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x562be54c3390 0x562be5873130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f313a608740 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x562be54a26b0 0x562be5872bf0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.000+0000 7f313a608740 1 --2- 172.21.15.31:0/1302793443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x562be549a4d0 0x562be5876070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.001+0000 7f313a608740 1 -- 172.21.15.31:0/1302793443 >> 172.21.15.31:0/1302793443 conn(0x562be54b74e0 msgr2=0x562be586eb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.001+0000 7f3114ff9640 0 client.16947 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.001+0000 7f313a608740 1 -- 172.21.15.31:0/1302793443 shutdown_connections 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.001+0000 7f313a608740 1 -- 172.21.15.31:0/1302793443 wait complete. 2024-04-25T16:07:49.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.001+0000 7f313a608740 20 client.16947 trim_cache size 0 max 16384 2024-04-25T16:07:49.006 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:07:49.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.035+0000 7f3132422640 1 -- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31180afcf0 msgr2=0x7f31180b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.035+0000 7f3132422640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31180afcf0 0x7f31180b2200 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x562be5865700 tx=0x7f3124009580 comp rx=0 tx=0).stop 2024-04-25T16:07:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.035+0000 7f3132422640 1 -- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f312c10b190 msgr2=0x7f312c1a93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.035+0000 7f3132422640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f312c10b190 0x7f312c1a93c0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f312800d6d0 tx=0x7f312800db90 comp rx=0 tx=0).stop 2024-04-25T16:07:49.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.036+0000 7f3132422640 1 -- 172.21.15.31:0/3627589823 shutdown_connections 2024-04-25T16:07:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.036+0000 7f3132422640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f31180afcf0 0x7f31180b2200 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.036+0000 7f3132422640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312c10fe90 0x7f312c1b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.036+0000 7f3132422640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f312c10b190 0x7f312c1a93c0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.036+0000 7f3132422640 1 --2- 172.21.15.31:0/3627589823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f312c10a340 0x7f312c1a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.036+0000 7f3132422640 1 -- 172.21.15.31:0/3627589823 >> 172.21.15.31:0/3627589823 conn(0x7f312c070320 msgr2=0x7f312c10be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.042+0000 7f3132422640 1 -- 172.21.15.31:0/3627589823 shutdown_connections 2024-04-25T16:07:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:49.043+0000 7f3132422640 1 -- 172.21.15.31:0/3627589823 wait complete. 2024-04-25T16:07:49.422 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:49 smithi031 ceph-mon[29140]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 873 MiB/s wr, 1.79k op/s 2024-04-25T16:07:49.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:49 smithi063 ceph-mon[32413]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 873 MiB/s wr, 1.79k op/s 2024-04-25T16:07:49.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:49 smithi186 ceph-mon[33258]: pgmap v546: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 873 MiB/s wr, 1.79k op/s 2024-04-25T16:07:50.126 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:07:50.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:07:50.126 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:07:50.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.566+0000 7f899ec42640 1 -- 172.21.15.31:0/253991326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8998110030 msgr2=0x7f8998112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:50.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.566+0000 7f899ec42640 1 --2- 172.21.15.31:0/253991326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8998110030 0x7f8998112450 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f8980009900 tx=0x7f898002a400 comp rx=0 tx=0).stop 2024-04-25T16:07:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.567+0000 7f899ec42640 1 -- 172.21.15.31:0/253991326 shutdown_connections 2024-04-25T16:07:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.567+0000 7f899ec42640 1 --2- 172.21.15.31:0/253991326 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8998110030 0x7f8998112450 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.567+0000 7f899ec42640 1 --2- 172.21.15.31:0/253991326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f899810b330 0x7f899810b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.567+0000 7f899ec42640 1 --2- 172.21.15.31:0/253991326 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f899810a3d0 0x7f899810adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.567+0000 7f899ec42640 1 -- 172.21.15.31:0/253991326 >> 172.21.15.31:0/253991326 conn(0x7f8998078de0 msgr2=0x7f899807b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:50.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.570+0000 7f899ec42640 1 -- 172.21.15.31:0/253991326 shutdown_connections 2024-04-25T16:07:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.574+0000 7f899ec42640 1 -- 172.21.15.31:0/253991326 wait complete. 2024-04-25T16:07:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.575+0000 7f899ec42640 1 Processor -- start 2024-04-25T16:07:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.575+0000 7f899ec42640 1 -- start start 2024-04-25T16:07:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.575+0000 7f899ec42640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f899810a3d0 0x7f8998081a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.575+0000 7f899ec42640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f899810b330 0x7f8998081f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.575+0000 7f899ec42640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8998110030 0x7f8998082e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f899ec42640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8998115c10 con 0x7f899810a3d0 2024-04-25T16:07:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f899ec42640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8998115a90 con 0x7f8998110030 2024-04-25T16:07:50.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f899ec42640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8998115d90 con 0x7f899810b330 2024-04-25T16:07:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f8997fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f899810b330 0x7f8998081f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f8997fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f899810b330 0x7f8998081f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54070/0 (socket says 172.21.15.31:54070) 2024-04-25T16:07:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f8997fff640 1 -- 172.21.15.31:0/393723491 learned_addr learned my addr 172.21.15.31:0/393723491 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f8997fff640 1 -- 172.21.15.31:0/393723491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8998110030 msgr2=0x7f8998082e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f8997fff640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8998110030 0x7f8998082e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f8997fff640 1 -- 172.21.15.31:0/393723491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f899810a3d0 msgr2=0x7f8998081a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f8997fff640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f899810a3d0 0x7f8998081a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.576+0000 7f8997fff640 1 -- 172.21.15.31:0/393723491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89980834d0 con 0x7f899810b330 2024-04-25T16:07:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.577+0000 7f8997fff640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f899810b330 0x7f8998081f70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f899000cce0 tx=0x7f899000a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.581+0000 7f8995ffb640 1 -- 172.21.15.31:0/393723491 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8990004b20 con 0x7f899810b330 2024-04-25T16:07:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.581+0000 7f899ec42640 1 -- 172.21.15.31:0/393723491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89980837c0 con 0x7f899810b330 2024-04-25T16:07:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.582+0000 7f899ec42640 1 -- 172.21.15.31:0/393723491 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8998083d10 con 0x7f899810b330 2024-04-25T16:07:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.582+0000 7f8995ffb640 1 -- 172.21.15.31:0/393723491 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8990004cc0 con 0x7f899810b330 2024-04-25T16:07:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.582+0000 7f8995ffb640 1 -- 172.21.15.31:0/393723491 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f899000f6b0 con 0x7f899810b330 2024-04-25T16:07:50.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.589+0000 7f8995ffb640 1 -- 172.21.15.31:0/393723491 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f899000f8d0 con 0x7f899810b330 2024-04-25T16:07:50.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.592+0000 7f8995ffb640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89840afd00 0x7f89840b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.592+0000 7f89a5bce740 1 Processor -- start 2024-04-25T16:07:50.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.593+0000 7f899c9de640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89840afd00 0x7f89840b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.593+0000 7f89a5bce740 1 -- start start 2024-04-25T16:07:50.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.594+0000 7f8995ffb640 1 -- 172.21.15.31:0/393723491 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f89900dfb90 con 0x7f899810b330 2024-04-25T16:07:50.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.595+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 0x55a8138974a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.595+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 0x55a813897a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.595+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a81389e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.596+0000 7f8997fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 0x55a813897a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.596+0000 7f8997fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 0x55a813897a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54078/0 (socket says 172.21.15.31:54078) 2024-04-25T16:07:50.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.596+0000 7f8997fff640 1 -- 172.21.15.31:0/3702660366 learned_addr learned my addr 172.21.15.31:0/3702660366 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.596+0000 7f89a5bce740 1 -- 172.21.15.31:0/3702660366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a8137fcb00 con 0x55a8134d5730 2024-04-25T16:07:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.596+0000 7f89a5bce740 1 -- 172.21.15.31:0/3702660366 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a813798920 con 0x55a8134f6410 2024-04-25T16:07:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.596+0000 7f89a5bce740 1 -- 172.21.15.31:0/3702660366 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a8137f8ac0 con 0x55a8134cd550 2024-04-25T16:07:50.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.597+0000 7f899d1df640 1 --2- 172.21.15.31:0/3702660366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a81389e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.598+0000 7f899d1df640 1 -- 172.21.15.31:0/3702660366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 msgr2=0x55a813897a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.598+0000 7f899d1df640 1 --2- 172.21.15.31:0/3702660366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 0x55a813897a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.598+0000 7f899d1df640 1 -- 172.21.15.31:0/3702660366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 msgr2=0x55a8138974a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.598+0000 7f899d1df640 1 --2- 172.21.15.31:0/3702660366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 0x55a8138974a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.598+0000 7f899d1df640 1 -- 172.21.15.31:0/3702660366 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8135d67e0 con 0x55a8134d5730 2024-04-25T16:07:50.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.598+0000 7f899d1df640 1 --2- 172.21.15.31:0/3702660366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a81389e920 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8980005f70 tx=0x7f8980005310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f2be53d192f65 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:50.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.602+0000 7f899c9de640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89840afd00 0x7f89840b2210 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8998109bf0 tx=0x7f8988005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:50.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.602+0000 7f898effd640 1 -- 172.21.15.31:0/3702660366 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898003e070 con 0x55a8134d5730 2024-04-25T16:07:50.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.602+0000 7f898effd640 1 -- 172.21.15.31:0/3702660366 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89800046c0 con 0x55a8134d5730 2024-04-25T16:07:50.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.604+0000 7f89a5bce740 1 -- 172.21.15.31:0/3702660366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 msgr2=0x55a81389e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:50.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.604+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3702660366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a81389e920 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f8980005f70 tx=0x7f8980005310 comp rx=0 tx=0).stop 2024-04-25T16:07:50.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.608+0000 7f89a5bce740 1 -- 172.21.15.31:0/3702660366 shutdown_connections 2024-04-25T16:07:50.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.608+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3702660366 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a81389e920 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.608+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3702660366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 0x55a813897a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.608+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3702660366 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 0x55a8138974a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.608+0000 7f89a5bce740 1 -- 172.21.15.31:0/3702660366 >> 172.21.15.31:0/3702660366 conn(0x55a8134ea560 msgr2=0x55a813892e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:50.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.612+0000 7f89a5bce740 1 -- 172.21.15.31:0/3702660366 shutdown_connections 2024-04-25T16:07:50.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.616+0000 7f89a5bce740 1 -- 172.21.15.31:0/3702660366 wait complete. 2024-04-25T16:07:50.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.616+0000 7f89a5bce740 1 -- start start 2024-04-25T16:07:50.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.616+0000 7f89a5bce740 1 Processor -- start 2024-04-25T16:07:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.617+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 0x55a8138a1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.617+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.617+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 0x55a8138a8720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.617+0000 7f89a5bce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a81389f360 con 0x55a8134d5730 2024-04-25T16:07:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.617+0000 7f89a5bce740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a8137965d0 con 0x55a8134f6410 2024-04-25T16:07:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.618+0000 7f89a5bce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a81389f4e0 con 0x55a8134cd550 2024-04-25T16:07:50.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.618+0000 7f89a5bce740 10 client.0 _ll_register_callbacks cb 0x55a813784080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.618+0000 7f899d1df640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 0x55a8138a8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.618+0000 7f899d1df640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 0x55a8138a8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48374/0 (socket says 172.21.15.31:48374) 2024-04-25T16:07:50.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.618+0000 7f899d1df640 1 -- 172.21.15.31:0/3834046609 learned_addr learned my addr 172.21.15.31:0/3834046609 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.619+0000 7f899d1df640 1 -- 172.21.15.31:0/3834046609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 msgr2=0x55a8138a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.619+0000 7f899d1df640 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 0x55a8138a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.619+0000 7f899d1df640 1 -- 172.21.15.31:0/3834046609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 msgr2=0x55a8138a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.619+0000 7f899d1df640 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.619+0000 7f899d1df640 1 -- 172.21.15.31:0/3834046609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a813636f70 con 0x55a8134f6410 2024-04-25T16:07:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.620+0000 7f899d1df640 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 0x55a8138a8720 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8980006010 tx=0x7f8980004870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64b7ce57abba1b59 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:50.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.620+0000 7f897ffff640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:50.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.629+0000 7f897ffff640 1 -- 172.21.15.31:0/3834046609 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898003e070 con 0x55a8134f6410 2024-04-25T16:07:50.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.629+0000 7f897ffff640 1 -- 172.21.15.31:0/3834046609 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8980004af0 con 0x55a8134f6410 2024-04-25T16:07:50.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.630+0000 7f89a5bce740 1 -- 172.21.15.31:0/3834046609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89800046c0 con 0x55a8134f6410 2024-04-25T16:07:50.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.634+0000 7f897ffff640 1 -- 172.21.15.31:0/3834046609 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8980053050 con 0x55a8134f6410 2024-04-25T16:07:50.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.634+0000 7f89a5bce740 10 client.26594 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:50.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.634+0000 7f89a5bce740 1 -- 172.21.15.31:0/3834046609 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8134ce170 con 0x55a8134f6410 2024-04-25T16:07:50.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.636+0000 7f897ffff640 1 -- 172.21.15.31:0/3834046609 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89800046c0 con 0x55a8134f6410 2024-04-25T16:07:50.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.636+0000 7f897ffff640 1 -- 172.21.15.31:0/3834046609 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8980043750 con 0x55a8134f6410 2024-04-25T16:07:50.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.636+0000 7f89a5bce740 10 client.26594 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:50.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.636+0000 7f89a5bce740 10 client.26594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:50.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.636+0000 7f89a5bce740 20 client.26594 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:50.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.636+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a8134b51f0 0x55a8138aee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.636+0000 7f89a5bce740 4 client.26594 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:50.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.636+0000 7f89a5bce740 1 -- 172.21.15.31:0/3834046609 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a81380a710 con 0x55a8134b51f0 2024-04-25T16:07:50.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.637+0000 7f899c9de640 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a8134b51f0 0x55a8138aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.639+0000 7f899c9de640 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a8134b51f0 0x55a8138aee70 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d91d35bab4f8c94 server_cookie=a75ec0eade1e7c27 in_seq=0 out_seq=0 2024-04-25T16:07:50.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.639+0000 7f897ffff640 10 client.26594 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:50.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.640+0000 7f897ffff640 1 -- 172.21.15.31:0/3834046609 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a81380a710 con 0x55a8134b51f0 2024-04-25T16:07:50.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.640+0000 7f897ffff640 10 client.26594 handle_command_reply: tid=0 2024-04-25T16:07:50.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.640+0000 7f89a5bce740 1 client.26594 shutdown 2024-04-25T16:07:50.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.641+0000 7f89a5bce740 1 -- 172.21.15.31:0/3834046609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 msgr2=0x55a8138a8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:50.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.641+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 0x55a8138a8720 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8980006010 tx=0x7f8980004870 comp rx=0 tx=0).stop 2024-04-25T16:07:50.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.641+0000 7f89a5bce740 1 -- 172.21.15.31:0/3834046609 shutdown_connections 2024-04-25T16:07:50.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.641+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a8134b51f0 0x55a8138aee70 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.641+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134f6410 0x55a8138a8720 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.641+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138a17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.641+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3834046609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134cd550 0x55a8138a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.641+0000 7f89a5bce740 1 -- 172.21.15.31:0/3834046609 >> 172.21.15.31:0/3834046609 conn(0x55a8134ea560 msgr2=0x55a813891ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:50.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.641+0000 7f897ffff640 0 client.26594 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:50.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.649+0000 7f89a5bce740 1 -- 172.21.15.31:0/3834046609 shutdown_connections 2024-04-25T16:07:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.668+0000 7f89a5bce740 1 -- 172.21.15.31:0/3834046609 wait complete. 2024-04-25T16:07:50.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.668+0000 7f89a5bce740 20 client.26594 trim_cache size 0 max 16384 2024-04-25T16:07:50.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.781+0000 7f89a5bce740 1 Processor -- start 2024-04-25T16:07:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.785+0000 7f89a5bce740 1 -- start start 2024-04-25T16:07:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.785+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138b3620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.785+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134cd550 0x55a8138a8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.785+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.785+0000 7f89a5bce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f897800e590 con 0x55a8134d5730 2024-04-25T16:07:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.785+0000 7f89a5bce740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f89780100d0 con 0x55a8134cd550 2024-04-25T16:07:50.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.785+0000 7f89a5bce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f897800ec60 con 0x55a8134ea560 2024-04-25T16:07:50.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.786+0000 7f899c9de640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.786+0000 7f899c9de640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138b3620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54106/0 (socket says 172.21.15.31:54106) 2024-04-25T16:07:50.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.786+0000 7f899c9de640 1 -- 172.21.15.31:0/3421410649 learned_addr learned my addr 172.21.15.31:0/3421410649 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:50.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.793+0000 7f899d1df640 1 --2- 172.21.15.31:0/3421410649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.794+0000 7f899c9de640 1 -- 172.21.15.31:0/3421410649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134cd550 msgr2=0x55a8138a8640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.794+0000 7f899c9de640 1 --2- 172.21.15.31:0/3421410649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134cd550 0x55a8138a8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.794+0000 7f899c9de640 1 -- 172.21.15.31:0/3421410649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 msgr2=0x55a8138b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.794+0000 7f899c9de640 1 --2- 172.21.15.31:0/3421410649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.794+0000 7f899c9de640 1 -- 172.21.15.31:0/3421410649 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8138a1e80 con 0x55a8134ea560 2024-04-25T16:07:50.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.796+0000 7f899c9de640 1 --2- 172.21.15.31:0/3421410649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138b3620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8988011040 tx=0x7f89880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=608e60ddc4bb03d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.797+0000 7f898e7fc640 1 -- 172.21.15.31:0/3421410649 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f898801ed20 con 0x55a8134ea560 2024-04-25T16:07:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.797+0000 7f898e7fc640 1 -- 172.21.15.31:0/3421410649 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8988016040 con 0x55a8134ea560 2024-04-25T16:07:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.799+0000 7f89a5bce740 1 -- 172.21.15.31:0/3421410649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 msgr2=0x55a8138b3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:50.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.799+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3421410649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138b3620 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8988011040 tx=0x7f89880046c0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.813+0000 7f89a5bce740 1 -- 172.21.15.31:0/3421410649 shutdown_connections 2024-04-25T16:07:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.813+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3421410649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.813+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3421410649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134cd550 0x55a8138a8640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.813+0000 7f89a5bce740 1 --2- 172.21.15.31:0/3421410649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138b3620 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.813+0000 7f89a5bce740 1 -- 172.21.15.31:0/3421410649 >> 172.21.15.31:0/3421410649 conn(0x55a8138a1a50 msgr2=0x55a81389b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.813+0000 7f89a5bce740 1 -- 172.21.15.31:0/3421410649 shutdown_connections 2024-04-25T16:07:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.814+0000 7f89a5bce740 1 -- 172.21.15.31:0/3421410649 wait complete. 2024-04-25T16:07:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.814+0000 7f89a5bce740 1 -- start start 2024-04-25T16:07:50.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.814+0000 7f89a5bce740 1 Processor -- start 2024-04-25T16:07:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.818+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134cd550 0x55a8138a5710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.818+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.818+0000 7f89a5bce740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138a6b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.818+0000 7f89a5bce740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f897800fa00 con 0x55a8134d5730 2024-04-25T16:07:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.818+0000 7f89a5bce740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f897800f330 con 0x55a8134cd550 2024-04-25T16:07:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.818+0000 7f89a5bce740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f897800c570 con 0x55a8134ea560 2024-04-25T16:07:50.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.818+0000 7f89a5bce740 10 client.0 _ll_register_callbacks cb 0x7f8980005510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:50.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.819+0000 7f8997fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.819+0000 7f8997fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138a5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60642/0 (socket says 172.21.15.31:60642) 2024-04-25T16:07:50.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.819+0000 7f8997fff640 1 -- 172.21.15.31:0/1143206002 learned_addr learned my addr 172.21.15.31:0/1143206002 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:50.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.821+0000 7f899d1df640 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138a6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.821+0000 7f899c9de640 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134cd550 0x55a8138a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.823+0000 7f8997fff640 1 -- 172.21.15.31:0/1143206002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 msgr2=0x55a8138a6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.823+0000 7f8997fff640 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138a6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.823+0000 7f8997fff640 1 -- 172.21.15.31:0/1143206002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134cd550 msgr2=0x55a8138a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:50.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.823+0000 7f8997fff640 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134cd550 0x55a8138a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.823+0000 7f8997fff640 1 -- 172.21.15.31:0/1143206002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8138a7140 con 0x55a8134d5730 2024-04-25T16:07:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.828+0000 7f8997fff640 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138a5c50 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8990007390 tx=0x7f89900a85f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a19f51bd417d3f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:50.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.828+0000 7f897ffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:07:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.832+0000 7f897ffff640 1 -- 172.21.15.31:0/1143206002 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89900c2060 con 0x55a8134d5730 2024-04-25T16:07:50.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.832+0000 7f89a5bce740 1 -- 172.21.15.31:0/1143206002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8988016040 con 0x55a8134d5730 2024-04-25T16:07:50.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.832+0000 7f897ffff640 1 -- 172.21.15.31:0/1143206002 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f89900a7280 con 0x55a8134d5730 2024-04-25T16:07:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.836+0000 7f897ffff640 1 -- 172.21.15.31:0/1143206002 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f89900c9050 con 0x55a8134d5730 2024-04-25T16:07:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.836+0000 7f89a5bce740 10 client.16959 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:50.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.836+0000 7f89a5bce740 1 -- 172.21.15.31:0/1143206002 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8138a84a0 con 0x55a8134d5730 2024-04-25T16:07:50.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.843+0000 7f897ffff640 1 -- 172.21.15.31:0/1143206002 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8988016040 con 0x55a8134d5730 2024-04-25T16:07:50.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.843+0000 7f897ffff640 1 -- 172.21.15.31:0/1143206002 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f89900a3990 con 0x55a8134d5730 2024-04-25T16:07:50.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.843+0000 7f89a5bce740 10 client.16959 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:50.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.843+0000 7f89a5bce740 10 client.16959 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:50.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.843+0000 7f89a5bce740 20 client.16959 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:50.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.843+0000 7f89a5bce740 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a8134f6410 0x55a813919530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:50.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.843+0000 7f89a5bce740 4 client.16959 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:07:50.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.843+0000 7f89a5bce740 1 -- 172.21.15.31:0/1143206002 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a813804580 con 0x55a8134f6410 2024-04-25T16:07:50.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.844+0000 7f899c9de640 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a8134f6410 0x55a813919530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:50.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.855+0000 7f899c9de640 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a8134f6410 0x55a813919530 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=481ebfe49516b620 server_cookie=61e68456425bef86 in_seq=0 out_seq=0 2024-04-25T16:07:50.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.855+0000 7f897ffff640 10 client.16959 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:50.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.862+0000 7f897ffff640 1 -- 172.21.15.31:0/1143206002 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a813804580 con 0x55a8134f6410 2024-04-25T16:07:50.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.862+0000 7f897ffff640 10 client.16959 handle_command_reply: tid=0 2024-04-25T16:07:50.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.862+0000 7f89a5bce740 1 client.16959 shutdown 2024-04-25T16:07:50.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.863+0000 7f89a5bce740 1 -- 172.21.15.31:0/1143206002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 msgr2=0x55a8138a5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:50.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.863+0000 7f89a5bce740 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138a5c50 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f8990007390 tx=0x7f89900a85f0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f89a5bce740 1 -- 172.21.15.31:0/1143206002 shutdown_connections 2024-04-25T16:07:50.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f89a5bce740 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a8134f6410 0x55a813919530 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f89a5bce740 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8134ea560 0x55a8138a6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f89a5bce740 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8134d5730 0x55a8138a5c50 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f89a5bce740 1 --2- 172.21.15.31:0/1143206002 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8134cd550 0x55a8138a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f89a5bce740 1 -- 172.21.15.31:0/1143206002 >> 172.21.15.31:0/1143206002 conn(0x55a8138a1a50 msgr2=0x55a813895c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:50.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f897ffff640 0 client.16959 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:50.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f89a5bce740 1 -- 172.21.15.31:0/1143206002 shutdown_connections 2024-04-25T16:07:50.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f89a5bce740 1 -- 172.21.15.31:0/1143206002 wait complete. 2024-04-25T16:07:50.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.866+0000 7f89a5bce740 20 client.16959 trim_cache size 0 max 16384 2024-04-25T16:07:50.873 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:07:50.873 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:07:50.873 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "4a85b5e2-36f7-4dab-8b0c-7c5eb3cf6ab0", 2024-04-25T16:07:50.873 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:07:50.873 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:07:50.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.876+0000 7f899ec42640 1 -- 172.21.15.31:0/393723491 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89840afd00 msgr2=0x7f89840b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:50.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.876+0000 7f899ec42640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89840afd00 0x7f89840b2210 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f8998109bf0 tx=0x7f8988005bf0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.876+0000 7f899ec42640 1 -- 172.21.15.31:0/393723491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f899810b330 msgr2=0x7f8998081f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:50.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.876+0000 7f899ec42640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f899810b330 0x7f8998081f70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f899000cce0 tx=0x7f899000a620 comp rx=0 tx=0).stop 2024-04-25T16:07:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.880+0000 7f899ec42640 1 -- 172.21.15.31:0/393723491 shutdown_connections 2024-04-25T16:07:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.880+0000 7f899ec42640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89840afd00 0x7f89840b2210 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.880+0000 7f899ec42640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8998110030 0x7f8998082e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.880+0000 7f899ec42640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f899810b330 0x7f8998081f70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.880+0000 7f899ec42640 1 --2- 172.21.15.31:0/393723491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f899810a3d0 0x7f8998081a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:50.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.880+0000 7f899ec42640 1 -- 172.21.15.31:0/393723491 >> 172.21.15.31:0/393723491 conn(0x7f8998078de0 msgr2=0x7f899810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:50.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.882+0000 7f899ec42640 1 -- 172.21.15.31:0/393723491 shutdown_connections 2024-04-25T16:07:50.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:50.882+0000 7f899ec42640 1 -- 172.21.15.31:0/393723491 wait complete. 2024-04-25T16:07:50.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:07:50.930 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:07:51.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.266+0000 7f97254b1640 1 -- 172.21.15.31:0/3134215428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 msgr2=0x7f9720114520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:51.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.266+0000 7f97254b1640 1 --2- 172.21.15.31:0/3134215428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 0x7f9720114520 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f9714009900 tx=0x7f971402a410 comp rx=0 tx=0).stop 2024-04-25T16:07:51.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.273+0000 7f97254b1640 1 -- 172.21.15.31:0/3134215428 shutdown_connections 2024-04-25T16:07:51.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.273+0000 7f97254b1640 1 --2- 172.21.15.31:0/3134215428 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 0x7f9720114520 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.273+0000 7f97254b1640 1 --2- 172.21.15.31:0/3134215428 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f972010f760 0x7f9720111b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.273+0000 7f97254b1640 1 --2- 172.21.15.31:0/3134215428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9720109fc0 0x7f972010a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.276 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.273+0000 7f97254b1640 1 -- 172.21.15.31:0/3134215428 >> 172.21.15.31:0/3134215428 conn(0x7f97200704e0 msgr2=0x7f9720072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:51.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.280+0000 7f97254b1640 1 -- 172.21.15.31:0/3134215428 shutdown_connections 2024-04-25T16:07:51.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.282+0000 7f97254b1640 1 -- 172.21.15.31:0/3134215428 wait complete. 2024-04-25T16:07:51.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.284+0000 7f97254b1640 1 Processor -- start 2024-04-25T16:07:51.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.284+0000 7f97254b1640 1 -- start start 2024-04-25T16:07:51.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.284+0000 7f97254b1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9720109fc0 0x7f97201a08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.284+0000 7f97254b1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f972010f760 0x7f97201a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.284+0000 7f97254b1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 0x7f97201b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f97254b1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9720117ef0 con 0x7f972010f760 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f97254b1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9720117d70 con 0x7f97201120c0 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f97254b1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9720118070 con 0x7f9720109fc0 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f971f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 0x7f97201b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f971f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 0x7f97201b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48418/0 (socket says 172.21.15.31:48418) 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f971f7fe640 1 -- 172.21.15.31:0/2176145496 learned_addr learned my addr 172.21.15.31:0/2176145496 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f971effd640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9720109fc0 0x7f97201a08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f971f7fe640 1 -- 172.21.15.31:0/2176145496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9720109fc0 msgr2=0x7f97201a08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f971f7fe640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9720109fc0 0x7f97201a08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.285+0000 7f971f7fe640 1 -- 172.21.15.31:0/2176145496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f972010f760 msgr2=0x7f97201a0df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:07:51.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.286+0000 7f971f7fe640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f972010f760 0x7f97201a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.286+0000 7f971f7fe640 1 -- 172.21.15.31:0/2176145496 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97201b0bb0 con 0x7f97201120c0 2024-04-25T16:07:51.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.286+0000 7f971f7fe640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 0x7f97201b0540 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9714030a30 tx=0x7f9714004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:51.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.289+0000 7f9707fff640 1 -- 172.21.15.31:0/2176145496 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714031600 con 0x7f97201120c0 2024-04-25T16:07:51.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.289+0000 7f9707fff640 1 -- 172.21.15.31:0/2176145496 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9714030e40 con 0x7f97201120c0 2024-04-25T16:07:51.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.290+0000 7f9707fff640 1 -- 172.21.15.31:0/2176145496 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714045410 con 0x7f97201120c0 2024-04-25T16:07:51.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.291+0000 7f97254b1640 1 -- 172.21.15.31:0/2176145496 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97201b0e40 con 0x7f97201120c0 2024-04-25T16:07:51.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.291+0000 7f97254b1640 1 -- 172.21.15.31:0/2176145496 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97201b11f0 con 0x7f97201120c0 2024-04-25T16:07:51.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.296+0000 7f9707fff640 1 -- 172.21.15.31:0/2176145496 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9714043020 con 0x7f97201120c0 2024-04-25T16:07:51.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.296+0000 7f972c438740 1 Processor -- start 2024-04-25T16:07:51.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.298+0000 7f9707fff640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f97080afb50 0x7f97080b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.298+0000 7f9707fff640 1 -- 172.21.15.31:0/2176145496 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f97140ff270 con 0x7f97201120c0 2024-04-25T16:07:51.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.299+0000 7f972c438740 1 -- start start 2024-04-25T16:07:51.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.299+0000 7f971e7fc640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f97080afb50 0x7f97080b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.300+0000 7f972c438740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414cf390 0x561b41870a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.300+0000 7f972c438740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41870fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.300+0000 7f972c438740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b41871c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.300+0000 7f971effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414cf390 0x561b41870a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.300+0000 7f971effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414cf390 0x561b41870a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54144/0 (socket says 172.21.15.31:54144) 2024-04-25T16:07:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.300+0000 7f971effd640 1 -- 172.21.15.31:0/562865090 learned_addr learned my addr 172.21.15.31:0/562865090 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.300+0000 7f972c438740 1 -- 172.21.15.31:0/562865090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561b417d5d80 con 0x561b414a64d0 2024-04-25T16:07:51.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.301+0000 7f971f7fe640 1 --2- 172.21.15.31:0/562865090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b41871c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.301+0000 7f971e7fc640 1 --2- 172.21.15.31:0/562865090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41870fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.301+0000 7f972c438740 1 -- 172.21.15.31:0/562865090 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561b4178f8b0 con 0x561b414ae6b0 2024-04-25T16:07:51.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.301+0000 7f972c438740 1 -- 172.21.15.31:0/562865090 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561b417d1d40 con 0x561b414cf390 2024-04-25T16:07:51.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.305+0000 7f971e7fc640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f97080afb50 0x7f97080b2060 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f97201a18b0 tx=0x7f9710006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.306+0000 7f971e7fc640 1 -- 172.21.15.31:0/562865090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414cf390 msgr2=0x561b41870a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.306+0000 7f971e7fc640 1 --2- 172.21.15.31:0/562865090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414cf390 0x561b41870a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.306+0000 7f971e7fc640 1 -- 172.21.15.31:0/562865090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 msgr2=0x561b41871c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.306+0000 7f971e7fc640 1 --2- 172.21.15.31:0/562865090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b41871c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.306+0000 7f971e7fc640 1 -- 172.21.15.31:0/562865090 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b41760a40 con 0x561b414a64d0 2024-04-25T16:07:51.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.311+0000 7f971e7fc640 1 --2- 172.21.15.31:0/562865090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41870fe0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f97201aa4a0 tx=0x7f9710016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6dee8684d5f1e6b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:51.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.317+0000 7f97057fa640 1 -- 172.21.15.31:0/562865090 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9710022070 con 0x561b414a64d0 2024-04-25T16:07:51.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.317+0000 7f97057fa640 1 -- 172.21.15.31:0/562865090 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9710019040 con 0x561b414a64d0 2024-04-25T16:07:51.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.317+0000 7f97057fa640 1 -- 172.21.15.31:0/562865090 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971001da60 con 0x561b414a64d0 2024-04-25T16:07:51.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.318+0000 7f972c438740 1 -- 172.21.15.31:0/562865090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 msgr2=0x561b41870fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:51.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.318+0000 7f972c438740 1 --2- 172.21.15.31:0/562865090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41870fe0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f97201aa4a0 tx=0x7f9710016000 comp rx=0 tx=0).stop 2024-04-25T16:07:51.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.319+0000 7f972c438740 1 -- 172.21.15.31:0/562865090 shutdown_connections 2024-04-25T16:07:51.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.319+0000 7f972c438740 1 --2- 172.21.15.31:0/562865090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b41871c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.319+0000 7f972c438740 1 --2- 172.21.15.31:0/562865090 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41870fe0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.319+0000 7f972c438740 1 --2- 172.21.15.31:0/562865090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414cf390 0x561b41870a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.319+0000 7f972c438740 1 -- 172.21.15.31:0/562865090 >> 172.21.15.31:0/562865090 conn(0x561b414c34e0 msgr2=0x561b4186c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:51.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.319+0000 7f972c438740 1 -- 172.21.15.31:0/562865090 shutdown_connections 2024-04-25T16:07:51.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.319+0000 7f972c438740 1 -- 172.21.15.31:0/562865090 wait complete. 2024-04-25T16:07:51.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.320+0000 7f972c438740 1 -- start start 2024-04-25T16:07:51.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.320+0000 7f972c438740 1 Processor -- start 2024-04-25T16:07:51.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.321+0000 7f972c438740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414a64d0 0x561b4187a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.321+0000 7f972c438740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b4187aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.321+0000 7f972c438740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414cf390 0x561b41881e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.321+0000 7f972c438740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x561b41872780 con 0x561b414cf390 2024-04-25T16:07:51.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.321+0000 7f972c438740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x561b4175cd10 con 0x561b414ae6b0 2024-04-25T16:07:51.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.321+0000 7f972c438740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x561b41872900 con 0x561b414a64d0 2024-04-25T16:07:51.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.321+0000 7f972c438740 10 client.0 _ll_register_callbacks cb 0x561b41789210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:51.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971e7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b4187aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971e7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b4187aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48440/0 (socket says 172.21.15.31:48440) 2024-04-25T16:07:51.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971e7fc640 1 -- 172.21.15.31:0/1988958667 learned_addr learned my addr 172.21.15.31:0/1988958667 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:51.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971f7fe640 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414cf390 0x561b41881e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971effd640 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414a64d0 0x561b4187a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971e7fc640 1 -- 172.21.15.31:0/1988958667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414a64d0 msgr2=0x561b4187a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:51.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971e7fc640 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414a64d0 0x561b4187a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971e7fc640 1 -- 172.21.15.31:0/1988958667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414cf390 msgr2=0x561b41881e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:51.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971e7fc640 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414cf390 0x561b41881e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.322+0000 7f971e7fc640 1 -- 172.21.15.31:0/1988958667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b415b17d0 con 0x561b414ae6b0 2024-04-25T16:07:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.328+0000 7f971e7fc640 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b4187aec0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f971001def0 tx=0x7f971001df20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93ac323bcd7dde1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.329+0000 7f9702ff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.329+0000 7f9702ff5640 1 -- 172.21.15.31:0/1988958667 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971002d070 con 0x561b414ae6b0 2024-04-25T16:07:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.329+0000 7f972c438740 1 -- 172.21.15.31:0/1988958667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9710019040 con 0x561b414ae6b0 2024-04-25T16:07:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.329+0000 7f9702ff5640 1 -- 172.21.15.31:0/1988958667 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9710014820 con 0x561b414ae6b0 2024-04-25T16:07:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.329+0000 7f9702ff5640 1 -- 172.21.15.31:0/1988958667 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9710039050 con 0x561b414ae6b0 2024-04-25T16:07:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.329+0000 7f972c438740 10 client.26609 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:51.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.329+0000 7f972c438740 1 -- 172.21.15.31:0/1988958667 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b414e73f0 con 0x561b414ae6b0 2024-04-25T16:07:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.330+0000 7f9702ff5640 1 -- 172.21.15.31:0/1988958667 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9710019040 con 0x561b414ae6b0 2024-04-25T16:07:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.331+0000 7f9702ff5640 1 -- 172.21.15.31:0/1988958667 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f9710028040 con 0x561b414ae6b0 2024-04-25T16:07:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.332+0000 7f972c438740 10 client.26609 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.332+0000 7f972c438740 10 client.26609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.332+0000 7f972c438740 20 client.26609 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.332+0000 7f972c438740 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561b4148df60 0x561b41887eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.332+0000 7f972c438740 4 client.26609 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:51.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.332+0000 7f972c438740 1 -- 172.21.15.31:0/1988958667 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561b417e3990 con 0x561b4148df60 2024-04-25T16:07:51.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.338+0000 7f971effd640 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561b4148df60 0x561b41887eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.340+0000 7f971effd640 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561b4148df60 0x561b41887eb0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fddeadbc55498488 server_cookie=b28e84070ac4835e in_seq=0 out_seq=0 2024-04-25T16:07:51.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.341+0000 7f9702ff5640 10 client.26609 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:51.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.343+0000 7f9702ff5640 1 -- 172.21.15.31:0/1988958667 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561b417e3990 con 0x561b4148df60 2024-04-25T16:07:51.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.343+0000 7f9702ff5640 10 client.26609 handle_command_reply: tid=0 2024-04-25T16:07:51.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.343+0000 7f972c438740 1 client.26609 shutdown 2024-04-25T16:07:51.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 -- 172.21.15.31:0/1988958667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 msgr2=0x561b4187aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:51.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b4187aec0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f971001def0 tx=0x7f971001df20 comp rx=0 tx=0).stop 2024-04-25T16:07:51.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 -- 172.21.15.31:0/1988958667 shutdown_connections 2024-04-25T16:07:51.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561b4148df60 0x561b41887eb0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414cf390 0x561b41881e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414ae6b0 0x561b4187aec0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 --2- 172.21.15.31:0/1988958667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414a64d0 0x561b4187a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 -- 172.21.15.31:0/1988958667 >> 172.21.15.31:0/1988958667 conn(0x561b414c34e0 msgr2=0x561b4186f180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:51.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f9702ff5640 0 client.26609 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:51.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 -- 172.21.15.31:0/1988958667 shutdown_connections 2024-04-25T16:07:51.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 1 -- 172.21.15.31:0/1988958667 wait complete. 2024-04-25T16:07:51.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.350+0000 7f972c438740 20 client.26609 trim_cache size 0 max 16384 2024-04-25T16:07:51.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.412+0000 7f972c438740 1 Processor -- start 2024-04-25T16:07:51.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.412+0000 7f972c438740 1 -- start start 2024-04-25T16:07:51.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.413+0000 7f972c438740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41875d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.413+0000 7f972c438740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4186c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.413+0000 7f972c438740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b41881d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.413+0000 7f972c438740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f97201b9da0 con 0x561b414a64d0 2024-04-25T16:07:51.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.414+0000 7f971e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4186c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.414+0000 7f971e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4186c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54156/0 (socket says 172.21.15.31:54156) 2024-04-25T16:07:51.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.414+0000 7f971e7fc640 1 -- 172.21.15.31:0/599247048 learned_addr learned my addr 172.21.15.31:0/599247048 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:51.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.414+0000 7f971f7fe640 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b41881d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.414+0000 7f972c438740 1 -- 172.21.15.31:0/599247048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f97201bb8e0 con 0x561b414cf390 2024-04-25T16:07:51.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.414+0000 7f972c438740 1 -- 172.21.15.31:0/599247048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97201ba470 con 0x561b414ae6b0 2024-04-25T16:07:51.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.414+0000 7f971effd640 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41875d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.415+0000 7f971f7fe640 1 -- 172.21.15.31:0/599247048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 msgr2=0x561b4186c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:51.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.416+0000 7f971f7fe640 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4186c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.416+0000 7f971f7fe640 1 -- 172.21.15.31:0/599247048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 msgr2=0x561b41875d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:51.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.416+0000 7f971f7fe640 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41875d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.416+0000 7f971f7fe640 1 -- 172.21.15.31:0/599247048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561b414a70f0 con 0x561b414cf390 2024-04-25T16:07:51.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.416+0000 7f971e7fc640 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4186c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:07:51.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.417+0000 7f971f7fe640 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b41881d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f97140c4970 tx=0x7f971402a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c28a1b76e810aac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:51.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.417+0000 7f9704ff9640 1 -- 172.21.15.31:0/599247048 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971404a040 con 0x561b414cf390 2024-04-25T16:07:51.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.417+0000 7f9704ff9640 1 -- 172.21.15.31:0/599247048 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97140c7700 con 0x561b414cf390 2024-04-25T16:07:51.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.419+0000 7f972c438740 1 -- 172.21.15.31:0/599247048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 msgr2=0x561b41881d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:51.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.419+0000 7f972c438740 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b41881d40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f97140c4970 tx=0x7f971402a410 comp rx=0 tx=0).stop 2024-04-25T16:07:51.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.420+0000 7f972c438740 1 -- 172.21.15.31:0/599247048 shutdown_connections 2024-04-25T16:07:51.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.420+0000 7f972c438740 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b41881d40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.420+0000 7f972c438740 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4186c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.420+0000 7f972c438740 1 --2- 172.21.15.31:0/599247048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41875d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.420+0000 7f972c438740 1 -- 172.21.15.31:0/599247048 >> 172.21.15.31:0/599247048 conn(0x561b414c34e0 msgr2=0x561b4186bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:51.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.423+0000 7f972c438740 1 -- 172.21.15.31:0/599247048 shutdown_connections 2024-04-25T16:07:51.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.423+0000 7f972c438740 1 -- 172.21.15.31:0/599247048 wait complete. 2024-04-25T16:07:51.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.424+0000 7f972c438740 1 -- start start 2024-04-25T16:07:51.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.424+0000 7f972c438740 1 Processor -- start 2024-04-25T16:07:51.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.424+0000 7f972c438740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41882550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.424+0000 7f972c438740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4187f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.424+0000 7f972c438740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b4187f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.424+0000 7f972c438740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f97201bb210 con 0x561b414a64d0 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.424+0000 7f972c438740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f97201bab40 con 0x561b414cf390 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.424+0000 7f972c438740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97201b7e30 con 0x561b414ae6b0 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.424+0000 7f972c438740 10 client.0 _ll_register_callbacks cb 0x7f9710020070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.425+0000 7f971e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4187f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.425+0000 7f971e7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4187f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54164/0 (socket says 172.21.15.31:54164) 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.425+0000 7f971e7fc640 1 -- 172.21.15.31:0/2100270460 learned_addr learned my addr 172.21.15.31:0/2100270460 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.425+0000 7f971f7fe640 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b4187f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.426+0000 7f971effd640 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41882550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.426+0000 7f971f7fe640 1 -- 172.21.15.31:0/2100270460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 msgr2=0x561b4187f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.426+0000 7f971f7fe640 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4187f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.426+0000 7f971f7fe640 1 -- 172.21.15.31:0/2100270460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 msgr2=0x561b41882550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.426+0000 7f971f7fe640 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41882550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.426+0000 7f971f7fe640 1 -- 172.21.15.31:0/2100270460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561b417964b0 con 0x561b414cf390 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.426+0000 7f971effd640 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41882550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.427+0000 7f971f7fe640 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b4187f660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97140c7400 tx=0x7f9714009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd22b91bf0f82b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.428+0000 7f9702ff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.428+0000 7f9702ff5640 1 -- 172.21.15.31:0/2100270460 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971404a070 con 0x561b414cf390 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.428+0000 7f972c438740 1 -- 172.21.15.31:0/2100270460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97140c7700 con 0x561b414cf390 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.428+0000 7f9702ff5640 1 -- 172.21.15.31:0/2100270460 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f97140c3cf0 con 0x561b414cf390 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.429+0000 7f9702ff5640 1 -- 172.21.15.31:0/2100270460 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f97140c3b30 con 0x561b414cf390 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.429+0000 7f972c438740 10 client.26615 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:51.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.429+0000 7f972c438740 1 -- 172.21.15.31:0/2100270460 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x561b41900080 con 0x561b414cf390 2024-04-25T16:07:51.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.429+0000 7f9702ff5640 1 -- 172.21.15.31:0/2100270460 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97140c7700 con 0x561b414cf390 2024-04-25T16:07:51.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.431+0000 7f9702ff5640 1 -- 172.21.15.31:0/2100270460 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f971404d060 con 0x561b414cf390 2024-04-25T16:07:51.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.432+0000 7f972c438740 10 client.26615 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:51.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.432+0000 7f972c438740 10 client.26615 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:51.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.432+0000 7f972c438740 20 client.26615 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:51.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.432+0000 7f972c438740 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561b414633a0 0x561b419027e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:51.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.432+0000 7f972c438740 4 client.26615 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:07:51.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.432+0000 7f972c438740 1 -- 172.21.15.31:0/2100270460 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561b417dd800 con 0x561b414633a0 2024-04-25T16:07:51.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.436+0000 7f971effd640 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561b414633a0 0x561b419027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:51.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.460+0000 7f971effd640 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561b414633a0 0x561b419027e0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82439eb841c65a59 server_cookie=264d6d3a3146eb5a in_seq=0 out_seq=0 2024-04-25T16:07:51.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.461+0000 7f9702ff5640 10 client.26615 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:51.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.464+0000 7f9702ff5640 1 -- 172.21.15.31:0/2100270460 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x561b417dd800 con 0x561b414633a0 2024-04-25T16:07:51.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.464+0000 7f9702ff5640 10 client.26615 handle_command_reply: tid=0 2024-04-25T16:07:51.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.464+0000 7f972c438740 1 client.26615 shutdown 2024-04-25T16:07:51.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.466+0000 7f972c438740 1 -- 172.21.15.31:0/2100270460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 msgr2=0x561b4187f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:51.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.466+0000 7f972c438740 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b4187f660 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f97140c7400 tx=0x7f9714009260 comp rx=0 tx=0).stop 2024-04-25T16:07:51.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.466+0000 7f972c438740 1 -- 172.21.15.31:0/2100270460 shutdown_connections 2024-04-25T16:07:51.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.466+0000 7f972c438740 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x561b414633a0 0x561b419027e0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.466+0000 7f972c438740 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x561b414cf390 0x561b4187f660 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.466+0000 7f972c438740 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x561b414ae6b0 0x561b4187f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.466+0000 7f972c438740 1 --2- 172.21.15.31:0/2100270460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x561b414a64d0 0x561b41882550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.466+0000 7f9702ff5640 0 client.26615 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:51.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.466+0000 7f972c438740 1 -- 172.21.15.31:0/2100270460 >> 172.21.15.31:0/2100270460 conn(0x561b414c34e0 msgr2=0x561b4186a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:51.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.474+0000 7f972c438740 1 -- 172.21.15.31:0/2100270460 shutdown_connections 2024-04-25T16:07:51.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.475+0000 7f972c438740 1 -- 172.21.15.31:0/2100270460 wait complete. 2024-04-25T16:07:51.481 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:07:51.481 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (5 inodes in the stack)", 2024-04-25T16:07:51.481 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:07:51.481 INFO:teuthology.orchestra.run.smithi031.stdout: "4a85b5e2-36f7-4dab-8b0c-7c5eb3cf6ab0": { 2024-04-25T16:07:51.482 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:07:51.482 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "4a85b5e2-36f7-4dab-8b0c-7c5eb3cf6ab0", 2024-04-25T16:07:51.482 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:07:51.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:07:51.482 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:07:51.482 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:07:51.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.478+0000 7f972c438740 20 client.26615 trim_cache size 0 max 16384 2024-04-25T16:07:51.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 -- 172.21.15.31:0/2176145496 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f97080afb50 msgr2=0x7f97080b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f97080afb50 0x7f97080b2060 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f97201a18b0 tx=0x7f9710006c90 comp rx=0 tx=0).stop 2024-04-25T16:07:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 -- 172.21.15.31:0/2176145496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 msgr2=0x7f97201b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 0x7f97201b0540 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9714030a30 tx=0x7f9714004270 comp rx=0 tx=0).stop 2024-04-25T16:07:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 -- 172.21.15.31:0/2176145496 shutdown_connections 2024-04-25T16:07:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f97080afb50 0x7f97080b2060 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f97201120c0 0x7f97201b0540 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f972010f760 0x7f97201a0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 --2- 172.21.15.31:0/2176145496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9720109fc0 0x7f97201a08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:51.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.480+0000 7f97254b1640 1 -- 172.21.15.31:0/2176145496 >> 172.21.15.31:0/2176145496 conn(0x7f97200704e0 msgr2=0x7f97201105a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:51.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.481+0000 7f97254b1640 1 -- 172.21.15.31:0/2176145496 shutdown_connections 2024-04-25T16:07:51.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:07:51.482+0000 7f97254b1640 1 -- 172.21.15.31:0/2176145496 wait complete. 2024-04-25T16:07:51.555 INFO:tasks.cephfs.filesystem:scrub status for tag:4a85b5e2-36f7-4dab-8b0c-7c5eb3cf6ab0 - {'path': '/', 'tag': '4a85b5e2-36f7-4dab-8b0c-7c5eb3cf6ab0', 'options': 'recursive,force'} 2024-04-25T16:07:51.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:51 smithi031 ceph-mon[29140]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 769 MiB/s wr, 1.72k op/s 2024-04-25T16:07:51.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:51 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:07:51.706 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:51 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:51.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:51 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:07:51.707 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:51 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:07:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:51 smithi063 ceph-mon[32413]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 769 MiB/s wr, 1.72k op/s 2024-04-25T16:07:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:51 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:07:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:51 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:51 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:07:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:51 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:07:51.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:51 smithi186 ceph-mon[33258]: pgmap v547: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 769 MiB/s wr, 1.72k op/s 2024-04-25T16:07:51.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:51 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:07:51.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:51 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:51.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:51 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:07:51.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:51 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:07:52.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:52 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:52.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:52 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:52.952 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:52 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:53.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:53 smithi063 ceph-mon[32413]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 711 MiB/s wr, 1.68k op/s 2024-04-25T16:07:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:53 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:53.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:53 smithi186 ceph-mon[33258]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 711 MiB/s wr, 1.68k op/s 2024-04-25T16:07:53.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:53 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:53.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:53 smithi031 ceph-mon[29140]: pgmap v548: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 711 MiB/s wr, 1.68k op/s 2024-04-25T16:07:53.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:53 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:07:55.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:55 smithi063 ceph-mon[32413]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 926 MiB/s wr, 2.41k op/s 2024-04-25T16:07:55.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:55 smithi186 ceph-mon[33258]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 926 MiB/s wr, 2.41k op/s 2024-04-25T16:07:55.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:55 smithi031 ceph-mon[29140]: pgmap v549: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 926 MiB/s wr, 2.41k op/s 2024-04-25T16:07:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.715+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2489853044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f5810a340 msgr2=0x7f5f5810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.715+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2489853044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f5810a340 0x7f5f5810ac50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009900 tx=0x7f5f4c02a440 comp rx=0 tx=0).stop 2024-04-25T16:07:56.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.716+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2489853044 shutdown_connections 2024-04-25T16:07:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.716+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2489853044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5810feb0 0x7f5f581102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.716+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2489853044 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f5810b190 0x7f5f5810b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.716+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2489853044 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f5810a340 0x7f5f5810ac50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.716+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2489853044 >> 172.21.15.31:0/2489853044 conn(0x7f5f58070330 msgr2=0x7f5f580727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:56.718 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.717+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2489853044 shutdown_connections 2024-04-25T16:07:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.722+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2489853044 wait complete. 2024-04-25T16:07:56.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.724+0000 7f5f5cf6c640 1 Processor -- start 2024-04-25T16:07:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.725+0000 7f5f5cf6c640 1 -- start start 2024-04-25T16:07:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.725+0000 7f5f5cf6c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f5810a340 0x7f5f581a9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.725+0000 7f5f5cf6c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f5810b190 0x7f5f581a9540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.725+0000 7f5f5cf6c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5810feb0 0x7f5f581b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.725+0000 7f5f5cf6c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5811e370 con 0x7f5f5810a340 2024-04-25T16:07:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.725+0000 7f5f5cf6c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5811e1f0 con 0x7f5f5810b190 2024-04-25T16:07:56.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.725+0000 7f5f5cf6c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f5811e4f0 con 0x7f5f5810feb0 2024-04-25T16:07:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.732+0000 7f5f56ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f5810b190 0x7f5f581a9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.733+0000 7f5f56ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f5810b190 0x7f5f581a9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:46140/0 (socket says 172.21.15.31:46140) 2024-04-25T16:07:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.733+0000 7f5f56ffd640 1 -- 172.21.15.31:0/2568013371 learned_addr learned my addr 172.21.15.31:0/2568013371 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:56.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.733+0000 7f5f577fe640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f5810a340 0x7f5f581a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.734+0000 7f5f57fff640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5810feb0 0x7f5f581b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.743+0000 7f5f57fff640 1 -- 172.21.15.31:0/2568013371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f5810b190 msgr2=0x7f5f581a9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.743+0000 7f5f57fff640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f5810b190 0x7f5f581a9540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.743+0000 7f5f57fff640 1 -- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f5810a340 msgr2=0x7f5f581a9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.743+0000 7f5f57fff640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f5810a340 0x7f5f581a9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.743+0000 7f5f57fff640 1 -- 172.21.15.31:0/2568013371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f58111c90 con 0x7f5f5810feb0 2024-04-25T16:07:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.752+0000 7f5f57fff640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5810feb0 0x7f5f581b04d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800b3e0 tx=0x7f5f4800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.752+0000 7f5f54ff9640 1 -- 172.21.15.31:0/2568013371 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48015020 con 0x7f5f5810feb0 2024-04-25T16:07:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.752+0000 7f5f54ff9640 1 -- 172.21.15.31:0/2568013371 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f48011ce0 con 0x7f5f5810feb0 2024-04-25T16:07:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.752+0000 7f5f54ff9640 1 -- 172.21.15.31:0/2568013371 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f48010e00 con 0x7f5f5810feb0 2024-04-25T16:07:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.752+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2568013371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f58111f80 con 0x7f5f5810feb0 2024-04-25T16:07:56.755 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.752+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2568013371 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f58112500 con 0x7f5f5810feb0 2024-04-25T16:07:56.763 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.759+0000 7f5f6507d740 1 Processor -- start 2024-04-25T16:07:56.771 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.768+0000 7f5f54ff9640 1 -- 172.21.15.31:0/2568013371 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5f48018020 con 0x7f5f5810feb0 2024-04-25T16:07:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.770+0000 7f5f54ff9640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f400afc30 0x7f5f400b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.770+0000 7f5f54ff9640 1 -- 172.21.15.31:0/2568013371 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5f480df270 con 0x7f5f5810feb0 2024-04-25T16:07:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.771+0000 7f5f6507d740 1 -- start start 2024-04-25T16:07:56.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.771+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd65b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.771+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd66090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.771+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd6cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.771+0000 7f5f6507d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f1fbccb460 con 0x55f1fb9c51b0 2024-04-25T16:07:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.771+0000 7f5f6507d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f1fbc85590 con 0x55f1fb9a44d0 2024-04-25T16:07:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.771+0000 7f5f6507d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f1fbcc7420 con 0x55f1fb99c2f0 2024-04-25T16:07:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.772+0000 7f5f577fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd65b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.772+0000 7f5f577fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd65b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52060/0 (socket says 172.21.15.31:52060) 2024-04-25T16:07:56.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.772+0000 7f5f577fe640 1 -- 172.21.15.31:0/4100631011 learned_addr learned my addr 172.21.15.31:0/4100631011 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:56.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.773+0000 7f5f577fe640 1 -- 172.21.15.31:0/4100631011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 msgr2=0x55f1fbd66090 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:07:56.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.773+0000 7f5f577fe640 1 --2- 172.21.15.31:0/4100631011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd66090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.773+0000 7f5f577fe640 1 -- 172.21.15.31:0/4100631011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 msgr2=0x55f1fbd6cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:56.774 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.773+0000 7f5f577fe640 1 --2- 172.21.15.31:0/4100631011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd6cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.777+0000 7f5f56ffd640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f400afc30 0x7f5f400b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.777+0000 7f5f577fe640 1 -- 172.21.15.31:0/4100631011 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1fbc920d0 con 0x55f1fb9c51b0 2024-04-25T16:07:56.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.785+0000 7f5f577fe640 1 --2- 172.21.15.31:0/4100631011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd65b10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5f58109c10 tx=0x7f5f4c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac6b76aad50994d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:56.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.786+0000 7f5f3dffb640 1 -- 172.21.15.31:0/4100631011 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c035030 con 0x55f1fb9c51b0 2024-04-25T16:07:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.786+0000 7f5f3dffb640 1 -- 172.21.15.31:0/4100631011 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f4c040070 con 0x55f1fb9c51b0 2024-04-25T16:07:56.789 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.786+0000 7f5f3dffb640 1 -- 172.21.15.31:0/4100631011 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c031c50 con 0x55f1fb9c51b0 2024-04-25T16:07:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.788+0000 7f5f6507d740 1 -- 172.21.15.31:0/4100631011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 msgr2=0x55f1fbd65b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.788+0000 7f5f6507d740 1 --2- 172.21.15.31:0/4100631011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd65b10 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5f58109c10 tx=0x7f5f4c005fd0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.790 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.788+0000 7f5f6507d740 1 -- 172.21.15.31:0/4100631011 shutdown_connections 2024-04-25T16:07:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.788+0000 7f5f6507d740 1 --2- 172.21.15.31:0/4100631011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd6cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.788+0000 7f5f6507d740 1 --2- 172.21.15.31:0/4100631011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd66090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.788+0000 7f5f6507d740 1 --2- 172.21.15.31:0/4100631011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd65b10 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.788+0000 7f5f6507d740 1 -- 172.21.15.31:0/4100631011 >> 172.21.15.31:0/4100631011 conn(0x55f1fb9b9300 msgr2=0x55f1fbd614a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:56.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.822+0000 7f5f56ffd640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f400afc30 0x7f5f400b2140 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x55f1fbd66ef0 tx=0x7f5f5000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.827+0000 7f5f6507d740 1 -- 172.21.15.31:0/4100631011 shutdown_connections 2024-04-25T16:07:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.830+0000 7f5f6507d740 1 -- 172.21.15.31:0/4100631011 wait complete. 2024-04-25T16:07:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.831+0000 7f5f6507d740 1 -- start start 2024-04-25T16:07:56.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.832+0000 7f5f6507d740 1 Processor -- start 2024-04-25T16:07:56.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.832+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.833+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd6fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.833+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd76cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.833+0000 7f5f6507d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f1fbd6d980 con 0x55f1fb9c51b0 2024-04-25T16:07:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.833+0000 7f5f6507d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f1fbc52b60 con 0x55f1fb9a44d0 2024-04-25T16:07:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.833+0000 7f5f6507d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f1fbd6db00 con 0x55f1fb99c2f0 2024-04-25T16:07:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.833+0000 7f5f6507d740 10 client.0 _ll_register_callbacks cb 0x55f1fbc772b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.833+0000 7f5f57fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd76cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.833+0000 7f5f57fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd76cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52064/0 (socket says 172.21.15.31:52064) 2024-04-25T16:07:56.835 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.833+0000 7f5f57fff640 1 -- 172.21.15.31:0/434513203 learned_addr learned my addr 172.21.15.31:0/434513203 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.834+0000 7f5f577fe640 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.837+0000 7f5f56ffd640 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd6fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.837+0000 7f5f577fe640 1 -- 172.21.15.31:0/434513203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 msgr2=0x55f1fbd6fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:56.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.837+0000 7f5f577fe640 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd6fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.837+0000 7f5f577fe640 1 -- 172.21.15.31:0/434513203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 msgr2=0x55f1fbd76cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.837+0000 7f5f577fe640 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd76cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.837+0000 7f5f577fe640 1 -- 172.21.15.31:0/434513203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1fb9dd210 con 0x55f1fb99c2f0 2024-04-25T16:07:56.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.838+0000 7f5f577fe640 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f810 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009850 tx=0x7f5f4c031b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f50102f056cfb878 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:56.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.843+0000 7f5f36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:56.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.843+0000 7f5f36ffd640 1 -- 172.21.15.31:0/434513203 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c053060 con 0x55f1fb99c2f0 2024-04-25T16:07:56.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.845+0000 7f5f36ffd640 1 -- 172.21.15.31:0/434513203 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5f4c0037f0 con 0x55f1fb99c2f0 2024-04-25T16:07:56.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.845+0000 7f5f6507d740 1 -- 172.21.15.31:0/434513203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f4c040070 con 0x55f1fb99c2f0 2024-04-25T16:07:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.849+0000 7f5f36ffd640 1 -- 172.21.15.31:0/434513203 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5f4c035030 con 0x55f1fb99c2f0 2024-04-25T16:07:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.849+0000 7f5f6507d740 10 client.26581 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:56.850 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.849+0000 7f5f6507d740 1 -- 172.21.15.31:0/434513203 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f1fbaa9620 con 0x55f1fb99c2f0 2024-04-25T16:07:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.851+0000 7f5f36ffd640 1 -- 172.21.15.31:0/434513203 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f4c040070 con 0x55f1fb99c2f0 2024-04-25T16:07:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.851+0000 7f5f36ffd640 1 -- 172.21.15.31:0/434513203 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5f4c04d030 con 0x55f1fb99c2f0 2024-04-25T16:07:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.851+0000 7f5f6507d740 10 client.26581 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.851+0000 7f5f6507d740 10 client.26581 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:56.852 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.851+0000 7f5f6507d740 20 client.26581 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.851+0000 7f5f6507d740 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1fb983d80 0x55f1fbd7d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.851+0000 7f5f6507d740 4 client.26581 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:07:56.853 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.851+0000 7f5f6507d740 1 -- 172.21.15.31:0/434513203 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1fbcd9070 con 0x55f1fb983d80 2024-04-25T16:07:56.861 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.860+0000 7f5f56ffd640 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1fb983d80 0x55f1fbd7d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.869+0000 7f5f56ffd640 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1fb983d80 0x55f1fbd7d610 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be98a912b6a86930 server_cookie=8f234355eba8d73a in_seq=0 out_seq=0 2024-04-25T16:07:56.871 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.870+0000 7f5f36ffd640 10 client.26581 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.876+0000 7f5f36ffd640 1 -- 172.21.15.31:0/434513203 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f1fbcd9070 con 0x55f1fb983d80 2024-04-25T16:07:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.876+0000 7f5f36ffd640 10 client.26581 handle_command_reply: tid=0 2024-04-25T16:07:56.879 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.876+0000 7f5f6507d740 1 client.26581 shutdown 2024-04-25T16:07:56.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 -- 172.21.15.31:0/434513203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 msgr2=0x55f1fbd6f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:56.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f810 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c009850 tx=0x7f5f4c031b80 comp rx=0 tx=0).stop 2024-04-25T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 -- 172.21.15.31:0/434513203 shutdown_connections 2024-04-25T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1fb983d80 0x55f1fbd7d610 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd76cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd6fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 --2- 172.21.15.31:0/434513203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f810 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 -- 172.21.15.31:0/434513203 >> 172.21.15.31:0/434513203 conn(0x55f1fb9b9300 msgr2=0x55f1fbd60500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f36ffd640 0 client.26581 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 -- 172.21.15.31:0/434513203 shutdown_connections 2024-04-25T16:07:56.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.881+0000 7f5f6507d740 1 -- 172.21.15.31:0/434513203 wait complete. 2024-04-25T16:07:56.884 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.882+0000 7f5f6507d740 20 client.26581 trim_cache size 0 max 16384 2024-04-25T16:07:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.956+0000 7f5f6507d740 1 Processor -- start 2024-04-25T16:07:56.958 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.957+0000 7f5f6507d740 1 -- start start 2024-04-25T16:07:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.957+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.958+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd76bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.958+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd6f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.958+0000 7f5f6507d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4400e4f0 con 0x55f1fb9c51b0 2024-04-25T16:07:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.958+0000 7f5f6507d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5f44010030 con 0x55f1fb9a44d0 2024-04-25T16:07:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.958+0000 7f5f6507d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4400ebc0 con 0x55f1fb99c2f0 2024-04-25T16:07:56.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.958+0000 7f5f577fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.958+0000 7f5f577fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:60672/0 (socket says 172.21.15.31:60672) 2024-04-25T16:07:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.958+0000 7f5f577fe640 1 -- 172.21.15.31:0/2089038317 learned_addr learned my addr 172.21.15.31:0/2089038317 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.958+0000 7f5f56ffd640 1 --2- 172.21.15.31:0/2089038317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd76bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.959+0000 7f5f577fe640 1 -- 172.21.15.31:0/2089038317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 msgr2=0x55f1fbd76bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:56.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.959+0000 7f5f577fe640 1 --2- 172.21.15.31:0/2089038317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd76bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.959+0000 7f5f577fe640 1 -- 172.21.15.31:0/2089038317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 msgr2=0x55f1fbd6f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.959+0000 7f5f577fe640 1 --2- 172.21.15.31:0/2089038317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd6f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.959+0000 7f5f577fe640 1 -- 172.21.15.31:0/2089038317 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1fbd6a690 con 0x55f1fb99c2f0 2024-04-25T16:07:56.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.960+0000 7f5f577fe640 1 --2- 172.21.15.31:0/2089038317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f3e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0092a0 tx=0x7f5f4c005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74b923a37ac2b7bc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:56.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.960+0000 7f5f3d7fa640 1 -- 172.21.15.31:0/2089038317 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c05e020 con 0x55f1fb99c2f0 2024-04-25T16:07:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.962+0000 7f5f3d7fa640 1 -- 172.21.15.31:0/2089038317 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f4c040040 con 0x55f1fb99c2f0 2024-04-25T16:07:56.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.971+0000 7f5f3d7fa640 1 -- 172.21.15.31:0/2089038317 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c053070 con 0x55f1fb99c2f0 2024-04-25T16:07:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.974+0000 7f5f6507d740 1 -- 172.21.15.31:0/2089038317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 msgr2=0x55f1fbd6f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:07:56.976 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.975+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2089038317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f3e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c0092a0 tx=0x7f5f4c005fd0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.979+0000 7f5f6507d740 1 -- 172.21.15.31:0/2089038317 shutdown_connections 2024-04-25T16:07:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.979+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2089038317 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd6f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.979+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2089038317 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd76bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.979+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2089038317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd6f3e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.979+0000 7f5f6507d740 1 -- 172.21.15.31:0/2089038317 >> 172.21.15.31:0/2089038317 conn(0x55f1fb9b9300 msgr2=0x55f1fbd65a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:07:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.985+0000 7f5f6507d740 1 -- 172.21.15.31:0/2089038317 shutdown_connections 2024-04-25T16:07:56.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.985+0000 7f5f6507d740 1 -- 172.21.15.31:0/2089038317 wait complete. 2024-04-25T16:07:56.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.987+0000 7f5f6507d740 1 -- start start 2024-04-25T16:07:56.990 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.988+0000 7f5f6507d740 1 Processor -- start 2024-04-25T16:07:56.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.990+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd71fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.991+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd72500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.991+0000 7f5f6507d740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd75440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.992+0000 7f5f6507d740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4400f960 con 0x55f1fb9a44d0 2024-04-25T16:07:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.992+0000 7f5f6507d740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4400f290 con 0x55f1fb9c51b0 2024-04-25T16:07:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.992+0000 7f5f577fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd71fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.992+0000 7f5f6507d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4400c4d0 con 0x55f1fb99c2f0 2024-04-25T16:07:56.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.992+0000 7f5f577fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd71fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:60674/0 (socket says 172.21.15.31:60674) 2024-04-25T16:07:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.992+0000 7f5f577fe640 1 -- 172.21.15.31:0/2754999086 learned_addr learned my addr 172.21.15.31:0/2754999086 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:07:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.993+0000 7f5f57fff640 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd75440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:56.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.993+0000 7f5f6507d740 10 client.0 _ll_register_callbacks cb 0x7f5f4c009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:07:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.993+0000 7f5f577fe640 1 -- 172.21.15.31:0/2754999086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9c51b0 msgr2=0x55f1fbd75440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.993+0000 7f5f577fe640 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd75440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.993+0000 7f5f577fe640 1 -- 172.21.15.31:0/2754999086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9a44d0 msgr2=0x55f1fbd72500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:07:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.993+0000 7f5f577fe640 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd72500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:07:56.995 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.994+0000 7f5f577fe640 1 -- 172.21.15.31:0/2754999086 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1fbd77110 con 0x55f1fb99c2f0 2024-04-25T16:07:56.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:56.995+0000 7f5f577fe640 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd71fc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c004650 tx=0x7f5f4c02abc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3071f5aa51a7f7c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:07:57.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.008+0000 7f5f36ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:07:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.008+0000 7f5f36ffd640 1 -- 172.21.15.31:0/2754999086 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c05e070 con 0x55f1fb99c2f0 2024-04-25T16:07:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.008+0000 7f5f36ffd640 1 -- 172.21.15.31:0/2754999086 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5f4c03c730 con 0x55f1fb99c2f0 2024-04-25T16:07:57.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.009+0000 7f5f6507d740 1 -- 172.21.15.31:0/2754999086 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f4c040040 con 0x55f1fb99c2f0 2024-04-25T16:07:57.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.010+0000 7f5f36ffd640 1 -- 172.21.15.31:0/2754999086 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5f4c053070 con 0x55f1fb99c2f0 2024-04-25T16:07:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.012+0000 7f5f6507d740 10 client.26587 fetch_fsmap learned FSMap version 29 2024-04-25T16:07:57.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.012+0000 7f5f6507d740 1 -- 172.21.15.31:0/2754999086 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f1fbd76aa0 con 0x55f1fb99c2f0 2024-04-25T16:07:57.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.014+0000 7f5f36ffd640 1 -- 172.21.15.31:0/2754999086 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f4c03cde0 con 0x55f1fb99c2f0 2024-04-25T16:07:57.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.016+0000 7f5f36ffd640 1 -- 172.21.15.31:0/2754999086 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5f4c04d070 con 0x55f1fb99c2f0 2024-04-25T16:07:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.019+0000 7f5f6507d740 10 client.26587 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:07:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.019+0000 7f5f6507d740 10 client.26587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:07:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.019+0000 7f5f6507d740 20 client.26587 populate_metadata read hostname 'smithi031' 2024-04-25T16:07:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.019+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1fb9591c0 0x55f1fbde7c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:07:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.019+0000 7f5f6507d740 4 client.26587 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:07:57.020 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.019+0000 7f5f6507d740 1 -- 172.21.15.31:0/2754999086 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5f4c040040 con 0x55f1fb9591c0 2024-04-25T16:07:57.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.022+0000 7f5f56ffd640 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1fb9591c0 0x55f1fbde7c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:07:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.024+0000 7f5f56ffd640 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1fb9591c0 0x55f1fbde7c90 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8749e28fea9073 server_cookie=1b71026fc2ea622d in_seq=0 out_seq=0 2024-04-25T16:07:57.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:07:57.025+0000 7f5f36ffd640 10 client.26587 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:07:57.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:57 smithi063 ceph-mon[32413]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 699 MiB/s wr, 1.76k op/s 2024-04-25T16:07:57.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:57 smithi186 ceph-mon[33258]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 699 MiB/s wr, 1.76k op/s 2024-04-25T16:07:57.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:57 smithi031 ceph-mon[29140]: pgmap v550: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 699 MiB/s wr, 1.76k op/s 2024-04-25T16:07:58.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:58 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:07:58.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:58 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:07:58.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:58 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:07:59.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:59 smithi063 ceph-mon[32413]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 836 MiB/s wr, 2.19k op/s 2024-04-25T16:07:59.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:07:59 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:07:59.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:59 smithi186 ceph-mon[33258]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 836 MiB/s wr, 2.19k op/s 2024-04-25T16:07:59.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:07:59 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:07:59.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:59 smithi031 ceph-mon[29140]: pgmap v551: 97 pgs: 97 active+clean; 10 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 836 MiB/s wr, 2.19k op/s 2024-04-25T16:07:59.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:07:59 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:08:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.406+0000 7f5f36ffd640 1 -- 172.21.15.31:0/2754999086 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5f4c040040 con 0x55f1fb9591c0 2024-04-25T16:08:00.409 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.406+0000 7f5f36ffd640 10 client.26587 handle_command_reply: tid=0 2024-04-25T16:08:00.410 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.408+0000 7f5f6507d740 1 client.26587 shutdown 2024-04-25T16:08:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.409+0000 7f5f6507d740 1 -- 172.21.15.31:0/2754999086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 msgr2=0x55f1fbd71fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:00.411 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.409+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd71fc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c004650 tx=0x7f5f4c02abc0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.417+0000 7f5f6507d740 1 -- 172.21.15.31:0/2754999086 shutdown_connections 2024-04-25T16:08:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.417+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1fb9591c0 0x55f1fbde7c90 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.417+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1fb9c51b0 0x55f1fbd75440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.417+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1fb9a44d0 0x55f1fbd72500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.417+0000 7f5f6507d740 1 --2- 172.21.15.31:0/2754999086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1fb99c2f0 0x55f1fbd71fc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.417+0000 7f5f6507d740 1 -- 172.21.15.31:0/2754999086 >> 172.21.15.31:0/2754999086 conn(0x55f1fb9b9300 msgr2=0x55f1fbd6f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.417+0000 7f5f36ffd640 0 client.26587 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.417+0000 7f5f6507d740 1 -- 172.21.15.31:0/2754999086 shutdown_connections 2024-04-25T16:08:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.417+0000 7f5f6507d740 1 -- 172.21.15.31:0/2754999086 wait complete. 2024-04-25T16:08:00.419 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.418+0000 7f5f6507d740 20 client.26587 trim_cache size 0 max 16384 2024-04-25T16:08:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:08:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:08:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:08:00.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:08:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.419+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f400afc30 msgr2=0x7f5f400b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.419+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f400afc30 0x7f5f400b2140 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x55f1fbd66ef0 tx=0x7f5f5000b040 comp rx=0 tx=0).stop 2024-04-25T16:08:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.419+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2568013371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5810feb0 msgr2=0x7f5f581b04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.419+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5810feb0 0x7f5f581b04d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5f4800b3e0 tx=0x7f5f4800b8a0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.420+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2568013371 shutdown_connections 2024-04-25T16:08:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.420+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f400afc30 0x7f5f400b2140 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.420+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f5810feb0 0x7f5f581b04d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.422 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.420+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f5810b190 0x7f5f581a9540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.420+0000 7f5f5cf6c640 1 --2- 172.21.15.31:0/2568013371 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f5810a340 0x7f5f581a9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:00.423 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.421+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2568013371 >> 172.21.15.31:0/2568013371 conn(0x7f5f58070330 msgr2=0x7f5f5810be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.425+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2568013371 shutdown_connections 2024-04-25T16:08:00.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:00.425+0000 7f5f5cf6c640 1 -- 172.21.15.31:0/2568013371 wait complete. 2024-04-25T16:08:01.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:01 smithi063 ceph-mon[32413]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 755 MiB/s wr, 2.21k op/s 2024-04-25T16:08:01.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:01 smithi186 ceph-mon[33258]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 755 MiB/s wr, 2.21k op/s 2024-04-25T16:08:01.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:01 smithi031 ceph-mon[29140]: pgmap v552: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 755 MiB/s wr, 2.21k op/s 2024-04-25T16:08:03.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:03 smithi186 ceph-mon[33258]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 706 MiB/s wr, 2.08k op/s 2024-04-25T16:08:03.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:03 smithi031 ceph-mon[29140]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 706 MiB/s wr, 2.08k op/s 2024-04-25T16:08:04.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:03 smithi063 ceph-mon[32413]: pgmap v553: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 706 MiB/s wr, 2.08k op/s 2024-04-25T16:08:05.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:05 smithi186 ceph-mon[33258]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 947 MiB/s wr, 2.42k op/s 2024-04-25T16:08:05.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:05 smithi031 ceph-mon[29140]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 947 MiB/s wr, 2.42k op/s 2024-04-25T16:08:06.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:05 smithi063 ceph-mon[32413]: pgmap v554: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 947 MiB/s wr, 2.42k op/s 2024-04-25T16:08:07.958 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:07 smithi031 ceph-mon[29140]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 733 MiB/s wr, 1.64k op/s 2024-04-25T16:08:08.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:07 smithi063 ceph-mon[32413]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 733 MiB/s wr, 1.64k op/s 2024-04-25T16:08:08.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:07 smithi186 ceph-mon[33258]: pgmap v555: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 733 MiB/s wr, 1.64k op/s 2024-04-25T16:08:09.950 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:09 smithi186 ceph-mon[33258]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 866 MiB/s wr, 1.77k op/s 2024-04-25T16:08:09.971 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:09 smithi031 ceph-mon[29140]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 866 MiB/s wr, 1.77k op/s 2024-04-25T16:08:10.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:09 smithi063 ceph-mon[32413]: pgmap v556: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 866 MiB/s wr, 1.77k op/s 2024-04-25T16:08:11.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:11 smithi031 ceph-mon[29140]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 788 MiB/s wr, 1.46k op/s 2024-04-25T16:08:12.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:11 smithi063 ceph-mon[32413]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 788 MiB/s wr, 1.46k op/s 2024-04-25T16:08:12.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:11 smithi186 ceph-mon[33258]: pgmap v557: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 788 MiB/s wr, 1.46k op/s 2024-04-25T16:08:13.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.789+0000 7efdd2287640 1 -- 172.21.15.31:0/1612536800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc10a340 msgr2=0x7efdcc10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.789+0000 7efdd2287640 1 --2- 172.21.15.31:0/1612536800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc10a340 0x7efdcc10ac50 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efdbc01a910 tx=0x7efdbc03b450 comp rx=0 tx=0).stop 2024-04-25T16:08:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.792+0000 7efdd2287640 1 -- 172.21.15.31:0/1612536800 shutdown_connections 2024-04-25T16:08:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.792+0000 7efdd2287640 1 --2- 172.21.15.31:0/1612536800 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc10feb0 0x7efdcc1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.792+0000 7efdd2287640 1 --2- 172.21.15.31:0/1612536800 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc10b190 0x7efdcc10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.792+0000 7efdd2287640 1 --2- 172.21.15.31:0/1612536800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc10a340 0x7efdcc10ac50 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.792+0000 7efdd2287640 1 -- 172.21.15.31:0/1612536800 >> 172.21.15.31:0/1612536800 conn(0x7efdcc070330 msgr2=0x7efdcc0727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:13.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.795+0000 7efdd2287640 1 -- 172.21.15.31:0/1612536800 shutdown_connections 2024-04-25T16:08:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.797+0000 7efdd2287640 1 -- 172.21.15.31:0/1612536800 wait complete. 2024-04-25T16:08:13.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.798+0000 7efdd2287640 1 Processor -- start 2024-04-25T16:08:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.799+0000 7efdd2287640 1 -- start start 2024-04-25T16:08:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.800+0000 7efdd2287640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc10a340 0x7efdcc1a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.800+0000 7efdd2287640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc10b190 0x7efdcc1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.800+0000 7efdd2287640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc10feb0 0x7efdcc1a7c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.800+0000 7efdd2287640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc113b40 con 0x7efdcc10a340 2024-04-25T16:08:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.800+0000 7efdd2287640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc1139c0 con 0x7efdcc10b190 2024-04-25T16:08:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.800+0000 7efdd2287640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc113cc0 con 0x7efdcc10feb0 2024-04-25T16:08:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.800+0000 7efdd1285640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc10a340 0x7efdcc1a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.802 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.800+0000 7efdd1285640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc10a340 0x7efdcc1a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45654/0 (socket says 172.21.15.31:45654) 2024-04-25T16:08:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.800+0000 7efdd1285640 1 -- 172.21.15.31:0/104221444 learned_addr learned my addr 172.21.15.31:0/104221444 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:13.803 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.802+0000 7efdd1285640 1 -- 172.21.15.31:0/104221444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc10feb0 msgr2=0x7efdcc1a7c90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:08:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.802+0000 7efdd1285640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc10feb0 0x7efdcc1a7c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.802+0000 7efdd1285640 1 -- 172.21.15.31:0/104221444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc10b190 msgr2=0x7efdcc1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.802+0000 7efdd1285640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc10b190 0x7efdcc1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.802+0000 7efdd1285640 1 -- 172.21.15.31:0/104221444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc1a8300 con 0x7efdcc10a340 2024-04-25T16:08:13.804 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.803+0000 7efdd1285640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc10a340 0x7efdcc1a07c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efdbc03ba40 tx=0x7efdbc004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:13.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.804+0000 7efdca7fc640 1 -- 172.21.15.31:0/104221444 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc04f070 con 0x7efdcc10a340 2024-04-25T16:08:13.806 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.805+0000 7efdd2287640 1 -- 172.21.15.31:0/104221444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc1a8590 con 0x7efdcc10a340 2024-04-25T16:08:13.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.806+0000 7efdd2287640 1 -- 172.21.15.31:0/104221444 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc1a8b00 con 0x7efdcc10a340 2024-04-25T16:08:13.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.806+0000 7efdca7fc640 1 -- 172.21.15.31:0/104221444 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdbc040bd0 con 0x7efdcc10a340 2024-04-25T16:08:13.807 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.806+0000 7efdca7fc640 1 -- 172.21.15.31:0/104221444 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc046030 con 0x7efdcc10a340 2024-04-25T16:08:13.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.814+0000 7efdca7fc640 1 -- 172.21.15.31:0/104221444 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7efdbc052070 con 0x7efdcc10a340 2024-04-25T16:08:13.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.818+0000 7efdca7fc640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdb40afc30 0x7efdb40b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.818+0000 7efdda395740 1 Processor -- start 2024-04-25T16:08:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.818+0000 7efdca7fc640 1 -- 172.21.15.31:0/104221444 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7efdbc040d80 con 0x7efdcc10a340 2024-04-25T16:08:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdda395740 1 -- start start 2024-04-25T16:08:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdda395740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d27529b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdda395740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 0x556d2752a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdda395740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27530fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdda395740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556d2748f460 con 0x556d271684d0 2024-04-25T16:08:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdda395740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556d27449590 con 0x556d271602f0 2024-04-25T16:08:13.820 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdda395740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556d2748b420 con 0x556d271891b0 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdd1285640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d27529b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdd1285640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d27529b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53364/0 (socket says 172.21.15.31:53364) 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdd1285640 1 -- 172.21.15.31:0/3063412268 learned_addr learned my addr 172.21.15.31:0/3063412268 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.819+0000 7efdd0a84640 1 --2- 172.21.15.31:0/3063412268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 0x556d2752a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.820+0000 7efdd0a84640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdb40afc30 0x7efdb40b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.820+0000 7efdd1285640 1 -- 172.21.15.31:0/3063412268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 msgr2=0x556d2752a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.820+0000 7efdd1285640 1 --2- 172.21.15.31:0/3063412268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 0x556d2752a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.820+0000 7efdd1285640 1 -- 172.21.15.31:0/3063412268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 msgr2=0x556d27530fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.820+0000 7efdd1285640 1 --2- 172.21.15.31:0/3063412268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27530fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.820+0000 7efdd1285640 1 -- 172.21.15.31:0/3063412268 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d274560d0 con 0x556d271891b0 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.820+0000 7efdd1285640 1 --2- 172.21.15.31:0/3063412268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d27529b20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efdbc041040 tx=0x7efdbc000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f6f44f2a673c025 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.821+0000 7efdb37fe640 1 -- 172.21.15.31:0/3063412268 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc059070 con 0x556d271891b0 2024-04-25T16:08:13.822 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.821+0000 7efdb37fe640 1 -- 172.21.15.31:0/3063412268 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdbc0e23d0 con 0x556d271891b0 2024-04-25T16:08:13.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.822+0000 7efdb37fe640 1 -- 172.21.15.31:0/3063412268 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc0e1f40 con 0x556d271891b0 2024-04-25T16:08:13.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.822+0000 7efdda395740 1 -- 172.21.15.31:0/3063412268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 msgr2=0x556d27529b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:13.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.822+0000 7efdda395740 1 --2- 172.21.15.31:0/3063412268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d27529b20 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efdbc041040 tx=0x7efdbc000bc0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.823+0000 7efdda395740 1 -- 172.21.15.31:0/3063412268 shutdown_connections 2024-04-25T16:08:13.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.823+0000 7efdda395740 1 --2- 172.21.15.31:0/3063412268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27530fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.823+0000 7efdda395740 1 --2- 172.21.15.31:0/3063412268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 0x556d2752a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.823+0000 7efdda395740 1 --2- 172.21.15.31:0/3063412268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d27529b20 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.824 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.823+0000 7efdda395740 1 -- 172.21.15.31:0/3063412268 >> 172.21.15.31:0/3063412268 conn(0x556d2717d300 msgr2=0x556d275254b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:13.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.824+0000 7efdd0a84640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdb40afc30 0x7efdb40b2140 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x556d2752af00 tx=0x7efdc4006ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:13.825 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.824+0000 7efdda395740 1 -- 172.21.15.31:0/3063412268 shutdown_connections 2024-04-25T16:08:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.824+0000 7efdda395740 1 -- 172.21.15.31:0/3063412268 wait complete. 2024-04-25T16:08:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.825+0000 7efdda395740 1 -- start start 2024-04-25T16:08:13.826 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.825+0000 7efdda395740 1 Processor -- start 2024-04-25T16:08:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.826+0000 7efdda395740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 0x556d275335c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.826+0000 7efdda395740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27533b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.826+0000 7efdda395740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d2753aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.826+0000 7efdda395740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556d275317b0 con 0x556d271684d0 2024-04-25T16:08:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.826+0000 7efdda395740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556d27416b60 con 0x556d271602f0 2024-04-25T16:08:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.826+0000 7efdda395740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556d27531930 con 0x556d271891b0 2024-04-25T16:08:13.828 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.826+0000 7efdda395740 10 client.0 _ll_register_callbacks cb 0x556d2743b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.827+0000 7efdd0a84640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27533b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.827+0000 7efdd0a84640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27533b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45676/0 (socket says 172.21.15.31:45676) 2024-04-25T16:08:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.827+0000 7efdd0a84640 1 -- 172.21.15.31:0/3101562626 learned_addr learned my addr 172.21.15.31:0/3101562626 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.828+0000 7efdd1a86640 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d2753aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.829+0000 7efdd1285640 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 0x556d275335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.829+0000 7efdd1a86640 1 -- 172.21.15.31:0/3101562626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 msgr2=0x556d275335c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.829+0000 7efdd1a86640 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 0x556d275335c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.832 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.829+0000 7efdd1a86640 1 -- 172.21.15.31:0/3101562626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 msgr2=0x556d27533b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:13.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.829+0000 7efdd1a86640 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27533b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.829+0000 7efdd1a86640 1 -- 172.21.15.31:0/3101562626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d271a1210 con 0x556d271891b0 2024-04-25T16:08:13.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.829+0000 7efdd1a86640 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d2753aa60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efdc000bd10 tx=0x7efdc0009430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a281c5026155ac75 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:13.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.830+0000 7efdb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:08:13.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.830+0000 7efdb0ff9640 1 -- 172.21.15.31:0/3101562626 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc0011440 con 0x556d271891b0 2024-04-25T16:08:13.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.830+0000 7efdda395740 1 -- 172.21.15.31:0/3101562626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdbc0e23d0 con 0x556d271891b0 2024-04-25T16:08:13.833 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.830+0000 7efdb0ff9640 1 -- 172.21.15.31:0/3101562626 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7efdc00115e0 con 0x556d271891b0 2024-04-25T16:08:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.832+0000 7efdb0ff9640 1 -- 172.21.15.31:0/3101562626 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efdc00117e0 con 0x556d271891b0 2024-04-25T16:08:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.832+0000 7efdda395740 10 client.26596 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:13.834 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.832+0000 7efdda395740 1 -- 172.21.15.31:0/3101562626 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556d2726d620 con 0x556d271891b0 2024-04-25T16:08:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.838+0000 7efdb0ff9640 1 -- 172.21.15.31:0/3101562626 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdbc0e23d0 con 0x556d271891b0 2024-04-25T16:08:13.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.838+0000 7efdb0ff9640 1 -- 172.21.15.31:0/3101562626 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7efdc001db40 con 0x556d271891b0 2024-04-25T16:08:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.838+0000 7efdda395740 10 client.26596 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.838+0000 7efdda395740 10 client.26596 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.838+0000 7efdda395740 20 client.26596 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.838+0000 7efdda395740 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556d27147d80 0x556d27541870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.838+0000 7efdda395740 4 client.26596 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:13.840 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.838+0000 7efdda395740 1 -- 172.21.15.31:0/3101562626 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d2749d070 con 0x556d27147d80 2024-04-25T16:08:13.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.839+0000 7efdd1285640 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556d27147d80 0x556d27541870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.840+0000 7efdd1285640 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556d27147d80 0x556d27541870 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6fc6aef0fb1fef9 server_cookie=339649cf51373cb2 in_seq=0 out_seq=0 2024-04-25T16:08:13.841 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.840+0000 7efdb0ff9640 10 client.26596 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:13.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.842+0000 7efdb0ff9640 1 -- 172.21.15.31:0/3101562626 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556d2749d070 con 0x556d27147d80 2024-04-25T16:08:13.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.842+0000 7efdb0ff9640 10 client.26596 handle_command_reply: tid=0 2024-04-25T16:08:13.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.842+0000 7efdda395740 1 client.26596 shutdown 2024-04-25T16:08:13.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.847+0000 7efdda395740 1 -- 172.21.15.31:0/3101562626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 msgr2=0x556d2753aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:13.848 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.847+0000 7efdda395740 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d2753aa60 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7efdc000bd10 tx=0x7efdc0009430 comp rx=0 tx=0).stop 2024-04-25T16:08:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.847+0000 7efdda395740 1 -- 172.21.15.31:0/3101562626 shutdown_connections 2024-04-25T16:08:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.847+0000 7efdda395740 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556d27147d80 0x556d27541870 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.847+0000 7efdda395740 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271891b0 0x556d2753aa60 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.847+0000 7efdda395740 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27533b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.847+0000 7efdda395740 1 --2- 172.21.15.31:0/3101562626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271602f0 0x556d275335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.847+0000 7efdda395740 1 -- 172.21.15.31:0/3101562626 >> 172.21.15.31:0/3101562626 conn(0x556d2717d300 msgr2=0x556d275283c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:13.849 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.848+0000 7efdb0ff9640 0 client.26596 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.849+0000 7efdda395740 1 -- 172.21.15.31:0/3101562626 shutdown_connections 2024-04-25T16:08:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.849+0000 7efdda395740 1 -- 172.21.15.31:0/3101562626 wait complete. 2024-04-25T16:08:13.851 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.850+0000 7efdda395740 20 client.26596 trim_cache size 0 max 16384 2024-04-25T16:08:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.888+0000 7efdda395740 1 Processor -- start 2024-04-25T16:08:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.888+0000 7efdda395740 1 -- start start 2024-04-25T16:08:13.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.888+0000 7efdda395740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27532f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.888+0000 7efdda395740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d2753a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.889+0000 7efdda395740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 0x556d275335c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.889+0000 7efdd0a84640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d2753a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.889+0000 7efdd1285640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27532f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.889+0000 7efdd1285640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27532f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53382/0 (socket says 172.21.15.31:53382) 2024-04-25T16:08:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.889+0000 7efdd0a84640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d2753a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45692/0 (socket says 172.21.15.31:45692) 2024-04-25T16:08:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.889+0000 7efdd1285640 1 -- 172.21.15.31:0/2613885549 learned_addr learned my addr 172.21.15.31:0/2613885549 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:13.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.890+0000 7efdda395740 1 -- 172.21.15.31:0/2613885549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800e500 con 0x556d271684d0 2024-04-25T16:08:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.890+0000 7efdda395740 1 -- 172.21.15.31:0/2613885549 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efdb8010040 con 0x556d271891b0 2024-04-25T16:08:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.890+0000 7efdda395740 1 -- 172.21.15.31:0/2613885549 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800ebd0 con 0x556d271602f0 2024-04-25T16:08:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.890+0000 7efdd1a86640 1 --2- 172.21.15.31:0/2613885549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 0x556d275335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.890+0000 7efdd0a84640 1 -- 172.21.15.31:0/2613885549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 msgr2=0x556d27532f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.890+0000 7efdd0a84640 1 --2- 172.21.15.31:0/2613885549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27532f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.890+0000 7efdd0a84640 1 -- 172.21.15.31:0/2613885549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 msgr2=0x556d275335c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.890+0000 7efdd0a84640 1 --2- 172.21.15.31:0/2613885549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 0x556d275335c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.890+0000 7efdd0a84640 1 -- 172.21.15.31:0/2613885549 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d27531360 con 0x556d271684d0 2024-04-25T16:08:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.891+0000 7efdd0a84640 1 --2- 172.21.15.31:0/2613885549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d2753a980 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7efdc4008fd0 tx=0x7efdc4005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b5e03aa6527c44 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.892+0000 7efdb2ffd640 1 -- 172.21.15.31:0/2613885549 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc4029070 con 0x556d271684d0 2024-04-25T16:08:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.892+0000 7efdb2ffd640 1 -- 172.21.15.31:0/2613885549 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdc4006870 con 0x556d271684d0 2024-04-25T16:08:13.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.892+0000 7efdb2ffd640 1 -- 172.21.15.31:0/2613885549 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdc40249c0 con 0x556d271684d0 2024-04-25T16:08:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.894+0000 7efdda395740 1 -- 172.21.15.31:0/2613885549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 msgr2=0x556d2753a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:13.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.894+0000 7efdda395740 1 --2- 172.21.15.31:0/2613885549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d2753a980 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7efdc4008fd0 tx=0x7efdc4005000 comp rx=0 tx=0).stop 2024-04-25T16:08:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.894+0000 7efdda395740 1 -- 172.21.15.31:0/2613885549 shutdown_connections 2024-04-25T16:08:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.894+0000 7efdda395740 1 --2- 172.21.15.31:0/2613885549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 0x556d275335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.894+0000 7efdda395740 1 --2- 172.21.15.31:0/2613885549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d2753a980 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.895+0000 7efdda395740 1 --2- 172.21.15.31:0/2613885549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27532f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.895+0000 7efdda395740 1 -- 172.21.15.31:0/2613885549 >> 172.21.15.31:0/2613885549 conn(0x556d2717d300 msgr2=0x556d27532870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.895+0000 7efdda395740 1 -- 172.21.15.31:0/2613885549 shutdown_connections 2024-04-25T16:08:13.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.895+0000 7efdda395740 1 -- 172.21.15.31:0/2613885549 wait complete. 2024-04-25T16:08:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.896+0000 7efdda395740 1 -- start start 2024-04-25T16:08:13.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.896+0000 7efdda395740 1 Processor -- start 2024-04-25T16:08:13.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.897+0000 7efdda395740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27537ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.898 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.897+0000 7efdda395740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27538220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.898+0000 7efdd1285640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27537ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.898+0000 7efdd1285640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27537ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53390/0 (socket says 172.21.15.31:53390) 2024-04-25T16:08:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.898+0000 7efdd1285640 1 -- 172.21.15.31:0/2360098445 learned_addr learned my addr 172.21.15.31:0/2360098445 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.898+0000 7efdda395740 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 0x556d27539150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.899 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.898+0000 7efdd0a84640 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27538220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.899+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800f970 con 0x556d271684d0 2024-04-25T16:08:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.899+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800f2a0 con 0x556d271891b0 2024-04-25T16:08:13.900 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.899+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efdb800c4e0 con 0x556d271602f0 2024-04-25T16:08:13.901 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.899+0000 7efdd1a86640 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 0x556d27539150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.900+0000 7efdd1285640 1 -- 172.21.15.31:0/2360098445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 msgr2=0x556d27539150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.900+0000 7efdd1285640 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 0x556d27539150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.900+0000 7efdd1285640 1 -- 172.21.15.31:0/2360098445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 msgr2=0x556d27538220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.900+0000 7efdd1285640 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27538220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.900+0000 7efdd1285640 1 -- 172.21.15.31:0/2360098445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d2753aec0 con 0x556d271602f0 2024-04-25T16:08:13.902 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.901+0000 7efdda395740 10 client.0 _ll_register_callbacks cb 0x7efdc0006df0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.901+0000 7efdd1285640 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27537ce0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdbc0d9500 tx=0x7efdbc03b450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf6eb6bb7d859640 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.901+0000 7efdb0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:08:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.901+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdc4006870 con 0x556d271602f0 2024-04-25T16:08:13.903 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.901+0000 7efdb0ff9640 1 -- 172.21.15.31:0/2360098445 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc0f3e60 con 0x556d271602f0 2024-04-25T16:08:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.902+0000 7efdb0ff9640 1 -- 172.21.15.31:0/2360098445 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7efdbc042d30 con 0x556d271602f0 2024-04-25T16:08:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.902+0000 7efdb0ff9640 1 -- 172.21.15.31:0/2360098445 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efdbc0e7060 con 0x556d271602f0 2024-04-25T16:08:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.902+0000 7efdda395740 10 client.26602 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:13.904 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.902+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=29}) -- 0x556d2752e650 con 0x556d271602f0 2024-04-25T16:08:13.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.904+0000 7efdb0ff9640 1 -- 172.21.15.31:0/2360098445 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7efdbc0d97d0 con 0x556d271602f0 2024-04-25T16:08:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.904+0000 7efdda395740 10 client.26602 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.904+0000 7efdda395740 10 client.26602 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.904+0000 7efdda395740 20 client.26602 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.904+0000 7efdda395740 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556d2711d1c0 0x556d275abc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.904+0000 7efdda395740 4 client.26602 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:08:13.906 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.904+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7efdc4006870 con 0x556d2711d1c0 2024-04-25T16:08:13.910 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.908+0000 7efdd0a84640 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556d2711d1c0 0x556d275abc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.911+0000 7efdd0a84640 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556d2711d1c0 0x556d275abc30 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd993417b3f4269c server_cookie=8f02e59b52574fb0 in_seq=0 out_seq=0 2024-04-25T16:08:13.914 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:13.911+0000 7efdb0ff9640 10 client.26602 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:14.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:13 smithi063 ceph-mon[32413]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 733 MiB/s wr, 1.25k op/s 2024-04-25T16:08:14.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:13 smithi186 ceph-mon[33258]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 733 MiB/s wr, 1.25k op/s 2024-04-25T16:08:14.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:13 smithi031 ceph-mon[29140]: pgmap v558: 97 pgs: 97 active+clean; 10 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 733 MiB/s wr, 1.25k op/s 2024-04-25T16:08:16.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:15 smithi063 ceph-mon[32413]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 996 MiB/s wr, 1.90k op/s 2024-04-25T16:08:16.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:15 smithi186 ceph-mon[33258]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 996 MiB/s wr, 1.90k op/s 2024-04-25T16:08:16.211 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:15 smithi031 ceph-mon[29140]: pgmap v559: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 996 MiB/s wr, 1.90k op/s 2024-04-25T16:08:18.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:17 smithi063 ceph-mon[32413]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 755 MiB/s wr, 1.60k op/s 2024-04-25T16:08:18.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:17 smithi186 ceph-mon[33258]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 755 MiB/s wr, 1.60k op/s 2024-04-25T16:08:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.197+0000 7efdb0ff9640 1 -- 172.21.15.31:0/2360098445 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7efdc4006870 con 0x556d2711d1c0 2024-04-25T16:08:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdb0ff9640 10 client.26602 handle_command_reply: tid=0 2024-04-25T16:08:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdda395740 1 client.26602 shutdown 2024-04-25T16:08:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 msgr2=0x556d27537ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdda395740 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27537ce0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdbc0d9500 tx=0x7efdbc03b450 comp rx=0 tx=0).stop 2024-04-25T16:08:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 shutdown_connections 2024-04-25T16:08:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdda395740 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556d2711d1c0 0x556d275abc30 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdda395740 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556d271891b0 0x556d27539150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.200 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdda395740 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556d271684d0 0x556d27538220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdda395740 1 --2- 172.21.15.31:0/2360098445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556d271602f0 0x556d27537ce0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 >> 172.21.15.31:0/2360098445 conn(0x556d2717d300 msgr2=0x556d27524e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.198+0000 7efdb0ff9640 0 client.26602 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.199+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 shutdown_connections 2024-04-25T16:08:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.199+0000 7efdda395740 1 -- 172.21.15.31:0/2360098445 wait complete. 2024-04-25T16:08:18.201 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.199+0000 7efdda395740 20 client.26602 trim_cache size 0 max 16384 2024-04-25T16:08:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:08:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:08:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:08:18.202 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:08:18.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:17 smithi031 ceph-mon[29140]: pgmap v560: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 755 MiB/s wr, 1.60k op/s 2024-04-25T16:08:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.207+0000 7efdd2287640 1 -- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdb40afc30 msgr2=0x7efdb40b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.207+0000 7efdd2287640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdb40afc30 0x7efdb40b2140 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x556d2752af00 tx=0x7efdc4006ce0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.207+0000 7efdd2287640 1 -- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc10a340 msgr2=0x7efdcc1a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:18.209 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.207+0000 7efdd2287640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc10a340 0x7efdcc1a07c0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7efdbc03ba40 tx=0x7efdbc004ac0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.212+0000 7efdd2287640 1 -- 172.21.15.31:0/104221444 shutdown_connections 2024-04-25T16:08:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.212+0000 7efdd2287640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdb40afc30 0x7efdb40b2140 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.212+0000 7efdd2287640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdcc10feb0 0x7efdcc1a7c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.212+0000 7efdd2287640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efdcc10b190 0x7efdcc1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.212+0000 7efdd2287640 1 --2- 172.21.15.31:0/104221444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efdcc10a340 0x7efdcc1a07c0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:18.213 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.212+0000 7efdd2287640 1 -- 172.21.15.31:0/104221444 >> 172.21.15.31:0/104221444 conn(0x7efdcc070330 msgr2=0x7efdcc10dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:18.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.217+0000 7efdd2287640 1 -- 172.21.15.31:0/104221444 shutdown_connections 2024-04-25T16:08:18.218 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:18.217+0000 7efdd2287640 1 -- 172.21.15.31:0/104221444 wait complete. 2024-04-25T16:08:20.007 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:19 smithi031 ceph-mon[29140]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 885 MiB/s wr, 2.02k op/s 2024-04-25T16:08:20.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:19 smithi063 ceph-mon[32413]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 885 MiB/s wr, 2.02k op/s 2024-04-25T16:08:20.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:19 smithi186 ceph-mon[33258]: pgmap v561: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 885 MiB/s wr, 2.02k op/s 2024-04-25T16:08:21.555 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:08:21.556 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:08:21.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:21 smithi031 ceph-mon[29140]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 807 MiB/s wr, 1.86k op/s 2024-04-25T16:08:22.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.012+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/4083685570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f806c860 msgr2=0x7fd8f806cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.012+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/4083685570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f806c860 0x7fd8f806cc70 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd8e8009860 tx=0x7fd8e802a360 comp rx=0 tx=0).stop 2024-04-25T16:08:22.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.016+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/4083685570 shutdown_connections 2024-04-25T16:08:22.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.016+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/4083685570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f8118a90 0x7fd8f811aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.016+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/4083685570 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f806d1b0 0x7fd8f8118550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.016+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/4083685570 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f806c860 0x7fd8f806cc70 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.016+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/4083685570 >> 172.21.15.31:0/4083685570 conn(0x7fd8f8072700 msgr2=0x7fd8f8109ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:22.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.022+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/4083685570 shutdown_connections 2024-04-25T16:08:22.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.024+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/4083685570 wait complete. 2024-04-25T16:08:22.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.026+0000 7fd8fc9a8640 1 Processor -- start 2024-04-25T16:08:22.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.027+0000 7fd8fc9a8640 1 -- start start 2024-04-25T16:08:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.027+0000 7fd8fc9a8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f806c860 0x7fd8f81a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.027+0000 7fd8fc9a8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f806d1b0 0x7fd8f81a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.027+0000 7fd8fc9a8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f8118a90 0x7fd8f81a7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.027+0000 7fd8fc9a8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f8112780 con 0x7fd8f8118a90 2024-04-25T16:08:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.027+0000 7fd8fc9a8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f8112600 con 0x7fd8f806c860 2024-04-25T16:08:22.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.027+0000 7fd8fc9a8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f8112900 con 0x7fd8f806d1b0 2024-04-25T16:08:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.028+0000 7fd8f6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f8118a90 0x7fd8f81a7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.028+0000 7fd8f6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f8118a90 0x7fd8f81a7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45722/0 (socket says 172.21.15.31:45722) 2024-04-25T16:08:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.028+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/1184121144 learned_addr learned my addr 172.21.15.31:0/1184121144 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.028+0000 7fd8f659c640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f806c860 0x7fd8f81a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.029+0000 7fd8f5d9b640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f806d1b0 0x7fd8f81a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.029+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/1184121144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f806d1b0 msgr2=0x7fd8f81a0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.029+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f806d1b0 0x7fd8f81a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.029+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/1184121144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f806c860 msgr2=0x7fd8f81a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.029+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f806c860 0x7fd8f81a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.029+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/1184121144 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8f81a8390 con 0x7fd8f8118a90 2024-04-25T16:08:22.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.029+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f8118a90 0x7fd8f81a7d20 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec01fcc0 tx=0x7fd8ec01d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.030+0000 7fd8e77fe640 1 -- 172.21.15.31:0/1184121144 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec01fdb0 con 0x7fd8f8118a90 2024-04-25T16:08:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.030+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/1184121144 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8f81a8680 con 0x7fd8f8118a90 2024-04-25T16:08:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.030+0000 7fd8e77fe640 1 -- 172.21.15.31:0/1184121144 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8ec0218f0 con 0x7fd8f8118a90 2024-04-25T16:08:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.030+0000 7fd8e77fe640 1 -- 172.21.15.31:0/1184121144 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec029920 con 0x7fd8f8118a90 2024-04-25T16:08:22.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.030+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/1184121144 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8f81a8bd0 con 0x7fd8f8118a90 2024-04-25T16:08:22.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.041+0000 7fd903856740 1 Processor -- start 2024-04-25T16:08:22.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.045+0000 7fd8e77fe640 1 -- 172.21.15.31:0/1184121144 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd8ec030020 con 0x7fd8f8118a90 2024-04-25T16:08:22.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.047+0000 7fd8e77fe640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afbf0 0x7fd8e00b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.048+0000 7fd8e77fe640 1 -- 172.21.15.31:0/1184121144 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd8ec0efe20 con 0x7fd8f8118a90 2024-04-25T16:08:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.048+0000 7fd903856740 1 -- start start 2024-04-25T16:08:22.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.048+0000 7fd903856740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d243841390 0x55d243be2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.048+0000 7fd903856740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 0x55d243be2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.048+0000 7fd903856740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2438206b0 0x55d243be3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.049+0000 7fd903856740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d243b47d80 con 0x55d2438184d0 2024-04-25T16:08:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.049+0000 7fd903856740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d243b018b0 con 0x55d2438206b0 2024-04-25T16:08:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.049+0000 7fd903856740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d243b43d40 con 0x55d243841390 2024-04-25T16:08:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.049+0000 7fd8f5d9b640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afbf0 0x7fd8e00b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.050+0000 7fd8f659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d243841390 0x55d243be2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.050+0000 7fd8f659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d243841390 0x55d243be2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53418/0 (socket says 172.21.15.31:53418) 2024-04-25T16:08:22.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.050+0000 7fd8f659c640 1 -- 172.21.15.31:0/891966620 learned_addr learned my addr 172.21.15.31:0/891966620 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.050+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2438206b0 0x55d243be3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.051+0000 7fd8f659c640 1 -- 172.21.15.31:0/891966620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2438206b0 msgr2=0x55d243be3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.051+0000 7fd8f659c640 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2438206b0 0x55d243be3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.051+0000 7fd8f659c640 1 -- 172.21.15.31:0/891966620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 msgr2=0x55d243be2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.051+0000 7fd8f5d9b640 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 0x55d243be2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.051+0000 7fd8f659c640 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 0x55d243be2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.051+0000 7fd8f659c640 1 -- 172.21.15.31:0/891966620 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d243ad2a40 con 0x55d243841390 2024-04-25T16:08:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.051+0000 7fd8f5d9b640 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 0x55d243be2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:08:22.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.053+0000 7fd8f659c640 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d243841390 0x55d243be2a60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd8f806e040 tx=0x7fd8e8030340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b5a0f37c5d21defc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:22.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.054+0000 7fd8e4ff9640 1 -- 172.21.15.31:0/891966620 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8041070 con 0x55d243841390 2024-04-25T16:08:22.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.054+0000 7fd8e4ff9640 1 -- 172.21.15.31:0/891966620 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8e8005040 con 0x55d243841390 2024-04-25T16:08:22.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.054+0000 7fd8e4ff9640 1 -- 172.21.15.31:0/891966620 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8e8037070 con 0x55d243841390 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.056+0000 7fd903856740 1 -- 172.21.15.31:0/891966620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d243841390 msgr2=0x55d243be2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.056+0000 7fd903856740 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d243841390 0x55d243be2a60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fd8f806e040 tx=0x7fd8e8030340 comp rx=0 tx=0).stop 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.057+0000 7fd903856740 1 -- 172.21.15.31:0/891966620 shutdown_connections 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.057+0000 7fd903856740 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2438206b0 0x55d243be3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.057+0000 7fd8f5d9b640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afbf0 0x7fd8e00b2100 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0005c40 tx=0x7fd8f0013820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.057+0000 7fd903856740 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 0x55d243be2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.057+0000 7fd903856740 1 --2- 172.21.15.31:0/891966620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d243841390 0x55d243be2a60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.057+0000 7fd903856740 1 -- 172.21.15.31:0/891966620 >> 172.21.15.31:0/891966620 conn(0x55d2438354e0 msgr2=0x55d243bde270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.057+0000 7fd903856740 1 -- 172.21.15.31:0/891966620 shutdown_connections 2024-04-25T16:08:22.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.057+0000 7fd903856740 1 -- 172.21.15.31:0/891966620 wait complete. 2024-04-25T16:08:22.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.059+0000 7fd903856740 1 -- start start 2024-04-25T16:08:22.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.059+0000 7fd903856740 1 Processor -- start 2024-04-25T16:08:22.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.060+0000 7fd903856740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 0x55d243bec8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.060+0000 7fd903856740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438206b0 0x55d243bece00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.060+0000 7fd903856740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d243841390 0x55d243bf3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.060+0000 7fd8f5d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438206b0 0x55d243bece00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.060+0000 7fd8f5d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438206b0 0x55d243bece00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53420/0 (socket says 172.21.15.31:53420) 2024-04-25T16:08:22.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.060+0000 7fd8f5d9b640 1 -- 172.21.15.31:0/3251909062 learned_addr learned my addr 172.21.15.31:0/3251909062 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:22.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.060+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d243841390 0x55d243bf3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.061+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d243be4720 con 0x55d2438184d0 2024-04-25T16:08:22.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.061+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d243aced10 con 0x55d243841390 2024-04-25T16:08:22.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.061+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d243be48a0 con 0x55d2438206b0 2024-04-25T16:08:22.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.061+0000 7fd8f659c640 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 0x55d243bec8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.061+0000 7fd8f5d9b640 1 -- 172.21.15.31:0/3251909062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d243841390 msgr2=0x55d243bf3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:22.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.061+0000 7fd8f5d9b640 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d243841390 0x55d243bf3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.061+0000 7fd8f5d9b640 1 -- 172.21.15.31:0/3251909062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 msgr2=0x55d243bec8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:22.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.061+0000 7fd8f5d9b640 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 0x55d243bec8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.061+0000 7fd8f5d9b640 1 -- 172.21.15.31:0/3251909062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2439237d0 con 0x55d2438206b0 2024-04-25T16:08:22.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.062+0000 7fd903856740 10 client.0 _ll_register_callbacks cb 0x55d243afb210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:22.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.062+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8e8005040 con 0x55d2438206b0 2024-04-25T16:08:22.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.064+0000 7fd8f5d9b640 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438206b0 0x55d243bece00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00096d0 tx=0x7fd8f0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=431d1e2e4cdd8ebc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:22.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.065+0000 7fd8ddffb640 10 client.26611 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:08:22.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.065+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3251909062 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8f0024030 con 0x55d2438206b0 2024-04-25T16:08:22.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.066+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3251909062 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd8f0004b20 con 0x55d2438206b0 2024-04-25T16:08:22.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.066+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3251909062 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd8f0004d20 con 0x55d2438206b0 2024-04-25T16:08:22.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.066+0000 7fd903856740 10 client.26611 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:22.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.067+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55d2438593f0 con 0x55d2438206b0 2024-04-25T16:08:22.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.067+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3251909062 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd8f0031060 con 0x55d2438206b0 2024-04-25T16:08:22.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.068+0000 7fd903856740 10 client.26611 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:22.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.068+0000 7fd903856740 10 client.26611 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:22.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.068+0000 7fd903856740 20 client.26611 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:22.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.068+0000 7fd903856740 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d2437fff60 0x55d243bf9fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.068+0000 7fd903856740 4 client.26611 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:22.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.068+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d243b55990 con 0x55d2437fff60 2024-04-25T16:08:22.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.069+0000 7fd8f659c640 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d2437fff60 0x55d243bf9fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.070+0000 7fd8ddffb640 10 client.26611 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:22.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.070+0000 7fd8f659c640 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d2437fff60 0x55d243bf9fd0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a32195e9576e44df server_cookie=42a5489ed7f16446 in_seq=0 out_seq=0 2024-04-25T16:08:22.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.078+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3251909062 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d243b55990 con 0x55d2437fff60 2024-04-25T16:08:22.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.078+0000 7fd8ddffb640 10 client.26611 handle_command_reply: tid=0 2024-04-25T16:08:22.081 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.080+0000 7fd903856740 1 client.26611 shutdown 2024-04-25T16:08:22.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.085+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438206b0 msgr2=0x55d243bece00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.085+0000 7fd903856740 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438206b0 0x55d243bece00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd8f00096d0 tx=0x7fd8f0004ab0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.085+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 shutdown_connections 2024-04-25T16:08:22.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.085+0000 7fd903856740 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d2437fff60 0x55d243bf9fd0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.085+0000 7fd903856740 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d243841390 0x55d243bf3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.085+0000 7fd8ddffb640 0 client.26611 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:22.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.086+0000 7fd903856740 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438206b0 0x55d243bece00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.086+0000 7fd903856740 1 --2- 172.21.15.31:0/3251909062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438184d0 0x55d243bec8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.086+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 >> 172.21.15.31:0/3251909062 conn(0x55d2438354e0 msgr2=0x55d243bdd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:22.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.086+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 shutdown_connections 2024-04-25T16:08:22.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.086+0000 7fd903856740 1 -- 172.21.15.31:0/3251909062 wait complete. 2024-04-25T16:08:22.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.086+0000 7fd903856740 20 client.26611 trim_cache size 0 max 16384 2024-04-25T16:08:22.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:21 smithi063 ceph-mon[32413]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 807 MiB/s wr, 1.86k op/s 2024-04-25T16:08:22.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.126+0000 7fd903856740 1 Processor -- start 2024-04-25T16:08:22.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.126+0000 7fd903856740 1 -- start start 2024-04-25T16:08:22.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.127+0000 7fd903856740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2437fff60 0x55d243beb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.127+0000 7fd903856740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438184d0 0x55d243beceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.127+0000 7fd903856740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243be7890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.128+0000 7fd903856740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81b19a0 con 0x55d2438206b0 2024-04-25T16:08:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.128+0000 7fd903856740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81b34e0 con 0x55d2437fff60 2024-04-25T16:08:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.128+0000 7fd903856740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81b2070 con 0x55d2438184d0 2024-04-25T16:08:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.128+0000 7fd8f5d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438184d0 0x55d243beceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.128+0000 7fd8f5d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438184d0 0x55d243beceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53430/0 (socket says 172.21.15.31:53430) 2024-04-25T16:08:22.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.128+0000 7fd8f5d9b640 1 -- 172.21.15.31:0/541640546 learned_addr learned my addr 172.21.15.31:0/541640546 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:22.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.128+0000 7fd8f659c640 1 --2- 172.21.15.31:0/541640546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2437fff60 0x55d243beb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.128+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/541640546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243be7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.129+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/541640546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438184d0 msgr2=0x55d243beceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.129+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/541640546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438184d0 0x55d243beceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.129+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/541640546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2437fff60 msgr2=0x55d243beb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.129+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/541640546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2437fff60 0x55d243beb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.129+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/541640546 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2438190f0 con 0x55d2438206b0 2024-04-25T16:08:22.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.130+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/541640546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243be7890 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0b5520 tx=0x7fd8ec01a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2089869b47a2de7a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:22.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.131+0000 7fd8dffff640 1 -- 172.21.15.31:0/541640546 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0d2070 con 0x55d2438206b0 2024-04-25T16:08:22.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.131+0000 7fd8dffff640 1 -- 172.21.15.31:0/541640546 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8ec0bc660 con 0x55d2438206b0 2024-04-25T16:08:22.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.131+0000 7fd8dffff640 1 -- 172.21.15.31:0/541640546 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0c0c80 con 0x55d2438206b0 2024-04-25T16:08:22.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.133+0000 7fd903856740 1 -- 172.21.15.31:0/541640546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 msgr2=0x55d243be7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.133+0000 7fd903856740 1 --2- 172.21.15.31:0/541640546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243be7890 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0b5520 tx=0x7fd8ec01a5f0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.134+0000 7fd903856740 1 -- 172.21.15.31:0/541640546 shutdown_connections 2024-04-25T16:08:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.134+0000 7fd903856740 1 --2- 172.21.15.31:0/541640546 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243be7890 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.134+0000 7fd903856740 1 --2- 172.21.15.31:0/541640546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2438184d0 0x55d243beceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.134+0000 7fd903856740 1 --2- 172.21.15.31:0/541640546 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2437fff60 0x55d243beb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.134+0000 7fd903856740 1 -- 172.21.15.31:0/541640546 >> 172.21.15.31:0/541640546 conn(0x55d2438354e0 msgr2=0x55d243be1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.134+0000 7fd903856740 1 -- 172.21.15.31:0/541640546 shutdown_connections 2024-04-25T16:08:22.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.134+0000 7fd903856740 1 -- 172.21.15.31:0/541640546 wait complete. 2024-04-25T16:08:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.134+0000 7fd903856740 1 -- start start 2024-04-25T16:08:22.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd903856740 1 Processor -- start 2024-04-25T16:08:22.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd903856740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2437fff60 0x55d243bf0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd903856740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2438184d0 0x55d243bf0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd903856740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243bf1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd903856740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81b2e10 con 0x55d2438206b0 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd903856740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81b2740 con 0x55d2438184d0 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd903856740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd8f81af980 con 0x55d2437fff60 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd903856740 10 client.0 _ll_register_callbacks cb 0x7fd8f00095c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd8f6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243bf1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd8f6d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243bf1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45770/0 (socket says 172.21.15.31:45770) 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.135+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/3641593699 learned_addr learned my addr 172.21.15.31:0/3641593699 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.136+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/3641593699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2437fff60 msgr2=0x55d243bf0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.136+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2437fff60 0x55d243bf0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.136+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/3641593699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2438184d0 msgr2=0x55d243bf0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.136+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2438184d0 0x55d243bf0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.136+0000 7fd8f6d9d640 1 -- 172.21.15.31:0/3641593699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d243b084b0 con 0x55d2438206b0 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.136+0000 7fd8f6d9d640 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243bf1c70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0b8240 tx=0x7fd8ec0b47c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86d9d55cf32899c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.136+0000 7fd8ddffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.136+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3641593699 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0d2070 con 0x55d2438206b0 2024-04-25T16:08:22.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.137+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3641593699 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd8ec0c0e90 con 0x55d2438206b0 2024-04-25T16:08:22.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.137+0000 7fd903856740 1 -- 172.21.15.31:0/3641593699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8ec0bc660 con 0x55d2438206b0 2024-04-25T16:08:22.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.137+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3641593699 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd8ec021aa0 con 0x55d2438206b0 2024-04-25T16:08:22.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.137+0000 7fd903856740 10 client.17007 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:22.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.137+0000 7fd903856740 1 -- 172.21.15.31:0/3641593699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d243bf3360 con 0x55d2438206b0 2024-04-25T16:08:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.137+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3641593699 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8ec0bc660 con 0x55d2438206b0 2024-04-25T16:08:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.138+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3641593699 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd8ec0bc200 con 0x55d2438206b0 2024-04-25T16:08:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.138+0000 7fd903856740 10 client.17007 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.138+0000 7fd903856740 10 client.17007 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.138+0000 7fd903856740 20 client.17007 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.138+0000 7fd903856740 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d243841390 0x55d243c647c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.138+0000 7fd903856740 4 client.17007 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:08:22.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.138+0000 7fd903856740 1 -- 172.21.15.31:0/3641593699 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d243b4f800 con 0x55d243841390 2024-04-25T16:08:22.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.147+0000 7fd8f659c640 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d243841390 0x55d243c647c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:21 smithi186 ceph-mon[33258]: pgmap v562: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 807 MiB/s wr, 1.86k op/s 2024-04-25T16:08:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.147+0000 7fd8f659c640 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d243841390 0x55d243c647c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f2f085814df1b51 server_cookie=2b13ce6716590e42 in_seq=0 out_seq=0 2024-04-25T16:08:22.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.147+0000 7fd8ddffb640 10 client.17007 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:22.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.151+0000 7fd8ddffb640 1 -- 172.21.15.31:0/3641593699 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d243b4f800 con 0x55d243841390 2024-04-25T16:08:22.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.151+0000 7fd8ddffb640 10 client.17007 handle_command_reply: tid=0 2024-04-25T16:08:22.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.151+0000 7fd903856740 1 client.17007 shutdown 2024-04-25T16:08:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.156+0000 7fd903856740 1 -- 172.21.15.31:0/3641593699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 msgr2=0x55d243bf1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.156+0000 7fd903856740 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243bf1c70 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec0b8240 tx=0x7fd8ec0b47c0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.159+0000 7fd903856740 1 -- 172.21.15.31:0/3641593699 shutdown_connections 2024-04-25T16:08:22.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.159+0000 7fd903856740 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d243841390 0x55d243c647c0 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.159+0000 7fd903856740 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d2438206b0 0x55d243bf1c70 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.159+0000 7fd903856740 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d2438184d0 0x55d243bf0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.159+0000 7fd8ddffb640 0 client.17007 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.159+0000 7fd903856740 1 --2- 172.21.15.31:0/3641593699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2437fff60 0x55d243bf0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.160+0000 7fd903856740 1 -- 172.21.15.31:0/3641593699 >> 172.21.15.31:0/3641593699 conn(0x55d2438354e0 msgr2=0x55d243be1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.160+0000 7fd903856740 1 -- 172.21.15.31:0/3641593699 shutdown_connections 2024-04-25T16:08:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.160+0000 7fd903856740 1 -- 172.21.15.31:0/3641593699 wait complete. 2024-04-25T16:08:22.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.160+0000 7fd903856740 20 client.17007 trim_cache size 0 max 16384 2024-04-25T16:08:22.165 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:08:22.165 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:08:22.165 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:08:22.165 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:08:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.165+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afbf0 msgr2=0x7fd8e00b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.165+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afbf0 0x7fd8e00b2100 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd8f0005c40 tx=0x7fd8f0013820 comp rx=0 tx=0).stop 2024-04-25T16:08:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.165+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f8118a90 msgr2=0x7fd8f81a7d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.165+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f8118a90 0x7fd8f81a7d20 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec01fcc0 tx=0x7fd8ec01d6b0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.165+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/1184121144 shutdown_connections 2024-04-25T16:08:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.166+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd8e00afbf0 0x7fd8e00b2100 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.166+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd8f8118a90 0x7fd8f81a7d20 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.166+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8f806d1b0 0x7fd8f81a0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.166+0000 7fd8fc9a8640 1 --2- 172.21.15.31:0/1184121144 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd8f806c860 0x7fd8f81a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.166+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/1184121144 >> 172.21.15.31:0/1184121144 conn(0x7fd8f8072700 msgr2=0x7fd8f810b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.166+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/1184121144 shutdown_connections 2024-04-25T16:08:22.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.166+0000 7fd8fc9a8640 1 -- 172.21.15.31:0/1184121144 wait complete. 2024-04-25T16:08:22.214 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:08:22.214 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:08:22.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.638+0000 7f7ecb706640 1 -- 172.21.15.31:0/1399555375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ec4110f50 msgr2=0x7f7ec4113430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.638+0000 7f7ecb706640 1 --2- 172.21.15.31:0/1399555375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ec4110f50 0x7f7ec4113430 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0009ca0 tx=0x7f7ec002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.651+0000 7f7ecb706640 1 -- 172.21.15.31:0/1399555375 shutdown_connections 2024-04-25T16:08:22.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.651+0000 7f7ecb706640 1 --2- 172.21.15.31:0/1399555375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ec4110f50 0x7f7ec4113430 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.651+0000 7f7ecb706640 1 --2- 172.21.15.31:0/1399555375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ec410e3e0 0x7f7ec41108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.651+0000 7f7ecb706640 1 --2- 172.21.15.31:0/1399555375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ec410b9f0 0x7f7ec410de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.651+0000 7f7ecb706640 1 -- 172.21.15.31:0/1399555375 >> 172.21.15.31:0/1399555375 conn(0x7f7ec40704e0 msgr2=0x7f7ec4072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:22.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.656+0000 7f7ecb706640 1 -- 172.21.15.31:0/1399555375 shutdown_connections 2024-04-25T16:08:22.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.656+0000 7f7ecb706640 1 -- 172.21.15.31:0/1399555375 wait complete. 2024-04-25T16:08:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.658+0000 7f7ecb706640 1 Processor -- start 2024-04-25T16:08:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.658+0000 7f7ecb706640 1 -- start start 2024-04-25T16:08:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.658+0000 7f7ecb706640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ec410b9f0 0x7f7ec41a9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.659+0000 7f7ecb706640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ec410e3e0 0x7f7ec41a9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.659+0000 7f7ecb706640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ec4110f50 0x7f7ec41b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.659+0000 7f7ecb706640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec4115d00 con 0x7f7ec4110f50 2024-04-25T16:08:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.659+0000 7f7ecb706640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec4115b80 con 0x7f7ec410b9f0 2024-04-25T16:08:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.659+0000 7f7ecb706640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec4115e80 con 0x7f7ec410e3e0 2024-04-25T16:08:22.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.659+0000 7f7ec9ca3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ec4110f50 0x7f7ec41b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.659+0000 7f7ec9ca3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ec4110f50 0x7f7ec41b04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:45798/0 (socket says 172.21.15.31:45798) 2024-04-25T16:08:22.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.659+0000 7f7ec9ca3640 1 -- 172.21.15.31:0/31636437 learned_addr learned my addr 172.21.15.31:0/31636437 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:22.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.660+0000 7f7ec94a2640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ec410b9f0 0x7f7ec41a9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.661+0000 7f7ec94a2640 1 -- 172.21.15.31:0/31636437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ec410e3e0 msgr2=0x7f7ec41a9560 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:08:22.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.661+0000 7f7ec94a2640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ec410e3e0 0x7f7ec41a9560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.661+0000 7f7ec94a2640 1 -- 172.21.15.31:0/31636437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ec4110f50 msgr2=0x7f7ec41b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:22.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.661+0000 7f7ec94a2640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ec4110f50 0x7f7ec41b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:22.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.661+0000 7f7ec94a2640 1 -- 172.21.15.31:0/31636437 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec41b0b60 con 0x7f7ec410b9f0 2024-04-25T16:08:22.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.662+0000 7f7ec94a2640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ec410b9f0 0x7f7ec41a9020 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7ec410a0b0 tx=0x7f7eb400cdd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:22.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.663+0000 7f7eba7fc640 1 -- 172.21.15.31:0/31636437 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb4007bb0 con 0x7f7ec410b9f0 2024-04-25T16:08:22.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.664+0000 7f7eba7fc640 1 -- 172.21.15.31:0/31636437 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7eb4007d50 con 0x7f7ec410b9f0 2024-04-25T16:08:22.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.664+0000 7f7eba7fc640 1 -- 172.21.15.31:0/31636437 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb4004da0 con 0x7f7ec410b9f0 2024-04-25T16:08:22.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.664+0000 7f7ecb706640 1 -- 172.21.15.31:0/31636437 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec41b0e50 con 0x7f7ec410b9f0 2024-04-25T16:08:22.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.664+0000 7f7ecb706640 1 -- 172.21.15.31:0/31636437 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec41b12f0 con 0x7f7ec410b9f0 2024-04-25T16:08:22.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.672+0000 7f7eba7fc640 1 -- 172.21.15.31:0/31636437 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7eb4004f60 con 0x7f7ec410b9f0 2024-04-25T16:08:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.674+0000 7f7ecb706640 1 -- 172.21.15.31:0/31636437 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ea80052c0 con 0x7f7ec410b9f0 2024-04-25T16:08:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.675+0000 7f7eba7fc640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7eb00afbf0 0x7f7eb00b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:22.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.675+0000 7f7eba7fc640 1 -- 172.21.15.31:0/31636437 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7eb40de880 con 0x7f7ec410b9f0 2024-04-25T16:08:22.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.684+0000 7f7ec8ca1640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7eb00afbf0 0x7f7eb00b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:22.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.685+0000 7f7eba7fc640 1 -- 172.21.15.31:0/31636437 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7eb4014030 con 0x7f7ec410b9f0 2024-04-25T16:08:22.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:22.697+0000 7f7ec8ca1640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7eb00afbf0 0x7f7eb00b2100 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7ec41aa0c0 tx=0x7f7ebc00b370 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:23.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.472+0000 7f7ecb706640 1 -- 172.21.15.31:0/31636437 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ea80055b0 con 0x7f7ec410b9f0 2024-04-25T16:08:23.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.474+0000 7f7eba7fc640 1 -- 172.21.15.31:0/31636437 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7f7eb408f030 con 0x7f7ec410b9f0 2024-04-25T16:08:23.475 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:08:23.476 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:08:23.480 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 -- 172.21.15.31:0/31636437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7eb00afbf0 msgr2=0x7f7eb00b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7eb00afbf0 0x7f7eb00b2100 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7ec41aa0c0 tx=0x7f7ebc00b370 comp rx=0 tx=0).stop 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 -- 172.21.15.31:0/31636437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ec410b9f0 msgr2=0x7f7ec41a9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ec410b9f0 0x7f7ec41a9020 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7ec410a0b0 tx=0x7f7eb400cdd0 comp rx=0 tx=0).stop 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 -- 172.21.15.31:0/31636437 shutdown_connections 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7eb00afbf0 0x7f7eb00b2100 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7ec4110f50 0x7f7ec41b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ec410e3e0 0x7f7ec41a9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 --2- 172.21.15.31:0/31636437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7ec410b9f0 0x7f7ec41a9020 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:23.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 -- 172.21.15.31:0/31636437 >> 172.21.15.31:0/31636437 conn(0x7f7ec40704e0 msgr2=0x7f7ec410c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 -- 172.21.15.31:0/31636437 shutdown_connections 2024-04-25T16:08:23.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.490+0000 7f7eaffff640 1 -- 172.21.15.31:0/31636437 wait complete. 2024-04-25T16:08:23.525 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:08:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.998+0000 7fc78647e640 1 -- 172.21.15.31:0/2088677905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc78010fe90 msgr2=0x7fc7801102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.998+0000 7fc777fff640 1 -- 172.21.15.31:0/2088677905 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc770036070 con 0x7fc78010fe90 2024-04-25T16:08:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:23.998+0000 7fc78647e640 1 --2- 172.21.15.31:0/2088677905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc78010fe90 0x7fc7801102a0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fc770009900 tx=0x7fc77002a420 comp rx=0 tx=0).stop 2024-04-25T16:08:24.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.006+0000 7fc78647e640 1 -- 172.21.15.31:0/2088677905 shutdown_connections 2024-04-25T16:08:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.006+0000 7fc78647e640 1 --2- 172.21.15.31:0/2088677905 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc78010fe90 0x7fc7801102a0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.006+0000 7fc78647e640 1 --2- 172.21.15.31:0/2088677905 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc78010b190 0x7fc78010b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.006+0000 7fc78647e640 1 --2- 172.21.15.31:0/2088677905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc78010a340 0x7fc78010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.006+0000 7fc78647e640 1 -- 172.21.15.31:0/2088677905 >> 172.21.15.31:0/2088677905 conn(0x7fc780070320 msgr2=0x7fc7800727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.007+0000 7fc78647e640 1 -- 172.21.15.31:0/2088677905 shutdown_connections 2024-04-25T16:08:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.007+0000 7fc78647e640 1 -- 172.21.15.31:0/2088677905 wait complete. 2024-04-25T16:08:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.008+0000 7fc78647e640 1 Processor -- start 2024-04-25T16:08:24.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.008+0000 7fc78647e640 1 -- start start 2024-04-25T16:08:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.010+0000 7fc78647e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc78010a340 0x7fc7801a8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.010+0000 7fc78647e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc78010b190 0x7fc7801a9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.010+0000 7fc78647e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc78010fe90 0x7fc7801b02f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.010+0000 7fc78647e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc78011e280 con 0x7fc78010a340 2024-04-25T16:08:24.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.010+0000 7fc78647e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc78011e100 con 0x7fc78010fe90 2024-04-25T16:08:24.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.010+0000 7fc78647e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc78011e400 con 0x7fc78010b190 2024-04-25T16:08:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.011+0000 7fc784c7b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc78010b190 0x7fc7801a9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.012+0000 7fc784c7b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc78010b190 0x7fc7801a9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53798/0 (socket says 172.21.15.31:53798) 2024-04-25T16:08:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.012+0000 7fc784c7b640 1 -- 172.21.15.31:0/1269475075 learned_addr learned my addr 172.21.15.31:0/1269475075 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:24.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.012+0000 7fc78547c640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc78010a340 0x7fc7801a8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.013+0000 7fc784c7b640 1 -- 172.21.15.31:0/1269475075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc78010fe90 msgr2=0x7fc7801b02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.013+0000 7fc784c7b640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc78010fe90 0x7fc7801b02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.013+0000 7fc784c7b640 1 -- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc78010a340 msgr2=0x7fc7801a8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.013+0000 7fc784c7b640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc78010a340 0x7fc7801a8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.013+0000 7fc784c7b640 1 -- 172.21.15.31:0/1269475075 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7801b0960 con 0x7fc78010b190 2024-04-25T16:08:24.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.014+0000 7fc784c7b640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc78010b190 0x7fc7801a9360 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00c890 tx=0x7fc77c00cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.019+0000 7fc7767fc640 1 -- 172.21.15.31:0/1269475075 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c004220 con 0x7fc78010b190 2024-04-25T16:08:24.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.019+0000 7fc78647e640 1 -- 172.21.15.31:0/1269475075 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7801b0c50 con 0x7fc78010b190 2024-04-25T16:08:24.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.019+0000 7fc78647e640 1 -- 172.21.15.31:0/1269475075 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7801b11a0 con 0x7fc78010b190 2024-04-25T16:08:24.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.019+0000 7fc7767fc640 1 -- 172.21.15.31:0/1269475075 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc77c0043c0 con 0x7fc78010b190 2024-04-25T16:08:24.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.020+0000 7fc7767fc640 1 -- 172.21.15.31:0/1269475075 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77c004a00 con 0x7fc78010b190 2024-04-25T16:08:24.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.027+0000 7fc7767fc640 1 -- 172.21.15.31:0/1269475075 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc77c004ba0 con 0x7fc78010b190 2024-04-25T16:08:24.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.028+0000 7fc7767fc640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc76c0afd00 0x7fc76c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.029+0000 7fc78547c640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc76c0afd00 0x7fc76c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.029+0000 7fc7767fc640 1 -- 172.21.15.31:0/1269475075 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc77c0df8c0 con 0x7fc78010b190 2024-04-25T16:08:24.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.031+0000 7fc78547c640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc76c0afd00 0x7fc76c0b2210 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc780109c10 tx=0x7fc778009870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.031+0000 7fc78e58b740 1 Processor -- start 2024-04-25T16:08:24.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.031+0000 7fc78e58b740 1 -- start start 2024-04-25T16:08:24.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.039+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07d07390 0x557a080a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.039+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.040+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 0x557a080aabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.040+0000 7fc78e58b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557a0800dd80 con 0x557a07cde4d0 2024-04-25T16:08:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.040+0000 7fc78e58b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557a07fc7a20 con 0x557a07d07390 2024-04-25T16:08:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.040+0000 7fc78e58b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557a08009d40 con 0x557a07ce66b0 2024-04-25T16:08:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.041+0000 7fc78547c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07d07390 0x557a080a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.041+0000 7fc78547c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07d07390 0x557a080a99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56966/0 (socket says 172.21.15.31:56966) 2024-04-25T16:08:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.041+0000 7fc78547c640 1 -- 172.21.15.31:0/2031382411 learned_addr learned my addr 172.21.15.31:0/2031382411 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.043+0000 7fc784c7b640 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.043+0000 7fc785c7d640 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 0x557a080aabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.044+0000 7fc78547c640 1 -- 172.21.15.31:0/2031382411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 msgr2=0x557a080aabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.044+0000 7fc78547c640 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 0x557a080aabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.044+0000 7fc78547c640 1 -- 172.21.15.31:0/2031382411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 msgr2=0x557a080a9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.044+0000 7fc78547c640 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080a9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.044+0000 7fc78547c640 1 -- 172.21.15.31:0/2031382411 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a07fd2c30 con 0x557a07d07390 2024-04-25T16:08:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.044+0000 7fc785c7d640 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 0x557a080aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:08:24.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.044+0000 7fc784c7b640 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:08:24.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.046+0000 7fc78547c640 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07d07390 0x557a080a99e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7780125e0 tx=0x7fc778013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bbb60fc06ccb398 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.049+0000 7fc76b7fe640 1 -- 172.21.15.31:0/2031382411 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778011720 con 0x557a07d07390 2024-04-25T16:08:24.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.049+0000 7fc76b7fe640 1 -- 172.21.15.31:0/2031382411 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc778024070 con 0x557a07d07390 2024-04-25T16:08:24.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.049+0000 7fc76b7fe640 1 -- 172.21.15.31:0/2031382411 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc77801fd50 con 0x557a07d07390 2024-04-25T16:08:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.051+0000 7fc78e58b740 1 -- 172.21.15.31:0/2031382411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07d07390 msgr2=0x557a080a99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.051+0000 7fc78e58b740 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07d07390 0x557a080a99e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc7780125e0 tx=0x7fc778013000 comp rx=0 tx=0).stop 2024-04-25T16:08:24.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.055+0000 7fc78e58b740 1 -- 172.21.15.31:0/2031382411 shutdown_connections 2024-04-25T16:08:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.055+0000 7fc78e58b740 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 0x557a080aabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.055+0000 7fc78e58b740 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080a9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.055+0000 7fc78e58b740 1 --2- 172.21.15.31:0/2031382411 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07d07390 0x557a080a99e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.055+0000 7fc78e58b740 1 -- 172.21.15.31:0/2031382411 >> 172.21.15.31:0/2031382411 conn(0x557a07cfb4e0 msgr2=0x557a080a51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.055+0000 7fc78e58b740 1 -- 172.21.15.31:0/2031382411 shutdown_connections 2024-04-25T16:08:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.056+0000 7fc78e58b740 1 -- 172.21.15.31:0/2031382411 wait complete. 2024-04-25T16:08:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.057+0000 7fc78e58b740 1 -- start start 2024-04-25T16:08:24.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.057+0000 7fc78e58b740 1 Processor -- start 2024-04-25T16:08:24.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.060+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.061+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07ce66b0 0x557a080c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.061+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 0x557a080cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.062+0000 7fc784c7b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07ce66b0 0x557a080c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.063+0000 7fc784c7b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07ce66b0 0x557a080c4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56970/0 (socket says 172.21.15.31:56970) 2024-04-25T16:08:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.063+0000 7fc78547c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.064+0000 7fc78547c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080c47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47206/0 (socket says 172.21.15.31:47206) 2024-04-25T16:08:24.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.065+0000 7fc784c7b640 1 -- 172.21.15.31:0/3931824755 learned_addr learned my addr 172.21.15.31:0/3931824755 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.066+0000 7fc785c7d640 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 0x557a080cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.067+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557a080ab700 con 0x557a07cde4d0 2024-04-25T16:08:24.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.067+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557a07f94dc0 con 0x557a07ce66b0 2024-04-25T16:08:24.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.069+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557a080ab880 con 0x557a07d07390 2024-04-25T16:08:24.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.071+0000 7fc78e58b740 10 client.0 _ll_register_callbacks cb 0x557a07fb0600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.076+0000 7fc785c7d640 1 -- 172.21.15.31:0/3931824755 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07ce66b0 msgr2=0x557a080c4d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.076+0000 7fc785c7d640 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07ce66b0 0x557a080c4d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.076+0000 7fc785c7d640 1 -- 172.21.15.31:0/3931824755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 msgr2=0x557a080c47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.076+0000 7fc785c7d640 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080c47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.076+0000 7fc785c7d640 1 -- 172.21.15.31:0/3931824755 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a07d1f3f0 con 0x557a07d07390 2024-04-25T16:08:24.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.078+0000 7fc785c7d640 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 0x557a080cbc80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc770034040 tx=0x7fc770004530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9b4fbbe8b26b05e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.079+0000 7fc768ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:08:24.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.081+0000 7fc768ff9640 1 -- 172.21.15.31:0/3931824755 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc770041020 con 0x557a07d07390 2024-04-25T16:08:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.082+0000 7fc768ff9640 1 -- 172.21.15.31:0/3931824755 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc770005450 con 0x557a07d07390 2024-04-25T16:08:24.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.082+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc778024070 con 0x557a07d07390 2024-04-25T16:08:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.083+0000 7fc768ff9640 1 -- 172.21.15.31:0/3931824755 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc770036070 con 0x557a07d07390 2024-04-25T16:08:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.083+0000 7fc78e58b740 10 client.26626 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:24.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.083+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557a07de97d0 con 0x557a07d07390 2024-04-25T16:08:24.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.084+0000 7fc768ff9640 1 -- 172.21.15.31:0/3931824755 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc778024070 con 0x557a07d07390 2024-04-25T16:08:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.086+0000 7fc768ff9640 1 -- 172.21.15.31:0/3931824755 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc770004970 con 0x557a07d07390 2024-04-25T16:08:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.087+0000 7fc78e58b740 10 client.26626 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.087+0000 7fc78e58b740 10 client.26626 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:08:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.087+0000 7fc78e58b740 20 client.26626 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.087+0000 7fc78e58b740 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557a07cc5f60 0x557a080d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.088+0000 7fc78e58b740 4 client.26626 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.088+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a0801b990 con 0x557a07cc5f60 2024-04-25T16:08:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.088+0000 7fc78547c640 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557a07cc5f60 0x557a080d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.089+0000 7fc78547c640 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557a07cc5f60 0x557a080d2840 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2e8a41f6b3b8e7ca server_cookie=c64bfd7234abf4d8 in_seq=0 out_seq=0 2024-04-25T16:08:24.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.089+0000 7fc768ff9640 10 client.26626 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:08:24.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.091+0000 7fc768ff9640 1 -- 172.21.15.31:0/3931824755 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557a0801b990 con 0x557a07cc5f60 2024-04-25T16:08:24.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.091+0000 7fc768ff9640 10 client.26626 handle_command_reply: tid=0 2024-04-25T16:08:24.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.091+0000 7fc78e58b740 1 client.26626 shutdown 2024-04-25T16:08:24.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.091+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 msgr2=0x557a080cbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.091+0000 7fc78e58b740 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 0x557a080cbc80 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc770034040 tx=0x7fc770004530 comp rx=0 tx=0).stop 2024-04-25T16:08:24.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.092+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 shutdown_connections 2024-04-25T16:08:24.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.092+0000 7fc78e58b740 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557a07cc5f60 0x557a080d2840 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.092+0000 7fc78e58b740 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 0x557a080cbc80 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.093+0000 7fc78e58b740 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07ce66b0 0x557a080c4d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.093+0000 7fc78e58b740 1 --2- 172.21.15.31:0/3931824755 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07cde4d0 0x557a080c47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.093+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 >> 172.21.15.31:0/3931824755 conn(0x557a07cfb4e0 msgr2=0x557a080a43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:24.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.093+0000 7fc768ff9640 0 client.26626 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:08:24.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.095+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 shutdown_connections 2024-04-25T16:08:24.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.097+0000 7fc78e58b740 1 -- 172.21.15.31:0/3931824755 wait complete. 2024-04-25T16:08:24.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.098+0000 7fc78e58b740 20 client.26626 trim_cache size 0 max 16384 2024-04-25T16:08:24.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:23 smithi063 ceph-mon[32413]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 753 MiB/s wr, 1.75k op/s 2024-04-25T16:08:24.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:23 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/31636437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:08:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.138+0000 7fc78e58b740 1 Processor -- start 2024-04-25T16:08:24.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.138+0000 7fc78e58b740 1 -- start start 2024-04-25T16:08:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080d7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07ce66b0 0x557a080d7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 0x557a080d7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc78547c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080d7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc78e58b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc76400e380 con 0x557a07ce66b0 2024-04-25T16:08:24.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc78e58b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc76400fec0 con 0x557a07cde4d0 2024-04-25T16:08:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc78e58b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc76400ea50 con 0x557a07d07390 2024-04-25T16:08:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc78547c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080d7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56982/0 (socket says 172.21.15.31:56982) 2024-04-25T16:08:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc78547c640 1 -- 172.21.15.31:0/1179755830 learned_addr learned my addr 172.21.15.31:0/1179755830 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.139+0000 7fc785c7d640 1 --2- 172.21.15.31:0/1179755830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 0x557a080d7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.140+0000 7fc78547c640 1 -- 172.21.15.31:0/1179755830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 msgr2=0x557a080d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.140+0000 7fc78547c640 1 --2- 172.21.15.31:0/1179755830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 0x557a080d7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.140+0000 7fc78547c640 1 -- 172.21.15.31:0/1179755830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07ce66b0 msgr2=0x557a080d7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.140+0000 7fc78547c640 1 --2- 172.21.15.31:0/1179755830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07ce66b0 0x557a080d7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.140+0000 7fc78547c640 1 -- 172.21.15.31:0/1179755830 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a080c5300 con 0x557a07cde4d0 2024-04-25T16:08:24.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.140+0000 7fc78547c640 1 --2- 172.21.15.31:0/1179755830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080d7030 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc778005fe0 tx=0x7fc77801f560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c02e08cb1bd1c14c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:23 smithi186 ceph-mon[33258]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 753 MiB/s wr, 1.75k op/s 2024-04-25T16:08:24.148 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:23 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/31636437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:08:24.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.149+0000 7fc76affd640 1 -- 172.21.15.31:0/1179755830 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778016040 con 0x557a07cde4d0 2024-04-25T16:08:24.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.149+0000 7fc76affd640 1 -- 172.21.15.31:0/1179755830 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc778024050 con 0x557a07cde4d0 2024-04-25T16:08:24.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.149+0000 7fc76affd640 1 -- 172.21.15.31:0/1179755830 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778036120 con 0x557a07cde4d0 2024-04-25T16:08:24.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.153+0000 7fc78e58b740 1 -- 172.21.15.31:0/1179755830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 msgr2=0x557a080d7030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.153+0000 7fc78e58b740 1 --2- 172.21.15.31:0/1179755830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080d7030 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc778005fe0 tx=0x7fc77801f560 comp rx=0 tx=0).stop 2024-04-25T16:08:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.159+0000 7fc78e58b740 1 -- 172.21.15.31:0/1179755830 shutdown_connections 2024-04-25T16:08:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.159+0000 7fc78e58b740 1 --2- 172.21.15.31:0/1179755830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07d07390 0x557a080d7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.159+0000 7fc78e58b740 1 --2- 172.21.15.31:0/1179755830 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07ce66b0 0x557a080d7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.159+0000 7fc78e58b740 1 --2- 172.21.15.31:0/1179755830 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080d7030 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.159+0000 7fc78e58b740 1 -- 172.21.15.31:0/1179755830 >> 172.21.15.31:0/1179755830 conn(0x557a07cfb4e0 msgr2=0x557a080ae4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.164+0000 7fc78e58b740 1 -- 172.21.15.31:0/1179755830 shutdown_connections 2024-04-25T16:08:24.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.164+0000 7fc78e58b740 1 -- 172.21.15.31:0/1179755830 wait complete. 2024-04-25T16:08:24.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.165+0000 7fc78e58b740 1 -- start start 2024-04-25T16:08:24.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.165+0000 7fc78e58b740 1 Processor -- start 2024-04-25T16:08:24.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.171+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.171+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 0x557a080c8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.171+0000 7fc78e58b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07d07390 0x557a08138be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.171+0000 7fc78e58b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc76400f7f0 con 0x557a07d07390 2024-04-25T16:08:24.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.171+0000 7fc78e58b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc76400f120 con 0x557a07cde4d0 2024-04-25T16:08:24.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.171+0000 7fc78e58b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc76400c410 con 0x557a07ce66b0 2024-04-25T16:08:24.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.171+0000 7fc78e58b740 10 client.0 _ll_register_callbacks cb 0x7fc770034070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:24.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.173+0000 7fc78547c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.173+0000 7fc784c7b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 0x557a080c8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.173+0000 7fc785c7d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07d07390 0x557a08138be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.173+0000 7fc78547c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56992/0 (socket says 172.21.15.31:56992) 2024-04-25T16:08:24.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.173+0000 7fc78547c640 1 -- 172.21.15.31:0/90899506 learned_addr learned my addr 172.21.15.31:0/90899506 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:24.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.174+0000 7fc78547c640 1 -- 172.21.15.31:0/90899506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 msgr2=0x557a080c8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.174+0000 7fc78547c640 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 0x557a080c8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.174+0000 7fc78547c640 1 -- 172.21.15.31:0/90899506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07d07390 msgr2=0x557a08138be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.174+0000 7fc78547c640 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07d07390 0x557a08138be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.174+0000 7fc78547c640 1 -- 172.21.15.31:0/90899506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a08139120 con 0x557a07cde4d0 2024-04-25T16:08:24.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.175+0000 7fc78547c640 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080c82d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc77800a010 tx=0x7fc778010ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=766c5f0d2ff9241f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.178+0000 7fc768ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:08:24.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.178+0000 7fc768ff9640 1 -- 172.21.15.31:0/90899506 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc778040060 con 0x557a07cde4d0 2024-04-25T16:08:24.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.178+0000 7fc768ff9640 1 -- 172.21.15.31:0/90899506 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc778011520 con 0x557a07cde4d0 2024-04-25T16:08:24.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.179+0000 7fc78e58b740 1 -- 172.21.15.31:0/90899506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc778024050 con 0x557a07cde4d0 2024-04-25T16:08:24.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.183+0000 7fc768ff9640 1 -- 172.21.15.31:0/90899506 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc778016040 con 0x557a07cde4d0 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.187+0000 7fc78e58b740 10 client.26666 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.187+0000 7fc78e58b740 1 -- 172.21.15.31:0/90899506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557a0813a300 con 0x557a07cde4d0 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.189+0000 7fc768ff9640 1 -- 172.21.15.31:0/90899506 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc778024050 con 0x557a07cde4d0 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.189+0000 7fc768ff9640 1 -- 172.21.15.31:0/90899506 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc77800f040 con 0x557a07cde4d0 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.190+0000 7fc78e58b740 10 client.26666 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.190+0000 7fc78e58b740 10 client.26666 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.190+0000 7fc78e58b740 20 client.26666 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.190+0000 7fc78e58b740 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557a07c9b3a0 0x557a0813c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.190+0000 7fc78e58b740 4 client.26666 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:08:24.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.190+0000 7fc78e58b740 1 -- 172.21.15.31:0/90899506 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557a08015800 con 0x557a07c9b3a0 2024-04-25T16:08:24.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:23 smithi031 ceph-mon[29140]: pgmap v563: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 753 MiB/s wr, 1.75k op/s 2024-04-25T16:08:24.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:23 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/31636437' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:08:24.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.191+0000 7fc784c7b640 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557a07c9b3a0 0x557a0813c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.208+0000 7fc784c7b640 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557a07c9b3a0 0x557a0813c9e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c43ee7e4eadc4f56 server_cookie=4a9c5861926cc3d0 in_seq=0 out_seq=0 2024-04-25T16:08:24.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.208+0000 7fc768ff9640 10 client.26666 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:08:24.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.216+0000 7fc768ff9640 1 -- 172.21.15.31:0/90899506 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557a08015800 con 0x557a07c9b3a0 2024-04-25T16:08:24.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.216+0000 7fc768ff9640 10 client.26666 handle_command_reply: tid=0 2024-04-25T16:08:24.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.218+0000 7fc78e58b740 1 client.26666 shutdown 2024-04-25T16:08:24.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.220+0000 7fc78e58b740 1 -- 172.21.15.31:0/90899506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 msgr2=0x557a080c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.220+0000 7fc78e58b740 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080c82d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc77800a010 tx=0x7fc778010ee0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.220+0000 7fc78e58b740 1 -- 172.21.15.31:0/90899506 shutdown_connections 2024-04-25T16:08:24.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.220+0000 7fc78e58b740 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557a07c9b3a0 0x557a0813c9e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.220+0000 7fc768ff9640 0 client.26666 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:08:24.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.220+0000 7fc78e58b740 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a07d07390 0x557a08138be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.220+0000 7fc78e58b740 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a07ce66b0 0x557a080c8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.220+0000 7fc78e58b740 1 --2- 172.21.15.31:0/90899506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a07cde4d0 0x557a080c82d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.220+0000 7fc78e58b740 1 -- 172.21.15.31:0/90899506 >> 172.21.15.31:0/90899506 conn(0x557a07cfb4e0 msgr2=0x557a080a7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:24.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.221+0000 7fc78e58b740 1 -- 172.21.15.31:0/90899506 shutdown_connections 2024-04-25T16:08:24.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.221+0000 7fc78e58b740 1 -- 172.21.15.31:0/90899506 wait complete. 2024-04-25T16:08:24.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.221+0000 7fc78e58b740 20 client.26666 trim_cache size 0 max 16384 2024-04-25T16:08:24.230 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:08:24.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.237+0000 7fc78647e640 1 -- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc76c0afd00 msgr2=0x7fc76c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.237+0000 7fc78647e640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc76c0afd00 0x7fc76c0b2210 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc780109c10 tx=0x7fc778009870 comp rx=0 tx=0).stop 2024-04-25T16:08:24.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.237+0000 7fc78647e640 1 -- 172.21.15.31:0/1269475075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc78010b190 msgr2=0x7fc7801a9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.237+0000 7fc78647e640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc78010b190 0x7fc7801a9360 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00c890 tx=0x7fc77c00cd50 comp rx=0 tx=0).stop 2024-04-25T16:08:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.245+0000 7fc78647e640 1 -- 172.21.15.31:0/1269475075 shutdown_connections 2024-04-25T16:08:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.245+0000 7fc78647e640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc76c0afd00 0x7fc76c0b2210 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.245+0000 7fc78647e640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc78010fe90 0x7fc7801b02f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.245+0000 7fc78647e640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc78010b190 0x7fc7801a9360 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.245+0000 7fc78647e640 1 --2- 172.21.15.31:0/1269475075 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc78010a340 0x7fc7801a8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.245+0000 7fc78647e640 1 -- 172.21.15.31:0/1269475075 >> 172.21.15.31:0/1269475075 conn(0x7fc780070320 msgr2=0x7fc78010be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:24.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.247+0000 7fc78647e640 1 -- 172.21.15.31:0/1269475075 shutdown_connections 2024-04-25T16:08:24.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.252+0000 7fc78647e640 1 -- 172.21.15.31:0/1269475075 wait complete. 2024-04-25T16:08:24.319 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:08:24.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.813+0000 7f50e700c640 1 -- 172.21.15.31:0/3265694757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e010a350 msgr2=0x7f50e010ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.813+0000 7f50e700c640 1 --2- 172.21.15.31:0/3265694757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e010a350 0x7f50e010ac90 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009900 tx=0x7f50d002a440 comp rx=0 tx=0).stop 2024-04-25T16:08:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.822+0000 7f50e700c640 1 -- 172.21.15.31:0/3265694757 shutdown_connections 2024-04-25T16:08:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.822+0000 7f50e700c640 1 --2- 172.21.15.31:0/3265694757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50e010fef0 0x7f50e0110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.822+0000 7f50e700c640 1 --2- 172.21.15.31:0/3265694757 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50e010b1d0 0x7f50e010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.822+0000 7f50e700c640 1 --2- 172.21.15.31:0/3265694757 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e010a350 0x7f50e010ac90 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.822+0000 7f50e700c640 1 -- 172.21.15.31:0/3265694757 >> 172.21.15.31:0/3265694757 conn(0x7f50e0070400 msgr2=0x7f50e0072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:24.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.826+0000 7f50e700c640 1 -- 172.21.15.31:0/3265694757 shutdown_connections 2024-04-25T16:08:24.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.829+0000 7f50e700c640 1 -- 172.21.15.31:0/3265694757 wait complete. 2024-04-25T16:08:24.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.830+0000 7f50e700c640 1 Processor -- start 2024-04-25T16:08:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.834+0000 7f50e700c640 1 -- start start 2024-04-25T16:08:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.834+0000 7f50e700c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50e010a350 0x7f50e0117100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.834+0000 7f50e700c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e010b1d0 0x7f50e0117640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.834+0000 7f50e700c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50e010fef0 0x7f50e01185a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.834+0000 7f50e700c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f50e011e260 con 0x7f50e010b1d0 2024-04-25T16:08:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.834+0000 7f50e700c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f50e011e0e0 con 0x7f50e010a350 2024-04-25T16:08:24.837 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.834+0000 7f50e700c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f50e011e3e0 con 0x7f50e010fef0 2024-04-25T16:08:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.835+0000 7f50e600a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50e010a350 0x7f50e0117100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.835+0000 7f50e600a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50e010a350 0x7f50e0117100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56998/0 (socket says 172.21.15.31:56998) 2024-04-25T16:08:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.835+0000 7f50e600a640 1 -- 172.21.15.31:0/3692675247 learned_addr learned my addr 172.21.15.31:0/3692675247 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.836+0000 7f50e600a640 1 -- 172.21.15.31:0/3692675247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50e010fef0 msgr2=0x7f50e01185a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:08:24.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.836+0000 7f50e600a640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50e010fef0 0x7f50e01185a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.836+0000 7f50e600a640 1 -- 172.21.15.31:0/3692675247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e010b1d0 msgr2=0x7f50e0117640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.836+0000 7f50e600a640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e010b1d0 0x7f50e0117640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.836+0000 7f50e600a640 1 -- 172.21.15.31:0/3692675247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e0118c10 con 0x7f50e010a350 2024-04-25T16:08:24.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.837+0000 7f50e600a640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50e010a350 0x7f50e0117100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009a20 tx=0x7f50d0004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.837+0000 7f50d77fe640 1 -- 172.21.15.31:0/3692675247 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0031600 con 0x7f50e010a350 2024-04-25T16:08:24.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.838+0000 7f50e700c640 1 -- 172.21.15.31:0/3692675247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e0118ea0 con 0x7f50e010a350 2024-04-25T16:08:24.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.838+0000 7f50e700c640 1 -- 172.21.15.31:0/3692675247 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e0119410 con 0x7f50e010a350 2024-04-25T16:08:24.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.838+0000 7f50d77fe640 1 -- 172.21.15.31:0/3692675247 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50d0030e40 con 0x7f50e010a350 2024-04-25T16:08:24.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.838+0000 7f50d77fe640 1 -- 172.21.15.31:0/3692675247 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d0045410 con 0x7f50e010a350 2024-04-25T16:08:24.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.850+0000 7f50ef1ff740 1 Processor -- start 2024-04-25T16:08:24.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.850+0000 7f50d77fe640 1 -- 172.21.15.31:0/3692675247 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f50d0043020 con 0x7f50e010a350 2024-04-25T16:08:24.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.855+0000 7f50ef1ff740 1 -- start start 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.855+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acd11390 0x55a8ad0b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.855+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acce84d0 0x55a8ad0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.855+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8accf06b0 0x55a8ad0b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.855+0000 7f50ef1ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a8ad017d80 con 0x55a8accf06b0 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.855+0000 7f50ef1ff740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a8acfd1a20 con 0x55a8acce84d0 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.855+0000 7f50ef1ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a8ad013d40 con 0x55a8acd11390 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.856+0000 7f50e600a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acce84d0 0x55a8ad0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.856+0000 7f50e600a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acce84d0 0x55a8ad0b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57000/0 (socket says 172.21.15.31:57000) 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.856+0000 7f50e600a640 1 -- 172.21.15.31:0/4052074348 learned_addr learned my addr 172.21.15.31:0/4052074348 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.856+0000 7f50d77fe640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50cc0afd00 0x7f50cc0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.857+0000 7f50e680b640 1 --2- 172.21.15.31:0/4052074348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acd11390 0x55a8ad0b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.857+0000 7f50e600a640 1 -- 172.21.15.31:0/4052074348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acd11390 msgr2=0x55a8ad0b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.857+0000 7f50e600a640 1 --2- 172.21.15.31:0/4052074348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acd11390 0x55a8ad0b3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.857+0000 7f50e600a640 1 -- 172.21.15.31:0/4052074348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8accf06b0 msgr2=0x55a8ad0b4c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:08:24.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.857+0000 7f50e600a640 1 --2- 172.21.15.31:0/4052074348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8accf06b0 0x55a8ad0b4c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.857+0000 7f50e600a640 1 -- 172.21.15.31:0/4052074348 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8acfdcc30 con 0x55a8acce84d0 2024-04-25T16:08:24.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.857+0000 7f50d77fe640 1 -- 172.21.15.31:0/3692675247 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f50d01002a0 con 0x7f50e010a350 2024-04-25T16:08:24.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.858+0000 7f50e600a640 1 --2- 172.21.15.31:0/4052074348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acce84d0 0x55a8ad0b3f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f50d00437b0 tx=0x7f50d0042ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42f3c1a6168f375 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.858+0000 7f50d4bfe640 1 -- 172.21.15.31:0/4052074348 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d00c89d0 con 0x55a8acce84d0 2024-04-25T16:08:24.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.858+0000 7f50e680b640 1 --2- 172.21.15.31:0/4052074348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acd11390 0x55a8ad0b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:08:24.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.858+0000 7f50d4bfe640 1 -- 172.21.15.31:0/4052074348 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50d0043310 con 0x55a8acce84d0 2024-04-25T16:08:24.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.860+0000 7f50ef1ff740 1 -- 172.21.15.31:0/4052074348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acce84d0 msgr2=0x55a8ad0b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.860+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/4052074348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acce84d0 0x55a8ad0b3f80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f50d00437b0 tx=0x7f50d0042ab0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.860+0000 7f50e680b640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50cc0afd00 0x7f50cc0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.869+0000 7f50ef1ff740 1 -- 172.21.15.31:0/4052074348 shutdown_connections 2024-04-25T16:08:24.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.869+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/4052074348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8accf06b0 0x55a8ad0b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.869+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/4052074348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acce84d0 0x55a8ad0b3f80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.869+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/4052074348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acd11390 0x55a8ad0b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.869+0000 7f50ef1ff740 1 -- 172.21.15.31:0/4052074348 >> 172.21.15.31:0/4052074348 conn(0x55a8acd054e0 msgr2=0x55a8ad0af210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:24.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.870+0000 7f50ef1ff740 1 -- 172.21.15.31:0/4052074348 shutdown_connections 2024-04-25T16:08:24.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.870+0000 7f50ef1ff740 1 -- 172.21.15.31:0/4052074348 wait complete. 2024-04-25T16:08:24.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.871+0000 7f50ef1ff740 1 -- start start 2024-04-25T16:08:24.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.871+0000 7f50ef1ff740 1 Processor -- start 2024-04-25T16:08:24.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acce84d0 0x55a8ad0bd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 0x55a8ad0bd910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0c4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50ef1ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a8acfd1a20 con 0x55a8acd11390 2024-04-25T16:08:24.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50ef1ff740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a8acf9edc0 con 0x55a8accf06b0 2024-04-25T16:08:24.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50ef1ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a8ad0b54e0 con 0x55a8acce84d0 2024-04-25T16:08:24.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50ef1ff740 10 client.0 _ll_register_callbacks cb 0x55a8acfba600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:24.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50e600a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acce84d0 0x55a8ad0bd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50e600a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acce84d0 0x55a8ad0bd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53858/0 (socket says 172.21.15.31:53858) 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.872+0000 7f50e600a640 1 -- 172.21.15.31:0/454204243 learned_addr learned my addr 172.21.15.31:0/454204243 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.873+0000 7f50e680b640 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0c4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.873+0000 7f50e600a640 1 -- 172.21.15.31:0/454204243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 msgr2=0x55a8ad0bd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.873+0000 7f50e600a640 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 0x55a8ad0bd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.873+0000 7f50e600a640 1 -- 172.21.15.31:0/454204243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 msgr2=0x55a8ad0c4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.873+0000 7f50e600a640 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0c4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.873+0000 7f50e600a640 1 -- 172.21.15.31:0/454204243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8acd293f0 con 0x55a8acce84d0 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.873+0000 7f50e680b640 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.873+0000 7f50e600a640 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acce84d0 0x55a8ad0bd3d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50d00092a0 tx=0x7f50d00c47a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd9f1430d7544c92 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.874+0000 7f50e680b640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50cc0afd00 0x7f50cc0b2210 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x55a8ad0b4a50 tx=0x7f50dc00ac00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.886+0000 7f50c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:08:24.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.886+0000 7f50c9ffb640 1 -- 172.21.15.31:0/454204243 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d004a6e0 con 0x55a8acce84d0 2024-04-25T16:08:24.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.886+0000 7f50ef1ff740 1 -- 172.21.15.31:0/454204243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f50d0043310 con 0x55a8acce84d0 2024-04-25T16:08:24.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.886+0000 7f50c9ffb640 1 -- 172.21.15.31:0/454204243 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f50d00d15d0 con 0x55a8acce84d0 2024-04-25T16:08:24.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.891+0000 7f50c9ffb640 1 -- 172.21.15.31:0/454204243 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50d00417e0 con 0x55a8acce84d0 2024-04-25T16:08:24.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.892+0000 7f50ef1ff740 10 client.26635 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:24.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.892+0000 7f50ef1ff740 1 -- 172.21.15.31:0/454204243 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8acdf37d0 con 0x55a8acce84d0 2024-04-25T16:08:24.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.898+0000 7f50c9ffb640 1 -- 172.21.15.31:0/454204243 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50d0043310 con 0x55a8acce84d0 2024-04-25T16:08:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.898+0000 7f50c9ffb640 1 -- 172.21.15.31:0/454204243 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f50d004d070 con 0x55a8acce84d0 2024-04-25T16:08:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.899+0000 7f50ef1ff740 10 client.26635 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.899+0000 7f50ef1ff740 10 client.26635 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:08:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.899+0000 7f50ef1ff740 20 client.26635 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.899+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55a8acccff60 0x55a8ad0cae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.899+0000 7f50ef1ff740 4 client.26635 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:24.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.899+0000 7f50ef1ff740 1 -- 172.21.15.31:0/454204243 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8ad025990 con 0x55a8acccff60 2024-04-25T16:08:24.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.904+0000 7f50e5809640 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55a8acccff60 0x55a8ad0cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.923+0000 7f50c9ffb640 10 client.26635 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:08:24.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.923+0000 7f50e5809640 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55a8acccff60 0x55a8ad0cae00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1cdb1e5577dce477 server_cookie=fb2ce5dc949e5caa in_seq=0 out_seq=0 2024-04-25T16:08:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.924+0000 7f50c9ffb640 1 -- 172.21.15.31:0/454204243 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a8ad025990 con 0x55a8acccff60 2024-04-25T16:08:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.924+0000 7f50c9ffb640 10 client.26635 handle_command_reply: tid=0 2024-04-25T16:08:24.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.924+0000 7f50ef1ff740 1 client.26635 shutdown 2024-04-25T16:08:24.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.927+0000 7f50ef1ff740 1 -- 172.21.15.31:0/454204243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acce84d0 msgr2=0x55a8ad0bd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.927+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acce84d0 0x55a8ad0bd3d0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50d00092a0 tx=0x7f50d00c47a0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.927+0000 7f50ef1ff740 1 -- 172.21.15.31:0/454204243 shutdown_connections 2024-04-25T16:08:24.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.927+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0c4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.927+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 0x55a8ad0bd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.927+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55a8acccff60 0x55a8ad0cae00 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.927+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/454204243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acce84d0 0x55a8ad0bd3d0 secure :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f50d00092a0 tx=0x7f50d00c47a0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.927+0000 7f50c9ffb640 0 client.26635 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:08:24.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.927+0000 7f50ef1ff740 1 -- 172.21.15.31:0/454204243 >> 172.21.15.31:0/454204243 conn(0x55a8acd054e0 msgr2=0x55a8ad0b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:24.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.933+0000 7f50ef1ff740 1 -- 172.21.15.31:0/454204243 shutdown_connections 2024-04-25T16:08:24.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.935+0000 7f50ef1ff740 1 -- 172.21.15.31:0/454204243 wait complete. 2024-04-25T16:08:24.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.939+0000 7f50ef1ff740 20 client.26635 trim_cache size 0 max 16384 2024-04-25T16:08:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.981+0000 7f50ef1ff740 1 Processor -- start 2024-04-25T16:08:24.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.982+0000 7f50ef1ff740 1 -- start start 2024-04-25T16:08:24.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.982+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 0x55a8ad0b2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.984+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0b2550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.984+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acca53a0 0x55a8ad0b2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:24.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.984+0000 7f50ef1ff740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f50c400e520 con 0x55a8acd11390 2024-04-25T16:08:24.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.984+0000 7f50ef1ff740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f50c4010060 con 0x55a8accf06b0 2024-04-25T16:08:24.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.984+0000 7f50ef1ff740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f50c400ebf0 con 0x55a8acca53a0 2024-04-25T16:08:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.985+0000 7f50e600a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 0x55a8ad0b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.985+0000 7f50e600a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 0x55a8ad0b2010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57006/0 (socket says 172.21.15.31:57006) 2024-04-25T16:08:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.985+0000 7f50e600a640 1 -- 172.21.15.31:0/3331814401 learned_addr learned my addr 172.21.15.31:0/3331814401 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:24.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.985+0000 7f50e680b640 1 --2- 172.21.15.31:0/3331814401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acca53a0 0x55a8ad0b2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.986+0000 7f50e5809640 1 --2- 172.21.15.31:0/3331814401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0b2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:24.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.989+0000 7f50e600a640 1 -- 172.21.15.31:0/3331814401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acca53a0 msgr2=0x55a8ad0b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.989+0000 7f50e600a640 1 --2- 172.21.15.31:0/3331814401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acca53a0 0x55a8ad0b2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.989+0000 7f50e600a640 1 -- 172.21.15.31:0/3331814401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 msgr2=0x55a8ad0b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:24.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.989+0000 7f50e600a640 1 --2- 172.21.15.31:0/3331814401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0b2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:24.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.989+0000 7f50e600a640 1 -- 172.21.15.31:0/3331814401 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8ad0b8cb0 con 0x55a8accf06b0 2024-04-25T16:08:24.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.993+0000 7f50e600a640 1 --2- 172.21.15.31:0/3331814401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 0x55a8ad0b2010 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f50d00c5e70 tx=0x7f50d00c4970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df70c3c9cacc6c13 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:24.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.995+0000 7f50cbfff640 1 -- 172.21.15.31:0/3331814401 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d00c8370 con 0x55a8accf06b0 2024-04-25T16:08:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.995+0000 7f50cbfff640 1 -- 172.21.15.31:0/3331814401 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f50d00c8c00 con 0x55a8accf06b0 2024-04-25T16:08:24.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.995+0000 7f50cbfff640 1 -- 172.21.15.31:0/3331814401 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d00c8550 con 0x55a8accf06b0 2024-04-25T16:08:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.997+0000 7f50ef1ff740 1 -- 172.21.15.31:0/3331814401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 msgr2=0x55a8ad0b2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:24.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:24.997+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/3331814401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 0x55a8ad0b2010 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f50d00c5e70 tx=0x7f50d00c4970 comp rx=0 tx=0).stop 2024-04-25T16:08:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.005+0000 7f50ef1ff740 1 -- 172.21.15.31:0/3331814401 shutdown_connections 2024-04-25T16:08:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.005+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/3331814401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8acca53a0 0x55a8ad0b2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.005+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/3331814401 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0b2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.005+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/3331814401 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8accf06b0 0x55a8ad0b2010 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.006+0000 7f50ef1ff740 1 -- 172.21.15.31:0/3331814401 >> 172.21.15.31:0/3331814401 conn(0x55a8acd054e0 msgr2=0x55a8ad0bd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.010+0000 7f50ef1ff740 1 -- 172.21.15.31:0/3331814401 shutdown_connections 2024-04-25T16:08:25.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.010+0000 7f50ef1ff740 1 -- 172.21.15.31:0/3331814401 wait complete. 2024-04-25T16:08:25.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.011+0000 7f50ef1ff740 1 -- start start 2024-04-25T16:08:25.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.011+0000 7f50ef1ff740 1 Processor -- start 2024-04-25T16:08:25.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.012+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acca53a0 0x55a8ad0c13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.013+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8accf06b0 0x55a8ad0c1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.013+0000 7f50ef1ff740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0c2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.013+0000 7f50e600a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acca53a0 0x55a8ad0c13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.013+0000 7f50e600a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acca53a0 0x55a8ad0c13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57022/0 (socket says 172.21.15.31:57022) 2024-04-25T16:08:25.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.013+0000 7f50e600a640 1 -- 172.21.15.31:0/1347710925 learned_addr learned my addr 172.21.15.31:0/1347710925 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:25.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.014+0000 7f50e5809640 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8accf06b0 0x55a8ad0c1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.014+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f50c400f990 con 0x55a8acd11390 2024-04-25T16:08:25.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.014+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f50c400f2c0 con 0x55a8acca53a0 2024-04-25T16:08:25.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.014+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f50c400c500 con 0x55a8accf06b0 2024-04-25T16:08:25.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.014+0000 7f50ef1ff740 10 client.0 _ll_register_callbacks cb 0x7f50d00c5250 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:25.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.015+0000 7f50e600a640 1 -- 172.21.15.31:0/1347710925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8accf06b0 msgr2=0x55a8ad0c1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:25.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.015+0000 7f50e600a640 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8accf06b0 0x55a8ad0c1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.015+0000 7f50e600a640 1 -- 172.21.15.31:0/1347710925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 msgr2=0x55a8ad0c2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:25.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.015+0000 7f50e600a640 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0c2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.015+0000 7f50e600a640 1 -- 172.21.15.31:0/1347710925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8ad0b2fd0 con 0x55a8acca53a0 2024-04-25T16:08:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.021+0000 7f50e600a640 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acca53a0 0x55a8ad0c13c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f50d0047040 tx=0x7f50d002ab40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29de1bf8c288431 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.029+0000 7f50c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:08:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.029+0000 7f50c9ffb640 1 -- 172.21.15.31:0/1347710925 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50d00d1ab0 con 0x55a8acca53a0 2024-04-25T16:08:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.029+0000 7f50c9ffb640 1 -- 172.21.15.31:0/1347710925 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f50d00c4570 con 0x55a8acca53a0 2024-04-25T16:08:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.029+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55a8ad01f800 con 0x55a8acca53a0 2024-04-25T16:08:25.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.031+0000 7f50c9ffb640 1 -- 172.21.15.31:0/1347710925 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f50d00c8370 con 0x55a8acca53a0 2024-04-25T16:08:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.032+0000 7f50ef1ff740 10 client.26681 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.032+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a8ad0b2fd0 con 0x55a8acca53a0 2024-04-25T16:08:25.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.034+0000 7f50c9ffb640 1 -- 172.21.15.31:0/1347710925 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55a8ad01f800 con 0x55a8acca53a0 2024-04-25T16:08:25.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.034+0000 7f50c9ffb640 1 -- 172.21.15.31:0/1347710925 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f50d004a240 con 0x55a8acca53a0 2024-04-25T16:08:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.034+0000 7f50ef1ff740 10 client.26681 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.034+0000 7f50ef1ff740 10 client.26681 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:08:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.034+0000 7f50ef1ff740 20 client.26681 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.034+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55a8ad1331c0 0x55a8ad1355e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.034+0000 7f50ef1ff740 4 client.26681 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:08:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.034+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a8acf9a610 con 0x55a8ad1331c0 2024-04-25T16:08:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.035+0000 7f50e5809640 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55a8ad1331c0 0x55a8ad1355e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.035+0000 7f50c9ffb640 10 client.26681 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:08:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.036+0000 7f50e5809640 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55a8ad1331c0 0x55a8ad1355e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=98920cb7d6bc3d3b server_cookie=5636a4c5e21e5244 in_seq=0 out_seq=0 2024-04-25T16:08:25.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.041+0000 7f50c9ffb640 1 -- 172.21.15.31:0/1347710925 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a8acf9a610 con 0x55a8ad1331c0 2024-04-25T16:08:25.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.041+0000 7f50c9ffb640 10 client.26681 handle_command_reply: tid=0 2024-04-25T16:08:25.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.041+0000 7f50ef1ff740 1 client.26681 shutdown 2024-04-25T16:08:25.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.042+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acca53a0 msgr2=0x55a8ad0c13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.042+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acca53a0 0x55a8ad0c13c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f50d0047040 tx=0x7f50d002ab40 comp rx=0 tx=0).stop 2024-04-25T16:08:25.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.044+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 shutdown_connections 2024-04-25T16:08:25.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.044+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a8acd11390 0x55a8ad0c2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.044+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55a8ad1331c0 0x55a8ad1355e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.044+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a8accf06b0 0x55a8ad0c1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.044+0000 7f50ef1ff740 1 --2- 172.21.15.31:0/1347710925 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a8acca53a0 0x55a8ad0c13c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.044+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 >> 172.21.15.31:0/1347710925 conn(0x55a8acd054e0 msgr2=0x55a8ad0b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:25.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.044+0000 7f50c9ffb640 0 client.26681 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:08:25.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.060+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 shutdown_connections 2024-04-25T16:08:25.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.066+0000 7f50ef1ff740 1 -- 172.21.15.31:0/1347710925 wait complete. 2024-04-25T16:08:25.069 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:08:25.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.067+0000 7f50ef1ff740 20 client.26681 trim_cache size 0 max 16384 2024-04-25T16:08:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 -- 172.21.15.31:0/3692675247 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50cc0afd00 msgr2=0x7f50cc0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50cc0afd00 0x7f50cc0b2210 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x55a8ad0b4a50 tx=0x7f50dc00ac00 comp rx=0 tx=0).stop 2024-04-25T16:08:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 -- 172.21.15.31:0/3692675247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50e010a350 msgr2=0x7f50e0117100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50e010a350 0x7f50e0117100 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009a20 tx=0x7f50d0004270 comp rx=0 tx=0).stop 2024-04-25T16:08:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 -- 172.21.15.31:0/3692675247 shutdown_connections 2024-04-25T16:08:25.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f50cc0afd00 0x7f50cc0b2210 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50e010fef0 0x7f50e01185a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f50e010b1d0 0x7f50e0117640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 --2- 172.21.15.31:0/3692675247 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f50e010a350 0x7f50e0117100 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009a20 tx=0x7f50d0004270 comp rx=0 tx=0).stop 2024-04-25T16:08:25.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.073+0000 7f50e700c640 1 -- 172.21.15.31:0/3692675247 >> 172.21.15.31:0/3692675247 conn(0x7f50e0070400 msgr2=0x7f50e010bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.075+0000 7f50e700c640 1 -- 172.21.15.31:0/3692675247 shutdown_connections 2024-04-25T16:08:25.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.077+0000 7f50e700c640 1 -- 172.21.15.31:0/3692675247 wait complete. 2024-04-25T16:08:25.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:24 smithi186 ceph-mon[33258]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1003 MiB/s wr, 2.35k op/s 2024-04-25T16:08:25.151 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:24 smithi031 ceph-mon[29140]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1003 MiB/s wr, 2.35k op/s 2024-04-25T16:08:25.154 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:08:25.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:24 smithi063 ceph-mon[32413]: pgmap v564: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1003 MiB/s wr, 2.35k op/s 2024-04-25T16:08:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.610+0000 7f8a44700640 1 -- 172.21.15.31:0/3406463306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a3c109fc0 msgr2=0x7f8a3c10a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.610+0000 7f8a44700640 1 --2- 172.21.15.31:0/3406463306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a3c109fc0 0x7f8a3c10a3d0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f8a34009ca0 tx=0x7f8a3402a8d0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.610+0000 7f8a44700640 1 -- 172.21.15.31:0/3406463306 shutdown_connections 2024-04-25T16:08:25.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.610+0000 7f8a44700640 1 --2- 172.21.15.31:0/3406463306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a3c1120c0 0x7f8a3c1144e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.610+0000 7f8a44700640 1 --2- 172.21.15.31:0/3406463306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a3c10f760 0x7f8a3c111b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.610+0000 7f8a44700640 1 --2- 172.21.15.31:0/3406463306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a3c109fc0 0x7f8a3c10a3d0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.610+0000 7f8a44700640 1 -- 172.21.15.31:0/3406463306 >> 172.21.15.31:0/3406463306 conn(0x7f8a3c070400 msgr2=0x7f8a3c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.611+0000 7f8a44700640 1 -- 172.21.15.31:0/3406463306 shutdown_connections 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.611+0000 7f8a44700640 1 -- 172.21.15.31:0/3406463306 wait complete. 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.612+0000 7f8a44700640 1 Processor -- start 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.612+0000 7f8a44700640 1 -- start start 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.613+0000 7f8a44700640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a3c109fc0 0x7f8a3c075150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.613+0000 7f8a44700640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a3c10f760 0x7f8a3c075690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.613+0000 7f8a44700640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a3c1120c0 0x7f8a3c07c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.613+0000 7f8a44700640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c10c0f0 con 0x7f8a3c10f760 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.613+0000 7f8a44700640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c10bf70 con 0x7f8a3c1120c0 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.613+0000 7f8a44700640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c10c270 con 0x7f8a3c109fc0 2024-04-25T16:08:25.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.614+0000 7f8a42c9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a3c1120c0 0x7f8a3c07c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.614+0000 7f8a42c9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a3c1120c0 0x7f8a3c07c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57052/0 (socket says 172.21.15.31:57052) 2024-04-25T16:08:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.614+0000 7f8a42c9d640 1 -- 172.21.15.31:0/1163094829 learned_addr learned my addr 172.21.15.31:0/1163094829 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.614+0000 7f8a4249c640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a3c109fc0 0x7f8a3c075150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.614+0000 7f8a41c9b640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a3c10f760 0x7f8a3c075690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.615+0000 7f8a4249c640 1 -- 172.21.15.31:0/1163094829 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a3c1120c0 msgr2=0x7f8a3c07c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.615+0000 7f8a4249c640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a3c1120c0 0x7f8a3c07c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.615+0000 7f8a4249c640 1 -- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a3c10f760 msgr2=0x7f8a3c075690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.615+0000 7f8a4249c640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a3c10f760 0x7f8a3c075690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.615+0000 7f8a4249c640 1 -- 172.21.15.31:0/1163094829 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3c07cc90 con 0x7f8a3c109fc0 2024-04-25T16:08:25.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.619+0000 7f8a4249c640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a3c109fc0 0x7f8a3c075150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8a34006010 tx=0x7f8a34003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:25.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.622+0000 7f8a337fe640 1 -- 172.21.15.31:0/1163094829 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34032ec0 con 0x7f8a3c109fc0 2024-04-25T16:08:25.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.623+0000 7f8a44700640 1 -- 172.21.15.31:0/1163094829 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3c07cf20 con 0x7f8a3c109fc0 2024-04-25T16:08:25.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.623+0000 7f8a337fe640 1 -- 172.21.15.31:0/1163094829 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a340044f0 con 0x7f8a3c109fc0 2024-04-25T16:08:25.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.623+0000 7f8a44700640 1 -- 172.21.15.31:0/1163094829 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a3c07d470 con 0x7f8a3c109fc0 2024-04-25T16:08:25.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.625+0000 7f8a337fe640 1 -- 172.21.15.31:0/1163094829 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a34030cd0 con 0x7f8a3c109fc0 2024-04-25T16:08:25.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.631+0000 7f8a4b5b2740 1 Processor -- start 2024-04-25T16:08:25.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.632+0000 7f8a4b5b2740 1 -- start start 2024-04-25T16:08:25.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.632+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b19309d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.633+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 0x55b19309df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.633+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 0x55b19309ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.633+0000 7f8a4b5b2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b193001d80 con 0x55b192cd24d0 2024-04-25T16:08:25.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.633+0000 7f8a4b5b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b192fbba20 con 0x55b192cda6b0 2024-04-25T16:08:25.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.633+0000 7f8a4b5b2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b192ffdd40 con 0x55b192cfb390 2024-04-25T16:08:25.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.635+0000 7f8a41c9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 0x55b19309ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.635+0000 7f8a41c9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 0x55b19309ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57054/0 (socket says 172.21.15.31:57054) 2024-04-25T16:08:25.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.635+0000 7f8a41c9b640 1 -- 172.21.15.31:0/4066276118 learned_addr learned my addr 172.21.15.31:0/4066276118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:25.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.636+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/4066276118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b19309d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.640+0000 7f8a42c9d640 1 -- 172.21.15.31:0/4066276118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 msgr2=0x55b19309ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.640+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/4066276118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 0x55b19309ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.640+0000 7f8a42c9d640 1 -- 172.21.15.31:0/4066276118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 msgr2=0x55b19309df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.640+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/4066276118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 0x55b19309df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.640+0000 7f8a42c9d640 1 -- 172.21.15.31:0/4066276118 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b192fc6c30 con 0x55b192cfb390 2024-04-25T16:08:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.641+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/4066276118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b19309d9e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800bc30 tx=0x7f8a3800f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7cfb800cd362af7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:25.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.643+0000 7f8a30ff9640 1 -- 172.21.15.31:0/4066276118 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38015070 con 0x55b192cfb390 2024-04-25T16:08:25.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.643+0000 7f8a30ff9640 1 -- 172.21.15.31:0/4066276118 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a380118f0 con 0x55b192cfb390 2024-04-25T16:08:25.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.643+0000 7f8a30ff9640 1 -- 172.21.15.31:0/4066276118 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38010820 con 0x55b192cfb390 2024-04-25T16:08:25.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.663+0000 7f8a337fe640 1 -- 172.21.15.31:0/1163094829 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8a34004040 con 0x7f8a3c109fc0 2024-04-25T16:08:25.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.665+0000 7f8a337fe640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a280afe60 0x7f8a280b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.665+0000 7f8a337fe640 1 -- 172.21.15.31:0/1163094829 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8a341020f0 con 0x7f8a3c109fc0 2024-04-25T16:08:25.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.667+0000 7f8a4249c640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a280afe60 0x7f8a280b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.668+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/4066276118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 msgr2=0x55b19309d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.668+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/4066276118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b19309d9e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800bc30 tx=0x7f8a3800f730 comp rx=0 tx=0).stop 2024-04-25T16:08:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.669+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/4066276118 shutdown_connections 2024-04-25T16:08:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.669+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/4066276118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 0x55b19309ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.669+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/4066276118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 0x55b19309df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.669+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/4066276118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b19309d9e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.669+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/4066276118 >> 172.21.15.31:0/4066276118 conn(0x55b192cef4e0 msgr2=0x55b1930991f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:25.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.675+0000 7f8a4249c640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a280afe60 0x7f8a280b2370 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8a34050030 tx=0x7f8a34005c30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:25.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.680+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/4066276118 shutdown_connections 2024-04-25T16:08:25.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.680+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/4066276118 wait complete. 2024-04-25T16:08:25.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.681+0000 7f8a4b5b2740 1 -- start start 2024-04-25T16:08:25.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.681+0000 7f8a4b5b2740 1 Processor -- start 2024-04-25T16:08:25.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.687+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 0x55b1930b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.687+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 0x55b1930b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.687+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b1930b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.687+0000 7f8a4b5b2740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b19309f700 con 0x55b192cd24d0 2024-04-25T16:08:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.687+0000 7f8a4b5b2740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b192f88dc0 con 0x55b192cda6b0 2024-04-25T16:08:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.687+0000 7f8a4b5b2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b19309f880 con 0x55b192cfb390 2024-04-25T16:08:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.687+0000 7f8a4b5b2740 10 client.0 _ll_register_callbacks cb 0x55b192fa4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.697+0000 7f8a42c9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 0x55b1930b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.697+0000 7f8a42c9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 0x55b1930b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47316/0 (socket says 172.21.15.31:47316) 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.697+0000 7f8a42c9d640 1 -- 172.21.15.31:0/2285197889 learned_addr learned my addr 172.21.15.31:0/2285197889 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.697+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b1930b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.698+0000 7f8a42c9d640 1 -- 172.21.15.31:0/2285197889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 msgr2=0x55b1930b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.698+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b1930b74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.698+0000 7f8a42c9d640 1 -- 172.21.15.31:0/2285197889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 msgr2=0x55b1930b0540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.698+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 0x55b1930b0540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.698+0000 7f8a42c9d640 1 -- 172.21.15.31:0/2285197889 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b192d133f0 con 0x55b192cd24d0 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.699+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 0x55b1930b0000 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f8a38000c00 tx=0x7f8a3800b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e831352e4c76ff1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.699+0000 7f8a25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:08:25.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.700+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2285197889 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38022040 con 0x55b192cd24d0 2024-04-25T16:08:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.700+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2285197889 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a380118f0 con 0x55b192cd24d0 2024-04-25T16:08:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.700+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2285197889 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8a38011d20 con 0x55b192cd24d0 2024-04-25T16:08:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.702+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2285197889 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8a38015040 con 0x55b192cd24d0 2024-04-25T16:08:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.702+0000 7f8a4b5b2740 10 client.17043 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.702+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2285197889 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b192ddd7d0 con 0x55b192cd24d0 2024-04-25T16:08:25.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.703+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2285197889 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a380118f0 con 0x55b192cd24d0 2024-04-25T16:08:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.705+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2285197889 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8a38004b90 con 0x55b192cd24d0 2024-04-25T16:08:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.705+0000 7f8a4b5b2740 10 client.17043 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.705+0000 7f8a4b5b2740 10 client.17043 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:08:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.705+0000 7f8a4b5b2740 20 client.17043 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.705+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55b192cb9f60 0x55b1930a98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.705+0000 7f8a4b5b2740 4 client.17043 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.705+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2285197889 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b19300f990 con 0x55b192cb9f60 2024-04-25T16:08:25.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.707+0000 7f8a41c9b640 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55b192cb9f60 0x55b1930a98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.722+0000 7f8a41c9b640 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55b192cb9f60 0x55b1930a98f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=15a77e9cf835483f server_cookie=b6df7334b48e3518 in_seq=0 out_seq=0 2024-04-25T16:08:25.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.723+0000 7f8a25ffb640 10 client.17043 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:08:25.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.725+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2285197889 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b19300f990 con 0x55b192cb9f60 2024-04-25T16:08:25.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.725+0000 7f8a25ffb640 10 client.17043 handle_command_reply: tid=0 2024-04-25T16:08:25.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.725+0000 7f8a4b5b2740 1 client.17043 shutdown 2024-04-25T16:08:25.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.727+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2285197889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 msgr2=0x55b1930b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.727+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 0x55b1930b0000 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f8a38000c00 tx=0x7f8a3800b740 comp rx=0 tx=0).stop 2024-04-25T16:08:25.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.728+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2285197889 shutdown_connections 2024-04-25T16:08:25.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.728+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b1930b74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.728+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55b192cb9f60 0x55b1930a98f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.728+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cda6b0 0x55b1930b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.728+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2285197889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cd24d0 0x55b1930b0000 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.728+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2285197889 >> 172.21.15.31:0/2285197889 conn(0x55b192cef4e0 msgr2=0x55b1930983f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:25.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.728+0000 7f8a25ffb640 0 client.17043 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:08:25.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.735+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2285197889 shutdown_connections 2024-04-25T16:08:25.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.736+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2285197889 wait complete. 2024-04-25T16:08:25.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.736+0000 7f8a4b5b2740 20 client.17043 trim_cache size 0 max 16384 2024-04-25T16:08:25.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.787+0000 7f8a4b5b2740 1 Processor -- start 2024-04-25T16:08:25.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.792+0000 7f8a4b5b2740 1 -- start start 2024-04-25T16:08:25.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.793+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cd24d0 0x55b1930afbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.793+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b0110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.793+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b1930ae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.793+0000 7f8a42c9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cd24d0 0x55b1930afbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.793+0000 7f8a42c9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cd24d0 0x55b1930afbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57076/0 (socket says 172.21.15.31:57076) 2024-04-25T16:08:25.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.794+0000 7f8a42c9d640 1 -- 172.21.15.31:0/1901803443 learned_addr learned my addr 172.21.15.31:0/1901803443 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:25.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.794+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/1901803443 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c081c00 con 0x55b192cda6b0 2024-04-25T16:08:25.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.794+0000 7f8a41c9b640 1 --2- 172.21.15.31:0/1901803443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.794+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/1901803443 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c083740 con 0x55b192cd24d0 2024-04-25T16:08:25.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.794+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/1901803443 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c0822d0 con 0x55b192cfb390 2024-04-25T16:08:25.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.795+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/1901803443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b1930ae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.797+0000 7f8a42c9d640 1 -- 172.21.15.31:0/1901803443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 msgr2=0x55b1930ae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:25.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.797+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/1901803443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b1930ae0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.797+0000 7f8a42c9d640 1 -- 172.21.15.31:0/1901803443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 msgr2=0x55b1930b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:25.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.797+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/1901803443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.797+0000 7f8a42c9d640 1 -- 172.21.15.31:0/1901803443 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b193099d80 con 0x55b192cd24d0 2024-04-25T16:08:25.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.803+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/1901803443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cd24d0 0x55b1930afbd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8a38011570 tx=0x7f8a38010920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=917f82bd4ec52411 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:25.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.808+0000 7f8a27fff640 1 -- 172.21.15.31:0/1901803443 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38010ce0 con 0x55b192cd24d0 2024-04-25T16:08:25.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.808+0000 7f8a27fff640 1 -- 172.21.15.31:0/1901803443 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a3803a040 con 0x55b192cd24d0 2024-04-25T16:08:25.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.808+0000 7f8a27fff640 1 -- 172.21.15.31:0/1901803443 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a38005040 con 0x55b192cd24d0 2024-04-25T16:08:25.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.809+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/1901803443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cd24d0 msgr2=0x55b1930afbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.813 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.809+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/1901803443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cd24d0 0x55b1930afbd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8a38011570 tx=0x7f8a38010920 comp rx=0 tx=0).stop 2024-04-25T16:08:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.814+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/1901803443 shutdown_connections 2024-04-25T16:08:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.814+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/1901803443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cfb390 0x55b1930ae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.814+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/1901803443 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.814+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/1901803443 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cd24d0 0x55b1930afbd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.814+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/1901803443 >> 172.21.15.31:0/1901803443 conn(0x55b192cef4e0 msgr2=0x55b1930a2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.817+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/1901803443 shutdown_connections 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.817+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/1901803443 wait complete. 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.817+0000 7f8a4b5b2740 1 -- start start 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.818+0000 7f8a4b5b2740 1 Processor -- start 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.818+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cd24d0 0x55b1930b25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.818+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.818+0000 7f8a4b5b2740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cfb390 0x55b1930b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.819+0000 7f8a41c9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.819+0000 7f8a41c9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47334/0 (socket says 172.21.15.31:47334) 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.819+0000 7f8a41c9b640 1 -- 172.21.15.31:0/2073521482 learned_addr learned my addr 172.21.15.31:0/2073521482 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:25.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.819+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c083070 con 0x55b192cda6b0 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.819+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c0829a0 con 0x55b192cfb390 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.819+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8a3c07fcd0 con 0x55b192cd24d0 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.819+0000 7f8a4b5b2740 10 client.0 _ll_register_callbacks cb 0x7f8a2c00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.821+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cd24d0 0x55b1930b25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.821+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cfb390 0x55b1930b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.822+0000 7f8a41c9b640 1 -- 172.21.15.31:0/2073521482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cd24d0 msgr2=0x55b1930b25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.822+0000 7f8a41c9b640 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cd24d0 0x55b1930b25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.822+0000 7f8a41c9b640 1 -- 172.21.15.31:0/2073521482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cfb390 msgr2=0x55b1930b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.822+0000 7f8a41c9b640 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cfb390 0x55b1930b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.822+0000 7f8a41c9b640 1 -- 172.21.15.31:0/2073521482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1930b5ff0 con 0x55b192cda6b0 2024-04-25T16:08:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.829+0000 7f8a41c9b640 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b2af0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x55b1930aa050 tx=0x7f8a2c00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=924b3abec8d119d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:25.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.829+0000 7f8a25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:08:25.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.830+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2073521482 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c01d3f0 con 0x55b192cda6b0 2024-04-25T16:08:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.831+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a3803a040 con 0x55b192cda6b0 2024-04-25T16:08:25.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.831+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2073521482 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8a2c01d590 con 0x55b192cda6b0 2024-04-25T16:08:25.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.832+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2073521482 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8a2c01d790 con 0x55b192cda6b0 2024-04-25T16:08:25.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.832+0000 7f8a4b5b2740 10 client.17049 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:25.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.832+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b1930baa20 con 0x55b192cda6b0 2024-04-25T16:08:25.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.833+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2073521482 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a3803a040 con 0x55b192cda6b0 2024-04-25T16:08:25.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.834+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2073521482 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8a2c047420 con 0x55b192cda6b0 2024-04-25T16:08:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.834+0000 7f8a4b5b2740 10 client.17049 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.834+0000 7f8a4b5b2740 10 client.17049 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:08:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.834+0000 7f8a4b5b2740 20 client.17049 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:25.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.834+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55b192c8f3a0 0x55b1930bd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.834+0000 7f8a4b5b2740 4 client.17049 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:08:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.834+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b193009800 con 0x55b192c8f3a0 2024-04-25T16:08:25.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.836+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55b192c8f3a0 0x55b1930bd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:25.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.845+0000 7f8a42c9d640 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55b192c8f3a0 0x55b1930bd420 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=46d6ec6ef51e7459 server_cookie=2cbff824ab791940 in_seq=0 out_seq=0 2024-04-25T16:08:25.846 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.845+0000 7f8a25ffb640 10 client.17049 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:08:25.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.846+0000 7f8a25ffb640 1 -- 172.21.15.31:0/2073521482 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b193009800 con 0x55b192c8f3a0 2024-04-25T16:08:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.846+0000 7f8a25ffb640 10 client.17049 handle_command_reply: tid=0 2024-04-25T16:08:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.846+0000 7f8a4b5b2740 1 client.17049 shutdown 2024-04-25T16:08:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.846+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 msgr2=0x55b1930b2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.846+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b2af0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x55b1930aa050 tx=0x7f8a2c00be60 comp rx=0 tx=0).stop 2024-04-25T16:08:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.847+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 shutdown_connections 2024-04-25T16:08:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.847+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55b192c8f3a0 0x55b1930bd420 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.847+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b192cfb390 0x55b1930b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.847+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b192cda6b0 0x55b1930b2af0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.847+0000 7f8a4b5b2740 1 --2- 172.21.15.31:0/2073521482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b192cd24d0 0x55b1930b25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.847+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 >> 172.21.15.31:0/2073521482 conn(0x55b192cef4e0 msgr2=0x55b1930a2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:25.850 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.849+0000 7f8a25ffb640 0 client.17049 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:08:25.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.849+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 shutdown_connections 2024-04-25T16:08:25.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.850+0000 7f8a4b5b2740 1 -- 172.21.15.31:0/2073521482 wait complete. 2024-04-25T16:08:25.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.850+0000 7f8a4b5b2740 20 client.17049 trim_cache size 0 max 16384 2024-04-25T16:08:25.861 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:08:25.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.864+0000 7f8a44700640 1 -- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a280afe60 msgr2=0x7f8a280b2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.864+0000 7f8a44700640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a280afe60 0x7f8a280b2370 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8a34050030 tx=0x7f8a34005c30 comp rx=0 tx=0).stop 2024-04-25T16:08:25.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.864+0000 7f8a44700640 1 -- 172.21.15.31:0/1163094829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a3c109fc0 msgr2=0x7f8a3c075150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:25.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.864+0000 7f8a44700640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a3c109fc0 0x7f8a3c075150 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8a34006010 tx=0x7f8a34003ba0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.867+0000 7f8a44700640 1 -- 172.21.15.31:0/1163094829 shutdown_connections 2024-04-25T16:08:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.867+0000 7f8a44700640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8a280afe60 0x7f8a280b2370 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.867+0000 7f8a44700640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8a3c1120c0 0x7f8a3c07c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.867+0000 7f8a44700640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8a3c10f760 0x7f8a3c075690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.868+0000 7f8a44700640 1 --2- 172.21.15.31:0/1163094829 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a3c109fc0 0x7f8a3c075150 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:25.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.868+0000 7f8a44700640 1 -- 172.21.15.31:0/1163094829 >> 172.21.15.31:0/1163094829 conn(0x7f8a3c070400 msgr2=0x7f8a3c072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:25.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.871+0000 7f8a44700640 1 -- 172.21.15.31:0/1163094829 shutdown_connections 2024-04-25T16:08:25.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:25.875+0000 7f8a44700640 1 -- 172.21.15.31:0/1163094829 wait complete. 2024-04-25T16:08:25.945 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:08:26.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.443+0000 7f871137d640 1 -- 172.21.15.31:0/966111553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 msgr2=0x7f870c10ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:26.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.443+0000 7f871137d640 1 --2- 172.21.15.31:0/966111553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 0x7f870c10ac90 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8704009ca0 tx=0x7f870402a8f0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.454+0000 7f871137d640 1 -- 172.21.15.31:0/966111553 shutdown_connections 2024-04-25T16:08:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.454+0000 7f871137d640 1 --2- 172.21.15.31:0/966111553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f870c10fef0 0x7f870c110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.454+0000 7f871137d640 1 --2- 172.21.15.31:0/966111553 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f870c10b1d0 0x7f870c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.454+0000 7f871137d640 1 --2- 172.21.15.31:0/966111553 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 0x7f870c10ac90 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.454+0000 7f871137d640 1 -- 172.21.15.31:0/966111553 >> 172.21.15.31:0/966111553 conn(0x7f870c070400 msgr2=0x7f870c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:26.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.458+0000 7f871137d640 1 -- 172.21.15.31:0/966111553 shutdown_connections 2024-04-25T16:08:26.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.461+0000 7f871137d640 1 -- 172.21.15.31:0/966111553 wait complete. 2024-04-25T16:08:26.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.462+0000 7f871137d640 1 Processor -- start 2024-04-25T16:08:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.464+0000 7f871137d640 1 -- start start 2024-04-25T16:08:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.464+0000 7f871137d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 0x7f870c117110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.465+0000 7f871137d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f870c10b1d0 0x7f870c117650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.465+0000 7f871137d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f870c10fef0 0x7f870c1185b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.465+0000 7f871137d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f870c11e390 con 0x7f870c10fef0 2024-04-25T16:08:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.465+0000 7f871137d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f870c11e210 con 0x7f870c10a3e0 2024-04-25T16:08:26.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.465+0000 7f871137d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f870c11e510 con 0x7f870c10b1d0 2024-04-25T16:08:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.467+0000 7f870affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 0x7f870c117110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.467+0000 7f870affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 0x7f870c117110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57096/0 (socket says 172.21.15.31:57096) 2024-04-25T16:08:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.467+0000 7f870affd640 1 -- 172.21.15.31:0/1031782187 learned_addr learned my addr 172.21.15.31:0/1031782187 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:26.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.467+0000 7f870b7fe640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f870c10fef0 0x7f870c1185b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.468+0000 7f870affd640 1 -- 172.21.15.31:0/1031782187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f870c10b1d0 msgr2=0x7f870c117650 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:08:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.468+0000 7f870affd640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f870c10b1d0 0x7f870c117650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.468+0000 7f870affd640 1 -- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f870c10fef0 msgr2=0x7f870c1185b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.468+0000 7f870affd640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f870c10fef0 0x7f870c1185b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.468+0000 7f870affd640 1 -- 172.21.15.31:0/1031782187 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f870c118c20 con 0x7f870c10a3e0 2024-04-25T16:08:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.469+0000 7f870affd640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 0x7f870c117110 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f870402aee0 tx=0x7f8704003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:26.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.470+0000 7f86f3fff640 1 -- 172.21.15.31:0/1031782187 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8704004040 con 0x7f870c10a3e0 2024-04-25T16:08:26.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.470+0000 7f86f3fff640 1 -- 172.21.15.31:0/1031782187 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f870402d070 con 0x7f870c10a3e0 2024-04-25T16:08:26.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.470+0000 7f86f3fff640 1 -- 172.21.15.31:0/1031782187 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870402fba0 con 0x7f870c10a3e0 2024-04-25T16:08:26.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.470+0000 7f871137d640 1 -- 172.21.15.31:0/1031782187 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f870c118eb0 con 0x7f870c10a3e0 2024-04-25T16:08:26.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.471+0000 7f871137d640 1 -- 172.21.15.31:0/1031782187 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f870c119430 con 0x7f870c10a3e0 2024-04-25T16:08:26.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.481+0000 7f871822b740 1 Processor -- start 2024-04-25T16:08:26.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.481+0000 7f871822b740 1 -- start start 2024-04-25T16:08:26.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.481+0000 7f871822b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd3a390 0x56421d0dc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.482+0000 7f871822b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421cd114d0 0x56421d0dcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.482+0000 7f871822b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd196b0 0x56421d0ddbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.482+0000 7f871822b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56421d040d80 con 0x56421cd3a390 2024-04-25T16:08:26.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.482+0000 7f871822b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56421cffaa20 con 0x56421cd114d0 2024-04-25T16:08:26.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.482+0000 7f871822b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56421d03cd40 con 0x56421cd196b0 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.484+0000 7f870b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd3a390 0x56421d0dc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.484+0000 7f870b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd3a390 0x56421d0dc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47354/0 (socket says 172.21.15.31:47354) 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.484+0000 7f870b7fe640 1 -- 172.21.15.31:0/756523729 learned_addr learned my addr 172.21.15.31:0/756523729 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.484+0000 7f86f3fff640 1 -- 172.21.15.31:0/1031782187 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f870402fd40 con 0x7f870c10a3e0 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.485+0000 7f870b7fe640 1 -- 172.21.15.31:0/756523729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd196b0 msgr2=0x56421d0ddbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.485+0000 7f870b7fe640 1 --2- 172.21.15.31:0/756523729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd196b0 0x56421d0ddbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.485+0000 7f86f3fff640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86f40afc70 0x7f86f40b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.485+0000 7f870b7fe640 1 -- 172.21.15.31:0/756523729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421cd114d0 msgr2=0x56421d0dcf60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.485+0000 7f870b7fe640 1 --2- 172.21.15.31:0/756523729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421cd114d0 0x56421d0dcf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.485+0000 7f870b7fe640 1 -- 172.21.15.31:0/756523729 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421d005c30 con 0x56421cd3a390 2024-04-25T16:08:26.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.487+0000 7f86f3fff640 1 -- 172.21.15.31:0/1031782187 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8704100120 con 0x7f870c10a3e0 2024-04-25T16:08:26.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.490+0000 7f870b7fe640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86f40afc70 0x7f86f40b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.490+0000 7f870b7fe640 1 --2- 172.21.15.31:0/756523729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd3a390 0x56421d0dc9e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f87000099c0 tx=0x7f8700009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2098fb05b50708 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:26.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.491+0000 7f86f17fa640 1 -- 172.21.15.31:0/756523729 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870000ecc0 con 0x56421cd3a390 2024-04-25T16:08:26.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.492+0000 7f86f17fa640 1 -- 172.21.15.31:0/756523729 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f870000ee60 con 0x56421cd3a390 2024-04-25T16:08:26.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.493+0000 7f871822b740 1 -- 172.21.15.31:0/756523729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd3a390 msgr2=0x56421d0dc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:26.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.493+0000 7f871822b740 1 --2- 172.21.15.31:0/756523729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd3a390 0x56421d0dc9e0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f87000099c0 tx=0x7f8700009e80 comp rx=0 tx=0).stop 2024-04-25T16:08:26.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.494+0000 7f871822b740 1 -- 172.21.15.31:0/756523729 shutdown_connections 2024-04-25T16:08:26.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.494+0000 7f871822b740 1 --2- 172.21.15.31:0/756523729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd196b0 0x56421d0ddbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.494+0000 7f871822b740 1 --2- 172.21.15.31:0/756523729 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421cd114d0 0x56421d0dcf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.494+0000 7f871822b740 1 --2- 172.21.15.31:0/756523729 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd3a390 0x56421d0dc9e0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.494+0000 7f871822b740 1 -- 172.21.15.31:0/756523729 >> 172.21.15.31:0/756523729 conn(0x56421cd2e4e0 msgr2=0x56421d0d81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:26.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.496+0000 7f870b7fe640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86f40afc70 0x7f86f40b2180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f87000099c0 tx=0x7f8700009e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:26.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.497+0000 7f871822b740 1 -- 172.21.15.31:0/756523729 shutdown_connections 2024-04-25T16:08:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.497+0000 7f871822b740 1 -- 172.21.15.31:0/756523729 wait complete. 2024-04-25T16:08:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.498+0000 7f871822b740 1 -- start start 2024-04-25T16:08:26.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.498+0000 7f871822b740 1 Processor -- start 2024-04-25T16:08:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f871822b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd114d0 0x56421d0e6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f871822b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421cd196b0 0x56421d0e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f871822b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421ccf8f60 0x56421d0edb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f871822b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56421d0de750 con 0x56421ccf8f60 2024-04-25T16:08:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f871822b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56421cfc7dc0 con 0x56421cd196b0 2024-04-25T16:08:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f871822b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56421d0de8d0 con 0x56421cd114d0 2024-04-25T16:08:26.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f871822b740 10 client.0 _ll_register_callbacks cb 0x56421cfe3600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:26.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f870affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd114d0 0x56421d0e6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f870affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd114d0 0x56421d0e6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53974/0 (socket says 172.21.15.31:53974) 2024-04-25T16:08:26.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.499+0000 7f870affd640 1 -- 172.21.15.31:0/3602062910 learned_addr learned my addr 172.21.15.31:0/3602062910 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:26.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.500+0000 7f870a7fc640 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421cd196b0 0x56421d0e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.500+0000 7f870affd640 1 -- 172.21.15.31:0/3602062910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421cd196b0 msgr2=0x56421d0e6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:26.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.500+0000 7f870affd640 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421cd196b0 0x56421d0e6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.500+0000 7f870affd640 1 -- 172.21.15.31:0/3602062910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421ccf8f60 msgr2=0x56421d0edb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:26.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.500+0000 7f870affd640 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421ccf8f60 0x56421d0edb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.500+0000 7f870affd640 1 -- 172.21.15.31:0/3602062910 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421cd523f0 con 0x56421cd114d0 2024-04-25T16:08:26.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.500+0000 7f870affd640 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd114d0 0x56421d0e6680 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f87040c5820 tx=0x7f87040c45a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e93db83d81227ba9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:26.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.501+0000 7f86eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:08:26.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.505+0000 7f86eeff5640 1 -- 172.21.15.31:0/3602062910 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87040c80c0 con 0x56421cd114d0 2024-04-25T16:08:26.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.505+0000 7f871822b740 1 -- 172.21.15.31:0/3602062910 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56421d04e990 con 0x56421cd114d0 2024-04-25T16:08:26.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.505+0000 7f86eeff5640 1 -- 172.21.15.31:0/3602062910 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f87040c8810 con 0x56421cd114d0 2024-04-25T16:08:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.506+0000 7f86eeff5640 1 -- 172.21.15.31:0/3602062910 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f87040c8cb0 con 0x56421cd114d0 2024-04-25T16:08:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.506+0000 7f871822b740 10 client.26659 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.506+0000 7f871822b740 1 -- 172.21.15.31:0/3602062910 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56421ce1c7d0 con 0x56421cd114d0 2024-04-25T16:08:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.507+0000 7f86eeff5640 1 -- 172.21.15.31:0/3602062910 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56421d04e990 con 0x56421cd114d0 2024-04-25T16:08:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.507+0000 7f86eeff5640 1 -- 172.21.15.31:0/3602062910 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f87040c3f20 con 0x56421cd114d0 2024-04-25T16:08:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.507+0000 7f871822b740 10 client.26659 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.507+0000 7f871822b740 10 client.26659 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:08:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.507+0000 7f871822b740 20 client.26659 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.507+0000 7f871822b740 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56421ccce3a0 0x56421d0f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.507+0000 7f871822b740 4 client.26659 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.507+0000 7f871822b740 1 -- 172.21.15.31:0/3602062910 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56421d048800 con 0x56421ccce3a0 2024-04-25T16:08:26.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.509+0000 7f870a7fc640 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56421ccce3a0 0x56421d0f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.517+0000 7f870a7fc640 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56421ccce3a0 0x56421d0f46c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb498fa167c74f3 server_cookie=d348d761d948d18b in_seq=0 out_seq=0 2024-04-25T16:08:26.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.517+0000 7f86eeff5640 10 client.26659 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:08:26.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.521+0000 7f86eeff5640 1 -- 172.21.15.31:0/3602062910 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56421d048800 con 0x56421ccce3a0 2024-04-25T16:08:26.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.521+0000 7f86eeff5640 10 client.26659 handle_command_reply: tid=0 2024-04-25T16:08:26.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.521+0000 7f871822b740 1 client.26659 shutdown 2024-04-25T16:08:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.526+0000 7f871822b740 1 -- 172.21.15.31:0/3602062910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd114d0 msgr2=0x56421d0e6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.526+0000 7f871822b740 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd114d0 0x56421d0e6680 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f87040c5820 tx=0x7f87040c45a0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.526+0000 7f871822b740 1 -- 172.21.15.31:0/3602062910 shutdown_connections 2024-04-25T16:08:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.526+0000 7f871822b740 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56421ccce3a0 0x56421d0f46c0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.526+0000 7f871822b740 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421ccf8f60 0x56421d0edb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.526+0000 7f871822b740 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421cd196b0 0x56421d0e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.526+0000 7f871822b740 1 --2- 172.21.15.31:0/3602062910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421cd114d0 0x56421d0e6680 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f87040c5820 tx=0x7f87040c45a0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.526+0000 7f871822b740 1 -- 172.21.15.31:0/3602062910 >> 172.21.15.31:0/3602062910 conn(0x56421cd2e4e0 msgr2=0x56421d0d73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:26.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.528+0000 7f86eeff5640 0 client.26659 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:08:26.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.528+0000 7f871822b740 1 -- 172.21.15.31:0/3602062910 shutdown_connections 2024-04-25T16:08:26.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.528+0000 7f871822b740 1 -- 172.21.15.31:0/3602062910 wait complete. 2024-04-25T16:08:26.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.529+0000 7f871822b740 20 client.26659 trim_cache size 0 max 16384 2024-04-25T16:08:26.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.564+0000 7f871822b740 1 Processor -- start 2024-04-25T16:08:26.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.564+0000 7f871822b740 1 -- start start 2024-04-25T16:08:26.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.565+0000 7f871822b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421ccf8f60 0x56421d0e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.566+0000 7f871822b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0edae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.566+0000 7f871822b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421d0d30e0 0x56421d0d7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.566+0000 7f870a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0edae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.566+0000 7f870a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0edae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47372/0 (socket says 172.21.15.31:47372) 2024-04-25T16:08:26.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.566+0000 7f870affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421ccf8f60 0x56421d0e56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.566+0000 7f870affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421ccf8f60 0x56421d0e56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57130/0 (socket says 172.21.15.31:57130) 2024-04-25T16:08:26.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.567+0000 7f870a7fc640 1 -- 172.21.15.31:0/1261955497 learned_addr learned my addr 172.21.15.31:0/1261955497 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:26.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.568+0000 7f871822b740 1 -- 172.21.15.31:0/1261955497 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86f800e570 con 0x56421cd196b0 2024-04-25T16:08:26.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.568+0000 7f871822b740 1 -- 172.21.15.31:0/1261955497 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f86f80100b0 con 0x56421ccf8f60 2024-04-25T16:08:26.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.568+0000 7f871822b740 1 -- 172.21.15.31:0/1261955497 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86f800ec40 con 0x56421d0d30e0 2024-04-25T16:08:26.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.569+0000 7f870b7fe640 1 --2- 172.21.15.31:0/1261955497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421d0d30e0 0x56421d0d7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.569+0000 7f870a7fc640 1 -- 172.21.15.31:0/1261955497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421d0d30e0 msgr2=0x56421d0d7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:26.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.570+0000 7f870a7fc640 1 --2- 172.21.15.31:0/1261955497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421d0d30e0 0x56421d0d7bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.570+0000 7f870a7fc640 1 -- 172.21.15.31:0/1261955497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421ccf8f60 msgr2=0x56421d0e56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:26.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.570+0000 7f870a7fc640 1 --2- 172.21.15.31:0/1261955497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421ccf8f60 0x56421d0e56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.570+0000 7f870a7fc640 1 -- 172.21.15.31:0/1261955497 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56421d0e1cd0 con 0x56421cd196b0 2024-04-25T16:08:26.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.571+0000 7f870a7fc640 1 --2- 172.21.15.31:0/1261955497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0edae0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f86fc004820 tx=0x7f86fc00bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6040453ab63042d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:26.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.573+0000 7f86f0ff9640 1 -- 172.21.15.31:0/1261955497 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc014af0 con 0x56421cd196b0 2024-04-25T16:08:26.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.574+0000 7f86f0ff9640 1 -- 172.21.15.31:0/1261955497 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86fc014c90 con 0x56421cd196b0 2024-04-25T16:08:26.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.574+0000 7f86f0ff9640 1 -- 172.21.15.31:0/1261955497 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc020690 con 0x56421cd196b0 2024-04-25T16:08:26.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.576+0000 7f871822b740 1 -- 172.21.15.31:0/1261955497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 msgr2=0x56421d0edae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:26.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.576+0000 7f871822b740 1 --2- 172.21.15.31:0/1261955497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0edae0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f86fc004820 tx=0x7f86fc00bc90 comp rx=0 tx=0).stop 2024-04-25T16:08:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.578+0000 7f871822b740 1 -- 172.21.15.31:0/1261955497 shutdown_connections 2024-04-25T16:08:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.578+0000 7f871822b740 1 --2- 172.21.15.31:0/1261955497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421d0d30e0 0x56421d0d7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.578+0000 7f871822b740 1 --2- 172.21.15.31:0/1261955497 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0edae0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.578+0000 7f871822b740 1 --2- 172.21.15.31:0/1261955497 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421ccf8f60 0x56421d0e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.578+0000 7f871822b740 1 -- 172.21.15.31:0/1261955497 >> 172.21.15.31:0/1261955497 conn(0x56421cd2e4e0 msgr2=0x56421d0e4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.583+0000 7f871822b740 1 -- 172.21.15.31:0/1261955497 shutdown_connections 2024-04-25T16:08:26.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.584+0000 7f871822b740 1 -- 172.21.15.31:0/1261955497 wait complete. 2024-04-25T16:08:26.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.585+0000 7f871822b740 1 -- start start 2024-04-25T16:08:26.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.585+0000 7f871822b740 1 Processor -- start 2024-04-25T16:08:26.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.586+0000 7f871822b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421ccf8f60 0x56421d0ea970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.586+0000 7f871822b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0eaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.587+0000 7f871822b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421d0d30e0 0x56421d15aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.587+0000 7f871822b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86f800f9e0 con 0x56421cd196b0 2024-04-25T16:08:26.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.587+0000 7f871822b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f86f800f310 con 0x56421d0d30e0 2024-04-25T16:08:26.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.587+0000 7f871822b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86f800c550 con 0x56421ccf8f60 2024-04-25T16:08:26.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.587+0000 7f871822b740 10 client.0 _ll_register_callbacks cb 0x7f87040095a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:26.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.588+0000 7f870affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421ccf8f60 0x56421d0ea970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.588+0000 7f870a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0eaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.588+0000 7f870a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0eaeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47384/0 (socket says 172.21.15.31:47384) 2024-04-25T16:08:26.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.588+0000 7f870a7fc640 1 -- 172.21.15.31:0/1454249252 learned_addr learned my addr 172.21.15.31:0/1454249252 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:26.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.588+0000 7f870affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421ccf8f60 0x56421d0ea970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53992/0 (socket says 172.21.15.31:53992) 2024-04-25T16:08:26.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.589+0000 7f870b7fe640 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421d0d30e0 0x56421d15aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.589+0000 7f870a7fc640 1 -- 172.21.15.31:0/1454249252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421ccf8f60 msgr2=0x56421d0ea970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:26.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.589+0000 7f870a7fc640 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421ccf8f60 0x56421d0ea970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.589+0000 7f870a7fc640 1 -- 172.21.15.31:0/1454249252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421d0d30e0 msgr2=0x56421d15aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:26.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.589+0000 7f870a7fc640 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421d0d30e0 0x56421d15aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.589+0000 7f870a7fc640 1 -- 172.21.15.31:0/1454249252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56421d0eb4a0 con 0x56421cd196b0 2024-04-25T16:08:26.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.592+0000 7f870a7fc640 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0eaeb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f86fc008010 tx=0x7f86fc0172e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c7a5c53d8a6819 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:26.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.593+0000 7f86eeff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:08:26.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.595+0000 7f86eeff5640 1 -- 172.21.15.31:0/1454249252 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86fc014af0 con 0x56421cd196b0 2024-04-25T16:08:26.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.595+0000 7f871822b740 1 -- 172.21.15.31:0/1454249252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86fc014c90 con 0x56421cd196b0 2024-04-25T16:08:26.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.596+0000 7f86eeff5640 1 -- 172.21.15.31:0/1454249252 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f86fc017610 con 0x56421cd196b0 2024-04-25T16:08:26.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.597+0000 7f86eeff5640 1 -- 172.21.15.31:0/1454249252 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86fc017850 con 0x56421cd196b0 2024-04-25T16:08:26.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.597+0000 7f871822b740 10 client.17061 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:26.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.597+0000 7f871822b740 1 -- 172.21.15.31:0/1454249252 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56421d15bfe0 con 0x56421cd196b0 2024-04-25T16:08:26.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.598+0000 7f86eeff5640 1 -- 172.21.15.31:0/1454249252 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86fc030030 con 0x56421cd196b0 2024-04-25T16:08:26.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.599+0000 7f86eeff5640 1 -- 172.21.15.31:0/1454249252 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f86fc029070 con 0x56421cd196b0 2024-04-25T16:08:26.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.599+0000 7f871822b740 10 client.17061 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:26.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.599+0000 7f871822b740 10 client.17061 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:08:26.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.599+0000 7f871822b740 20 client.17061 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:26.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.599+0000 7f871822b740 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56421d15c850 0x56421d15ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:26.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.600+0000 7f871822b740 4 client.17061 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:08:26.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.600+0000 7f871822b740 1 -- 172.21.15.31:0/1454249252 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f86fc014c90 con 0x56421d15c850 2024-04-25T16:08:26.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.600+0000 7f870affd640 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56421d15c850 0x56421d15ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:26.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.602+0000 7f870affd640 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56421d15c850 0x56421d15ec70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=28051fa0b0aaf23e server_cookie=3be9b511e939de8 in_seq=0 out_seq=0 2024-04-25T16:08:26.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.602+0000 7f86eeff5640 10 client.17061 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:08:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.603+0000 7f86eeff5640 1 -- 172.21.15.31:0/1454249252 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f86fc014c90 con 0x56421d15c850 2024-04-25T16:08:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.603+0000 7f86eeff5640 10 client.17061 handle_command_reply: tid=0 2024-04-25T16:08:26.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.603+0000 7f871822b740 1 client.17061 shutdown 2024-04-25T16:08:26.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 -- 172.21.15.31:0/1454249252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 msgr2=0x56421d0eaeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:26.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0eaeb0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f86fc008010 tx=0x7f86fc0172e0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 -- 172.21.15.31:0/1454249252 shutdown_connections 2024-04-25T16:08:26.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56421d0d30e0 0x56421d15aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x56421d15c850 0x56421d15ec70 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56421cd196b0 0x56421d0eaeb0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 --2- 172.21.15.31:0/1454249252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56421ccf8f60 0x56421d0ea970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 -- 172.21.15.31:0/1454249252 >> 172.21.15.31:0/1454249252 conn(0x56421cd2e4e0 msgr2=0x56421d0d7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:26.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f86eeff5640 0 client.17061 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:08:26.607 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:08:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 -- 172.21.15.31:0/1454249252 shutdown_connections 2024-04-25T16:08:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.605+0000 7f871822b740 1 -- 172.21.15.31:0/1454249252 wait complete. 2024-04-25T16:08:26.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.606+0000 7f871822b740 20 client.17061 trim_cache size 0 max 16384 2024-04-25T16:08:26.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.609+0000 7f871137d640 1 -- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86f40afc70 msgr2=0x7f86f40b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:26.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.609+0000 7f871137d640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86f40afc70 0x7f86f40b2180 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f87000099c0 tx=0x7f8700009e80 comp rx=0 tx=0).stop 2024-04-25T16:08:26.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.609+0000 7f871137d640 1 -- 172.21.15.31:0/1031782187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 msgr2=0x7f870c117110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:26.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.610+0000 7f871137d640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 0x7f870c117110 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f870402aee0 tx=0x7f8704003ba0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.611+0000 7f871137d640 1 -- 172.21.15.31:0/1031782187 shutdown_connections 2024-04-25T16:08:26.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.611+0000 7f871137d640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86f40afc70 0x7f86f40b2180 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.611+0000 7f871137d640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f870c10fef0 0x7f870c1185b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.611+0000 7f871137d640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f870c10b1d0 0x7f870c117650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.611+0000 7f871137d640 1 --2- 172.21.15.31:0/1031782187 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f870c10a3e0 0x7f870c117110 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:26.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.611+0000 7f871137d640 1 -- 172.21.15.31:0/1031782187 >> 172.21.15.31:0/1031782187 conn(0x7f870c070400 msgr2=0x7f870c10bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:26.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.617+0000 7f871137d640 1 -- 172.21.15.31:0/1031782187 shutdown_connections 2024-04-25T16:08:26.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:26.621+0000 7f871137d640 1 -- 172.21.15.31:0/1031782187 wait complete. 2024-04-25T16:08:26.660 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:08:26.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:26 smithi031 ceph-mon[29140]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 747 MiB/s wr, 1.85k op/s 2024-04-25T16:08:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.311+0000 7f833272d640 1 -- 172.21.15.31:0/2311686804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c10b310 msgr2=0x7f832c10b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.311+0000 7f833272d640 1 --2- 172.21.15.31:0/2311686804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c10b310 0x7f832c10b720 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8320009860 tx=0x7f832002a380 comp rx=0 tx=0).stop 2024-04-25T16:08:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.313+0000 7f833272d640 1 -- 172.21.15.31:0/2311686804 shutdown_connections 2024-04-25T16:08:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.313+0000 7f833272d640 1 --2- 172.21.15.31:0/2311686804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f832c110030 0x7f832c110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.313+0000 7f833272d640 1 --2- 172.21.15.31:0/2311686804 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c10b310 0x7f832c10b720 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.313+0000 7f833272d640 1 --2- 172.21.15.31:0/2311686804 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c10a420 0x7f832c10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.313+0000 7f833272d640 1 -- 172.21.15.31:0/2311686804 >> 172.21.15.31:0/2311686804 conn(0x7f832c0704e0 msgr2=0x7f832c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:27.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.319+0000 7f833272d640 1 -- 172.21.15.31:0/2311686804 shutdown_connections 2024-04-25T16:08:27.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.322+0000 7f833272d640 1 -- 172.21.15.31:0/2311686804 wait complete. 2024-04-25T16:08:27.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.323+0000 7f833272d640 1 Processor -- start 2024-04-25T16:08:27.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.326+0000 7f833272d640 1 -- start start 2024-04-25T16:08:27.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.327+0000 7f833272d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f832c10a420 0x7f832c1a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.327+0000 7f833272d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c10b310 0x7f832c1a9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.327+0000 7f833272d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c110030 0x7f832c1b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.327+0000 7f833272d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f832c11e4e0 con 0x7f832c110030 2024-04-25T16:08:27.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.327+0000 7f833272d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f832c11e360 con 0x7f832c10b310 2024-04-25T16:08:27.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.327+0000 7f833272d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f832c11e660 con 0x7f832c10a420 2024-04-25T16:08:27.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.332+0000 7f832b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c10b310 0x7f832c1a9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.334+0000 7f832b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c10b310 0x7f832c1a9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57160/0 (socket says 172.21.15.31:57160) 2024-04-25T16:08:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.334+0000 7f832b7fe640 1 -- 172.21.15.31:0/3717176062 learned_addr learned my addr 172.21.15.31:0/3717176062 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.334+0000 7f832bfff640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f832c10a420 0x7f832c1a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.335+0000 7f8330cca640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c110030 0x7f832c1b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.335+0000 7f8330cca640 1 -- 172.21.15.31:0/3717176062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f832c10a420 msgr2=0x7f832c1a90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.335+0000 7f8330cca640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f832c10a420 0x7f832c1a90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.335+0000 7f8330cca640 1 -- 172.21.15.31:0/3717176062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c10b310 msgr2=0x7f832c1a9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.335+0000 7f8330cca640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c10b310 0x7f832c1a9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.335+0000 7f8330cca640 1 -- 172.21.15.31:0/3717176062 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832c111c90 con 0x7f832c110030 2024-04-25T16:08:27.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.336+0000 7f8330cca640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c110030 0x7f832c1b05c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f832400ef00 tx=0x7f832400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:27.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.340+0000 7f83297fa640 1 -- 172.21.15.31:0/3717176062 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8324004b20 con 0x7f832c110030 2024-04-25T16:08:27.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.340+0000 7f83297fa640 1 -- 172.21.15.31:0/3717176062 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8324003e20 con 0x7f832c110030 2024-04-25T16:08:27.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.340+0000 7f83297fa640 1 -- 172.21.15.31:0/3717176062 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832400b570 con 0x7f832c110030 2024-04-25T16:08:27.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.341+0000 7f833272d640 1 -- 172.21.15.31:0/3717176062 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f832c111f80 con 0x7f832c110030 2024-04-25T16:08:27.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.341+0000 7f833272d640 1 -- 172.21.15.31:0/3717176062 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f832c112480 con 0x7f832c110030 2024-04-25T16:08:27.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.350+0000 7f83297fa640 1 -- 172.21.15.31:0/3717176062 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f832401d020 con 0x7f832c110030 2024-04-25T16:08:27.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.355+0000 7f83297fa640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83140afd00 0x7f83140b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.360 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.358+0000 7f83395de740 1 Processor -- start 2024-04-25T16:08:27.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.359+0000 7f83297fa640 1 -- 172.21.15.31:0/3717176062 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f83240dea10 con 0x7f832c110030 2024-04-25T16:08:27.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.362+0000 7f832bfff640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83140afd00 0x7f83140b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.363+0000 7f83395de740 1 -- start start 2024-04-25T16:08:27.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.363+0000 7f83395de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88d008390 0x55c88d3aa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.363+0000 7f83395de740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfdf4d0 0x55c88d3aad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.364+0000 7f83395de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfe76b0 0x55c88d3aba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.364+0000 7f83395de740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c88d30ed80 con 0x55c88cfdf4d0 2024-04-25T16:08:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.364+0000 7f83395de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c88d2c8a20 con 0x55c88cfe76b0 2024-04-25T16:08:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.364+0000 7f83395de740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c88d30ad40 con 0x55c88d008390 2024-04-25T16:08:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.367+0000 7f8330cca640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfe76b0 0x55c88d3aba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.367+0000 7f8330cca640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfe76b0 0x55c88d3aba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57176/0 (socket says 172.21.15.31:57176) 2024-04-25T16:08:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.367+0000 7f8330cca640 1 -- 172.21.15.31:0/1675875404 learned_addr learned my addr 172.21.15.31:0/1675875404 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:27.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.367+0000 7f832bfff640 1 --2- 172.21.15.31:0/1675875404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88d008390 0x55c88d3aa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:26 smithi063 ceph-mon[32413]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 747 MiB/s wr, 1.85k op/s 2024-04-25T16:08:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.370+0000 7f832b7fe640 1 --2- 172.21.15.31:0/1675875404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfdf4d0 0x55c88d3aad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.370+0000 7f8330cca640 1 -- 172.21.15.31:0/1675875404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88d008390 msgr2=0x55c88d3aa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.370+0000 7f8330cca640 1 --2- 172.21.15.31:0/1675875404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88d008390 0x55c88d3aa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.370+0000 7f8330cca640 1 -- 172.21.15.31:0/1675875404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfdf4d0 msgr2=0x55c88d3aad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:27.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.370+0000 7f8330cca640 1 --2- 172.21.15.31:0/1675875404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfdf4d0 0x55c88d3aad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.370+0000 7f8330cca640 1 -- 172.21.15.31:0/1675875404 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c88d2d3c30 con 0x55c88cfe76b0 2024-04-25T16:08:27.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.371+0000 7f8330cca640 1 --2- 172.21.15.31:0/1675875404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfe76b0 0x55c88d3aba20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f83240a2f20 tx=0x7f83240a6af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec14b2a0d567c9d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:27.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.371+0000 7f83127fc640 1 -- 172.21.15.31:0/1675875404 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f832401a040 con 0x55c88cfe76b0 2024-04-25T16:08:27.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.372+0000 7f83127fc640 1 -- 172.21.15.31:0/1675875404 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83240ab250 con 0x55c88cfe76b0 2024-04-25T16:08:27.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.372+0000 7f83127fc640 1 -- 172.21.15.31:0/1675875404 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83240af870 con 0x55c88cfe76b0 2024-04-25T16:08:27.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.377+0000 7f83395de740 1 -- 172.21.15.31:0/1675875404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfe76b0 msgr2=0x55c88d3aba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:27.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.378+0000 7f83395de740 1 --2- 172.21.15.31:0/1675875404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfe76b0 0x55c88d3aba20 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f83240a2f20 tx=0x7f83240a6af0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.382+0000 7f83395de740 1 -- 172.21.15.31:0/1675875404 shutdown_connections 2024-04-25T16:08:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.382+0000 7f83395de740 1 --2- 172.21.15.31:0/1675875404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfe76b0 0x55c88d3aba20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.382+0000 7f83395de740 1 --2- 172.21.15.31:0/1675875404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfdf4d0 0x55c88d3aad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.382+0000 7f83395de740 1 --2- 172.21.15.31:0/1675875404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88d008390 0x55c88d3aa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.382+0000 7f83395de740 1 -- 172.21.15.31:0/1675875404 >> 172.21.15.31:0/1675875404 conn(0x55c88cffc4e0 msgr2=0x55c88d3a6020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:27.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.383+0000 7f832bfff640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83140afd00 0x7f83140b2210 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f832c109c60 tx=0x7f831c015850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:27.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.387+0000 7f83395de740 1 -- 172.21.15.31:0/1675875404 shutdown_connections 2024-04-25T16:08:27.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.389+0000 7f83395de740 1 -- 172.21.15.31:0/1675875404 wait complete. 2024-04-25T16:08:27.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.390+0000 7f83395de740 1 -- start start 2024-04-25T16:08:27.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.391+0000 7f83395de740 1 Processor -- start 2024-04-25T16:08:27.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.391+0000 7f83395de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.392+0000 7f83395de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 0x55c88d3b4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.392+0000 7f83395de740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88d008390 0x55c88d3bb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.392+0000 7f83395de740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c88d3ac3b0 con 0x55c88d008390 2024-04-25T16:08:27.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.392+0000 7f83395de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c88d295dc0 con 0x55c88cfdf4d0 2024-04-25T16:08:27.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.392+0000 7f83395de740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c88d3ac530 con 0x55c88cfe76b0 2024-04-25T16:08:27.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.392+0000 7f83395de740 10 client.0 _ll_register_callbacks cb 0x55c88d2b1600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:27.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.394+0000 7f832bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.394+0000 7f832bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b4210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57188/0 (socket says 172.21.15.31:57188) 2024-04-25T16:08:27.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.394+0000 7f832bfff640 1 -- 172.21.15.31:0/2762722544 learned_addr learned my addr 172.21.15.31:0/2762722544 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:27.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.395+0000 7f832b7fe640 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 0x55c88d3b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.396+0000 7f832b7fe640 1 -- 172.21.15.31:0/2762722544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 msgr2=0x55c88d3b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:27.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.396+0000 7f832b7fe640 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.396+0000 7f832b7fe640 1 -- 172.21.15.31:0/2762722544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88d008390 msgr2=0x55c88d3bb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:27.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.396+0000 7f832b7fe640 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88d008390 0x55c88d3bb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.396+0000 7f832b7fe640 1 -- 172.21.15.31:0/2762722544 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c88d0203f0 con 0x55c88cfe76b0 2024-04-25T16:08:27.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:26 smithi186 ceph-mon[33258]: pgmap v565: 97 pgs: 97 active+clean; 10 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 747 MiB/s wr, 1.85k op/s 2024-04-25T16:08:27.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.398+0000 7f832b7fe640 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 0x55c88d3b4750 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8320009830 tx=0x7f83200307b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b949e28c5e679dc server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:27.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.399+0000 7f830b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:08:27.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.399+0000 7f830b7fe640 1 -- 172.21.15.31:0/2762722544 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8320037040 con 0x55c88cfe76b0 2024-04-25T16:08:27.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.400+0000 7f83395de740 1 -- 172.21.15.31:0/2762722544 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83240ab250 con 0x55c88cfe76b0 2024-04-25T16:08:27.402 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.400+0000 7f830b7fe640 1 -- 172.21.15.31:0/2762722544 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8320004450 con 0x55c88cfe76b0 2024-04-25T16:08:27.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.401+0000 7f830b7fe640 1 -- 172.21.15.31:0/2762722544 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f832004f050 con 0x55c88cfe76b0 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.401+0000 7f83395de740 10 client.26674 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.401+0000 7f83395de740 1 -- 172.21.15.31:0/2762722544 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c88d0ea7d0 con 0x55c88cfe76b0 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.402+0000 7f830b7fe640 1 -- 172.21.15.31:0/2762722544 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f83240ab250 con 0x55c88cfe76b0 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.402+0000 7f830b7fe640 1 -- 172.21.15.31:0/2762722544 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f832003d040 con 0x55c88cfe76b0 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.403+0000 7f83395de740 10 client.26674 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.403+0000 7f83395de740 10 client.26674 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.403+0000 7f83395de740 20 client.26674 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.403+0000 7f83395de740 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c88cfc6f60 0x55c88d3c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.403+0000 7f83395de740 4 client.26674 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:27.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.403+0000 7f83395de740 1 -- 172.21.15.31:0/2762722544 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c88d31c990 con 0x55c88cfc6f60 2024-04-25T16:08:27.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.404+0000 7f832bfff640 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c88cfc6f60 0x55c88d3c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.406+0000 7f832bfff640 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c88cfc6f60 0x55c88d3c24a0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43ca9454c2776a0f server_cookie=bf5c816d26fb43a2 in_seq=0 out_seq=0 2024-04-25T16:08:27.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.407+0000 7f830b7fe640 10 client.26674 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:27.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.409+0000 7f830b7fe640 1 -- 172.21.15.31:0/2762722544 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c88d31c990 con 0x55c88cfc6f60 2024-04-25T16:08:27.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.409+0000 7f830b7fe640 10 client.26674 handle_command_reply: tid=0 2024-04-25T16:08:27.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.409+0000 7f83395de740 1 client.26674 shutdown 2024-04-25T16:08:27.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.413+0000 7f83395de740 1 -- 172.21.15.31:0/2762722544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 msgr2=0x55c88d3b4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:27.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.413+0000 7f83395de740 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 0x55c88d3b4750 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8320009830 tx=0x7f83200307b0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.416+0000 7f83395de740 1 -- 172.21.15.31:0/2762722544 shutdown_connections 2024-04-25T16:08:27.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.416+0000 7f83395de740 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c88cfc6f60 0x55c88d3c24a0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.416+0000 7f83395de740 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88d008390 0x55c88d3bb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.416+0000 7f83395de740 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 0x55c88d3b4750 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.416+0000 7f83395de740 1 --2- 172.21.15.31:0/2762722544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.418 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.416+0000 7f83395de740 1 -- 172.21.15.31:0/2762722544 >> 172.21.15.31:0/2762722544 conn(0x55c88cffc4e0 msgr2=0x55c88d3a5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.424+0000 7f830b7fe640 0 client.26674 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.425+0000 7f83395de740 1 -- 172.21.15.31:0/2762722544 shutdown_connections 2024-04-25T16:08:27.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.425+0000 7f83395de740 1 -- 172.21.15.31:0/2762722544 wait complete. 2024-04-25T16:08:27.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.425+0000 7f83395de740 20 client.26674 trim_cache size 0 max 16384 2024-04-25T16:08:27.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.461+0000 7f83395de740 1 Processor -- start 2024-04-25T16:08:27.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.462+0000 7f83395de740 1 -- start start 2024-04-25T16:08:27.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.463+0000 7f83395de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.463+0000 7f83395de740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfe76b0 0x55c88d3bb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.464+0000 7f83395de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88d008390 0x55c88d3b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.464+0000 7f83395de740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f831800e550 con 0x55c88cfe76b0 2024-04-25T16:08:27.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.464+0000 7f83395de740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8318010090 con 0x55c88d008390 2024-04-25T16:08:27.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.464+0000 7f83395de740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f831800ec20 con 0x55c88cfdf4d0 2024-04-25T16:08:27.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.464+0000 7f832bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.464+0000 7f832bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54026/0 (socket says 172.21.15.31:54026) 2024-04-25T16:08:27.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.464+0000 7f832bfff640 1 -- 172.21.15.31:0/3089259695 learned_addr learned my addr 172.21.15.31:0/3089259695 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:27.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.468+0000 7f832b7fe640 1 --2- 172.21.15.31:0/3089259695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfe76b0 0x55c88d3bb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.469+0000 7f8330cca640 1 --2- 172.21.15.31:0/3089259695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88d008390 0x55c88d3b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.472+0000 7f832bfff640 1 -- 172.21.15.31:0/3089259695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88d008390 msgr2=0x55c88d3b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:27.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.472+0000 7f832bfff640 1 --2- 172.21.15.31:0/3089259695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88d008390 0x55c88d3b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.472+0000 7f832bfff640 1 -- 172.21.15.31:0/3089259695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfe76b0 msgr2=0x55c88d3bb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:27.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.472+0000 7f832bfff640 1 --2- 172.21.15.31:0/3089259695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfe76b0 0x55c88d3bb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.475 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.472+0000 7f832bfff640 1 -- 172.21.15.31:0/3089259695 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c88d3bbb10 con 0x55c88cfdf4d0 2024-04-25T16:08:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.477+0000 7f832bfff640 1 --2- 172.21.15.31:0/3089259695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b27f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f831c005810 tx=0x7f831c018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=109c21b53261b002 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:27.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.479+0000 7f8311ffb640 1 -- 172.21.15.31:0/3089259695 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c012ce0 con 0x55c88cfdf4d0 2024-04-25T16:08:27.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.479+0000 7f8311ffb640 1 -- 172.21.15.31:0/3089259695 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f831c029410 con 0x55c88cfdf4d0 2024-04-25T16:08:27.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.479+0000 7f8311ffb640 1 -- 172.21.15.31:0/3089259695 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c003850 con 0x55c88cfdf4d0 2024-04-25T16:08:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.481+0000 7f83395de740 1 -- 172.21.15.31:0/3089259695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfdf4d0 msgr2=0x55c88d3b27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.481+0000 7f83395de740 1 --2- 172.21.15.31:0/3089259695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b27f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f831c005810 tx=0x7f831c018000 comp rx=0 tx=0).stop 2024-04-25T16:08:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.483+0000 7f83395de740 1 -- 172.21.15.31:0/3089259695 shutdown_connections 2024-04-25T16:08:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.483+0000 7f83395de740 1 --2- 172.21.15.31:0/3089259695 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88d008390 0x55c88d3b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.483+0000 7f83395de740 1 --2- 172.21.15.31:0/3089259695 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88cfe76b0 0x55c88d3bb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.483+0000 7f83395de740 1 --2- 172.21.15.31:0/3089259695 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfdf4d0 0x55c88d3b27f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.483+0000 7f83395de740 1 -- 172.21.15.31:0/3089259695 >> 172.21.15.31:0/3089259695 conn(0x55c88cffc4e0 msgr2=0x55c88d3af0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.483+0000 7f83395de740 1 -- 172.21.15.31:0/3089259695 shutdown_connections 2024-04-25T16:08:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.483+0000 7f83395de740 1 -- 172.21.15.31:0/3089259695 wait complete. 2024-04-25T16:08:27.489 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.485+0000 7f83395de740 1 -- start start 2024-04-25T16:08:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.486+0000 7f83395de740 1 Processor -- start 2024-04-25T16:08:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.487+0000 7f83395de740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3bb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.487+0000 7f83395de740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 0x55c88d3b7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.488+0000 7f83395de740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88d008390 0x55c88d3b8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.488+0000 7f83395de740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f831800f9c0 con 0x55c88d008390 2024-04-25T16:08:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.488+0000 7f832bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3bb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.488+0000 7f832bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3bb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57210/0 (socket says 172.21.15.31:57210) 2024-04-25T16:08:27.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.488+0000 7f832bfff640 1 -- 172.21.15.31:0/203838984 learned_addr learned my addr 172.21.15.31:0/203838984 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:27.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.488+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f831800f2f0 con 0x55c88cfdf4d0 2024-04-25T16:08:27.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.488+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f831800c530 con 0x55c88cfe76b0 2024-04-25T16:08:27.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.488+0000 7f832b7fe640 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 0x55c88d3b7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.488+0000 7f83395de740 10 client.0 _ll_register_callbacks cb 0x7f8324009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:27.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.489+0000 7f832bfff640 1 -- 172.21.15.31:0/203838984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 msgr2=0x55c88d3b7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:27.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.489+0000 7f832bfff640 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 0x55c88d3b7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.489+0000 7f832bfff640 1 -- 172.21.15.31:0/203838984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88d008390 msgr2=0x55c88d3b8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:27.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.489+0000 7f832bfff640 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88d008390 0x55c88d3b8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.489+0000 7f832bfff640 1 -- 172.21.15.31:0/203838984 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c88d3b87c0 con 0x55c88cfdf4d0 2024-04-25T16:08:27.503 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.500+0000 7f832bfff640 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3bb1c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f831c006000 tx=0x7f831c0295c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23baf2d9fe5068f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:27.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.501+0000 7f830b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:08:27.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.507+0000 7f830b7fe640 1 -- 172.21.15.31:0/203838984 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c029a30 con 0x55c88cfdf4d0 2024-04-25T16:08:27.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.507+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f831c029410 con 0x55c88cfdf4d0 2024-04-25T16:08:27.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.507+0000 7f830b7fe640 1 -- 172.21.15.31:0/203838984 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f831c005040 con 0x55c88cfdf4d0 2024-04-25T16:08:27.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.510+0000 7f830b7fe640 1 -- 172.21.15.31:0/203838984 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f831c0052c0 con 0x55c88cfdf4d0 2024-04-25T16:08:27.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.510+0000 7f83395de740 10 client.26726 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:27.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.511+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c88d429770 con 0x55c88cfdf4d0 2024-04-25T16:08:27.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.514+0000 7f830b7fe640 1 -- 172.21.15.31:0/203838984 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f831c029410 con 0x55c88cfdf4d0 2024-04-25T16:08:27.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.514+0000 7f830b7fe640 1 -- 172.21.15.31:0/203838984 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f831c029ce0 con 0x55c88cfdf4d0 2024-04-25T16:08:27.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.516+0000 7f83395de740 10 client.26726 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:27.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.516+0000 7f83395de740 10 client.26726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:27.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.516+0000 7f83395de740 20 client.26726 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:27.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.516+0000 7f83395de740 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c88cf9c3a0 0x55c88d42c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:27.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.516+0000 7f83395de740 4 client.26726 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:08:27.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.516+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c88d316800 con 0x55c88cf9c3a0 2024-04-25T16:08:27.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.527+0000 7f832b7fe640 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c88cf9c3a0 0x55c88d42c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:27.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.541+0000 7f830b7fe640 10 client.26726 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:27.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.541+0000 7f832b7fe640 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c88cf9c3a0 0x55c88d42c0a0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ce0bdc3fa13de95 server_cookie=51c74f50a1374b7 in_seq=0 out_seq=0 2024-04-25T16:08:27.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.545+0000 7f830b7fe640 1 -- 172.21.15.31:0/203838984 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c88d316800 con 0x55c88cf9c3a0 2024-04-25T16:08:27.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.545+0000 7f830b7fe640 10 client.26726 handle_command_reply: tid=0 2024-04-25T16:08:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.545+0000 7f83395de740 1 client.26726 shutdown 2024-04-25T16:08:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 msgr2=0x55c88d3bb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3bb1c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f831c006000 tx=0x7f831c0295c0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 shutdown_connections 2024-04-25T16:08:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55c88cf9c3a0 0x55c88d42c0a0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c88d008390 0x55c88d3b8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c88cfe76b0 0x55c88d3b7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 --2- 172.21.15.31:0/203838984 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c88cfdf4d0 0x55c88d3bb1c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 >> 172.21.15.31:0/203838984 conn(0x55c88cffc4e0 msgr2=0x55c88d3a8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f830b7fe640 0 client.26726 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:27.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 shutdown_connections 2024-04-25T16:08:27.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.548+0000 7f83395de740 1 -- 172.21.15.31:0/203838984 wait complete. 2024-04-25T16:08:27.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.551+0000 7f83395de740 20 client.26726 trim_cache size 0 max 16384 2024-04-25T16:08:27.553 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:08:27.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.556+0000 7f833272d640 1 -- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83140afd00 msgr2=0x7f83140b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:27.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.556+0000 7f833272d640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83140afd00 0x7f83140b2210 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f832c109c60 tx=0x7f831c015850 comp rx=0 tx=0).stop 2024-04-25T16:08:27.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.557+0000 7f833272d640 1 -- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c110030 msgr2=0x7f832c1b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:27.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.557+0000 7f833272d640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c110030 0x7f832c1b05c0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f832400ef00 tx=0x7f832400c740 comp rx=0 tx=0).stop 2024-04-25T16:08:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.559+0000 7f833272d640 1 -- 172.21.15.31:0/3717176062 shutdown_connections 2024-04-25T16:08:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.559+0000 7f833272d640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f83140afd00 0x7f83140b2210 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.559+0000 7f833272d640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f832c110030 0x7f832c1b05c0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.559+0000 7f833272d640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f832c10b310 0x7f832c1a9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.559+0000 7f833272d640 1 --2- 172.21.15.31:0/3717176062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f832c10a420 0x7f832c1a90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.559+0000 7f833272d640 1 -- 172.21.15.31:0/3717176062 >> 172.21.15.31:0/3717176062 conn(0x7f832c0704e0 msgr2=0x7f832c10c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.560+0000 7f833272d640 1 -- 172.21.15.31:0/3717176062 shutdown_connections 2024-04-25T16:08:27.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:27.560+0000 7f833272d640 1 -- 172.21.15.31:0/3717176062 wait complete. 2024-04-25T16:08:27.918 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:27 smithi031 ceph-mon[29140]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 885 MiB/s wr, 2.25k op/s 2024-04-25T16:08:28.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:27 smithi063 ceph-mon[32413]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 885 MiB/s wr, 2.25k op/s 2024-04-25T16:08:28.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:27 smithi186 ceph-mon[33258]: pgmap v566: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 885 MiB/s wr, 2.25k op/s 2024-04-25T16:08:28.623 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:08:28.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:08:28.623 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:08:29.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.188+0000 7efde7627640 1 -- 172.21.15.31:0/4119280449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde010fef0 msgr2=0x7efde0110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:29.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.188+0000 7efde7627640 1 --2- 172.21.15.31:0/4119280449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde010fef0 0x7efde0110300 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009900 tx=0x7efdd002a440 comp rx=0 tx=0).stop 2024-04-25T16:08:29.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.196+0000 7efde7627640 1 -- 172.21.15.31:0/4119280449 shutdown_connections 2024-04-25T16:08:29.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.196+0000 7efde7627640 1 --2- 172.21.15.31:0/4119280449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde010fef0 0x7efde0110300 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.196+0000 7efde7627640 1 --2- 172.21.15.31:0/4119280449 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efde010b1d0 0x7efde010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.196+0000 7efde7627640 1 --2- 172.21.15.31:0/4119280449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efde010a3e0 0x7efde010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.196+0000 7efde7627640 1 -- 172.21.15.31:0/4119280449 >> 172.21.15.31:0/4119280449 conn(0x7efde0070400 msgr2=0x7efde0072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:29.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.205+0000 7efde7627640 1 -- 172.21.15.31:0/4119280449 shutdown_connections 2024-04-25T16:08:29.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.205+0000 7efde7627640 1 -- 172.21.15.31:0/4119280449 wait complete. 2024-04-25T16:08:29.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.206+0000 7efde7627640 1 Processor -- start 2024-04-25T16:08:29.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.206+0000 7efde7627640 1 -- start start 2024-04-25T16:08:29.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.207+0000 7efde7627640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde010a3e0 0x7efde0117090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.207+0000 7efde7627640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efde010b1d0 0x7efde01175d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.207+0000 7efde7627640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efde010fef0 0x7efde01184e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.207+0000 7efde7627640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efde011e290 con 0x7efde010b1d0 2024-04-25T16:08:29.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.207+0000 7efde7627640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efde011e110 con 0x7efde010fef0 2024-04-25T16:08:29.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.208+0000 7efde7627640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efde011e410 con 0x7efde010a3e0 2024-04-25T16:08:29.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.208+0000 7efde4bc2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efde010b1d0 0x7efde01175d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.208+0000 7efde4bc2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efde010b1d0 0x7efde01175d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47468/0 (socket says 172.21.15.31:47468) 2024-04-25T16:08:29.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.208+0000 7efde4bc2640 1 -- 172.21.15.31:0/1838084626 learned_addr learned my addr 172.21.15.31:0/1838084626 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:29.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.211+0000 7efde53c3640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde010a3e0 0x7efde0117090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.211+0000 7efde5bc4640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efde010fef0 0x7efde01184e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.211+0000 7efde4bc2640 1 -- 172.21.15.31:0/1838084626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde010a3e0 msgr2=0x7efde0117090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.212+0000 7efde4bc2640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde010a3e0 0x7efde0117090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.212+0000 7efde4bc2640 1 -- 172.21.15.31:0/1838084626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efde010fef0 msgr2=0x7efde01184e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.212+0000 7efde4bc2640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efde010fef0 0x7efde01184e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.212+0000 7efde4bc2640 1 -- 172.21.15.31:0/1838084626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efde0118b50 con 0x7efde010b1d0 2024-04-25T16:08:29.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.216+0000 7efde4bc2640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efde010b1d0 0x7efde01175d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7efdd400d950 tx=0x7efdd400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:29.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.221+0000 7efdde7fc640 1 -- 172.21.15.31:0/1838084626 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd4017070 con 0x7efde010b1d0 2024-04-25T16:08:29.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.221+0000 7efdde7fc640 1 -- 172.21.15.31:0/1838084626 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdd4009c30 con 0x7efde010b1d0 2024-04-25T16:08:29.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.221+0000 7efde7627640 1 -- 172.21.15.31:0/1838084626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efde0118e40 con 0x7efde010b1d0 2024-04-25T16:08:29.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.221+0000 7efde7627640 1 -- 172.21.15.31:0/1838084626 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efde0119410 con 0x7efde010b1d0 2024-04-25T16:08:29.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.222+0000 7efdde7fc640 1 -- 172.21.15.31:0/1838084626 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd400f620 con 0x7efde010b1d0 2024-04-25T16:08:29.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.230+0000 7efdde7fc640 1 -- 172.21.15.31:0/1838084626 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7efdd400f7c0 con 0x7efde010b1d0 2024-04-25T16:08:29.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.232+0000 7efdee5b8740 1 Processor -- start 2024-04-25T16:08:29.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.233+0000 7efdde7fc640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdc80afd00 0x7efdc80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.233+0000 7efdde7fc640 1 -- 172.21.15.31:0/1838084626 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7efdd40de790 con 0x7efde010b1d0 2024-04-25T16:08:29.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.236+0000 7efdee5b8740 1 -- start start 2024-04-25T16:08:29.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.237+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb914f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.237+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb914fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.237+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d8d730 0x55efb9156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.237+0000 7efdee5b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55efb90b4b00 con 0x55efb8d8d730 2024-04-25T16:08:29.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.237+0000 7efdee5b8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55efb9050920 con 0x55efb8dae410 2024-04-25T16:08:29.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.237+0000 7efdee5b8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55efb90b0ac0 con 0x55efb8d85550 2024-04-25T16:08:29.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.238+0000 7efde4bc2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb914fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.239+0000 7efde4bc2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb914fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54058/0 (socket says 172.21.15.31:54058) 2024-04-25T16:08:29.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.239+0000 7efde4bc2640 1 -- 172.21.15.31:0/4031058129 learned_addr learned my addr 172.21.15.31:0/4031058129 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:29.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.239+0000 7efde5bc4640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdc80afd00 0x7efdc80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.239+0000 7efde53c3640 1 --2- 172.21.15.31:0/4031058129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb914f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.239+0000 7efde5bc4640 1 --2- 172.21.15.31:0/4031058129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d8d730 0x55efb9156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.239+0000 7efde4bc2640 1 -- 172.21.15.31:0/4031058129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 msgr2=0x55efb914f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:29.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.239+0000 7efde4bc2640 1 --2- 172.21.15.31:0/4031058129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb914f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.239+0000 7efde4bc2640 1 -- 172.21.15.31:0/4031058129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d8d730 msgr2=0x55efb9156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:29.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.239+0000 7efde4bc2640 1 --2- 172.21.15.31:0/4031058129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d8d730 0x55efb9156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.240+0000 7efde4bc2640 1 -- 172.21.15.31:0/4031058129 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb8e8e7e0 con 0x55efb8d85550 2024-04-25T16:08:29.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.248+0000 7efde4bc2640 1 --2- 172.21.15.31:0/4031058129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb914fbe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009610 tx=0x7efdd40af3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=643f8ce17f79da0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:29.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.249+0000 7efde5bc4640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdc80afd00 0x7efdc80b2210 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efdd00096c0 tx=0x7efdd0009630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:29.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.250+0000 7efdc77fe640 1 -- 172.21.15.31:0/4031058129 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd4020040 con 0x55efb8d85550 2024-04-25T16:08:29.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.251+0000 7efdc77fe640 1 -- 172.21.15.31:0/4031058129 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdd40a6c20 con 0x55efb8d85550 2024-04-25T16:08:29.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.252+0000 7efdee5b8740 1 -- 172.21.15.31:0/4031058129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 msgr2=0x55efb914fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:29.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.252+0000 7efdee5b8740 1 --2- 172.21.15.31:0/4031058129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb914fbe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efdd4009610 tx=0x7efdd40af3c0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.253+0000 7efdee5b8740 1 -- 172.21.15.31:0/4031058129 shutdown_connections 2024-04-25T16:08:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.253+0000 7efdee5b8740 1 --2- 172.21.15.31:0/4031058129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d8d730 0x55efb9156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.253+0000 7efdee5b8740 1 --2- 172.21.15.31:0/4031058129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb914fbe0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.253+0000 7efdee5b8740 1 --2- 172.21.15.31:0/4031058129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb914f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.253+0000 7efdee5b8740 1 -- 172.21.15.31:0/4031058129 >> 172.21.15.31:0/4031058129 conn(0x55efb8da2560 msgr2=0x55efb914aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:29.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.253+0000 7efdee5b8740 1 -- 172.21.15.31:0/4031058129 shutdown_connections 2024-04-25T16:08:29.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.253+0000 7efdee5b8740 1 -- 172.21.15.31:0/4031058129 wait complete. 2024-04-25T16:08:29.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.254+0000 7efdee5b8740 1 -- start start 2024-04-25T16:08:29.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.255+0000 7efdee5b8740 1 Processor -- start 2024-04-25T16:08:29.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.256+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb9159460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.256+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb91599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.256+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8dae410 0x55efb9160900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.256+0000 7efde4bc2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb91599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.258 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.257+0000 7efde4bc2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb91599a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57238/0 (socket says 172.21.15.31:57238) 2024-04-25T16:08:29.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.257+0000 7efde4bc2640 1 -- 172.21.15.31:0/655219249 learned_addr learned my addr 172.21.15.31:0/655219249 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:29.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.257+0000 7efdee5b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55efb9157520 con 0x55efb8dae410 2024-04-25T16:08:29.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.257+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55efb904e5d0 con 0x55efb8d8d730 2024-04-25T16:08:29.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.257+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55efb91576a0 con 0x55efb8d85550 2024-04-25T16:08:29.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.257+0000 7efde53c3640 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb9159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.257+0000 7efdee5b8740 10 client.0 _ll_register_callbacks cb 0x55efb903c080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:29.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.258+0000 7efde4bc2640 1 -- 172.21.15.31:0/655219249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 msgr2=0x55efb9159460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:29.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.258+0000 7efde4bc2640 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb9159460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.258+0000 7efde4bc2640 1 -- 172.21.15.31:0/655219249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8dae410 msgr2=0x55efb9160900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:29.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.258+0000 7efde4bc2640 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8dae410 0x55efb9160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.258+0000 7efde4bc2640 1 -- 172.21.15.31:0/655219249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb8eeef70 con 0x55efb8d8d730 2024-04-25T16:08:29.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.258+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdd40a6c20 con 0x55efb8d8d730 2024-04-25T16:08:29.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.258+0000 7efde4bc2640 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb91599a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efdd40a2ca0 tx=0x7efdd40a62e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33306ea5149f2280 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.282+0000 7efdc4ff9640 10 client.26735 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:08:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.282+0000 7efdc4ff9640 1 -- 172.21.15.31:0/655219249 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd4020070 con 0x55efb8d8d730 2024-04-25T16:08:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.282+0000 7efdc4ff9640 1 -- 172.21.15.31:0/655219249 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7efdd40a58f0 con 0x55efb8d8d730 2024-04-25T16:08:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.282+0000 7efdc4ff9640 1 -- 172.21.15.31:0/655219249 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efdd40c1310 con 0x55efb8d8d730 2024-04-25T16:08:29.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.283+0000 7efdee5b8740 10 client.26735 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:29.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.283+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55efb8d86170 con 0x55efb8d8d730 2024-04-25T16:08:29.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.284+0000 7efdc4ff9640 1 -- 172.21.15.31:0/655219249 <== mon.2 v2:172.21.15.63:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7efdd40ac160 con 0x55efb8d8d730 2024-04-25T16:08:29.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.288+0000 7efdee5b8740 10 client.26735 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:29.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.288+0000 7efdee5b8740 10 client.26735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:29.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.288+0000 7efdee5b8740 20 client.26735 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:29.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.288+0000 7efdee5b8740 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55efb8d6d1f0 0x55efb91668c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.288+0000 7efdee5b8740 4 client.26735 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:29.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.288+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55efb90c2710 con 0x55efb8d6d1f0 2024-04-25T16:08:29.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.289+0000 7efde53c3640 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55efb8d6d1f0 0x55efb91668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.292+0000 7efde53c3640 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55efb8d6d1f0 0x55efb91668c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f4b8b2a943926 server_cookie=a30f4362f1e7966b in_seq=0 out_seq=0 2024-04-25T16:08:29.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.292+0000 7efdc4ff9640 10 client.26735 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:29.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.295+0000 7efdc4ff9640 1 -- 172.21.15.31:0/655219249 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55efb90c2710 con 0x55efb8d6d1f0 2024-04-25T16:08:29.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.295+0000 7efdc4ff9640 10 client.26735 handle_command_reply: tid=0 2024-04-25T16:08:29.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.295+0000 7efdee5b8740 1 client.26735 shutdown 2024-04-25T16:08:29.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.297+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 msgr2=0x55efb91599a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:29.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.297+0000 7efdee5b8740 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb91599a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7efdd40a2ca0 tx=0x7efdd40a62e0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.297+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 shutdown_connections 2024-04-25T16:08:29.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.297+0000 7efdee5b8740 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55efb8d6d1f0 0x55efb91668c0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.297+0000 7efdee5b8740 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8dae410 0x55efb9160900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.297+0000 7efdee5b8740 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb91599a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.297+0000 7efdee5b8740 1 --2- 172.21.15.31:0/655219249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d85550 0x55efb9159460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.297+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 >> 172.21.15.31:0/655219249 conn(0x55efb8da2560 msgr2=0x55efb9149f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:29.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.297+0000 7efdc4ff9640 0 client.26735 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:29.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.298+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 shutdown_connections 2024-04-25T16:08:29.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.298+0000 7efdee5b8740 1 -- 172.21.15.31:0/655219249 wait complete. 2024-04-25T16:08:29.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.299+0000 7efdee5b8740 20 client.26735 trim_cache size 0 max 16384 2024-04-25T16:08:29.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.363+0000 7efdee5b8740 1 Processor -- start 2024-04-25T16:08:29.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.366+0000 7efdee5b8740 1 -- start start 2024-04-25T16:08:29.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.366+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb9158e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.367+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d8d730 0x55efb9160820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.367+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb9160d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.367+0000 7efdee5b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc00e530 con 0x55efb8d85550 2024-04-25T16:08:29.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.367+0000 7efdee5b8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc010070 con 0x55efb8dae410 2024-04-25T16:08:29.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.367+0000 7efdee5b8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc00ec00 con 0x55efb8d8d730 2024-04-25T16:08:29.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.367+0000 7efde5bc4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb9160d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.367+0000 7efde5bc4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb9160d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57244/0 (socket says 172.21.15.31:57244) 2024-04-25T16:08:29.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.367+0000 7efde5bc4640 1 -- 172.21.15.31:0/462569631 learned_addr learned my addr 172.21.15.31:0/462569631 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:29.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.367+0000 7efde4bc2640 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d8d730 0x55efb9160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.368+0000 7efde53c3640 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb9158e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.376 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.372+0000 7efde5bc4640 1 -- 172.21.15.31:0/462569631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d8d730 msgr2=0x55efb9160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:29.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.372+0000 7efde5bc4640 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d8d730 0x55efb9160820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.372+0000 7efde5bc4640 1 -- 172.21.15.31:0/462569631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 msgr2=0x55efb9158e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:29.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.372+0000 7efde5bc4640 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb9158e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.372+0000 7efde5bc4640 1 -- 172.21.15.31:0/462569631 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efb9154240 con 0x55efb8dae410 2024-04-25T16:08:29.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.373+0000 7efde5bc4640 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb9160d60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009900 tx=0x7efdd0043040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12b9c59c79d28999 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:29.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.374+0000 7efdc6ffd640 1 -- 172.21.15.31:0/462569631 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd0005500 con 0x55efb8dae410 2024-04-25T16:08:29.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.374+0000 7efdc6ffd640 1 -- 172.21.15.31:0/462569631 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdd0050070 con 0x55efb8dae410 2024-04-25T16:08:29.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.374+0000 7efdc6ffd640 1 -- 172.21.15.31:0/462569631 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd00416c0 con 0x55efb8dae410 2024-04-25T16:08:29.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.375+0000 7efde53c3640 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb9158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:08:29.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.377+0000 7efdee5b8740 1 -- 172.21.15.31:0/462569631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 msgr2=0x55efb9160d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:29.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.377+0000 7efdee5b8740 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb9160d60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7efdd0009900 tx=0x7efdd0043040 comp rx=0 tx=0).stop 2024-04-25T16:08:29.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.378+0000 7efdee5b8740 1 -- 172.21.15.31:0/462569631 shutdown_connections 2024-04-25T16:08:29.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.378+0000 7efdee5b8740 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8dae410 0x55efb9160d60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.378+0000 7efdee5b8740 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8d8d730 0x55efb9160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.378+0000 7efdee5b8740 1 --2- 172.21.15.31:0/462569631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb9158e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.378+0000 7efdee5b8740 1 -- 172.21.15.31:0/462569631 >> 172.21.15.31:0/462569631 conn(0x55efb8da2560 msgr2=0x55efb91586f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:29.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.381+0000 7efdee5b8740 1 -- 172.21.15.31:0/462569631 shutdown_connections 2024-04-25T16:08:29.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.381+0000 7efdee5b8740 1 -- 172.21.15.31:0/462569631 wait complete. 2024-04-25T16:08:29.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.382+0000 7efdee5b8740 1 -- start start 2024-04-25T16:08:29.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.383+0000 7efdee5b8740 1 Processor -- start 2024-04-25T16:08:29.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.384+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb915e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.384+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb915ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.384+0000 7efdee5b8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8dae410 0x55efb915fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.384+0000 7efdee5b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc00f9a0 con 0x55efb8d85550 2024-04-25T16:08:29.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.384+0000 7efdee5b8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc00f2d0 con 0x55efb8d8d730 2024-04-25T16:08:29.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.384+0000 7efdee5b8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc00c510 con 0x55efb8dae410 2024-04-25T16:08:29.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.384+0000 7efdee5b8740 10 client.0 _ll_register_callbacks cb 0x7efdd80096d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:29.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.385+0000 7efde53c3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb915e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.385+0000 7efde53c3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb915e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47502/0 (socket says 172.21.15.31:47502) 2024-04-25T16:08:29.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.385+0000 7efde53c3640 1 -- 172.21.15.31:0/2821336761 learned_addr learned my addr 172.21.15.31:0/2821336761 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.386+0000 7efde4bc2640 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb915ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.386+0000 7efde5bc4640 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8dae410 0x55efb915fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.386+0000 7efde4bc2640 1 -- 172.21.15.31:0/2821336761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8dae410 msgr2=0x55efb915fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.386+0000 7efde4bc2640 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8dae410 0x55efb915fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.386+0000 7efde4bc2640 1 -- 172.21.15.31:0/2821336761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 msgr2=0x55efb915e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.386+0000 7efde4bc2640 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb915e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.386+0000 7efde4bc2640 1 -- 172.21.15.31:0/2821336761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efb9160040 con 0x55efb8d8d730 2024-04-25T16:08:29.390 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.387+0000 7efde4bc2640 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb915ebf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efdd4012040 tx=0x7efdd40a2470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8259d640a69d4165 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:29.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.391+0000 7efdc4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:08:29.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.391+0000 7efdc4ff9640 1 -- 172.21.15.31:0/2821336761 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdd40afad0 con 0x55efb8d8d730 2024-04-25T16:08:29.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.391+0000 7efdc4ff9640 1 -- 172.21.15.31:0/2821336761 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7efdd400fcc0 con 0x55efb8d8d730 2024-04-25T16:08:29.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.391+0000 7efdee5b8740 1 -- 172.21.15.31:0/2821336761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efdd0050070 con 0x55efb8d8d730 2024-04-25T16:08:29.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.394+0000 7efdc4ff9640 1 -- 172.21.15.31:0/2821336761 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7efdd40a5ca0 con 0x55efb8d8d730 2024-04-25T16:08:29.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.394+0000 7efdee5b8740 10 client.26741 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:29.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.394+0000 7efdee5b8740 1 -- 172.21.15.31:0/2821336761 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55efb9161300 con 0x55efb8d8d730 2024-04-25T16:08:29.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.396+0000 7efdc4ff9640 1 -- 172.21.15.31:0/2821336761 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdd0050070 con 0x55efb8d8d730 2024-04-25T16:08:29.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.396+0000 7efdc4ff9640 1 -- 172.21.15.31:0/2821336761 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7efdd40de190 con 0x55efb8d8d730 2024-04-25T16:08:29.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.396+0000 7efdee5b8740 10 client.26741 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:29.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.396+0000 7efdee5b8740 10 client.26741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:29.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.396+0000 7efdee5b8740 20 client.26741 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:29.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.397+0000 7efdee5b8740 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55efb8d42630 0x55efb91d0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:29.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.397+0000 7efdee5b8740 4 client.26741 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:08:29.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.397+0000 7efdee5b8740 1 -- 172.21.15.31:0/2821336761 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55efb90bc580 con 0x55efb8d42630 2024-04-25T16:08:29.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.397+0000 7efde53c3640 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55efb8d42630 0x55efb91d0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:29.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.404+0000 7efde53c3640 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55efb8d42630 0x55efb91d0ff0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b9e2cbd4ca00063 server_cookie=12b3ee72fb081b14 in_seq=0 out_seq=0 2024-04-25T16:08:29.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.411+0000 7efdc4ff9640 10 client.26741 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:29.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdc4ff9640 1 -- 172.21.15.31:0/2821336761 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55efb90bc580 con 0x55efb8d42630 2024-04-25T16:08:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdc4ff9640 10 client.26741 handle_command_reply: tid=0 2024-04-25T16:08:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdee5b8740 1 client.26741 shutdown 2024-04-25T16:08:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdee5b8740 1 -- 172.21.15.31:0/2821336761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 msgr2=0x55efb915ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:29.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdee5b8740 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb915ebf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efdd4012040 tx=0x7efdd40a2470 comp rx=0 tx=0).stop 2024-04-25T16:08:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdee5b8740 1 -- 172.21.15.31:0/2821336761 shutdown_connections 2024-04-25T16:08:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdee5b8740 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55efb8d42630 0x55efb91d0ff0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdee5b8740 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55efb8dae410 0x55efb915fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdee5b8740 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55efb8d8d730 0x55efb915ebf0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdee5b8740 1 --2- 172.21.15.31:0/2821336761 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55efb8d85550 0x55efb915e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdee5b8740 1 -- 172.21.15.31:0/2821336761 >> 172.21.15.31:0/2821336761 conn(0x55efb8da2560 msgr2=0x55efb9149560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:29.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.594+0000 7efdc4ff9640 0 client.26741 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.595+0000 7efdee5b8740 1 -- 172.21.15.31:0/2821336761 shutdown_connections 2024-04-25T16:08:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.595+0000 7efdee5b8740 1 -- 172.21.15.31:0/2821336761 wait complete. 2024-04-25T16:08:29.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.596+0000 7efdee5b8740 20 client.26741 trim_cache size 0 max 16384 2024-04-25T16:08:29.599 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:08:29.599 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:08:29.599 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "8b37eebc-a9a5-48e4-9e55-a524ebe55cef", 2024-04-25T16:08:29.599 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:08:29.599 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:08:29.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.598+0000 7efde7627640 1 -- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdc80afd00 msgr2=0x7efdc80b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:29.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.598+0000 7efde7627640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdc80afd00 0x7efdc80b2210 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7efdd00096c0 tx=0x7efdd0009630 comp rx=0 tx=0).stop 2024-04-25T16:08:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.598+0000 7efde7627640 1 -- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efde010b1d0 msgr2=0x7efde01175d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.598+0000 7efde7627640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efde010b1d0 0x7efde01175d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7efdd400d950 tx=0x7efdd400de10 comp rx=0 tx=0).stop 2024-04-25T16:08:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.599+0000 7efde7627640 1 -- 172.21.15.31:0/1838084626 shutdown_connections 2024-04-25T16:08:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.599+0000 7efde7627640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7efdc80afd00 0x7efdc80b2210 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.599+0000 7efde7627640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efde010fef0 0x7efde01184e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.599+0000 7efde7627640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7efde010b1d0 0x7efde01175d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.599+0000 7efde7627640 1 --2- 172.21.15.31:0/1838084626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efde010a3e0 0x7efde0117090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:29.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.599+0000 7efde7627640 1 -- 172.21.15.31:0/1838084626 >> 172.21.15.31:0/1838084626 conn(0x7efde0070400 msgr2=0x7efde010bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.602+0000 7efde7627640 1 -- 172.21.15.31:0/1838084626 shutdown_connections 2024-04-25T16:08:29.605 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:29.604+0000 7efde7627640 1 -- 172.21.15.31:0/1838084626 wait complete. 2024-04-25T16:08:29.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:08:29.653 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:08:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.159+0000 7f3c64c30640 1 -- 172.21.15.31:0/1580006060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c60110030 msgr2=0x7f3c60110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.159+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1580006060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c60110030 0x7f3c60110440 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009860 tx=0x7f3c5402a340 comp rx=0 tx=0).stop 2024-04-25T16:08:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.159+0000 7f3c64c30640 1 -- 172.21.15.31:0/1580006060 shutdown_connections 2024-04-25T16:08:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.159+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1580006060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c60110030 0x7f3c60110440 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.159+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1580006060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c6010b310 0x7f3c6010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.159+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1580006060 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6010a420 0x7f3c6010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.159+0000 7f3c64c30640 1 -- 172.21.15.31:0/1580006060 >> 172.21.15.31:0/1580006060 conn(0x7f3c600704e0 msgr2=0x7f3c60072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:30.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.159+0000 7f3c64c30640 1 -- 172.21.15.31:0/1580006060 shutdown_connections 2024-04-25T16:08:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.162+0000 7f3c64c30640 1 -- 172.21.15.31:0/1580006060 wait complete. 2024-04-25T16:08:30.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.163+0000 7f3c64c30640 1 Processor -- start 2024-04-25T16:08:30.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.165+0000 7f3c64c30640 1 -- start start 2024-04-25T16:08:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.166+0000 7f3c64c30640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6010a420 0x7f3c601a90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.166+0000 7f3c64c30640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6010b310 0x7f3c601a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.166+0000 7f3c64c30640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c60110030 0x7f3c601b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.166+0000 7f3c64c30640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6011e490 con 0x7f3c6010b310 2024-04-25T16:08:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.166+0000 7f3c64c30640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6011e310 con 0x7f3c60110030 2024-04-25T16:08:30.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.166+0000 7f3c64c30640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c6011e610 con 0x7f3c6010a420 2024-04-25T16:08:30.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.168+0000 7f3c5ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c60110030 0x7f3c601b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.168+0000 7f3c5ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c60110030 0x7f3c601b0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57268/0 (socket says 172.21.15.31:57268) 2024-04-25T16:08:30.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.168+0000 7f3c5ed9d640 1 -- 172.21.15.31:0/1604530530 learned_addr learned my addr 172.21.15.31:0/1604530530 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:30.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.170+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6010b310 0x7f3c601a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.170+0000 7f3c5e59c640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6010a420 0x7f3c601a90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.170+0000 7f3c5ed9d640 1 -- 172.21.15.31:0/1604530530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6010a420 msgr2=0x7f3c601a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.170+0000 7f3c5ed9d640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6010a420 0x7f3c601a90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.170+0000 7f3c5ed9d640 1 -- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6010b310 msgr2=0x7f3c601a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.170+0000 7f3c5ed9d640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6010b310 0x7f3c601a95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.170+0000 7f3c5ed9d640 1 -- 172.21.15.31:0/1604530530 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c60111c90 con 0x7f3c60110030 2024-04-25T16:08:30.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.173+0000 7f3c5ed9d640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c60110030 0x7f3c601b0520 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3c540303f0 tx=0x7f3c54004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:30.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.175+0000 7f3c4f7fe640 1 -- 172.21.15.31:0/1604530530 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54037040 con 0x7f3c60110030 2024-04-25T16:08:30.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.175+0000 7f3c64c30640 1 -- 172.21.15.31:0/1604530530 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c60111f20 con 0x7f3c60110030 2024-04-25T16:08:30.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.175+0000 7f3c64c30640 1 -- 172.21.15.31:0/1604530530 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c601123b0 con 0x7f3c60110030 2024-04-25T16:08:30.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.176+0000 7f3c4f7fe640 1 -- 172.21.15.31:0/1604530530 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c54041070 con 0x7f3c60110030 2024-04-25T16:08:30.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.176+0000 7f3c4f7fe640 1 -- 172.21.15.31:0/1604530530 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c54004d30 con 0x7f3c60110030 2024-04-25T16:08:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.183+0000 7f3c6bbb6740 1 Processor -- start 2024-04-25T16:08:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.183+0000 7f3c6bbb6740 1 -- start start 2024-04-25T16:08:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.183+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d47324a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.184+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f5a4d0 0x564d47324fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.184+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 0x564d47325c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.184+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564d47289d80 con 0x564d46f5a4d0 2024-04-25T16:08:30.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.184+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564d472438b0 con 0x564d46f83390 2024-04-25T16:08:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.184+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d47285d40 con 0x564d46f626b0 2024-04-25T16:08:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.184+0000 7f3c5e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f5a4d0 0x564d47324fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.184+0000 7f3c5e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f5a4d0 0x564d47324fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47520/0 (socket says 172.21.15.31:47520) 2024-04-25T16:08:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.184+0000 7f3c5e59c640 1 -- 172.21.15.31:0/1163030051 learned_addr learned my addr 172.21.15.31:0/1163030051 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.185+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/1163030051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d47324a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.185+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/1163030051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 0x564d47325c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.186+0000 7f3c5dd9b640 1 -- 172.21.15.31:0/1163030051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 msgr2=0x564d47324a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:30.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.186+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/1163030051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d47324a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.186+0000 7f3c5dd9b640 1 -- 172.21.15.31:0/1163030051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f5a4d0 msgr2=0x564d47324fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:30.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.186+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/1163030051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f5a4d0 0x564d47324fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.186+0000 7f3c5dd9b640 1 -- 172.21.15.31:0/1163030051 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d47214a40 con 0x564d46f626b0 2024-04-25T16:08:30.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.186+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/1163030051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 0x564d47325c70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c601aa0f0 tx=0x7f3c580095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfa6d3c164989fec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:30.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.187+0000 7f3c4cff9640 1 -- 172.21.15.31:0/1163030051 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c58015050 con 0x564d46f626b0 2024-04-25T16:08:30.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.187+0000 7f3c4cff9640 1 -- 172.21.15.31:0/1163030051 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c5800de40 con 0x564d46f626b0 2024-04-25T16:08:30.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.187+0000 7f3c4cff9640 1 -- 172.21.15.31:0/1163030051 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c58018040 con 0x564d46f626b0 2024-04-25T16:08:30.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/1163030051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 msgr2=0x564d47325c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:30.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/1163030051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 0x564d47325c70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3c601aa0f0 tx=0x7f3c580095c0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/1163030051 shutdown_connections 2024-04-25T16:08:30.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/1163030051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 0x564d47325c70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/1163030051 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f5a4d0 0x564d47324fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/1163030051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d47324a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/1163030051 >> 172.21.15.31:0/1163030051 conn(0x564d46f774e0 msgr2=0x564d47320270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:30.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/1163030051 shutdown_connections 2024-04-25T16:08:30.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/1163030051 wait complete. 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.190+0000 7f3c6bbb6740 1 -- start start 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c6bbb6740 1 Processor -- start 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 0x564d4732e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d4732eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d47335e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564d47326780 con 0x564d46f626b0 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564d47210d10 con 0x564d46f83390 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d47326900 con 0x564d46f5a4d0 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c6bbb6740 10 client.0 _ll_register_callbacks cb 0x564d4723d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c4f7fe640 1 -- 172.21.15.31:0/1604530530 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3c54005050 con 0x7f3c60110030 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c5e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d4732eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c5e59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d4732eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47532/0 (socket says 172.21.15.31:47532) 2024-04-25T16:08:30.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.191+0000 7f3c5e59c640 1 -- 172.21.15.31:0/159517438 learned_addr learned my addr 172.21.15.31:0/159517438 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:30.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.192+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 0x564d4732e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.192+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d47335e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.192+0000 7f3c5e59c640 1 -- 172.21.15.31:0/159517438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 msgr2=0x564d4732e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.192+0000 7f3c5e59c640 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 0x564d4732e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.192+0000 7f3c5e59c640 1 -- 172.21.15.31:0/159517438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 msgr2=0x564d47335e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.192+0000 7f3c5e59c640 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d47335e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.192+0000 7f3c5e59c640 1 -- 172.21.15.31:0/159517438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d470657d0 con 0x564d46f626b0 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.193+0000 7f3c5e59c640 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d4732eeb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009db0 tx=0x7f3c500117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5029178c32eecb4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.193+0000 7f3c45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.193+0000 7f3c45ffb640 1 -- 172.21.15.31:0/159517438 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c50012860 con 0x564d46f626b0 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.193+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/159517438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564d47297990 con 0x564d46f626b0 2024-04-25T16:08:30.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.193+0000 7f3c45ffb640 1 -- 172.21.15.31:0/159517438 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3c50012a00 con 0x564d46f626b0 2024-04-25T16:08:30.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.194+0000 7f3c45ffb640 1 -- 172.21.15.31:0/159517438 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c50012c00 con 0x564d46f626b0 2024-04-25T16:08:30.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.194+0000 7f3c6bbb6740 10 client.17097 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:30.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.194+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/159517438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564d46f9b3f0 con 0x564d46f626b0 2024-04-25T16:08:30.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.196+0000 7f3c4f7fe640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c480afc10 0x7f3c480b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.197+0000 7f3c45ffb640 1 -- 172.21.15.31:0/159517438 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x564d47297990 con 0x564d46f626b0 2024-04-25T16:08:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.197+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c480afc10 0x7f3c480b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.198+0000 7f3c45ffb640 1 -- 172.21.15.31:0/159517438 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3c5000f040 con 0x564d46f626b0 2024-04-25T16:08:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.198+0000 7f3c6bbb6740 10 client.17097 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.198+0000 7f3c6bbb6740 10 client.17097 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.198+0000 7f3c6bbb6740 20 client.17097 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.198+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564d46f41f60 0x564d4733c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.198+0000 7f3c6bbb6740 4 client.17097 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:30.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.198+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/159517438 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d47291800 con 0x564d46f41f60 2024-04-25T16:08:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.198+0000 7f3c4f7fe640 1 -- 172.21.15.31:0/1604530530 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3c540fe890 con 0x7f3c60110030 2024-04-25T16:08:30.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.199+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564d46f41f60 0x564d4733c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:29 smithi031 ceph-mon[29140]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 809 MiB/s wr, 2.01k op/s 2024-04-25T16:08:30.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.205+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c480afc10 0x7f3c480b2120 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800fa60 tx=0x7f3c5801ff30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:30.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.205+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564d46f41f60 0x564d4733c490 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4cb19211093dcc4 server_cookie=d16d0337d22bfd7f in_seq=0 out_seq=0 2024-04-25T16:08:30.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.206+0000 7f3c45ffb640 10 client.17097 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:30.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.208+0000 7f3c45ffb640 1 -- 172.21.15.31:0/159517438 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564d47291800 con 0x564d46f41f60 2024-04-25T16:08:30.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.208+0000 7f3c45ffb640 10 client.17097 handle_command_reply: tid=0 2024-04-25T16:08:30.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.208+0000 7f3c6bbb6740 1 client.17097 shutdown 2024-04-25T16:08:30.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.209+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/159517438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 msgr2=0x564d4732eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:30.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.209+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d4732eeb0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009db0 tx=0x7f3c500117b0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.209+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/159517438 shutdown_connections 2024-04-25T16:08:30.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.209+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564d46f41f60 0x564d4733c490 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.209+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d47335e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.209+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d4732eeb0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.209+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/159517438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 0x564d4732e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.209+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/159517438 >> 172.21.15.31:0/159517438 conn(0x564d46f774e0 msgr2=0x564d4731f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:30.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.209+0000 7f3c45ffb640 0 client.17097 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:30.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.210+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/159517438 shutdown_connections 2024-04-25T16:08:30.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.211+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/159517438 wait complete. 2024-04-25T16:08:30.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.211+0000 7f3c6bbb6740 20 client.17097 trim_cache size 0 max 16384 2024-04-25T16:08:30.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.282+0000 7f3c6bbb6740 1 Processor -- start 2024-04-25T16:08:30.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.290+0000 7f3c6bbb6740 1 -- start start 2024-04-25T16:08:30.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.291+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f5a4d0 0x564d4732d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.291+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 0x564d47335da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.291+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f83390 0x564d473362e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.291+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4000e4f0 con 0x564d46f83390 2024-04-25T16:08:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.292+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3c40010030 con 0x564d46f5a4d0 2024-04-25T16:08:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.292+0000 7f3c5e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f5a4d0 0x564d4732d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.292+0000 7f3c5e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f5a4d0 0x564d4732d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57294/0 (socket says 172.21.15.31:57294) 2024-04-25T16:08:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.292+0000 7f3c5e59c640 1 -- 172.21.15.31:0/4190491069 learned_addr learned my addr 172.21.15.31:0/4190491069 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:30.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.292+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/4190491069 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4000ebc0 con 0x564d46f626b0 2024-04-25T16:08:30.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.293+0000 7f3c5ed9d640 1 --2- 172.21.15.31:0/4190491069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f83390 0x564d473362e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.296+0000 7f3c5ed9d640 1 -- 172.21.15.31:0/4190491069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 msgr2=0x564d47335da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.296+0000 7f3c5ed9d640 1 --2- 172.21.15.31:0/4190491069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 0x564d47335da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.296+0000 7f3c5ed9d640 1 -- 172.21.15.31:0/4190491069 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f5a4d0 msgr2=0x564d4732d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.296+0000 7f3c5ed9d640 1 --2- 172.21.15.31:0/4190491069 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f5a4d0 0x564d4732d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.296+0000 7f3c5ed9d640 1 -- 172.21.15.31:0/4190491069 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d46f5b0f0 con 0x564d46f83390 2024-04-25T16:08:30.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.297+0000 7f3c5ed9d640 1 --2- 172.21.15.31:0/4190491069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f83390 0x564d473362e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3c540c6aa0 tx=0x7f3c5402a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=640f048e1bada9ed server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.310+0000 7f3c47fff640 1 -- 172.21.15.31:0/4190491069 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c540ca070 con 0x564d46f83390 2024-04-25T16:08:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.311+0000 7f3c47fff640 1 -- 172.21.15.31:0/4190491069 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c540e1480 con 0x564d46f83390 2024-04-25T16:08:30.313 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.311+0000 7f3c47fff640 1 -- 172.21.15.31:0/4190491069 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c540cfc00 con 0x564d46f83390 2024-04-25T16:08:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.313+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/4190491069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f83390 msgr2=0x564d473362e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:30.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.313+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/4190491069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f83390 0x564d473362e0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f3c540c6aa0 tx=0x7f3c5402a340 comp rx=0 tx=0).stop 2024-04-25T16:08:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.314+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/4190491069 shutdown_connections 2024-04-25T16:08:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.314+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/4190491069 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f83390 0x564d473362e0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.314+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/4190491069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f626b0 0x564d47335da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.314+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/4190491069 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f5a4d0 0x564d4732d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.314+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/4190491069 >> 172.21.15.31:0/4190491069 conn(0x564d46f774e0 msgr2=0x564d4732cf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:30.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.330+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/4190491069 shutdown_connections 2024-04-25T16:08:30.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.330+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/4190491069 wait complete. 2024-04-25T16:08:30.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.331+0000 7f3c6bbb6740 1 -- start start 2024-04-25T16:08:30.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.332+0000 7f3c6bbb6740 1 Processor -- start 2024-04-25T16:08:30.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.333+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 0x564d473331e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.333+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d47333720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.333+0000 7f3c6bbb6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d473a2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.333+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4000f960 con 0x564d46f626b0 2024-04-25T16:08:30.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.333+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4000f290 con 0x564d46f83390 2024-04-25T16:08:30.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.333+0000 7f3c6bbb6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4000c4d0 con 0x564d46f5a4d0 2024-04-25T16:08:30.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.334+0000 7f3c6bbb6740 10 client.0 _ll_register_callbacks cb 0x7f3c50005c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:30.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.337+0000 7f3c5dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d47333720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.337+0000 7f3c5dd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d47333720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47550/0 (socket says 172.21.15.31:47550) 2024-04-25T16:08:30.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.337+0000 7f3c5dd9b640 1 -- 172.21.15.31:0/2844842306 learned_addr learned my addr 172.21.15.31:0/2844842306 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:30.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.338+0000 7f3c5ed9d640 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d473a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.338+0000 7f3c5e59c640 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 0x564d473331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.343+0000 7f3c5e59c640 1 -- 172.21.15.31:0/2844842306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 msgr2=0x564d473a2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:30.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.343+0000 7f3c5e59c640 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d473a2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.343+0000 7f3c5e59c640 1 -- 172.21.15.31:0/2844842306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 msgr2=0x564d47333720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:30.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.343+0000 7f3c5e59c640 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d47333720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.343+0000 7f3c5e59c640 1 -- 172.21.15.31:0/2844842306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d4724a4b0 con 0x564d46f5a4d0 2024-04-25T16:08:30.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.351+0000 7f3c5e59c640 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 0x564d473331e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009d80 tx=0x7f3c50005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7532e9bcee8ba74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.351+0000 7f3c45ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:08:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.353+0000 7f3c45ffb640 1 -- 172.21.15.31:0/2844842306 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c50031030 con 0x564d46f5a4d0 2024-04-25T16:08:30.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.354+0000 7f3c45ffb640 1 -- 172.21.15.31:0/2844842306 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3c50009900 con 0x564d46f5a4d0 2024-04-25T16:08:30.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.354+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/2844842306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c540e1480 con 0x564d46f5a4d0 2024-04-25T16:08:30.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.364+0000 7f3c45ffb640 1 -- 172.21.15.31:0/2844842306 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3c50029040 con 0x564d46f5a4d0 2024-04-25T16:08:30.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.364+0000 7f3c6bbb6740 10 client.26710 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:30.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.364+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/2844842306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564d473a3a60 con 0x564d46f5a4d0 2024-04-25T16:08:30.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:29 smithi063 ceph-mon[32413]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 809 MiB/s wr, 2.01k op/s 2024-04-25T16:08:30.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.376+0000 7f3c45ffb640 1 -- 172.21.15.31:0/2844842306 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3c500054f0 con 0x564d46f5a4d0 2024-04-25T16:08:30.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.378+0000 7f3c45ffb640 1 -- 172.21.15.31:0/2844842306 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3c5002c030 con 0x564d46f5a4d0 2024-04-25T16:08:30.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.378+0000 7f3c6bbb6740 10 client.26710 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:30.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.378+0000 7f3c6bbb6740 10 client.26710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:30.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.379+0000 7f3c6bbb6740 20 client.26710 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:30.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.379+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564d46f173a0 0x564d473a6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:30.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.379+0000 7f3c6bbb6740 4 client.26710 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:08:30.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.379+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/2844842306 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f3c540e1480 con 0x564d46f173a0 2024-04-25T16:08:30.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.390+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564d46f173a0 0x564d473a6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:30.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:29 smithi186 ceph-mon[33258]: pgmap v567: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 809 MiB/s wr, 2.01k op/s 2024-04-25T16:08:30.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.411+0000 7f3c5dd9b640 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564d46f173a0 0x564d473a6420 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b908679349713620 server_cookie=b3d1c591d2bf97d8 in_seq=0 out_seq=0 2024-04-25T16:08:30.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.413+0000 7f3c45ffb640 10 client.26710 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:30.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.420+0000 7f3c45ffb640 1 -- 172.21.15.31:0/2844842306 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x7f3c540e1480 con 0x564d46f173a0 2024-04-25T16:08:30.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.420+0000 7f3c45ffb640 10 client.26710 handle_command_reply: tid=0 2024-04-25T16:08:30.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.420+0000 7f3c6bbb6740 1 client.26710 shutdown 2024-04-25T16:08:30.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.421+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/2844842306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 msgr2=0x564d473331e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:30.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.421+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 0x564d473331e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3c50009d80 tx=0x7f3c50005850 comp rx=0 tx=0).stop 2024-04-25T16:08:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.422+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/2844842306 shutdown_connections 2024-04-25T16:08:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.422+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564d46f173a0 0x564d473a6420 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.422+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564d46f83390 0x564d473a2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.422+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564d46f626b0 0x564d47333720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.422+0000 7f3c6bbb6740 1 --2- 172.21.15.31:0/2844842306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d46f5a4d0 0x564d473331e0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.422+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/2844842306 >> 172.21.15.31:0/2844842306 conn(0x564d46f774e0 msgr2=0x564d4732dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:30.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.436+0000 7f3c45ffb640 0 client.26710 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:30.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.436+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/2844842306 shutdown_connections 2024-04-25T16:08:30.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.436+0000 7f3c6bbb6740 1 -- 172.21.15.31:0/2844842306 wait complete. 2024-04-25T16:08:30.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.436+0000 7f3c6bbb6740 20 client.26710 trim_cache size 0 max 16384 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (5 inodes in the stack)", 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout: "8b37eebc-a9a5-48e4-9e55-a524ebe55cef": { 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "8b37eebc-a9a5-48e4-9e55-a524ebe55cef", 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:08:30.441 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:08:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 -- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c480afc10 msgr2=0x7f3c480b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c480afc10 0x7f3c480b2120 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f3c5800fa60 tx=0x7f3c5801ff30 comp rx=0 tx=0).stop 2024-04-25T16:08:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 -- 172.21.15.31:0/1604530530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c60110030 msgr2=0x7f3c601b0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:30.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c60110030 0x7f3c601b0520 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3c540303f0 tx=0x7f3c54004270 comp rx=0 tx=0).stop 2024-04-25T16:08:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 -- 172.21.15.31:0/1604530530 shutdown_connections 2024-04-25T16:08:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3c480afc10 0x7f3c480b2120 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3c60110030 0x7f3c601b0520 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3c6010b310 0x7f3c601a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 --2- 172.21.15.31:0/1604530530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c6010a420 0x7f3c601a90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:30.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.445+0000 7f3c64c30640 1 -- 172.21.15.31:0/1604530530 >> 172.21.15.31:0/1604530530 conn(0x7f3c600704e0 msgr2=0x7f3c6010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:30.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.448+0000 7f3c64c30640 1 -- 172.21.15.31:0/1604530530 shutdown_connections 2024-04-25T16:08:30.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:08:30.449+0000 7f3c64c30640 1 -- 172.21.15.31:0/1604530530 wait complete. 2024-04-25T16:08:30.516 INFO:tasks.cephfs.filesystem:scrub status for tag:8b37eebc-a9a5-48e4-9e55-a524ebe55cef - {'path': '/', 'tag': '8b37eebc-a9a5-48e4-9e55-a524ebe55cef', 'options': 'recursive,force'} 2024-04-25T16:08:31.204 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:30 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:08:31.204 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:30 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:08:31.204 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:30 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:08:31.204 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:30 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:08:31.204 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:30 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:08:31.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:30 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:08:31.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:30 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:08:31.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:30 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:08:31.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:30 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:08:31.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:30 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:08:31.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:30 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:08:31.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:30 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:08:31.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:30 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:08:31.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:30 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:08:31.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:30 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:08:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.590+0000 7f3ffe19f640 1 -- 172.21.15.31:0/3471741544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ff00a6a40 msgr2=0x7f3ff00a6e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:31.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.590+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/3471741544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ff00a6a40 0x7f3ff00a6e50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8009860 tx=0x7f3fe802a360 comp rx=0 tx=0).stop 2024-04-25T16:08:31.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.595+0000 7f3ffe19f640 1 -- 172.21.15.31:0/3471741544 shutdown_connections 2024-04-25T16:08:31.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.595+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/3471741544 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ff00ab760 0x7f3ff00abb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.595+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/3471741544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ff00a6a40 0x7f3ff00a6e50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.595+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/3471741544 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ff00a5940 0x7f3ff00a6500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.595+0000 7f3ffe19f640 1 -- 172.21.15.31:0/3471741544 >> 172.21.15.31:0/3471741544 conn(0x7f3ff00a09b0 msgr2=0x7f3ff00a2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:31.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.631+0000 7f3ffe19f640 1 -- 172.21.15.31:0/3471741544 shutdown_connections 2024-04-25T16:08:31.645 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.642+0000 7f3ffe19f640 1 -- 172.21.15.31:0/3471741544 wait complete. 2024-04-25T16:08:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.644+0000 7f3ffe19f640 1 Processor -- start 2024-04-25T16:08:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffe19f640 1 -- start start 2024-04-25T16:08:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffe19f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ff00a5940 0x7f3ff00b25e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.646 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffe19f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ff00a6a40 0x7f3ff00b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffe19f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ff00ab760 0x7f3ff00b3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffe19f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff00b9b30 con 0x7f3ff00a5940 2024-04-25T16:08:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffe19f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff00b99b0 con 0x7f3ff00ab760 2024-04-25T16:08:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffe19f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff00b9cb0 con 0x7f3ff00a6a40 2024-04-25T16:08:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffd19d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ff00a5940 0x7f3ff00b25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffd19d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ff00a5940 0x7f3ff00b25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47574/0 (socket says 172.21.15.31:47574) 2024-04-25T16:08:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.645+0000 7f3ffd19d640 1 -- 172.21.15.31:0/1428476400 learned_addr learned my addr 172.21.15.31:0/1428476400 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.646+0000 7f3ffc99c640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ff00a6a40 0x7f3ff00b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.646+0000 7f3ffd19d640 1 -- 172.21.15.31:0/1428476400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ff00a6a40 msgr2=0x7f3ff00b2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.646+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ff00a6a40 0x7f3ff00b2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.646+0000 7f3ffd19d640 1 -- 172.21.15.31:0/1428476400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ff00ab760 msgr2=0x7f3ff00b3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.646+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ff00ab760 0x7f3ff00b3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.648 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.646+0000 7f3ffd19d640 1 -- 172.21.15.31:0/1428476400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ff00b40f0 con 0x7f3ff00a5940 2024-04-25T16:08:31.649 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.648+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ff00a5940 0x7f3ff00b25e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8068520 tx=0x7f3ff8073a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:31.672 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.669+0000 7f3fee7fc640 1 -- 172.21.15.31:0/1428476400 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff807a070 con 0x7f3ff00a5940 2024-04-25T16:08:31.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.669+0000 7f3fee7fc640 1 -- 172.21.15.31:0/1428476400 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff8069b90 con 0x7f3ff00a5940 2024-04-25T16:08:31.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.669+0000 7f3fee7fc640 1 -- 172.21.15.31:0/1428476400 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8076b10 con 0x7f3ff00a5940 2024-04-25T16:08:31.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.669+0000 7f3ffe19f640 1 -- 172.21.15.31:0/1428476400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ff00b43e0 con 0x7f3ff00a5940 2024-04-25T16:08:31.673 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.669+0000 7f3ffe19f640 1 -- 172.21.15.31:0/1428476400 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ff00b48b0 con 0x7f3ff00a5940 2024-04-25T16:08:31.678 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.677+0000 7f400638a740 1 Processor -- start 2024-04-25T16:08:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.678+0000 7f400638a740 1 -- start start 2024-04-25T16:08:31.679 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.678+0000 7f400638a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56034986a1b0 0x560349c0ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.679+0000 7f400638a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603498412f0 0x560349c0b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.679+0000 7f400638a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498494d0 0x560349c11fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.680 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.679+0000 7f400638a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560349b70460 con 0x56034986a1b0 2024-04-25T16:08:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.679+0000 7f400638a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560349b2a590 con 0x5603498412f0 2024-04-25T16:08:31.681 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.679+0000 7f400638a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560349b6c420 con 0x5603498494d0 2024-04-25T16:08:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.681+0000 7f3ffc99c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498494d0 0x560349c11fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.681+0000 7f3ffd19d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603498412f0 0x560349c0b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.682 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.681+0000 7f3ffd99e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56034986a1b0 0x560349c0ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.681+0000 7f3ffd99e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56034986a1b0 0x560349c0ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47582/0 (socket says 172.21.15.31:47582) 2024-04-25T16:08:31.683 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.681+0000 7f3ffd99e640 1 -- 172.21.15.31:0/2516066889 learned_addr learned my addr 172.21.15.31:0/2516066889 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.683+0000 7f3ffd99e640 1 -- 172.21.15.31:0/2516066889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498494d0 msgr2=0x560349c11fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.683+0000 7f3ffd99e640 1 --2- 172.21.15.31:0/2516066889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498494d0 0x560349c11fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.684 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.683+0000 7f3ffd99e640 1 -- 172.21.15.31:0/2516066889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603498412f0 msgr2=0x560349c0b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:31.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.683+0000 7f3ffd99e640 1 --2- 172.21.15.31:0/2516066889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603498412f0 0x560349c0b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.685 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.683+0000 7f3ffd99e640 1 -- 172.21.15.31:0/2516066889 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560349b370d0 con 0x56034986a1b0 2024-04-25T16:08:31.687 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.685+0000 7f3ffd99e640 1 --2- 172.21.15.31:0/2516066889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56034986a1b0 0x560349c0ab20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400a960 tx=0x7f3ff400ae20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12db2e2d9bfcf6a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.686+0000 7f3fe37fe640 1 -- 172.21.15.31:0/2516066889 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff400cd80 con 0x56034986a1b0 2024-04-25T16:08:31.688 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.687+0000 7f3fe37fe640 1 -- 172.21.15.31:0/2516066889 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff400f900 con 0x56034986a1b0 2024-04-25T16:08:31.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.690+0000 7f400638a740 1 -- 172.21.15.31:0/2516066889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56034986a1b0 msgr2=0x560349c0ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:31.691 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.690+0000 7f400638a740 1 --2- 172.21.15.31:0/2516066889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56034986a1b0 0x560349c0ab20 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400a960 tx=0x7f3ff400ae20 comp rx=0 tx=0).stop 2024-04-25T16:08:31.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.690+0000 7f400638a740 1 -- 172.21.15.31:0/2516066889 shutdown_connections 2024-04-25T16:08:31.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.690+0000 7f400638a740 1 --2- 172.21.15.31:0/2516066889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498494d0 0x560349c11fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.690+0000 7f400638a740 1 --2- 172.21.15.31:0/2516066889 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603498412f0 0x560349c0b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.690+0000 7f400638a740 1 --2- 172.21.15.31:0/2516066889 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56034986a1b0 0x560349c0ab20 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.690+0000 7f400638a740 1 -- 172.21.15.31:0/2516066889 >> 172.21.15.31:0/2516066889 conn(0x56034985e300 msgr2=0x560349c064b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:31.692 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.691+0000 7f400638a740 1 -- 172.21.15.31:0/2516066889 shutdown_connections 2024-04-25T16:08:31.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.691+0000 7f400638a740 1 -- 172.21.15.31:0/2516066889 wait complete. 2024-04-25T16:08:31.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.691+0000 7f400638a740 1 -- start start 2024-04-25T16:08:31.693 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.691+0000 7f400638a740 1 Processor -- start 2024-04-25T16:08:31.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.693+0000 7f400638a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 0x560349c146f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.694 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.693+0000 7f400638a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498494d0 0x560349c14c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.695 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.694+0000 7f400638a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 0x560349c1bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.694+0000 7f3ffd99e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 0x560349c146f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.694+0000 7f3ffd99e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 0x560349c146f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54182/0 (socket says 172.21.15.31:54182) 2024-04-25T16:08:31.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.694+0000 7f3ffd99e640 1 -- 172.21.15.31:0/1140800332 learned_addr learned my addr 172.21.15.31:0/1140800332 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:31.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.694+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560349c12a40 con 0x5603498494d0 2024-04-25T16:08:31.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.694+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560349af7b60 con 0x560349828d80 2024-04-25T16:08:31.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.694+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560349c12bc0 con 0x5603498412f0 2024-04-25T16:08:31.696 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.694+0000 7f3ffc99c640 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 0x560349c1bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.701+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498494d0 0x560349c14c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.701+0000 7f3fee7fc640 1 -- 172.21.15.31:0/1428476400 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3ff8069690 con 0x7f3ff00a5940 2024-04-25T16:08:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.704+0000 7f3fee7fc640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fe40afbb0 0x7f3fe40b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.705+0000 7f3fee7fc640 1 -- 172.21.15.31:0/1428476400 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3ff8146060 con 0x7f3ff00a5940 2024-04-25T16:08:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.705+0000 7f3ffc99c640 1 -- 172.21.15.31:0/1140800332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 msgr2=0x560349c146f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.705+0000 7f3ffc99c640 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 0x560349c146f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.705+0000 7f3ffc99c640 1 -- 172.21.15.31:0/1140800332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498494d0 msgr2=0x560349c14c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.705+0000 7f3ffc99c640 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498494d0 0x560349c14c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.705+0000 7f3ffc99c640 1 -- 172.21.15.31:0/1140800332 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560349882210 con 0x560349828d80 2024-04-25T16:08:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.706+0000 7f3ffc99c640 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 0x560349c1bb90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00b3680 tx=0x7f3fe8030de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2af25ffd712f2721 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.706+0000 7f400638a740 10 client.0 _ll_register_callbacks cb 0x560349b1c2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.706+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560349b7e070 con 0x560349828d80 2024-04-25T16:08:31.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.707+0000 7f3fe0ff9640 10 client.26762 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:08:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.709+0000 7f3ffd99e640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fe40afbb0 0x7f3fe40b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.710+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/1140800332 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe804b050 con 0x560349828d80 2024-04-25T16:08:31.711 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.710+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/1140800332 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3fe8004450 con 0x560349828d80 2024-04-25T16:08:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.710+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/1140800332 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3fe8041070 con 0x560349828d80 2024-04-25T16:08:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.710+0000 7f400638a740 10 client.26762 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:31.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.710+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({fsmap=29}) -- 0x56034994e620 con 0x560349828d80 2024-04-25T16:08:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.712+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/1140800332 <== mon.2 v2:172.21.15.63:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3fe804e020 con 0x560349828d80 2024-04-25T16:08:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.712+0000 7f400638a740 10 client.26762 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.712+0000 7f400638a740 10 client.26762 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.712+0000 7f400638a740 20 client.26762 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.712+0000 7f400638a740 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5603497fe1c0 0x560349c228e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.712+0000 7f400638a740 4 client.26762 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.712+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560349b7e070 con 0x5603497fe1c0 2024-04-25T16:08:31.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.713+0000 7f3ffd99e640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fe40afbb0 0x7f3fe40b20c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400c6a0 tx=0x7f3ff401c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:31.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.715+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5603497fe1c0 0x560349c228e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.716+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5603497fe1c0 0x560349c228e0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2049f575adfcb8e2 server_cookie=e3aaa10b9fbf2b7d in_seq=0 out_seq=0 2024-04-25T16:08:31.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.716+0000 7f3fe0ff9640 10 client.26762 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:31.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.723+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/1140800332 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560349b7e070 con 0x5603497fe1c0 2024-04-25T16:08:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.723+0000 7f3fe0ff9640 10 client.26762 handle_command_reply: tid=0 2024-04-25T16:08:31.725 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.723+0000 7f400638a740 1 client.26762 shutdown 2024-04-25T16:08:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.724+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 msgr2=0x560349c1bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.724+0000 7f400638a740 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 0x560349c1bb90 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3ff00b3680 tx=0x7f3fe8030de0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.724+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 shutdown_connections 2024-04-25T16:08:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.724+0000 7f400638a740 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5603497fe1c0 0x560349c228e0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.724+0000 7f400638a740 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 0x560349c1bb90 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.724+0000 7f400638a740 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498494d0 0x560349c14c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.724+0000 7f400638a740 1 --2- 172.21.15.31:0/1140800332 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 0x560349c146f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.724+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 >> 172.21.15.31:0/1140800332 conn(0x56034985e300 msgr2=0x560349c09430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:31.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.724+0000 7f3fe0ff9640 0 client.26762 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:31.729 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.725+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 shutdown_connections 2024-04-25T16:08:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.732+0000 7f400638a740 1 -- 172.21.15.31:0/1140800332 wait complete. 2024-04-25T16:08:31.735 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.732+0000 7f400638a740 20 client.26762 trim_cache size 0 max 16384 2024-04-25T16:08:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.790+0000 7f400638a740 1 Processor -- start 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.790+0000 7f400638a740 1 -- start start 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.791+0000 7f400638a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560349828d80 0x560349c22a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.791+0000 7f400638a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 0x560349c1bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.792+0000 7f3ffd19d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560349828d80 0x560349c22a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.792+0000 7f3ffd19d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560349828d80 0x560349c22a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47598/0 (socket says 172.21.15.31:47598) 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.792+0000 7f3ffd19d640 1 -- 172.21.15.31:0/1857589795 learned_addr learned my addr 172.21.15.31:0/1857589795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.792+0000 7f400638a740 1 --2- 172.21.15.31:0/1857589795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603498494d0 0x560349c05d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.792+0000 7f400638a740 1 -- 172.21.15.31:0/1857589795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff015e080 con 0x560349828d80 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.792+0000 7f400638a740 1 -- 172.21.15.31:0/1857589795 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff015fbc0 con 0x5603498494d0 2024-04-25T16:08:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.792+0000 7f400638a740 1 -- 172.21.15.31:0/1857589795 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff015e750 con 0x5603498412f0 2024-04-25T16:08:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.795+0000 7f3ffd19d640 1 -- 172.21.15.31:0/1857589795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 msgr2=0x560349c1bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.795+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/1857589795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 0x560349c1bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.795+0000 7f3ffd19d640 1 -- 172.21.15.31:0/1857589795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603498494d0 msgr2=0x560349c05d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:08:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.795+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/1857589795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603498494d0 0x560349c05d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.795+0000 7f3ffd19d640 1 -- 172.21.15.31:0/1857589795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603497fe3f0 con 0x560349828d80 2024-04-25T16:08:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.799+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/1857589795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560349828d80 0x560349c22a20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80520f0 tx=0x7f3ff8068e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32a795ec662a131f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.805+0000 7f3fe2ffd640 1 -- 172.21.15.31:0/1857589795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8076460 con 0x560349828d80 2024-04-25T16:08:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.806+0000 7f3fe2ffd640 1 -- 172.21.15.31:0/1857589795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff810e000 con 0x560349828d80 2024-04-25T16:08:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.806+0000 7f3fe2ffd640 1 -- 172.21.15.31:0/1857589795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8143c20 con 0x560349828d80 2024-04-25T16:08:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.807+0000 7f400638a740 1 -- 172.21.15.31:0/1857589795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560349828d80 msgr2=0x560349c22a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.807+0000 7f400638a740 1 --2- 172.21.15.31:0/1857589795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560349828d80 0x560349c22a20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f3ff80520f0 tx=0x7f3ff8068e60 comp rx=0 tx=0).stop 2024-04-25T16:08:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.807+0000 7f400638a740 1 -- 172.21.15.31:0/1857589795 shutdown_connections 2024-04-25T16:08:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.807+0000 7f400638a740 1 --2- 172.21.15.31:0/1857589795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5603498494d0 0x560349c05d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.807+0000 7f400638a740 1 --2- 172.21.15.31:0/1857589795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498412f0 0x560349c1bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.807+0000 7f400638a740 1 --2- 172.21.15.31:0/1857589795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560349828d80 0x560349c22a20 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.807+0000 7f400638a740 1 -- 172.21.15.31:0/1857589795 >> 172.21.15.31:0/1857589795 conn(0x56034985e300 msgr2=0x560349c0eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.807+0000 7f400638a740 1 -- 172.21.15.31:0/1857589795 shutdown_connections 2024-04-25T16:08:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.807+0000 7f400638a740 1 -- 172.21.15.31:0/1857589795 wait complete. 2024-04-25T16:08:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.808+0000 7f400638a740 1 -- start start 2024-04-25T16:08:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.808+0000 7f400638a740 1 Processor -- start 2024-04-25T16:08:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.809+0000 7f400638a740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 0x560349c18910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.809+0000 7f400638a740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498412f0 0x560349c18e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.810+0000 7f400638a740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498494d0 0x560349c88c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.810+0000 7f400638a740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff015f4f0 con 0x5603498412f0 2024-04-25T16:08:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.810+0000 7f400638a740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff015ee20 con 0x560349828d80 2024-04-25T16:08:31.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.810+0000 7f400638a740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ff015c060 con 0x5603498494d0 2024-04-25T16:08:31.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.810+0000 7f400638a740 10 client.0 _ll_register_callbacks cb 0x7f3fe80096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:31.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.811+0000 7f3ffc99c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498412f0 0x560349c18e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.811+0000 7f3ffc99c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498412f0 0x560349c18e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:47600/0 (socket says 172.21.15.31:47600) 2024-04-25T16:08:31.812 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.811+0000 7f3ffc99c640 1 -- 172.21.15.31:0/575404224 learned_addr learned my addr 172.21.15.31:0/575404224 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.812+0000 7f3ffc99c640 1 -- 172.21.15.31:0/575404224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498494d0 msgr2=0x560349c88c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:08:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.812+0000 7f3ffc99c640 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498494d0 0x560349c88c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.812+0000 7f3ffc99c640 1 -- 172.21.15.31:0/575404224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 msgr2=0x560349c18910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:08:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.812+0000 7f3ffc99c640 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 0x560349c18910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:31.813 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.812+0000 7f3ffc99c640 1 -- 172.21.15.31:0/575404224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560349c19410 con 0x5603498412f0 2024-04-25T16:08:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.812+0000 7f3ffc99c640 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498412f0 0x560349c18e50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8030a60 tx=0x7f3fe802a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92780993439ba0ec server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.813+0000 7f3fe0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:08:31.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.813+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/575404224 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fe8055060 con 0x5603498412f0 2024-04-25T16:08:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.813+0000 7f400638a740 1 -- 172.21.15.31:0/575404224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ff810e000 con 0x5603498412f0 2024-04-25T16:08:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.814+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/575404224 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3fe80054b0 con 0x5603498412f0 2024-04-25T16:08:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.814+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/575404224 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3fe804b070 con 0x5603498412f0 2024-04-25T16:08:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.814+0000 7f400638a740 10 client.17121 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.814+0000 7f400638a740 1 -- 172.21.15.31:0/575404224 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560349c8a250 con 0x5603498412f0 2024-04-25T16:08:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.815+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/575404224 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ff810e000 con 0x5603498412f0 2024-04-25T16:08:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.817+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/575404224 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3fe804e030 con 0x5603498412f0 2024-04-25T16:08:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.818+0000 7f400638a740 10 client.17121 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.818+0000 7f400638a740 10 client.17121 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.818+0000 7f400638a740 20 client.17121 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.818+0000 7f400638a740 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560349c8ac40 0x560349c8d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.818+0000 7f400638a740 4 client.17121 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:08:31.819 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.818+0000 7f400638a740 1 -- 172.21.15.31:0/575404224 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560349b77ee0 con 0x560349c8ac40 2024-04-25T16:08:31.823 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.822+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560349c8ac40 0x560349c8d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.835+0000 7f3fe0ff9640 10 client.17121 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:31.839 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:31.835+0000 7f3ffd19d640 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560349c8ac40 0x560349c8d150 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abd688bdf3ff7e60 server_cookie=4f079d963dd8a705 in_seq=0 out_seq=0 2024-04-25T16:08:32.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:31 smithi063 ceph-mon[32413]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 754 MiB/s wr, 1.89k op/s 2024-04-25T16:08:32.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:31 smithi186 ceph-mon[33258]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 754 MiB/s wr, 1.89k op/s 2024-04-25T16:08:32.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:31 smithi031 ceph-mon[29140]: pgmap v568: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 754 MiB/s wr, 1.89k op/s 2024-04-25T16:08:33.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:32 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:08:33.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:32 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:08:33.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:32 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:08:34.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:34 smithi063 ceph-mon[32413]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1005 MiB/s wr, 2.35k op/s 2024-04-25T16:08:34.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:34 smithi186 ceph-mon[33258]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1005 MiB/s wr, 2.35k op/s 2024-04-25T16:08:34.458 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:34 smithi031 ceph-mon[29140]: pgmap v569: 97 pgs: 97 active+clean; 10 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1005 MiB/s wr, 2.35k op/s 2024-04-25T16:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.014+0000 7f3fe0ff9640 1 -- 172.21.15.31:0/575404224 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560349b77ee0 con 0x560349c8ac40 2024-04-25T16:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.014+0000 7f3fe0ff9640 10 client.17121 handle_command_reply: tid=0 2024-04-25T16:08:35.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.014+0000 7f400638a740 1 client.17121 shutdown 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.015+0000 7f400638a740 1 -- 172.21.15.31:0/575404224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498412f0 msgr2=0x560349c18e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.015+0000 7f400638a740 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498412f0 0x560349c18e50 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f3fe8030a60 tx=0x7f3fe802a360 comp rx=0 tx=0).stop 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.015+0000 7f400638a740 1 -- 172.21.15.31:0/575404224 shutdown_connections 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.015+0000 7f400638a740 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560349c8ac40 0x560349c8d150 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.015+0000 7f400638a740 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5603498494d0 0x560349c88c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.015+0000 7f400638a740 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5603498412f0 0x560349c18e50 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.015+0000 7f400638a740 1 --2- 172.21.15.31:0/575404224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560349828d80 0x560349c18910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.015+0000 7f400638a740 1 -- 172.21.15.31:0/575404224 >> 172.21.15.31:0/575404224 conn(0x56034985e300 msgr2=0x560349c09360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.015+0000 7f3fe0ff9640 0 client.17121 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:35.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.016+0000 7f400638a740 1 -- 172.21.15.31:0/575404224 shutdown_connections 2024-04-25T16:08:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.016+0000 7f400638a740 1 -- 172.21.15.31:0/575404224 wait complete. 2024-04-25T16:08:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.016+0000 7f400638a740 20 client.17121 trim_cache size 0 max 16384 2024-04-25T16:08:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:08:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:08:35.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:08:35.019 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:08:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.026+0000 7f3ffe19f640 1 -- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fe40afbb0 msgr2=0x7f3fe40b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:35.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.026+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fe40afbb0 0x7f3fe40b20c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f3ff400c6a0 tx=0x7f3ff401c040 comp rx=0 tx=0).stop 2024-04-25T16:08:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.027+0000 7f3ffe19f640 1 -- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ff00a5940 msgr2=0x7f3ff00b25e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:35.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.027+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ff00a5940 0x7f3ff00b25e0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8068520 tx=0x7f3ff8073a20 comp rx=0 tx=0).stop 2024-04-25T16:08:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.027+0000 7f3ffe19f640 1 -- 172.21.15.31:0/1428476400 shutdown_connections 2024-04-25T16:08:35.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.027+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3fe40afbb0 0x7f3fe40b20c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.028+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ff00ab760 0x7f3ff00b3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.028+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ff00a6a40 0x7f3ff00b2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.028+0000 7f3ffe19f640 1 --2- 172.21.15.31:0/1428476400 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ff00a5940 0x7f3ff00b25e0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.028+0000 7f3ffe19f640 1 -- 172.21.15.31:0/1428476400 >> 172.21.15.31:0/1428476400 conn(0x7f3ff00a09b0 msgr2=0x7f3ff00a7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.028+0000 7f3ffe19f640 1 -- 172.21.15.31:0/1428476400 shutdown_connections 2024-04-25T16:08:35.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:35.028+0000 7f3ffe19f640 1 -- 172.21.15.31:0/1428476400 wait complete. 2024-04-25T16:08:36.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:35 smithi063 ceph-mon[32413]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 753 MiB/s wr, 1.79k op/s 2024-04-25T16:08:36.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:35 smithi186 ceph-mon[33258]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 753 MiB/s wr, 1.79k op/s 2024-04-25T16:08:36.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:35 smithi031 ceph-mon[29140]: pgmap v570: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 753 MiB/s wr, 1.79k op/s 2024-04-25T16:08:38.202 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:37 smithi031 ceph-mon[29140]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 849 MiB/s wr, 1.91k op/s 2024-04-25T16:08:38.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:37 smithi063 ceph-mon[32413]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 849 MiB/s wr, 1.91k op/s 2024-04-25T16:08:38.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:37 smithi186 ceph-mon[33258]: pgmap v571: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 849 MiB/s wr, 1.91k op/s 2024-04-25T16:08:39.205 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:38 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:08:39.270 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:38 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:08:39.299 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:38 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:08:40.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:39 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:08:40.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:39 smithi063 ceph-mon[32413]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 803 MiB/s wr, 1.62k op/s 2024-04-25T16:08:40.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:39 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:08:40.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:39 smithi186 ceph-mon[33258]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 803 MiB/s wr, 1.62k op/s 2024-04-25T16:08:40.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:39 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:08:40.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:39 smithi031 ceph-mon[29140]: pgmap v572: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 803 MiB/s wr, 1.62k op/s 2024-04-25T16:08:42.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:41 smithi063 ceph-mon[32413]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 749 MiB/s wr, 1.45k op/s 2024-04-25T16:08:42.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:41 smithi186 ceph-mon[33258]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 749 MiB/s wr, 1.45k op/s 2024-04-25T16:08:42.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:41 smithi031 ceph-mon[29140]: pgmap v573: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 749 MiB/s wr, 1.45k op/s 2024-04-25T16:08:44.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:43 smithi063 ceph-mon[32413]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 993 MiB/s wr, 2.29k op/s 2024-04-25T16:08:44.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:43 smithi186 ceph-mon[33258]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 993 MiB/s wr, 2.29k op/s 2024-04-25T16:08:44.456 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:43 smithi031 ceph-mon[29140]: pgmap v574: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 993 MiB/s wr, 2.29k op/s 2024-04-25T16:08:46.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:45 smithi031 ceph-mon[29140]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 724 MiB/s wr, 1.85k op/s 2024-04-25T16:08:46.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:45 smithi063 ceph-mon[32413]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 724 MiB/s wr, 1.85k op/s 2024-04-25T16:08:46.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:45 smithi186 ceph-mon[33258]: pgmap v575: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 724 MiB/s wr, 1.85k op/s 2024-04-25T16:08:47.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:47 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:08:47.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:47 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:08:47.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:47 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:08:48.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:48 smithi031 ceph-mon[29140]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 848 MiB/s wr, 2.25k op/s 2024-04-25T16:08:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.554+0000 7fa93c26e640 1 -- 172.21.15.31:0/103243896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410a350 msgr2=0x7fa93410ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:48.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.554+0000 7fa93c26e640 1 --2- 172.21.15.31:0/103243896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410a350 0x7fa93410ac90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa930009860 tx=0x7fa93002a380 comp rx=0 tx=0).stop 2024-04-25T16:08:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.567+0000 7fa93c26e640 1 -- 172.21.15.31:0/103243896 shutdown_connections 2024-04-25T16:08:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.567+0000 7fa93c26e640 1 --2- 172.21.15.31:0/103243896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93410fef0 0x7fa934110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.567+0000 7fa93c26e640 1 --2- 172.21.15.31:0/103243896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa93410b1d0 0x7fa93410b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.567+0000 7fa93c26e640 1 --2- 172.21.15.31:0/103243896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410a350 0x7fa93410ac90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.570 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.567+0000 7fa93c26e640 1 -- 172.21.15.31:0/103243896 >> 172.21.15.31:0/103243896 conn(0x7fa934070390 msgr2=0x7fa934072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.591+0000 7fa93c26e640 1 -- 172.21.15.31:0/103243896 shutdown_connections 2024-04-25T16:08:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.592+0000 7fa93c26e640 1 -- 172.21.15.31:0/103243896 wait complete. 2024-04-25T16:08:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.593+0000 7fa93c26e640 1 Processor -- start 2024-04-25T16:08:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.594+0000 7fa93c26e640 1 -- start start 2024-04-25T16:08:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.594+0000 7fa93c26e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa93410a350 0x7fa93411b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.594+0000 7fa93c26e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93410b1d0 0x7fa934119fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.594+0000 7fa93c26e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410fef0 0x7fa93411a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.594+0000 7fa93c26e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa934112360 con 0x7fa93410b1d0 2024-04-25T16:08:48.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.594+0000 7fa93c26e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa9341121e0 con 0x7fa93410a350 2024-04-25T16:08:48.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.594+0000 7fa93c26e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa9341124e0 con 0x7fa93410fef0 2024-04-25T16:08:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.595+0000 7fa93a80b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410fef0 0x7fa93411a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.595+0000 7fa93a80b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410fef0 0x7fa93411a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:35582/0 (socket says 172.21.15.31:35582) 2024-04-25T16:08:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.595+0000 7fa93a80b640 1 -- 172.21.15.31:0/699253369 learned_addr learned my addr 172.21.15.31:0/699253369 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.596+0000 7fa93a80b640 1 -- 172.21.15.31:0/699253369 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa93410a350 msgr2=0x7fa93411b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.596+0000 7fa93a80b640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa93410a350 0x7fa93411b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.596+0000 7fa93a80b640 1 -- 172.21.15.31:0/699253369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93410b1d0 msgr2=0x7fa934119fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.596+0000 7fa93a80b640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93410b1d0 0x7fa934119fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.596+0000 7fa93a80b640 1 -- 172.21.15.31:0/699253369 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa93411aa60 con 0x7fa93410fef0 2024-04-25T16:08:48.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.597+0000 7fa93a80b640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410fef0 0x7fa93411a520 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa92400d750 tx=0x7fa92400dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:48.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.598+0000 7fa92b7fe640 1 -- 172.21.15.31:0/699253369 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa924017070 con 0x7fa93410fef0 2024-04-25T16:08:48.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.598+0000 7fa93c26e640 1 -- 172.21.15.31:0/699253369 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa93411ad50 con 0x7fa93410fef0 2024-04-25T16:08:48.602 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.598+0000 7fa93c26e640 1 -- 172.21.15.31:0/699253369 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa934123d20 con 0x7fa93410fef0 2024-04-25T16:08:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.606+0000 7fa943204740 1 Processor -- start 2024-04-25T16:08:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.610+0000 7fa943204740 1 -- start start 2024-04-25T16:08:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.610+0000 7fa943204740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbd91b0 0x56458df79b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.610+0000 7fa943204740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbb02f0 0x56458df7a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.611+0000 7fa943204740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df80fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.611+0000 7fa943204740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56458dedf460 con 0x56458dbd91b0 2024-04-25T16:08:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.611+0000 7fa943204740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56458de99590 con 0x56458dbb84d0 2024-04-25T16:08:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.611+0000 7fa943204740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56458dedb420 con 0x56458dbb02f0 2024-04-25T16:08:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.611+0000 7fa92b7fe640 1 -- 172.21.15.31:0/699253369 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9240054d0 con 0x7fa93410fef0 2024-04-25T16:08:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.612+0000 7fa93a00a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbb02f0 0x56458df7a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.612+0000 7fa93a00a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbb02f0 0x56458df7a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:35592/0 (socket says 172.21.15.31:35592) 2024-04-25T16:08:48.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.612+0000 7fa93a00a640 1 -- 172.21.15.31:0/1131540708 learned_addr learned my addr 172.21.15.31:0/1131540708 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:48.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.613+0000 7fa93a00a640 1 -- 172.21.15.31:0/1131540708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 msgr2=0x56458df80fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:08:48.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.613+0000 7fa93a00a640 1 --2- 172.21.15.31:0/1131540708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df80fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.613+0000 7fa93a00a640 1 -- 172.21.15.31:0/1131540708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbd91b0 msgr2=0x56458df79b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:48.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.613+0000 7fa93a00a640 1 --2- 172.21.15.31:0/1131540708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbd91b0 0x56458df79b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:48 smithi063 ceph-mon[32413]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 848 MiB/s wr, 2.25k op/s 2024-04-25T16:08:48.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.619+0000 7fa93a00a640 1 -- 172.21.15.31:0/1131540708 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56458dea60d0 con 0x56458dbb02f0 2024-04-25T16:08:48.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.619+0000 7fa92b7fe640 1 -- 172.21.15.31:0/699253369 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa924004cb0 con 0x7fa93410fef0 2024-04-25T16:08:48.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.619+0000 7fa92b7fe640 1 -- 172.21.15.31:0/699253369 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa924010460 con 0x7fa93410fef0 2024-04-25T16:08:48.621 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.619+0000 7fa93a00a640 1 --2- 172.21.15.31:0/1131540708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbb02f0 0x56458df7a0a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa930030520 tx=0x7fa930005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a40af2cb056ac523 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.620+0000 7fa928bfe640 1 -- 172.21.15.31:0/1131540708 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930037040 con 0x56458dbb02f0 2024-04-25T16:08:48.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.620+0000 7fa928bfe640 1 -- 172.21.15.31:0/1131540708 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa930040070 con 0x56458dbb02f0 2024-04-25T16:08:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.620+0000 7fa92b7fe640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9200afbd0 0x7fa9200b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.624 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.621+0000 7fa92b7fe640 1 -- 172.21.15.31:0/699253369 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa9240df580 con 0x7fa93410fef0 2024-04-25T16:08:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.623+0000 7fa943204740 1 -- 172.21.15.31:0/1131540708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbb02f0 msgr2=0x56458df7a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.623+0000 7fa943204740 1 --2- 172.21.15.31:0/1131540708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbb02f0 0x56458df7a0a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa930030520 tx=0x7fa930005e40 comp rx=0 tx=0).stop 2024-04-25T16:08:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.623+0000 7fa943204740 1 -- 172.21.15.31:0/1131540708 shutdown_connections 2024-04-25T16:08:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.623+0000 7fa943204740 1 --2- 172.21.15.31:0/1131540708 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df80fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.623+0000 7fa943204740 1 --2- 172.21.15.31:0/1131540708 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbb02f0 0x56458df7a0a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.625 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.623+0000 7fa943204740 1 --2- 172.21.15.31:0/1131540708 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbd91b0 0x56458df79b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.623+0000 7fa943204740 1 -- 172.21.15.31:0/1131540708 >> 172.21.15.31:0/1131540708 conn(0x56458dbcd300 msgr2=0x56458df754b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.623+0000 7fa943204740 1 -- 172.21.15.31:0/1131540708 shutdown_connections 2024-04-25T16:08:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.623+0000 7fa943204740 1 -- 172.21.15.31:0/1131540708 wait complete. 2024-04-25T16:08:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.624+0000 7fa943204740 1 -- start start 2024-04-25T16:08:48.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.624+0000 7fa943204740 1 Processor -- start 2024-04-25T16:08:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.624+0000 7fa943204740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df836c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.624+0000 7fa943204740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbd91b0 0x56458df83c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.624+0000 7fa943204740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458db97d80 0x56458df8ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.625+0000 7fa943204740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56458df819e0 con 0x56458dbd91b0 2024-04-25T16:08:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.625+0000 7fa943204740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56458de66b60 con 0x56458dbb84d0 2024-04-25T16:08:48.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.625+0000 7fa943204740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56458df81b60 con 0x56458db97d80 2024-04-25T16:08:48.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.625+0000 7fa943204740 10 client.0 _ll_register_callbacks cb 0x56458de8b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:48.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.625+0000 7fa93a80b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458db97d80 0x56458df8ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.625+0000 7fa93a00a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df836c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.632 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.625+0000 7fa93a80b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458db97d80 0x56458df8ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:35608/0 (socket says 172.21.15.31:35608) 2024-04-25T16:08:48.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.625+0000 7fa93a80b640 1 -- 172.21.15.31:0/3054294752 learned_addr learned my addr 172.21.15.31:0/3054294752 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:48.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.626+0000 7fa93a00a640 1 -- 172.21.15.31:0/3054294752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458db97d80 msgr2=0x56458df8ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:48.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.629+0000 7fa93a00a640 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458db97d80 0x56458df8ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.629+0000 7fa93a00a640 1 -- 172.21.15.31:0/3054294752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbd91b0 msgr2=0x56458df83c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:48.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.629+0000 7fa93a00a640 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbd91b0 0x56458df83c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.629+0000 7fa93a00a640 1 -- 172.21.15.31:0/3054294752 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56458dbf1210 con 0x56458dbb84d0 2024-04-25T16:08:48.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.631+0000 7fa93a00a640 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df836c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56458df76240 tx=0x7fa93002ab00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f41725b467001030 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:48.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.632+0000 7fa939809640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9200afbd0 0x7fa9200b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.632+0000 7fa91dffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:08:48.635 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.633+0000 7fa91dffb640 1 -- 172.21.15.31:0/3054294752 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa930037040 con 0x56458dbb84d0 2024-04-25T16:08:48.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.633+0000 7fa91dffb640 1 -- 172.21.15.31:0/3054294752 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa930030750 con 0x56458dbb84d0 2024-04-25T16:08:48.636 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.633+0000 7fa943204740 1 -- 172.21.15.31:0/3054294752 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56458deed070 con 0x56458dbb84d0 2024-04-25T16:08:48.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.634+0000 7fa91dffb640 1 -- 172.21.15.31:0/3054294752 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa930030950 con 0x56458dbb84d0 2024-04-25T16:08:48.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.635+0000 7fa943204740 10 client.26768 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:48.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.635+0000 7fa943204740 1 -- 172.21.15.31:0/3054294752 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56458dcbd620 con 0x56458dbb84d0 2024-04-25T16:08:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.638+0000 7fa91dffb640 1 -- 172.21.15.31:0/3054294752 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56458deed070 con 0x56458dbb84d0 2024-04-25T16:08:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.638+0000 7fa91dffb640 1 -- 172.21.15.31:0/3054294752 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa93003a070 con 0x56458dbb84d0 2024-04-25T16:08:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.638+0000 7fa943204740 10 client.26768 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.638+0000 7fa943204740 10 client.26768 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.638+0000 7fa943204740 20 client.26768 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.638+0000 7fa943204740 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56458db6d1c0 0x56458df91040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.638+0000 7fa943204740 4 client.26768 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:08:48.640 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.638+0000 7fa943204740 1 -- 172.21.15.31:0/3054294752 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56458dee6ee0 con 0x56458db6d1c0 2024-04-25T16:08:48.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:48 smithi186 ceph-mon[33258]: pgmap v576: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 848 MiB/s wr, 2.25k op/s 2024-04-25T16:08:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.645+0000 7fa939809640 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56458db6d1c0 0x56458df91040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.648+0000 7fa939809640 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56458db6d1c0 0x56458df91040 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27005295705b0b05 server_cookie=feb5d39e2bdd1eb9 in_seq=0 out_seq=0 2024-04-25T16:08:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.648+0000 7fa939809640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9200afbd0 0x7fa9200b20e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009810 tx=0x7fa92c009f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.648+0000 7fa91dffb640 10 client.26768 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.651+0000 7fa91dffb640 1 -- 172.21.15.31:0/3054294752 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56458dee6ee0 con 0x56458db6d1c0 2024-04-25T16:08:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.651+0000 7fa91dffb640 10 client.26768 handle_command_reply: tid=0 2024-04-25T16:08:48.654 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.652+0000 7fa943204740 1 client.26768 shutdown 2024-04-25T16:08:48.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.661+0000 7fa943204740 1 -- 172.21.15.31:0/3054294752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 msgr2=0x56458df836c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:48.663 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.661+0000 7fa943204740 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df836c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56458df76240 tx=0x7fa93002ab00 comp rx=0 tx=0).stop 2024-04-25T16:08:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.663+0000 7fa943204740 1 -- 172.21.15.31:0/3054294752 shutdown_connections 2024-04-25T16:08:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.663+0000 7fa943204740 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56458db6d1c0 0x56458df91040 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.665 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.663+0000 7fa943204740 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458db97d80 0x56458df8ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.663+0000 7fa943204740 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbd91b0 0x56458df83c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.663+0000 7fa943204740 1 --2- 172.21.15.31:0/3054294752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df836c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.663+0000 7fa943204740 1 -- 172.21.15.31:0/3054294752 >> 172.21.15.31:0/3054294752 conn(0x56458dbcd300 msgr2=0x56458df78430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:48.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.663+0000 7fa91dffb640 0 client.26768 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:48.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.663+0000 7fa943204740 1 -- 172.21.15.31:0/3054294752 shutdown_connections 2024-04-25T16:08:48.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.663+0000 7fa943204740 1 -- 172.21.15.31:0/3054294752 wait complete. 2024-04-25T16:08:48.666 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.664+0000 7fa943204740 20 client.26768 trim_cache size 0 max 16384 2024-04-25T16:08:48.720 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.718+0000 7fa943204740 1 Processor -- start 2024-04-25T16:08:48.728 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.726+0000 7fa943204740 1 -- start start 2024-04-25T16:08:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.726+0000 7fa943204740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458db97d80 0x56458df95760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.726+0000 7fa943204740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbb84d0 0x56458df8ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.726+0000 7fa943204740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df95ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.726+0000 7fa943204740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa91800e590 con 0x56458dbb84d0 2024-04-25T16:08:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.726+0000 7fa943204740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa9180100d0 con 0x56458db97d80 2024-04-25T16:08:48.738 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.726+0000 7fa943204740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa91800ec60 con 0x56458dbd91b0 2024-04-25T16:08:48.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.736+0000 7fa93a80b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df95ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.736+0000 7fa93a00a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458db97d80 0x56458df95760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.736+0000 7fa93a80b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df95ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:35622/0 (socket says 172.21.15.31:35622) 2024-04-25T16:08:48.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.736+0000 7fa93a80b640 1 -- 172.21.15.31:0/4025075539 learned_addr learned my addr 172.21.15.31:0/4025075539 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:48.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.741+0000 7fa939809640 1 --2- 172.21.15.31:0/4025075539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbb84d0 0x56458df8ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.741+0000 7fa93a80b640 1 -- 172.21.15.31:0/4025075539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458db97d80 msgr2=0x56458df95760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.741+0000 7fa93a80b640 1 --2- 172.21.15.31:0/4025075539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458db97d80 0x56458df95760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.741+0000 7fa93a80b640 1 -- 172.21.15.31:0/4025075539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbb84d0 msgr2=0x56458df8ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.741+0000 7fa93a80b640 1 --2- 172.21.15.31:0/4025075539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbb84d0 0x56458df8ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.741+0000 7fa93a80b640 1 -- 172.21.15.31:0/4025075539 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56458df842c0 con 0x56458dbd91b0 2024-04-25T16:08:48.747 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.744+0000 7fa93a80b640 1 --2- 172.21.15.31:0/4025075539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df95ca0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa924005f80 tx=0x7fa9240b02f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df8560125cbab29d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.755+0000 7fa91ffff640 1 -- 172.21.15.31:0/4025075539 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9240c1070 con 0x56458dbd91b0 2024-04-25T16:08:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.755+0000 7fa91ffff640 1 -- 172.21.15.31:0/4025075539 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9240a7a90 con 0x56458dbd91b0 2024-04-25T16:08:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.757+0000 7fa943204740 1 -- 172.21.15.31:0/4025075539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 msgr2=0x56458df95ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.757+0000 7fa943204740 1 --2- 172.21.15.31:0/4025075539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df95ca0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa924005f80 tx=0x7fa9240b02f0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.760 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.758+0000 7fa943204740 1 -- 172.21.15.31:0/4025075539 shutdown_connections 2024-04-25T16:08:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.758+0000 7fa943204740 1 --2- 172.21.15.31:0/4025075539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df95ca0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.758+0000 7fa943204740 1 --2- 172.21.15.31:0/4025075539 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458dbb84d0 0x56458df8ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.758+0000 7fa943204740 1 --2- 172.21.15.31:0/4025075539 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458db97d80 0x56458df95760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.758+0000 7fa943204740 1 -- 172.21.15.31:0/4025075539 >> 172.21.15.31:0/4025075539 conn(0x56458dbcd300 msgr2=0x56458df82970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:48.761 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.759+0000 7fa943204740 1 -- 172.21.15.31:0/4025075539 shutdown_connections 2024-04-25T16:08:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.759+0000 7fa943204740 1 -- 172.21.15.31:0/4025075539 wait complete. 2024-04-25T16:08:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.760+0000 7fa943204740 1 -- start start 2024-04-25T16:08:48.762 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.760+0000 7fa943204740 1 Processor -- start 2024-04-25T16:08:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.760+0000 7fa943204740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458db97d80 0x56458df87dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.765 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.760+0000 7fa943204740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df88310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.763+0000 7fa943204740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df89240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.763+0000 7fa943204740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa91800fa00 con 0x56458db97d80 2024-04-25T16:08:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.763+0000 7fa943204740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa91800f330 con 0x56458dbb84d0 2024-04-25T16:08:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.763+0000 7fa943204740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa91800c570 con 0x56458dbd91b0 2024-04-25T16:08:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.763+0000 7fa943204740 10 client.0 _ll_register_callbacks cb 0x7fa92c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:08:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.764+0000 7fa939809640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df88310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.764+0000 7fa93a80b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df89240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.764+0000 7fa93a80b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df89240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:35626/0 (socket says 172.21.15.31:35626) 2024-04-25T16:08:48.766 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.764+0000 7fa93a80b640 1 -- 172.21.15.31:0/2515481169 learned_addr learned my addr 172.21.15.31:0/2515481169 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:08:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.764+0000 7fa93a80b640 1 -- 172.21.15.31:0/2515481169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 msgr2=0x56458df88310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.764+0000 7fa93a80b640 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df88310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.764+0000 7fa93a80b640 1 -- 172.21.15.31:0/2515481169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458db97d80 msgr2=0x56458df87dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:08:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.764+0000 7fa93a80b640 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458db97d80 0x56458df87dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.764+0000 7fa93a80b640 1 -- 172.21.15.31:0/2515481169 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56458df89800 con 0x56458dbd91b0 2024-04-25T16:08:48.767 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.765+0000 7fa93a80b640 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df89240 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa9240a3b10 tx=0x7fa9240a3fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7853aa6b98a394e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:08:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.771+0000 7fa91dffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:08:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.774+0000 7fa91dffb640 1 -- 172.21.15.31:0/2515481169 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9240c1040 con 0x56458dbd91b0 2024-04-25T16:08:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.774+0000 7fa943204740 1 -- 172.21.15.31:0/2515481169 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9240a7a90 con 0x56458dbd91b0 2024-04-25T16:08:48.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.774+0000 7fa91dffb640 1 -- 172.21.15.31:0/2515481169 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa9240df0f0 con 0x56458dbd91b0 2024-04-25T16:08:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.774+0000 7fa91dffb640 1 -- 172.21.15.31:0/2515481169 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa9240dd1c0 con 0x56458dbd91b0 2024-04-25T16:08:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.775+0000 7fa943204740 10 client.26740 fetch_fsmap learned FSMap version 29 2024-04-25T16:08:48.777 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.775+0000 7fa943204740 1 -- 172.21.15.31:0/2515481169 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56458df8ab00 con 0x56458dbd91b0 2024-04-25T16:08:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.776+0000 7fa91dffb640 1 -- 172.21.15.31:0/2515481169 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9240a7a90 con 0x56458dbd91b0 2024-04-25T16:08:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.776+0000 7fa91dffb640 1 -- 172.21.15.31:0/2515481169 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa9240ca0d0 con 0x56458dbd91b0 2024-04-25T16:08:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.776+0000 7fa943204740 10 client.26740 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:08:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.776+0000 7fa943204740 10 client.26740 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:08:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.776+0000 7fa943204740 20 client.26740 populate_metadata read hostname 'smithi031' 2024-04-25T16:08:48.778 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.776+0000 7fa943204740 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56458dff9690 0x56458dffbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:08:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.776+0000 7fa943204740 4 client.26740 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:08:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.776+0000 7fa943204740 1 -- 172.21.15.31:0/2515481169 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56458de625f0 con 0x56458dff9690 2024-04-25T16:08:48.779 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.777+0000 7fa93a00a640 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56458dff9690 0x56458dffbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:08:48.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.782+0000 7fa91dffb640 10 client.26740 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:48.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:48.783+0000 7fa93a00a640 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56458dff9690 0x56458dffbab0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9803ffd6fd15609f server_cookie=1d0474a82afc51a0 in_seq=0 out_seq=0 2024-04-25T16:08:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:49 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:49 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-25T16:08:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:49 smithi063 ceph-mon[32413]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-25T16:08:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:49 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:49 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-25T16:08:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:49 smithi186 ceph-mon[33258]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-25T16:08:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:49 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:49.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:49 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi186 to 176.2M 2024-04-25T16:08:49.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:49 smithi031 ceph-mon[29140]: Unable to set osd_memory_target on smithi186 to 184788172: error parsing value: Value '184788172' is below minimum 939524096 2024-04-25T16:08:50.605 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:50 smithi031 ceph-mon[29140]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 843 MiB/s wr, 2.31k op/s 2024-04-25T16:08:50.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:50 smithi063 ceph-mon[32413]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 843 MiB/s wr, 2.31k op/s 2024-04-25T16:08:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:50 smithi186 ceph-mon[33258]: pgmap v577: 97 pgs: 97 active+clean; 10 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 843 MiB/s wr, 2.31k op/s 2024-04-25T16:08:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.971+0000 7fa91dffb640 1 -- 172.21.15.31:0/2515481169 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56458de625f0 con 0x56458dff9690 2024-04-25T16:08:50.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.971+0000 7fa91dffb640 10 client.26740 handle_command_reply: tid=0 2024-04-25T16:08:50.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.971+0000 7fa943204740 1 client.26740 shutdown 2024-04-25T16:08:50.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.978+0000 7fa943204740 1 -- 172.21.15.31:0/2515481169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 msgr2=0x56458df89240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:50.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.978+0000 7fa943204740 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df89240 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa9240a3b10 tx=0x7fa9240a3fd0 comp rx=0 tx=0).stop 2024-04-25T16:08:50.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.978+0000 7fa943204740 1 -- 172.21.15.31:0/2515481169 shutdown_connections 2024-04-25T16:08:50.980 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.978+0000 7fa943204740 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x56458dff9690 0x56458dffbab0 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:50.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.978+0000 7fa943204740 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56458dbd91b0 0x56458df89240 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:50.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.978+0000 7fa943204740 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56458dbb84d0 0x56458df88310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:50.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.978+0000 7fa943204740 1 --2- 172.21.15.31:0/2515481169 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56458db97d80 0x56458df87dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:50.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.978+0000 7fa943204740 1 -- 172.21.15.31:0/2515481169 >> 172.21.15.31:0/2515481169 conn(0x56458dbcd300 msgr2=0x56458df78a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:50.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.979+0000 7fa91dffb640 0 client.26740 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:08:50.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.979+0000 7fa943204740 1 -- 172.21.15.31:0/2515481169 shutdown_connections 2024-04-25T16:08:50.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.979+0000 7fa943204740 1 -- 172.21.15.31:0/2515481169 wait complete. 2024-04-25T16:08:50.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.979+0000 7fa943204740 20 client.26740 trim_cache size 0 max 16384 2024-04-25T16:08:50.981 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:08:50.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:08:50.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:08:50.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:08:50.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.981+0000 7fa93c26e640 1 -- 172.21.15.31:0/699253369 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9200afbd0 msgr2=0x7fa9200b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:50.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.981+0000 7fa93c26e640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9200afbd0 0x7fa9200b20e0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa92c009810 tx=0x7fa92c009f70 comp rx=0 tx=0).stop 2024-04-25T16:08:50.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.981+0000 7fa93c26e640 1 -- 172.21.15.31:0/699253369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410fef0 msgr2=0x7fa93411a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:08:50.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.981+0000 7fa93c26e640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410fef0 0x7fa93411a520 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa92400d750 tx=0x7fa92400dc10 comp rx=0 tx=0).stop 2024-04-25T16:08:50.983 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.981+0000 7fa93c26e640 1 -- 172.21.15.31:0/699253369 shutdown_connections 2024-04-25T16:08:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.982+0000 7fa93c26e640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa9200afbd0 0x7fa9200b20e0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.982+0000 7fa93c26e640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa93410fef0 0x7fa93411a520 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.982+0000 7fa93c26e640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa93410b1d0 0x7fa934119fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.982+0000 7fa93c26e640 1 --2- 172.21.15.31:0/699253369 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa93410a350 0x7fa93411b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:08:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.982+0000 7fa93c26e640 1 -- 172.21.15.31:0/699253369 >> 172.21.15.31:0/699253369 conn(0x7fa934070390 msgr2=0x7fa93410dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:08:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.983+0000 7fa93c26e640 1 -- 172.21.15.31:0/699253369 shutdown_connections 2024-04-25T16:08:50.984 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:08:50.983+0000 7fa93c26e640 1 -- 172.21.15.31:0/699253369 wait complete. 2024-04-25T16:08:51.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:51 smithi063 ceph-mon[32413]: Adjusting osd_memory_target on smithi063 to 176.2M 2024-04-25T16:08:51.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:51 smithi063 ceph-mon[32413]: Unable to set osd_memory_target on smithi063 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-04-25T16:08:51.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:08:51.621 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:08:51.621 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:08:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:51 smithi186 ceph-mon[33258]: Adjusting osd_memory_target on smithi063 to 176.2M 2024-04-25T16:08:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:51 smithi186 ceph-mon[33258]: Unable to set osd_memory_target on smithi063 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-04-25T16:08:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:08:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:08:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:08:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-25T16:08:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:51 smithi031 ceph-mon[29140]: Adjusting osd_memory_target on smithi063 to 176.2M 2024-04-25T16:08:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:51 smithi031 ceph-mon[29140]: Unable to set osd_memory_target on smithi063 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-04-25T16:08:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:08:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:08:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:08:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:52 smithi063 ceph-mon[32413]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 753 MiB/s wr, 2.18k op/s 2024-04-25T16:08:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:52 smithi186 ceph-mon[33258]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 753 MiB/s wr, 2.18k op/s 2024-04-25T16:08:52.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:52 smithi031 ceph-mon[29140]: pgmap v578: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 753 MiB/s wr, 2.18k op/s 2024-04-25T16:08:54.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:53 smithi063 ceph-mon[32413]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 935 MiB/s wr, 2.48k op/s 2024-04-25T16:08:54.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:53 smithi186 ceph-mon[33258]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 935 MiB/s wr, 2.48k op/s 2024-04-25T16:08:54.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:53 smithi031 ceph-mon[29140]: pgmap v579: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 935 MiB/s wr, 2.48k op/s 2024-04-25T16:08:56.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:55 smithi063 ceph-mon[32413]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 713 MiB/s wr, 1.71k op/s 2024-04-25T16:08:56.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:55 smithi186 ceph-mon[33258]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 713 MiB/s wr, 1.71k op/s 2024-04-25T16:08:56.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:55 smithi031 ceph-mon[29140]: pgmap v580: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 713 MiB/s wr, 1.71k op/s 2024-04-25T16:08:58.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:57 smithi063 ceph-mon[32413]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 867 MiB/s wr, 2.02k op/s 2024-04-25T16:08:58.401 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:57 smithi186 ceph-mon[33258]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 867 MiB/s wr, 2.02k op/s 2024-04-25T16:08:58.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:57 smithi031 ceph-mon[29140]: pgmap v581: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 867 MiB/s wr, 2.02k op/s 2024-04-25T16:09:00.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:08:59 smithi063 ceph-mon[32413]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 840 MiB/s wr, 2.01k op/s 2024-04-25T16:09:00.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:08:59 smithi186 ceph-mon[33258]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 840 MiB/s wr, 2.01k op/s 2024-04-25T16:09:00.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:08:59 smithi031 ceph-mon[29140]: pgmap v582: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 840 MiB/s wr, 2.01k op/s 2024-04-25T16:09:00.517 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:09:00.518 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:09:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.023+0000 7fe977ac6640 1 -- 172.21.15.31:0/888482493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97010fef0 msgr2=0x7fe970110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.023+0000 7fe977ac6640 1 --2- 172.21.15.31:0/888482493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97010fef0 0x7fe970110300 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe960009900 tx=0x7fe96002a440 comp rx=0 tx=0).stop 2024-04-25T16:09:01.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.024+0000 7fe977ac6640 1 -- 172.21.15.31:0/888482493 shutdown_connections 2024-04-25T16:09:01.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.024+0000 7fe977ac6640 1 --2- 172.21.15.31:0/888482493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97010fef0 0x7fe970110300 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.024+0000 7fe977ac6640 1 --2- 172.21.15.31:0/888482493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97010b1d0 0x7fe97010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.024+0000 7fe977ac6640 1 --2- 172.21.15.31:0/888482493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97010a3e0 0x7fe97010ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.024+0000 7fe977ac6640 1 -- 172.21.15.31:0/888482493 >> 172.21.15.31:0/888482493 conn(0x7fe970070400 msgr2=0x7fe970072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:01.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.033+0000 7fe977ac6640 1 -- 172.21.15.31:0/888482493 shutdown_connections 2024-04-25T16:09:01.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.035+0000 7fe977ac6640 1 -- 172.21.15.31:0/888482493 wait complete. 2024-04-25T16:09:01.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.036+0000 7fe977ac6640 1 Processor -- start 2024-04-25T16:09:01.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.036+0000 7fe977ac6640 1 -- start start 2024-04-25T16:09:01.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.036+0000 7fe977ac6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97010a3e0 0x7fe9701a8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.036+0000 7fe977ac6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97010b1d0 0x7fe9701a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.036+0000 7fe977ac6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97010fef0 0x7fe9701b0460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.036+0000 7fe977ac6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe97011e330 con 0x7fe97010b1d0 2024-04-25T16:09:01.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.036+0000 7fe977ac6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe97011e1b0 con 0x7fe97010a3e0 2024-04-25T16:09:01.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.036+0000 7fe977ac6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe97011e4b0 con 0x7fe97010fef0 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.037+0000 7fe975862640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97010a3e0 0x7fe9701a8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.037+0000 7fe975862640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97010a3e0 0x7fe9701a8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:33100/0 (socket says 172.21.15.31:33100) 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.037+0000 7fe975862640 1 -- 172.21.15.31:0/1051798499 learned_addr learned my addr 172.21.15.31:0/1051798499 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.037+0000 7fe976063640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97010fef0 0x7fe9701b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.037+0000 7fe976063640 1 -- 172.21.15.31:0/1051798499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97010a3e0 msgr2=0x7fe9701a8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.037+0000 7fe976063640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97010a3e0 0x7fe9701a8f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.037+0000 7fe976063640 1 -- 172.21.15.31:0/1051798499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97010b1d0 msgr2=0x7fe9701a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.037+0000 7fe976063640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97010b1d0 0x7fe9701a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.037+0000 7fe976063640 1 -- 172.21.15.31:0/1051798499 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe970111c90 con 0x7fe97010fef0 2024-04-25T16:09:01.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.038+0000 7fe976063640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97010fef0 0x7fe9701b0460 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe960009230 tx=0x7fe960003cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.045+0000 7fe966ffd640 1 -- 172.21.15.31:0/1051798499 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96003f070 con 0x7fe97010fef0 2024-04-25T16:09:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.045+0000 7fe966ffd640 1 -- 172.21.15.31:0/1051798499 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe960031c50 con 0x7fe97010fef0 2024-04-25T16:09:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.045+0000 7fe966ffd640 1 -- 172.21.15.31:0/1051798499 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe960035030 con 0x7fe97010fef0 2024-04-25T16:09:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.045+0000 7fe977ac6640 1 -- 172.21.15.31:0/1051798499 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe970111f20 con 0x7fe97010fef0 2024-04-25T16:09:01.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.045+0000 7fe977ac6640 1 -- 172.21.15.31:0/1051798499 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe970112440 con 0x7fe97010fef0 2024-04-25T16:09:01.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.052+0000 7fe97e977740 1 Processor -- start 2024-04-25T16:09:01.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.053+0000 7fe97e977740 1 -- start start 2024-04-25T16:09:01.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.054+0000 7fe97e977740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c017c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.054+0000 7fe97e977740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc4d4d0 0x55d91c0181d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.054+0000 7fe97e977740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc556b0 0x55d91c018e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.055+0000 7fe97e977740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d91bf7cd80 con 0x55d91bc4d4d0 2024-04-25T16:09:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.055+0000 7fe975862640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc4d4d0 0x55d91c0181d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.055+0000 7fe966ffd640 1 -- 172.21.15.31:0/1051798499 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe960042070 con 0x7fe97010fef0 2024-04-25T16:09:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.055+0000 7fe97e977740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d91bf368b0 con 0x55d91bc556b0 2024-04-25T16:09:01.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.055+0000 7fe97e977740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d91bf78d40 con 0x55d91bc76390 2024-04-25T16:09:01.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.056+0000 7fe966ffd640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe95c0afd00 0x7fe95c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.057+0000 7fe966ffd640 1 -- 172.21.15.31:0/1051798499 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe9600ff550 con 0x7fe97010fef0 2024-04-25T16:09:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.060+0000 7fe975862640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc4d4d0 0x55d91c0181d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52544/0 (socket says 172.21.15.31:52544) 2024-04-25T16:09:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.060+0000 7fe975862640 1 -- 172.21.15.31:0/1283667471 learned_addr learned my addr 172.21.15.31:0/1283667471 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.060+0000 7fe975061640 1 --2- 172.21.15.31:0/1283667471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c017c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.060+0000 7fe975061640 1 --2- 172.21.15.31:0/1283667471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc556b0 0x55d91c018e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.061+0000 7fe976063640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe95c0afd00 0x7fe95c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.064+0000 7fe975862640 1 -- 172.21.15.31:0/1283667471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 msgr2=0x55d91c017c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:01.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.064+0000 7fe975862640 1 --2- 172.21.15.31:0/1283667471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c017c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.064+0000 7fe975862640 1 -- 172.21.15.31:0/1283667471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc556b0 msgr2=0x55d91c018e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:01.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.064+0000 7fe975862640 1 --2- 172.21.15.31:0/1283667471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc556b0 0x55d91c018e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.064+0000 7fe975862640 1 -- 172.21.15.31:0/1283667471 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d91bf07a40 con 0x55d91bc4d4d0 2024-04-25T16:09:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.065+0000 7fe975862640 1 --2- 172.21.15.31:0/1283667471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc4d4d0 0x55d91c0181d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009ac0 tx=0x7fe96c009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45019fc6473bce9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:01.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.066+0000 7fe95bfff640 1 -- 172.21.15.31:0/1283667471 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c017070 con 0x55d91bc4d4d0 2024-04-25T16:09:01.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.066+0000 7fe95bfff640 1 -- 172.21.15.31:0/1283667471 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe96c00eba0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.067+0000 7fe95bfff640 1 -- 172.21.15.31:0/1283667471 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c012630 con 0x55d91bc4d4d0 2024-04-25T16:09:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.068+0000 7fe97e977740 1 -- 172.21.15.31:0/1283667471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc4d4d0 msgr2=0x55d91c0181d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.068+0000 7fe97e977740 1 --2- 172.21.15.31:0/1283667471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc4d4d0 0x55d91c0181d0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fe96c009ac0 tx=0x7fe96c009f80 comp rx=0 tx=0).stop 2024-04-25T16:09:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.069+0000 7fe97e977740 1 -- 172.21.15.31:0/1283667471 shutdown_connections 2024-04-25T16:09:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.069+0000 7fe97e977740 1 --2- 172.21.15.31:0/1283667471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc556b0 0x55d91c018e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.069+0000 7fe97e977740 1 --2- 172.21.15.31:0/1283667471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc4d4d0 0x55d91c0181d0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.069+0000 7fe97e977740 1 --2- 172.21.15.31:0/1283667471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c017c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.069+0000 7fe97e977740 1 -- 172.21.15.31:0/1283667471 >> 172.21.15.31:0/1283667471 conn(0x55d91bc6a4e0 msgr2=0x55d91c013460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.070+0000 7fe97e977740 1 -- 172.21.15.31:0/1283667471 shutdown_connections 2024-04-25T16:09:01.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.070+0000 7fe97e977740 1 -- 172.21.15.31:0/1283667471 wait complete. 2024-04-25T16:09:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.071+0000 7fe97e977740 1 -- start start 2024-04-25T16:09:01.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.071+0000 7fe97e977740 1 Processor -- start 2024-04-25T16:09:01.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.072+0000 7fe97e977740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc4d4d0 0x55d91c021be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.072+0000 7fe97e977740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c022120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.073+0000 7fe975061640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc4d4d0 0x55d91c021be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.073+0000 7fe975061640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc4d4d0 0x55d91c021be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53678/0 (socket says 172.21.15.31:53678) 2024-04-25T16:09:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.073+0000 7fe975061640 1 -- 172.21.15.31:0/2327727156 learned_addr learned my addr 172.21.15.31:0/2327727156 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.073+0000 7fe97e977740 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc76390 0x55d91c029010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.073+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d91c0199c0 con 0x55d91bc556b0 2024-04-25T16:09:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.073+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d91bf03d10 con 0x55d91bc76390 2024-04-25T16:09:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.073+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d91c019b40 con 0x55d91bc4d4d0 2024-04-25T16:09:01.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.074+0000 7fe97e977740 10 client.0 _ll_register_callbacks cb 0x55d91bf30210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.078+0000 7fe975061640 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc76390 0x55d91c029010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.078+0000 7fe975061640 1 -- 172.21.15.31:0/2327727156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc76390 msgr2=0x55d91c029010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.078+0000 7fe975061640 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc76390 0x55d91c029010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.078+0000 7fe975061640 1 -- 172.21.15.31:0/2327727156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 msgr2=0x55d91c022120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.078+0000 7fe975061640 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c022120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.078+0000 7fe975061640 1 -- 172.21.15.31:0/2327727156 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d91bd587d0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.081+0000 7fe975061640 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc4d4d0 0x55d91c021be0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe968017f10 tx=0x7fe9680022b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=384f3b62d24b90ad server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:01.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.087+0000 7fe976063640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe95c0afd00 0x7fe95c0b2210 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe9600c39d0 tx=0x7fe96002a440 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:01.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.088+0000 7fe9597fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:01.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.088+0000 7fe9597fa640 1 -- 172.21.15.31:0/2327727156 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96801f060 con 0x55d91bc4d4d0 2024-04-25T16:09:01.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.088+0000 7fe9597fa640 1 -- 172.21.15.31:0/2327727156 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe96801a620 con 0x55d91bc4d4d0 2024-04-25T16:09:01.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.088+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe96c00eba0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.091+0000 7fe9597fa640 1 -- 172.21.15.31:0/2327727156 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe968010040 con 0x55d91bc4d4d0 2024-04-25T16:09:01.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.095+0000 7fe97e977740 10 client.26752 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:01.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.095+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d91bc8e3f0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.097+0000 7fe9597fa640 1 -- 172.21.15.31:0/2327727156 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe96c00eba0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.097+0000 7fe9597fa640 1 -- 172.21.15.31:0/2327727156 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe9680194d0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.097+0000 7fe97e977740 10 client.26752 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:01.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.097+0000 7fe97e977740 10 client.26752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:01.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.097+0000 7fe97e977740 20 client.26752 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:01.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.097+0000 7fe97e977740 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d91bc34f60 0x55d91c02fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.097+0000 7fe97e977740 4 client.26752 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:01.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.097+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d91bf8a990 con 0x55d91bc34f60 2024-04-25T16:09:01.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.099+0000 7fe975862640 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d91bc34f60 0x55d91c02fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.106+0000 7fe975862640 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d91bc34f60 0x55d91c02fa50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d70dd99443a4b1 server_cookie=f1e46a464e6c9623 in_seq=0 out_seq=0 2024-04-25T16:09:01.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.106+0000 7fe9597fa640 10 client.26752 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:01.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.108+0000 7fe9597fa640 1 -- 172.21.15.31:0/2327727156 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d91bf8a990 con 0x55d91bc34f60 2024-04-25T16:09:01.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.108+0000 7fe9597fa640 10 client.26752 handle_command_reply: tid=0 2024-04-25T16:09:01.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.108+0000 7fe97e977740 1 client.26752 shutdown 2024-04-25T16:09:01.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.109+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc4d4d0 msgr2=0x55d91c021be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.109+0000 7fe97e977740 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc4d4d0 0x55d91c021be0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe968017f10 tx=0x7fe9680022b0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.110+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 shutdown_connections 2024-04-25T16:09:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.110+0000 7fe97e977740 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d91bc34f60 0x55d91c02fa50 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.110+0000 7fe97e977740 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc76390 0x55d91c029010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.110+0000 7fe97e977740 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c022120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.110+0000 7fe97e977740 1 --2- 172.21.15.31:0/2327727156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc4d4d0 0x55d91c021be0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.110+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 >> 172.21.15.31:0/2327727156 conn(0x55d91bc6a4e0 msgr2=0x55d91c016370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:01.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.111+0000 7fe9597fa640 0 client.26752 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:01.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.111+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 shutdown_connections 2024-04-25T16:09:01.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.113+0000 7fe97e977740 1 -- 172.21.15.31:0/2327727156 wait complete. 2024-04-25T16:09:01.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.114+0000 7fe97e977740 20 client.26752 trim_cache size 0 max 16384 2024-04-25T16:09:01.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.196+0000 7fe97e977740 1 Processor -- start 2024-04-25T16:09:01.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.197+0000 7fe97e977740 1 -- start start 2024-04-25T16:09:01.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.198+0000 7fe97e977740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c020160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.198+0000 7fe97e977740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c0206a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.198+0000 7fe97e977740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c028fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.198+0000 7fe97e977740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9701c2940 con 0x55d91bc556b0 2024-04-25T16:09:01.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.198+0000 7fe97e977740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe9701c4480 con 0x55d91bc4d4d0 2024-04-25T16:09:01.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.199+0000 7fe97e977740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe9701c3010 con 0x55d91bc76390 2024-04-25T16:09:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.201+0000 7fe975862640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c0206a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.201+0000 7fe975862640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c0206a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52558/0 (socket says 172.21.15.31:52558) 2024-04-25T16:09:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.201+0000 7fe975862640 1 -- 172.21.15.31:0/387186434 learned_addr learned my addr 172.21.15.31:0/387186434 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:01.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.202+0000 7fe975061640 1 --2- 172.21.15.31:0/387186434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c020160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.202+0000 7fe975862640 1 -- 172.21.15.31:0/387186434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 msgr2=0x55d91c028fa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.202+0000 7fe975862640 1 --2- 172.21.15.31:0/387186434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c028fa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.202+0000 7fe975862640 1 -- 172.21.15.31:0/387186434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 msgr2=0x55d91c020160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.202+0000 7fe975862640 1 --2- 172.21.15.31:0/387186434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c020160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.202+0000 7fe975862640 1 -- 172.21.15.31:0/387186434 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d91bc4e0f0 con 0x55d91bc556b0 2024-04-25T16:09:01.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.209+0000 7fe975862640 1 --2- 172.21.15.31:0/387186434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c0206a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x55d91c017970 tx=0x7fe96c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=657abb3fa34d4f31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.210+0000 7fe95b7fe640 1 -- 172.21.15.31:0/387186434 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c017040 con 0x55d91bc556b0 2024-04-25T16:09:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.210+0000 7fe95b7fe640 1 -- 172.21.15.31:0/387186434 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe96c029070 con 0x55d91bc556b0 2024-04-25T16:09:01.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.210+0000 7fe95b7fe640 1 -- 172.21.15.31:0/387186434 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96c024630 con 0x55d91bc556b0 2024-04-25T16:09:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.212+0000 7fe97e977740 1 -- 172.21.15.31:0/387186434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 msgr2=0x55d91c0206a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.212+0000 7fe97e977740 1 --2- 172.21.15.31:0/387186434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c0206a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x55d91c017970 tx=0x7fe96c000bc0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.212+0000 7fe97e977740 1 -- 172.21.15.31:0/387186434 shutdown_connections 2024-04-25T16:09:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.212+0000 7fe97e977740 1 --2- 172.21.15.31:0/387186434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c028fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.212+0000 7fe97e977740 1 --2- 172.21.15.31:0/387186434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c0206a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.212+0000 7fe97e977740 1 --2- 172.21.15.31:0/387186434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c020160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.212+0000 7fe97e977740 1 -- 172.21.15.31:0/387186434 >> 172.21.15.31:0/387186434 conn(0x55d91bc6a4e0 msgr2=0x55d91c01c8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:01.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.220+0000 7fe97e977740 1 -- 172.21.15.31:0/387186434 shutdown_connections 2024-04-25T16:09:01.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.221+0000 7fe97e977740 1 -- 172.21.15.31:0/387186434 wait complete. 2024-04-25T16:09:01.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.222+0000 7fe97e977740 1 -- start start 2024-04-25T16:09:01.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.222+0000 7fe97e977740 1 Processor -- start 2024-04-25T16:09:01.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.223+0000 7fe97e977740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c0259f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.224+0000 7fe97e977740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c025f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.224+0000 7fe97e977740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c028e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.224+0000 7fe975061640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c0259f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.224+0000 7fe975061640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c0259f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:33152/0 (socket says 172.21.15.31:33152) 2024-04-25T16:09:01.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.224+0000 7fe975061640 1 -- 172.21.15.31:0/3508218324 learned_addr learned my addr 172.21.15.31:0/3508218324 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:01.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.225+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe9701c3db0 con 0x55d91bc556b0 2024-04-25T16:09:01.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.225+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe9701c36e0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.225+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe9701192d0 con 0x55d91bc76390 2024-04-25T16:09:01.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.225+0000 7fe97e977740 10 client.0 _ll_register_callbacks cb 0x7fe968001e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:01.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.226+0000 7fe975862640 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c025f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.228+0000 7fe975061640 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c028e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.228+0000 7fe975061640 1 -- 172.21.15.31:0/3508218324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 msgr2=0x55d91c028e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:01.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.228+0000 7fe975061640 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c028e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.228+0000 7fe975061640 1 -- 172.21.15.31:0/3508218324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 msgr2=0x55d91c025f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:01.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.228+0000 7fe975061640 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c025f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.228+0000 7fe975061640 1 -- 172.21.15.31:0/3508218324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d91bf3d4b0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.232+0000 7fe975061640 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c0259f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe968019a40 tx=0x7fe968001a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1725e7131633dde3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:01.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.233+0000 7fe9597fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:01.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.234+0000 7fe9597fa640 1 -- 172.21.15.31:0/3508218324 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe96802a070 con 0x55d91bc4d4d0 2024-04-25T16:09:01.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.239+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe96c029070 con 0x55d91bc4d4d0 2024-04-25T16:09:01.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.239+0000 7fe9597fa640 1 -- 172.21.15.31:0/3508218324 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe9680195c0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.241+0000 7fe9597fa640 1 -- 172.21.15.31:0/3508218324 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe96801f020 con 0x55d91bc4d4d0 2024-04-25T16:09:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.243+0000 7fe97e977740 10 client.26786 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:01.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.243+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d91c0965f0 con 0x55d91bc4d4d0 2024-04-25T16:09:01.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.245+0000 7fe9597fa640 1 -- 172.21.15.31:0/3508218324 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe96c029070 con 0x55d91bc4d4d0 2024-04-25T16:09:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.246+0000 7fe9597fa640 1 -- 172.21.15.31:0/3508218324 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe96801a420 con 0x55d91bc4d4d0 2024-04-25T16:09:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.247+0000 7fe97e977740 10 client.26786 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:01.249 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.247+0000 7fe97e977740 10 client.26786 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.247+0000 7fe97e977740 20 client.26786 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.247+0000 7fe97e977740 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d91bc0a3a0 0x55d91c098f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.247+0000 7fe97e977740 4 client.26786 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:09:01.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.247+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d91bf84800 con 0x55d91bc0a3a0 2024-04-25T16:09:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.248+0000 7fe975862640 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d91bc0a3a0 0x55d91c098f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.249+0000 7fe975862640 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d91bc0a3a0 0x55d91c098f90 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd1cc7b564866a23 server_cookie=e7659b4561d6f06f in_seq=0 out_seq=0 2024-04-25T16:09:01.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.251+0000 7fe9597fa640 10 client.26786 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:01.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.252+0000 7fe9597fa640 1 -- 172.21.15.31:0/3508218324 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d91bf84800 con 0x55d91bc0a3a0 2024-04-25T16:09:01.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.252+0000 7fe9597fa640 10 client.26786 handle_command_reply: tid=0 2024-04-25T16:09:01.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.252+0000 7fe97e977740 1 client.26786 shutdown 2024-04-25T16:09:01.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.253+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 msgr2=0x55d91c0259f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe97e977740 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c0259f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe968019a40 tx=0x7fe968001a00 comp rx=0 tx=0).stop 2024-04-25T16:09:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 shutdown_connections 2024-04-25T16:09:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe97e977740 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d91bc0a3a0 0x55d91c098f90 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe9597fa640 0 client.26786 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe97e977740 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d91bc76390 0x55d91c028e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe97e977740 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d91bc556b0 0x55d91c025f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe97e977740 1 --2- 172.21.15.31:0/3508218324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d91bc4d4d0 0x55d91c0259f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 >> 172.21.15.31:0/3508218324 conn(0x55d91bc6a4e0 msgr2=0x55d91c020160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 shutdown_connections 2024-04-25T16:09:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.254+0000 7fe97e977740 1 -- 172.21.15.31:0/3508218324 wait complete. 2024-04-25T16:09:01.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.255+0000 7fe97e977740 20 client.26786 trim_cache size 0 max 16384 2024-04-25T16:09:01.257 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:09:01.257 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:09:01.257 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:09:01.257 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:09:01.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.262+0000 7fe977ac6640 1 -- 172.21.15.31:0/1051798499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe95c0afd00 msgr2=0x7fe95c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.262+0000 7fe977ac6640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe95c0afd00 0x7fe95c0b2210 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fe9600c39d0 tx=0x7fe96002a440 comp rx=0 tx=0).stop 2024-04-25T16:09:01.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.262+0000 7fe977ac6640 1 -- 172.21.15.31:0/1051798499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97010fef0 msgr2=0x7fe9701b0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.262+0000 7fe977ac6640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97010fef0 0x7fe9701b0460 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe960009230 tx=0x7fe960003cd0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.267+0000 7fe977ac6640 1 -- 172.21.15.31:0/1051798499 shutdown_connections 2024-04-25T16:09:01.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.267+0000 7fe977ac6640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe95c0afd00 0x7fe95c0b2210 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.267+0000 7fe977ac6640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe97010fef0 0x7fe9701b0460 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.267+0000 7fe977ac6640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe97010b1d0 0x7fe9701a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.267+0000 7fe977ac6640 1 --2- 172.21.15.31:0/1051798499 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe97010a3e0 0x7fe9701a8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.267+0000 7fe977ac6640 1 -- 172.21.15.31:0/1051798499 >> 172.21.15.31:0/1051798499 conn(0x7fe970070400 msgr2=0x7fe97010bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:01.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.272+0000 7fe977ac6640 1 -- 172.21.15.31:0/1051798499 shutdown_connections 2024-04-25T16:09:01.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.277+0000 7fe977ac6640 1 -- 172.21.15.31:0/1051798499 wait complete. 2024-04-25T16:09:01.394 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:09:01.394 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:09:01.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.814+0000 7fea5359e640 1 -- 172.21.15.31:0/3523571855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0abb70 msgr2=0x7fea4c0adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.814+0000 7fea5359e640 1 --2- 172.21.15.31:0/3523571855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0abb70 0x7fea4c0adf90 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fea54050a10 tx=0x7fea54090fe0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.820+0000 7fea5359e640 1 -- 172.21.15.31:0/3523571855 shutdown_connections 2024-04-25T16:09:01.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.820+0000 7fea5359e640 1 --2- 172.21.15.31:0/3523571855 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0abb70 0x7fea4c0adf90 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.820+0000 7fea5359e640 1 --2- 172.21.15.31:0/3523571855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea4c0a6e70 0x7fea4c0a7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.820+0000 7fea5359e640 1 --2- 172.21.15.31:0/3523571855 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c0a5c30 0x7fea4c0a6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.820+0000 7fea5359e640 1 -- 172.21.15.31:0/3523571855 >> 172.21.15.31:0/3523571855 conn(0x7fea4c0a09b0 msgr2=0x7fea4c0a2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:01.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.821+0000 7fea5359e640 1 -- 172.21.15.31:0/3523571855 shutdown_connections 2024-04-25T16:09:01.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.821+0000 7fea5359e640 1 -- 172.21.15.31:0/3523571855 wait complete. 2024-04-25T16:09:01.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.824+0000 7fea5359e640 1 Processor -- start 2024-04-25T16:09:01.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.824+0000 7fea5359e640 1 -- start start 2024-04-25T16:09:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.825+0000 7fea5359e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0a5c30 0x7fea4c13be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.825+0000 7fea5359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea4c0a6e70 0x7fea4c13c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.826+0000 7fea5259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0a5c30 0x7fea4c13be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.826+0000 7fea5259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0a5c30 0x7fea4c13be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52576/0 (socket says 172.21.15.31:52576) 2024-04-25T16:09:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.826+0000 7fea5259c640 1 -- 172.21.15.31:0/2568715551 learned_addr learned my addr 172.21.15.31:0/2568715551 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:01.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.826+0000 7fea5359e640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c0abb70 0x7fea4c143340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.826+0000 7fea51d9b640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea4c0a6e70 0x7fea4c13c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.826+0000 7fea5359e640 1 -- 172.21.15.31:0/2568715551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c0b12a0 con 0x7fea4c0a5c30 2024-04-25T16:09:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.827+0000 7fea5359e640 1 -- 172.21.15.31:0/2568715551 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c0b1120 con 0x7fea4c0abb70 2024-04-25T16:09:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.827+0000 7fea5359e640 1 -- 172.21.15.31:0/2568715551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea4c0b1420 con 0x7fea4c0a6e70 2024-04-25T16:09:01.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.827+0000 7fea52d9d640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c0abb70 0x7fea4c143340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.828+0000 7fea5259c640 1 -- 172.21.15.31:0/2568715551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea4c0a6e70 msgr2=0x7fea4c13c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.828+0000 7fea5259c640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea4c0a6e70 0x7fea4c13c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.828+0000 7fea5259c640 1 -- 172.21.15.31:0/2568715551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c0abb70 msgr2=0x7fea4c143340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:01.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.828+0000 7fea5259c640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c0abb70 0x7fea4c143340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:01.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.828+0000 7fea5259c640 1 -- 172.21.15.31:0/2568715551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea4c1439b0 con 0x7fea4c0a5c30 2024-04-25T16:09:01.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.829+0000 7fea5259c640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0a5c30 0x7fea4c13be70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fea4800b450 tx=0x7fea4800b850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.833+0000 7fea437fe640 1 -- 172.21.15.31:0/2568715551 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea4801d070 con 0x7fea4c0a5c30 2024-04-25T16:09:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.833+0000 7fea437fe640 1 -- 172.21.15.31:0/2568715551 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea48018440 con 0x7fea4c0a5c30 2024-04-25T16:09:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.833+0000 7fea437fe640 1 -- 172.21.15.31:0/2568715551 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea48016580 con 0x7fea4c0a5c30 2024-04-25T16:09:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.835+0000 7fea5359e640 1 -- 172.21.15.31:0/2568715551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea4c143ca0 con 0x7fea4c0a5c30 2024-04-25T16:09:01.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.835+0000 7fea5359e640 1 -- 172.21.15.31:0/2568715551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea4c144110 con 0x7fea4c0a5c30 2024-04-25T16:09:01.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.838+0000 7fea417fa640 1 -- 172.21.15.31:0/2568715551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea4c0035e0 con 0x7fea4c0a5c30 2024-04-25T16:09:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.846+0000 7fea437fe640 1 -- 172.21.15.31:0/2568715551 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fea480185f0 con 0x7fea4c0a5c30 2024-04-25T16:09:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.847+0000 7fea437fe640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fea3c0afd00 0x7fea3c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:01.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.847+0000 7fea437fe640 1 -- 172.21.15.31:0/2568715551 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fea480e2310 con 0x7fea4c0a5c30 2024-04-25T16:09:01.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.850+0000 7fea51d9b640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fea3c0afd00 0x7fea3c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:01.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.855+0000 7fea51d9b640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fea3c0afd00 0x7fea3c0b2210 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fea4c13cf10 tx=0x7fea4400c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:01.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:01.856+0000 7fea437fe640 1 -- 172.21.15.31:0/2568715551 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea480aa8d0 con 0x7fea4c0a5c30 2024-04-25T16:09:02.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:02 smithi063 ceph-mon[32413]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 747 MiB/s wr, 1.77k op/s 2024-04-25T16:09:02.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:02 smithi186 ceph-mon[33258]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 747 MiB/s wr, 1.77k op/s 2024-04-25T16:09:02.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:02 smithi031 ceph-mon[29140]: pgmap v583: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 747 MiB/s wr, 1.77k op/s 2024-04-25T16:09:02.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.628+0000 7fea417fa640 1 -- 172.21.15.31:0/2568715551 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea4c003890 con 0x7fea4c0a5c30 2024-04-25T16:09:02.639 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:09:02.639 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:09:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.633+0000 7fea437fe640 1 -- 172.21.15.31:0/2568715551 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7fea480aebd0 con 0x7fea4c0a5c30 2024-04-25T16:09:02.640 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 -- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fea3c0afd00 msgr2=0x7fea3c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fea3c0afd00 0x7fea3c0b2210 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fea4c13cf10 tx=0x7fea4400c040 comp rx=0 tx=0).stop 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 -- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0a5c30 msgr2=0x7fea4c13be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0a5c30 0x7fea4c13be70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fea4800b450 tx=0x7fea4800b850 comp rx=0 tx=0).stop 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 -- 172.21.15.31:0/2568715551 shutdown_connections 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fea3c0afd00 0x7fea3c0b2210 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fea4c0abb70 0x7fea4c143340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea4c0a6e70 0x7fea4c13c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 --2- 172.21.15.31:0/2568715551 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fea4c0a5c30 0x7fea4c13be70 secure :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fea4800b450 tx=0x7fea4800b850 comp rx=0 tx=0).stop 2024-04-25T16:09:02.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.641+0000 7fea417fa640 1 -- 172.21.15.31:0/2568715551 >> 172.21.15.31:0/2568715551 conn(0x7fea4c0a09b0 msgr2=0x7fea4c0a7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:02.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.646+0000 7fea417fa640 1 -- 172.21.15.31:0/2568715551 shutdown_connections 2024-04-25T16:09:02.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:02.646+0000 7fea417fa640 1 -- 172.21.15.31:0/2568715551 wait complete. 2024-04-25T16:09:02.752 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:09:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.146+0000 7f62d35ce640 1 -- 172.21.15.31:0/1489674313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62cc10b310 msgr2=0x7f62cc10b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:03.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.146+0000 7f62d35ce640 1 --2- 172.21.15.31:0/1489674313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62cc10b310 0x7f62cc10b720 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f62c401acb0 tx=0x7f62c403b920 comp rx=0 tx=0).stop 2024-04-25T16:09:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.146+0000 7f62d35ce640 1 -- 172.21.15.31:0/1489674313 shutdown_connections 2024-04-25T16:09:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.146+0000 7f62d35ce640 1 --2- 172.21.15.31:0/1489674313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62cc110030 0x7f62cc110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.146+0000 7f62d35ce640 1 --2- 172.21.15.31:0/1489674313 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62cc10b310 0x7f62cc10b720 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.146+0000 7f62d35ce640 1 --2- 172.21.15.31:0/1489674313 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62cc10a420 0x7f62cc10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.146+0000 7f62d35ce640 1 -- 172.21.15.31:0/1489674313 >> 172.21.15.31:0/1489674313 conn(0x7f62cc0704e0 msgr2=0x7f62cc072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:03.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.148+0000 7f62d35ce640 1 -- 172.21.15.31:0/1489674313 shutdown_connections 2024-04-25T16:09:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.149+0000 7f62d35ce640 1 -- 172.21.15.31:0/1489674313 wait complete. 2024-04-25T16:09:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.150+0000 7f62d35ce640 1 Processor -- start 2024-04-25T16:09:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.150+0000 7f62d35ce640 1 -- start start 2024-04-25T16:09:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.150+0000 7f62d35ce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62cc10a420 0x7f62cc1a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.150+0000 7f62d35ce640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62cc10b310 0x7f62cc1a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.150+0000 7f62d35ce640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62cc110030 0x7f62cc1a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.150+0000 7f62d35ce640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc113cc0 con 0x7f62cc110030 2024-04-25T16:09:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.150+0000 7f62d35ce640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc113b40 con 0x7f62cc10b310 2024-04-25T16:09:03.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.150+0000 7f62d35ce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62cc113e40 con 0x7f62cc10a420 2024-04-25T16:09:03.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.154+0000 7f62d136a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62cc10a420 0x7f62cc1a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.155+0000 7f62d136a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62cc10a420 0x7f62cc1a08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53722/0 (socket says 172.21.15.31:53722) 2024-04-25T16:09:03.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.155+0000 7f62d136a640 1 -- 172.21.15.31:0/4237444397 learned_addr learned my addr 172.21.15.31:0/4237444397 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:03.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.156+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62cc110030 0x7f62cc1a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.156+0000 7f62d0b69640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62cc10b310 0x7f62cc1a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.158+0000 7f62d136a640 1 -- 172.21.15.31:0/4237444397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62cc10b310 msgr2=0x7f62cc1a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.158+0000 7f62d136a640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62cc10b310 0x7f62cc1a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.158+0000 7f62d136a640 1 -- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62cc110030 msgr2=0x7f62cc1a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.158+0000 7f62d136a640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62cc110030 0x7f62cc1a7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.158+0000 7f62d136a640 1 -- 172.21.15.31:0/4237444397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62cc1a8410 con 0x7f62cc10a420 2024-04-25T16:09:03.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.167+0000 7f62d136a640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62cc10a420 0x7f62cc1a08d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f62c800c970 tx=0x7f62c800ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.167+0000 7f62c27fc640 1 -- 172.21.15.31:0/4237444397 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c8007bb0 con 0x7f62cc10a420 2024-04-25T16:09:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.168+0000 7f62c27fc640 1 -- 172.21.15.31:0/4237444397 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c8007d50 con 0x7f62cc10a420 2024-04-25T16:09:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.168+0000 7f62c27fc640 1 -- 172.21.15.31:0/4237444397 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c800f450 con 0x7f62cc10a420 2024-04-25T16:09:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.168+0000 7f62d35ce640 1 -- 172.21.15.31:0/4237444397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62cc1a8700 con 0x7f62cc10a420 2024-04-25T16:09:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.169+0000 7f62d35ce640 1 -- 172.21.15.31:0/4237444397 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62cc1a8c00 con 0x7f62cc10a420 2024-04-25T16:09:03.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.173+0000 7f62da47e740 1 Processor -- start 2024-04-25T16:09:03.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.173+0000 7f62da47e740 1 -- start start 2024-04-25T16:09:03.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.174+0000 7f62da47e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5a2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.182+0000 7f62c27fc640 1 -- 172.21.15.31:0/4237444397 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f62c8004f20 con 0x7f62cc10a420 2024-04-25T16:09:03.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.183+0000 7f62c27fc640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62b80afd00 0x7f62b80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.184+0000 7f62c27fc640 1 -- 172.21.15.31:0/4237444397 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f62c80debc0 con 0x7f62cc10a420 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62d1b6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62da47e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62da47e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62da47e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d9ee506d80 con 0x55d9ee200390 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62da47e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d9ee4c0a20 con 0x55d9ee1d74d0 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62da47e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d9ee502d40 con 0x55d9ee1df6b0 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62d0b69640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62d1b6b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62d1b6b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5a3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53734/0 (socket says 172.21.15.31:53734) 2024-04-25T16:09:03.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.185+0000 7f62d1b6b640 1 -- 172.21.15.31:0/3419236866 learned_addr learned my addr 172.21.15.31:0/3419236866 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.186+0000 7f62d1b6b640 1 -- 172.21.15.31:0/3419236866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 msgr2=0x55d9ee5a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.186+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/3419236866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.186+0000 7f62d1b6b640 1 -- 172.21.15.31:0/3419236866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 msgr2=0x55d9ee5a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.186+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/3419236866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5a2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.186+0000 7f62d1b6b640 1 -- 172.21.15.31:0/3419236866 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9ee4cbc30 con 0x55d9ee1df6b0 2024-04-25T16:09:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.187+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/3419236866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5a3a20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f62bc005fd0 tx=0x7f62bc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4cc1ca575ac631b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.187+0000 7f62b77fe640 1 -- 172.21.15.31:0/3419236866 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc012650 con 0x55d9ee1df6b0 2024-04-25T16:09:03.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.187+0000 7f62b77fe640 1 -- 172.21.15.31:0/3419236866 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62bc016070 con 0x55d9ee1df6b0 2024-04-25T16:09:03.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.187+0000 7f62b77fe640 1 -- 172.21.15.31:0/3419236866 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc00eb50 con 0x55d9ee1df6b0 2024-04-25T16:09:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.189+0000 7f62da47e740 1 -- 172.21.15.31:0/3419236866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 msgr2=0x55d9ee5a3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.189+0000 7f62da47e740 1 --2- 172.21.15.31:0/3419236866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5a3a20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f62bc005fd0 tx=0x7f62bc004730 comp rx=0 tx=0).stop 2024-04-25T16:09:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.189+0000 7f62d136a640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62b80afd00 0x7f62b80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.189+0000 7f62da47e740 1 -- 172.21.15.31:0/3419236866 shutdown_connections 2024-04-25T16:09:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.189+0000 7f62da47e740 1 --2- 172.21.15.31:0/3419236866 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5a3a20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.189+0000 7f62da47e740 1 --2- 172.21.15.31:0/3419236866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.189+0000 7f62da47e740 1 --2- 172.21.15.31:0/3419236866 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.189+0000 7f62da47e740 1 -- 172.21.15.31:0/3419236866 >> 172.21.15.31:0/3419236866 conn(0x55d9ee1f44e0 msgr2=0x55d9ee59e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:03.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.193+0000 7f62da47e740 1 -- 172.21.15.31:0/3419236866 shutdown_connections 2024-04-25T16:09:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.196+0000 7f62da47e740 1 -- 172.21.15.31:0/3419236866 wait complete. 2024-04-25T16:09:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.196+0000 7f62da47e740 1 -- start start 2024-04-25T16:09:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.197+0000 7f62da47e740 1 Processor -- start 2024-04-25T16:09:03.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.197+0000 7f62da47e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5ac460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.198+0000 7f62da47e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5ac9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.198+0000 7f62da47e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee200390 0x55d9ee5b3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.198+0000 7f62da47e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d9ee5a4530 con 0x55d9ee1d74d0 2024-04-25T16:09:03.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.198+0000 7f62da47e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d9ee48ddc0 con 0x55d9ee1df6b0 2024-04-25T16:09:03.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.198+0000 7f62da47e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d9ee5a46b0 con 0x55d9ee200390 2024-04-25T16:09:03.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.198+0000 7f62da47e740 10 client.0 _ll_register_callbacks cb 0x55d9ee4a9600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:03.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.200+0000 7f62d136a640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62b80afd00 0x7f62b80b2210 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f62c800afd0 tx=0x7f62c800cf60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:03.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.205+0000 7f62d0b69640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5ac9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.205+0000 7f62d1b6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5ac460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.205+0000 7f62d1b6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5ac460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52614/0 (socket says 172.21.15.31:52614) 2024-04-25T16:09:03.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.205+0000 7f62d1b6b640 1 -- 172.21.15.31:0/1611091791 learned_addr learned my addr 172.21.15.31:0/1611091791 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:03.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.205+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee200390 0x55d9ee5b3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.211+0000 7f62d0b69640 1 -- 172.21.15.31:0/1611091791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee200390 msgr2=0x55d9ee5b3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:03.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.211+0000 7f62d0b69640 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee200390 0x55d9ee5b3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.211+0000 7f62d0b69640 1 -- 172.21.15.31:0/1611091791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee1d74d0 msgr2=0x55d9ee5ac460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:03.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.211+0000 7f62d0b69640 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5ac460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.211+0000 7f62d0b69640 1 -- 172.21.15.31:0/1611091791 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9ee2183f0 con 0x55d9ee1df6b0 2024-04-25T16:09:03.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.214+0000 7f62d0b69640 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5ac9a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62cc1a1970 tx=0x7f62c401aa20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad3a0e5dc824be67 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:03.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.260+0000 7f62b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:03.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.260+0000 7f62b4ff9640 1 -- 172.21.15.31:0/1611091791 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4045040 con 0x55d9ee1df6b0 2024-04-25T16:09:03.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.261+0000 7f62da47e740 1 -- 172.21.15.31:0/1611091791 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62bc016070 con 0x55d9ee1df6b0 2024-04-25T16:09:03.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.261+0000 7f62b4ff9640 1 -- 172.21.15.31:0/1611091791 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f62c4019040 con 0x55d9ee1df6b0 2024-04-25T16:09:03.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.263+0000 7f62b4ff9640 1 -- 172.21.15.31:0/1611091791 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62c4041830 con 0x55d9ee1df6b0 2024-04-25T16:09:03.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.264+0000 7f62da47e740 10 client.26798 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:03.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.264+0000 7f62da47e740 1 -- 172.21.15.31:0/1611091791 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d9ee2e27d0 con 0x55d9ee1df6b0 2024-04-25T16:09:03.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.270+0000 7f62b4ff9640 1 -- 172.21.15.31:0/1611091791 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62bc016070 con 0x55d9ee1df6b0 2024-04-25T16:09:03.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.272+0000 7f62b4ff9640 1 -- 172.21.15.31:0/1611091791 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f62c404bc40 con 0x55d9ee1df6b0 2024-04-25T16:09:03.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.272+0000 7f62da47e740 10 client.26798 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:03.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.272+0000 7f62da47e740 10 client.26798 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:09:03.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.272+0000 7f62da47e740 20 client.26798 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:03.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.272+0000 7f62da47e740 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d9ee1bef60 0x55d9ee5ba600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.272+0000 7f62da47e740 4 client.26798 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:03.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.272+0000 7f62da47e740 1 -- 172.21.15.31:0/1611091791 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d9ee514990 con 0x55d9ee1bef60 2024-04-25T16:09:03.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.274+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d9ee1bef60 0x55d9ee5ba600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.279+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d9ee1bef60 0x55d9ee5ba600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3813bb724ea0836b server_cookie=416a98e7750fb178 in_seq=0 out_seq=0 2024-04-25T16:09:03.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.279+0000 7f62b4ff9640 10 client.26798 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:09:03.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.284+0000 7f62b4ff9640 1 -- 172.21.15.31:0/1611091791 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d9ee514990 con 0x55d9ee1bef60 2024-04-25T16:09:03.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.284+0000 7f62b4ff9640 10 client.26798 handle_command_reply: tid=0 2024-04-25T16:09:03.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.284+0000 7f62da47e740 1 client.26798 shutdown 2024-04-25T16:09:03.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.285+0000 7f62da47e740 1 -- 172.21.15.31:0/1611091791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1df6b0 msgr2=0x55d9ee5ac9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:03.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.286+0000 7f62da47e740 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5ac9a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f62cc1a1970 tx=0x7f62c401aa20 comp rx=0 tx=0).stop 2024-04-25T16:09:03.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.286+0000 7f62da47e740 1 -- 172.21.15.31:0/1611091791 shutdown_connections 2024-04-25T16:09:03.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.286+0000 7f62da47e740 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d9ee1bef60 0x55d9ee5ba600 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.286+0000 7f62da47e740 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee200390 0x55d9ee5b3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.286+0000 7f62da47e740 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5ac9a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.286+0000 7f62da47e740 1 --2- 172.21.15.31:0/1611091791 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5ac460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.286+0000 7f62da47e740 1 -- 172.21.15.31:0/1611091791 >> 172.21.15.31:0/1611091791 conn(0x55d9ee1f44e0 msgr2=0x55d9ee5a0f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:03.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.286+0000 7f62b4ff9640 0 client.26798 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:09:03.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.287+0000 7f62da47e740 1 -- 172.21.15.31:0/1611091791 shutdown_connections 2024-04-25T16:09:03.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.288+0000 7f62da47e740 1 -- 172.21.15.31:0/1611091791 wait complete. 2024-04-25T16:09:03.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.288+0000 7f62da47e740 20 client.26798 trim_cache size 0 max 16384 2024-04-25T16:09:03.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.345+0000 7f62da47e740 1 Processor -- start 2024-04-25T16:09:03.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.345+0000 7f62da47e740 1 -- start start 2024-04-25T16:09:03.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.346+0000 7f62da47e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5aad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.346+0000 7f62da47e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.346+0000 7f62da47e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5a7120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.347+0000 7f62d1b6b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5aad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.347+0000 7f62d0b69640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.347+0000 7f62d0b69640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53738/0 (socket says 172.21.15.31:53738) 2024-04-25T16:09:03.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.347+0000 7f62d1b6b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5aad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:33198/0 (socket says 172.21.15.31:33198) 2024-04-25T16:09:03.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.347+0000 7f62d0b69640 1 -- 172.21.15.31:0/624265337 learned_addr learned my addr 172.21.15.31:0/624265337 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:03.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.348+0000 7f62da47e740 1 -- 172.21.15.31:0/624265337 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f62b000e3c0 con 0x55d9ee200390 2024-04-25T16:09:03.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.348+0000 7f62da47e740 1 -- 172.21.15.31:0/624265337 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f62b000ff00 con 0x55d9ee1d74d0 2024-04-25T16:09:03.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.348+0000 7f62da47e740 1 -- 172.21.15.31:0/624265337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62b000ea90 con 0x55d9ee1df6b0 2024-04-25T16:09:03.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.352+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/624265337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5a7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.360+0000 7f62d0b69640 1 -- 172.21.15.31:0/624265337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 msgr2=0x55d9ee5aad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.360+0000 7f62d0b69640 1 --2- 172.21.15.31:0/624265337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5aad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.360+0000 7f62d0b69640 1 -- 172.21.15.31:0/624265337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 msgr2=0x55d9ee5a7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.360+0000 7f62d0b69640 1 --2- 172.21.15.31:0/624265337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5a7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.360+0000 7f62d0b69640 1 -- 172.21.15.31:0/624265337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d9ee5ac210 con 0x55d9ee1df6b0 2024-04-25T16:09:03.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.365+0000 7f62d0b69640 1 --2- 172.21.15.31:0/624265337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b3820 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f62c401a4f0 tx=0x7f62c4000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad8fcae2c0654c31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.367+0000 7f62b6ffd640 1 -- 172.21.15.31:0/624265337 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c4045070 con 0x55d9ee1df6b0 2024-04-25T16:09:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.367+0000 7f62b6ffd640 1 -- 172.21.15.31:0/624265337 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62c4064020 con 0x55d9ee1df6b0 2024-04-25T16:09:03.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.368+0000 7f62b6ffd640 1 -- 172.21.15.31:0/624265337 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62c404a9b0 con 0x55d9ee1df6b0 2024-04-25T16:09:03.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:03 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2568715551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:09:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.369+0000 7f62da47e740 1 -- 172.21.15.31:0/624265337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 msgr2=0x55d9ee5b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:03.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.369+0000 7f62da47e740 1 --2- 172.21.15.31:0/624265337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b3820 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f62c401a4f0 tx=0x7f62c4000f40 comp rx=0 tx=0).stop 2024-04-25T16:09:03.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.373+0000 7f62da47e740 1 -- 172.21.15.31:0/624265337 shutdown_connections 2024-04-25T16:09:03.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.373+0000 7f62da47e740 1 --2- 172.21.15.31:0/624265337 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5a7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.373+0000 7f62da47e740 1 --2- 172.21.15.31:0/624265337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b3820 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.373+0000 7f62da47e740 1 --2- 172.21.15.31:0/624265337 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5aad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.373+0000 7f62da47e740 1 -- 172.21.15.31:0/624265337 >> 172.21.15.31:0/624265337 conn(0x55d9ee1f44e0 msgr2=0x55d9ee5a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.373+0000 7f62da47e740 1 -- 172.21.15.31:0/624265337 shutdown_connections 2024-04-25T16:09:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.373+0000 7f62da47e740 1 -- 172.21.15.31:0/624265337 wait complete. 2024-04-25T16:09:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.374+0000 7f62da47e740 1 -- start start 2024-04-25T16:09:03.375 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.374+0000 7f62da47e740 1 Processor -- start 2024-04-25T16:09:03.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.376+0000 7f62da47e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.376+0000 7f62da47e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.376+0000 7f62da47e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.377+0000 7f62da47e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f62b000f830 con 0x55d9ee200390 2024-04-25T16:09:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.377+0000 7f62da47e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f62b000f160 con 0x55d9ee1d74d0 2024-04-25T16:09:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.377+0000 7f62da47e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f62b000c450 con 0x55d9ee1df6b0 2024-04-25T16:09:03.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.377+0000 7f62da47e740 10 client.0 _ll_register_callbacks cb 0x7f62c4040d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.378+0000 7f62d1b6b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.378+0000 7f62d1b6b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5b36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:33210/0 (socket says 172.21.15.31:33210) 2024-04-25T16:09:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.378+0000 7f62d1b6b640 1 -- 172.21.15.31:0/2885081308 learned_addr learned my addr 172.21.15.31:0/2885081308 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:03.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.380+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.381+0000 7f62d0b69640 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.383+0000 7f62d1b6b640 1 -- 172.21.15.31:0/2885081308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 msgr2=0x55d9ee5b0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.383+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.383+0000 7f62d1b6b640 1 -- 172.21.15.31:0/2885081308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 msgr2=0x55d9ee5b0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.383+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5b0760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.383+0000 7f62d1b6b640 1 -- 172.21.15.31:0/2885081308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d9ee6209a0 con 0x55d9ee1d74d0 2024-04-25T16:09:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.387+0000 7f62d1b6b640 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5b36a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62bc010010 tx=0x7f62c4040cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e12fd7389301072 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:03.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.387+0000 7f62b4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.388+0000 7f62b4ff9640 1 -- 172.21.15.31:0/2885081308 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62bc022040 con 0x55d9ee1d74d0 2024-04-25T16:09:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.388+0000 7f62da47e740 1 -- 172.21.15.31:0/2885081308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62c4064020 con 0x55d9ee1d74d0 2024-04-25T16:09:03.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.388+0000 7f62b4ff9640 1 -- 172.21.15.31:0/2885081308 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f62bc012650 con 0x55d9ee1d74d0 2024-04-25T16:09:03.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.392+0000 7f62b4ff9640 1 -- 172.21.15.31:0/2885081308 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f62bc00ee50 con 0x55d9ee1d74d0 2024-04-25T16:09:03.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.393+0000 7f62da47e740 10 client.26804 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:03.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.393+0000 7f62da47e740 1 -- 172.21.15.31:0/2885081308 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d9ee621ad0 con 0x55d9ee1d74d0 2024-04-25T16:09:03.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:03 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2568715551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:09:03.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.400+0000 7f62b4ff9640 1 -- 172.21.15.31:0/2885081308 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62bc016050 con 0x55d9ee1d74d0 2024-04-25T16:09:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.402+0000 7f62b4ff9640 1 -- 172.21.15.31:0/2885081308 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f62bc02b890 con 0x55d9ee1d74d0 2024-04-25T16:09:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.402+0000 7f62da47e740 10 client.26804 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.402+0000 7f62da47e740 10 client.26804 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:09:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.402+0000 7f62da47e740 20 client.26804 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.402+0000 7f62da47e740 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d9ee1943a0 0x55d9ee624370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.402+0000 7f62da47e740 4 client.26804 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:03.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.402+0000 7f62da47e740 1 -- 172.21.15.31:0/2885081308 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f62c4064020 con 0x55d9ee1943a0 2024-04-25T16:09:03.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.405+0000 7f62d0b69640 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d9ee1943a0 0x55d9ee624370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:03.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.414+0000 7f62d0b69640 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d9ee1943a0 0x55d9ee624370 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a8868cd710f97534 server_cookie=f214fae5b677c14a in_seq=0 out_seq=0 2024-04-25T16:09:03.417 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.414+0000 7f62b4ff9640 10 client.26804 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:09:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.416+0000 7f62b4ff9640 1 -- 172.21.15.31:0/2885081308 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f62c4064020 con 0x55d9ee1943a0 2024-04-25T16:09:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.416+0000 7f62b4ff9640 10 client.26804 handle_command_reply: tid=0 2024-04-25T16:09:03.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.416+0000 7f62da47e740 1 client.26804 shutdown 2024-04-25T16:09:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.422+0000 7f62da47e740 1 -- 172.21.15.31:0/2885081308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 msgr2=0x55d9ee5b36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:03.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.422+0000 7f62da47e740 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5b36a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f62bc010010 tx=0x7f62c4040cc0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.428+0000 7f62da47e740 1 -- 172.21.15.31:0/2885081308 shutdown_connections 2024-04-25T16:09:03.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.428+0000 7f62da47e740 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d9ee1943a0 0x55d9ee624370 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.428+0000 7f62da47e740 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d9ee200390 0x55d9ee5b0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.428+0000 7f62da47e740 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d9ee1df6b0 0x55d9ee5b0220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.428+0000 7f62da47e740 1 --2- 172.21.15.31:0/2885081308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d9ee1d74d0 0x55d9ee5b36a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.428+0000 7f62da47e740 1 -- 172.21.15.31:0/2885081308 >> 172.21.15.31:0/2885081308 conn(0x55d9ee1f44e0 msgr2=0x55d9ee5a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:03.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.429+0000 7f62b4ff9640 0 client.26804 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:09:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.436+0000 7f62da47e740 1 -- 172.21.15.31:0/2885081308 shutdown_connections 2024-04-25T16:09:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.436+0000 7f62da47e740 1 -- 172.21.15.31:0/2885081308 wait complete. 2024-04-25T16:09:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.436+0000 7f62da47e740 20 client.26804 trim_cache size 0 max 16384 2024-04-25T16:09:03.442 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:03.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.449+0000 7f62d35ce640 1 -- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62b80afd00 msgr2=0x7f62b80b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:03.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.449+0000 7f62d35ce640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62b80afd00 0x7f62b80b2210 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f62c800afd0 tx=0x7f62c800cf60 comp rx=0 tx=0).stop 2024-04-25T16:09:03.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:03 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2568715551' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:09:03.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.449+0000 7f62d35ce640 1 -- 172.21.15.31:0/4237444397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62cc10a420 msgr2=0x7f62cc1a08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:03.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.449+0000 7f62d35ce640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62cc10a420 0x7f62cc1a08d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f62c800c970 tx=0x7f62c800ce30 comp rx=0 tx=0).stop 2024-04-25T16:09:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.452+0000 7f62d35ce640 1 -- 172.21.15.31:0/4237444397 shutdown_connections 2024-04-25T16:09:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.452+0000 7f62d35ce640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62b80afd00 0x7f62b80b2210 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.452+0000 7f62d35ce640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f62cc110030 0x7f62cc1a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.452+0000 7f62d35ce640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62cc10b310 0x7f62cc1a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.452+0000 7f62d35ce640 1 --2- 172.21.15.31:0/4237444397 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62cc10a420 0x7f62cc1a08d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:03.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.452+0000 7f62d35ce640 1 -- 172.21.15.31:0/4237444397 >> 172.21.15.31:0/4237444397 conn(0x7f62cc0704e0 msgr2=0x7f62cc10e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.461+0000 7f62d35ce640 1 -- 172.21.15.31:0/4237444397 shutdown_connections 2024-04-25T16:09:03.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:03.465+0000 7f62d35ce640 1 -- 172.21.15.31:0/4237444397 wait complete. 2024-04-25T16:09:03.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:09:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.011+0000 7feb06934640 1 -- 172.21.15.31:0/3836793141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb0010a420 msgr2=0x7feb0010add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.011+0000 7feb06934640 1 --2- 172.21.15.31:0/3836793141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb0010a420 0x7feb0010add0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009be0 tx=0x7feaf002a6c0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.012+0000 7feb06934640 1 -- 172.21.15.31:0/3836793141 shutdown_connections 2024-04-25T16:09:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.012+0000 7feb06934640 1 --2- 172.21.15.31:0/3836793141 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb00110030 0x7feb00110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.012+0000 7feb06934640 1 --2- 172.21.15.31:0/3836793141 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb0010b310 0x7feb0010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.012+0000 7feb06934640 1 --2- 172.21.15.31:0/3836793141 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb0010a420 0x7feb0010add0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.012+0000 7feb06934640 1 -- 172.21.15.31:0/3836793141 >> 172.21.15.31:0/3836793141 conn(0x7feb000704e0 msgr2=0x7feb00072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.021+0000 7feb06934640 1 -- 172.21.15.31:0/3836793141 shutdown_connections 2024-04-25T16:09:04.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.022+0000 7feb06934640 1 -- 172.21.15.31:0/3836793141 wait complete. 2024-04-25T16:09:04.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.023+0000 7feb06934640 1 Processor -- start 2024-04-25T16:09:04.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.023+0000 7feb06934640 1 -- start start 2024-04-25T16:09:04.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.024+0000 7feb06934640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb0010a420 0x7feb0011b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.024+0000 7feb06934640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb0010b310 0x7feb0011a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.024+0000 7feb06934640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb00110030 0x7feb0011a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.024+0000 7feb06934640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feb00112380 con 0x7feb00110030 2024-04-25T16:09:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.024+0000 7feb06934640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7feb00112200 con 0x7feb0010a420 2024-04-25T16:09:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.024+0000 7feb06934640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb00112500 con 0x7feb0010b310 2024-04-25T16:09:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.025+0000 7feaff7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb0010b310 0x7feb0011a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.025+0000 7feaff7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb0010b310 0x7feb0011a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45498/0 (socket says 172.21.15.31:45498) 2024-04-25T16:09:04.026 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.025+0000 7feaff7fe640 1 -- 172.21.15.31:0/1442215465 learned_addr learned my addr 172.21.15.31:0/1442215465 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.025+0000 7feafffff640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb0010a420 0x7feb0011b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.025+0000 7feb04ed1640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb00110030 0x7feb0011a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.026+0000 7feaff7fe640 1 -- 172.21.15.31:0/1442215465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb0010a420 msgr2=0x7feb0011b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.026+0000 7feaff7fe640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb0010a420 0x7feb0011b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.026+0000 7feaff7fe640 1 -- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb00110030 msgr2=0x7feb0011a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.026+0000 7feaff7fe640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb00110030 0x7feb0011a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.026+0000 7feaff7fe640 1 -- 172.21.15.31:0/1442215465 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb0011aaa0 con 0x7feb0010b310 2024-04-25T16:09:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.027+0000 7feb04ed1640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb00110030 0x7feb0011a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:09:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.027+0000 7feafffff640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb0010a420 0x7feb0011b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:09:04.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.027+0000 7feaff7fe640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb0010b310 0x7feb0011a020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7feae800e9f0 tx=0x7feae800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.028+0000 7feafd7fa640 1 -- 172.21.15.31:0/1442215465 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae800ccc0 con 0x7feb0010b310 2024-04-25T16:09:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.028+0000 7feb06934640 1 -- 172.21.15.31:0/1442215465 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb0011ad90 con 0x7feb0010b310 2024-04-25T16:09:04.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.028+0000 7feb06934640 1 -- 172.21.15.31:0/1442215465 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb001b9cb0 con 0x7feb0010b310 2024-04-25T16:09:04.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.031+0000 7feafd7fa640 1 -- 172.21.15.31:0/1442215465 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feae80108e0 con 0x7feb0010b310 2024-04-25T16:09:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.034+0000 7feb0d7e8740 1 Processor -- start 2024-04-25T16:09:04.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.034+0000 7feafd7fa640 1 -- 172.21.15.31:0/1442215465 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae8018780 con 0x7feb0010b310 2024-04-25T16:09:04.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.035+0000 7feafd7fa640 1 -- 172.21.15.31:0/1442215465 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7feae80189e0 con 0x7feb0010b310 2024-04-25T16:09:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.036+0000 7feafd7fa640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feaec0afa20 0x7feaec0b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.036+0000 7feb0d7e8740 1 -- start start 2024-04-25T16:09:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.037+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f902b390 0x55b5f93cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.037+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 0x55b5f93cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.037+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f900a6b0 0x55b5f93cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.037+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b5f9331d80 con 0x55b5f902b390 2024-04-25T16:09:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.037+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b5f92eba20 con 0x55b5f90024d0 2024-04-25T16:09:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.037+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b5f932dd40 con 0x55b5f900a6b0 2024-04-25T16:09:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.037+0000 7feb04ed1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f900a6b0 0x55b5f93cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.037+0000 7feb04ed1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f900a6b0 0x55b5f93cea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45506/0 (socket says 172.21.15.31:45506) 2024-04-25T16:09:04.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.037+0000 7feb04ed1640 1 -- 172.21.15.31:0/341301978 learned_addr learned my addr 172.21.15.31:0/341301978 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.038+0000 7feb04ed1640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feaec0afa20 0x7feaec0b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.039+0000 7feb04ed1640 1 -- 172.21.15.31:0/341301978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 msgr2=0x55b5f93cdd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:09:04.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.039+0000 7feb04ed1640 1 --2- 172.21.15.31:0/341301978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 0x55b5f93cdd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.039+0000 7feb04ed1640 1 -- 172.21.15.31:0/341301978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f902b390 msgr2=0x55b5f93cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.039+0000 7feb04ed1640 1 --2- 172.21.15.31:0/341301978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f902b390 0x55b5f93cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.039+0000 7feb04ed1640 1 -- 172.21.15.31:0/341301978 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f92f6c30 con 0x55b5f900a6b0 2024-04-25T16:09:04.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.039+0000 7feb04ed1640 1 --2- 172.21.15.31:0/341301978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f900a6b0 0x55b5f93cea20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feaf8011e70 tx=0x7feaf800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30dc4ff9dc9cc1af server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.039+0000 7feb04ed1640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feaec0afa20 0x7feaec0b1f30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7feaf800fc90 tx=0x7feaf8018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.040+0000 7feafd7fa640 1 -- 172.21.15.31:0/1442215465 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7feae80a4b80 con 0x7feb0010b310 2024-04-25T16:09:04.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.041+0000 7feaf67fc640 1 -- 172.21.15.31:0/341301978 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8022070 con 0x55b5f900a6b0 2024-04-25T16:09:04.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.041+0000 7feaf67fc640 1 -- 172.21.15.31:0/341301978 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaf800ed10 con 0x55b5f900a6b0 2024-04-25T16:09:04.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.042+0000 7feaf67fc640 1 -- 172.21.15.31:0/341301978 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8010d00 con 0x55b5f900a6b0 2024-04-25T16:09:04.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.043+0000 7feb0d7e8740 1 -- 172.21.15.31:0/341301978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f900a6b0 msgr2=0x55b5f93cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.043+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/341301978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f900a6b0 0x55b5f93cea20 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feaf8011e70 tx=0x7feaf800f790 comp rx=0 tx=0).stop 2024-04-25T16:09:04.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.043+0000 7feb0d7e8740 1 -- 172.21.15.31:0/341301978 shutdown_connections 2024-04-25T16:09:04.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.043+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/341301978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f900a6b0 0x55b5f93cea20 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.043+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/341301978 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 0x55b5f93cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.043+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/341301978 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f902b390 0x55b5f93cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.043+0000 7feb0d7e8740 1 -- 172.21.15.31:0/341301978 >> 172.21.15.31:0/341301978 conn(0x55b5f901f4e0 msgr2=0x55b5f93c9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.047+0000 7feb0d7e8740 1 -- 172.21.15.31:0/341301978 shutdown_connections 2024-04-25T16:09:04.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.048+0000 7feb0d7e8740 1 -- 172.21.15.31:0/341301978 wait complete. 2024-04-25T16:09:04.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.048+0000 7feb0d7e8740 1 -- start start 2024-04-25T16:09:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.049+0000 7feb0d7e8740 1 Processor -- start 2024-04-25T16:09:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.049+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 0x55b5f93d74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.049+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 0x55b5f93d7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.049+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f902b390 0x55b5f93de970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.049+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b5f93cf520 con 0x55b5f900a6b0 2024-04-25T16:09:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.049+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b5f92b8dc0 con 0x55b5f90024d0 2024-04-25T16:09:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.049+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b5f93cf6a0 con 0x55b5f902b390 2024-04-25T16:09:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.050+0000 7feb0d7e8740 10 client.0 _ll_register_callbacks cb 0x55b5f92d4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.051+0000 7feb04ed1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f902b390 0x55b5f93de970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.051+0000 7feafffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 0x55b5f93d74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.051+0000 7feafffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 0x55b5f93d74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56002/0 (socket says 172.21.15.31:56002) 2024-04-25T16:09:04.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.051+0000 7feafffff640 1 -- 172.21.15.31:0/2464159802 learned_addr learned my addr 172.21.15.31:0/2464159802 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.052+0000 7feafffff640 1 -- 172.21.15.31:0/2464159802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f902b390 msgr2=0x55b5f93de970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.052+0000 7feafffff640 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f902b390 0x55b5f93de970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.052+0000 7feafffff640 1 -- 172.21.15.31:0/2464159802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 msgr2=0x55b5f93d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.052+0000 7feafffff640 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 0x55b5f93d7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.052+0000 7feafffff640 1 -- 172.21.15.31:0/2464159802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5f90433f0 con 0x55b5f90024d0 2024-04-25T16:09:04.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.053+0000 7feafffff640 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 0x55b5f93d74d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009be0 tx=0x7feaf0030c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=621f77e9f776d2e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.060+0000 7feadf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:04.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.061+0000 7feadf7fe640 1 -- 172.21.15.31:0/2464159802 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf0041030 con 0x55b5f90024d0 2024-04-25T16:09:04.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.061+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2464159802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7feaf800ed10 con 0x55b5f90024d0 2024-04-25T16:09:04.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.061+0000 7feadf7fe640 1 -- 172.21.15.31:0/2464159802 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7feaf0030d20 con 0x55b5f90024d0 2024-04-25T16:09:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.070+0000 7feadf7fe640 1 -- 172.21.15.31:0/2464159802 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feaf0036040 con 0x55b5f90024d0 2024-04-25T16:09:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.070+0000 7feb0d7e8740 10 client.26813 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:04.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.070+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2464159802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5f910d7d0 con 0x55b5f90024d0 2024-04-25T16:09:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.071+0000 7feadf7fe640 1 -- 172.21.15.31:0/2464159802 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaf800ed10 con 0x55b5f90024d0 2024-04-25T16:09:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.071+0000 7feadf7fe640 1 -- 172.21.15.31:0/2464159802 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7feaf0004460 con 0x55b5f90024d0 2024-04-25T16:09:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.071+0000 7feb0d7e8740 10 client.26813 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.071+0000 7feb0d7e8740 10 client.26813 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:09:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.071+0000 7feb0d7e8740 20 client.26813 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.071+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55b5f8fe9f60 0x55b5f93e5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.071+0000 7feb0d7e8740 4 client.26813 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:04.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.071+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2464159802 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5f933f990 con 0x55b5f8fe9f60 2024-04-25T16:09:04.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.072+0000 7feaff7fe640 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55b5f8fe9f60 0x55b5f93e5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.073+0000 7feadf7fe640 10 client.26813 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:09:04.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.073+0000 7feaff7fe640 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55b5f8fe9f60 0x55b5f93e5730 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=658d38af9ed5e1a4 server_cookie=5724bfa6feb33b42 in_seq=0 out_seq=0 2024-04-25T16:09:04.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.082+0000 7feadf7fe640 1 -- 172.21.15.31:0/2464159802 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b5f933f990 con 0x55b5f8fe9f60 2024-04-25T16:09:04.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.082+0000 7feadf7fe640 10 client.26813 handle_command_reply: tid=0 2024-04-25T16:09:04.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.082+0000 7feb0d7e8740 1 client.26813 shutdown 2024-04-25T16:09:04.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.084+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2464159802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 msgr2=0x55b5f93d74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.084+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 0x55b5f93d74d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7feaf0009be0 tx=0x7feaf0030c30 comp rx=0 tx=0).stop 2024-04-25T16:09:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.088+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2464159802 shutdown_connections 2024-04-25T16:09:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.088+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55b5f8fe9f60 0x55b5f93e5730 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.088+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f902b390 0x55b5f93de970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.088+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 0x55b5f93d7a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.088+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2464159802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f90024d0 0x55b5f93d74d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.088+0000 7feadf7fe640 0 client.26813 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:09:04.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.088+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2464159802 >> 172.21.15.31:0/2464159802 conn(0x55b5f901f4e0 msgr2=0x55b5f93c8040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.088+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2464159802 shutdown_connections 2024-04-25T16:09:04.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.088+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2464159802 wait complete. 2024-04-25T16:09:04.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.089+0000 7feb0d7e8740 20 client.26813 trim_cache size 0 max 16384 2024-04-25T16:09:04.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.133+0000 7feb0d7e8740 1 Processor -- start 2024-04-25T16:09:04.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.133+0000 7feb0d7e8740 1 -- start start 2024-04-25T16:09:04.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.135+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 0x55b5f93c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.135+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f900a6b0 0x55b5f93de890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.135+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f902b390 0x55b5f93d70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.135+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feae000e500 con 0x55b5f902b390 2024-04-25T16:09:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.135+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7feae0010040 con 0x55b5f900a6b0 2024-04-25T16:09:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.135+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feae000ebd0 con 0x55b5f90024d0 2024-04-25T16:09:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.135+0000 7feafffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 0x55b5f93c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.136+0000 7feaff7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f900a6b0 0x55b5f93de890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.136+0000 7feaff7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f900a6b0 0x55b5f93de890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56018/0 (socket says 172.21.15.31:56018) 2024-04-25T16:09:04.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.136+0000 7feaff7fe640 1 -- 172.21.15.31:0/1184948780 learned_addr learned my addr 172.21.15.31:0/1184948780 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.136+0000 7feaff7fe640 1 -- 172.21.15.31:0/1184948780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 msgr2=0x55b5f93c3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.136+0000 7feaff7fe640 1 --2- 172.21.15.31:0/1184948780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 0x55b5f93c3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.136+0000 7feaff7fe640 1 -- 172.21.15.31:0/1184948780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f902b390 msgr2=0x55b5f93d70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.136+0000 7feaff7fe640 1 --2- 172.21.15.31:0/1184948780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f902b390 0x55b5f93d70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.136+0000 7feaff7fe640 1 -- 172.21.15.31:0/1184948780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5f93d80d0 con 0x55b5f900a6b0 2024-04-25T16:09:04.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.137+0000 7feaff7fe640 1 --2- 172.21.15.31:0/1184948780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f900a6b0 0x55b5f93de890 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7feae80a5430 tx=0x7feae80afc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb0433a335a664c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.137+0000 7feaf5ffb640 1 -- 172.21.15.31:0/1184948780 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae80a4d80 con 0x55b5f900a6b0 2024-04-25T16:09:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.137+0000 7feaf5ffb640 1 -- 172.21.15.31:0/1184948780 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feae80910c0 con 0x55b5f900a6b0 2024-04-25T16:09:04.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.138+0000 7feaf5ffb640 1 -- 172.21.15.31:0/1184948780 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae80a4f20 con 0x55b5f900a6b0 2024-04-25T16:09:04.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.139+0000 7feb0d7e8740 1 -- 172.21.15.31:0/1184948780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f900a6b0 msgr2=0x55b5f93de890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.139+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/1184948780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f900a6b0 0x55b5f93de890 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7feae80a5430 tx=0x7feae80afc50 comp rx=0 tx=0).stop 2024-04-25T16:09:04.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.145+0000 7feb0d7e8740 1 -- 172.21.15.31:0/1184948780 shutdown_connections 2024-04-25T16:09:04.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.145+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/1184948780 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f902b390 0x55b5f93d70a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.145+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/1184948780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f900a6b0 0x55b5f93de890 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.145+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/1184948780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 0x55b5f93c3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.145+0000 7feb0d7e8740 1 -- 172.21.15.31:0/1184948780 >> 172.21.15.31:0/1184948780 conn(0x55b5f901f4e0 msgr2=0x55b5f93d2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.145+0000 7feb0d7e8740 1 -- 172.21.15.31:0/1184948780 shutdown_connections 2024-04-25T16:09:04.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.145+0000 7feb0d7e8740 1 -- 172.21.15.31:0/1184948780 wait complete. 2024-04-25T16:09:04.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.146+0000 7feb0d7e8740 1 -- start start 2024-04-25T16:09:04.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.147+0000 7feb0d7e8740 1 Processor -- start 2024-04-25T16:09:04.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.149+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 0x55b5f93dbac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.149+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 0x55b5f93dc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.149+0000 7feb0d7e8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f902b390 0x55b5f93dcf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.149+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feae000f970 con 0x55b5f900a6b0 2024-04-25T16:09:04.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.149+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7feae000f2a0 con 0x55b5f902b390 2024-04-25T16:09:04.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.150+0000 7feb0d7e8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feae000c4e0 con 0x55b5f90024d0 2024-04-25T16:09:04.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.150+0000 7feb0d7e8740 10 client.0 _ll_register_callbacks cb 0x7feaf8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:04.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.150+0000 7feb04ed1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f902b390 0x55b5f93dcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.150+0000 7feb04ed1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f902b390 0x55b5f93dcf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56022/0 (socket says 172.21.15.31:56022) 2024-04-25T16:09:04.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.150+0000 7feb04ed1640 1 -- 172.21.15.31:0/2497264960 learned_addr learned my addr 172.21.15.31:0/2497264960 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.151+0000 7feafffff640 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 0x55b5f93dbac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.151+0000 7feaff7fe640 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 0x55b5f93dc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.151+0000 7feb04ed1640 1 -- 172.21.15.31:0/2497264960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 msgr2=0x55b5f93dbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.151+0000 7feb04ed1640 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 0x55b5f93dbac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.151+0000 7feb04ed1640 1 -- 172.21.15.31:0/2497264960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 msgr2=0x55b5f93dc000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.151+0000 7feb04ed1640 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 0x55b5f93dc000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.151+0000 7feb04ed1640 1 -- 172.21.15.31:0/2497264960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5f93dd4f0 con 0x55b5f902b390 2024-04-25T16:09:04.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.151+0000 7feaff7fe640 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 0x55b5f93dc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:09:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.152+0000 7feb04ed1640 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f902b390 0x55b5f93dcf30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feaf80054f0 tx=0x7feaf8005520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b819a1b8d52d3482 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.152+0000 7feadf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.152+0000 7feadf7fe640 1 -- 172.21.15.31:0/2497264960 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaf8022070 con 0x55b5f902b390 2024-04-25T16:09:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.153+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2497264960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b5f9339800 con 0x55b5f902b390 2024-04-25T16:09:04.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.153+0000 7feadf7fe640 1 -- 172.21.15.31:0/2497264960 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7feaf8005740 con 0x55b5f902b390 2024-04-25T16:09:04.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.153+0000 7feadf7fe640 1 -- 172.21.15.31:0/2497264960 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7feaf802b040 con 0x55b5f902b390 2024-04-25T16:09:04.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.153+0000 7feb0d7e8740 10 client.26819 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:04.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.153+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2497264960 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b5f93de7e0 con 0x55b5f902b390 2024-04-25T16:09:04.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.154+0000 7feadf7fe640 1 -- 172.21.15.31:0/2497264960 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55b5f9339800 con 0x55b5f902b390 2024-04-25T16:09:04.155 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.154+0000 7feadf7fe640 1 -- 172.21.15.31:0/2497264960 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7feaf8003ba0 con 0x55b5f902b390 2024-04-25T16:09:04.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.154+0000 7feb0d7e8740 10 client.26819 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:04.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.154+0000 7feb0d7e8740 10 client.26819 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:09:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.154+0000 7feb0d7e8740 20 client.26819 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.154+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55b5f8fbf3a0 0x55b5f944fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.154+0000 7feb0d7e8740 4 client.26819 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.154+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2497264960 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b5f92b4610 con 0x55b5f8fbf3a0 2024-04-25T16:09:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.155+0000 7feafffff640 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55b5f8fbf3a0 0x55b5f944fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.155+0000 7feafffff640 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55b5f8fbf3a0 0x55b5f944fef0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e9b8cbc51fb477d2 server_cookie=f2939e3028a214d5 in_seq=0 out_seq=0 2024-04-25T16:09:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.155+0000 7feadf7fe640 10 client.26819 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:09:04.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.156+0000 7feadf7fe640 1 -- 172.21.15.31:0/2497264960 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b5f92b4610 con 0x55b5f8fbf3a0 2024-04-25T16:09:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.156+0000 7feadf7fe640 10 client.26819 handle_command_reply: tid=0 2024-04-25T16:09:04.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.156+0000 7feb0d7e8740 1 client.26819 shutdown 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.156+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2497264960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f902b390 msgr2=0x55b5f93dcf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.156+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f902b390 0x55b5f93dcf30 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7feaf80054f0 tx=0x7feaf8005520 comp rx=0 tx=0).stop 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.157+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2497264960 shutdown_connections 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.157+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b5f902b390 0x55b5f93dcf30 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.157+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b5f900a6b0 0x55b5f93dc000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.157+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55b5f8fbf3a0 0x55b5f944fef0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.157+0000 7feb0d7e8740 1 --2- 172.21.15.31:0/2497264960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b5f90024d0 0x55b5f93dbac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.157+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2497264960 >> 172.21.15.31:0/2497264960 conn(0x55b5f901f4e0 msgr2=0x55b5f93d7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.157+0000 7feadf7fe640 0 client.26819 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.160+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2497264960 shutdown_connections 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.160+0000 7feb0d7e8740 1 -- 172.21.15.31:0/2497264960 wait complete. 2024-04-25T16:09:04.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.160+0000 7feb0d7e8740 20 client.26819 trim_cache size 0 max 16384 2024-04-25T16:09:04.162 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.161+0000 7feb06934640 1 -- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feaec0afa20 msgr2=0x7feaec0b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.161+0000 7feb06934640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feaec0afa20 0x7feaec0b1f30 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7feaf800fc90 tx=0x7feaf8018000 comp rx=0 tx=0).stop 2024-04-25T16:09:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.161+0000 7feb06934640 1 -- 172.21.15.31:0/1442215465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb0010b310 msgr2=0x7feb0011a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.161+0000 7feb06934640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb0010b310 0x7feb0011a020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7feae800e9f0 tx=0x7feae800edf0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.161+0000 7feb06934640 1 -- 172.21.15.31:0/1442215465 shutdown_connections 2024-04-25T16:09:04.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.161+0000 7feb06934640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feaec0afa20 0x7feaec0b1f30 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.161+0000 7feb06934640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7feb00110030 0x7feb0011a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.162+0000 7feb06934640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb0010b310 0x7feb0011a020 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.162+0000 7feb06934640 1 --2- 172.21.15.31:0/1442215465 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7feb0010a420 0x7feb0011b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.162+0000 7feb06934640 1 -- 172.21.15.31:0/1442215465 >> 172.21.15.31:0/1442215465 conn(0x7feb000704e0 msgr2=0x7feb0010e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.162+0000 7feb06934640 1 -- 172.21.15.31:0/1442215465 shutdown_connections 2024-04-25T16:09:04.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.162+0000 7feb06934640 1 -- 172.21.15.31:0/1442215465 wait complete. 2024-04-25T16:09:04.201 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:09:04.331 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:04 smithi031 ceph-mon[29140]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 895 MiB/s wr, 2.12k op/s 2024-04-25T16:09:04.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:04 smithi063 ceph-mon[32413]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 895 MiB/s wr, 2.12k op/s 2024-04-25T16:09:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.386+0000 7fec6c9d6640 1 -- 172.21.15.31:0/1588430451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec6810b190 msgr2=0x7fec6810b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.386+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/1588430451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec6810b190 0x7fec6810b5a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fec58009900 tx=0x7fec5802a440 comp rx=0 tx=0).stop 2024-04-25T16:09:04.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.387+0000 7fec6c9d6640 1 -- 172.21.15.31:0/1588430451 shutdown_connections 2024-04-25T16:09:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.387+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/1588430451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec6810feb0 0x7fec681102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.387+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/1588430451 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec6810b190 0x7fec6810b5a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.387+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/1588430451 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec6810a340 0x7fec6810ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.392 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.387+0000 7fec6c9d6640 1 -- 172.21.15.31:0/1588430451 >> 172.21.15.31:0/1588430451 conn(0x7fec68070330 msgr2=0x7fec680727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:04 smithi186 ceph-mon[33258]: pgmap v584: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 895 MiB/s wr, 2.12k op/s 2024-04-25T16:09:04.400 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.398+0000 7fec6c9d6640 1 -- 172.21.15.31:0/1588430451 shutdown_connections 2024-04-25T16:09:04.412 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.410+0000 7fec6c9d6640 1 -- 172.21.15.31:0/1588430451 wait complete. 2024-04-25T16:09:04.414 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.412+0000 7fec6c9d6640 1 Processor -- start 2024-04-25T16:09:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.413+0000 7fec6c9d6640 1 -- start start 2024-04-25T16:09:04.415 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.413+0000 7fec6c9d6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec6810a340 0x7fec681a8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.413+0000 7fec6c9d6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec6810b190 0x7fec681a9430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.413+0000 7fec6c9d6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec6810feb0 0x7fec681b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.413+0000 7fec6c9d6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec6811e2f0 con 0x7fec6810feb0 2024-04-25T16:09:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.413+0000 7fec6c9d6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec6811e170 con 0x7fec6810a340 2024-04-25T16:09:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.413+0000 7fec6c9d6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec6811e470 con 0x7fec6810b190 2024-04-25T16:09:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.414+0000 7fec6659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec6810b190 0x7fec681a9430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.415+0000 7fec6659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec6810b190 0x7fec681a9430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45552/0 (socket says 172.21.15.31:45552) 2024-04-25T16:09:04.416 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.415+0000 7fec6659c640 1 -- 172.21.15.31:0/431743631 learned_addr learned my addr 172.21.15.31:0/431743631 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.416+0000 7fec6659c640 1 -- 172.21.15.31:0/431743631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec6810a340 msgr2=0x7fec681a8ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.416+0000 7fec6659c640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec6810a340 0x7fec681a8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.416+0000 7fec6659c640 1 -- 172.21.15.31:0/431743631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec6810feb0 msgr2=0x7fec681b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.416+0000 7fec6659c640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec6810feb0 0x7fec681b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.417 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.416+0000 7fec6659c640 1 -- 172.21.15.31:0/431743631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec681b0a30 con 0x7fec6810b190 2024-04-25T16:09:04.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.416+0000 7fec6659c640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec6810b190 0x7fec681a9430 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fec5802aa30 tx=0x7fec58003e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.430 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.429+0000 7fec4ffff640 1 -- 172.21.15.31:0/431743631 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec58040030 con 0x7fec6810b190 2024-04-25T16:09:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.429+0000 7fec6c9d6640 1 -- 172.21.15.31:0/431743631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec681b0cc0 con 0x7fec6810b190 2024-04-25T16:09:04.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.429+0000 7fec6c9d6640 1 -- 172.21.15.31:0/431743631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec681b1230 con 0x7fec6810b190 2024-04-25T16:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.437+0000 7fec74166740 1 Processor -- start 2024-04-25T16:09:04.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.437+0000 7fec74166740 1 -- start start 2024-04-25T16:09:04.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.438+0000 7fec74166740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55730fb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.439+0000 7fec74166740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b5573100a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.439+0000 7fec74166740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b557316fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.439+0000 7fec74166740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b557275460 con 0x55b556f462f0 2024-04-25T16:09:04.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.439+0000 7fec74166740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b55722f590 con 0x55b556f4e4d0 2024-04-25T16:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.439+0000 7fec74166740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b557271420 con 0x55b556f6f1b0 2024-04-25T16:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.439+0000 7fec4ffff640 1 -- 172.21.15.31:0/431743631 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec58005160 con 0x7fec6810b190 2024-04-25T16:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.439+0000 7fec6759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55730fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.440+0000 7fec6759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55730fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45558/0 (socket says 172.21.15.31:45558) 2024-04-25T16:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.440+0000 7fec6759e640 1 -- 172.21.15.31:0/1876999529 learned_addr learned my addr 172.21.15.31:0/1876999529 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.440+0000 7fec4ffff640 1 -- 172.21.15.31:0/431743631 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5803c910 con 0x7fec6810b190 2024-04-25T16:09:04.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.440+0000 7fec4ffff640 1 -- 172.21.15.31:0/431743631 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fec58043030 con 0x7fec6810b190 2024-04-25T16:09:04.442 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.440+0000 7fec66d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b5573100a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.441+0000 7fec6759e640 1 -- 172.21.15.31:0/1876999529 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 msgr2=0x55b557316fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.443+0000 7fec6759e640 1 --2- 172.21.15.31:0/1876999529 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b557316fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.443+0000 7fec6759e640 1 -- 172.21.15.31:0/1876999529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 msgr2=0x55b5573100a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.443+0000 7fec6759e640 1 --2- 172.21.15.31:0/1876999529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b5573100a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.443+0000 7fec6759e640 1 -- 172.21.15.31:0/1876999529 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b55723c0d0 con 0x55b556f6f1b0 2024-04-25T16:09:04.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.443+0000 7fec66d9d640 1 --2- 172.21.15.31:0/1876999529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b5573100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:09:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.444+0000 7fec6759e640 1 --2- 172.21.15.31:0/1876999529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55730fb20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fec60009d80 tx=0x7fec6000e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbabc6aaa654c2b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.445 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.444+0000 7fec4d7fa640 1 -- 172.21.15.31:0/1876999529 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60016070 con 0x55b556f6f1b0 2024-04-25T16:09:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.445+0000 7fec4d7fa640 1 -- 172.21.15.31:0/1876999529 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec60005020 con 0x55b556f6f1b0 2024-04-25T16:09:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.445+0000 7fec4d7fa640 1 -- 172.21.15.31:0/1876999529 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec600116d0 con 0x55b556f6f1b0 2024-04-25T16:09:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.446+0000 7fec74166740 1 -- 172.21.15.31:0/1876999529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 msgr2=0x55b55730fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.447 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.446+0000 7fec74166740 1 --2- 172.21.15.31:0/1876999529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55730fb20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fec60009d80 tx=0x7fec6000e6c0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.447+0000 7fec4ffff640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec500afd20 0x7fec500b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.449 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.448+0000 7fec4ffff640 1 -- 172.21.15.31:0/431743631 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fec581004d0 con 0x7fec6810b190 2024-04-25T16:09:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.449+0000 7fec74166740 1 -- 172.21.15.31:0/1876999529 shutdown_connections 2024-04-25T16:09:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.449+0000 7fec74166740 1 --2- 172.21.15.31:0/1876999529 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b557316fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.449+0000 7fec74166740 1 --2- 172.21.15.31:0/1876999529 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b5573100a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.450 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.449+0000 7fec74166740 1 --2- 172.21.15.31:0/1876999529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55730fb20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.449+0000 7fec74166740 1 -- 172.21.15.31:0/1876999529 >> 172.21.15.31:0/1876999529 conn(0x55b556f63300 msgr2=0x55b55730b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.449+0000 7fec74166740 1 -- 172.21.15.31:0/1876999529 shutdown_connections 2024-04-25T16:09:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.449+0000 7fec74166740 1 -- 172.21.15.31:0/1876999529 wait complete. 2024-04-25T16:09:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.450+0000 7fec74166740 1 -- start start 2024-04-25T16:09:04.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.450+0000 7fec74166740 1 Processor -- start 2024-04-25T16:09:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.451+0000 7fec74166740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b557319910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.451+0000 7fec74166740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.451+0000 7fec74166740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f6f1b0 0x55b557320db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.451+0000 7fec74166740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b5573179f0 con 0x55b556f462f0 2024-04-25T16:09:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.451+0000 7fec74166740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b5571fcb60 con 0x55b556f6f1b0 2024-04-25T16:09:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.451+0000 7fec74166740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b557317b70 con 0x55b556f4e4d0 2024-04-25T16:09:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.451+0000 7fec74166740 10 client.0 _ll_register_callbacks cb 0x55b5572212b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.452+0000 7fec6759e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f6f1b0 0x55b557320db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.452+0000 7fec6759e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f6f1b0 0x55b557320db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56052/0 (socket says 172.21.15.31:56052) 2024-04-25T16:09:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.452+0000 7fec6759e640 1 -- 172.21.15.31:0/3394599043 learned_addr learned my addr 172.21.15.31:0/3394599043 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.452+0000 7fec6659c640 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.453+0000 7fec6759e640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec500afd20 0x7fec500b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.453+0000 7fec6759e640 1 -- 172.21.15.31:0/3394599043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f4e4d0 msgr2=0x55b557319e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.453+0000 7fec6759e640 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.453+0000 7fec6759e640 1 -- 172.21.15.31:0/3394599043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 msgr2=0x55b557319910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.453+0000 7fec6759e640 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b557319910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.453+0000 7fec6759e640 1 -- 172.21.15.31:0/3394599043 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b556f87210 con 0x55b556f6f1b0 2024-04-25T16:09:04.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.454+0000 7fec6759e640 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f6f1b0 0x55b557320db0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fec600049d0 tx=0x7fec60005570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12a39491984fbb54 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.461+0000 7fec4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.461+0000 7fec4aff5640 1 -- 172.21.15.31:0/3394599043 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60016050 con 0x55b556f6f1b0 2024-04-25T16:09:04.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.461+0000 7fec74166740 1 -- 172.21.15.31:0/3394599043 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fec60005020 con 0x55b556f6f1b0 2024-04-25T16:09:04.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.461+0000 7fec4aff5640 1 -- 172.21.15.31:0/3394599043 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fec60011680 con 0x55b556f6f1b0 2024-04-25T16:09:04.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.465+0000 7fec4aff5640 1 -- 172.21.15.31:0/3394599043 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fec60011a60 con 0x55b556f6f1b0 2024-04-25T16:09:04.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.471+0000 7fec74166740 10 client.26822 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.471+0000 7fec74166740 1 -- 172.21.15.31:0/3394599043 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b557053620 con 0x55b556f6f1b0 2024-04-25T16:09:04.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.472+0000 7fec4aff5640 1 -- 172.21.15.31:0/3394599043 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec60005020 con 0x55b556f6f1b0 2024-04-25T16:09:04.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.478+0000 7fec4aff5640 1 -- 172.21.15.31:0/3394599043 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fec6000f040 con 0x55b556f6f1b0 2024-04-25T16:09:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.481+0000 7fec74166740 10 client.26822 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.481+0000 7fec74166740 10 client.26822 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.481+0000 7fec74166740 20 client.26822 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.481+0000 7fec74166740 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b556f2dd80 0x55b557327460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.481+0000 7fec74166740 4 client.26822 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:04.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.481+0000 7fec74166740 1 -- 172.21.15.31:0/3394599043 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b557283070 con 0x55b556f2dd80 2024-04-25T16:09:04.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.482+0000 7fec66d9d640 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b556f2dd80 0x55b557327460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.487+0000 7fec66d9d640 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b556f2dd80 0x55b557327460 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c05f67376d6850b6 server_cookie=ca907055f73fdb21 in_seq=0 out_seq=0 2024-04-25T16:09:04.490 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.487+0000 7fec4aff5640 10 client.26822 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:04.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.492+0000 7fec6759e640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec500afd20 0x7fec500b2230 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55b5573240e0 tx=0x7fec60028000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.496+0000 7fec4aff5640 1 -- 172.21.15.31:0/3394599043 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b557283070 con 0x55b556f2dd80 2024-04-25T16:09:04.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.497+0000 7fec4aff5640 10 client.26822 handle_command_reply: tid=0 2024-04-25T16:09:04.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.500+0000 7fec74166740 1 client.26822 shutdown 2024-04-25T16:09:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.501+0000 7fec74166740 1 -- 172.21.15.31:0/3394599043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f6f1b0 msgr2=0x55b557320db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.501+0000 7fec74166740 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f6f1b0 0x55b557320db0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fec600049d0 tx=0x7fec60005570 comp rx=0 tx=0).stop 2024-04-25T16:09:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.504+0000 7fec74166740 1 -- 172.21.15.31:0/3394599043 shutdown_connections 2024-04-25T16:09:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.504+0000 7fec74166740 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b556f2dd80 0x55b557327460 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.504+0000 7fec74166740 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f6f1b0 0x55b557320db0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.504+0000 7fec74166740 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.504+0000 7fec74166740 1 --2- 172.21.15.31:0/3394599043 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b557319910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.504+0000 7fec4aff5640 0 client.26822 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:04.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.504+0000 7fec74166740 1 -- 172.21.15.31:0/3394599043 >> 172.21.15.31:0/3394599043 conn(0x55b556f63300 msgr2=0x55b55730e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.505+0000 7fec74166740 1 -- 172.21.15.31:0/3394599043 shutdown_connections 2024-04-25T16:09:04.509 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.507+0000 7fec74166740 1 -- 172.21.15.31:0/3394599043 wait complete. 2024-04-25T16:09:04.511 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.508+0000 7fec74166740 20 client.26822 trim_cache size 0 max 16384 2024-04-25T16:09:04.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.585+0000 7fec74166740 1 Processor -- start 2024-04-25T16:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.592+0000 7fec74166740 1 -- start start 2024-04-25T16:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.593+0000 7fec74166740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f462f0 0x55b557319910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.593+0000 7fec74166740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.593+0000 7fec74166740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f6f1b0 0x55b55732bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.593+0000 7fec74166740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec5400e3b0 con 0x55b556f6f1b0 2024-04-25T16:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.593+0000 7fec74166740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec5400fef0 con 0x55b556f4e4d0 2024-04-25T16:09:04.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.593+0000 7fec74166740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec5400ea80 con 0x55b556f462f0 2024-04-25T16:09:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.594+0000 7fec6659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.594+0000 7fec6659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56060/0 (socket says 172.21.15.31:56060) 2024-04-25T16:09:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.594+0000 7fec6659c640 1 -- 172.21.15.31:0/2309263297 learned_addr learned my addr 172.21.15.31:0/2309263297 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.596+0000 7fec6659c640 1 -- 172.21.15.31:0/2309263297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f462f0 msgr2=0x55b557319910 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.596+0000 7fec6659c640 1 --2- 172.21.15.31:0/2309263297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f462f0 0x55b557319910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.596+0000 7fec6659c640 1 -- 172.21.15.31:0/2309263297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f6f1b0 msgr2=0x55b55732bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.596+0000 7fec6759e640 1 --2- 172.21.15.31:0/2309263297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f6f1b0 0x55b55732bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.596+0000 7fec6659c640 1 --2- 172.21.15.31:0/2309263297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f6f1b0 0x55b55732bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.597+0000 7fec6659c640 1 -- 172.21.15.31:0/2309263297 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b55730be10 con 0x55b556f4e4d0 2024-04-25T16:09:04.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.598+0000 7fec6659c640 1 --2- 172.21.15.31:0/2309263297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fec58004650 tx=0x7fec580c88c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3382b21fc4ee9563 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.606+0000 7fec4cff9640 1 -- 172.21.15.31:0/2309263297 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec5804a040 con 0x55b556f4e4d0 2024-04-25T16:09:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.606+0000 7fec4cff9640 1 -- 172.21.15.31:0/2309263297 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec580c84f0 con 0x55b556f4e4d0 2024-04-25T16:09:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.607+0000 7fec74166740 1 -- 172.21.15.31:0/2309263297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 msgr2=0x55b557319e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.607+0000 7fec74166740 1 --2- 172.21.15.31:0/2309263297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fec58004650 tx=0x7fec580c88c0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.607+0000 7fec74166740 1 -- 172.21.15.31:0/2309263297 shutdown_connections 2024-04-25T16:09:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.607+0000 7fec74166740 1 --2- 172.21.15.31:0/2309263297 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f6f1b0 0x55b55732bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.607+0000 7fec74166740 1 --2- 172.21.15.31:0/2309263297 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b557319e50 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.608+0000 7fec74166740 1 --2- 172.21.15.31:0/2309263297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f462f0 0x55b557319910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.608+0000 7fec74166740 1 -- 172.21.15.31:0/2309263297 >> 172.21.15.31:0/2309263297 conn(0x55b556f63300 msgr2=0x55b557317fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.610+0000 7fec74166740 1 -- 172.21.15.31:0/2309263297 shutdown_connections 2024-04-25T16:09:04.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.610+0000 7fec74166740 1 -- 172.21.15.31:0/2309263297 wait complete. 2024-04-25T16:09:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.611+0000 7fec74166740 1 -- start start 2024-04-25T16:09:04.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.611+0000 7fec74166740 1 Processor -- start 2024-04-25T16:09:04.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.612+0000 7fec74166740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b55731d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.612+0000 7fec74166740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b55731de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.612+0000 7fec74166740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55731ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.613+0000 7fec6659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b55731de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.613+0000 7fec6659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b55731de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56076/0 (socket says 172.21.15.31:56076) 2024-04-25T16:09:04.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.613+0000 7fec6659c640 1 -- 172.21.15.31:0/2258377697 learned_addr learned my addr 172.21.15.31:0/2258377697 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.613+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fec5400f820 con 0x55b556f462f0 2024-04-25T16:09:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.613+0000 7fec6759e640 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55731ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.613+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fec5400f150 con 0x55b556f4e4d0 2024-04-25T16:09:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.613+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fec5400c440 con 0x55b556f6f1b0 2024-04-25T16:09:04.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.614+0000 7fec74166740 10 client.0 _ll_register_callbacks cb 0x7fec60009c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.621+0000 7fec6759e640 1 -- 172.21.15.31:0/2258377697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 msgr2=0x55b55731de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.621+0000 7fec6759e640 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b55731de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.622 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.621+0000 7fec6759e640 1 -- 172.21.15.31:0/2258377697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 msgr2=0x55b55731d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.621+0000 7fec6759e640 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b55731d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.623 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.621+0000 7fec6759e640 1 -- 172.21.15.31:0/2258377697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b55731f300 con 0x55b556f6f1b0 2024-04-25T16:09:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.624+0000 7fec6759e640 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55731ed40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fec6001a650 tx=0x7fec60005c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3d12e0fc50cea63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.624+0000 7fec4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:04.626 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.625+0000 7fec4aff5640 1 -- 172.21.15.31:0/2258377697 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec60016070 con 0x55b556f6f1b0 2024-04-25T16:09:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.626+0000 7fec4aff5640 1 -- 172.21.15.31:0/2258377697 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fec60011d00 con 0x55b556f6f1b0 2024-04-25T16:09:04.627 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.626+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b55727cee0 con 0x55b556f6f1b0 2024-04-25T16:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.627+0000 7fec4aff5640 1 -- 172.21.15.31:0/2258377697 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fec6003a080 con 0x55b556f6f1b0 2024-04-25T16:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.628+0000 7fec74166740 10 client.26809 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:04.629 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.628+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b557320670 con 0x55b556f6f1b0 2024-04-25T16:09:04.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.629+0000 7fec4aff5640 1 -- 172.21.15.31:0/2258377697 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fec60004e30 con 0x55b556f6f1b0 2024-04-25T16:09:04.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.630+0000 7fec4aff5640 1 -- 172.21.15.31:0/2258377697 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fec6001ab70 con 0x55b556f6f1b0 2024-04-25T16:09:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.631+0000 7fec74166740 10 client.26809 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.631+0000 7fec74166740 10 client.26809 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.631+0000 7fec74166740 20 client.26809 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.632+0000 7fec74166740 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b556f031c0 0x55b557391eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.632+0000 7fec74166740 4 client.26809 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:09:04.633 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.632+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b55727cee0 con 0x55b556f031c0 2024-04-25T16:09:04.634 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.632+0000 7fec66d9d640 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b556f031c0 0x55b557391eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.636+0000 7fec4aff5640 10 client.26809 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:04.637 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:04.636+0000 7fec66d9d640 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b556f031c0 0x55b557391eb0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=877a6b71ce6079d1 server_cookie=8a08616e3049cced in_seq=0 out_seq=0 2024-04-25T16:09:04.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.716+0000 7fa04de80640 1 -- 172.21.15.31:0/3463130810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa04810a340 msgr2=0x7fa04810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.716+0000 7fa04de80640 1 --2- 172.21.15.31:0/3463130810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa04810a340 0x7fa04810ac50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fa038009900 tx=0x7fa03802a440 comp rx=0 tx=0).stop 2024-04-25T16:09:04.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.721+0000 7fa04de80640 1 -- 172.21.15.31:0/3463130810 shutdown_connections 2024-04-25T16:09:04.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.721+0000 7fa04de80640 1 --2- 172.21.15.31:0/3463130810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa04810fe90 0x7fa0481102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.721+0000 7fa04de80640 1 --2- 172.21.15.31:0/3463130810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04810b190 0x7fa04810b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.721+0000 7fa04de80640 1 --2- 172.21.15.31:0/3463130810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa04810a340 0x7fa04810ac50 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.721+0000 7fa04de80640 1 -- 172.21.15.31:0/3463130810 >> 172.21.15.31:0/3463130810 conn(0x7fa048070320 msgr2=0x7fa0480727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.728+0000 7fa04de80640 1 -- 172.21.15.31:0/3463130810 shutdown_connections 2024-04-25T16:09:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.728+0000 7fa04de80640 1 -- 172.21.15.31:0/3463130810 wait complete. 2024-04-25T16:09:04.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.729+0000 7fa04de80640 1 Processor -- start 2024-04-25T16:09:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa04de80640 1 -- start start 2024-04-25T16:09:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa04de80640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04810a340 0x7fa0481a8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa04de80640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa04810b190 0x7fa0481a9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa04de80640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa04810fe90 0x7fa0481b03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa04de80640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa04811e380 con 0x7fa04810fe90 2024-04-25T16:09:04.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa04de80640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa04811e200 con 0x7fa04810b190 2024-04-25T16:09:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa04de80640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa04811e500 con 0x7fa04810a340 2024-04-25T16:09:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa047fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa04810b190 0x7fa0481a9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa047fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa04810b190 0x7fa0481a9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56094/0 (socket says 172.21.15.31:56094) 2024-04-25T16:09:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.730+0000 7fa047fff640 1 -- 172.21.15.31:0/2586341574 learned_addr learned my addr 172.21.15.31:0/2586341574 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.731+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04810a340 0x7fa0481a8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.731+0000 7fa04ce7e640 1 -- 172.21.15.31:0/2586341574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa04810b190 msgr2=0x7fa0481a9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.731+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa04810b190 0x7fa0481a9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.731+0000 7fa04ce7e640 1 -- 172.21.15.31:0/2586341574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa04810fe90 msgr2=0x7fa0481b03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.731+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa04810fe90 0x7fa0481b03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.731+0000 7fa04ce7e640 1 -- 172.21.15.31:0/2586341574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa048111c90 con 0x7fa04810a340 2024-04-25T16:09:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.732+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04810a340 0x7fa0481a8f20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa038009850 tx=0x7fa0380308f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.737+0000 7fa045ffb640 1 -- 172.21.15.31:0/2586341574 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038040020 con 0x7fa04810a340 2024-04-25T16:09:04.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.737+0000 7fa04de80640 1 -- 172.21.15.31:0/2586341574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa048111f20 con 0x7fa04810a340 2024-04-25T16:09:04.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.737+0000 7fa04de80640 1 -- 172.21.15.31:0/2586341574 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa048112410 con 0x7fa04810a340 2024-04-25T16:09:04.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.737+0000 7fa045ffb640 1 -- 172.21.15.31:0/2586341574 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0380035f0 con 0x7fa04810a340 2024-04-25T16:09:04.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.737+0000 7fa045ffb640 1 -- 172.21.15.31:0/2586341574 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa038035070 con 0x7fa04810a340 2024-04-25T16:09:04.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.745+0000 7fa045ffb640 1 -- 172.21.15.31:0/2586341574 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa038043020 con 0x7fa04810a340 2024-04-25T16:09:04.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.747+0000 7fa045ffb640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0300afcf0 0x7fa0300b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.747+0000 7fa055f8f740 1 Processor -- start 2024-04-25T16:09:04.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.747+0000 7fa055f8f740 1 -- start start 2024-04-25T16:09:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.748+0000 7fa045ffb640 1 -- 172.21.15.31:0/2586341574 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa0381007a0 con 0x7fa04810a340 2024-04-25T16:09:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.748+0000 7fa047fff640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0300afcf0 0x7fa0300b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.748+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7de9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.748+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7def60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.748+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 0x55867b7dfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.748+0000 7fa055f8f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55867b742d80 con 0x55867b4134d0 2024-04-25T16:09:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.748+0000 7fa055f8f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55867b6fca20 con 0x55867b41b6b0 2024-04-25T16:09:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.748+0000 7fa055f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55867b73ed40 con 0x55867b43c390 2024-04-25T16:09:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.749+0000 7fa04ce7e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7de9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.749+0000 7fa04d67f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 0x55867b7dfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.749+0000 7fa04ce7e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7de9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45602/0 (socket says 172.21.15.31:45602) 2024-04-25T16:09:04.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.749+0000 7fa04ce7e640 1 -- 172.21.15.31:0/1712566962 learned_addr learned my addr 172.21.15.31:0/1712566962 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.749+0000 7fa047fff640 1 --2- 172.21.15.31:0/1712566962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7def60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.750+0000 7fa04d67f640 1 -- 172.21.15.31:0/1712566962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 msgr2=0x55867b7de9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.750+0000 7fa04d67f640 1 --2- 172.21.15.31:0/1712566962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7de9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.750+0000 7fa04d67f640 1 -- 172.21.15.31:0/1712566962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 msgr2=0x55867b7def60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.750+0000 7fa04d67f640 1 --2- 172.21.15.31:0/1712566962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7def60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.750+0000 7fa04d67f640 1 -- 172.21.15.31:0/1712566962 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55867b707c30 con 0x55867b41b6b0 2024-04-25T16:09:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.750+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/1712566962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7de9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:09:04.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.750+0000 7fa04d67f640 1 --2- 172.21.15.31:0/1712566962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 0x55867b7dfbf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa040009c40 tx=0x7fa04000e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f41b3f10812a7f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.755+0000 7fa036ffd640 1 -- 172.21.15.31:0/1712566962 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040016070 con 0x55867b41b6b0 2024-04-25T16:09:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.755+0000 7fa036ffd640 1 -- 172.21.15.31:0/1712566962 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa040005020 con 0x55867b41b6b0 2024-04-25T16:09:04.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.755+0000 7fa036ffd640 1 -- 172.21.15.31:0/1712566962 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0400116d0 con 0x55867b41b6b0 2024-04-25T16:09:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.757+0000 7fa055f8f740 1 -- 172.21.15.31:0/1712566962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 msgr2=0x55867b7dfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.757+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1712566962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 0x55867b7dfbf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa040009c40 tx=0x7fa04000e6c0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.758+0000 7fa055f8f740 1 -- 172.21.15.31:0/1712566962 shutdown_connections 2024-04-25T16:09:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.758+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1712566962 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 0x55867b7dfbf0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.758+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1712566962 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7def60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.758+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1712566962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7de9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.758+0000 7fa055f8f740 1 -- 172.21.15.31:0/1712566962 >> 172.21.15.31:0/1712566962 conn(0x55867b4304e0 msgr2=0x55867b7da1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.759+0000 7fa047fff640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0300afcf0 0x7fa0300b2200 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0058c0 tx=0x7fa03c00abd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.760+0000 7fa055f8f740 1 -- 172.21.15.31:0/1712566962 shutdown_connections 2024-04-25T16:09:04.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.760+0000 7fa055f8f740 1 -- 172.21.15.31:0/1712566962 wait complete. 2024-04-25T16:09:04.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.764+0000 7fa055f8f740 1 -- start start 2024-04-25T16:09:04.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.765+0000 7fa055f8f740 1 Processor -- start 2024-04-25T16:09:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.766+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b4134d0 0x55867b7e8560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.766+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 0x55867b7e8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.766+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b43c390 0x55867b7efa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.766+0000 7fa055f8f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55867b7e0700 con 0x55867b43c390 2024-04-25T16:09:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.766+0000 7fa055f8f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55867b6c9dc0 con 0x55867b4134d0 2024-04-25T16:09:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.766+0000 7fa055f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55867b7e0880 con 0x55867b41b6b0 2024-04-25T16:09:04.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.766+0000 7fa055f8f740 10 client.0 _ll_register_callbacks cb 0x55867b6e5600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:04.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.767+0000 7fa04ce7e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b4134d0 0x55867b7e8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.767+0000 7fa04ce7e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b4134d0 0x55867b7e8560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56114/0 (socket says 172.21.15.31:56114) 2024-04-25T16:09:04.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.767+0000 7fa04ce7e640 1 -- 172.21.15.31:0/1269954363 learned_addr learned my addr 172.21.15.31:0/1269954363 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.768+0000 7fa04d67f640 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b43c390 0x55867b7efa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.769+0000 7fa04d67f640 1 -- 172.21.15.31:0/1269954363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 msgr2=0x55867b7e8aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:04.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.769+0000 7fa04d67f640 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 0x55867b7e8aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.769+0000 7fa04d67f640 1 -- 172.21.15.31:0/1269954363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b4134d0 msgr2=0x55867b7e8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.769+0000 7fa04d67f640 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b4134d0 0x55867b7e8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.769+0000 7fa04d67f640 1 -- 172.21.15.31:0/1269954363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55867b4543f0 con 0x55867b43c390 2024-04-25T16:09:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.770+0000 7fa04d67f640 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b43c390 0x55867b7efa00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa040009c40 tx=0x7fa040005400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43906c2eef1c5cd7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.770+0000 7fa02bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:04.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.770+0000 7fa02bfff640 1 -- 172.21.15.31:0/1269954363 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040016040 con 0x55867b43c390 2024-04-25T16:09:04.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.771+0000 7fa02bfff640 1 -- 172.21.15.31:0/1269954363 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa040005700 con 0x55867b43c390 2024-04-25T16:09:04.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.771+0000 7fa055f8f740 1 -- 172.21.15.31:0/1269954363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa040005020 con 0x55867b43c390 2024-04-25T16:09:04.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.771+0000 7fa02bfff640 1 -- 172.21.15.31:0/1269954363 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa040004de0 con 0x55867b43c390 2024-04-25T16:09:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.772+0000 7fa055f8f740 10 client.17178 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:04.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.772+0000 7fa055f8f740 1 -- 172.21.15.31:0/1269954363 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55867b51e7d0 con 0x55867b43c390 2024-04-25T16:09:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.774+0000 7fa02bfff640 1 -- 172.21.15.31:0/1269954363 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa040005020 con 0x55867b43c390 2024-04-25T16:09:04.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.775+0000 7fa02bfff640 1 -- 172.21.15.31:0/1269954363 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa04000f040 con 0x55867b43c390 2024-04-25T16:09:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.775+0000 7fa055f8f740 10 client.17178 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.775+0000 7fa055f8f740 10 client.17178 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:09:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.775+0000 7fa055f8f740 20 client.17178 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.776+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55867b3faf60 0x55867b7f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.776+0000 7fa055f8f740 4 client.17178 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:04.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.776+0000 7fa055f8f740 1 -- 172.21.15.31:0/1269954363 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55867b750990 con 0x55867b3faf60 2024-04-25T16:09:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.778+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55867b3faf60 0x55867b7f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.783+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55867b3faf60 0x55867b7f5ee0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=db1940f5c18e1a57 server_cookie=fe113ac0c20f9589 in_seq=0 out_seq=0 2024-04-25T16:09:04.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.783+0000 7fa02bfff640 10 client.17178 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:09:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.784+0000 7fa02bfff640 1 -- 172.21.15.31:0/1269954363 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55867b750990 con 0x55867b3faf60 2024-04-25T16:09:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.785+0000 7fa02bfff640 10 client.17178 handle_command_reply: tid=0 2024-04-25T16:09:04.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.785+0000 7fa055f8f740 1 client.17178 shutdown 2024-04-25T16:09:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.785+0000 7fa055f8f740 1 -- 172.21.15.31:0/1269954363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b43c390 msgr2=0x55867b7efa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.785+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b43c390 0x55867b7efa00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fa040009c40 tx=0x7fa040005400 comp rx=0 tx=0).stop 2024-04-25T16:09:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.786+0000 7fa055f8f740 1 -- 172.21.15.31:0/1269954363 shutdown_connections 2024-04-25T16:09:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.786+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b43c390 0x55867b7efa00 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.786+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 0x55867b7e8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.786+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55867b3faf60 0x55867b7f5ee0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.786+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1269954363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b4134d0 0x55867b7e8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.786+0000 7fa055f8f740 1 -- 172.21.15.31:0/1269954363 >> 172.21.15.31:0/1269954363 conn(0x55867b4304e0 msgr2=0x55867b7d93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.786+0000 7fa02bfff640 0 client.17178 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:09:04.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.794+0000 7fa055f8f740 1 -- 172.21.15.31:0/1269954363 shutdown_connections 2024-04-25T16:09:04.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.796+0000 7fa055f8f740 1 -- 172.21.15.31:0/1269954363 wait complete. 2024-04-25T16:09:04.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.796+0000 7fa055f8f740 20 client.17178 trim_cache size 0 max 16384 2024-04-25T16:09:04.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.862+0000 7fa055f8f740 1 Processor -- start 2024-04-25T16:09:04.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.874+0000 7fa055f8f740 1 -- start start 2024-04-25T16:09:04.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.877+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7e6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.877+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 0x55867b7e7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.877+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7e7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.877+0000 7fa055f8f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c00e290 con 0x55867b4134d0 2024-04-25T16:09:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.877+0000 7fa055f8f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c00fdd0 con 0x55867b41b6b0 2024-04-25T16:09:04.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.877+0000 7fa055f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c00e960 con 0x55867b43c390 2024-04-25T16:09:04.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.879+0000 7fa04d67f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.880+0000 7fa047fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 0x55867b7e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.880+0000 7fa04d67f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7e7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45620/0 (socket says 172.21.15.31:45620) 2024-04-25T16:09:04.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.880+0000 7fa04d67f640 1 -- 172.21.15.31:0/898588139 learned_addr learned my addr 172.21.15.31:0/898588139 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.882+0000 7fa04d67f640 1 -- 172.21.15.31:0/898588139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 msgr2=0x55867b7e7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.882+0000 7fa04d67f640 1 --2- 172.21.15.31:0/898588139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 0x55867b7e7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.882+0000 7fa04d67f640 1 -- 172.21.15.31:0/898588139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 msgr2=0x55867b7e6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.882+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/898588139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7e6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.882+0000 7fa04d67f640 1 --2- 172.21.15.31:0/898588139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7e6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.882+0000 7fa04d67f640 1 -- 172.21.15.31:0/898588139 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55867b7cbba0 con 0x55867b43c390 2024-04-25T16:09:04.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.883+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/898588139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7e6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:09:04.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.893+0000 7fa04d67f640 1 --2- 172.21.15.31:0/898588139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7e7970 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa0400095c0 tx=0x7fa04000e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ae408ce166ca4d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.893+0000 7fa0367fc640 1 -- 172.21.15.31:0/898588139 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040016070 con 0x55867b43c390 2024-04-25T16:09:04.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.894+0000 7fa0367fc640 1 -- 172.21.15.31:0/898588139 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa040023770 con 0x55867b43c390 2024-04-25T16:09:04.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.907+0000 7fa0367fc640 1 -- 172.21.15.31:0/898588139 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa040011950 con 0x55867b43c390 2024-04-25T16:09:04.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.908+0000 7fa055f8f740 1 -- 172.21.15.31:0/898588139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 msgr2=0x55867b7e7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.908+0000 7fa055f8f740 1 --2- 172.21.15.31:0/898588139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7e7970 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa0400095c0 tx=0x7fa04000e6c0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.909+0000 7fa055f8f740 1 -- 172.21.15.31:0/898588139 shutdown_connections 2024-04-25T16:09:04.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.909+0000 7fa055f8f740 1 --2- 172.21.15.31:0/898588139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b43c390 0x55867b7e7970 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.909+0000 7fa055f8f740 1 --2- 172.21.15.31:0/898588139 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b41b6b0 0x55867b7e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.909+0000 7fa055f8f740 1 --2- 172.21.15.31:0/898588139 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7e6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.909+0000 7fa055f8f740 1 -- 172.21.15.31:0/898588139 >> 172.21.15.31:0/898588139 conn(0x55867b4304e0 msgr2=0x55867b7e3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.909+0000 7fa055f8f740 1 -- 172.21.15.31:0/898588139 shutdown_connections 2024-04-25T16:09:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.911+0000 7fa055f8f740 1 -- 172.21.15.31:0/898588139 wait complete. 2024-04-25T16:09:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.912+0000 7fa055f8f740 1 -- start start 2024-04-25T16:09:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.912+0000 7fa055f8f740 1 Processor -- start 2024-04-25T16:09:04.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.913+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7ebca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.913+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 0x55867b7ec1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.913+0000 7fa055f8f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b43c390 0x55867b85c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.913+0000 7fa055f8f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c00f700 con 0x55867b4134d0 2024-04-25T16:09:04.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.913+0000 7fa055f8f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c00f030 con 0x55867b43c390 2024-04-25T16:09:04.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.913+0000 7fa055f8f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa02c00c320 con 0x55867b41b6b0 2024-04-25T16:09:04.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.913+0000 7fa055f8f740 10 client.0 _ll_register_callbacks cb 0x7fa03c00af20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:04.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.914+0000 7fa047fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 0x55867b7ec1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.914+0000 7fa047fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 0x55867b7ec1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45634/0 (socket says 172.21.15.31:45634) 2024-04-25T16:09:04.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.914+0000 7fa047fff640 1 -- 172.21.15.31:0/1058825 learned_addr learned my addr 172.21.15.31:0/1058825 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.914+0000 7fa04d67f640 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b43c390 0x55867b85c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.915+0000 7fa047fff640 1 -- 172.21.15.31:0/1058825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b43c390 msgr2=0x55867b85c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.915+0000 7fa047fff640 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b43c390 0x55867b85c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.915+0000 7fa047fff640 1 -- 172.21.15.31:0/1058825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 msgr2=0x55867b7ebca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.915+0000 7fa047fff640 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7ebca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.915+0000 7fa047fff640 1 -- 172.21.15.31:0/1058825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55867b7ef1a0 con 0x55867b41b6b0 2024-04-25T16:09:04.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.916+0000 7fa047fff640 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 0x55867b7ec1e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa03c014410 tx=0x7fa03c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b3139618c98ded2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.923+0000 7fa02bfff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.923+0000 7fa02bfff640 1 -- 172.21.15.31:0/1058825 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa03c005020 con 0x55867b41b6b0 2024-04-25T16:09:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.923+0000 7fa02bfff640 1 -- 172.21.15.31:0/1058825 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa03c0051c0 con 0x55867b41b6b0 2024-04-25T16:09:04.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.936+0000 7fa055f8f740 1 -- 172.21.15.31:0/1058825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa040023770 con 0x55867b41b6b0 2024-04-25T16:09:04.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.937+0000 7fa02bfff640 1 -- 172.21.15.31:0/1058825 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa03c0053c0 con 0x55867b41b6b0 2024-04-25T16:09:04.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.937+0000 7fa055f8f740 10 client.26821 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:04.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.937+0000 7fa055f8f740 1 -- 172.21.15.31:0/1058825 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55867b85d620 con 0x55867b41b6b0 2024-04-25T16:09:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.947+0000 7fa02bfff640 1 -- 172.21.15.31:0/1058825 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa040023770 con 0x55867b41b6b0 2024-04-25T16:09:04.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.947+0000 7fa02bfff640 1 -- 172.21.15.31:0/1058825 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa03c0049a0 con 0x55867b41b6b0 2024-04-25T16:09:04.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.947+0000 7fa055f8f740 10 client.26821 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:04.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.947+0000 7fa055f8f740 10 client.26821 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:09:04.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.947+0000 7fa055f8f740 20 client.26821 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:04.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.947+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55867b3d03a0 0x55867b85ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:04.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.947+0000 7fa055f8f740 4 client.26821 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:04.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.947+0000 7fa055f8f740 1 -- 172.21.15.31:0/1058825 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55867b74a800 con 0x55867b3d03a0 2024-04-25T16:09:04.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.953+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55867b3d03a0 0x55867b85ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.973+0000 7fa04ce7e640 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55867b3d03a0 0x55867b85ffd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a90b0f73861d4c79 server_cookie=23f7300fe30c7b2b in_seq=0 out_seq=0 2024-04-25T16:09:04.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.973+0000 7fa02bfff640 10 client.26821 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:09:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.974+0000 7fa02bfff640 1 -- 172.21.15.31:0/1058825 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55867b74a800 con 0x55867b3d03a0 2024-04-25T16:09:04.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.974+0000 7fa02bfff640 10 client.26821 handle_command_reply: tid=0 2024-04-25T16:09:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.983+0000 7fa055f8f740 1 client.26821 shutdown 2024-04-25T16:09:04.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.983+0000 7fa055f8f740 1 -- 172.21.15.31:0/1058825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 msgr2=0x55867b7ec1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.983+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 0x55867b7ec1e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa03c014410 tx=0x7fa03c004730 comp rx=0 tx=0).stop 2024-04-25T16:09:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.984+0000 7fa055f8f740 1 -- 172.21.15.31:0/1058825 shutdown_connections 2024-04-25T16:09:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.984+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55867b3d03a0 0x55867b85ffd0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.984+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55867b43c390 0x55867b85c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.984+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55867b41b6b0 0x55867b7ec1e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.984+0000 7fa055f8f740 1 --2- 172.21.15.31:0/1058825 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55867b4134d0 0x55867b7ebca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:04.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.984+0000 7fa055f8f740 1 -- 172.21.15.31:0/1058825 >> 172.21.15.31:0/1058825 conn(0x55867b4304e0 msgr2=0x55867b7dcff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.986+0000 7fa02bfff640 0 client.26821 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:09:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.988+0000 7fa055f8f740 1 -- 172.21.15.31:0/1058825 shutdown_connections 2024-04-25T16:09:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.988+0000 7fa055f8f740 1 -- 172.21.15.31:0/1058825 wait complete. 2024-04-25T16:09:04.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:04.988+0000 7fa055f8f740 20 client.26821 trim_cache size 0 max 16384 2024-04-25T16:09:04.995 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:05.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.001+0000 7fa04de80640 1 -- 172.21.15.31:0/2586341574 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0300afcf0 msgr2=0x7fa0300b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.001+0000 7fa04de80640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0300afcf0 0x7fa0300b2200 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fa03c0058c0 tx=0x7fa03c00abd0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.001+0000 7fa04de80640 1 -- 172.21.15.31:0/2586341574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04810a340 msgr2=0x7fa0481a8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.001+0000 7fa04de80640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04810a340 0x7fa0481a8f20 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa038009850 tx=0x7fa0380308f0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.007+0000 7fa04de80640 1 -- 172.21.15.31:0/2586341574 shutdown_connections 2024-04-25T16:09:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.008+0000 7fa04de80640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0300afcf0 0x7fa0300b2200 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.008+0000 7fa04de80640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa04810fe90 0x7fa0481b03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.008+0000 7fa04de80640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa04810b190 0x7fa0481a9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.008+0000 7fa04de80640 1 --2- 172.21.15.31:0/2586341574 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa04810a340 0x7fa0481a8f20 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.008+0000 7fa04de80640 1 -- 172.21.15.31:0/2586341574 >> 172.21.15.31:0/2586341574 conn(0x7fa048070320 msgr2=0x7fa04810be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.009+0000 7fa04de80640 1 -- 172.21.15.31:0/2586341574 shutdown_connections 2024-04-25T16:09:05.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.009+0000 7fa04de80640 1 -- 172.21.15.31:0/2586341574 wait complete. 2024-04-25T16:09:05.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:09:05.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.494+0000 7f9e6759e640 1 -- 172.21.15.31:0/3298176773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e600a6d20 msgr2=0x7f9e600a4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.494+0000 7f9e6759e640 1 --2- 172.21.15.31:0/3298176773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e600a6d20 0x7f9e600a4fa0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009cb0 tx=0x7f9e5c02a780 comp rx=0 tx=0).stop 2024-04-25T16:09:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.497+0000 7f9e6759e640 1 -- 172.21.15.31:0/3298176773 shutdown_connections 2024-04-25T16:09:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.497+0000 7f9e6759e640 1 --2- 172.21.15.31:0/3298176773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e60009e90 0x7f9e6000c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.497+0000 7f9e6759e640 1 --2- 172.21.15.31:0/3298176773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e600a54e0 0x7f9e600a58f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.497+0000 7f9e6759e640 1 --2- 172.21.15.31:0/3298176773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e600a6d20 0x7f9e600a4fa0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.497+0000 7f9e6759e640 1 -- 172.21.15.31:0/3298176773 >> 172.21.15.31:0/3298176773 conn(0x7f9e600a09b0 msgr2=0x7f9e600a2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.497+0000 7f9e6759e640 1 -- 172.21.15.31:0/3298176773 shutdown_connections 2024-04-25T16:09:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.497+0000 7f9e6759e640 1 -- 172.21.15.31:0/3298176773 wait complete. 2024-04-25T16:09:05.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.499+0000 7f9e6759e640 1 Processor -- start 2024-04-25T16:09:05.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.499+0000 7f9e6759e640 1 -- start start 2024-04-25T16:09:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.499+0000 7f9e6759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e60009e90 0x7f9e60010f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.499+0000 7f9e6759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e600a54e0 0x7f9e60011480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.499+0000 7f9e6759e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e600a6d20 0x7f9e60156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.499+0000 7f9e6759e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6000fac0 con 0x7f9e60009e90 2024-04-25T16:09:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.499+0000 7f9e6759e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6000f940 con 0x7f9e600a6d20 2024-04-25T16:09:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.499+0000 7f9e6759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6000fc40 con 0x7f9e600a54e0 2024-04-25T16:09:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.500+0000 7f9e6659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e60009e90 0x7f9e60010f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.500+0000 7f9e6659c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e60009e90 0x7f9e60010f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56020/0 (socket says 172.21.15.31:56020) 2024-04-25T16:09:05.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.500+0000 7f9e6659c640 1 -- 172.21.15.31:0/2044455980 learned_addr learned my addr 172.21.15.31:0/2044455980 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:05.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.500+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e600a6d20 0x7f9e60156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.500+0000 7f9e6659c640 1 -- 172.21.15.31:0/2044455980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e600a54e0 msgr2=0x7f9e60011480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.500+0000 7f9e6659c640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e600a54e0 0x7f9e60011480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.500+0000 7f9e6659c640 1 -- 172.21.15.31:0/2044455980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e600a6d20 msgr2=0x7f9e60156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.500+0000 7f9e6659c640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e600a6d20 0x7f9e60156660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.500+0000 7f9e6659c640 1 -- 172.21.15.31:0/2044455980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e60156bf0 con 0x7f9e60009e90 2024-04-25T16:09:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.501+0000 7f9e6659c640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e60009e90 0x7f9e60010f40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0305a0 tx=0x7f9e5c0305d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.501+0000 7f9e577fe640 1 -- 172.21.15.31:0/2044455980 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c040020 con 0x7f9e60009e90 2024-04-25T16:09:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.501+0000 7f9e577fe640 1 -- 172.21.15.31:0/2044455980 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e5c038040 con 0x7f9e60009e90 2024-04-25T16:09:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.501+0000 7f9e577fe640 1 -- 172.21.15.31:0/2044455980 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c037070 con 0x7f9e60009e90 2024-04-25T16:09:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.502+0000 7f9e6759e640 1 -- 172.21.15.31:0/2044455980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e60156e80 con 0x7f9e60009e90 2024-04-25T16:09:05.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.502+0000 7f9e6759e640 1 -- 172.21.15.31:0/2044455980 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e60157230 con 0x7f9e60009e90 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.505+0000 7f9e73e40740 1 Processor -- start 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.506+0000 7f9e73e40740 1 -- start start 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.506+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc3b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.506+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8704d0 0x55c7fcc3bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.506+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8786b0 0x55c7fcc3cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.506+0000 7f9e73e40740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c7fcb9fd80 con 0x55c7fc8704d0 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.506+0000 7f9e73e40740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c7fcb59a20 con 0x55c7fc899390 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.506+0000 7f9e73e40740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c7fcb9bd40 con 0x55c7fc8786b0 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.507+0000 7f9e66d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc3b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.507+0000 7f9e66d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc3b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56154/0 (socket says 172.21.15.31:56154) 2024-04-25T16:09:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.507+0000 7f9e66d9d640 1 -- 172.21.15.31:0/3945805507 learned_addr learned my addr 172.21.15.31:0/3945805507 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:05.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.508+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/3945805507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8786b0 0x55c7fcc3cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.509+0000 7f9e66d9d640 1 -- 172.21.15.31:0/3945805507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8786b0 msgr2=0x55c7fcc3cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:05.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.509+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/3945805507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8786b0 0x55c7fcc3cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.509+0000 7f9e66d9d640 1 -- 172.21.15.31:0/3945805507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8704d0 msgr2=0x55c7fcc3bf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:09:05.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.509+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/3945805507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8704d0 0x55c7fcc3bf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.509+0000 7f9e66d9d640 1 -- 172.21.15.31:0/3945805507 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7fcb64c30 con 0x55c7fc899390 2024-04-25T16:09:05.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.509+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/3945805507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8786b0 0x55c7fcc3cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:09:05.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.511+0000 7f9e577fe640 1 -- 172.21.15.31:0/2044455980 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9e5c043020 con 0x7f9e60009e90 2024-04-25T16:09:05.513 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.512+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/3945805507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc3b9e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9e680520f0 tx=0x7f9e68064120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=318dd5b2675ad33a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.512+0000 7f9e577fe640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e500afba0 0x7f9e500b20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.512+0000 7f9e54ff9640 1 -- 172.21.15.31:0/3945805507 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e68082070 con 0x55c7fc899390 2024-04-25T16:09:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.512+0000 7f9e54ff9640 1 -- 172.21.15.31:0/3945805507 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e6806fd80 con 0x55c7fc899390 2024-04-25T16:09:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.512+0000 7f9e54ff9640 1 -- 172.21.15.31:0/3945805507 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e6806e440 con 0x55c7fc899390 2024-04-25T16:09:05.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.513+0000 7f9e577fe640 1 -- 172.21.15.31:0/2044455980 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9e5c0fe990 con 0x7f9e60009e90 2024-04-25T16:09:05.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.514+0000 7f9e6659c640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e500afba0 0x7f9e500b20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.514+0000 7f9e73e40740 1 -- 172.21.15.31:0/3945805507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 msgr2=0x55c7fcc3b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.515 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.514+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3945805507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc3b9e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9e680520f0 tx=0x7f9e68064120 comp rx=0 tx=0).stop 2024-04-25T16:09:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.516+0000 7f9e73e40740 1 -- 172.21.15.31:0/3945805507 shutdown_connections 2024-04-25T16:09:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.516+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3945805507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8786b0 0x55c7fcc3cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.516+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3945805507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8704d0 0x55c7fcc3bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.516+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3945805507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc3b9e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.516+0000 7f9e73e40740 1 -- 172.21.15.31:0/3945805507 >> 172.21.15.31:0/3945805507 conn(0x55c7fc88d4e0 msgr2=0x55c7fcc371f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:05.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.517+0000 7f9e6659c640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e500afba0 0x7f9e500b20b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c030570 tx=0x7f9e5c02ac80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.517+0000 7f9e73e40740 1 -- 172.21.15.31:0/3945805507 shutdown_connections 2024-04-25T16:09:05.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.517+0000 7f9e73e40740 1 -- 172.21.15.31:0/3945805507 wait complete. 2024-04-25T16:09:05.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.524+0000 7f9e73e40740 1 -- start start 2024-04-25T16:09:05.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.524+0000 7f9e73e40740 1 Processor -- start 2024-04-25T16:09:05.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.529+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 0x55c7fcc45590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.529+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8786b0 0x55c7fcc45ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.529+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc4ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.529+0000 7f9e73e40740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55c7fcc3d700 con 0x55c7fc8786b0 2024-04-25T16:09:05.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.530+0000 7f9e73e40740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55c7fcb26dc0 con 0x55c7fc899390 2024-04-25T16:09:05.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.530+0000 7f9e66d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 0x55c7fcc45590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.530+0000 7f9e66d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 0x55c7fcc45590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45658/0 (socket says 172.21.15.31:45658) 2024-04-25T16:09:05.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.530+0000 7f9e66d9d640 1 -- 172.21.15.31:0/218166311 learned_addr learned my addr 172.21.15.31:0/218166311 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:05.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.530+0000 7f9e73e40740 1 -- 172.21.15.31:0/218166311 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55c7fcc3d880 con 0x55c7fc8704d0 2024-04-25T16:09:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.530+0000 7f9e73e40740 10 client.0 _ll_register_callbacks cb 0x55c7fcb42600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:05.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.531+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8786b0 0x55c7fcc45ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.532+0000 7f9e65d9b640 1 -- 172.21.15.31:0/218166311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 msgr2=0x55c7fcc45590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.532+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 0x55c7fcc45590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.532+0000 7f9e65d9b640 1 -- 172.21.15.31:0/218166311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 msgr2=0x55c7fcc4ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.532+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc4ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.532+0000 7f9e65d9b640 1 -- 172.21.15.31:0/218166311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7fc8b13f0 con 0x55c7fc8786b0 2024-04-25T16:09:05.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.535+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8786b0 0x55c7fcc45ad0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009ea0 tx=0x7f9e58009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967b12ab7a238df9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:05.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.535+0000 7f9e4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.537+0000 7f9e4dffb640 1 -- 172.21.15.31:0/218166311 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e58015d20 con 0x55c7fc8786b0 2024-04-25T16:09:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.537+0000 7f9e4dffb640 1 -- 172.21.15.31:0/218166311 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9e58004020 con 0x55c7fc8786b0 2024-04-25T16:09:05.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.538+0000 7f9e73e40740 1 -- 172.21.15.31:0/218166311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e6806fd80 con 0x55c7fc8786b0 2024-04-25T16:09:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.544+0000 7f9e4dffb640 1 -- 172.21.15.31:0/218166311 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e58004220 con 0x55c7fc8786b0 2024-04-25T16:09:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.544+0000 7f9e73e40740 10 client.17187 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:05.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.544+0000 7f9e73e40740 1 -- 172.21.15.31:0/218166311 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c7fc97b7d0 con 0x55c7fc8786b0 2024-04-25T16:09:05.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.554+0000 7f9e4dffb640 1 -- 172.21.15.31:0/218166311 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e6806fd80 con 0x55c7fc8786b0 2024-04-25T16:09:05.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.554+0000 7f9e4dffb640 1 -- 172.21.15.31:0/218166311 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f9e5800f040 con 0x55c7fc8786b0 2024-04-25T16:09:05.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.556+0000 7f9e73e40740 10 client.17187 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:05.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.556+0000 7f9e73e40740 10 client.17187 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:09:05.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.556+0000 7f9e73e40740 20 client.17187 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:05.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.557+0000 7f9e73e40740 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c7fc857f60 0x55c7fcc53280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.557+0000 7f9e73e40740 4 client.17187 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:05.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.557+0000 7f9e73e40740 1 -- 172.21.15.31:0/218166311 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c7fcbad990 con 0x55c7fc857f60 2024-04-25T16:09:05.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.564+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c7fc857f60 0x55c7fcc53280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.576+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c7fc857f60 0x55c7fcc53280 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf72cec697026c18 server_cookie=97033e1e4fbbf6c3 in_seq=0 out_seq=0 2024-04-25T16:09:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.577+0000 7f9e4dffb640 10 client.17187 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:09:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.579+0000 7f9e4dffb640 1 -- 172.21.15.31:0/218166311 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c7fcbad990 con 0x55c7fc857f60 2024-04-25T16:09:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.579+0000 7f9e4dffb640 10 client.17187 handle_command_reply: tid=0 2024-04-25T16:09:05.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.579+0000 7f9e73e40740 1 client.17187 shutdown 2024-04-25T16:09:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.585+0000 7f9e73e40740 1 -- 172.21.15.31:0/218166311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8786b0 msgr2=0x55c7fcc45ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.586+0000 7f9e73e40740 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8786b0 0x55c7fcc45ad0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009ea0 tx=0x7f9e58009f90 comp rx=0 tx=0).stop 2024-04-25T16:09:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.586+0000 7f9e73e40740 1 -- 172.21.15.31:0/218166311 shutdown_connections 2024-04-25T16:09:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.586+0000 7f9e73e40740 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc4ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.586+0000 7f9e73e40740 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c7fc857f60 0x55c7fcc53280 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.586+0000 7f9e73e40740 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8786b0 0x55c7fcc45ad0 secure :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f9e58009ea0 tx=0x7f9e58009f90 comp rx=0 tx=0).stop 2024-04-25T16:09:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.586+0000 7f9e73e40740 1 --2- 172.21.15.31:0/218166311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 0x55c7fcc45590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.586+0000 7f9e73e40740 1 -- 172.21.15.31:0/218166311 >> 172.21.15.31:0/218166311 conn(0x55c7fc88d4e0 msgr2=0x55c7fcc363f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:05.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.587+0000 7f9e4dffb640 0 client.17187 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:09:05.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.587+0000 7f9e73e40740 1 -- 172.21.15.31:0/218166311 shutdown_connections 2024-04-25T16:09:05.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.587+0000 7f9e73e40740 1 -- 172.21.15.31:0/218166311 wait complete. 2024-04-25T16:09:05.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.587+0000 7f9e73e40740 20 client.17187 trim_cache size 0 max 16384 2024-04-25T16:09:05.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.625+0000 7f9e73e40740 1 Processor -- start 2024-04-25T16:09:05.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.625+0000 7f9e73e40740 1 -- start start 2024-04-25T16:09:05.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.626+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8704d0 0x55c7fcc3c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.626+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc4c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.626+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc36bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.626+0000 7f9e73e40740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6015df90 con 0x55c7fc8704d0 2024-04-25T16:09:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.626+0000 7f9e73e40740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6015fad0 con 0x55c7fc899390 2024-04-25T16:09:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.626+0000 7f9e73e40740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6015e660 con 0x55c7fc82d3a0 2024-04-25T16:09:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.627+0000 7f9e65d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc4c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.627+0000 7f9e65d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc4c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56182/0 (socket says 172.21.15.31:56182) 2024-04-25T16:09:05.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.627+0000 7f9e65d9b640 1 -- 172.21.15.31:0/2012686558 learned_addr learned my addr 172.21.15.31:0/2012686558 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:05.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.627+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/2012686558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc36bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.627+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/2012686558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8704d0 0x55c7fcc3c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.628+0000 7f9e65d9b640 1 -- 172.21.15.31:0/2012686558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc82d3a0 msgr2=0x55c7fcc36bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:05.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.628+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/2012686558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc36bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.628+0000 7f9e65d9b640 1 -- 172.21.15.31:0/2012686558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8704d0 msgr2=0x55c7fcc3c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:05.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.628+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/2012686558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8704d0 0x55c7fcc3c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.628+0000 7f9e65d9b640 1 -- 172.21.15.31:0/2012686558 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c7fcc28ba0 con 0x55c7fc899390 2024-04-25T16:09:05.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.631+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/2012686558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc4c9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9e58027600 tx=0x7f9e58009c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=131c99c7a8f5df6c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:05.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.632+0000 7f9e4ffff640 1 -- 172.21.15.31:0/2012686558 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e580043c0 con 0x55c7fc899390 2024-04-25T16:09:05.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.632+0000 7f9e4ffff640 1 -- 172.21.15.31:0/2012686558 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e58004560 con 0x55c7fc899390 2024-04-25T16:09:05.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.634+0000 7f9e73e40740 1 -- 172.21.15.31:0/2012686558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 msgr2=0x55c7fcc4c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.634+0000 7f9e73e40740 1 --2- 172.21.15.31:0/2012686558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc4c9f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9e58027600 tx=0x7f9e58009c60 comp rx=0 tx=0).stop 2024-04-25T16:09:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.660+0000 7f9e73e40740 1 -- 172.21.15.31:0/2012686558 shutdown_connections 2024-04-25T16:09:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.660+0000 7f9e73e40740 1 --2- 172.21.15.31:0/2012686558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc36bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.660+0000 7f9e73e40740 1 --2- 172.21.15.31:0/2012686558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fcc4c9f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.660+0000 7f9e73e40740 1 --2- 172.21.15.31:0/2012686558 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc8704d0 0x55c7fcc3c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.660+0000 7f9e73e40740 1 -- 172.21.15.31:0/2012686558 >> 172.21.15.31:0/2012686558 conn(0x55c7fc88d4e0 msgr2=0x55c7fcc402a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:05.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.670+0000 7f9e73e40740 1 -- 172.21.15.31:0/2012686558 shutdown_connections 2024-04-25T16:09:05.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.670+0000 7f9e73e40740 1 -- 172.21.15.31:0/2012686558 wait complete. 2024-04-25T16:09:05.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.671+0000 7f9e73e40740 1 -- start start 2024-04-25T16:09:05.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.671+0000 7f9e73e40740 1 Processor -- start 2024-04-25T16:09:05.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.672+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc49910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.672+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 0x55c7fcc49e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.672+0000 7f9e73e40740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fccb95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.672+0000 7f9e73e40740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6015f400 con 0x55c7fc82d3a0 2024-04-25T16:09:05.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.672+0000 7f9e73e40740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6015ed30 con 0x55c7fc899390 2024-04-25T16:09:05.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.672+0000 7f9e73e40740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9e6015bf70 con 0x55c7fc8704d0 2024-04-25T16:09:05.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.672+0000 7f9e73e40740 10 client.0 _ll_register_callbacks cb 0x7f9e58009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:05.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.674+0000 7f9e66d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc49910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.674+0000 7f9e66d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc49910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56052/0 (socket says 172.21.15.31:56052) 2024-04-25T16:09:05.676 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.674+0000 7f9e66d9d640 1 -- 172.21.15.31:0/3334821001 learned_addr learned my addr 172.21.15.31:0/3334821001 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.676+0000 7f9e66d9d640 1 -- 172.21.15.31:0/3334821001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 msgr2=0x55c7fcc49e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.676+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 0x55c7fcc49e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.676+0000 7f9e66d9d640 1 -- 172.21.15.31:0/3334821001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 msgr2=0x55c7fccb95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.676+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fccb95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.676+0000 7f9e66d9d640 1 -- 172.21.15.31:0/3334821001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c7fccb9b20 con 0x55c7fc82d3a0 2024-04-25T16:09:05.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.677+0000 7f9e66d9d640 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc49910 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9e680520f0 tx=0x7f9e6807c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc141b01e7dd2e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:05.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.683+0000 7f9e4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:05.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.683+0000 7f9e4dffb640 1 -- 172.21.15.31:0/3334821001 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e68082040 con 0x55c7fc82d3a0 2024-04-25T16:09:05.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.683+0000 7f9e73e40740 1 -- 172.21.15.31:0/3334821001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9e58004560 con 0x55c7fc82d3a0 2024-04-25T16:09:05.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.683+0000 7f9e4dffb640 1 -- 172.21.15.31:0/3334821001 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9e6807ca20 con 0x55c7fc82d3a0 2024-04-25T16:09:05.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.685+0000 7f9e4dffb640 1 -- 172.21.15.31:0/3334821001 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9e680a0050 con 0x55c7fc82d3a0 2024-04-25T16:09:05.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.685+0000 7f9e73e40740 10 client.17193 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:05.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.686+0000 7f9e73e40740 1 -- 172.21.15.31:0/3334821001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55c7fccbadd0 con 0x55c7fc82d3a0 2024-04-25T16:09:05.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.687+0000 7f9e4dffb640 1 -- 172.21.15.31:0/3334821001 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e58004560 con 0x55c7fc82d3a0 2024-04-25T16:09:05.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.687+0000 7f9e4dffb640 1 -- 172.21.15.31:0/3334821001 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f9e6809b450 con 0x55c7fc82d3a0 2024-04-25T16:09:05.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.687+0000 7f9e73e40740 10 client.17193 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:05.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.687+0000 7f9e73e40740 10 client.17193 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:09:05.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.687+0000 7f9e73e40740 20 client.17193 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:05.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.687+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c7fccbb760 0x55c7fcccede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:05.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.687+0000 7f9e73e40740 4 client.17193 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:05.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.687+0000 7f9e73e40740 1 -- 172.21.15.31:0/3334821001 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c7fcba7800 con 0x55c7fccbb760 2024-04-25T16:09:05.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.699+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c7fccbb760 0x55c7fcccede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:05.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.707+0000 7f9e65d9b640 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c7fccbb760 0x55c7fcccede0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=26f99a8f7e3ec742 server_cookie=be7a3faf9df256b in_seq=0 out_seq=0 2024-04-25T16:09:05.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.710+0000 7f9e4dffb640 10 client.17193 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:09:05.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.710+0000 7f9e4dffb640 1 -- 172.21.15.31:0/3334821001 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c7fcba7800 con 0x55c7fccbb760 2024-04-25T16:09:05.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.710+0000 7f9e4dffb640 10 client.17193 handle_command_reply: tid=0 2024-04-25T16:09:05.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.710+0000 7f9e73e40740 1 client.17193 shutdown 2024-04-25T16:09:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.722+0000 7f9e73e40740 1 -- 172.21.15.31:0/3334821001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc82d3a0 msgr2=0x55c7fcc49910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.722+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc49910 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f9e680520f0 tx=0x7f9e6807c520 comp rx=0 tx=0).stop 2024-04-25T16:09:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.723+0000 7f9e73e40740 1 -- 172.21.15.31:0/3334821001 shutdown_connections 2024-04-25T16:09:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.723+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55c7fc899390 0x55c7fccb95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.723+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55c7fc8704d0 0x55c7fcc49e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.723+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55c7fccbb760 0x55c7fcccede0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.723+0000 7f9e73e40740 1 --2- 172.21.15.31:0/3334821001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55c7fc82d3a0 0x55c7fcc49910 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.723+0000 7f9e73e40740 1 -- 172.21.15.31:0/3334821001 >> 172.21.15.31:0/3334821001 conn(0x55c7fc88d4e0 msgr2=0x55c7fcc39ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:05.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.723+0000 7f9e4dffb640 0 client.17193 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:09:05.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.729+0000 7f9e73e40740 1 -- 172.21.15.31:0/3334821001 shutdown_connections 2024-04-25T16:09:05.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.738+0000 7f9e73e40740 1 -- 172.21.15.31:0/3334821001 wait complete. 2024-04-25T16:09:05.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.742+0000 7f9e73e40740 20 client.17193 trim_cache size 0 max 16384 2024-04-25T16:09:05.745 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:05.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.745+0000 7f9e6759e640 1 -- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e500afba0 msgr2=0x7f9e500b20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.746+0000 7f9e6759e640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e500afba0 0x7f9e500b20b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c030570 tx=0x7f9e5c02ac80 comp rx=0 tx=0).stop 2024-04-25T16:09:05.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.746+0000 7f9e6759e640 1 -- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e60009e90 msgr2=0x7f9e60010f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:05.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.746+0000 7f9e6759e640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e60009e90 0x7f9e60010f40 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0305a0 tx=0x7f9e5c0305d0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.747+0000 7f9e6759e640 1 -- 172.21.15.31:0/2044455980 shutdown_connections 2024-04-25T16:09:05.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.749+0000 7f9e6759e640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9e500afba0 0x7f9e500b20b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.749+0000 7f9e6759e640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9e600a6d20 0x7f9e60156660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.750+0000 7f9e6759e640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e600a54e0 0x7f9e60011480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.750+0000 7f9e6759e640 1 --2- 172.21.15.31:0/2044455980 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9e60009e90 0x7f9e60010f40 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:05.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.750+0000 7f9e6759e640 1 -- 172.21.15.31:0/2044455980 >> 172.21.15.31:0/2044455980 conn(0x7f9e600a09b0 msgr2=0x7f9e600a2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:05.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.754+0000 7f9e6759e640 1 -- 172.21.15.31:0/2044455980 shutdown_connections 2024-04-25T16:09:05.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:05.754+0000 7f9e6759e640 1 -- 172.21.15.31:0/2044455980 wait complete. 2024-04-25T16:09:05.838 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:09:06.109 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:06 smithi031 ceph-mon[29140]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 741 MiB/s wr, 1.82k op/s 2024-04-25T16:09:06.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:06 smithi063 ceph-mon[32413]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 741 MiB/s wr, 1.82k op/s 2024-04-25T16:09:06.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:06 smithi186 ceph-mon[33258]: pgmap v585: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 741 MiB/s wr, 1.82k op/s 2024-04-25T16:09:06.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.417+0000 7fcdf851f640 1 -- 172.21.15.31:0/417550646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf010a520 msgr2=0x7fcdf010b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:06.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.417+0000 7fcdf851f640 1 --2- 172.21.15.31:0/417550646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf010a520 0x7fcdf010b220 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcde8005fd0 tx=0x7fcde802b3e0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.417+0000 7fcdf851f640 1 -- 172.21.15.31:0/417550646 shutdown_connections 2024-04-25T16:09:06.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.417+0000 7fcdf851f640 1 --2- 172.21.15.31:0/417550646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf0110460 0x7fcdf0112880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.418+0000 7fcdf851f640 1 --2- 172.21.15.31:0/417550646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf010b760 0x7fcdf010bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.418+0000 7fcdf851f640 1 --2- 172.21.15.31:0/417550646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf010a520 0x7fcdf010b220 secure :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fcde8005fd0 tx=0x7fcde802b3e0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.418+0000 7fcdf851f640 1 -- 172.21.15.31:0/417550646 >> 172.21.15.31:0/417550646 conn(0x7fcdf0070940 msgr2=0x7fcdf0072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:06.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.423+0000 7fcdf851f640 1 -- 172.21.15.31:0/417550646 shutdown_connections 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.424+0000 7fcdf851f640 1 -- 172.21.15.31:0/417550646 wait complete. 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.425+0000 7fcdf851f640 1 Processor -- start 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.425+0000 7fcdf851f640 1 -- start start 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf851f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf010b760 0x7fcdf01a9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf851f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf0110460 0x7fcdf01a9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf851f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf0078df0 0x7fcdf007b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf851f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf01157c0 con 0x7fcdf010b760 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf851f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf0115640 con 0x7fcdf0110460 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf851f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf0115940 con 0x7fcdf0078df0 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf6abc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf0078df0 0x7fcdf007b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf6abc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf0078df0 0x7fcdf007b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45712/0 (socket says 172.21.15.31:45712) 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf5aba640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf0110460 0x7fcdf01a9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.426+0000 7fcdf6abc640 1 -- 172.21.15.31:0/3967521224 learned_addr learned my addr 172.21.15.31:0/3967521224 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.427+0000 7fcdf6abc640 1 -- 172.21.15.31:0/3967521224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf0110460 msgr2=0x7fcdf01a9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.427+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf0110460 0x7fcdf01a9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.427+0000 7fcdf6abc640 1 -- 172.21.15.31:0/3967521224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf010b760 msgr2=0x7fcdf01a9070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.427+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf010b760 0x7fcdf01a9070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.427+0000 7fcdf6abc640 1 -- 172.21.15.31:0/3967521224 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf007b940 con 0x7fcdf0078df0 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.427+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf0078df0 0x7fcdf007b210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcde000e970 tx=0x7fcde000ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.428+0000 7fcde77fe640 1 -- 172.21.15.31:0/3967521224 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde000ccc0 con 0x7fcdf0078df0 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.428+0000 7fcde77fe640 1 -- 172.21.15.31:0/3967521224 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcde000ce60 con 0x7fcdf0078df0 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.428+0000 7fcde77fe640 1 -- 172.21.15.31:0/3967521224 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde0010430 con 0x7fcdf0078df0 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.428+0000 7fcdf851f640 1 -- 172.21.15.31:0/3967521224 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf007bc30 con 0x7fcdf0078df0 2024-04-25T16:09:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.428+0000 7fcdf851f640 1 -- 172.21.15.31:0/3967521224 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf007c130 con 0x7fcdf0078df0 2024-04-25T16:09:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.432+0000 7fcde77fe640 1 -- 172.21.15.31:0/3967521224 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcde00105d0 con 0x7fcdf0078df0 2024-04-25T16:09:06.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.433+0000 7fcde77fe640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcddc0afc40 0x7fcddc0b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.434+0000 7fcde77fe640 1 -- 172.21.15.31:0/3967521224 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcde00dfe40 con 0x7fcdf0078df0 2024-04-25T16:09:06.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.437+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcddc0afc40 0x7fcddc0b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.439+0000 7fcdff3d5740 1 Processor -- start 2024-04-25T16:09:06.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.439+0000 7fcdff3d5740 1 -- start start 2024-04-25T16:09:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.440+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.440+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71814d0 0x556ac755dda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.440+0000 7fcdf62bb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.440+0000 7fcdf62bb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56208/0 (socket says 172.21.15.31:56208) 2024-04-25T16:09:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.440+0000 7fcdf62bb640 1 -- 172.21.15.31:0/1958908872 learned_addr learned my addr 172.21.15.31:0/1958908872 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.440+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1958908872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71896b0 0x556ac755ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.440+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1958908872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556ac74b0d80 con 0x556ac71814d0 2024-04-25T16:09:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.440+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1958908872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556ac746aa20 con 0x556ac71aa390 2024-04-25T16:09:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.440+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1958908872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556ac74acd40 con 0x556ac71896b0 2024-04-25T16:09:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.441+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/1958908872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71896b0 0x556ac755ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.441+0000 7fcdf62bb640 1 -- 172.21.15.31:0/1958908872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71896b0 msgr2=0x556ac755ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:06.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.441+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/1958908872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71896b0 0x556ac755ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.441+0000 7fcdf62bb640 1 -- 172.21.15.31:0/1958908872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71814d0 msgr2=0x556ac755dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.441+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/1958908872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71814d0 0x556ac755dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.441+0000 7fcdf62bb640 1 -- 172.21.15.31:0/1958908872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac7475c30 con 0x556ac71aa390 2024-04-25T16:09:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.442+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/1958908872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755d820 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcde800a040 tx=0x7fcde803a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bedb9bec0c47e8d4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.443+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcddc0afc40 0x7fcddc0b2150 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcde8009000 tx=0x7fcde8000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.444+0000 7fcde4bfe640 1 -- 172.21.15.31:0/1958908872 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde8037070 con 0x556ac71aa390 2024-04-25T16:09:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.444+0000 7fcde4bfe640 1 -- 172.21.15.31:0/1958908872 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcde8063070 con 0x556ac71aa390 2024-04-25T16:09:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.446+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1958908872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 msgr2=0x556ac755d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:06.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.446+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1958908872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755d820 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcde800a040 tx=0x7fcde803a000 comp rx=0 tx=0).stop 2024-04-25T16:09:06.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.456+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1958908872 shutdown_connections 2024-04-25T16:09:06.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.456+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1958908872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71896b0 0x556ac755ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.456+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1958908872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71814d0 0x556ac755dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.456+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1958908872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755d820 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.456+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1958908872 >> 172.21.15.31:0/1958908872 conn(0x556ac719e4e0 msgr2=0x556ac7548020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.457+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1958908872 shutdown_connections 2024-04-25T16:09:06.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.457+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1958908872 wait complete. 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.457+0000 7fcdff3d5740 1 -- start start 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.457+0000 7fcdff3d5740 1 Processor -- start 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71814d0 0x556ac7553b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71896b0 0x556ac75540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdff3d5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556ac755f4e0 con 0x556ac71896b0 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdff3d5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556ac7437dc0 con 0x556ac71aa390 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdff3d5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556ac755f660 con 0x556ac71814d0 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdff3d5740 10 client.0 _ll_register_callbacks cb 0x556ac7453600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdf6abc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdf6abc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56224/0 (socket says 172.21.15.31:56224) 2024-04-25T16:09:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdf6abc640 1 -- 172.21.15.31:0/1717545810 learned_addr learned my addr 172.21.15.31:0/1717545810 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.460+0000 7fcdf5aba640 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71896b0 0x556ac75540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.461+0000 7fcdf6abc640 1 -- 172.21.15.31:0/1717545810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71814d0 msgr2=0x556ac7553b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.461+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71814d0 0x556ac7553b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.461+0000 7fcdf6abc640 1 -- 172.21.15.31:0/1717545810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71896b0 msgr2=0x556ac75540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.461+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71896b0 0x556ac75540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.461+0000 7fcdf6abc640 1 -- 172.21.15.31:0/1717545810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac71c23f0 con 0x556ac71aa390 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.461+0000 7fcdf5aba640 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71896b0 0x556ac75540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.461+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755b020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcde00a8030 tx=0x7fcde00a7e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5929da4230fd88e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.462+0000 7fcdd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.462+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/1717545810 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde00c1070 con 0x556ac71aa390 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.463+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/1717545810 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcde00b0f10 con 0x556ac71aa390 2024-04-25T16:09:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.463+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1717545810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcde8063070 con 0x556ac71aa390 2024-04-25T16:09:06.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.464+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/1717545810 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcde00ac6e0 con 0x556ac71aa390 2024-04-25T16:09:06.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.469+0000 7fcdff3d5740 10 client.26867 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:06.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.469+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1717545810 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556ac728c7d0 con 0x556ac71aa390 2024-04-25T16:09:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.475+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/1717545810 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcde8063070 con 0x556ac71aa390 2024-04-25T16:09:06.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.475+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/1717545810 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcde00a3ca0 con 0x556ac71aa390 2024-04-25T16:09:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.475+0000 7fcdff3d5740 10 client.26867 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.475+0000 7fcdff3d5740 10 client.26867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.475+0000 7fcdff3d5740 20 client.26867 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.475+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556ac7168f60 0x556ac75641d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:06.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.475+0000 7fcdff3d5740 4 client.26867 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:06.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.475+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1717545810 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556ac74be990 con 0x556ac7168f60 2024-04-25T16:09:06.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.476+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556ac7168f60 0x556ac75641d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:06.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.487+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556ac7168f60 0x556ac75641d0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce0ed97f299a3f90 server_cookie=5cfed7b14a32d1 in_seq=0 out_seq=0 2024-04-25T16:09:06.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:06.492+0000 7fcdd9ffb640 10 client.26867 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:08.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:08 smithi063 ceph-mon[32413]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 879 MiB/s wr, 2.12k op/s 2024-04-25T16:09:08.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:08 smithi186 ceph-mon[33258]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 879 MiB/s wr, 2.12k op/s 2024-04-25T16:09:08.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:08 smithi031 ceph-mon[29140]: pgmap v586: 97 pgs: 97 active+clean; 10 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 879 MiB/s wr, 2.12k op/s 2024-04-25T16:09:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.177+0000 7fec4aff5640 1 -- 172.21.15.31:0/2258377697 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b55727cee0 con 0x55b556f031c0 2024-04-25T16:09:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.177+0000 7fec4aff5640 10 client.26809 handle_command_reply: tid=0 2024-04-25T16:09:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 client.26809 shutdown 2024-04-25T16:09:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 msgr2=0x55b55731ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:10.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55731ed40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fec6001a650 tx=0x7fec60005c10 comp rx=0 tx=0).stop 2024-04-25T16:09:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 shutdown_connections 2024-04-25T16:09:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b556f031c0 0x55b557391eb0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b556f6f1b0 0x55b55731ed40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b556f4e4d0 0x55b55731de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 --2- 172.21.15.31:0/2258377697 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b556f462f0 0x55b55731d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 >> 172.21.15.31:0/2258377697 conn(0x55b556f63300 msgr2=0x55b557317fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec4aff5640 0 client.26809 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:10.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 shutdown_connections 2024-04-25T16:09:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fec74166740 1 -- 172.21.15.31:0/2258377697 wait complete. 2024-04-25T16:09:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/1717545810 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556ac74be990 con 0x556ac7168f60 2024-04-25T16:09:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fcdd9ffb640 10 client.26867 handle_command_reply: tid=0 2024-04-25T16:09:10.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.178+0000 7fcdff3d5740 1 client.26867 shutdown 2024-04-25T16:09:10.181 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fec74166740 20 client.26809 trim_cache size 0 max 16384 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1717545810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 msgr2=0x556ac755b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755b020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fcde00a8030 tx=0x7fcde00a7e40 comp rx=0 tx=0).stop 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1717545810 shutdown_connections 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556ac7168f60 0x556ac75641d0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71aa390 0x556ac755b020 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71896b0 0x556ac75540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/1717545810 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71814d0 0x556ac7553b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1717545810 >> 172.21.15.31:0/1717545810 conn(0x556ac719e4e0 msgr2=0x556ac7547040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdd9ffb640 0 client.26867 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1717545810 shutdown_connections 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 1 -- 172.21.15.31:0/1717545810 wait complete. 2024-04-25T16:09:10.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.179+0000 7fcdff3d5740 20 client.26867 trim_cache size 0 max 16384 2024-04-25T16:09:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:09:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:09:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:09:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:09:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.182+0000 7fec6c9d6640 1 -- 172.21.15.31:0/431743631 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec500afd20 msgr2=0x7fec500b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.182+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec500afd20 0x7fec500b2230 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x55b5573240e0 tx=0x7fec60028000 comp rx=0 tx=0).stop 2024-04-25T16:09:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.182+0000 7fec6c9d6640 1 -- 172.21.15.31:0/431743631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec6810b190 msgr2=0x7fec681a9430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:10.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.182+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec6810b190 0x7fec681a9430 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fec5802aa30 tx=0x7fec58003e90 comp rx=0 tx=0).stop 2024-04-25T16:09:10.185 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.184+0000 7fec6c9d6640 1 -- 172.21.15.31:0/431743631 shutdown_connections 2024-04-25T16:09:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.184+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fec500afd20 0x7fec500b2230 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.184+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fec6810feb0 0x7fec681b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.184+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec6810b190 0x7fec681a9430 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.185+0000 7fec6c9d6640 1 --2- 172.21.15.31:0/431743631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fec6810a340 0x7fec681a8ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.185+0000 7fec6c9d6640 1 -- 172.21.15.31:0/431743631 >> 172.21.15.31:0/431743631 conn(0x7fec68070330 msgr2=0x7fec6810be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.185+0000 7fec6c9d6640 1 -- 172.21.15.31:0/431743631 shutdown_connections 2024-04-25T16:09:10.186 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:10.185+0000 7fec6c9d6640 1 -- 172.21.15.31:0/431743631 wait complete. 2024-04-25T16:09:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.209+0000 7fcdff3d5740 1 Processor -- start 2024-04-25T16:09:10.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.209+0000 7fcdff3d5740 1 -- start start 2024-04-25T16:09:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.209+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71814d0 0x556ac75521a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.209+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac755af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.209+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71aa390 0x556ac75526e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.209+0000 7fcdff3d5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd400e590 con 0x556ac71814d0 2024-04-25T16:09:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.209+0000 7fcdff3d5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd40100d0 con 0x556ac71896b0 2024-04-25T16:09:10.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.209+0000 7fcdff3d5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd400ec60 con 0x556ac71aa390 2024-04-25T16:09:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.211+0000 7fcdf5aba640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac755af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:10.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.211+0000 7fcdf5aba640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac755af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56236/0 (socket says 172.21.15.31:56236) 2024-04-25T16:09:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.211+0000 7fcdf5aba640 1 -- 172.21.15.31:0/3518799444 learned_addr learned my addr 172.21.15.31:0/3518799444 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:10.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.211+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/3518799444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71aa390 0x556ac75526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.211+0000 7fcdf5aba640 1 -- 172.21.15.31:0/3518799444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71aa390 msgr2=0x556ac75526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.211+0000 7fcdf5aba640 1 --2- 172.21.15.31:0/3518799444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71aa390 0x556ac75526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.211+0000 7fcdf5aba640 1 -- 172.21.15.31:0/3518799444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71814d0 msgr2=0x556ac75521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.211+0000 7fcdf5aba640 1 --2- 172.21.15.31:0/3518799444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71814d0 0x556ac75521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.211+0000 7fcdf5aba640 1 -- 172.21.15.31:0/3518799444 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556ac7169190 con 0x556ac71896b0 2024-04-25T16:09:10.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.212+0000 7fcdf5aba640 1 --2- 172.21.15.31:0/3518799444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac755af40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00ab40 tx=0x7fcdec00af40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7575916a1d67075e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:10.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.221+0000 7fcddbfff640 1 -- 172.21.15.31:0/3518799444 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec004020 con 0x556ac71896b0 2024-04-25T16:09:10.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.221+0000 7fcddbfff640 1 -- 172.21.15.31:0/3518799444 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdec0041c0 con 0x556ac71896b0 2024-04-25T16:09:10.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.224+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3518799444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 msgr2=0x556ac755af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:10.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.224+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3518799444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac755af40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fcdec00ab40 tx=0x7fcdec00af40 comp rx=0 tx=0).stop 2024-04-25T16:09:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.227+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3518799444 shutdown_connections 2024-04-25T16:09:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.227+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3518799444 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71aa390 0x556ac75526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.227+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3518799444 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac755af40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.227+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3518799444 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71814d0 0x556ac75521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.227+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3518799444 >> 172.21.15.31:0/3518799444 conn(0x556ac719e4e0 msgr2=0x556ac754ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:10.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.227+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3518799444 shutdown_connections 2024-04-25T16:09:10.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.227+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3518799444 wait complete. 2024-04-25T16:09:10.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.228+0000 7fcdff3d5740 1 -- start start 2024-04-25T16:09:10.229 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.228+0000 7fcdff3d5740 1 Processor -- start 2024-04-25T16:09:10.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71814d0 0x556ac7557d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:10.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac7558270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:10.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdff3d5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71aa390 0x556ac75ca530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdff3d5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd400fa00 con 0x556ac71aa390 2024-04-25T16:09:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdff3d5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd400f330 con 0x556ac71896b0 2024-04-25T16:09:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdff3d5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd400c570 con 0x556ac71814d0 2024-04-25T16:09:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdff3d5740 10 client.0 _ll_register_callbacks cb 0x7fcde00a8190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdf5aba640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac7558270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdf5aba640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac7558270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56248/0 (socket says 172.21.15.31:56248) 2024-04-25T16:09:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdf5aba640 1 -- 172.21.15.31:0/3437911922 learned_addr learned my addr 172.21.15.31:0/3437911922 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:10.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.229+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71814d0 0x556ac7557d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.233+0000 7fcdf5aba640 1 -- 172.21.15.31:0/3437911922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71814d0 msgr2=0x556ac7557d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.233+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71aa390 0x556ac75ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.233+0000 7fcdf5aba640 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71814d0 0x556ac7557d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.233+0000 7fcdf5aba640 1 -- 172.21.15.31:0/3437911922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71aa390 msgr2=0x556ac75ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:10.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.233+0000 7fcdf5aba640 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71aa390 0x556ac75ca530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.233+0000 7fcdf5aba640 1 -- 172.21.15.31:0/3437911922 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556ac755b1c0 con 0x556ac71896b0 2024-04-25T16:09:10.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.233+0000 7fcdf5aba640 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac7558270 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcdec003fd0 tx=0x7fcdec0045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41f303d86b37eeae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:10.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.234+0000 7fcdf6abc640 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71aa390 0x556ac75ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:09:10.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.234+0000 7fcdd9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:10.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.235+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/3437911922 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdec028020 con 0x556ac71896b0 2024-04-25T16:09:10.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.235+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/3437911922 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcdec012b70 con 0x556ac71896b0 2024-04-25T16:09:10.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.236+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3437911922 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcdec0041c0 con 0x556ac71896b0 2024-04-25T16:09:10.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.239+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/3437911922 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcdec01f040 con 0x556ac71896b0 2024-04-25T16:09:10.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.240+0000 7fcdff3d5740 10 client.26873 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:10.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.240+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3437911922 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556ac75cbad0 con 0x556ac71896b0 2024-04-25T16:09:10.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.247+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/3437911922 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcdec0041c0 con 0x556ac71896b0 2024-04-25T16:09:10.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.248+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/3437911922 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcdec024790 con 0x556ac71896b0 2024-04-25T16:09:10.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.249+0000 7fcdff3d5740 10 client.26873 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:10.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.249+0000 7fcdff3d5740 10 client.26873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:10.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.249+0000 7fcdff3d5740 20 client.26873 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:10.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.249+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556ac713e3a0 0x556ac75ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:10.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.249+0000 7fcdff3d5740 4 client.26873 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:10.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.249+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3437911922 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556ac74b8800 con 0x556ac713e3a0 2024-04-25T16:09:10.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.254+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556ac713e3a0 0x556ac75ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.255+0000 7fcdf62bb640 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556ac713e3a0 0x556ac75ce420 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=970a9f9618be49d5 server_cookie=ac2d2efbe41d46a5 in_seq=0 out_seq=0 2024-04-25T16:09:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.255+0000 7fcdd9ffb640 10 client.26873 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:10.268 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:10.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.262+0000 7fcdd9ffb640 1 -- 172.21.15.31:0/3437911922 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556ac74b8800 con 0x556ac713e3a0 2024-04-25T16:09:10.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.262+0000 7fcdd9ffb640 10 client.26873 handle_command_reply: tid=0 2024-04-25T16:09:10.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.262+0000 7fcdff3d5740 1 client.26873 shutdown 2024-04-25T16:09:10.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3437911922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 msgr2=0x556ac7558270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac7558270 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcdec003fd0 tx=0x7fcdec0045a0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3437911922 shutdown_connections 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556ac713e3a0 0x556ac75ce420 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556ac71aa390 0x556ac75ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556ac71896b0 0x556ac7558270 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 --2- 172.21.15.31:0/3437911922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556ac71814d0 0x556ac7557d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3437911922 >> 172.21.15.31:0/3437911922 conn(0x556ac719e4e0 msgr2=0x556ac7551e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdd9ffb640 0 client.26873 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3437911922 shutdown_connections 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.264+0000 7fcdff3d5740 1 -- 172.21.15.31:0/3437911922 wait complete. 2024-04-25T16:09:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.265+0000 7fcdff3d5740 20 client.26873 trim_cache size 0 max 16384 2024-04-25T16:09:10.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.270+0000 7fcdf851f640 1 -- 172.21.15.31:0/3967521224 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcddc0afc40 msgr2=0x7fcddc0b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:10.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.270+0000 7fcdf851f640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcddc0afc40 0x7fcddc0b2150 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fcde8009000 tx=0x7fcde8000f40 comp rx=0 tx=0).stop 2024-04-25T16:09:10.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.270+0000 7fcdf851f640 1 -- 172.21.15.31:0/3967521224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf0078df0 msgr2=0x7fcdf007b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:10.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.270+0000 7fcdf851f640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf0078df0 0x7fcdf007b210 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcde000e970 tx=0x7fcde000ed70 comp rx=0 tx=0).stop 2024-04-25T16:09:10.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.271+0000 7fcdf851f640 1 -- 172.21.15.31:0/3967521224 shutdown_connections 2024-04-25T16:09:10.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.271+0000 7fcdf851f640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcddc0afc40 0x7fcddc0b2150 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.271+0000 7fcdf851f640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf0078df0 0x7fcdf007b210 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.271+0000 7fcdf851f640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcdf0110460 0x7fcdf01a9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.271+0000 7fcdf851f640 1 --2- 172.21.15.31:0/3967521224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcdf010b760 0x7fcdf01a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:10.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.271+0000 7fcdf851f640 1 -- 172.21.15.31:0/3967521224 >> 172.21.15.31:0/3967521224 conn(0x7fcdf0070940 msgr2=0x7fcdf010c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:10.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.273+0000 7fcdf851f640 1 -- 172.21.15.31:0/3967521224 shutdown_connections 2024-04-25T16:09:10.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:10.274+0000 7fcdf851f640 1 -- 172.21.15.31:0/3967521224 wait complete. 2024-04-25T16:09:10.327 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:10 smithi031 ceph-mon[29140]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 831 MiB/s wr, 2.06k op/s 2024-04-25T16:09:10.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:10 smithi063 ceph-mon[32413]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 831 MiB/s wr, 2.06k op/s 2024-04-25T16:09:10.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:10 smithi186 ceph-mon[33258]: pgmap v587: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 831 MiB/s wr, 2.06k op/s 2024-04-25T16:09:11.327 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:09:11.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:09:11.328 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:09:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.960+0000 7fd949ad9640 1 -- 172.21.15.31:0/287288683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd94410fef0 msgr2=0x7fd944110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:11.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.960+0000 7fd949ad9640 1 --2- 172.21.15.31:0/287288683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd94410fef0 0x7fd944110300 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd938009900 tx=0x7fd93802a400 comp rx=0 tx=0).stop 2024-04-25T16:09:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.960+0000 7fd949ad9640 1 -- 172.21.15.31:0/287288683 shutdown_connections 2024-04-25T16:09:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.960+0000 7fd949ad9640 1 --2- 172.21.15.31:0/287288683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd94410fef0 0x7fd944110300 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.960+0000 7fd949ad9640 1 --2- 172.21.15.31:0/287288683 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd94410b1d0 0x7fd94410b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.960+0000 7fd949ad9640 1 --2- 172.21.15.31:0/287288683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94410a3e0 0x7fd94410ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:11.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.960+0000 7fd949ad9640 1 -- 172.21.15.31:0/287288683 >> 172.21.15.31:0/287288683 conn(0x7fd944070400 msgr2=0x7fd944072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:11.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.963+0000 7fd949ad9640 1 -- 172.21.15.31:0/287288683 shutdown_connections 2024-04-25T16:09:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.963+0000 7fd949ad9640 1 -- 172.21.15.31:0/287288683 wait complete. 2024-04-25T16:09:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.965+0000 7fd949ad9640 1 Processor -- start 2024-04-25T16:09:11.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.973+0000 7fd949ad9640 1 -- start start 2024-04-25T16:09:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.974+0000 7fd949ad9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd94410a3e0 0x7fd9441a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.974+0000 7fd949ad9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94410b1d0 0x7fd9441a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.975+0000 7fd949ad9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd94410fef0 0x7fd9441b0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.975+0000 7fd949ad9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd94411e2b0 con 0x7fd94410fef0 2024-04-25T16:09:11.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.975+0000 7fd949ad9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd94411e130 con 0x7fd94410a3e0 2024-04-25T16:09:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.975+0000 7fd949ad9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd94411e430 con 0x7fd94410b1d0 2024-04-25T16:09:11.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.976+0000 7fd942ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94410b1d0 0x7fd9441a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.976+0000 7fd942ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94410b1d0 0x7fd9441a93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45774/0 (socket says 172.21.15.31:45774) 2024-04-25T16:09:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.976+0000 7fd942ffd640 1 -- 172.21.15.31:0/1404493076 learned_addr learned my addr 172.21.15.31:0/1404493076 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:11.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.976+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd94410a3e0 0x7fd9441a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:11.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.978+0000 7fd943fff640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd94410fef0 0x7fd9441b0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:11.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.979+0000 7fd9437fe640 1 -- 172.21.15.31:0/1404493076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94410b1d0 msgr2=0x7fd9441a93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:11.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.979+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94410b1d0 0x7fd9441a93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:11.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.979+0000 7fd9437fe640 1 -- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd94410fef0 msgr2=0x7fd9441b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.979+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd94410fef0 0x7fd9441b0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:11.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.979+0000 7fd9437fe640 1 -- 172.21.15.31:0/1404493076 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9441b09c0 con 0x7fd94410a3e0 2024-04-25T16:09:11.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.983+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd94410a3e0 0x7fd9441a8e80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd944074760 tx=0x7fd93c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:11.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.984+0000 7fd940ff9640 1 -- 172.21.15.31:0/1404493076 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c00cd50 con 0x7fd94410a3e0 2024-04-25T16:09:11.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.984+0000 7fd949ad9640 1 -- 172.21.15.31:0/1404493076 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9441b0cb0 con 0x7fd94410a3e0 2024-04-25T16:09:11.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.984+0000 7fd940ff9640 1 -- 172.21.15.31:0/1404493076 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd93c011920 con 0x7fd94410a3e0 2024-04-25T16:09:11.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.984+0000 7fd940ff9640 1 -- 172.21.15.31:0/1404493076 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93c01a940 con 0x7fd94410a3e0 2024-04-25T16:09:11.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.984+0000 7fd949ad9640 1 -- 172.21.15.31:0/1404493076 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9441b1220 con 0x7fd94410a3e0 2024-04-25T16:09:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.995+0000 7fd950981740 1 Processor -- start 2024-04-25T16:09:11.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.995+0000 7fd950981740 1 -- start start 2024-04-25T16:09:11.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:11.996+0000 7fd950981740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f685f410 0x55b1f6c00660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.000+0000 7fd943fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f685f410 0x55b1f6c00660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.000+0000 7fd950981740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6c00be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.000+0000 7fd950981740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f683e730 0x55b1f6c07ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.001+0000 7fd943fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f685f410 0x55b1f6c00660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45784/0 (socket says 172.21.15.31:45784) 2024-04-25T16:09:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.001+0000 7fd943fff640 1 -- 172.21.15.31:0/4149530127 learned_addr learned my addr 172.21.15.31:0/4149530127 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.001+0000 7fd940ff9640 1 -- 172.21.15.31:0/1404493076 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd93c011470 con 0x7fd94410a3e0 2024-04-25T16:09:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.001+0000 7fd950981740 1 -- 172.21.15.31:0/4149530127 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b1f6b65b00 con 0x55b1f6836550 2024-04-25T16:09:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.002+0000 7fd950981740 1 -- 172.21.15.31:0/4149530127 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b1f6b01920 con 0x55b1f683e730 2024-04-25T16:09:12.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.002+0000 7fd950981740 1 -- 172.21.15.31:0/4149530127 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b1f6b61ac0 con 0x55b1f685f410 2024-04-25T16:09:12.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.003+0000 7fd942ffd640 1 --2- 172.21.15.31:0/4149530127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f683e730 0x55b1f6c07ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.004+0000 7fd943fff640 1 -- 172.21.15.31:0/4149530127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f683e730 msgr2=0x55b1f6c07ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.005+0000 7fd940ff9640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd92c0afbb0 0x7fd92c0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.005+0000 7fd940ff9640 1 -- 172.21.15.31:0/1404493076 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd93c0df0a0 con 0x7fd94410a3e0 2024-04-25T16:09:12.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.004+0000 7fd943fff640 1 --2- 172.21.15.31:0/4149530127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f683e730 0x55b1f6c07ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.006+0000 7fd943fff640 1 -- 172.21.15.31:0/4149530127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 msgr2=0x55b1f6c00be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.006+0000 7fd943fff640 1 --2- 172.21.15.31:0/4149530127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6c00be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.006+0000 7fd943fff640 1 -- 172.21.15.31:0/4149530127 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1f693f7e0 con 0x55b1f685f410 2024-04-25T16:09:12.009 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.007+0000 7fd943fff640 1 --2- 172.21.15.31:0/4149530127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f685f410 0x55b1f6c00660 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd93802a9f0 tx=0x7fd938004310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cd5d68d829b2f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:12.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.009+0000 7fd929ffb640 1 -- 172.21.15.31:0/4149530127 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938035030 con 0x55b1f685f410 2024-04-25T16:09:12.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.010+0000 7fd929ffb640 1 -- 172.21.15.31:0/4149530127 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9380054f0 con 0x55b1f685f410 2024-04-25T16:09:12.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.010+0000 7fd929ffb640 1 -- 172.21.15.31:0/4149530127 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938003850 con 0x55b1f685f410 2024-04-25T16:09:12.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.014+0000 7fd950981740 1 -- 172.21.15.31:0/4149530127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f685f410 msgr2=0x55b1f6c00660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.014+0000 7fd950981740 1 --2- 172.21.15.31:0/4149530127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f685f410 0x55b1f6c00660 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd93802a9f0 tx=0x7fd938004310 comp rx=0 tx=0).stop 2024-04-25T16:09:12.017 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.015+0000 7fd943fff640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd92c0afbb0 0x7fd92c0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.018+0000 7fd950981740 1 -- 172.21.15.31:0/4149530127 shutdown_connections 2024-04-25T16:09:12.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.018+0000 7fd950981740 1 --2- 172.21.15.31:0/4149530127 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f683e730 0x55b1f6c07ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.018+0000 7fd950981740 1 --2- 172.21.15.31:0/4149530127 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6c00be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.018+0000 7fd950981740 1 --2- 172.21.15.31:0/4149530127 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f685f410 0x55b1f6c00660 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.018+0000 7fd950981740 1 -- 172.21.15.31:0/4149530127 >> 172.21.15.31:0/4149530127 conn(0x55b1f6853560 msgr2=0x55b1f6bfbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:12.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.019+0000 7fd950981740 1 -- 172.21.15.31:0/4149530127 shutdown_connections 2024-04-25T16:09:12.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.019+0000 7fd950981740 1 -- 172.21.15.31:0/4149530127 wait complete. 2024-04-25T16:09:12.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.022+0000 7fd950981740 1 -- start start 2024-04-25T16:09:12.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.024+0000 7fd950981740 1 Processor -- start 2024-04-25T16:09:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.026+0000 7fd950981740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6c0a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.027 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.026+0000 7fd943fff640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd92c0afbb0 0x7fd92c0b20c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd93802a9f0 tx=0x7fd938004310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.026+0000 7fd950981740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c0aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.027+0000 7fd950981740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 0x55b1f6c119d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.027+0000 7fd950981740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b1f6c08520 con 0x55b1f6836550 2024-04-25T16:09:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.027+0000 7fd950981740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b1f6aff5d0 con 0x55b1f685f410 2024-04-25T16:09:12.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.027+0000 7fd950981740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b1f6c086a0 con 0x55b1f683e730 2024-04-25T16:09:12.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.027+0000 7fd942ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c0aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.028+0000 7fd942ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c0aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45788/0 (socket says 172.21.15.31:45788) 2024-04-25T16:09:12.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.028+0000 7fd942ffd640 1 -- 172.21.15.31:0/1127406524 learned_addr learned my addr 172.21.15.31:0/1127406524 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:12.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.028+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6c0a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.029+0000 7fd943fff640 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 0x55b1f6c119d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.029+0000 7fd942ffd640 1 -- 172.21.15.31:0/1127406524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 msgr2=0x55b1f6c119d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.029+0000 7fd942ffd640 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 0x55b1f6c119d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.029+0000 7fd942ffd640 1 -- 172.21.15.31:0/1127406524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 msgr2=0x55b1f6c0a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.029+0000 7fd942ffd640 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6c0a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.029+0000 7fd942ffd640 1 -- 172.21.15.31:0/1127406524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1f699ff70 con 0x55b1f683e730 2024-04-25T16:09:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.030+0000 7fd942ffd640 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c0aa70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd9340115b0 tx=0x7fd934011a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be51455619fc1eea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.031+0000 7fd950981740 10 client.0 _ll_register_callbacks cb 0x55b1f6aed080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.031+0000 7fd950981740 1 -- 172.21.15.31:0/1127406524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd9380054f0 con 0x55b1f683e730 2024-04-25T16:09:12.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.031+0000 7fd9277f6640 10 client.26854 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:12.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.031+0000 7fd9277f6640 1 -- 172.21.15.31:0/1127406524 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93400f6f0 con 0x55b1f683e730 2024-04-25T16:09:12.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.032+0000 7fd9277f6640 1 -- 172.21.15.31:0/1127406524 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd93400f890 con 0x55b1f683e730 2024-04-25T16:09:12.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.048+0000 7fd9277f6640 1 -- 172.21.15.31:0/1127406524 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd93400fa90 con 0x55b1f683e730 2024-04-25T16:09:12.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.049+0000 7fd950981740 10 client.26854 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:12.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.049+0000 7fd950981740 1 -- 172.21.15.31:0/1127406524 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55b1f6837170 con 0x55b1f683e730 2024-04-25T16:09:12.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.049+0000 7fd9277f6640 1 -- 172.21.15.31:0/1127406524 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd93400fc30 con 0x55b1f683e730 2024-04-25T16:09:12.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.050+0000 7fd950981740 10 client.26854 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:12.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.050+0000 7fd950981740 10 client.26854 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:12.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.050+0000 7fd950981740 20 client.26854 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:12.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.050+0000 7fd950981740 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b1f681e1f0 0x55b1f6c18210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.050+0000 7fd950981740 4 client.26854 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:12.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.050+0000 7fd950981740 1 -- 172.21.15.31:0/1127406524 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd9380054f0 con 0x55b1f681e1f0 2024-04-25T16:09:12.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.050+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b1f681e1f0 0x55b1f6c18210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.052+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b1f681e1f0 0x55b1f6c18210 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92b46a24ba79ffb9 server_cookie=d645bf15a0f4a8fe in_seq=0 out_seq=0 2024-04-25T16:09:12.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.052+0000 7fd9277f6640 10 client.26854 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:12.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.055+0000 7fd9277f6640 1 -- 172.21.15.31:0/1127406524 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fd9380054f0 con 0x55b1f681e1f0 2024-04-25T16:09:12.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.055+0000 7fd9277f6640 10 client.26854 handle_command_reply: tid=0 2024-04-25T16:09:12.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.055+0000 7fd950981740 1 client.26854 shutdown 2024-04-25T16:09:12.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.056+0000 7fd950981740 1 -- 172.21.15.31:0/1127406524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 msgr2=0x55b1f6c0aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.056+0000 7fd950981740 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c0aa70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fd9340115b0 tx=0x7fd934011a70 comp rx=0 tx=0).stop 2024-04-25T16:09:12.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.057+0000 7fd950981740 1 -- 172.21.15.31:0/1127406524 shutdown_connections 2024-04-25T16:09:12.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.057+0000 7fd950981740 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b1f681e1f0 0x55b1f6c18210 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.057+0000 7fd950981740 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 0x55b1f6c119d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.057+0000 7fd950981740 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c0aa70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.057+0000 7fd950981740 1 --2- 172.21.15.31:0/1127406524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6c0a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.057+0000 7fd950981740 1 -- 172.21.15.31:0/1127406524 >> 172.21.15.31:0/1127406524 conn(0x55b1f6853560 msgr2=0x55b1f6bfaf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.057+0000 7fd9277f6640 0 client.26854 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.058+0000 7fd950981740 1 -- 172.21.15.31:0/1127406524 shutdown_connections 2024-04-25T16:09:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.058+0000 7fd950981740 1 -- 172.21.15.31:0/1127406524 wait complete. 2024-04-25T16:09:12.059 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.058+0000 7fd950981740 20 client.26854 trim_cache size 0 max 16384 2024-04-25T16:09:12.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.117+0000 7fd950981740 1 Processor -- start 2024-04-25T16:09:12.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.117+0000 7fd950981740 1 -- start start 2024-04-25T16:09:12.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.118+0000 7fd950981740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6bfc030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.118+0000 7fd950981740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c118f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.118+0000 7fd950981740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 0x55b1f6c11e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.118+0000 7fd950981740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441b9f10 con 0x55b1f6836550 2024-04-25T16:09:12.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.118+0000 7fd950981740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441bba50 con 0x55b1f685f410 2024-04-25T16:09:12.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.119+0000 7fd950981740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441ba5e0 con 0x55b1f683e730 2024-04-25T16:09:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.119+0000 7fd943fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 0x55b1f6c11e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.119+0000 7fd943fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 0x55b1f6c11e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56288/0 (socket says 172.21.15.31:56288) 2024-04-25T16:09:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.119+0000 7fd943fff640 1 -- 172.21.15.31:0/1550047404 learned_addr learned my addr 172.21.15.31:0/1550047404 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.119+0000 7fd942ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c118f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.119+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1550047404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6bfc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.120+0000 7fd942ffd640 1 -- 172.21.15.31:0/1550047404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 msgr2=0x55b1f6c11e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.120+0000 7fd942ffd640 1 --2- 172.21.15.31:0/1550047404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 0x55b1f6c11e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.120+0000 7fd942ffd640 1 -- 172.21.15.31:0/1550047404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 msgr2=0x55b1f6bfc030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.120+0000 7fd942ffd640 1 --2- 172.21.15.31:0/1550047404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6bfc030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.120+0000 7fd942ffd640 1 -- 172.21.15.31:0/1550047404 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b1f6bfcb80 con 0x55b1f683e730 2024-04-25T16:09:12.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.121+0000 7fd942ffd640 1 --2- 172.21.15.31:0/1550047404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c118f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55b1f6bfcd50 tx=0x7fd934016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e23aa51f67765226 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:12.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.122+0000 7fd9297fa640 1 -- 172.21.15.31:0/1550047404 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93400f6f0 con 0x55b1f683e730 2024-04-25T16:09:12.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.131+0000 7fd9297fa640 1 -- 172.21.15.31:0/1550047404 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd93400f890 con 0x55b1f683e730 2024-04-25T16:09:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.132+0000 7fd950981740 1 -- 172.21.15.31:0/1550047404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 msgr2=0x55b1f6c118f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.132+0000 7fd950981740 1 --2- 172.21.15.31:0/1550047404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c118f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55b1f6bfcd50 tx=0x7fd934016000 comp rx=0 tx=0).stop 2024-04-25T16:09:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.133+0000 7fd950981740 1 -- 172.21.15.31:0/1550047404 shutdown_connections 2024-04-25T16:09:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.133+0000 7fd950981740 1 --2- 172.21.15.31:0/1550047404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f685f410 0x55b1f6c11e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.133+0000 7fd950981740 1 --2- 172.21.15.31:0/1550047404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f683e730 0x55b1f6c118f0 secure :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0x55b1f6bfcd50 tx=0x7fd934016000 comp rx=0 tx=0).stop 2024-04-25T16:09:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.133+0000 7fd950981740 1 --2- 172.21.15.31:0/1550047404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f6836550 0x55b1f6bfc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.133+0000 7fd950981740 1 -- 172.21.15.31:0/1550047404 >> 172.21.15.31:0/1550047404 conn(0x55b1f6853560 msgr2=0x55b1f6bfb920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:12.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.135+0000 7fd950981740 1 -- 172.21.15.31:0/1550047404 shutdown_connections 2024-04-25T16:09:12.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.138+0000 7fd950981740 1 -- 172.21.15.31:0/1550047404 wait complete. 2024-04-25T16:09:12.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.139+0000 7fd950981740 1 -- start start 2024-04-25T16:09:12.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.139+0000 7fd950981740 1 Processor -- start 2024-04-25T16:09:12.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.140+0000 7fd950981740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f6836550 0x55b1f6c0f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.140+0000 7fd950981740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f685f410 0x55b1f6c0fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.140+0000 7fd950981740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f67f3630 0x55b1f6c10cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.140+0000 7fd950981740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441bb380 con 0x55b1f685f410 2024-04-25T16:09:12.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.140+0000 7fd950981740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441bacb0 con 0x55b1f6836550 2024-04-25T16:09:12.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.140+0000 7fd950981740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd9441b7ef0 con 0x55b1f67f3630 2024-04-25T16:09:12.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.140+0000 7fd950981740 10 client.0 _ll_register_callbacks cb 0x7fd934009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:12.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.141+0000 7fd943fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f67f3630 0x55b1f6c10cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.141+0000 7fd943fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f67f3630 0x55b1f6c10cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45818/0 (socket says 172.21.15.31:45818) 2024-04-25T16:09:12.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.141+0000 7fd943fff640 1 -- 172.21.15.31:0/1156463605 learned_addr learned my addr 172.21.15.31:0/1156463605 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:12.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.142+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f6836550 0x55b1f6c0f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.142+0000 7fd943fff640 1 -- 172.21.15.31:0/1156463605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f6836550 msgr2=0x55b1f6c0f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.142+0000 7fd943fff640 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f6836550 0x55b1f6c0f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.142+0000 7fd943fff640 1 -- 172.21.15.31:0/1156463605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f685f410 msgr2=0x55b1f6c0fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.142+0000 7fd943fff640 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f685f410 0x55b1f6c0fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.142+0000 7fd943fff640 1 -- 172.21.15.31:0/1156463605 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b1f6c05370 con 0x55b1f67f3630 2024-04-25T16:09:12.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.147+0000 7fd943fff640 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f67f3630 0x55b1f6c10cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd938043000 tx=0x7fd9380314c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ae8f1c1fb0e9285 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.148+0000 7fd9277f6640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:12.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.150+0000 7fd9277f6640 1 -- 172.21.15.31:0/1156463605 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938035070 con 0x55b1f67f3630 2024-04-25T16:09:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.151+0000 7fd950981740 1 -- 172.21.15.31:0/1156463605 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd93400f890 con 0x55b1f67f3630 2024-04-25T16:09:12.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.151+0000 7fd9277f6640 1 -- 172.21.15.31:0/1156463605 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd9380048d0 con 0x55b1f67f3630 2024-04-25T16:09:12.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.156+0000 7fd9277f6640 1 -- 172.21.15.31:0/1156463605 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd938004ad0 con 0x55b1f67f3630 2024-04-25T16:09:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.156+0000 7fd950981740 10 client.26860 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:12.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.156+0000 7fd950981740 1 -- 172.21.15.31:0/1156463605 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b1f6c12430 con 0x55b1f67f3630 2024-04-25T16:09:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.158+0000 7fd9277f6640 1 -- 172.21.15.31:0/1156463605 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd93400f890 con 0x55b1f67f3630 2024-04-25T16:09:12.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.159+0000 7fd9277f6640 1 -- 172.21.15.31:0/1156463605 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd938031c10 con 0x55b1f67f3630 2024-04-25T16:09:12.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.159+0000 7fd950981740 10 client.26860 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:12.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.159+0000 7fd950981740 10 client.26860 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:12.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.159+0000 7fd950981740 20 client.26860 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:12.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.159+0000 7fd950981740 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b1f6c80bd0 0x55b1f6c83010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.159+0000 7fd950981740 4 client.26860 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:09:12.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.159+0000 7fd950981740 1 -- 172.21.15.31:0/1156463605 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55b1f6bf74f0 con 0x55b1f6c80bd0 2024-04-25T16:09:12.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.163+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b1f6c80bd0 0x55b1f6c83010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.167+0000 7fd9437fe640 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b1f6c80bd0 0x55b1f6c83010 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a6bbf1025ebbd8f server_cookie=2b5e13daad5b3c71 in_seq=0 out_seq=0 2024-04-25T16:09:12.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.167+0000 7fd9277f6640 10 client.26860 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:12.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.171+0000 7fd9277f6640 1 -- 172.21.15.31:0/1156463605 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55b1f6bf74f0 con 0x55b1f6c80bd0 2024-04-25T16:09:12.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.171+0000 7fd9277f6640 10 client.26860 handle_command_reply: tid=0 2024-04-25T16:09:12.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.171+0000 7fd950981740 1 client.26860 shutdown 2024-04-25T16:09:12.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.171+0000 7fd950981740 1 -- 172.21.15.31:0/1156463605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f67f3630 msgr2=0x55b1f6c10cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.171+0000 7fd950981740 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f67f3630 0x55b1f6c10cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd938043000 tx=0x7fd9380314c0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.187+0000 7fd950981740 1 -- 172.21.15.31:0/1156463605 shutdown_connections 2024-04-25T16:09:12.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.187+0000 7fd950981740 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b1f6c80bd0 0x55b1f6c83010 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.187+0000 7fd950981740 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b1f67f3630 0x55b1f6c10cc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.187+0000 7fd950981740 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b1f685f410 0x55b1f6c0fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.187+0000 7fd950981740 1 --2- 172.21.15.31:0/1156463605 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b1f6836550 0x55b1f6c0f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.187+0000 7fd950981740 1 -- 172.21.15.31:0/1156463605 >> 172.21.15.31:0/1156463605 conn(0x55b1f6853560 msgr2=0x55b1f6bfa560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:12.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.187+0000 7fd9277f6640 0 client.26860 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:12.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.192+0000 7fd950981740 1 -- 172.21.15.31:0/1156463605 shutdown_connections 2024-04-25T16:09:12.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.192+0000 7fd950981740 1 -- 172.21.15.31:0/1156463605 wait complete. 2024-04-25T16:09:12.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.193+0000 7fd950981740 20 client.26860 trim_cache size 0 max 16384 2024-04-25T16:09:12.199 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:09:12.199 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:09:12.199 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "0b7969e2-cf36-4973-92e5-9146a356363c", 2024-04-25T16:09:12.199 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:09:12.199 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:09:12.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.200+0000 7fd949ad9640 1 -- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd92c0afbb0 msgr2=0x7fd92c0b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.200+0000 7fd949ad9640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd92c0afbb0 0x7fd92c0b20c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd93802a9f0 tx=0x7fd938004310 comp rx=0 tx=0).stop 2024-04-25T16:09:12.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.201+0000 7fd949ad9640 1 -- 172.21.15.31:0/1404493076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd94410a3e0 msgr2=0x7fd9441a8e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.201+0000 7fd949ad9640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd94410a3e0 0x7fd9441a8e80 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd944074760 tx=0x7fd93c00ee80 comp rx=0 tx=0).stop 2024-04-25T16:09:12.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.201+0000 7fd949ad9640 1 -- 172.21.15.31:0/1404493076 shutdown_connections 2024-04-25T16:09:12.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.201+0000 7fd949ad9640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd92c0afbb0 0x7fd92c0b20c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.201+0000 7fd949ad9640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd94410fef0 0x7fd9441b0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.201+0000 7fd949ad9640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd94410b1d0 0x7fd9441a93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.202+0000 7fd949ad9640 1 --2- 172.21.15.31:0/1404493076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd94410a3e0 0x7fd9441a8e80 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.202+0000 7fd949ad9640 1 -- 172.21.15.31:0/1404493076 >> 172.21.15.31:0/1404493076 conn(0x7fd944070400 msgr2=0x7fd94410bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:12.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.202+0000 7fd949ad9640 1 -- 172.21.15.31:0/1404493076 shutdown_connections 2024-04-25T16:09:12.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.202+0000 7fd949ad9640 1 -- 172.21.15.31:0/1404493076 wait complete. 2024-04-25T16:09:12.298 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:09:12.298 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:09:12.298 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:12 smithi031 ceph-mon[29140]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 737 MiB/s wr, 1.74k op/s 2024-04-25T16:09:12.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:12 smithi063 ceph-mon[32413]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 737 MiB/s wr, 1.74k op/s 2024-04-25T16:09:12.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:12 smithi186 ceph-mon[33258]: pgmap v588: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 737 MiB/s wr, 1.74k op/s 2024-04-25T16:09:12.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.788+0000 7fcb7abe3640 1 -- 172.21.15.31:0/2851263559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7410a0e0 msgr2=0x7fcb7410a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.788+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/2851263559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7410a0e0 0x7fcb7410a4f0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009860 tx=0x7fcb6802a360 comp rx=0 tx=0).stop 2024-04-25T16:09:12.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.800+0000 7fcb7abe3640 1 -- 172.21.15.31:0/2851263559 shutdown_connections 2024-04-25T16:09:12.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.800+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/2851263559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb74078d00 0x7fcb7407b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.800+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/2851263559 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7410a0e0 0x7fcb7410a4f0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.800+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/2851263559 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb7410b6b0 0x7fcb74109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.800+0000 7fcb7abe3640 1 -- 172.21.15.31:0/2851263559 >> 172.21.15.31:0/2851263559 conn(0x7fcb740704e0 msgr2=0x7fcb74072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:12.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.802+0000 7fcb7abe3640 1 -- 172.21.15.31:0/2851263559 shutdown_connections 2024-04-25T16:09:12.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.803+0000 7fcb7abe3640 1 -- 172.21.15.31:0/2851263559 wait complete. 2024-04-25T16:09:12.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.804+0000 7fcb7abe3640 1 Processor -- start 2024-04-25T16:09:12.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.814+0000 7fcb7abe3640 1 -- start start 2024-04-25T16:09:12.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.815+0000 7fcb7abe3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb74078d00 0x7fcb74084ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.815+0000 7fcb7abe3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7410a0e0 0x7fcb74083340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.815+0000 7fcb7abe3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb7410b6b0 0x7fcb74083880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.815+0000 7fcb7abe3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7407d6c0 con 0x7fcb7410a0e0 2024-04-25T16:09:12.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.815+0000 7fcb7abe3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7407d540 con 0x7fcb7410b6b0 2024-04-25T16:09:12.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.815+0000 7fcb7abe3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7407d840 con 0x7fcb74078d00 2024-04-25T16:09:12.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.824+0000 7fcb79180640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb7410b6b0 0x7fcb74083880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.824+0000 7fcb79180640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb7410b6b0 0x7fcb74083880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56324/0 (socket says 172.21.15.31:56324) 2024-04-25T16:09:12.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.824+0000 7fcb79180640 1 -- 172.21.15.31:0/1574210530 learned_addr learned my addr 172.21.15.31:0/1574210530 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:12.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.826+0000 7fcb7897f640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb74078d00 0x7fcb74084ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.827+0000 7fcb79180640 1 -- 172.21.15.31:0/1574210530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb74078d00 msgr2=0x7fcb74084ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.827+0000 7fcb79180640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb74078d00 0x7fcb74084ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.827+0000 7fcb79180640 1 -- 172.21.15.31:0/1574210530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7410a0e0 msgr2=0x7fcb74083340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.827+0000 7fcb79180640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7410a0e0 0x7fcb74083340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.827+0000 7fcb79180640 1 -- 172.21.15.31:0/1574210530 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb74083dc0 con 0x7fcb7410b6b0 2024-04-25T16:09:12.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.845+0000 7fcb79180640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb7410b6b0 0x7fcb74083880 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcb6400e970 tx=0x7fcb6400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:12.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.848+0000 7fcb71ffb640 1 -- 172.21.15.31:0/1574210530 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6400ccc0 con 0x7fcb7410b6b0 2024-04-25T16:09:12.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.848+0000 7fcb71ffb640 1 -- 172.21.15.31:0/1574210530 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb640054d0 con 0x7fcb7410b6b0 2024-04-25T16:09:12.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.848+0000 7fcb71ffb640 1 -- 172.21.15.31:0/1574210530 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb64010430 con 0x7fcb7410b6b0 2024-04-25T16:09:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.849+0000 7fcb7abe3640 1 -- 172.21.15.31:0/1574210530 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb740840b0 con 0x7fcb7410b6b0 2024-04-25T16:09:12.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.850+0000 7fcb7abe3640 1 -- 172.21.15.31:0/1574210530 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb741cabd0 con 0x7fcb7410b6b0 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.865+0000 7fcb71ffb640 1 -- 172.21.15.31:0/1574210530 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcb64004f00 con 0x7fcb7410b6b0 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.867+0000 7fcb71ffb640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb5c0afc30 0x7fcb5c0b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.867+0000 7fcb71ffb640 1 -- 172.21.15.31:0/1574210530 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcb640df000 con 0x7fcb7410b6b0 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.868+0000 7fcb81b6b740 1 Processor -- start 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.868+0000 7fcb81b6b740 1 -- start start 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.868+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f154328390 0x55f1546c9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.868+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 0x55f1546c9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.868+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 0x55f1546cac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.869+0000 7fcb73fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 0x55f1546c9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.869+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f15462ed80 con 0x55f1542ff4d0 2024-04-25T16:09:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.869+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f1545e88b0 con 0x55f1543076b0 2024-04-25T16:09:12.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.869+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f15462ad40 con 0x55f154328390 2024-04-25T16:09:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.870+0000 7fcb79180640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 0x55f1546cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.870+0000 7fcb79180640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 0x55f1546cac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:56336/0 (socket says 172.21.15.31:56336) 2024-04-25T16:09:12.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.870+0000 7fcb79180640 1 -- 172.21.15.31:0/2849757666 learned_addr learned my addr 172.21.15.31:0/2849757666 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.873+0000 7fcb79180640 1 -- 172.21.15.31:0/2849757666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f154328390 msgr2=0x55f1546c9a60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.873+0000 7fcb79180640 1 --2- 172.21.15.31:0/2849757666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f154328390 0x55f1546c9a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.873+0000 7fcb79180640 1 -- 172.21.15.31:0/2849757666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 msgr2=0x55f1546c9fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.873+0000 7fcb79180640 1 --2- 172.21.15.31:0/2849757666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 0x55f1546c9fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.873+0000 7fcb79180640 1 -- 172.21.15.31:0/2849757666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1545b9a40 con 0x55f1543076b0 2024-04-25T16:09:12.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.874+0000 7fcb79180640 1 --2- 172.21.15.31:0/2849757666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 0x55f1546cac70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcb640a3510 tx=0x7fcb640094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bd35e98f4836a7c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.882+0000 7fcb5affd640 1 -- 172.21.15.31:0/2849757666 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb640c1070 con 0x55f1543076b0 2024-04-25T16:09:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.882+0000 7fcb5affd640 1 -- 172.21.15.31:0/2849757666 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb640a7b90 con 0x55f1543076b0 2024-04-25T16:09:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.885+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2849757666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 msgr2=0x55f1546cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.885+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2849757666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 0x55f1546cac70 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcb640a3510 tx=0x7fcb640094a0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.889 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.888+0000 7fcb7897f640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb5c0afc30 0x7fcb5c0b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.889+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2849757666 shutdown_connections 2024-04-25T16:09:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.889+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2849757666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 0x55f1546cac70 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.889+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2849757666 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 0x55f1546c9fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.889+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2849757666 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f154328390 0x55f1546c9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.889+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2849757666 >> 172.21.15.31:0/2849757666 conn(0x55f15431c4e0 msgr2=0x55f1546c5270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.891+0000 7fcb7897f640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb5c0afc30 0x7fcb5c0b2140 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x55f1546caab0 tx=0x7fcb6c00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.893+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2849757666 shutdown_connections 2024-04-25T16:09:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.894+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2849757666 wait complete. 2024-04-25T16:09:12.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.895+0000 7fcb81b6b740 1 -- start start 2024-04-25T16:09:12.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.895+0000 7fcb81b6b740 1 Processor -- start 2024-04-25T16:09:12.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.897+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1542ff4d0 0x55f1546d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.897+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 0x55f1546d3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.897+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 0x55f1546dadd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.897+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f1546cb7d0 con 0x55f154328390 2024-04-25T16:09:12.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.897+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f1545b5d10 con 0x55f1542ff4d0 2024-04-25T16:09:12.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.897+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f1546cb950 con 0x55f1543076b0 2024-04-25T16:09:12.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.897+0000 7fcb81b6b740 10 client.0 _ll_register_callbacks cb 0x55f1545e2210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:12.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.898+0000 7fcb79180640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 0x55f1546dadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.898+0000 7fcb79180640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 0x55f1546dadd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56190/0 (socket says 172.21.15.31:56190) 2024-04-25T16:09:12.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.898+0000 7fcb79180640 1 -- 172.21.15.31:0/3188359192 learned_addr learned my addr 172.21.15.31:0/3188359192 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:12.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.899+0000 7fcb7897f640 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1542ff4d0 0x55f1546d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.899+0000 7fcb79180640 1 -- 172.21.15.31:0/3188359192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 msgr2=0x55f1546d3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.899+0000 7fcb79180640 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 0x55f1546d3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.899+0000 7fcb79180640 1 -- 172.21.15.31:0/3188359192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1542ff4d0 msgr2=0x55f1546d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.899+0000 7fcb79180640 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1542ff4d0 0x55f1546d3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.899+0000 7fcb79180640 1 -- 172.21.15.31:0/3188359192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f15440a7d0 con 0x55f154328390 2024-04-25T16:09:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.900+0000 7fcb79180640 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 0x55f1546dadd0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcb640b04b0 tx=0x7fcb640aedf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907432b7b78cef42 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:12.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.900+0000 7fcb53fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.900+0000 7fcb53fff640 1 -- 172.21.15.31:0/3188359192 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb640c6060 con 0x55f154328390 2024-04-25T16:09:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.901+0000 7fcb81b6b740 1 -- 172.21.15.31:0/3188359192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb640a7b90 con 0x55f154328390 2024-04-25T16:09:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.901+0000 7fcb53fff640 1 -- 172.21.15.31:0/3188359192 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcb640a6900 con 0x55f154328390 2024-04-25T16:09:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.901+0000 7fcb53fff640 1 -- 172.21.15.31:0/3188359192 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcb64087050 con 0x55f154328390 2024-04-25T16:09:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.901+0000 7fcb81b6b740 10 client.17214 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.901+0000 7fcb81b6b740 1 -- 172.21.15.31:0/3188359192 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f1543403f0 con 0x55f154328390 2024-04-25T16:09:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.902+0000 7fcb53fff640 1 -- 172.21.15.31:0/3188359192 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb640a7b90 con 0x55f154328390 2024-04-25T16:09:12.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.903+0000 7fcb53fff640 1 -- 172.21.15.31:0/3188359192 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcb640a6160 con 0x55f154328390 2024-04-25T16:09:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.903+0000 7fcb81b6b740 10 client.17214 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:12.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.903+0000 7fcb81b6b740 10 client.17214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.903+0000 7fcb81b6b740 20 client.17214 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.903+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1542e6f60 0x55f1546e0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.903+0000 7fcb81b6b740 4 client.17214 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.903+0000 7fcb81b6b740 1 -- 172.21.15.31:0/3188359192 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f15463c990 con 0x55f1542e6f60 2024-04-25T16:09:12.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.903+0000 7fcb7897f640 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1542e6f60 0x55f1546e0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:12.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.909+0000 7fcb7897f640 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1542e6f60 0x55f1546e0e90 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b5c1c6ccb2dc54a server_cookie=b81650ae21bf67f6 in_seq=0 out_seq=0 2024-04-25T16:09:12.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.910+0000 7fcb53fff640 10 client.17214 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:12.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.923+0000 7fcb53fff640 1 -- 172.21.15.31:0/3188359192 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f15463c990 con 0x55f1542e6f60 2024-04-25T16:09:12.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.923+0000 7fcb53fff640 10 client.17214 handle_command_reply: tid=0 2024-04-25T16:09:12.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.923+0000 7fcb81b6b740 1 client.17214 shutdown 2024-04-25T16:09:12.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.926+0000 7fcb81b6b740 1 -- 172.21.15.31:0/3188359192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 msgr2=0x55f1546dadd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:12.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.926+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 0x55f1546dadd0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fcb640b04b0 tx=0x7fcb640aedf0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.929+0000 7fcb81b6b740 1 -- 172.21.15.31:0/3188359192 shutdown_connections 2024-04-25T16:09:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.929+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1542e6f60 0x55f1546e0e90 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.929+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 0x55f1546dadd0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.929+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 0x55f1546d3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.929+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/3188359192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1542ff4d0 0x55f1546d3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.929+0000 7fcb81b6b740 1 -- 172.21.15.31:0/3188359192 >> 172.21.15.31:0/3188359192 conn(0x55f15431c4e0 msgr2=0x55f1546c43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:12.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.929+0000 7fcb53fff640 0 client.17214 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:12.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.933+0000 7fcb81b6b740 1 -- 172.21.15.31:0/3188359192 shutdown_connections 2024-04-25T16:09:12.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.933+0000 7fcb81b6b740 1 -- 172.21.15.31:0/3188359192 wait complete. 2024-04-25T16:09:12.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:12.933+0000 7fcb81b6b740 20 client.17214 trim_cache size 0 max 16384 2024-04-25T16:09:13.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.044+0000 7fcb81b6b740 1 Processor -- start 2024-04-25T16:09:13.050 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.048+0000 7fcb81b6b740 1 -- start start 2024-04-25T16:09:13.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.049+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 0x55f1546d1f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:13.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.049+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 0x55f1546d24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:13.051 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.050+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f154328390 0x55f1546e5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:13.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.050+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb741d39b0 con 0x55f1542ff4d0 2024-04-25T16:09:13.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.052+0000 7fcb7897f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 0x55f1546d1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:13.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.052+0000 7fcb7897f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 0x55f1546d1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:56204/0 (socket says 172.21.15.31:56204) 2024-04-25T16:09:13.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.052+0000 7fcb7897f640 1 -- 172.21.15.31:0/2599692117 learned_addr learned my addr 172.21.15.31:0/2599692117 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:13.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.054+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2599692117 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcb741d54f0 con 0x55f154328390 2024-04-25T16:09:13.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.054+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2599692117 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb741d4080 con 0x55f1543076b0 2024-04-25T16:09:13.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.058+0000 7fcb73fff640 1 --2- 172.21.15.31:0/2599692117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 0x55f1546d24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:13.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.059+0000 7fcb73fff640 1 -- 172.21.15.31:0/2599692117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f154328390 msgr2=0x55f1546e5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:13.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.059+0000 7fcb73fff640 1 --2- 172.21.15.31:0/2599692117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f154328390 0x55f1546e5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.059+0000 7fcb73fff640 1 -- 172.21.15.31:0/2599692117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 msgr2=0x55f1546d1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:13.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.059+0000 7fcb73fff640 1 --2- 172.21.15.31:0/2599692117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 0x55f1546d1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.059+0000 7fcb73fff640 1 -- 172.21.15.31:0/2599692117 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1543000f0 con 0x55f1543076b0 2024-04-25T16:09:13.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.060+0000 7fcb73fff640 1 --2- 172.21.15.31:0/2599692117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 0x55f1546d24b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcb68035040 tx=0x7fcb68004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14bbb56e0c1a12b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:13.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.061+0000 7fcb5a7fc640 1 -- 172.21.15.31:0/2599692117 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68040020 con 0x55f1543076b0 2024-04-25T16:09:13.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.062+0000 7fcb5a7fc640 1 -- 172.21.15.31:0/2599692117 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb680043b0 con 0x55f1543076b0 2024-04-25T16:09:13.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.066+0000 7fcb5a7fc640 1 -- 172.21.15.31:0/2599692117 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb68037040 con 0x55f1543076b0 2024-04-25T16:09:13.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.067+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2599692117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 msgr2=0x55f1546d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:13.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.067+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2599692117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 0x55f1546d24b0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fcb68035040 tx=0x7fcb68004040 comp rx=0 tx=0).stop 2024-04-25T16:09:13.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.070+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2599692117 shutdown_connections 2024-04-25T16:09:13.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.070+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2599692117 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f154328390 0x55f1546e5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.070+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2599692117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1543076b0 0x55f1546d24b0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.070+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2599692117 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f1542ff4d0 0x55f1546d1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.070+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2599692117 >> 172.21.15.31:0/2599692117 conn(0x55f15431c4e0 msgr2=0x55f1546ce5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:13.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.075+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2599692117 shutdown_connections 2024-04-25T16:09:13.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.079+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2599692117 wait complete. 2024-04-25T16:09:13.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.079+0000 7fcb81b6b740 1 -- start start 2024-04-25T16:09:13.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.079+0000 7fcb81b6b740 1 Processor -- start 2024-04-25T16:09:13.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.083+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1542ff4d0 0x55f1546d7590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:13.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.083+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 0x55f1546d7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:13.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.083+0000 7fcb81b6b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 0x55f1546d8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:13.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.083+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb741d4e20 con 0x55f154328390 2024-04-25T16:09:13.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.083+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcb741d4750 con 0x55f1543076b0 2024-04-25T16:09:13.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.083+0000 7fcb81b6b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb741d1990 con 0x55f1542ff4d0 2024-04-25T16:09:13.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.083+0000 7fcb81b6b740 10 client.0 _ll_register_callbacks cb 0x7fcb640bf070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:13.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.084+0000 7fcb7897f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1542ff4d0 0x55f1546d7590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:13.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.084+0000 7fcb7897f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1542ff4d0 0x55f1546d7590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45860/0 (socket says 172.21.15.31:45860) 2024-04-25T16:09:13.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.084+0000 7fcb7897f640 1 -- 172.21.15.31:0/2400070249 learned_addr learned my addr 172.21.15.31:0/2400070249 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:13.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.087+0000 7fcb7897f640 1 -- 172.21.15.31:0/2400070249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 msgr2=0x55f1546d7ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:09:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.087+0000 7fcb7897f640 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 0x55f1546d7ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.087+0000 7fcb7897f640 1 -- 172.21.15.31:0/2400070249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 msgr2=0x55f1546d8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.087+0000 7fcb7897f640 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 0x55f1546d8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.087+0000 7fcb7897f640 1 -- 172.21.15.31:0/2400070249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1545ef4b0 con 0x55f1542ff4d0 2024-04-25T16:09:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.091+0000 7fcb7897f640 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1542ff4d0 0x55f1546d7590 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcb7410aed0 tx=0x7fcb6c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47cceb7d1dbe5961 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:13.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.091+0000 7fcb53fff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:13.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.098+0000 7fcb53fff640 1 -- 172.21.15.31:0/2400070249 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c004d60 con 0x55f1542ff4d0 2024-04-25T16:09:13.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.099+0000 7fcb53fff640 1 -- 172.21.15.31:0/2400070249 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcb6c0273c0 con 0x55f1542ff4d0 2024-04-25T16:09:13.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.109+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2400070249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb680043b0 con 0x55f1542ff4d0 2024-04-25T16:09:13.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.112+0000 7fcb53fff640 1 -- 172.21.15.31:0/2400070249 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcb6c0275c0 con 0x55f1542ff4d0 2024-04-25T16:09:13.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.112+0000 7fcb81b6b740 10 client.26872 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:13.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.112+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2400070249 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f1546da180 con 0x55f1542ff4d0 2024-04-25T16:09:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.118+0000 7fcb53fff640 1 -- 172.21.15.31:0/2400070249 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb680043b0 con 0x55f1542ff4d0 2024-04-25T16:09:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.118+0000 7fcb53fff640 1 -- 172.21.15.31:0/2400070249 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcb6c0058a0 con 0x55f1542ff4d0 2024-04-25T16:09:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.119+0000 7fcb81b6b740 10 client.26872 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.119+0000 7fcb81b6b740 10 client.26872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:13.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.119+0000 7fcb81b6b740 20 client.26872 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:13.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.119+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1542bc3a0 0x55f15474b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:13.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.119+0000 7fcb81b6b740 4 client.26872 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:09:13.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.119+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2400070249 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f154636800 con 0x55f1542bc3a0 2024-04-25T16:09:13.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.120+0000 7fcb73fff640 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1542bc3a0 0x55f15474b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:13.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.121+0000 7fcb73fff640 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1542bc3a0 0x55f15474b2f0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2ab11150f33631e server_cookie=73d4ba4f56d6488a in_seq=0 out_seq=0 2024-04-25T16:09:13.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.124+0000 7fcb53fff640 10 client.26872 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:13.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.127+0000 7fcb53fff640 1 -- 172.21.15.31:0/2400070249 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55f154636800 con 0x55f1542bc3a0 2024-04-25T16:09:13.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.128+0000 7fcb53fff640 10 client.26872 handle_command_reply: tid=0 2024-04-25T16:09:13.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.128+0000 7fcb81b6b740 1 client.26872 shutdown 2024-04-25T16:09:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2400070249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1542ff4d0 msgr2=0x55f1546d7590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1542ff4d0 0x55f1546d7590 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fcb7410aed0 tx=0x7fcb6c004ab0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2400070249 shutdown_connections 2024-04-25T16:09:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f1542bc3a0 0x55f15474b2f0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f154328390 0x55f1546d8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f1543076b0 0x55f1546d7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 --2- 172.21.15.31:0/2400070249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f1542ff4d0 0x55f1546d7590 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2400070249 >> 172.21.15.31:0/2400070249 conn(0x55f15431c4e0 msgr2=0x55f1546d1f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:13.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb53fff640 0 client.26872 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2400070249 shutdown_connections 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.132+0000 7fcb81b6b740 1 -- 172.21.15.31:0/2400070249 wait complete. 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.133+0000 7fcb81b6b740 20 client.26872 trim_cache size 0 max 16384 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stdout: "0b7969e2-cf36-4973-92e5-9146a356363c": { 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "0b7969e2-cf36-4973-92e5-9146a356363c", 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:09:13.135 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:09:13.136 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 -- 172.21.15.31:0/1574210530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb5c0afc30 msgr2=0x7fcb5c0b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb5c0afc30 0x7fcb5c0b2140 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x55f1546caab0 tx=0x7fcb6c00a3b0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 -- 172.21.15.31:0/1574210530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb7410b6b0 msgr2=0x7fcb74083880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb7410b6b0 0x7fcb74083880 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fcb6400e970 tx=0x7fcb6400ee30 comp rx=0 tx=0).stop 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 -- 172.21.15.31:0/1574210530 shutdown_connections 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb5c0afc30 0x7fcb5c0b2140 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb7410b6b0 0x7fcb74083880 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb7410a0e0 0x7fcb74083340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 --2- 172.21.15.31:0/1574210530 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb74078d00 0x7fcb74084ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:13.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.144+0000 7fcb7abe3640 1 -- 172.21.15.31:0/1574210530 >> 172.21.15.31:0/1574210530 conn(0x7fcb740704e0 msgr2=0x7fcb74074ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:13.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.145+0000 7fcb7abe3640 1 -- 172.21.15.31:0/1574210530 shutdown_connections 2024-04-25T16:09:13.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:13.145+0000 7fcb7abe3640 1 -- 172.21.15.31:0/1574210530 wait complete. 2024-04-25T16:09:13.192 INFO:tasks.cephfs.filesystem:scrub status for tag:0b7969e2-cf36-4973-92e5-9146a356363c - {'path': '/', 'tag': '0b7969e2-cf36-4973-92e5-9146a356363c', 'options': 'recursive,force'} 2024-04-25T16:09:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:13 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:09:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:13 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:13 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:09:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:13 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:09:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:13 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:13 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:09:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:13 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:09:13.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:13 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:13.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:13 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:09:14.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:14 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:09:14.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:14 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:14.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:14 smithi063 ceph-mon[32413]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 903 MiB/s wr, 2.27k op/s 2024-04-25T16:09:14.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:14 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:09:14.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:14 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:14.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:14 smithi186 ceph-mon[33258]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 903 MiB/s wr, 2.27k op/s 2024-04-25T16:09:14.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:14 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:09:14.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:14 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:14.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:14 smithi031 ceph-mon[29140]: pgmap v589: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 903 MiB/s wr, 2.27k op/s 2024-04-25T16:09:15.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:15 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:15.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:15 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:15.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:15 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:16.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:16 smithi186 ceph-mon[33258]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 784 MiB/s wr, 1.96k op/s 2024-04-25T16:09:16.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:16 smithi031 ceph-mon[29140]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 784 MiB/s wr, 1.96k op/s 2024-04-25T16:09:16.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:16 smithi063 ceph-mon[32413]: pgmap v590: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 784 MiB/s wr, 1.96k op/s 2024-04-25T16:09:18.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:18 smithi063 ceph-mon[32413]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 918 MiB/s wr, 2.43k op/s 2024-04-25T16:09:18.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:18 smithi186 ceph-mon[33258]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 918 MiB/s wr, 2.43k op/s 2024-04-25T16:09:18.455 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:18 smithi031 ceph-mon[29140]: pgmap v591: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 918 MiB/s wr, 2.43k op/s 2024-04-25T16:09:19.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:19 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:09:19.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:19 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:09:19.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:19 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:09:20.352 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:20 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:09:20.353 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:20 smithi031 ceph-mon[29140]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 811 MiB/s wr, 2.28k op/s 2024-04-25T16:09:20.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:20 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:09:20.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:20 smithi063 ceph-mon[32413]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 811 MiB/s wr, 2.28k op/s 2024-04-25T16:09:20.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:20 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:09:20.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:20 smithi186 ceph-mon[33258]: pgmap v592: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 811 MiB/s wr, 2.28k op/s 2024-04-25T16:09:22.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:22 smithi063 ceph-mon[32413]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 727 MiB/s wr, 2.01k op/s 2024-04-25T16:09:22.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:22 smithi186 ceph-mon[33258]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 727 MiB/s wr, 2.01k op/s 2024-04-25T16:09:22.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:22 smithi031 ceph-mon[29140]: pgmap v593: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 727 MiB/s wr, 2.01k op/s 2024-04-25T16:09:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.522+0000 7fe0896d8640 1 -- 172.21.15.31:0/4087530427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08410a420 msgr2=0x7fe08410ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.522+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4087530427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08410a420 0x7fe08410ae10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7fe078009900 tx=0x7fe07802a440 comp rx=0 tx=0).stop 2024-04-25T16:09:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.522+0000 7fe0896d8640 1 -- 172.21.15.31:0/4087530427 shutdown_connections 2024-04-25T16:09:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.522+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4087530427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe084110070 0x7fe084110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.522+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4087530427 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe08410b350 0x7fe08410b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.526 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.522+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4087530427 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08410a420 0x7fe08410ae10 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.527 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.522+0000 7fe0896d8640 1 -- 172.21.15.31:0/4087530427 >> 172.21.15.31:0/4087530427 conn(0x7fe084070540 msgr2=0x7fe0840729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.534+0000 7fe0896d8640 1 -- 172.21.15.31:0/4087530427 shutdown_connections 2024-04-25T16:09:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.534+0000 7fe0896d8640 1 -- 172.21.15.31:0/4087530427 wait complete. 2024-04-25T16:09:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.535+0000 7fe0896d8640 1 Processor -- start 2024-04-25T16:09:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.535+0000 7fe0896d8640 1 -- start start 2024-04-25T16:09:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.536+0000 7fe0896d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08410a420 0x7fe0841a90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.536+0000 7fe0896d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe08410b350 0x7fe0841a9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.536+0000 7fe0896d8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe084110070 0x7fe0841b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.536+0000 7fe0896d8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe08411e620 con 0x7fe08410a420 2024-04-25T16:09:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.536+0000 7fe0896d8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe08411e4a0 con 0x7fe08410b350 2024-04-25T16:09:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.536+0000 7fe0896d8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe08411e7a0 con 0x7fe084110070 2024-04-25T16:09:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.536+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe084110070 0x7fe0841b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.536+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe084110070 0x7fe0841b0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37194/0 (socket says 172.21.15.31:37194) 2024-04-25T16:09:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.536+0000 7fe0837fe640 1 -- 172.21.15.31:0/4168817475 learned_addr learned my addr 172.21.15.31:0/4168817475 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.537+0000 7fe082ffd640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08410a420 0x7fe0841a90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.537+0000 7fe0837fe640 1 -- 172.21.15.31:0/4168817475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe08410b350 msgr2=0x7fe0841a9600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:09:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.537+0000 7fe0837fe640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe08410b350 0x7fe0841a9600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.537+0000 7fe0837fe640 1 -- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08410a420 msgr2=0x7fe0841a90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.537+0000 7fe0837fe640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08410a420 0x7fe0841a90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.537+0000 7fe0837fe640 1 -- 172.21.15.31:0/4168817475 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0841b0c00 con 0x7fe084110070 2024-04-25T16:09:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.537+0000 7fe0837fe640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe084110070 0x7fe0841b0590 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00b990 tx=0x7fe07c00be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.538+0000 7fe06bfff640 1 -- 172.21.15.31:0/4168817475 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c019070 con 0x7fe084110070 2024-04-25T16:09:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.538+0000 7fe06bfff640 1 -- 172.21.15.31:0/4168817475 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe07c00cba0 con 0x7fe084110070 2024-04-25T16:09:23.540 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.539+0000 7fe06bfff640 1 -- 172.21.15.31:0/4168817475 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07c0075d0 con 0x7fe084110070 2024-04-25T16:09:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.540+0000 7fe0896d8640 1 -- 172.21.15.31:0/4168817475 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0841b0ef0 con 0x7fe084110070 2024-04-25T16:09:23.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.540+0000 7fe0896d8640 1 -- 172.21.15.31:0/4168817475 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0841b12a0 con 0x7fe084110070 2024-04-25T16:09:23.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.554+0000 7fe06bfff640 1 -- 172.21.15.31:0/4168817475 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe07c014440 con 0x7fe084110070 2024-04-25T16:09:23.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.554+0000 7fe090583740 1 Processor -- start 2024-04-25T16:09:23.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.554+0000 7fe090583740 1 -- start start 2024-04-25T16:09:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.564+0000 7fe090583740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70d01b0 0x55bde7470970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.564+0000 7fe090583740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70a72f0 0x55bde7470ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.564+0000 7fe090583740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70af4d0 0x55bde7477df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.564+0000 7fe090583740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bde73d6460 con 0x55bde70af4d0 2024-04-25T16:09:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.564+0000 7fe090583740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bde7390590 con 0x55bde70a72f0 2024-04-25T16:09:23.566 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.564+0000 7fe090583740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bde73d2420 con 0x55bde70d01b0 2024-04-25T16:09:23.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.565+0000 7fe082ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70a72f0 0x55bde7470ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.565+0000 7fe082ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70a72f0 0x55bde7470ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:60680/0 (socket says 172.21.15.31:60680) 2024-04-25T16:09:23.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.565+0000 7fe082ffd640 1 -- 172.21.15.31:0/2882917493 learned_addr learned my addr 172.21.15.31:0/2882917493 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:23.567 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.565+0000 7fe0827fc640 1 --2- 172.21.15.31:0/2882917493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70d01b0 0x55bde7470970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.566+0000 7fe06bfff640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe06c0afbd0 0x7fe06c0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.566+0000 7fe082ffd640 1 -- 172.21.15.31:0/2882917493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70d01b0 msgr2=0x55bde7470970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.566+0000 7fe082ffd640 1 --2- 172.21.15.31:0/2882917493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70d01b0 0x55bde7470970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.566+0000 7fe082ffd640 1 -- 172.21.15.31:0/2882917493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70af4d0 msgr2=0x55bde7477df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.566+0000 7fe082ffd640 1 --2- 172.21.15.31:0/2882917493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70af4d0 0x55bde7477df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.566+0000 7fe082ffd640 1 -- 172.21.15.31:0/2882917493 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bde739d0d0 con 0x55bde70a72f0 2024-04-25T16:09:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.567+0000 7fe082ffd640 1 --2- 172.21.15.31:0/2882917493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70a72f0 0x55bde7470ef0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe078009900 tx=0x7fe078004220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b197005c1ad7f1f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:23.568 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.567+0000 7fe06bfff640 1 -- 172.21.15.31:0/4168817475 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe07c0de130 con 0x7fe084110070 2024-04-25T16:09:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.567+0000 7fe0697fa640 1 -- 172.21.15.31:0/2882917493 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078040070 con 0x55bde70a72f0 2024-04-25T16:09:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.567+0000 7fe0697fa640 1 -- 172.21.15.31:0/2882917493 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0780047a0 con 0x55bde70a72f0 2024-04-25T16:09:23.569 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.567+0000 7fe0697fa640 1 -- 172.21.15.31:0/2882917493 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078035040 con 0x55bde70a72f0 2024-04-25T16:09:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.569+0000 7fe090583740 1 -- 172.21.15.31:0/2882917493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70a72f0 msgr2=0x55bde7470ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.569+0000 7fe090583740 1 --2- 172.21.15.31:0/2882917493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70a72f0 0x55bde7470ef0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe078009900 tx=0x7fe078004220 comp rx=0 tx=0).stop 2024-04-25T16:09:23.571 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.569+0000 7fe0837fe640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe06c0afbd0 0x7fe06c0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.578+0000 7fe090583740 1 -- 172.21.15.31:0/2882917493 shutdown_connections 2024-04-25T16:09:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.578+0000 7fe090583740 1 --2- 172.21.15.31:0/2882917493 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70af4d0 0x55bde7477df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.578+0000 7fe090583740 1 --2- 172.21.15.31:0/2882917493 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70a72f0 0x55bde7470ef0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.578+0000 7fe090583740 1 --2- 172.21.15.31:0/2882917493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70d01b0 0x55bde7470970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.578+0000 7fe090583740 1 -- 172.21.15.31:0/2882917493 >> 172.21.15.31:0/2882917493 conn(0x55bde70c4300 msgr2=0x55bde746c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:23.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.579+0000 7fe090583740 1 -- 172.21.15.31:0/2882917493 shutdown_connections 2024-04-25T16:09:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.579+0000 7fe090583740 1 -- 172.21.15.31:0/2882917493 wait complete. 2024-04-25T16:09:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.580+0000 7fe090583740 1 -- start start 2024-04-25T16:09:23.582 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.580+0000 7fe090583740 1 Processor -- start 2024-04-25T16:09:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.581+0000 7fe090583740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 0x55bde747a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.581+0000 7fe090583740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70af4d0 0x55bde747ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.581+0000 7fe090583740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 0x55bde7481ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.581+0000 7fe090583740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bde74787e0 con 0x55bde70a72f0 2024-04-25T16:09:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.581+0000 7fe090583740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bde735db60 con 0x55bde70d01b0 2024-04-25T16:09:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.581+0000 7fe090583740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bde7478960 con 0x55bde70af4d0 2024-04-25T16:09:23.583 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.581+0000 7fe090583740 10 client.0 _ll_register_callbacks cb 0x55bde73822b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.582+0000 7fe082ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70af4d0 0x55bde747ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.582+0000 7fe082ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70af4d0 0x55bde747ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:37216/0 (socket says 172.21.15.31:37216) 2024-04-25T16:09:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.582+0000 7fe082ffd640 1 -- 172.21.15.31:0/2795503489 learned_addr learned my addr 172.21.15.31:0/2795503489 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:23.584 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.582+0000 7fe0827fc640 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 0x55bde7481ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.583+0000 7fe0827fc640 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 0x55bde747a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.583+0000 7fe0827fc640 1 -- 172.21.15.31:0/2795503489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70af4d0 msgr2=0x55bde747ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.583+0000 7fe0827fc640 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70af4d0 0x55bde747ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.583+0000 7fe0827fc640 1 -- 172.21.15.31:0/2795503489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 msgr2=0x55bde747a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.583+0000 7fe0827fc640 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 0x55bde747a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.583+0000 7fe0827fc640 1 -- 172.21.15.31:0/2795503489 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bde70e8210 con 0x55bde70d01b0 2024-04-25T16:09:23.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.584+0000 7fe0827fc640 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 0x55bde7481ad0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe0841aa160 tx=0x7fe0740061c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed57a4757481700e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:23.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.585+0000 7fe0837fe640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe06c0afbd0 0x7fe06c0b20e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe07c014bd0 tx=0x7fe07c004a60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.595+0000 7fe066ff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.595+0000 7fe066ff5640 1 -- 172.21.15.31:0/2795503489 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07401d060 con 0x55bde70d01b0 2024-04-25T16:09:23.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.595+0000 7fe066ff5640 1 -- 172.21.15.31:0/2795503489 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe074015610 con 0x55bde70d01b0 2024-04-25T16:09:23.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.595+0000 7fe090583740 1 -- 172.21.15.31:0/2795503489 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0780047a0 con 0x55bde70d01b0 2024-04-25T16:09:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.596+0000 7fe066ff5640 1 -- 172.21.15.31:0/2795503489 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe074015810 con 0x55bde70d01b0 2024-04-25T16:09:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.596+0000 7fe090583740 10 client.26903 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.596+0000 7fe090583740 1 -- 172.21.15.31:0/2795503489 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bde71b4620 con 0x55bde70d01b0 2024-04-25T16:09:23.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.597+0000 7fe066ff5640 1 -- 172.21.15.31:0/2795503489 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0780047a0 con 0x55bde70d01b0 2024-04-25T16:09:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.597+0000 7fe066ff5640 1 -- 172.21.15.31:0/2795503489 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe0740239c0 con 0x55bde70d01b0 2024-04-25T16:09:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.598+0000 7fe090583740 10 client.26903 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.598+0000 7fe090583740 10 client.26903 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.598+0000 7fe090583740 20 client.26903 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.598+0000 7fe090583740 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bde708ed80 0x55bde7499390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.598+0000 7fe090583740 4 client.26903 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:23.599 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.598+0000 7fe090583740 1 -- 172.21.15.31:0/2795503489 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bde73e4070 con 0x55bde708ed80 2024-04-25T16:09:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.598+0000 7fe082ffd640 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bde708ed80 0x55bde7499390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.599+0000 7fe066ff5640 10 client.26903 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:23.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.607+0000 7fe082ffd640 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bde708ed80 0x55bde7499390 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d6c95b051fe5f6b server_cookie=e4a4c9c5eae62041 in_seq=0 out_seq=0 2024-04-25T16:09:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.610+0000 7fe066ff5640 1 -- 172.21.15.31:0/2795503489 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bde73e4070 con 0x55bde708ed80 2024-04-25T16:09:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.610+0000 7fe066ff5640 10 client.26903 handle_command_reply: tid=0 2024-04-25T16:09:23.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.610+0000 7fe090583740 1 client.26903 shutdown 2024-04-25T16:09:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.611+0000 7fe090583740 1 -- 172.21.15.31:0/2795503489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 msgr2=0x55bde7481ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:23.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.611+0000 7fe090583740 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 0x55bde7481ad0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe0841aa160 tx=0x7fe0740061c0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.612+0000 7fe090583740 1 -- 172.21.15.31:0/2795503489 shutdown_connections 2024-04-25T16:09:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.612+0000 7fe090583740 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bde708ed80 0x55bde7499390 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.612+0000 7fe090583740 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 0x55bde7481ad0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.612+0000 7fe090583740 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70af4d0 0x55bde747ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.612+0000 7fe090583740 1 --2- 172.21.15.31:0/2795503489 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 0x55bde747a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.612+0000 7fe066ff5640 0 client.26903 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:23.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.612+0000 7fe090583740 1 -- 172.21.15.31:0/2795503489 >> 172.21.15.31:0/2795503489 conn(0x55bde70c4300 msgr2=0x55bde746f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.613+0000 7fe090583740 1 -- 172.21.15.31:0/2795503489 shutdown_connections 2024-04-25T16:09:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.613+0000 7fe090583740 1 -- 172.21.15.31:0/2795503489 wait complete. 2024-04-25T16:09:23.615 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.613+0000 7fe090583740 20 client.26903 trim_cache size 0 max 16384 2024-04-25T16:09:23.700 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.696+0000 7fe090583740 1 Processor -- start 2024-04-25T16:09:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.697+0000 7fe090583740 1 -- start start 2024-04-25T16:09:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.698+0000 7fe090583740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70a72f0 0x55bde7479fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.701 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.698+0000 7fe090583740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70af4d0 0x55bde746bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.698+0000 7fe090583740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 0x55bde746c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.698+0000 7fe090583740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841b1700 con 0x55bde70af4d0 2024-04-25T16:09:23.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.698+0000 7fe090583740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841b3240 con 0x55bde70d01b0 2024-04-25T16:09:23.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.698+0000 7fe090583740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841b1dd0 con 0x55bde70a72f0 2024-04-25T16:09:23.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.701+0000 7fe0827fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70a72f0 0x55bde7479fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.701+0000 7fe0827fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70a72f0 0x55bde7479fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:55506/0 (socket says 172.21.15.31:55506) 2024-04-25T16:09:23.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.701+0000 7fe0827fc640 1 -- 172.21.15.31:0/4133283288 learned_addr learned my addr 172.21.15.31:0/4133283288 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:23.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.702+0000 7fe0827fc640 1 --2- 172.21.15.31:0/4133283288 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 0x55bde746c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.702+0000 7fe0827fc640 1 -- 172.21.15.31:0/4133283288 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 msgr2=0x55bde746c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.702+0000 7fe0827fc640 1 --2- 172.21.15.31:0/4133283288 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 0x55bde746c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.702+0000 7fe0827fc640 1 -- 172.21.15.31:0/4133283288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70af4d0 msgr2=0x55bde746bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.702+0000 7fe0827fc640 1 --2- 172.21.15.31:0/4133283288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70af4d0 0x55bde746bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.702+0000 7fe0827fc640 1 -- 172.21.15.31:0/4133283288 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bde7475540 con 0x55bde70a72f0 2024-04-25T16:09:23.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.703+0000 7fe0827fc640 1 --2- 172.21.15.31:0/4133283288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70a72f0 0x55bde7479fd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe07400b2d0 tx=0x7fe074002000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c91d6fb332ae4186 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.711+0000 7fe068ff9640 1 -- 172.21.15.31:0/4133283288 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074028070 con 0x55bde70a72f0 2024-04-25T16:09:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.711+0000 7fe068ff9640 1 -- 172.21.15.31:0/4133283288 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe074015e60 con 0x55bde70a72f0 2024-04-25T16:09:23.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.711+0000 7fe068ff9640 1 -- 172.21.15.31:0/4133283288 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074018450 con 0x55bde70a72f0 2024-04-25T16:09:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.714+0000 7fe090583740 1 -- 172.21.15.31:0/4133283288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70a72f0 msgr2=0x55bde7479fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.714+0000 7fe090583740 1 --2- 172.21.15.31:0/4133283288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70a72f0 0x55bde7479fd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe07400b2d0 tx=0x7fe074002000 comp rx=0 tx=0).stop 2024-04-25T16:09:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.714+0000 7fe090583740 1 -- 172.21.15.31:0/4133283288 shutdown_connections 2024-04-25T16:09:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.714+0000 7fe090583740 1 --2- 172.21.15.31:0/4133283288 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70d01b0 0x55bde746c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.714+0000 7fe090583740 1 --2- 172.21.15.31:0/4133283288 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70af4d0 0x55bde746bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.714+0000 7fe090583740 1 --2- 172.21.15.31:0/4133283288 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70a72f0 0x55bde7479fd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.714+0000 7fe090583740 1 -- 172.21.15.31:0/4133283288 >> 172.21.15.31:0/4133283288 conn(0x55bde70c4300 msgr2=0x55bde74798c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.715+0000 7fe090583740 1 -- 172.21.15.31:0/4133283288 shutdown_connections 2024-04-25T16:09:23.716 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.715+0000 7fe090583740 1 -- 172.21.15.31:0/4133283288 wait complete. 2024-04-25T16:09:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.716+0000 7fe090583740 1 -- start start 2024-04-25T16:09:23.717 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.716+0000 7fe090583740 1 Processor -- start 2024-04-25T16:09:23.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.720+0000 7fe090583740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 0x55bde747f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.722 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.721+0000 7fe090583740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70af4d0 0x55bde747f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.721+0000 7fe090583740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70d01b0 0x55bde7480470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.721+0000 7fe090583740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841b2b70 con 0x55bde70a72f0 2024-04-25T16:09:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.721+0000 7fe090583740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0841b24a0 con 0x55bde70af4d0 2024-04-25T16:09:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.721+0000 7fe090583740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe084118580 con 0x55bde70d01b0 2024-04-25T16:09:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.722+0000 7fe082ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70af4d0 0x55bde747f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.722+0000 7fe082ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70af4d0 0x55bde747f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:59410/0 (socket says 172.21.15.31:59410) 2024-04-25T16:09:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.722+0000 7fe082ffd640 1 -- 172.21.15.31:0/565537808 learned_addr learned my addr 172.21.15.31:0/565537808 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:23.723 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.722+0000 7fe090583740 10 client.0 _ll_register_callbacks cb 0x7fe074006110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:23.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.722+0000 7fe0827fc640 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 0x55bde747f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.723+0000 7fe082ffd640 1 -- 172.21.15.31:0/565537808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70d01b0 msgr2=0x55bde7480470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:23.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.723+0000 7fe082ffd640 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70d01b0 0x55bde7480470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.723+0000 7fe082ffd640 1 -- 172.21.15.31:0/565537808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 msgr2=0x55bde747f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:23.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.723+0000 7fe082ffd640 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 0x55bde747f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:23.724 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.723+0000 7fe082ffd640 1 -- 172.21.15.31:0/565537808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bde7480a30 con 0x55bde70af4d0 2024-04-25T16:09:23.726 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.725+0000 7fe082ffd640 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70af4d0 0x55bde747f5b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe0780040d0 tx=0x7fe078005d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25f7da0e4e3adc4a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:23.730 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.727+0000 7fe066ff5640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:23.731 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.728+0000 7fe066ff5640 1 -- 172.21.15.31:0/565537808 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe078035040 con 0x55bde70af4d0 2024-04-25T16:09:23.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.729+0000 7fe066ff5640 1 -- 172.21.15.31:0/565537808 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe0780037f0 con 0x55bde70af4d0 2024-04-25T16:09:23.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.730+0000 7fe090583740 1 -- 172.21.15.31:0/565537808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe074015e60 con 0x55bde70af4d0 2024-04-25T16:09:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.730+0000 7fe066ff5640 1 -- 172.21.15.31:0/565537808 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe078040040 con 0x55bde70af4d0 2024-04-25T16:09:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.730+0000 7fe090583740 10 client.26909 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.730+0000 7fe090583740 1 -- 172.21.15.31:0/565537808 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bde7501e00 con 0x55bde70af4d0 2024-04-25T16:09:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.731+0000 7fe066ff5640 1 -- 172.21.15.31:0/565537808 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0780314a0 con 0x55bde70af4d0 2024-04-25T16:09:23.734 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.731+0000 7fe066ff5640 1 -- 172.21.15.31:0/565537808 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe0780510d0 con 0x55bde70af4d0 2024-04-25T16:09:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.731+0000 7fe090583740 10 client.26909 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.731+0000 7fe090583740 10 client.26909 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.731+0000 7fe090583740 20 client.26909 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.731+0000 7fe090583740 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bde70641c0 0x55bde7504780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.731+0000 7fe090583740 4 client.26909 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:09:23.737 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.731+0000 7fe090583740 1 -- 172.21.15.31:0/565537808 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fe074015e60 con 0x55bde70641c0 2024-04-25T16:09:23.753 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.751+0000 7fe0827fc640 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bde70641c0 0x55bde7504780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.775+0000 7fe0827fc640 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bde70641c0 0x55bde7504780 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d92d045cd0424924 server_cookie=542420c34f36dfeb in_seq=0 out_seq=0 2024-04-25T16:09:23.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:23.776+0000 7fe066ff5640 10 client.26909 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:24.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:24 smithi063 ceph-mon[32413]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 954 MiB/s wr, 2.35k op/s 2024-04-25T16:09:24.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:24 smithi186 ceph-mon[33258]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 954 MiB/s wr, 2.35k op/s 2024-04-25T16:09:24.456 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:24 smithi031 ceph-mon[29140]: pgmap v594: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 954 MiB/s wr, 2.35k op/s 2024-04-25T16:09:26.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:26 smithi063 ceph-mon[32413]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 777 MiB/s wr, 1.76k op/s 2024-04-25T16:09:26.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:26 smithi186 ceph-mon[33258]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 777 MiB/s wr, 1.76k op/s 2024-04-25T16:09:26.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:26 smithi031 ceph-mon[29140]: pgmap v595: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 777 MiB/s wr, 1.76k op/s 2024-04-25T16:09:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.533+0000 7fe066ff5640 1 -- 172.21.15.31:0/565537808 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fe074015e60 con 0x55bde70641c0 2024-04-25T16:09:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.533+0000 7fe066ff5640 10 client.26909 handle_command_reply: tid=0 2024-04-25T16:09:26.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.533+0000 7fe090583740 1 client.26909 shutdown 2024-04-25T16:09:26.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.533+0000 7fe090583740 1 -- 172.21.15.31:0/565537808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70af4d0 msgr2=0x55bde747f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:26.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.533+0000 7fe090583740 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70af4d0 0x55bde747f5b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe0780040d0 tx=0x7fe078005d10 comp rx=0 tx=0).stop 2024-04-25T16:09:26.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.547+0000 7fe090583740 1 -- 172.21.15.31:0/565537808 shutdown_connections 2024-04-25T16:09:26.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.547+0000 7fe090583740 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bde70641c0 0x55bde7504780 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:26.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.547+0000 7fe090583740 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bde70d01b0 0x55bde7480470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:26.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.547+0000 7fe090583740 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bde70af4d0 0x55bde747f5b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:26.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.547+0000 7fe090583740 1 --2- 172.21.15.31:0/565537808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bde70a72f0 0x55bde747f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:26.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.547+0000 7fe090583740 1 -- 172.21.15.31:0/565537808 >> 172.21.15.31:0/565537808 conn(0x55bde70c4300 msgr2=0x55bde746f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:26.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.547+0000 7fe066ff5640 0 client.26909 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:26.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.549+0000 7fe090583740 1 -- 172.21.15.31:0/565537808 shutdown_connections 2024-04-25T16:09:26.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.549+0000 7fe090583740 1 -- 172.21.15.31:0/565537808 wait complete. 2024-04-25T16:09:26.552 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.550+0000 7fe090583740 20 client.26909 trim_cache size 0 max 16384 2024-04-25T16:09:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:09:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:09:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:09:26.556 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:09:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 -- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe06c0afbd0 msgr2=0x7fe06c0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe06c0afbd0 0x7fe06c0b20e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe07c014bd0 tx=0x7fe07c004a60 comp rx=0 tx=0).stop 2024-04-25T16:09:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 -- 172.21.15.31:0/4168817475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe084110070 msgr2=0x7fe0841b0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe084110070 0x7fe0841b0590 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe07c00b990 tx=0x7fe07c00be50 comp rx=0 tx=0).stop 2024-04-25T16:09:26.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 -- 172.21.15.31:0/4168817475 shutdown_connections 2024-04-25T16:09:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe06c0afbd0 0x7fe06c0b20e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe084110070 0x7fe0841b0590 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe08410b350 0x7fe0841a9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 --2- 172.21.15.31:0/4168817475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe08410a420 0x7fe0841a90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:26.574 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.571+0000 7fe0896d8640 1 -- 172.21.15.31:0/4168817475 >> 172.21.15.31:0/4168817475 conn(0x7fe084070540 msgr2=0x7fe08410c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.585+0000 7fe0896d8640 1 -- 172.21.15.31:0/4168817475 shutdown_connections 2024-04-25T16:09:26.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:26.586+0000 7fe0896d8640 1 -- 172.21.15.31:0/4168817475 wait complete. 2024-04-25T16:09:28.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:28 smithi063 ceph-mon[32413]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 924 MiB/s wr, 2.17k op/s 2024-04-25T16:09:28.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:28 smithi186 ceph-mon[33258]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 924 MiB/s wr, 2.17k op/s 2024-04-25T16:09:28.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:28 smithi031 ceph-mon[29140]: pgmap v596: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 924 MiB/s wr, 2.17k op/s 2024-04-25T16:09:30.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:30 smithi063 ceph-mon[32413]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 820 MiB/s wr, 1.82k op/s 2024-04-25T16:09:30.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:30 smithi186 ceph-mon[33258]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 820 MiB/s wr, 1.82k op/s 2024-04-25T16:09:30.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:30 smithi031 ceph-mon[29140]: pgmap v597: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 820 MiB/s wr, 1.82k op/s 2024-04-25T16:09:32.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:32 smithi063 ceph-mon[32413]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 770 MiB/s wr, 1.63k op/s 2024-04-25T16:09:32.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:32 smithi186 ceph-mon[33258]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 770 MiB/s wr, 1.63k op/s 2024-04-25T16:09:32.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:32 smithi031 ceph-mon[29140]: pgmap v598: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 770 MiB/s wr, 1.63k op/s 2024-04-25T16:09:34.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:34 smithi063 ceph-mon[32413]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 954 MiB/s wr, 2.05k op/s 2024-04-25T16:09:34.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:34 smithi186 ceph-mon[33258]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 954 MiB/s wr, 2.05k op/s 2024-04-25T16:09:34.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:34 smithi031 ceph-mon[29140]: pgmap v599: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 954 MiB/s wr, 2.05k op/s 2024-04-25T16:09:37.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:37 smithi063 ceph-mon[32413]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 751 MiB/s wr, 1.69k op/s 2024-04-25T16:09:37.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:37 smithi186 ceph-mon[33258]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 751 MiB/s wr, 1.69k op/s 2024-04-25T16:09:37.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:37 smithi031 ceph-mon[29140]: pgmap v600: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 751 MiB/s wr, 1.69k op/s 2024-04-25T16:09:39.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:39 smithi063 ceph-mon[32413]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 942 MiB/s wr, 2.22k op/s 2024-04-25T16:09:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:39 smithi186 ceph-mon[33258]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 942 MiB/s wr, 2.22k op/s 2024-04-25T16:09:39.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:39 smithi031 ceph-mon[29140]: pgmap v601: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 942 MiB/s wr, 2.22k op/s 2024-04-25T16:09:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.158+0000 7f7f68d4d640 1 -- 172.21.15.31:0/4152800750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f64110070 msgr2=0x7f7f64110480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:40.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.158+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/4152800750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f64110070 0x7f7f64110480 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009ca0 tx=0x7f7f5c02a8b0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.166+0000 7f7f68d4d640 1 -- 172.21.15.31:0/4152800750 shutdown_connections 2024-04-25T16:09:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.166+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/4152800750 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f64110070 0x7f7f64110480 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.166+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/4152800750 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f6410b350 0x7f7f6410b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.166+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/4152800750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f6410a420 0x7f7f6410ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.166+0000 7f7f68d4d640 1 -- 172.21.15.31:0/4152800750 >> 172.21.15.31:0/4152800750 conn(0x7f7f64070540 msgr2=0x7f7f640729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:40.176 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.169+0000 7f7f68d4d640 1 -- 172.21.15.31:0/4152800750 shutdown_connections 2024-04-25T16:09:40.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.173+0000 7f7f68d4d640 1 -- 172.21.15.31:0/4152800750 wait complete. 2024-04-25T16:09:40.177 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.174+0000 7f7f68d4d640 1 Processor -- start 2024-04-25T16:09:40.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.177+0000 7f7f68d4d640 1 -- start start 2024-04-25T16:09:40.187 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.177+0000 7f7f68d4d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f6410a420 0x7f7f64117000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.177+0000 7f7f68d4d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f6410b350 0x7f7f64117540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.177+0000 7f7f68d4d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64110070 0x7f7f641184a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.177+0000 7f7f68d4d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6411e4d0 con 0x7f7f6410a420 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.177+0000 7f7f68d4d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6411e350 con 0x7f7f6410b350 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.177+0000 7f7f68d4d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6411e650 con 0x7f7f64110070 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.178+0000 7f7f62d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64110070 0x7f7f641184a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.178+0000 7f7f62d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64110070 0x7f7f641184a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39590/0 (socket says 172.21.15.31:39590) 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.178+0000 7f7f62d9d640 1 -- 172.21.15.31:0/3062930975 learned_addr learned my addr 172.21.15.31:0/3062930975 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.179+0000 7f7f62d9d640 1 -- 172.21.15.31:0/3062930975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f6410b350 msgr2=0x7f7f64117540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.179+0000 7f7f62d9d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f6410b350 0x7f7f64117540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.179+0000 7f7f62d9d640 1 -- 172.21.15.31:0/3062930975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f6410a420 msgr2=0x7f7f64117000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:40.188 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.179+0000 7f7f62d9d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f6410a420 0x7f7f64117000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.179+0000 7f7f62d9d640 1 -- 172.21.15.31:0/3062930975 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f64118b10 con 0x7f7f64110070 2024-04-25T16:09:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.180+0000 7f7f62d9d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64110070 0x7f7f641184a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c005e30 tx=0x7f7f5c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.183+0000 7f7f537fe640 1 -- 172.21.15.31:0/3062930975 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c003e60 con 0x7f7f64110070 2024-04-25T16:09:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.183+0000 7f7f68d4d640 1 -- 172.21.15.31:0/3062930975 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f64118da0 con 0x7f7f64110070 2024-04-25T16:09:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.183+0000 7f7f68d4d640 1 -- 172.21.15.31:0/3062930975 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f641192f0 con 0x7f7f64110070 2024-04-25T16:09:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.183+0000 7f7f537fe640 1 -- 172.21.15.31:0/3062930975 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f5c040070 con 0x7f7f64110070 2024-04-25T16:09:40.189 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.185+0000 7f7f537fe640 1 -- 172.21.15.31:0/3062930975 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c031b20 con 0x7f7f64110070 2024-04-25T16:09:40.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.209+0000 7f7f537fe640 1 -- 172.21.15.31:0/3062930975 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7f5c031cc0 con 0x7f7f64110070 2024-04-25T16:09:40.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.209+0000 7f7f6fbfe740 1 Processor -- start 2024-04-25T16:09:40.211 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.209+0000 7f7f6fbfe740 1 -- start start 2024-04-25T16:09:40.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.212+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038124970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.212+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 0x55f038124ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.214 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.212+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 0x55f03812bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.213+0000 7f7f61d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038124970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.215 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.213+0000 7f7f61d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038124970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39602/0 (socket says 172.21.15.31:39602) 2024-04-25T16:09:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.213+0000 7f7f61d9b640 1 -- 172.21.15.31:0/1131841692 learned_addr learned my addr 172.21.15.31:0/1131841692 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.213+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/1131841692 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f03808a460 con 0x55f037d634d0 2024-04-25T16:09:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.213+0000 7f7f6259c640 1 --2- 172.21.15.31:0/1131841692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 0x55f038124ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.213+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/1131841692 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f038044590 con 0x55f037d5b2f0 2024-04-25T16:09:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.213+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/1131841692 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f038086420 con 0x55f037d841b0 2024-04-25T16:09:40.216 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.213+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/1131841692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 0x55f03812bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.221 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.219+0000 7f7f537fe640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7f4c0afbd0 0x7f7f4c0b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.222 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.220+0000 7f7f537fe640 1 -- 172.21.15.31:0/3062930975 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7f5c0ffbd0 con 0x7f7f64110070 2024-04-25T16:09:40.225 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.223+0000 7f7f62d9d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7f4c0afbd0 0x7f7f4c0b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.223+0000 7f7f6259c640 1 -- 172.21.15.31:0/1131841692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 msgr2=0x55f038124970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.223+0000 7f7f6259c640 1 --2- 172.21.15.31:0/1131841692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038124970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.223+0000 7f7f6259c640 1 -- 172.21.15.31:0/1131841692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 msgr2=0x55f03812bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.223+0000 7f7f6259c640 1 --2- 172.21.15.31:0/1131841692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 0x55f03812bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.226 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.223+0000 7f7f6259c640 1 -- 172.21.15.31:0/1131841692 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0380510d0 con 0x55f037d5b2f0 2024-04-25T16:09:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.226+0000 7f7f6259c640 1 --2- 172.21.15.31:0/1131841692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 0x55f038124ef0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009b10 tx=0x7f7f5400e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e206968eb4f62aac server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.229+0000 7f7f50ff9640 1 -- 172.21.15.31:0/1131841692 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5400ed50 con 0x55f037d5b2f0 2024-04-25T16:09:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.229+0000 7f7f50ff9640 1 -- 172.21.15.31:0/1131841692 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f54005020 con 0x55f037d5b2f0 2024-04-25T16:09:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.229+0000 7f7f50ff9640 1 -- 172.21.15.31:0/1131841692 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54004b10 con 0x55f037d5b2f0 2024-04-25T16:09:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.231+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/1131841692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 msgr2=0x55f038124ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.231+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/1131841692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 0x55f038124ef0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009b10 tx=0x7f7f5400e350 comp rx=0 tx=0).stop 2024-04-25T16:09:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.231+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/1131841692 shutdown_connections 2024-04-25T16:09:40.234 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.231+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/1131841692 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 0x55f03812bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.231+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/1131841692 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 0x55f038124ef0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.231+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/1131841692 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038124970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.235 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.231+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/1131841692 >> 172.21.15.31:0/1131841692 conn(0x55f037d78300 msgr2=0x55f038120300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.240+0000 7f7f62d9d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7f4c0afbd0 0x7f7f4c0b20e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c031fd0 tx=0x7f7f5c02adb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:40.242 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.240+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/1131841692 shutdown_connections 2024-04-25T16:09:40.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.240+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/1131841692 wait complete. 2024-04-25T16:09:40.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.241+0000 7f7f6fbfe740 1 -- start start 2024-04-25T16:09:40.243 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.241+0000 7f7f6fbfe740 1 Processor -- start 2024-04-25T16:09:40.244 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.242+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f03812e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.243+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f03812ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.243+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d841b0 0x55f038135b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.243+0000 7f7f6fbfe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f03812c840 con 0x55f037d841b0 2024-04-25T16:09:40.245 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.243+0000 7f7f6fbfe740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f038011b60 con 0x55f037d634d0 2024-04-25T16:09:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.243+0000 7f7f6fbfe740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f03812c9c0 con 0x55f037d5b2f0 2024-04-25T16:09:40.246 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.243+0000 7f7f61d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f03812e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.243+0000 7f7f61d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f03812e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39618/0 (socket says 172.21.15.31:39618) 2024-04-25T16:09:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.243+0000 7f7f61d9b640 1 -- 172.21.15.31:0/194595523 learned_addr learned my addr 172.21.15.31:0/194595523 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.244+0000 7f7f6259c640 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f03812ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.244+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d841b0 0x55f038135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.244+0000 7f7f61d9b640 1 -- 172.21.15.31:0/194595523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 msgr2=0x55f03812ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:40.265 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.244+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f03812ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.244+0000 7f7f61d9b640 1 -- 172.21.15.31:0/194595523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d841b0 msgr2=0x55f038135b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.244+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d841b0 0x55f038135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.244+0000 7f7f61d9b640 1 -- 172.21.15.31:0/194595523 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f037d9c210 con 0x55f037d5b2f0 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.245+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f03812e6e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7f58004640 tx=0x7f7f58004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51b33a81d4c0c9fb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.260+0000 7f7f49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.260+0000 7f7f49ffb640 1 -- 172.21.15.31:0/194595523 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f58004020 con 0x55f037d5b2f0 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.260+0000 7f7f49ffb640 1 -- 172.21.15.31:0/194595523 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7f580041c0 con 0x55f037d5b2f0 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.261+0000 7f7f49ffb640 10 client.0 inactive, discarding osd_map(77..77 src has 1..77) 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.261+0000 7f7f6fbfe740 10 client.0 _ll_register_callbacks cb 0x55f0380362b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:40.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.261+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/194595523 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f54005020 con 0x55f037d5b2f0 2024-04-25T16:09:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.264+0000 7f7f49ffb640 1 -- 172.21.15.31:0/194595523 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7f58024050 con 0x55f037d5b2f0 2024-04-25T16:09:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.264+0000 7f7f6fbfe740 10 client.26896 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:40.268 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.265+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/194595523 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55f037e68620 con 0x55f037d5b2f0 2024-04-25T16:09:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.267+0000 7f7f49ffb640 1 -- 172.21.15.31:0/194595523 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7f580035f0 con 0x55f037d5b2f0 2024-04-25T16:09:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.267+0000 7f7f6fbfe740 10 client.26896 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.267+0000 7f7f6fbfe740 10 client.26896 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.267+0000 7f7f6fbfe740 20 client.26896 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.267+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f037d42d80 0x55f03813c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.268+0000 7f7f6fbfe740 4 client.26896 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:40.270 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.268+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/194595523 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f038098070 con 0x55f037d42d80 2024-04-25T16:09:40.275 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.273+0000 7f7f6259c640 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f037d42d80 0x55f03813c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.289+0000 7f7f6259c640 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f037d42d80 0x55f03813c600 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13565b06c78b3a83 server_cookie=d566cf3d4bc7cbaa in_seq=0 out_seq=0 2024-04-25T16:09:40.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.289+0000 7f7f49ffb640 10 client.26896 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.297+0000 7f7f49ffb640 1 -- 172.21.15.31:0/194595523 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f038098070 con 0x55f037d42d80 2024-04-25T16:09:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.297+0000 7f7f49ffb640 10 client.26896 handle_command_reply: tid=0 2024-04-25T16:09:40.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.297+0000 7f7f6fbfe740 1 client.26896 shutdown 2024-04-25T16:09:40.309 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.306+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/194595523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 msgr2=0x55f03812e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:40.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.306+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f03812e6e0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7f58004640 tx=0x7f7f58004730 comp rx=0 tx=0).stop 2024-04-25T16:09:40.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.308+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/194595523 shutdown_connections 2024-04-25T16:09:40.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.308+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f037d42d80 0x55f03813c600 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.308+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d841b0 0x55f038135b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.308+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f03812ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.308+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/194595523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f03812e6e0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.308+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/194595523 >> 172.21.15.31:0/194595523 conn(0x55f037d78300 msgr2=0x55f038123210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:40.313 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.310+0000 7f7f49ffb640 0 client.26896 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:40.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.319+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/194595523 shutdown_connections 2024-04-25T16:09:40.328 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.323+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/194595523 wait complete. 2024-04-25T16:09:40.330 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.326+0000 7f7f6fbfe740 20 client.26896 trim_cache size 0 max 16384 2024-04-25T16:09:40.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:40 smithi063 ceph-mon[32413]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 824 MiB/s wr, 1.97k op/s 2024-04-25T16:09:40.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:40 smithi186 ceph-mon[33258]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 824 MiB/s wr, 1.97k op/s 2024-04-25T16:09:40.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.417+0000 7f7f6fbfe740 1 Processor -- start 2024-04-25T16:09:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.421+0000 7f7f6fbfe740 1 -- start start 2024-04-25T16:09:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.421+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 0x55f03811fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.421+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 0x55f038135b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.422+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038120150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.422+0000 7f7f6fbfe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4400e470 con 0x55f037d634d0 2024-04-25T16:09:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.422+0000 7f7f6fbfe740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4400ffb0 con 0x55f037d5b2f0 2024-04-25T16:09:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.422+0000 7f7f6fbfe740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4400eb40 con 0x55f037d841b0 2024-04-25T16:09:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.424+0000 7f7f61d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038120150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.424+0000 7f7f61d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 0x55f03811fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.426+0000 7f7f61d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038120150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:39632/0 (socket says 172.21.15.31:39632) 2024-04-25T16:09:40.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.426+0000 7f7f61d9b640 1 -- 172.21.15.31:0/591957076 learned_addr learned my addr 172.21.15.31:0/591957076 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.431+0000 7f7f6259c640 1 --2- 172.21.15.31:0/591957076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 0x55f038135b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.433+0000 7f7f61d9b640 1 -- 172.21.15.31:0/591957076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 msgr2=0x55f03811fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.433+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/591957076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 0x55f03811fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.433+0000 7f7f61d9b640 1 -- 172.21.15.31:0/591957076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 msgr2=0x55f038135b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.433+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/591957076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 0x55f038135b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.437 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.433+0000 7f7f61d9b640 1 -- 172.21.15.31:0/591957076 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f03812be90 con 0x55f037d841b0 2024-04-25T16:09:40.444 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.438+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/591957076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038120150 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7f580118b0 tx=0x7f7f5800b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5f91cad93b2e118 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.442+0000 7f7f4bfff640 1 -- 172.21.15.31:0/591957076 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f580036a0 con 0x55f037d841b0 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.442+0000 7f7f4bfff640 1 -- 172.21.15.31:0/591957076 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f54005020 con 0x55f037d841b0 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.442+0000 7f7f4bfff640 1 -- 172.21.15.31:0/591957076 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5801d6d0 con 0x55f037d841b0 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.444+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/591957076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 msgr2=0x55f038120150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.444+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/591957076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038120150 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7f580118b0 tx=0x7f7f5800b000 comp rx=0 tx=0).stop 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.444+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/591957076 shutdown_connections 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.444+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/591957076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d841b0 0x55f038120150 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7f580118b0 tx=0x7f7f5800b000 comp rx=0 tx=0).stop 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.444+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/591957076 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d634d0 0x55f038135b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.444+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/591957076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d5b2f0 0x55f03811fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.448 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.444+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/591957076 >> 172.21.15.31:0/591957076 conn(0x55f037d78300 msgr2=0x55f038128ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:40.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:40 smithi031 ceph-mon[29140]: pgmap v602: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 824 MiB/s wr, 1.97k op/s 2024-04-25T16:09:40.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.449+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/591957076 shutdown_connections 2024-04-25T16:09:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.452+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/591957076 wait complete. 2024-04-25T16:09:40.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.453+0000 7f7f6fbfe740 1 -- start start 2024-04-25T16:09:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.453+0000 7f7f6fbfe740 1 Processor -- start 2024-04-25T16:09:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.454+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f038132800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.454+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f038132d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.454+0000 7f7f6fbfe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d181c0 0x55f0381a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.454+0000 7f7f6fbfe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4400f8e0 con 0x55f037d181c0 2024-04-25T16:09:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.454+0000 7f7f6fbfe740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4400f210 con 0x55f037d634d0 2024-04-25T16:09:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.454+0000 7f7f6fbfe740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4400c450 con 0x55f037d5b2f0 2024-04-25T16:09:40.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.454+0000 7f7f6fbfe740 10 client.0 _ll_register_callbacks cb 0x7f7f58011970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.457+0000 7f7f6259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f038132d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.457+0000 7f7f6259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f038132d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:37294/0 (socket says 172.21.15.31:37294) 2024-04-25T16:09:40.460 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.457+0000 7f7f6259c640 1 -- 172.21.15.31:0/2821236419 learned_addr learned my addr 172.21.15.31:0/2821236419 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.459+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d181c0 0x55f0381a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.459+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f038132800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.462+0000 7f7f6259c640 1 -- 172.21.15.31:0/2821236419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 msgr2=0x55f038132800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.462+0000 7f7f6259c640 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f038132800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.462+0000 7f7f6259c640 1 -- 172.21.15.31:0/2821236419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d181c0 msgr2=0x55f0381a29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.462+0000 7f7f6259c640 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d181c0 0x55f0381a29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:40.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.462+0000 7f7f6259c640 1 -- 172.21.15.31:0/2821236419 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f038135d00 con 0x55f037d634d0 2024-04-25T16:09:40.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.472+0000 7f7f6259c640 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f038132d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009fd0 tx=0x7f7f5400eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d646dde6340fd9e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:40.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.473+0000 7f7f49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:40.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.473+0000 7f7f49ffb640 1 -- 172.21.15.31:0/2821236419 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f540053e0 con 0x55f037d634d0 2024-04-25T16:09:40.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.473+0000 7f7f49ffb640 1 -- 172.21.15.31:0/2821236419 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7f54005580 con 0x55f037d634d0 2024-04-25T16:09:40.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.476+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/2821236419 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f54005020 con 0x55f037d634d0 2024-04-25T16:09:40.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.479+0000 7f7f49ffb640 1 -- 172.21.15.31:0/2821236419 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7f54037050 con 0x55f037d634d0 2024-04-25T16:09:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.486+0000 7f7f6fbfe740 10 client.26924 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:40.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.486+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/2821236419 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f0381a3e80 con 0x55f037d634d0 2024-04-25T16:09:40.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.493+0000 7f7f49ffb640 1 -- 172.21.15.31:0/2821236419 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7f54005020 con 0x55f037d634d0 2024-04-25T16:09:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.493+0000 7f7f49ffb640 1 -- 172.21.15.31:0/2821236419 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7f54004cd0 con 0x55f037d634d0 2024-04-25T16:09:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.494+0000 7f7f6fbfe740 10 client.26924 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.494+0000 7f7f6fbfe740 10 client.26924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:40.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.494+0000 7f7f6fbfe740 20 client.26924 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.494+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f0381a46d0 0x55f0381a6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.494+0000 7f7f6fbfe740 4 client.26924 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:09:40.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.494+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/2821236419 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f038091ee0 con 0x55f0381a46d0 2024-04-25T16:09:40.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.497+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f0381a46d0 0x55f0381a6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:40.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.499+0000 7f7f61d9b640 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f0381a46d0 0x55f0381a6af0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d42638dae2439c7 server_cookie=63c9caab1b0bc508 in_seq=0 out_seq=0 2024-04-25T16:09:40.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:40.500+0000 7f7f49ffb640 10 client.26924 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:43.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:09:43.194 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:09:43.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:43 smithi063 ceph-mon[32413]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 761 MiB/s wr, 1.87k op/s 2024-04-25T16:09:43.395 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:43 smithi031 ceph-mon[29140]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 761 MiB/s wr, 1.87k op/s 2024-04-25T16:09:43.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:43 smithi186 ceph-mon[33258]: pgmap v603: 97 pgs: 97 active+clean; 10 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 761 MiB/s wr, 1.87k op/s 2024-04-25T16:09:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.716+0000 7fac9759e640 1 -- 172.21.15.31:0/2128612240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac9810fe90 msgr2=0x7fac981102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:43.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.716+0000 7fac9759e640 1 --2- 172.21.15.31:0/2128612240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac9810fe90 0x7fac981102a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009860 tx=0x7fac8c02a380 comp rx=0 tx=0).stop 2024-04-25T16:09:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.716+0000 7fac9759e640 1 -- 172.21.15.31:0/2128612240 shutdown_connections 2024-04-25T16:09:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.716+0000 7fac9759e640 1 --2- 172.21.15.31:0/2128612240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac9810fe90 0x7fac981102a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.716+0000 7fac9759e640 1 --2- 172.21.15.31:0/2128612240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac9810b190 0x7fac9810b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.716+0000 7fac9759e640 1 --2- 172.21.15.31:0/2128612240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9810a340 0x7fac9810ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.716+0000 7fac9759e640 1 -- 172.21.15.31:0/2128612240 >> 172.21.15.31:0/2128612240 conn(0x7fac98070320 msgr2=0x7fac980727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:43.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.719+0000 7fac9759e640 1 -- 172.21.15.31:0/2128612240 shutdown_connections 2024-04-25T16:09:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.719+0000 7fac9759e640 1 -- 172.21.15.31:0/2128612240 wait complete. 2024-04-25T16:09:43.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.721+0000 7fac9759e640 1 Processor -- start 2024-04-25T16:09:43.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac9759e640 1 -- start start 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac9759e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac9810a340 0x7fac981a8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac9759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac9810b190 0x7fac981a9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac9759e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9810fe90 0x7fac981b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac9759e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fac9811e2a0 con 0x7fac9810fe90 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac9759e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fac9811e120 con 0x7fac9810a340 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac9759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fac9811e420 con 0x7fac9810b190 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac96d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9810fe90 0x7fac981b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac96d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9810fe90 0x7fac981b03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60080/0 (socket says 172.21.15.31:60080) 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac96d9d640 1 -- 172.21.15.31:0/1836134778 learned_addr learned my addr 172.21.15.31:0/1836134778 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:43.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.722+0000 7fac95d9b640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac9810b190 0x7fac981a9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.723+0000 7fac95d9b640 1 -- 172.21.15.31:0/1836134778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac9810a340 msgr2=0x7fac981a8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.723+0000 7fac95d9b640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac9810a340 0x7fac981a8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.723+0000 7fac95d9b640 1 -- 172.21.15.31:0/1836134778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9810fe90 msgr2=0x7fac981b03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.723+0000 7fac95d9b640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9810fe90 0x7fac981b03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.723+0000 7fac95d9b640 1 -- 172.21.15.31:0/1836134778 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac981b0a20 con 0x7fac9810b190 2024-04-25T16:09:43.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.724+0000 7fac95d9b640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac9810b190 0x7fac981a9420 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fac9000ed70 tx=0x7fac9000c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:43.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.731+0000 7fac877fe640 1 -- 172.21.15.31:0/1836134778 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9001c070 con 0x7fac9810b190 2024-04-25T16:09:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.731+0000 7fac9759e640 1 -- 172.21.15.31:0/1836134778 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac981b0d10 con 0x7fac9810b190 2024-04-25T16:09:43.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.731+0000 7fac9759e640 1 -- 172.21.15.31:0/1836134778 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac981b11b0 con 0x7fac9810b190 2024-04-25T16:09:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.738+0000 7faca3972740 1 Processor -- start 2024-04-25T16:09:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.738+0000 7fac877fe640 1 -- 172.21.15.31:0/1836134778 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac90017af0 con 0x7fac9810b190 2024-04-25T16:09:43.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.738+0000 7fac877fe640 1 -- 172.21.15.31:0/1836134778 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9000fb10 con 0x7fac9810b190 2024-04-25T16:09:43.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.742+0000 7faca3972740 1 -- start start 2024-04-25T16:09:43.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.743+0000 7faca3972740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c25ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.743+0000 7faca3972740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be944d0 0x55930c25f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.743+0000 7faca3972740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 0x55930c25fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.743+0000 7faca3972740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55930c1c3d80 con 0x55930be9c6b0 2024-04-25T16:09:43.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.743+0000 7faca3972740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55930c17d8b0 con 0x55930be944d0 2024-04-25T16:09:43.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.743+0000 7faca3972740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55930c1bfd40 con 0x55930bebd390 2024-04-25T16:09:43.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.743+0000 7fac877fe640 1 -- 172.21.15.31:0/1836134778 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fac9001f070 con 0x7fac9810b190 2024-04-25T16:09:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.744+0000 7fac9659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be944d0 0x55930c25f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.744+0000 7fac9659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be944d0 0x55930c25f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58086/0 (socket says 172.21.15.31:58086) 2024-04-25T16:09:43.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.745+0000 7fac9659c640 1 -- 172.21.15.31:0/1663427731 learned_addr learned my addr 172.21.15.31:0/1663427731 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.745+0000 7fac96d9d640 1 --2- 172.21.15.31:0/1663427731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c25ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.745+0000 7fac95d9b640 1 --2- 172.21.15.31:0/1663427731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 0x55930c25fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.746+0000 7fac9659c640 1 -- 172.21.15.31:0/1663427731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 msgr2=0x55930c25ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.746+0000 7fac9659c640 1 --2- 172.21.15.31:0/1663427731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c25ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.746+0000 7fac9659c640 1 -- 172.21.15.31:0/1663427731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 msgr2=0x55930c25fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.746+0000 7fac9659c640 1 --2- 172.21.15.31:0/1663427731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 0x55930c25fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.746+0000 7fac9659c640 1 -- 172.21.15.31:0/1663427731 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55930c14ea40 con 0x55930be944d0 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.747+0000 7fac9659c640 1 --2- 172.21.15.31:0/1663427731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be944d0 0x55930c25f1d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fac880097b0 tx=0x7fac88009c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=831534debe3175d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.751+0000 7fac877fe640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fac800afcf0 0x7fac800b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.751+0000 7fac877fe640 1 -- 172.21.15.31:0/1836134778 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fac900dea70 con 0x7fac9810b190 2024-04-25T16:09:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.753+0000 7fac84ff9640 1 -- 172.21.15.31:0/1663427731 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac88015020 con 0x55930be944d0 2024-04-25T16:09:43.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.753+0000 7fac84ff9640 1 -- 172.21.15.31:0/1663427731 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac88011420 con 0x55930be944d0 2024-04-25T16:09:43.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.754+0000 7faca3972740 1 -- 172.21.15.31:0/1663427731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be944d0 msgr2=0x55930c25f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:43.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.754+0000 7faca3972740 1 --2- 172.21.15.31:0/1663427731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be944d0 0x55930c25f1d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fac880097b0 tx=0x7fac88009c70 comp rx=0 tx=0).stop 2024-04-25T16:09:43.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.761+0000 7faca3972740 1 -- 172.21.15.31:0/1663427731 shutdown_connections 2024-04-25T16:09:43.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.761+0000 7faca3972740 1 --2- 172.21.15.31:0/1663427731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 0x55930c25fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.761+0000 7faca3972740 1 --2- 172.21.15.31:0/1663427731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be944d0 0x55930c25f1d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.761+0000 7faca3972740 1 --2- 172.21.15.31:0/1663427731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c25ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.761+0000 7faca3972740 1 -- 172.21.15.31:0/1663427731 >> 172.21.15.31:0/1663427731 conn(0x55930beb14e0 msgr2=0x55930c25a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:43.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.762+0000 7faca3972740 1 -- 172.21.15.31:0/1663427731 shutdown_connections 2024-04-25T16:09:43.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.767+0000 7faca3972740 1 -- 172.21.15.31:0/1663427731 wait complete. 2024-04-25T16:09:43.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.767+0000 7faca3972740 1 -- start start 2024-04-25T16:09:43.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.768+0000 7faca3972740 1 Processor -- start 2024-04-25T16:09:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.768+0000 7faca3972740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930be944d0 0x55930c279b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.768+0000 7faca3972740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 0x55930c27a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.768+0000 7faca3972740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930bebd390 0x55930c280fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.768+0000 7faca3972740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55930c2609c0 con 0x55930be9c6b0 2024-04-25T16:09:43.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.768+0000 7faca3972740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55930c14ad10 con 0x55930bebd390 2024-04-25T16:09:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.768+0000 7faca3972740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55930c260b40 con 0x55930be944d0 2024-04-25T16:09:43.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.768+0000 7faca3972740 10 client.0 _ll_register_callbacks cb 0x55930c177210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:43.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.773+0000 7fac96d9d640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fac800afcf0 0x7fac800b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.773+0000 7fac96d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930bebd390 0x55930c280fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.773+0000 7fac96d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930bebd390 0x55930c280fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58102/0 (socket says 172.21.15.31:58102) 2024-04-25T16:09:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.773+0000 7fac96d9d640 1 -- 172.21.15.31:0/2403458159 learned_addr learned my addr 172.21.15.31:0/2403458159 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.773+0000 7fac95d9b640 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 0x55930c27a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.774+0000 7fac9659c640 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930be944d0 0x55930c279b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.778+0000 7fac95d9b640 1 -- 172.21.15.31:0/2403458159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930be944d0 msgr2=0x55930c279b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:43.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.778+0000 7fac95d9b640 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930be944d0 0x55930c279b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.778+0000 7fac95d9b640 1 -- 172.21.15.31:0/2403458159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930bebd390 msgr2=0x55930c280fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:43.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.778+0000 7fac95d9b640 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930bebd390 0x55930c280fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:43.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.778+0000 7fac95d9b640 1 -- 172.21.15.31:0/2403458159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55930bf9f7d0 con 0x55930be9c6b0 2024-04-25T16:09:43.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.779+0000 7fac95d9b640 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 0x55930c27a050 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fac9000efd0 tx=0x7fac900a75d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9b32c139d986b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:43.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.783+0000 7fac7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:43.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.786+0000 7fac96d9d640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fac800afcf0 0x7fac800b2200 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009830 tx=0x7fac8c030e60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:43.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.786+0000 7fac7dffb640 1 -- 172.21.15.31:0/2403458159 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9000fcb0 con 0x55930be9c6b0 2024-04-25T16:09:43.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.787+0000 7faca3972740 1 -- 172.21.15.31:0/2403458159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac88011420 con 0x55930be9c6b0 2024-04-25T16:09:43.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.787+0000 7fac7dffb640 1 -- 172.21.15.31:0/2403458159 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fac900a6210 con 0x55930be9c6b0 2024-04-25T16:09:43.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.787+0000 7fac7dffb640 1 -- 172.21.15.31:0/2403458159 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac900a6ac0 con 0x55930be9c6b0 2024-04-25T16:09:43.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.788+0000 7faca3972740 10 client.17250 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:43.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.788+0000 7faca3972740 1 -- 172.21.15.31:0/2403458159 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55930bed53f0 con 0x55930be9c6b0 2024-04-25T16:09:43.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.789+0000 7fac7dffb640 1 -- 172.21.15.31:0/2403458159 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac88011420 con 0x55930be9c6b0 2024-04-25T16:09:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.790+0000 7fac7dffb640 1 -- 172.21.15.31:0/2403458159 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fac9000b030 con 0x55930be9c6b0 2024-04-25T16:09:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.791+0000 7faca3972740 10 client.17250 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.791+0000 7faca3972740 10 client.17250 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.791+0000 7faca3972740 20 client.17250 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.791+0000 7faca3972740 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55930be7bf60 0x55930c2875a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.791+0000 7faca3972740 4 client.17250 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:43.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.791+0000 7faca3972740 1 -- 172.21.15.31:0/2403458159 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55930c1d1990 con 0x55930be7bf60 2024-04-25T16:09:43.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.794+0000 7fac9659c640 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55930be7bf60 0x55930c2875a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:43.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.815+0000 7fac9659c640 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55930be7bf60 0x55930c2875a0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9996d26c9379fe8 server_cookie=f6bb69a4393f8646 in_seq=0 out_seq=0 2024-04-25T16:09:43.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:43.816+0000 7fac7dffb640 10 client.17250 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:45.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:45 smithi063 ceph-mon[32413]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 937 MiB/s wr, 2.36k op/s 2024-04-25T16:09:45.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:45 smithi186 ceph-mon[33258]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 937 MiB/s wr, 2.36k op/s 2024-04-25T16:09:45.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:45 smithi031 ceph-mon[29140]: pgmap v604: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 937 MiB/s wr, 2.36k op/s 2024-04-25T16:09:45.534 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.532+0000 7f7f49ffb640 1 -- 172.21.15.31:0/2821236419 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f038091ee0 con 0x55f0381a46d0 2024-04-25T16:09:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.532+0000 7f7f49ffb640 10 client.26924 handle_command_reply: tid=0 2024-04-25T16:09:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 client.26924 shutdown 2024-04-25T16:09:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/2821236419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 msgr2=0x55f038132d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:45.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f038132d40 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009fd0 tx=0x7f7f5400eec0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/2821236419 shutdown_connections 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f0381a46d0 0x55f0381a6af0 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f037d181c0 0x55f0381a29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f037d634d0 0x55f038132d40 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 --2- 172.21.15.31:0/2821236419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f037d5b2f0 0x55f038132800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/2821236419 >> 172.21.15.31:0/2821236419 conn(0x55f037d78300 msgr2=0x55f038123100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f49ffb640 0 client.26924 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/2821236419 shutdown_connections 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.533+0000 7f7f6fbfe740 1 -- 172.21.15.31:0/2821236419 wait complete. 2024-04-25T16:09:45.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.534+0000 7f7f6fbfe740 20 client.26924 trim_cache size 0 max 16384 2024-04-25T16:09:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:09:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:09:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:09:45.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:09:45.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.537+0000 7fac7dffb640 1 -- 172.21.15.31:0/2403458159 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55930c1d1990 con 0x55930be7bf60 2024-04-25T16:09:45.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.537+0000 7fac7dffb640 10 client.17250 handle_command_reply: tid=0 2024-04-25T16:09:45.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.537+0000 7faca3972740 1 client.17250 shutdown 2024-04-25T16:09:45.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 -- 172.21.15.31:0/2403458159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 msgr2=0x55930c27a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:45.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 0x55930c27a050 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fac9000efd0 tx=0x7fac900a75d0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.538+0000 7f7f68d4d640 1 -- 172.21.15.31:0/3062930975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7f4c0afbd0 msgr2=0x7f7f4c0b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:45.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.538+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7f4c0afbd0 0x7f7f4c0b20e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c031fd0 tx=0x7f7f5c02adb0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.538+0000 7f7f68d4d640 1 -- 172.21.15.31:0/3062930975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64110070 msgr2=0x7f7f641184a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:45.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.538+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64110070 0x7f7f641184a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c005e30 tx=0x7f7f5c004ab0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 -- 172.21.15.31:0/2403458159 shutdown_connections 2024-04-25T16:09:45.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55930be7bf60 0x55930c2875a0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930bebd390 0x55930c280fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be9c6b0 0x55930c27a050 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 --2- 172.21.15.31:0/2403458159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930be944d0 0x55930c279b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 -- 172.21.15.31:0/2403458159 >> 172.21.15.31:0/2403458159 conn(0x55930beb14e0 msgr2=0x55930c25d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:45.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7fac7dffb640 0 client.17250 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:45.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 -- 172.21.15.31:0/2403458159 shutdown_connections 2024-04-25T16:09:45.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.538+0000 7faca3972740 1 -- 172.21.15.31:0/2403458159 wait complete. 2024-04-25T16:09:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.538+0000 7f7f68d4d640 1 -- 172.21.15.31:0/3062930975 shutdown_connections 2024-04-25T16:09:45.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.538+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7f4c0afbd0 0x7f7f4c0b20e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.539+0000 7faca3972740 20 client.17250 trim_cache size 0 max 16384 2024-04-25T16:09:45.547 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.539+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f64110070 0x7f7f641184a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.545+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7f6410b350 0x7f7f64117540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.545+0000 7f7f68d4d640 1 --2- 172.21.15.31:0/3062930975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7f6410a420 0x7f7f64117000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.545+0000 7f7f68d4d640 1 -- 172.21.15.31:0/3062930975 >> 172.21.15.31:0/3062930975 conn(0x7f7f64070540 msgr2=0x7f7f6410c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.545+0000 7f7f68d4d640 1 -- 172.21.15.31:0/3062930975 shutdown_connections 2024-04-25T16:09:45.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:45.545+0000 7f7f68d4d640 1 -- 172.21.15.31:0/3062930975 wait complete. 2024-04-25T16:09:45.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.587+0000 7faca3972740 1 Processor -- start 2024-04-25T16:09:45.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.587+0000 7faca3972740 1 -- start start 2024-04-25T16:09:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.596+0000 7faca3972740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be944d0 0x55930c25ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.596+0000 7faca3972740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c28cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:45.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.597+0000 7faca3972740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c28d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.597+0000 7fac95d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c28cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.597+0000 7fac95d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c28cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58114/0 (socket says 172.21.15.31:58114) 2024-04-25T16:09:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.597+0000 7fac95d9b640 1 -- 172.21.15.31:0/123315344 learned_addr learned my addr 172.21.15.31:0/123315344 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.597+0000 7faca3972740 1 -- 172.21.15.31:0/123315344 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fac7800e590 con 0x55930be944d0 2024-04-25T16:09:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.598+0000 7faca3972740 1 -- 172.21.15.31:0/123315344 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fac780100d0 con 0x55930be9c6b0 2024-04-25T16:09:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.598+0000 7fac96d9d640 1 --2- 172.21.15.31:0/123315344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c28d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:45.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.598+0000 7faca3972740 1 -- 172.21.15.31:0/123315344 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fac7800ec60 con 0x55930bebd390 2024-04-25T16:09:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.605+0000 7fac9659c640 1 --2- 172.21.15.31:0/123315344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be944d0 0x55930c25ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.605+0000 7fac95d9b640 1 -- 172.21.15.31:0/123315344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 msgr2=0x55930c28d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.605+0000 7fac95d9b640 1 --2- 172.21.15.31:0/123315344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c28d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.605+0000 7fac95d9b640 1 -- 172.21.15.31:0/123315344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be944d0 msgr2=0x55930c25ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:45.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.605+0000 7fac95d9b640 1 --2- 172.21.15.31:0/123315344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be944d0 0x55930c25ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.605+0000 7fac95d9b640 1 -- 172.21.15.31:0/123315344 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55930c263e90 con 0x55930be9c6b0 2024-04-25T16:09:45.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.610+0000 7fac95d9b640 1 --2- 172.21.15.31:0/123315344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c28cd60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fac900a32e0 tx=0x7fac900098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76aea94f2f226177 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.612+0000 7fac7ffff640 1 -- 172.21.15.31:0/123315344 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac9000fcb0 con 0x55930be9c6b0 2024-04-25T16:09:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.612+0000 7fac7ffff640 1 -- 172.21.15.31:0/123315344 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac900a2f30 con 0x55930be9c6b0 2024-04-25T16:09:45.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.612+0000 7fac7ffff640 1 -- 172.21.15.31:0/123315344 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac900a62c0 con 0x55930be9c6b0 2024-04-25T16:09:45.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.614+0000 7faca3972740 1 -- 172.21.15.31:0/123315344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 msgr2=0x55930c28cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:45.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.614+0000 7faca3972740 1 --2- 172.21.15.31:0/123315344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c28cd60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fac900a32e0 tx=0x7fac900098f0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.615+0000 7faca3972740 1 -- 172.21.15.31:0/123315344 shutdown_connections 2024-04-25T16:09:45.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.615+0000 7faca3972740 1 --2- 172.21.15.31:0/123315344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c28d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.615+0000 7faca3972740 1 --2- 172.21.15.31:0/123315344 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c28cd60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.615+0000 7faca3972740 1 --2- 172.21.15.31:0/123315344 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be944d0 0x55930c25ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.615+0000 7faca3972740 1 -- 172.21.15.31:0/123315344 >> 172.21.15.31:0/123315344 conn(0x55930beb14e0 msgr2=0x55930c278460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.615+0000 7faca3972740 1 -- 172.21.15.31:0/123315344 shutdown_connections 2024-04-25T16:09:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.615+0000 7faca3972740 1 -- 172.21.15.31:0/123315344 wait complete. 2024-04-25T16:09:45.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.616+0000 7faca3972740 1 -- start start 2024-04-25T16:09:45.619 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.617+0000 7faca3972740 1 Processor -- start 2024-04-25T16:09:45.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.618+0000 7faca3972740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be944d0 0x55930c280600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:45.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.618+0000 7faca3972740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c2eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.618+0000 7faca3972740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c2edfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.619+0000 7faca3972740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fac7800fa00 con 0x55930be944d0 2024-04-25T16:09:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.619+0000 7faca3972740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fac7800f330 con 0x55930be9c6b0 2024-04-25T16:09:45.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.619+0000 7fac95d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c2eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.619+0000 7fac95d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c2eb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58128/0 (socket says 172.21.15.31:58128) 2024-04-25T16:09:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.619+0000 7fac95d9b640 1 -- 172.21.15.31:0/1457407263 learned_addr learned my addr 172.21.15.31:0/1457407263 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.619+0000 7faca3972740 1 -- 172.21.15.31:0/1457407263 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fac7800c570 con 0x55930bebd390 2024-04-25T16:09:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.619+0000 7fac96d9d640 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c2edfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:45.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.620+0000 7faca3972740 10 client.0 _ll_register_callbacks cb 0x7fac900a6e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:45.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.622+0000 7fac95d9b640 1 -- 172.21.15.31:0/1457407263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 msgr2=0x55930c2edfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:45.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.622+0000 7fac95d9b640 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c2edfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.622+0000 7fac95d9b640 1 -- 172.21.15.31:0/1457407263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be944d0 msgr2=0x55930c280600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:45.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.622+0000 7fac95d9b640 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be944d0 0x55930c280600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.622+0000 7fac95d9b640 1 -- 172.21.15.31:0/1457407263 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55930c25f2b0 con 0x55930be9c6b0 2024-04-25T16:09:45.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.623+0000 7fac95d9b640 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c2eb8f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac900a6e20 tx=0x7fac900dc5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e8cac2110c0b03d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:45.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.623+0000 7fac7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:45.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.623+0000 7fac7dffb640 1 -- 172.21.15.31:0/1457407263 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac900c5070 con 0x55930be9c6b0 2024-04-25T16:09:45.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.624+0000 7fac7dffb640 1 -- 172.21.15.31:0/1457407263 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fac900a6280 con 0x55930be9c6b0 2024-04-25T16:09:45.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.624+0000 7faca3972740 1 -- 172.21.15.31:0/1457407263 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac900a2f30 con 0x55930be9c6b0 2024-04-25T16:09:45.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.625+0000 7fac7dffb640 1 -- 172.21.15.31:0/1457407263 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac9000fcb0 con 0x55930be9c6b0 2024-04-25T16:09:45.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.646+0000 7faca3972740 10 client.26939 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.646+0000 7faca3972740 1 -- 172.21.15.31:0/1457407263 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55930be950f0 con 0x55930be9c6b0 2024-04-25T16:09:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.647+0000 7fac7dffb640 1 -- 172.21.15.31:0/1457407263 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac900a6fb0 con 0x55930be9c6b0 2024-04-25T16:09:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.647+0000 7fac7dffb640 1 -- 172.21.15.31:0/1457407263 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fac900dc9e0 con 0x55930be9c6b0 2024-04-25T16:09:45.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.647+0000 7faca3972740 10 client.26939 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.647+0000 7faca3972740 10 client.26939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.647+0000 7faca3972740 20 client.26939 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.647+0000 7faca3972740 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55930be513a0 0x55930c2f1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.647+0000 7faca3972740 4 client.26939 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:09:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.647+0000 7faca3972740 1 -- 172.21.15.31:0/1457407263 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fac900a2f30 con 0x55930be513a0 2024-04-25T16:09:45.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.648+0000 7fac9659c640 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55930be513a0 0x55930c2f1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.649+0000 7fac9659c640 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55930be513a0 0x55930c2f1ce0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aad39f522bb5c01e server_cookie=82ba6892ad67837c in_seq=0 out_seq=0 2024-04-25T16:09:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.649+0000 7fac7dffb640 10 client.26939 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:45.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.651+0000 7fac7dffb640 1 -- 172.21.15.31:0/1457407263 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7fac900a2f30 con 0x55930be513a0 2024-04-25T16:09:45.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.651+0000 7fac7dffb640 10 client.26939 handle_command_reply: tid=0 2024-04-25T16:09:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.651+0000 7faca3972740 1 client.26939 shutdown 2024-04-25T16:09:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.651+0000 7faca3972740 1 -- 172.21.15.31:0/1457407263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 msgr2=0x55930c2eb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.651+0000 7faca3972740 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c2eb8f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fac900a6e20 tx=0x7fac900dc5a0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.652+0000 7faca3972740 1 -- 172.21.15.31:0/1457407263 shutdown_connections 2024-04-25T16:09:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.652+0000 7faca3972740 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55930be513a0 0x55930c2f1ce0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.652+0000 7faca3972740 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55930bebd390 0x55930c2edfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.652+0000 7faca3972740 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55930be9c6b0 0x55930c2eb8f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.652+0000 7faca3972740 1 --2- 172.21.15.31:0/1457407263 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55930be944d0 0x55930c280600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.652+0000 7faca3972740 1 -- 172.21.15.31:0/1457407263 >> 172.21.15.31:0/1457407263 conn(0x55930beb14e0 msgr2=0x55930c278460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:45.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.656+0000 7fac7dffb640 0 client.26939 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:45.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.656+0000 7faca3972740 1 -- 172.21.15.31:0/1457407263 shutdown_connections 2024-04-25T16:09:45.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.656+0000 7faca3972740 1 -- 172.21.15.31:0/1457407263 wait complete. 2024-04-25T16:09:45.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.660+0000 7faca3972740 20 client.26939 trim_cache size 0 max 16384 2024-04-25T16:09:45.669 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:09:45.669 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:09:45.669 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:09:45.669 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:09:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.676+0000 7fac9759e640 1 -- 172.21.15.31:0/1836134778 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fac800afcf0 msgr2=0x7fac800b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.676+0000 7fac9759e640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fac800afcf0 0x7fac800b2200 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fac8c009830 tx=0x7fac8c030e60 comp rx=0 tx=0).stop 2024-04-25T16:09:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.676+0000 7fac9759e640 1 -- 172.21.15.31:0/1836134778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac9810b190 msgr2=0x7fac981a9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:45.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.676+0000 7fac9759e640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac9810b190 0x7fac981a9420 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fac9000ed70 tx=0x7fac9000c6e0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.677+0000 7fac9759e640 1 -- 172.21.15.31:0/1836134778 shutdown_connections 2024-04-25T16:09:45.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.677+0000 7fac9759e640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fac800afcf0 0x7fac800b2200 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.677+0000 7fac9759e640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fac9810fe90 0x7fac981b03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.677+0000 7fac9759e640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac9810b190 0x7fac981a9420 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.677+0000 7fac9759e640 1 --2- 172.21.15.31:0/1836134778 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac9810a340 0x7fac981a8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:45.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.677+0000 7fac9759e640 1 -- 172.21.15.31:0/1836134778 >> 172.21.15.31:0/1836134778 conn(0x7fac98070320 msgr2=0x7fac9810be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:45.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.677+0000 7fac9759e640 1 -- 172.21.15.31:0/1836134778 shutdown_connections 2024-04-25T16:09:45.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:45.677+0000 7fac9759e640 1 -- 172.21.15.31:0/1836134778 wait complete. 2024-04-25T16:09:45.734 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:09:45.735 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:09:46.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.272+0000 7f2adb59e640 1 -- 172.21.15.31:0/3995747874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2adc10a340 msgr2=0x7f2adc10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:46.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.272+0000 7f2adb59e640 1 --2- 172.21.15.31:0/3995747874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2adc10a340 0x7f2adc10ac50 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009860 tx=0x7f2acc02a380 comp rx=0 tx=0).stop 2024-04-25T16:09:46.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.285+0000 7f2adb59e640 1 -- 172.21.15.31:0/3995747874 shutdown_connections 2024-04-25T16:09:46.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.285+0000 7f2adb59e640 1 --2- 172.21.15.31:0/3995747874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2adc10fe90 0x7f2adc1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:46.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.285+0000 7f2adb59e640 1 --2- 172.21.15.31:0/3995747874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2adc10b190 0x7f2adc10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:46.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.285+0000 7f2adb59e640 1 --2- 172.21.15.31:0/3995747874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2adc10a340 0x7f2adc10ac50 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:46.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.285+0000 7f2adb59e640 1 -- 172.21.15.31:0/3995747874 >> 172.21.15.31:0/3995747874 conn(0x7f2adc070320 msgr2=0x7f2adc0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:46.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.293+0000 7f2adb59e640 1 -- 172.21.15.31:0/3995747874 shutdown_connections 2024-04-25T16:09:46.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.293+0000 7f2adb59e640 1 -- 172.21.15.31:0/3995747874 wait complete. 2024-04-25T16:09:46.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.295+0000 7f2adb59e640 1 Processor -- start 2024-04-25T16:09:46.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.295+0000 7f2adb59e640 1 -- start start 2024-04-25T16:09:46.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.295+0000 7f2adb59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2adc10a340 0x7f2adc1a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:46.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.295+0000 7f2adb59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2adc10b190 0x7f2adc1a9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:46.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.295+0000 7f2adb59e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2adc10fe90 0x7f2adc1b03a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:46.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.295+0000 7f2adb59e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc11e280 con 0x7f2adc10a340 2024-04-25T16:09:46.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.296+0000 7f2adb59e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc11e100 con 0x7f2adc10fe90 2024-04-25T16:09:46.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.296+0000 7f2adb59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2adc11e400 con 0x7f2adc10b190 2024-04-25T16:09:46.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.298+0000 7f2adad9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2adc10fe90 0x7f2adc1b03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:46.299 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.298+0000 7f2adad9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2adc10fe90 0x7f2adc1b03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58146/0 (socket says 172.21.15.31:58146) 2024-04-25T16:09:46.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.298+0000 7f2adad9d640 1 -- 172.21.15.31:0/2550857977 learned_addr learned my addr 172.21.15.31:0/2550857977 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:46.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.298+0000 7f2ad9d9b640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2adc10b190 0x7f2adc1a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:46.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.298+0000 7f2adad9d640 1 -- 172.21.15.31:0/2550857977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2adc10b190 msgr2=0x7f2adc1a9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:46.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.298+0000 7f2adad9d640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2adc10b190 0x7f2adc1a9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:46.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.298+0000 7f2adad9d640 1 -- 172.21.15.31:0/2550857977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2adc10a340 msgr2=0x7f2adc1a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:46.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.298+0000 7f2adad9d640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2adc10a340 0x7f2adc1a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:46.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.298+0000 7f2adad9d640 1 -- 172.21.15.31:0/2550857977 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2adc111c90 con 0x7f2adc10fe90 2024-04-25T16:09:46.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.299+0000 7f2adad9d640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2adc10fe90 0x7f2adc1b03a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400e970 tx=0x7f2ac400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:46.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.304+0000 7f2ad37fe640 1 -- 172.21.15.31:0/2550857977 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac400ccc0 con 0x7f2adc10fe90 2024-04-25T16:09:46.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.305+0000 7f2adb59e640 1 -- 172.21.15.31:0/2550857977 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2adc111f80 con 0x7f2adc10fe90 2024-04-25T16:09:46.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.305+0000 7f2adb59e640 1 -- 172.21.15.31:0/2550857977 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2adc1124d0 con 0x7f2adc10fe90 2024-04-25T16:09:46.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.309+0000 7f2ad37fe640 1 -- 172.21.15.31:0/2550857977 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2ac400ce60 con 0x7f2adc10fe90 2024-04-25T16:09:46.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.309+0000 7f2adb59e640 1 -- 172.21.15.31:0/2550857977 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2adc10eca0 con 0x7f2adc10fe90 2024-04-25T16:09:46.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.309+0000 7f2ad37fe640 1 -- 172.21.15.31:0/2550857977 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4010430 con 0x7f2adc10fe90 2024-04-25T16:09:46.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.328+0000 7f2ad37fe640 1 -- 172.21.15.31:0/2550857977 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2ac4004fb0 con 0x7f2adc10fe90 2024-04-25T16:09:46.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.332+0000 7f2ad37fe640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2ac80afbb0 0x7f2ac80b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:46.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.332+0000 7f2ad37fe640 1 -- 172.21.15.31:0/2550857977 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f2ac40df4f0 con 0x7f2adc10fe90 2024-04-25T16:09:46.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.333+0000 7f2ad37fe640 1 -- 172.21.15.31:0/2550857977 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2ac40dfa10 con 0x7f2adc10fe90 2024-04-25T16:09:46.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.350+0000 7f2ada59c640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2ac80afbb0 0x7f2ac80b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:46.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:46.380+0000 7f2ada59c640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2ac80afbb0 0x7f2ac80b20c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009780 tx=0x7f2acc005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:47.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.042+0000 7f2adb59e640 1 -- 172.21.15.31:0/2550857977 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2adc02de00 con 0x7f2adc10fe90 2024-04-25T16:09:47.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.047+0000 7f2ad37fe640 1 -- 172.21.15.31:0/2550857977 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7f2ac40a80e0 con 0x7f2adc10fe90 2024-04-25T16:09:47.050 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:09:47.050 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:09:47.050 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.054+0000 7f2ad17fa640 1 -- 172.21.15.31:0/2550857977 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2ac80afbb0 msgr2=0x7f2ac80b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.054+0000 7f2ad17fa640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2ac80afbb0 0x7f2ac80b20c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009780 tx=0x7f2acc005e40 comp rx=0 tx=0).stop 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.054+0000 7f2ad17fa640 1 -- 172.21.15.31:0/2550857977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2adc10fe90 msgr2=0x7f2adc1b03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.054+0000 7f2ad17fa640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2adc10fe90 0x7f2adc1b03a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400e970 tx=0x7f2ac400ed70 comp rx=0 tx=0).stop 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.055+0000 7f2ad17fa640 1 -- 172.21.15.31:0/2550857977 shutdown_connections 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.055+0000 7f2ad17fa640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f2ac80afbb0 0x7f2ac80b20c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.055+0000 7f2ad17fa640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2adc10fe90 0x7f2adc1b03a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.055+0000 7f2ad17fa640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2adc10b190 0x7f2adc1a9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.055+0000 7f2ad17fa640 1 --2- 172.21.15.31:0/2550857977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2adc10a340 0x7f2adc1a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.055+0000 7f2ad17fa640 1 -- 172.21.15.31:0/2550857977 >> 172.21.15.31:0/2550857977 conn(0x7f2adc070320 msgr2=0x7f2adc10be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:47.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.059+0000 7f2ad17fa640 1 -- 172.21.15.31:0/2550857977 shutdown_connections 2024-04-25T16:09:47.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.059+0000 7f2ad17fa640 1 -- 172.21.15.31:0/2550857977 wait complete. 2024-04-25T16:09:47.095 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:09:47.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:47 smithi063 ceph-mon[32413]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 763 MiB/s wr, 2.04k op/s 2024-04-25T16:09:47.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:47 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2550857977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:09:47.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:47 smithi186 ceph-mon[33258]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 763 MiB/s wr, 2.04k op/s 2024-04-25T16:09:47.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:47 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2550857977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:09:47.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:47 smithi031 ceph-mon[29140]: pgmap v605: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 763 MiB/s wr, 2.04k op/s 2024-04-25T16:09:47.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:47 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2550857977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:09:47.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.510+0000 7f7a4220c640 1 -- 172.21.15.31:0/352325426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10b190 msgr2=0x7f7a3c10b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.510+0000 7f7a4220c640 1 --2- 172.21.15.31:0/352325426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10b190 0x7f7a3c10b5a0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7a3801a910 tx=0x7f7a3803b470 comp rx=0 tx=0).stop 2024-04-25T16:09:47.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.511+0000 7f7a4220c640 1 -- 172.21.15.31:0/352325426 shutdown_connections 2024-04-25T16:09:47.516 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.511+0000 7f7a4220c640 1 --2- 172.21.15.31:0/352325426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a3c10fe90 0x7f7a3c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.511+0000 7f7a4220c640 1 --2- 172.21.15.31:0/352325426 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10b190 0x7f7a3c10b5a0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.511+0000 7f7a4220c640 1 --2- 172.21.15.31:0/352325426 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a3c10a340 0x7f7a3c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.511+0000 7f7a4220c640 1 -- 172.21.15.31:0/352325426 >> 172.21.15.31:0/352325426 conn(0x7f7a3c070320 msgr2=0x7f7a3c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:47.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.511+0000 7f7a4220c640 1 -- 172.21.15.31:0/352325426 shutdown_connections 2024-04-25T16:09:47.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.512+0000 7f7a4220c640 1 -- 172.21.15.31:0/352325426 wait complete. 2024-04-25T16:09:47.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.513+0000 7f7a4220c640 1 Processor -- start 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.513+0000 7f7a4220c640 1 -- start start 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.514+0000 7f7a4220c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10a340 0x7f7a3c07c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.514+0000 7f7a4220c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a3c10b190 0x7f7a3c07c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.514+0000 7f7a4220c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a3c10fe90 0x7f7a3c083580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.514+0000 7f7a4220c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c113c50 con 0x7f7a3c10a340 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.514+0000 7f7a4220c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c113ad0 con 0x7f7a3c10b190 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.514+0000 7f7a4220c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c113dd0 con 0x7f7a3c10fe90 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.515+0000 7f7a4120a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10a340 0x7f7a3c07c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.515+0000 7f7a4120a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10a340 0x7f7a3c07c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60122/0 (socket says 172.21.15.31:60122) 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.515+0000 7f7a4120a640 1 -- 172.21.15.31:0/1066441261 learned_addr learned my addr 172.21.15.31:0/1066441261 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:47.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.515+0000 7f7a41a0b640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a3c10fe90 0x7f7a3c083580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.515+0000 7f7a40a09640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a3c10b190 0x7f7a3c07c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.516+0000 7f7a4120a640 1 -- 172.21.15.31:0/1066441261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a3c10fe90 msgr2=0x7f7a3c083580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.516+0000 7f7a4120a640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a3c10fe90 0x7f7a3c083580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.516+0000 7f7a4120a640 1 -- 172.21.15.31:0/1066441261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a3c10b190 msgr2=0x7f7a3c07c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.516+0000 7f7a4120a640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a3c10b190 0x7f7a3c07c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.516+0000 7f7a4120a640 1 -- 172.21.15.31:0/1066441261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3c083bf0 con 0x7f7a3c10a340 2024-04-25T16:09:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.516+0000 7f7a4120a640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10a340 0x7f7a3c07c0b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00b450 tx=0x7f7a2c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:47.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.517+0000 7f7a327fc640 1 -- 172.21.15.31:0/1066441261 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c015020 con 0x7f7a3c10a340 2024-04-25T16:09:47.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.517+0000 7f7a327fc640 1 -- 172.21.15.31:0/1066441261 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a2c011d40 con 0x7f7a3c10a340 2024-04-25T16:09:47.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.518+0000 7f7a327fc640 1 -- 172.21.15.31:0/1066441261 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c00f340 con 0x7f7a3c10a340 2024-04-25T16:09:47.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.519+0000 7f7a4220c640 1 -- 172.21.15.31:0/1066441261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a3c083ee0 con 0x7f7a3c10a340 2024-04-25T16:09:47.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.519+0000 7f7a4220c640 1 -- 172.21.15.31:0/1066441261 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3c084430 con 0x7f7a3c10a340 2024-04-25T16:09:47.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.527+0000 7f7a4a3fe740 1 Processor -- start 2024-04-25T16:09:47.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.530+0000 7f7a4a3fe740 1 -- start start 2024-04-25T16:09:47.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.530+0000 7f7a327fc640 1 -- 172.21.15.31:0/1066441261 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7a2c018020 con 0x7f7a3c10a340 2024-04-25T16:09:47.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.530+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fdd69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.530+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.530+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 0x559f9fdd7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.530+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559f9fd3ad80 con 0x559f9fa136b0 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.530+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559f9fcf4a20 con 0x559f9fa34390 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.530+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559f9fd36d40 con 0x559f9fa0b4d0 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a4120a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a4120a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47742/0 (socket says 172.21.15.31:47742) 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a4120a640 1 -- 172.21.15.31:0/1084420506 learned_addr learned my addr 172.21.15.31:0/1084420506 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a4120a640 1 -- 172.21.15.31:0/1084420506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 msgr2=0x559f9fdd69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a4120a640 1 --2- 172.21.15.31:0/1084420506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fdd69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a4120a640 1 -- 172.21.15.31:0/1084420506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 msgr2=0x559f9fdd7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:47.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a4120a640 1 --2- 172.21.15.31:0/1084420506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 0x559f9fdd7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a4120a640 1 -- 172.21.15.31:0/1084420506 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f9fcffc30 con 0x559f9fa0b4d0 2024-04-25T16:09:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a327fc640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7a280afbd0 0x7f7a280b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.531+0000 7f7a4120a640 1 --2- 172.21.15.31:0/1084420506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd6f60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c108590 tx=0x7f7a2c108a50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9df71ef91e8502c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.532+0000 7f7a277fe640 1 -- 172.21.15.31:0/1084420506 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0a27c0 con 0x559f9fa0b4d0 2024-04-25T16:09:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.532+0000 7f7a327fc640 1 -- 172.21.15.31:0/1066441261 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7a2c0de470 con 0x7f7a3c10a340 2024-04-25T16:09:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.534+0000 7f7a41a0b640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7a280afbd0 0x7f7a280b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.537+0000 7f7a277fe640 1 -- 172.21.15.31:0/1084420506 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a2c0a6410 con 0x559f9fa0b4d0 2024-04-25T16:09:47.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.539+0000 7f7a277fe640 1 -- 172.21.15.31:0/1084420506 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0a6070 con 0x559f9fa0b4d0 2024-04-25T16:09:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.539+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/1084420506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 msgr2=0x559f9fdd6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.539+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/1084420506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd6f60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c108590 tx=0x7f7a2c108a50 comp rx=0 tx=0).stop 2024-04-25T16:09:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.541+0000 7f7a41a0b640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7a280afbd0 0x7f7a280b20e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x559f9fdd7a30 tx=0x7f7a3400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.541+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/1084420506 shutdown_connections 2024-04-25T16:09:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.541+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/1084420506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 0x559f9fdd7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.541+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/1084420506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd6f60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.541+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/1084420506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fdd69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.541+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/1084420506 >> 172.21.15.31:0/1084420506 conn(0x559f9fa284e0 msgr2=0x559f9fdd21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:47.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.542+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/1084420506 shutdown_connections 2024-04-25T16:09:47.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.542+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/1084420506 wait complete. 2024-04-25T16:09:47.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.543+0000 7f7a4a3fe740 1 -- start start 2024-04-25T16:09:47.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.543+0000 7f7a4a3fe740 1 Processor -- start 2024-04-25T16:09:47.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fde0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa136b0 0x559f9fde0c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa34390 0x559f9fde7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559f9fdd8750 con 0x559f9fa34390 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559f9fcc1dc0 con 0x559f9fa136b0 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559f9fdd88d0 con 0x559f9fa0b4d0 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a4a3fe740 10 client.0 _ll_register_callbacks cb 0x559f9fcdd600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a41a0b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa34390 0x559f9fde7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a41a0b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa34390 0x559f9fde7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60132/0 (socket says 172.21.15.31:60132) 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a41a0b640 1 -- 172.21.15.31:0/3359126128 learned_addr learned my addr 172.21.15.31:0/3359126128 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.544+0000 7f7a40a09640 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa136b0 0x559f9fde0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.545+0000 7f7a4120a640 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fde0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.545+0000 7f7a4120a640 1 -- 172.21.15.31:0/3359126128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa136b0 msgr2=0x559f9fde0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.545+0000 7f7a4120a640 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa136b0 0x559f9fde0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.545+0000 7f7a4120a640 1 -- 172.21.15.31:0/3359126128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa34390 msgr2=0x559f9fde7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:47.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.545+0000 7f7a4120a640 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa34390 0x559f9fde7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.545+0000 7f7a4120a640 1 -- 172.21.15.31:0/3359126128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f9fa4c3f0 con 0x559f9fa0b4d0 2024-04-25T16:09:47.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.545+0000 7f7a4120a640 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fde0720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c108590 tx=0x7f7a2c0a2e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c0febe4e2510fb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:47.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.546+0000 7f7a24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:47.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.546+0000 7f7a24ff9640 1 -- 172.21.15.31:0/3359126128 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c0a6250 con 0x559f9fa0b4d0 2024-04-25T16:09:47.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.546+0000 7f7a24ff9640 1 -- 172.21.15.31:0/3359126128 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7a2c06cdf0 con 0x559f9fa0b4d0 2024-04-25T16:09:47.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.546+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/3359126128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a2c0a6410 con 0x559f9fa0b4d0 2024-04-25T16:09:47.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.549+0000 7f7a24ff9640 1 -- 172.21.15.31:0/3359126128 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7a2c075050 con 0x559f9fa0b4d0 2024-04-25T16:09:47.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.549+0000 7f7a4a3fe740 10 client.26929 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:47.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.549+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/3359126128 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559f9fb167d0 con 0x559f9fa0b4d0 2024-04-25T16:09:47.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.552+0000 7f7a24ff9640 1 -- 172.21.15.31:0/3359126128 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a2c0a6410 con 0x559f9fa0b4d0 2024-04-25T16:09:47.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.554+0000 7f7a24ff9640 1 -- 172.21.15.31:0/3359126128 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7a2c022060 con 0x559f9fa0b4d0 2024-04-25T16:09:47.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.555+0000 7f7a4a3fe740 10 client.26929 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:47.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.555+0000 7f7a4a3fe740 10 client.26929 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:09:47.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.555+0000 7f7a4a3fe740 20 client.26929 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:47.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.555+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x559f9f9f2f60 0x559f9fdee350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.555+0000 7f7a4a3fe740 4 client.26929 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:47.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.555+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/3359126128 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559f9fd48990 con 0x559f9f9f2f60 2024-04-25T16:09:47.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.558+0000 7f7a40a09640 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x559f9f9f2f60 0x559f9fdee350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.571+0000 7f7a40a09640 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x559f9f9f2f60 0x559f9fdee350 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=223f30443bf76615 server_cookie=21f994ba8941138f in_seq=0 out_seq=0 2024-04-25T16:09:47.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.571+0000 7f7a24ff9640 10 client.26929 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:09:47.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.573+0000 7f7a24ff9640 1 -- 172.21.15.31:0/3359126128 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559f9fd48990 con 0x559f9f9f2f60 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.573+0000 7f7a24ff9640 10 client.26929 handle_command_reply: tid=0 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.574+0000 7f7a4a3fe740 1 client.26929 shutdown 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.575+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/3359126128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 msgr2=0x559f9fde0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.575+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fde0720 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c108590 tx=0x7f7a2c0a2e30 comp rx=0 tx=0).stop 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.576+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/3359126128 shutdown_connections 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.576+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x559f9f9f2f60 0x559f9fdee350 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.576+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa34390 0x559f9fde7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.576+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa136b0 0x559f9fde0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.576+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/3359126128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fde0720 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.576+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/3359126128 >> 172.21.15.31:0/3359126128 conn(0x559f9fa284e0 msgr2=0x559f9fdd5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:47.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.577+0000 7f7a24ff9640 0 client.26929 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:09:47.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.577+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/3359126128 shutdown_connections 2024-04-25T16:09:47.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.583+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/3359126128 wait complete. 2024-04-25T16:09:47.585 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.583+0000 7f7a4a3fe740 20 client.26929 trim_cache size 0 max 16384 2024-04-25T16:09:47.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.633+0000 7f7a4a3fe740 1 Processor -- start 2024-04-25T16:09:47.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.633+0000 7f7a4a3fe740 1 -- start start 2024-04-25T16:09:47.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.633+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd5700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.633+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa136b0 0x559f9fdd5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.633+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fdf2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.633+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c1c2370 con 0x559f9fa0b4d0 2024-04-25T16:09:47.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.633+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c1c3eb0 con 0x559f9fa34390 2024-04-25T16:09:47.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.633+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c1c2a40 con 0x559f9fa136b0 2024-04-25T16:09:47.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.636+0000 7f7a41a0b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fdf2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.636+0000 7f7a40a09640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa136b0 0x559f9fdd5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.636+0000 7f7a40a09640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa136b0 0x559f9fdd5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47748/0 (socket says 172.21.15.31:47748) 2024-04-25T16:09:47.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.636+0000 7f7a40a09640 1 -- 172.21.15.31:0/2092461812 learned_addr learned my addr 172.21.15.31:0/2092461812 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:47.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.639+0000 7f7a4120a640 1 --2- 172.21.15.31:0/2092461812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.640+0000 7f7a40a09640 1 -- 172.21.15.31:0/2092461812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 msgr2=0x559f9fdf2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:47.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.640+0000 7f7a40a09640 1 --2- 172.21.15.31:0/2092461812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fdf2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.640+0000 7f7a40a09640 1 -- 172.21.15.31:0/2092461812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa0b4d0 msgr2=0x559f9fdd5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:47.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.640+0000 7f7a40a09640 1 --2- 172.21.15.31:0/2092461812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.640+0000 7f7a40a09640 1 -- 172.21.15.31:0/2092461812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559f9fddbdb0 con 0x559f9fa136b0 2024-04-25T16:09:47.648 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.646+0000 7f7a40a09640 1 --2- 172.21.15.31:0/2092461812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa136b0 0x559f9fdd5c40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c07d150 tx=0x7f7a38004b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=265d0d9c34dd6dbe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:47.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.651+0000 7f7a26ffd640 1 -- 172.21.15.31:0/2092461812 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38014020 con 0x559f9fa136b0 2024-04-25T16:09:47.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.651+0000 7f7a26ffd640 1 -- 172.21.15.31:0/2092461812 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a38004e20 con 0x559f9fa136b0 2024-04-25T16:09:47.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.651+0000 7f7a26ffd640 1 -- 172.21.15.31:0/2092461812 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38046040 con 0x559f9fa136b0 2024-04-25T16:09:47.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.653+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/2092461812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa136b0 msgr2=0x559f9fdd5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.653+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/2092461812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa136b0 0x559f9fdd5c40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c07d150 tx=0x7f7a38004b90 comp rx=0 tx=0).stop 2024-04-25T16:09:47.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.653+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/2092461812 shutdown_connections 2024-04-25T16:09:47.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.653+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/2092461812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fdf2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.653+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/2092461812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa136b0 0x559f9fdd5c40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.653+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/2092461812 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa0b4d0 0x559f9fdd5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.653+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/2092461812 >> 172.21.15.31:0/2092461812 conn(0x559f9fa284e0 msgr2=0x559f9fdd4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:47.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.657+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/2092461812 shutdown_connections 2024-04-25T16:09:47.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.661+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/2092461812 wait complete. 2024-04-25T16:09:47.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.661+0000 7f7a4a3fe740 1 -- start start 2024-04-25T16:09:47.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.662+0000 7f7a4a3fe740 1 Processor -- start 2024-04-25T16:09:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.664+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fde6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.664+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 0x559f9fde3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.664+0000 7f7a4a3fe740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fde40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.665+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c1c37e0 con 0x559f9fa136b0 2024-04-25T16:09:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.665+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c1c3110 con 0x559f9fa34390 2024-04-25T16:09:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.665+0000 7f7a4a3fe740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c1c0400 con 0x559f9fa0b4d0 2024-04-25T16:09:47.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.665+0000 7f7a4a3fe740 10 client.0 _ll_register_callbacks cb 0x7f7a2c108410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:47.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.665+0000 7f7a40a09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 0x559f9fde3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.665+0000 7f7a40a09640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 0x559f9fde3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60152/0 (socket says 172.21.15.31:60152) 2024-04-25T16:09:47.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.665+0000 7f7a40a09640 1 -- 172.21.15.31:0/222116629 learned_addr learned my addr 172.21.15.31:0/222116629 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:47.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.665+0000 7f7a41a0b640 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fde40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.666+0000 7f7a41a0b640 1 -- 172.21.15.31:0/222116629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 msgr2=0x559f9fde6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:47.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.666+0000 7f7a41a0b640 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fde6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.666+0000 7f7a41a0b640 1 -- 172.21.15.31:0/222116629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 msgr2=0x559f9fde3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.666+0000 7f7a41a0b640 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 0x559f9fde3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.666+0000 7f7a41a0b640 1 -- 172.21.15.31:0/222116629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559f9fde46b0 con 0x559f9fa34390 2024-04-25T16:09:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.667+0000 7f7a41a0b640 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fde40c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400bcb0 tx=0x7f7a34009630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5f698dc09842f8e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.667+0000 7f7a24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.668+0000 7f7a24ff9640 1 -- 172.21.15.31:0/222116629 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34023660 con 0x559f9fa34390 2024-04-25T16:09:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.668+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/222116629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a38004e20 con 0x559f9fa34390 2024-04-25T16:09:47.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.668+0000 7f7a24ff9640 1 -- 172.21.15.31:0/222116629 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f7a34023800 con 0x559f9fa34390 2024-04-25T16:09:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.670+0000 7f7a24ff9640 1 -- 172.21.15.31:0/222116629 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f7a34023a00 con 0x559f9fa34390 2024-04-25T16:09:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.670+0000 7f7a4a3fe740 10 client.26954 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:47.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.670+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/222116629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559f9fe557d0 con 0x559f9fa34390 2024-04-25T16:09:47.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.675+0000 7f7a24ff9640 1 -- 172.21.15.31:0/222116629 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a38004e20 con 0x559f9fa34390 2024-04-25T16:09:47.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.675+0000 7f7a24ff9640 1 -- 172.21.15.31:0/222116629 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f7a3402cc60 con 0x559f9fa34390 2024-04-25T16:09:47.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.675+0000 7f7a4a3fe740 10 client.26954 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:47.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.675+0000 7f7a4a3fe740 10 client.26954 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:09:47.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.675+0000 7f7a4a3fe740 20 client.26954 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:47.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.675+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x559f9f9c83a0 0x559f9fe58220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:47.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.675+0000 7f7a4a3fe740 4 client.26954 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:47.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.675+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/222116629 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559f9fd42800 con 0x559f9f9c83a0 2024-04-25T16:09:47.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.676+0000 7f7a4120a640 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x559f9f9c83a0 0x559f9fe58220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:47.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.679+0000 7f7a4120a640 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x559f9f9c83a0 0x559f9fe58220 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ceb2a91fac33661d server_cookie=78317d2c915db7e5 in_seq=0 out_seq=0 2024-04-25T16:09:47.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.679+0000 7f7a24ff9640 10 client.26954 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:09:47.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.680+0000 7f7a24ff9640 1 -- 172.21.15.31:0/222116629 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559f9fd42800 con 0x559f9f9c83a0 2024-04-25T16:09:47.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.680+0000 7f7a24ff9640 10 client.26954 handle_command_reply: tid=0 2024-04-25T16:09:47.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.680+0000 7f7a4a3fe740 1 client.26954 shutdown 2024-04-25T16:09:47.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.684+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/222116629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 msgr2=0x559f9fde40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.684+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fde40c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7a3400bcb0 tx=0x7f7a34009630 comp rx=0 tx=0).stop 2024-04-25T16:09:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.687+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/222116629 shutdown_connections 2024-04-25T16:09:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.687+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x559f9f9c83a0 0x559f9fe58220 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.687+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559f9fa34390 0x559f9fde40c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.687+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559f9fa136b0 0x559f9fde3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.687+0000 7f7a4a3fe740 1 --2- 172.21.15.31:0/222116629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559f9fa0b4d0 0x559f9fde6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.687+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/222116629 >> 172.21.15.31:0/222116629 conn(0x559f9fa284e0 msgr2=0x559f9fdd5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:47.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.688+0000 7f7a24ff9640 0 client.26954 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:09:47.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.690+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/222116629 shutdown_connections 2024-04-25T16:09:47.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.690+0000 7f7a4a3fe740 1 -- 172.21.15.31:0/222116629 wait complete. 2024-04-25T16:09:47.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.693+0000 7f7a4a3fe740 20 client.26954 trim_cache size 0 max 16384 2024-04-25T16:09:47.699 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 -- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7a280afbd0 msgr2=0x7f7a280b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7a280afbd0 0x7f7a280b20e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x559f9fdd7a30 tx=0x7f7a3400d040 comp rx=0 tx=0).stop 2024-04-25T16:09:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 -- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10a340 msgr2=0x7f7a3c07c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:47.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10a340 0x7f7a3c07c0b0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00b450 tx=0x7f7a2c00b910 comp rx=0 tx=0).stop 2024-04-25T16:09:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 -- 172.21.15.31:0/1066441261 shutdown_connections 2024-04-25T16:09:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f7a280afbd0 0x7f7a280b20e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a3c10fe90 0x7f7a3c083580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a3c10b190 0x7f7a3c07c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 --2- 172.21.15.31:0/1066441261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7a3c10a340 0x7f7a3c07c0b0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.700+0000 7f7a4220c640 1 -- 172.21.15.31:0/1066441261 >> 172.21.15.31:0/1066441261 conn(0x7f7a3c070320 msgr2=0x7f7a3c072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.701+0000 7f7a4220c640 1 -- 172.21.15.31:0/1066441261 shutdown_connections 2024-04-25T16:09:47.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:47.701+0000 7f7a4220c640 1 -- 172.21.15.31:0/1066441261 wait complete. 2024-04-25T16:09:47.740 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:09:48.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.175+0000 7f51c6ffd640 1 -- 172.21.15.31:0/1272893179 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c002f340 con 0x7f51c810b310 2024-04-25T16:09:48.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.176+0000 7f51ce5e3640 1 -- 172.21.15.31:0/1272893179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c810b310 msgr2=0x7f51c810b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.176+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/1272893179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c810b310 0x7f51c810b720 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009ca0 tx=0x7f51c002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.176+0000 7f51ce5e3640 1 -- 172.21.15.31:0/1272893179 shutdown_connections 2024-04-25T16:09:48.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.176+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/1272893179 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c8110030 0x7f51c8110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.176+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/1272893179 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c810b310 0x7f51c810b720 secure :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009ca0 tx=0x7f51c002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.176+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/1272893179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c810a420 0x7f51c810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.176+0000 7f51ce5e3640 1 -- 172.21.15.31:0/1272893179 >> 172.21.15.31:0/1272893179 conn(0x7f51c80704e0 msgr2=0x7f51c8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:48.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.178+0000 7f51ce5e3640 1 -- 172.21.15.31:0/1272893179 shutdown_connections 2024-04-25T16:09:48.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.178+0000 7f51ce5e3640 1 -- 172.21.15.31:0/1272893179 wait complete. 2024-04-25T16:09:48.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.179+0000 7f51ce5e3640 1 Processor -- start 2024-04-25T16:09:48.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.180+0000 7f51ce5e3640 1 -- start start 2024-04-25T16:09:48.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.180+0000 7f51ce5e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c810a420 0x7f51c81a92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.180+0000 7f51ce5e3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c8110030 0x7f51c81a9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.180+0000 7f51ce5e3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c81ae790 0x7f51c81b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.180+0000 7f51ce5e3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f51c811e5e0 con 0x7f51c810a420 2024-04-25T16:09:48.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.180+0000 7f51ce5e3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f51c811e460 con 0x7f51c81ae790 2024-04-25T16:09:48.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.180+0000 7f51ce5e3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51c811e760 con 0x7f51c8110030 2024-04-25T16:09:48.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.181+0000 7f51c7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c810a420 0x7f51c81a92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.181+0000 7f51ccb80640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c81ae790 0x7f51c81b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.181+0000 7f51ccb80640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c81ae790 0x7f51c81b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58220/0 (socket says 172.21.15.31:58220) 2024-04-25T16:09:48.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.181+0000 7f51ccb80640 1 -- 172.21.15.31:0/2896779033 learned_addr learned my addr 172.21.15.31:0/2896779033 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:48.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.183+0000 7f51c77fe640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c8110030 0x7f51c81a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.183+0000 7f51ccb80640 1 -- 172.21.15.31:0/2896779033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c8110030 msgr2=0x7f51c81a9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.183+0000 7f51ccb80640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c8110030 0x7f51c81a9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.183+0000 7f51ccb80640 1 -- 172.21.15.31:0/2896779033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c810a420 msgr2=0x7f51c81a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.183+0000 7f51ccb80640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c810a420 0x7f51c81a92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.183+0000 7f51ccb80640 1 -- 172.21.15.31:0/2896779033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51c8111c90 con 0x7f51c81ae790 2024-04-25T16:09:48.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.187+0000 7f51ccb80640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c81ae790 0x7f51c81b0bb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f51b800b3f0 tx=0x7f51b800b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.188+0000 7f51c57fa640 1 -- 172.21.15.31:0/2896779033 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8015020 con 0x7f51c81ae790 2024-04-25T16:09:48.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.188+0000 7f51c57fa640 1 -- 172.21.15.31:0/2896779033 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51b8011ce0 con 0x7f51c81ae790 2024-04-25T16:09:48.192 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.188+0000 7f51c57fa640 1 -- 172.21.15.31:0/2896779033 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8010e10 con 0x7f51c81ae790 2024-04-25T16:09:48.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.188+0000 7f51ce5e3640 1 -- 172.21.15.31:0/2896779033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51c8111f80 con 0x7f51c81ae790 2024-04-25T16:09:48.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.189+0000 7f51ce5e3640 1 -- 172.21.15.31:0/2896779033 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51c8112500 con 0x7f51c81ae790 2024-04-25T16:09:48.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.192+0000 7f51d5569740 1 Processor -- start 2024-04-25T16:09:48.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.196+0000 7f51d5569740 1 -- start start 2024-04-25T16:09:48.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.196+0000 7f51c57fa640 1 -- 172.21.15.31:0/2896779033 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f51b8018020 con 0x7f51c81ae790 2024-04-25T16:09:48.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.197+0000 7f51d5569740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63e8810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.197+0000 7f51d5569740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63e8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.197+0000 7f51d5569740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63e9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.197+0000 7f51d5569740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559fb634cd80 con 0x559fb60256b0 2024-04-25T16:09:48.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.197+0000 7f51d5569740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559fb6306a20 con 0x559fb6046390 2024-04-25T16:09:48.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.197+0000 7f51d5569740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559fb6348d40 con 0x559fb601d4d0 2024-04-25T16:09:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.198+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63e8810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.198+0000 7f51c7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63e8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.198+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63e9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.198+0000 7f51c7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63e8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47782/0 (socket says 172.21.15.31:47782) 2024-04-25T16:09:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.198+0000 7f51c7fff640 1 -- 172.21.15.31:0/2954703887 learned_addr learned my addr 172.21.15.31:0/2954703887 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.199+0000 7f51c57fa640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f51b00afc30 0x7f51b00b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.199+0000 7f51c57fa640 1 -- 172.21.15.31:0/2896779033 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f51b80de510 con 0x7f51c81ae790 2024-04-25T16:09:48.203 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.200+0000 7f51ccb80640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f51b00afc30 0x7f51b00b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.201+0000 7f51c77fe640 1 -- 172.21.15.31:0/2954703887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 msgr2=0x559fb63e8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:48.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.201+0000 7f51c77fe640 1 --2- 172.21.15.31:0/2954703887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63e8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.201+0000 7f51c77fe640 1 -- 172.21.15.31:0/2954703887 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 msgr2=0x559fb63e8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:48.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.201+0000 7f51c77fe640 1 --2- 172.21.15.31:0/2954703887 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63e8810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.201+0000 7f51c77fe640 1 -- 172.21.15.31:0/2954703887 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fb6311c30 con 0x559fb60256b0 2024-04-25T16:09:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.201+0000 7f51ccb80640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f51b00afc30 0x7f51b00b2140 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f51b80a2a20 tx=0x7f51b8010870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.201+0000 7f51c77fe640 1 --2- 172.21.15.31:0/2954703887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63e9a20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f51c0008000 tx=0x7f51c0031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f5c2cd3bb97e04a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.203+0000 7f51ae7fc640 1 -- 172.21.15.31:0/2954703887 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0031660 con 0x559fb60256b0 2024-04-25T16:09:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.203+0000 7f51ae7fc640 1 -- 172.21.15.31:0/2954703887 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51c0036040 con 0x559fb60256b0 2024-04-25T16:09:48.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.203+0000 7f51ae7fc640 1 -- 172.21.15.31:0/2954703887 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c00046d0 con 0x559fb60256b0 2024-04-25T16:09:48.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.204+0000 7f51d5569740 1 -- 172.21.15.31:0/2954703887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 msgr2=0x559fb63e9a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.204+0000 7f51d5569740 1 --2- 172.21.15.31:0/2954703887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63e9a20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f51c0008000 tx=0x7f51c0031a50 comp rx=0 tx=0).stop 2024-04-25T16:09:48.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.204+0000 7f51d5569740 1 -- 172.21.15.31:0/2954703887 shutdown_connections 2024-04-25T16:09:48.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.204+0000 7f51d5569740 1 --2- 172.21.15.31:0/2954703887 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63e9a20 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.204+0000 7f51d5569740 1 --2- 172.21.15.31:0/2954703887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63e8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.204+0000 7f51d5569740 1 --2- 172.21.15.31:0/2954703887 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63e8810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.204+0000 7f51d5569740 1 -- 172.21.15.31:0/2954703887 >> 172.21.15.31:0/2954703887 conn(0x559fb603a4e0 msgr2=0x559fb63e4020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:48.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.205+0000 7f51d5569740 1 -- 172.21.15.31:0/2954703887 shutdown_connections 2024-04-25T16:09:48.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.205+0000 7f51d5569740 1 -- 172.21.15.31:0/2954703887 wait complete. 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.206+0000 7f51d5569740 1 -- start start 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.206+0000 7f51d5569740 1 Processor -- start 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.207+0000 7f51d5569740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63f25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.207+0000 7f51d5569740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.207+0000 7f51d5569740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.207+0000 7f51d5569740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559fb63ea4d0 con 0x559fb60256b0 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.207+0000 7f51d5569740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x559fb62d3dc0 con 0x559fb6046390 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.207+0000 7f51d5569740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559fb63ea650 con 0x559fb601d4d0 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.207+0000 7f51d5569740 10 client.0 _ll_register_callbacks cb 0x559fb62ef600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.208+0000 7f51c7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.208+0000 7f51c7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60186/0 (socket says 172.21.15.31:60186) 2024-04-25T16:09:48.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.208+0000 7f51c7fff640 1 -- 172.21.15.31:0/2611111432 learned_addr learned my addr 172.21.15.31:0/2611111432 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:48.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.209+0000 7f51c7fff640 1 -- 172.21.15.31:0/2611111432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 msgr2=0x559fb63f25c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:48.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.209+0000 7f51c7fff640 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63f25c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.209+0000 7f51c7fff640 1 -- 172.21.15.31:0/2611111432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 msgr2=0x559fb63f9a60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:48.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.209+0000 7f51c7fff640 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f9a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.209+0000 7f51c7fff640 1 -- 172.21.15.31:0/2611111432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fb605e3f0 con 0x559fb60256b0 2024-04-25T16:09:48.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.209+0000 7f51c7fff640 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f2b00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00ae30 tx=0x7f51bc00f8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed28428527b10ed8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.217+0000 7f51a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:48.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.217+0000 7f51a77fe640 1 -- 172.21.15.31:0/2611111432 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51bc035860 con 0x559fb60256b0 2024-04-25T16:09:48.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.217+0000 7f51a77fe640 1 -- 172.21.15.31:0/2611111432 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f51bc035a00 con 0x559fb60256b0 2024-04-25T16:09:48.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.217+0000 7f51d5569740 1 -- 172.21.15.31:0/2611111432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c0036040 con 0x559fb60256b0 2024-04-25T16:09:48.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.220+0000 7f51a77fe640 1 -- 172.21.15.31:0/2611111432 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51bc035c00 con 0x559fb60256b0 2024-04-25T16:09:48.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.221+0000 7f51d5569740 10 client.17280 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:48.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.221+0000 7f51d5569740 1 -- 172.21.15.31:0/2611111432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559fb61287d0 con 0x559fb60256b0 2024-04-25T16:09:48.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.222+0000 7f51a77fe640 1 -- 172.21.15.31:0/2611111432 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51c0036040 con 0x559fb60256b0 2024-04-25T16:09:48.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.223+0000 7f51a77fe640 1 -- 172.21.15.31:0/2611111432 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f51bc0333f0 con 0x559fb60256b0 2024-04-25T16:09:48.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.224+0000 7f51d5569740 10 client.17280 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:48.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.225+0000 7f51d5569740 10 client.17280 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:09:48.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.225+0000 7f51d5569740 20 client.17280 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:48.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.225+0000 7f51d5569740 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x559fb6004f60 0x559fb6400390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.225+0000 7f51d5569740 4 client.17280 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:48.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.225+0000 7f51d5569740 1 -- 172.21.15.31:0/2611111432 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fb635a990 con 0x559fb6004f60 2024-04-25T16:09:48.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.231+0000 7f51c77fe640 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x559fb6004f60 0x559fb6400390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.250+0000 7f51a77fe640 10 client.17280 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:09:48.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.250+0000 7f51c77fe640 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x559fb6004f60 0x559fb6400390 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d9b91b412877d541 server_cookie=2cf65a9f8f968206 in_seq=0 out_seq=0 2024-04-25T16:09:48.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.252+0000 7f51a77fe640 1 -- 172.21.15.31:0/2611111432 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559fb635a990 con 0x559fb6004f60 2024-04-25T16:09:48.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.252+0000 7f51a77fe640 10 client.17280 handle_command_reply: tid=0 2024-04-25T16:09:48.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.257+0000 7f51d5569740 1 client.17280 shutdown 2024-04-25T16:09:48.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.260+0000 7f51d5569740 1 -- 172.21.15.31:0/2611111432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 msgr2=0x559fb63f2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.260+0000 7f51d5569740 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f2b00 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f51bc00ae30 tx=0x7f51bc00f8c0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.265+0000 7f51d5569740 1 -- 172.21.15.31:0/2611111432 shutdown_connections 2024-04-25T16:09:48.267 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.265+0000 7f51d5569740 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.265+0000 7f51d5569740 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f2b00 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.265+0000 7f51d5569740 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x559fb6004f60 0x559fb6400390 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.265+0000 7f51d5569740 1 --2- 172.21.15.31:0/2611111432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63f25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.265+0000 7f51a77fe640 0 client.17280 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:09:48.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.265+0000 7f51d5569740 1 -- 172.21.15.31:0/2611111432 >> 172.21.15.31:0/2611111432 conn(0x559fb603a4e0 msgr2=0x559fb63e3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:48.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.269+0000 7f51d5569740 1 -- 172.21.15.31:0/2611111432 shutdown_connections 2024-04-25T16:09:48.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.270+0000 7f51d5569740 1 -- 172.21.15.31:0/2611111432 wait complete. 2024-04-25T16:09:48.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.271+0000 7f51d5569740 20 client.17280 trim_cache size 0 max 16384 2024-04-25T16:09:48.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.312+0000 7f51d5569740 1 Processor -- start 2024-04-25T16:09:48.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.312+0000 7f51d5569740 1 -- start start 2024-04-25T16:09:48.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.313+0000 7f51d5569740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.314+0000 7f51d5569740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.314+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.314+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63f0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47802/0 (socket says 172.21.15.31:47802) 2024-04-25T16:09:48.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.314+0000 7f51c77fe640 1 -- 172.21.15.31:0/1136909547 learned_addr learned my addr 172.21.15.31:0/1136909547 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:48.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.314+0000 7f51d5569740 1 --2- 172.21.15.31:0/1136909547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.315+0000 7f51d5569740 1 -- 172.21.15.31:0/1136909547 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f51c8076030 con 0x559fb60256b0 2024-04-25T16:09:48.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.315+0000 7f51d5569740 1 -- 172.21.15.31:0/1136909547 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f51c8077b70 con 0x559fb6046390 2024-04-25T16:09:48.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.315+0000 7f51d5569740 1 -- 172.21.15.31:0/1136909547 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51c8076700 con 0x559fb601d4d0 2024-04-25T16:09:48.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.315+0000 7f51c7fff640 1 --2- 172.21.15.31:0/1136909547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.316+0000 7f51c77fe640 1 -- 172.21.15.31:0/1136909547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 msgr2=0x559fb63f11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:48.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.316+0000 7f51c77fe640 1 --2- 172.21.15.31:0/1136909547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.316+0000 7f51c77fe640 1 -- 172.21.15.31:0/1136909547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 msgr2=0x559fb63f9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:48.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.316+0000 7f51c77fe640 1 --2- 172.21.15.31:0/1136909547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.316+0000 7f51c77fe640 1 -- 172.21.15.31:0/1136909547 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fb63f0a50 con 0x559fb601d4d0 2024-04-25T16:09:48.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.321+0000 7f51c77fe640 1 --2- 172.21.15.31:0/1136909547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63f0cb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009bf0 tx=0x7f51c002a8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b07cfeebfa973671 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.326+0000 7f51adffb640 1 -- 172.21.15.31:0/1136909547 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0057070 con 0x559fb601d4d0 2024-04-25T16:09:48.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.326+0000 7f51adffb640 1 -- 172.21.15.31:0/1136909547 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51c0036070 con 0x559fb601d4d0 2024-04-25T16:09:48.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.326+0000 7f51adffb640 1 -- 172.21.15.31:0/1136909547 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c00505b0 con 0x559fb601d4d0 2024-04-25T16:09:48.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.328+0000 7f51d5569740 1 -- 172.21.15.31:0/1136909547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 msgr2=0x559fb63f0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.328+0000 7f51d5569740 1 --2- 172.21.15.31:0/1136909547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63f0cb0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009bf0 tx=0x7f51c002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.328+0000 7f51d5569740 1 -- 172.21.15.31:0/1136909547 shutdown_connections 2024-04-25T16:09:48.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.329+0000 7f51d5569740 1 --2- 172.21.15.31:0/1136909547 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.329+0000 7f51d5569740 1 --2- 172.21.15.31:0/1136909547 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb60256b0 0x559fb63f9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.329+0000 7f51d5569740 1 --2- 172.21.15.31:0/1136909547 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb601d4d0 0x559fb63f0cb0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.329+0000 7f51d5569740 1 -- 172.21.15.31:0/1136909547 >> 172.21.15.31:0/1136909547 conn(0x559fb603a4e0 msgr2=0x559fb63ed310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:48.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.331+0000 7f51d5569740 1 -- 172.21.15.31:0/1136909547 shutdown_connections 2024-04-25T16:09:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.331+0000 7f51d5569740 1 -- 172.21.15.31:0/1136909547 wait complete. 2024-04-25T16:09:48.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.333+0000 7f51d5569740 1 -- start start 2024-04-25T16:09:48.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.334+0000 7f51d5569740 1 Processor -- start 2024-04-25T16:09:48.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.335+0000 7f51d5569740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb601d4d0 0x559fb63f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.335+0000 7f51d5569740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb60256b0 0x559fb63f7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.336+0000 7f51d5569740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.336+0000 7f51d5569740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f51c80774a0 con 0x559fb601d4d0 2024-04-25T16:09:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.336+0000 7f51d5569740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f51c8076dd0 con 0x559fb6046390 2024-04-25T16:09:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.336+0000 7f51d5569740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f51c8119220 con 0x559fb60256b0 2024-04-25T16:09:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.336+0000 7f51d5569740 10 client.0 _ll_register_callbacks cb 0x7f51bc00a3f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.337+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb601d4d0 0x559fb63f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.337+0000 7f51c77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb601d4d0 0x559fb63f6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60208/0 (socket says 172.21.15.31:60208) 2024-04-25T16:09:48.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.337+0000 7f51c77fe640 1 -- 172.21.15.31:0/597659773 learned_addr learned my addr 172.21.15.31:0/597659773 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:48.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.337+0000 7f51c77fe640 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.338+0000 7f51c7fff640 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb60256b0 0x559fb63f7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.338+0000 7f51c77fe640 1 -- 172.21.15.31:0/597659773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb60256b0 msgr2=0x559fb63f7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:48.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.338+0000 7f51c77fe640 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb60256b0 0x559fb63f7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.338+0000 7f51c77fe640 1 -- 172.21.15.31:0/597659773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 msgr2=0x559fb63f8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:48.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.338+0000 7f51c77fe640 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.338+0000 7f51c77fe640 1 -- 172.21.15.31:0/597659773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fb63f9ec0 con 0x559fb601d4d0 2024-04-25T16:09:48.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.339+0000 7f51c77fe640 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb601d4d0 0x559fb63f6d40 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f51c00097b0 tx=0x7f51c0000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30456c3f7aa2579 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.342+0000 7f51a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.342+0000 7f51a77fe640 1 -- 172.21.15.31:0/597659773 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c0057040 con 0x559fb601d4d0 2024-04-25T16:09:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.342+0000 7f51a77fe640 1 -- 172.21.15.31:0/597659773 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f51c00507a0 con 0x559fb601d4d0 2024-04-25T16:09:48.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.342+0000 7f51d5569740 1 -- 172.21.15.31:0/597659773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51c0036070 con 0x559fb601d4d0 2024-04-25T16:09:48.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.343+0000 7f51a77fe640 1 -- 172.21.15.31:0/597659773 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f51c0065050 con 0x559fb601d4d0 2024-04-25T16:09:48.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.343+0000 7f51d5569740 10 client.17286 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:48.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.343+0000 7f51d5569740 1 -- 172.21.15.31:0/597659773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559fb63f97d0 con 0x559fb601d4d0 2024-04-25T16:09:48.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.345+0000 7f51a77fe640 1 -- 172.21.15.31:0/597659773 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51c0036070 con 0x559fb601d4d0 2024-04-25T16:09:48.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.345+0000 7f51a77fe640 1 -- 172.21.15.31:0/597659773 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f51c004b430 con 0x559fb601d4d0 2024-04-25T16:09:48.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.346+0000 7f51d5569740 10 client.17286 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:48.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.346+0000 7f51d5569740 10 client.17286 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:09:48.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.346+0000 7f51d5569740 20 client.17286 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:48.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.346+0000 7f51d5569740 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x559fb5fda3a0 0x559fb646a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.346+0000 7f51d5569740 4 client.17286 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:48.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.346+0000 7f51d5569740 1 -- 172.21.15.31:0/597659773 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559fb6354800 con 0x559fb5fda3a0 2024-04-25T16:09:48.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.354+0000 7f51c7fff640 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x559fb5fda3a0 0x559fb646a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.359+0000 7f51c7fff640 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x559fb5fda3a0 0x559fb646a7d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=262ccd2a75f0b83e server_cookie=c28f97e4b83f2d9c in_seq=0 out_seq=0 2024-04-25T16:09:48.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.362+0000 7f51a77fe640 10 client.17286 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:09:48.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.362+0000 7f51a77fe640 1 -- 172.21.15.31:0/597659773 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559fb6354800 con 0x559fb5fda3a0 2024-04-25T16:09:48.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.362+0000 7f51a77fe640 10 client.17286 handle_command_reply: tid=0 2024-04-25T16:09:48.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.362+0000 7f51d5569740 1 client.17286 shutdown 2024-04-25T16:09:48.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.364+0000 7f51d5569740 1 -- 172.21.15.31:0/597659773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb601d4d0 msgr2=0x559fb63f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.364+0000 7f51d5569740 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb601d4d0 0x559fb63f6d40 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f51c00097b0 tx=0x7f51c0000f40 comp rx=0 tx=0).stop 2024-04-25T16:09:48.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.366+0000 7f51d5569740 1 -- 172.21.15.31:0/597659773 shutdown_connections 2024-04-25T16:09:48.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.367+0000 7f51d5569740 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559fb6046390 0x559fb63f8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.367+0000 7f51d5569740 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x559fb5fda3a0 0x559fb646a7d0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.367+0000 7f51d5569740 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559fb60256b0 0x559fb63f7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.367+0000 7f51d5569740 1 --2- 172.21.15.31:0/597659773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559fb601d4d0 0x559fb63f6d40 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.367+0000 7f51d5569740 1 -- 172.21.15.31:0/597659773 >> 172.21.15.31:0/597659773 conn(0x559fb603a4e0 msgr2=0x559fb63f0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:48.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.367+0000 7f51a77fe640 0 client.17286 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:09:48.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.367+0000 7f51d5569740 1 -- 172.21.15.31:0/597659773 shutdown_connections 2024-04-25T16:09:48.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.368+0000 7f51d5569740 1 -- 172.21.15.31:0/597659773 wait complete. 2024-04-25T16:09:48.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.371+0000 7f51d5569740 20 client.17286 trim_cache size 0 max 16384 2024-04-25T16:09:48.374 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:48.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.382+0000 7f51ce5e3640 1 -- 172.21.15.31:0/2896779033 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f51b00afc30 msgr2=0x7f51b00b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.382+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f51b00afc30 0x7f51b00b2140 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f51b80a2a20 tx=0x7f51b8010870 comp rx=0 tx=0).stop 2024-04-25T16:09:48.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.382+0000 7f51ce5e3640 1 -- 172.21.15.31:0/2896779033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c81ae790 msgr2=0x7f51c81b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.385 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.382+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c81ae790 0x7f51c81b0bb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f51b800b3f0 tx=0x7f51b800b8b0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.383+0000 7f51ce5e3640 1 -- 172.21.15.31:0/2896779033 shutdown_connections 2024-04-25T16:09:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.383+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f51b00afc30 0x7f51b00b2140 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.383+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f51c81ae790 0x7f51c81b0bb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.383+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51c8110030 0x7f51c81a9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.383+0000 7f51ce5e3640 1 --2- 172.21.15.31:0/2896779033 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f51c810a420 0x7f51c81a92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.383+0000 7f51ce5e3640 1 -- 172.21.15.31:0/2896779033 >> 172.21.15.31:0/2896779033 conn(0x7f51c80704e0 msgr2=0x7f51c810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:48.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.384+0000 7f51ce5e3640 1 -- 172.21.15.31:0/2896779033 shutdown_connections 2024-04-25T16:09:48.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.386+0000 7f51ce5e3640 1 -- 172.21.15.31:0/2896779033 wait complete. 2024-04-25T16:09:48.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:09:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.894+0000 7f629200c640 1 -- 172.21.15.31:0/775905115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10a340 msgr2=0x7f628c10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.894+0000 7f629200c640 1 --2- 172.21.15.31:0/775905115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10a340 0x7f628c10ac50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f627c009860 tx=0x7f627c02a380 comp rx=0 tx=0).stop 2024-04-25T16:09:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.894+0000 7f629200c640 1 -- 172.21.15.31:0/775905115 shutdown_connections 2024-04-25T16:09:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.894+0000 7f629200c640 1 --2- 172.21.15.31:0/775905115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628c10fe90 0x7f628c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.894+0000 7f629200c640 1 --2- 172.21.15.31:0/775905115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f628c10b190 0x7f628c10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.894+0000 7f629200c640 1 --2- 172.21.15.31:0/775905115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10a340 0x7f628c10ac50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.894+0000 7f629200c640 1 -- 172.21.15.31:0/775905115 >> 172.21.15.31:0/775905115 conn(0x7f628c070320 msgr2=0x7f628c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:48.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.895+0000 7f629200c640 1 -- 172.21.15.31:0/775905115 shutdown_connections 2024-04-25T16:09:48.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.895+0000 7f629200c640 1 -- 172.21.15.31:0/775905115 wait complete. 2024-04-25T16:09:48.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.896+0000 7f629200c640 1 Processor -- start 2024-04-25T16:09:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.896+0000 7f629200c640 1 -- start start 2024-04-25T16:09:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.896+0000 7f629200c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10b190 0x7f628c1a8f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.897+0000 7f629200c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628c10fe90 0x7f628c1a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.897+0000 7f629200c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f628c1ae450 0x7f628c1b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.897+0000 7f629200c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f628c11e240 con 0x7f628c1ae450 2024-04-25T16:09:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.897+0000 7f629200c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f628c11e0c0 con 0x7f628c10b190 2024-04-25T16:09:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.897+0000 7f629200c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f628c11e3c0 con 0x7f628c10fe90 2024-04-25T16:09:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.897+0000 7f629100a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10b190 0x7f628c1a8f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.898+0000 7f629100a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10b190 0x7f628c1a8f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58262/0 (socket says 172.21.15.31:58262) 2024-04-25T16:09:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.898+0000 7f629100a640 1 -- 172.21.15.31:0/4044969192 learned_addr learned my addr 172.21.15.31:0/4044969192 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.898+0000 7f6290809640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628c10fe90 0x7f628c1a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.899+0000 7f629100a640 1 -- 172.21.15.31:0/4044969192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628c10fe90 msgr2=0x7f628c1a9550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.899+0000 7f629100a640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628c10fe90 0x7f628c1a9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.899+0000 7f629100a640 1 -- 172.21.15.31:0/4044969192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f628c1ae450 msgr2=0x7f628c1b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.899+0000 7f629100a640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f628c1ae450 0x7f628c1b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.899+0000 7f629100a640 1 -- 172.21.15.31:0/4044969192 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628c111c90 con 0x7f628c10b190 2024-04-25T16:09:48.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.907+0000 7f629100a640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10b190 0x7f628c1a8f90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f627c02a8f0 tx=0x7f627c0042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.909+0000 7f62827fc640 1 -- 172.21.15.31:0/4044969192 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c037050 con 0x7f628c10b190 2024-04-25T16:09:48.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.909+0000 7f629200c640 1 -- 172.21.15.31:0/4044969192 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f628c111f20 con 0x7f628c10b190 2024-04-25T16:09:48.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.909+0000 7f629200c640 1 -- 172.21.15.31:0/4044969192 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f628c112330 con 0x7f628c10b190 2024-04-25T16:09:48.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.910+0000 7f62827fc640 1 -- 172.21.15.31:0/4044969192 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f627c038040 con 0x7f628c10b190 2024-04-25T16:09:48.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.910+0000 7f62827fc640 1 -- 172.21.15.31:0/4044969192 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c003850 con 0x7f628c10b190 2024-04-25T16:09:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.918+0000 7f629a201740 1 Processor -- start 2024-04-25T16:09:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.919+0000 7f629a201740 1 -- start start 2024-04-25T16:09:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.919+0000 7f629a201740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755aa390 0x55e27594c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.919+0000 7f629a201740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 0x55e27594cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.919+0000 7f629a201740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755896b0 0x55e27594dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.919+0000 7f629a201740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e2758b0d80 con 0x55e2755814d0 2024-04-25T16:09:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.919+0000 7f629a201740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e27586aa20 con 0x55e2755896b0 2024-04-25T16:09:48.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.919+0000 7f629a201740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e2758acd40 con 0x55e2755aa390 2024-04-25T16:09:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.922+0000 7f629180b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755aa390 0x55e27594c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.922+0000 7f629180b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755aa390 0x55e27594c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47838/0 (socket says 172.21.15.31:47838) 2024-04-25T16:09:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.922+0000 7f629180b640 1 -- 172.21.15.31:0/2398983261 learned_addr learned my addr 172.21.15.31:0/2398983261 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:48.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.923+0000 7f6290809640 1 --2- 172.21.15.31:0/2398983261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755896b0 0x55e27594dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.923+0000 7f62827fc640 1 -- 172.21.15.31:0/4044969192 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f627c005220 con 0x7f628c10b190 2024-04-25T16:09:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.924+0000 7f629180b640 1 -- 172.21.15.31:0/2398983261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755896b0 msgr2=0x55e27594dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.924+0000 7f629180b640 1 --2- 172.21.15.31:0/2398983261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755896b0 0x55e27594dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.924+0000 7f629180b640 1 -- 172.21.15.31:0/2398983261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 msgr2=0x55e27594cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.924+0000 7f629180b640 1 --2- 172.21.15.31:0/2398983261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 0x55e27594cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.924+0000 7f629180b640 1 -- 172.21.15.31:0/2398983261 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e275875c30 con 0x55e2755aa390 2024-04-25T16:09:48.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.925+0000 7f629180b640 1 --2- 172.21.15.31:0/2398983261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755aa390 0x55e27594c9e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f628800ba90 tx=0x7f628800bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c724596b2da5b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.926+0000 7f62777fe640 1 -- 172.21.15.31:0/2398983261 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288016070 con 0x55e2755aa390 2024-04-25T16:09:48.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.926+0000 7f62777fe640 1 -- 172.21.15.31:0/2398983261 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62880054d0 con 0x55e2755aa390 2024-04-25T16:09:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.928+0000 7f629a201740 1 -- 172.21.15.31:0/2398983261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755aa390 msgr2=0x55e27594c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.928+0000 7f629a201740 1 --2- 172.21.15.31:0/2398983261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755aa390 0x55e27594c9e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f628800ba90 tx=0x7f628800bf50 comp rx=0 tx=0).stop 2024-04-25T16:09:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.929+0000 7f629a201740 1 -- 172.21.15.31:0/2398983261 shutdown_connections 2024-04-25T16:09:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.929+0000 7f629a201740 1 --2- 172.21.15.31:0/2398983261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755896b0 0x55e27594dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.929+0000 7f629a201740 1 --2- 172.21.15.31:0/2398983261 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 0x55e27594cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.929+0000 7f629a201740 1 --2- 172.21.15.31:0/2398983261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755aa390 0x55e27594c9e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.929+0000 7f629a201740 1 -- 172.21.15.31:0/2398983261 >> 172.21.15.31:0/2398983261 conn(0x55e27559e4e0 msgr2=0x55e2759481f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:48.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.929+0000 7f62827fc640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62780afcf0 0x7f62780b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.930+0000 7f62827fc640 1 -- 172.21.15.31:0/4044969192 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f627c0febd0 con 0x7f628c10b190 2024-04-25T16:09:48.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.935+0000 7f629a201740 1 -- 172.21.15.31:0/2398983261 shutdown_connections 2024-04-25T16:09:48.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.936+0000 7f6290809640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62780afcf0 0x7f62780b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.939+0000 7f629a201740 1 -- 172.21.15.31:0/2398983261 wait complete. 2024-04-25T16:09:48.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.939+0000 7f629a201740 1 -- start start 2024-04-25T16:09:48.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.940+0000 7f629a201740 1 Processor -- start 2024-04-25T16:09:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.941+0000 7f629a201740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755814d0 0x55e275956780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.941+0000 7f629a201740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 0x55e275956cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.941+0000 7f629a201740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e27595dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.941+0000 7f629a201740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e27594e750 con 0x55e2755896b0 2024-04-25T16:09:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.941+0000 7f629a201740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e275837dc0 con 0x55e2755aa390 2024-04-25T16:09:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.941+0000 7f629a201740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e27594e8d0 con 0x55e2755814d0 2024-04-25T16:09:48.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.941+0000 7f629a201740 10 client.0 _ll_register_callbacks cb 0x55e275853600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:48.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.943+0000 7f629180b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e27595dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.943+0000 7f629180b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e27595dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58286/0 (socket says 172.21.15.31:58286) 2024-04-25T16:09:48.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.943+0000 7f629180b640 1 -- 172.21.15.31:0/1291884345 learned_addr learned my addr 172.21.15.31:0/1291884345 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:48.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.943+0000 7f629100a640 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755814d0 0x55e275956780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.944+0000 7f629180b640 1 -- 172.21.15.31:0/1291884345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755814d0 msgr2=0x55e275956780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:48.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.944+0000 7f629180b640 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755814d0 0x55e275956780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.944+0000 7f629180b640 1 -- 172.21.15.31:0/1291884345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 msgr2=0x55e275956cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:09:48.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.944+0000 7f629180b640 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 0x55e275956cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.944+0000 7f629180b640 1 -- 172.21.15.31:0/1291884345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2755c23f0 con 0x55e2755aa390 2024-04-25T16:09:48.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.944+0000 7f629180b640 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e27595dc20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6288004fb0 tx=0x7f628800b660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5331d3068b8b684 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.944+0000 7f6274ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:48.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.945+0000 7f6274ff9640 1 -- 172.21.15.31:0/1291884345 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6288016070 con 0x55e2755aa390 2024-04-25T16:09:48.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.945+0000 7f629a201740 1 -- 172.21.15.31:0/1291884345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62880054d0 con 0x55e2755aa390 2024-04-25T16:09:48.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.945+0000 7f6274ff9640 1 -- 172.21.15.31:0/1291884345 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f628800dca0 con 0x55e2755aa390 2024-04-25T16:09:48.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.948+0000 7f6274ff9640 1 -- 172.21.15.31:0/1291884345 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6288004030 con 0x55e2755aa390 2024-04-25T16:09:48.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.952+0000 7f629a201740 10 client.26975 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:48.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.952+0000 7f629a201740 1 -- 172.21.15.31:0/1291884345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e27568c7d0 con 0x55e2755aa390 2024-04-25T16:09:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.962+0000 7f6274ff9640 1 -- 172.21.15.31:0/1291884345 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f62880054d0 con 0x55e2755aa390 2024-04-25T16:09:48.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.962+0000 7f6274ff9640 1 -- 172.21.15.31:0/1291884345 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f62880035f0 con 0x55e2755aa390 2024-04-25T16:09:48.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.962+0000 7f629a201740 10 client.26975 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:48.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.962+0000 7f629a201740 10 client.26975 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:09:48.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.962+0000 7f629a201740 20 client.26975 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:48.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.962+0000 7f629a201740 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55e275568f60 0x55e275964890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:48.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.962+0000 7f629a201740 4 client.26975 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:48.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.962+0000 7f629a201740 1 -- 172.21.15.31:0/1291884345 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2758be990 con 0x55e275568f60 2024-04-25T16:09:48.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.963+0000 7f629100a640 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55e275568f60 0x55e275964890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:48.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.966+0000 7f629100a640 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55e275568f60 0x55e275964890 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8e23077e82f84368 server_cookie=77cb53fd2cc2c2e7 in_seq=0 out_seq=0 2024-04-25T16:09:48.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.966+0000 7f6290809640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62780afcf0 0x7f62780b2200 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x55e2759577f0 tx=0x7f62840095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:48.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.966+0000 7f6274ff9640 10 client.26975 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:09:48.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.970+0000 7f6274ff9640 1 -- 172.21.15.31:0/1291884345 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e2758be990 con 0x55e275568f60 2024-04-25T16:09:48.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.970+0000 7f6274ff9640 10 client.26975 handle_command_reply: tid=0 2024-04-25T16:09:48.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.971+0000 7f629a201740 1 client.26975 shutdown 2024-04-25T16:09:48.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.975+0000 7f629a201740 1 -- 172.21.15.31:0/1291884345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 msgr2=0x55e27595dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:48.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.975+0000 7f629a201740 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e27595dc20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6288004fb0 tx=0x7f628800b660 comp rx=0 tx=0).stop 2024-04-25T16:09:48.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.975+0000 7f629a201740 1 -- 172.21.15.31:0/1291884345 shutdown_connections 2024-04-25T16:09:48.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.975+0000 7f629a201740 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55e275568f60 0x55e275964890 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.975+0000 7f629a201740 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e27595dc20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.975+0000 7f629a201740 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 0x55e275956cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.975+0000 7f629a201740 1 --2- 172.21.15.31:0/1291884345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755814d0 0x55e275956780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:48.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.975+0000 7f629a201740 1 -- 172.21.15.31:0/1291884345 >> 172.21.15.31:0/1291884345 conn(0x55e27559e4e0 msgr2=0x55e2759473f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:48.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.976+0000 7f6274ff9640 0 client.26975 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:09:48.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.980+0000 7f629a201740 1 -- 172.21.15.31:0/1291884345 shutdown_connections 2024-04-25T16:09:48.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.981+0000 7f629a201740 1 -- 172.21.15.31:0/1291884345 wait complete. 2024-04-25T16:09:48.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:48.981+0000 7f629a201740 20 client.26975 trim_cache size 0 max 16384 2024-04-25T16:09:49.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.028+0000 7f629a201740 1 Processor -- start 2024-04-25T16:09:49.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.031+0000 7f629a201740 1 -- start start 2024-04-25T16:09:49.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.031+0000 7f629a201740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755814d0 0x55e275956780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.031+0000 7f629a201740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 0x55e27595db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.031+0000 7f629a201740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e275956cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.031+0000 7f629a201740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f627000e4c0 con 0x55e2755896b0 2024-04-25T16:09:49.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.031+0000 7f629a201740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f6270010000 con 0x55e2755aa390 2024-04-25T16:09:49.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.031+0000 7f629a201740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f627000eb90 con 0x55e2755814d0 2024-04-25T16:09:49.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.031+0000 7f6290809640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 0x55e27595db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.032+0000 7f6290809640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 0x55e27595db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60248/0 (socket says 172.21.15.31:60248) 2024-04-25T16:09:49.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.032+0000 7f6290809640 1 -- 172.21.15.31:0/400272147 learned_addr learned my addr 172.21.15.31:0/400272147 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:49.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.032+0000 7f6290809640 1 -- 172.21.15.31:0/400272147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755814d0 msgr2=0x55e275956780 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:49.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.032+0000 7f6290809640 1 --2- 172.21.15.31:0/400272147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755814d0 0x55e275956780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.032+0000 7f6290809640 1 -- 172.21.15.31:0/400272147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 msgr2=0x55e275956cc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:09:49.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.032+0000 7f6290809640 1 --2- 172.21.15.31:0/400272147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e275956cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.032+0000 7f6290809640 1 -- 172.21.15.31:0/400272147 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e275960df0 con 0x55e2755896b0 2024-04-25T16:09:49.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.034+0000 7f6290809640 1 --2- 172.21.15.31:0/400272147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 0x55e27595db40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f628401c740 tx=0x7f6284015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216e0b2bf98722a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.037+0000 7f6276ffd640 1 -- 172.21.15.31:0/400272147 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628402c070 con 0x55e2755896b0 2024-04-25T16:09:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.037+0000 7f6276ffd640 1 -- 172.21.15.31:0/400272147 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f628401cdf0 con 0x55e2755896b0 2024-04-25T16:09:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.037+0000 7f6276ffd640 1 -- 172.21.15.31:0/400272147 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f628400e5b0 con 0x55e2755896b0 2024-04-25T16:09:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.039+0000 7f629a201740 1 -- 172.21.15.31:0/400272147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 msgr2=0x55e27595db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.039+0000 7f629a201740 1 --2- 172.21.15.31:0/400272147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 0x55e27595db40 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f628401c740 tx=0x7f6284015000 comp rx=0 tx=0).stop 2024-04-25T16:09:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.040+0000 7f629a201740 1 -- 172.21.15.31:0/400272147 shutdown_connections 2024-04-25T16:09:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.040+0000 7f629a201740 1 --2- 172.21.15.31:0/400272147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e275956cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.040+0000 7f629a201740 1 --2- 172.21.15.31:0/400272147 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755896b0 0x55e27595db40 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.040+0000 7f629a201740 1 --2- 172.21.15.31:0/400272147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755814d0 0x55e275956780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.040+0000 7f629a201740 1 -- 172.21.15.31:0/400272147 >> 172.21.15.31:0/400272147 conn(0x55e27559e4e0 msgr2=0x55e2759513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:49.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.042+0000 7f629a201740 1 -- 172.21.15.31:0/400272147 shutdown_connections 2024-04-25T16:09:49.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.045+0000 7f629a201740 1 -- 172.21.15.31:0/400272147 wait complete. 2024-04-25T16:09:49.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.046+0000 7f629a201740 1 -- start start 2024-04-25T16:09:49.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.046+0000 7f629a201740 1 Processor -- start 2024-04-25T16:09:49.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.047+0000 7f629a201740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 0x55e27595da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.047+0000 7f629a201740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755896b0 0x55e27595a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.047+0000 7f629a201740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e27595aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.047+0000 7f629a201740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f627000f930 con 0x55e2755814d0 2024-04-25T16:09:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.047+0000 7f629a201740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f627000f260 con 0x55e2755aa390 2024-04-25T16:09:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.047+0000 7f629a201740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f627000c4a0 con 0x55e2755896b0 2024-04-25T16:09:49.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.047+0000 7f629a201740 10 client.0 _ll_register_callbacks cb 0x7f628800b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.047+0000 7f629100a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 0x55e27595da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.048+0000 7f629100a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 0x55e27595da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60262/0 (socket says 172.21.15.31:60262) 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.048+0000 7f629100a640 1 -- 172.21.15.31:0/3911869308 learned_addr learned my addr 172.21.15.31:0/3911869308 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.049+0000 7f629100a640 1 -- 172.21.15.31:0/3911869308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755896b0 msgr2=0x55e27595a590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.049+0000 7f629100a640 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755896b0 0x55e27595a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.049+0000 7f629100a640 1 -- 172.21.15.31:0/3911869308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 msgr2=0x55e27595aad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.049+0000 7f629100a640 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e27595aad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.049+0000 7f629100a640 1 -- 172.21.15.31:0/3911869308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e275948d80 con 0x55e2755814d0 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.050+0000 7f629100a640 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 0x55e27595da10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f627c009780 tx=0x7f627c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec9fcbba2b8581d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:49.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.052+0000 7f6274ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:49.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.053+0000 7f6274ff9640 1 -- 172.21.15.31:0/3911869308 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627c0cb490 con 0x55e2755814d0 2024-04-25T16:09:49.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.055+0000 7f6274ff9640 1 -- 172.21.15.31:0/3911869308 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f627c005510 con 0x55e2755814d0 2024-04-25T16:09:49.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.058+0000 7f629a201740 1 -- 172.21.15.31:0/3911869308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f628401cdf0 con 0x55e2755814d0 2024-04-25T16:09:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.062+0000 7f6274ff9640 1 -- 172.21.15.31:0/3911869308 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f627c0cff40 con 0x55e2755814d0 2024-04-25T16:09:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.063+0000 7f629a201740 10 client.17295 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:49.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.063+0000 7f629a201740 1 -- 172.21.15.31:0/3911869308 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e2759cbbd0 con 0x55e2755814d0 2024-04-25T16:09:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.063+0000 7f6274ff9640 1 -- 172.21.15.31:0/3911869308 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f628401cdf0 con 0x55e2755814d0 2024-04-25T16:09:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.064+0000 7f6274ff9640 1 -- 172.21.15.31:0/3911869308 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f627c0cfab0 con 0x55e2755814d0 2024-04-25T16:09:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.064+0000 7f629a201740 10 client.17295 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.064+0000 7f629a201740 10 client.17295 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:09:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.064+0000 7f629a201740 20 client.17295 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.064+0000 7f629a201740 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55e27553e3a0 0x55e2759ce690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.064+0000 7f629a201740 4 client.17295 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:49.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.064+0000 7f629a201740 1 -- 172.21.15.31:0/3911869308 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e2758b8800 con 0x55e27553e3a0 2024-04-25T16:09:49.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.066+0000 7f6290809640 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55e27553e3a0 0x55e2759ce690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.081+0000 7f6290809640 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55e27553e3a0 0x55e2759ce690 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bdb5f922f51a404 server_cookie=8e4e99725b40501d in_seq=0 out_seq=0 2024-04-25T16:09:49.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.081+0000 7f6274ff9640 10 client.17295 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:09:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.083+0000 7f6274ff9640 1 -- 172.21.15.31:0/3911869308 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e2758b8800 con 0x55e27553e3a0 2024-04-25T16:09:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.083+0000 7f6274ff9640 10 client.17295 handle_command_reply: tid=0 2024-04-25T16:09:49.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.083+0000 7f629a201740 1 client.17295 shutdown 2024-04-25T16:09:49.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.085+0000 7f629a201740 1 -- 172.21.15.31:0/3911869308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 msgr2=0x55e27595da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.085+0000 7f629a201740 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 0x55e27595da10 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f627c009780 tx=0x7f627c005f90 comp rx=0 tx=0).stop 2024-04-25T16:09:49.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.088+0000 7f629a201740 1 -- 172.21.15.31:0/3911869308 shutdown_connections 2024-04-25T16:09:49.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.088+0000 7f629a201740 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55e27553e3a0 0x55e2759ce690 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.088+0000 7f629a201740 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e2755aa390 0x55e27595aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.088+0000 7f629a201740 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e2755896b0 0x55e27595a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.088+0000 7f629a201740 1 --2- 172.21.15.31:0/3911869308 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e2755814d0 0x55e27595da10 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.088+0000 7f629a201740 1 -- 172.21.15.31:0/3911869308 >> 172.21.15.31:0/3911869308 conn(0x55e27559e4e0 msgr2=0x55e2759513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:49.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.089+0000 7f6274ff9640 0 client.17295 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:09:49.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.090+0000 7f629a201740 1 -- 172.21.15.31:0/3911869308 shutdown_connections 2024-04-25T16:09:49.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.091+0000 7f629a201740 1 -- 172.21.15.31:0/3911869308 wait complete. 2024-04-25T16:09:49.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.091+0000 7f629a201740 20 client.17295 trim_cache size 0 max 16384 2024-04-25T16:09:49.098 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.103+0000 7f629200c640 1 -- 172.21.15.31:0/4044969192 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62780afcf0 msgr2=0x7f62780b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.103+0000 7f629200c640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62780afcf0 0x7f62780b2200 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x55e2759577f0 tx=0x7f62840095c0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.103+0000 7f629200c640 1 -- 172.21.15.31:0/4044969192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10b190 msgr2=0x7f628c1a8f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.103+0000 7f629200c640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10b190 0x7f628c1a8f90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f627c02a8f0 tx=0x7f627c0042e0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.106+0000 7f629200c640 1 -- 172.21.15.31:0/4044969192 shutdown_connections 2024-04-25T16:09:49.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.106+0000 7f629200c640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f62780afcf0 0x7f62780b2200 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.106+0000 7f629200c640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f628c1ae450 0x7f628c1b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.106+0000 7f629200c640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628c10fe90 0x7f628c1a9550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.106+0000 7f629200c640 1 --2- 172.21.15.31:0/4044969192 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f628c10b190 0x7f628c1a8f90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.106+0000 7f629200c640 1 -- 172.21.15.31:0/4044969192 >> 172.21.15.31:0/4044969192 conn(0x7f628c070320 msgr2=0x7f628c10be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:49.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.109+0000 7f629200c640 1 -- 172.21.15.31:0/4044969192 shutdown_connections 2024-04-25T16:09:49.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.110+0000 7f629200c640 1 -- 172.21.15.31:0/4044969192 wait complete. 2024-04-25T16:09:49.199 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:09:49.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:49 smithi063 ceph-mon[32413]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 918 MiB/s wr, 2.47k op/s 2024-04-25T16:09:49.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:49 smithi186 ceph-mon[33258]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 918 MiB/s wr, 2.47k op/s 2024-04-25T16:09:49.401 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:49 smithi031 ceph-mon[29140]: pgmap v606: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 918 MiB/s wr, 2.47k op/s 2024-04-25T16:09:49.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.695+0000 7fc14f146640 1 -- 172.21.15.31:0/3028047352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14810f760 msgr2=0x7fc148111b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.695+0000 7fc14f146640 1 --2- 172.21.15.31:0/3028047352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14810f760 0x7fc148111b80 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009860 tx=0x7fc13c02a380 comp rx=0 tx=0).stop 2024-04-25T16:09:49.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.695+0000 7fc14f146640 1 -- 172.21.15.31:0/3028047352 shutdown_connections 2024-04-25T16:09:49.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.695+0000 7fc14f146640 1 --2- 172.21.15.31:0/3028047352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc1481120c0 0x7fc148114520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.696+0000 7fc14f146640 1 --2- 172.21.15.31:0/3028047352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc14810f760 0x7fc148111b80 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.696+0000 7fc14f146640 1 --2- 172.21.15.31:0/3028047352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc148109fc0 0x7fc14810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.696+0000 7fc14f146640 1 -- 172.21.15.31:0/3028047352 >> 172.21.15.31:0/3028047352 conn(0x7fc1480704e0 msgr2=0x7fc148072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:49.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.703+0000 7fc14f146640 1 -- 172.21.15.31:0/3028047352 shutdown_connections 2024-04-25T16:09:49.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.709+0000 7fc14f146640 1 -- 172.21.15.31:0/3028047352 wait complete. 2024-04-25T16:09:49.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.710+0000 7fc14f146640 1 Processor -- start 2024-04-25T16:09:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.717+0000 7fc14f146640 1 -- start start 2024-04-25T16:09:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.717+0000 7fc14f146640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc148109fc0 0x7fc14810cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.717+0000 7fc14f146640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14810f760 0x7fc14810b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.717+0000 7fc14f146640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1481120c0 0x7fc14810bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.717+0000 7fc14f146640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc148117f40 con 0x7fc1481120c0 2024-04-25T16:09:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.717+0000 7fc14f146640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc148117dc0 con 0x7fc148109fc0 2024-04-25T16:09:49.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.717+0000 7fc14f146640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc1481180c0 con 0x7fc14810f760 2024-04-25T16:09:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.722+0000 7fc147fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14810f760 0x7fc14810b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.722+0000 7fc147fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14810f760 0x7fc14810b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47880/0 (socket says 172.21.15.31:47880) 2024-04-25T16:09:49.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.722+0000 7fc147fff640 1 -- 172.21.15.31:0/3261096840 learned_addr learned my addr 172.21.15.31:0/3261096840 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:49.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.724+0000 7fc147fff640 1 -- 172.21.15.31:0/3261096840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc148109fc0 msgr2=0x7fc14810cfc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:09:49.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.724+0000 7fc147fff640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc148109fc0 0x7fc14810cfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.724+0000 7fc147fff640 1 -- 172.21.15.31:0/3261096840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1481120c0 msgr2=0x7fc14810bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.724+0000 7fc147fff640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1481120c0 0x7fc14810bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.724+0000 7fc147fff640 1 -- 172.21.15.31:0/3261096840 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14810c2d0 con 0x7fc14810f760 2024-04-25T16:09:49.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.726+0000 7fc147fff640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14810f760 0x7fc14810b660 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc13c035040 tx=0x7fc13c032cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.751+0000 7fc145ffb640 1 -- 172.21.15.31:0/3261096840 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c037050 con 0x7fc14810f760 2024-04-25T16:09:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.751+0000 7fc145ffb640 1 -- 172.21.15.31:0/3261096840 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc13c043070 con 0x7fc14810f760 2024-04-25T16:09:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.751+0000 7fc145ffb640 1 -- 172.21.15.31:0/3261096840 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c003d10 con 0x7fc14810f760 2024-04-25T16:09:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.751+0000 7fc14f146640 1 -- 172.21.15.31:0/3261096840 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1481acc30 con 0x7fc14810f760 2024-04-25T16:09:49.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.751+0000 7fc14f146640 1 -- 172.21.15.31:0/3261096840 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1481acfe0 con 0x7fc14810f760 2024-04-25T16:09:49.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.775+0000 7fc155ff7740 1 Processor -- start 2024-04-25T16:09:49.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.775+0000 7fc155ff7740 1 -- start start 2024-04-25T16:09:49.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.776+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c073390 0x560d2c41e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.776+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c04a4d0 0x560d2c41e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.776+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c41f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.776+0000 7fc155ff7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560d2c379d80 con 0x560d2c0526b0 2024-04-25T16:09:49.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.776+0000 7fc155ff7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560d2c333a20 con 0x560d2c073390 2024-04-25T16:09:49.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.776+0000 7fc155ff7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d2c375d40 con 0x560d2c04a4d0 2024-04-25T16:09:49.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.778+0000 7fc145ffb640 1 -- 172.21.15.31:0/3261096840 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc13c004040 con 0x7fc14810f760 2024-04-25T16:09:49.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.779+0000 7fc14d6e3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c073390 0x560d2c41e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.781+0000 7fc147fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c41f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.783+0000 7fc145ffb640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1300afd00 0x7fc1300b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.785+0000 7fc147fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c41f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60288/0 (socket says 172.21.15.31:60288) 2024-04-25T16:09:49.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.785+0000 7fc147fff640 1 -- 172.21.15.31:0/2510338129 learned_addr learned my addr 172.21.15.31:0/2510338129 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:49.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.785+0000 7fc14cee2640 1 --2- 172.21.15.31:0/2510338129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c04a4d0 0x560d2c41e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.785+0000 7fc145ffb640 1 -- 172.21.15.31:0/3261096840 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc13c0ff1d0 con 0x7fc14810f760 2024-04-25T16:09:49.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.793+0000 7fc147fff640 1 -- 172.21.15.31:0/2510338129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c04a4d0 msgr2=0x560d2c41e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:49.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.793+0000 7fc147fff640 1 --2- 172.21.15.31:0/2510338129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c04a4d0 0x560d2c41e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.793+0000 7fc147fff640 1 -- 172.21.15.31:0/2510338129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c073390 msgr2=0x560d2c41e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:49.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.793+0000 7fc147fff640 1 --2- 172.21.15.31:0/2510338129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c073390 0x560d2c41e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.793+0000 7fc147fff640 1 -- 172.21.15.31:0/2510338129 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d2c33ec30 con 0x560d2c0526b0 2024-04-25T16:09:49.807 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.802+0000 7fc147fff640 1 --2- 172.21.15.31:0/2510338129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c41f230 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009830 tx=0x7fc13c02a380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79363b59ca0fc63 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.808+0000 7fc14d6e3640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1300afd00 0x7fc1300b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.809+0000 7fc12effd640 1 -- 172.21.15.31:0/2510338129 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0c6fb0 con 0x560d2c0526b0 2024-04-25T16:09:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.809+0000 7fc12effd640 1 -- 172.21.15.31:0/2510338129 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc13c038040 con 0x560d2c0526b0 2024-04-25T16:09:49.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.809+0000 7fc12effd640 1 -- 172.21.15.31:0/2510338129 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0c6dd0 con 0x560d2c0526b0 2024-04-25T16:09:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.810+0000 7fc155ff7740 1 -- 172.21.15.31:0/2510338129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 msgr2=0x560d2c41f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.812 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.810+0000 7fc155ff7740 1 --2- 172.21.15.31:0/2510338129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c41f230 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fc13c009830 tx=0x7fc13c02a380 comp rx=0 tx=0).stop 2024-04-25T16:09:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.826+0000 7fc155ff7740 1 -- 172.21.15.31:0/2510338129 shutdown_connections 2024-04-25T16:09:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.826+0000 7fc155ff7740 1 --2- 172.21.15.31:0/2510338129 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c41f230 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.826+0000 7fc155ff7740 1 --2- 172.21.15.31:0/2510338129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c04a4d0 0x560d2c41e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.826+0000 7fc155ff7740 1 --2- 172.21.15.31:0/2510338129 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c073390 0x560d2c41e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.826+0000 7fc155ff7740 1 -- 172.21.15.31:0/2510338129 >> 172.21.15.31:0/2510338129 conn(0x560d2c0674e0 msgr2=0x560d2c411020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:49.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.828+0000 7fc155ff7740 1 -- 172.21.15.31:0/2510338129 shutdown_connections 2024-04-25T16:09:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.828+0000 7fc155ff7740 1 -- 172.21.15.31:0/2510338129 wait complete. 2024-04-25T16:09:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.828+0000 7fc155ff7740 1 -- start start 2024-04-25T16:09:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.828+0000 7fc155ff7740 1 Processor -- start 2024-04-25T16:09:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.829+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c416b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.829+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c417070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.829+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c41bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.829+0000 7fc155ff7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560d2c41fb00 con 0x560d2c0526b0 2024-04-25T16:09:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.829+0000 7fc155ff7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560d2c300dc0 con 0x560d2c04a4d0 2024-04-25T16:09:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.829+0000 7fc155ff7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560d2c41fc80 con 0x560d2c073390 2024-04-25T16:09:49.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.829+0000 7fc155ff7740 10 client.0 _ll_register_callbacks cb 0x560d2c31c600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:49.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.832+0000 7fc14cee2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c416b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.833+0000 7fc14cee2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c416b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58340/0 (socket says 172.21.15.31:58340) 2024-04-25T16:09:49.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.833+0000 7fc14cee2640 1 -- 172.21.15.31:0/3777639766 learned_addr learned my addr 172.21.15.31:0/3777639766 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:49.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.833+0000 7fc147fff640 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c417070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.834+0000 7fc147fff640 1 -- 172.21.15.31:0/3777639766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 msgr2=0x560d2c41bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.834+0000 7fc147fff640 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c41bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.834+0000 7fc147fff640 1 -- 172.21.15.31:0/3777639766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 msgr2=0x560d2c416b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.834+0000 7fc147fff640 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c416b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.834+0000 7fc147fff640 1 -- 172.21.15.31:0/3777639766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d2c08b3f0 con 0x560d2c0526b0 2024-04-25T16:09:49.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.835+0000 7fc147fff640 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c417070 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000c00 tx=0x7fc13c0c7660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edfb87d4013a8587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:49.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.850+0000 7fc127fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:49.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.850+0000 7fc127fff640 1 -- 172.21.15.31:0/3777639766 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc13c0c7d60 con 0x560d2c0526b0 2024-04-25T16:09:49.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.850+0000 7fc127fff640 1 -- 172.21.15.31:0/3777639766 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc13c0c38f0 con 0x560d2c0526b0 2024-04-25T16:09:49.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.850+0000 7fc155ff7740 1 -- 172.21.15.31:0/3777639766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc13c038040 con 0x560d2c0526b0 2024-04-25T16:09:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.854+0000 7fc127fff640 1 -- 172.21.15.31:0/3777639766 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc13c0d04c0 con 0x560d2c0526b0 2024-04-25T16:09:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.854+0000 7fc155ff7740 10 client.17304 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:49.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.854+0000 7fc155ff7740 1 -- 172.21.15.31:0/3777639766 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560d2c1557d0 con 0x560d2c0526b0 2024-04-25T16:09:49.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.859+0000 7fc127fff640 1 -- 172.21.15.31:0/3777639766 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc13c038040 con 0x560d2c0526b0 2024-04-25T16:09:49.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.859+0000 7fc127fff640 1 -- 172.21.15.31:0/3777639766 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc13c0dcab0 con 0x560d2c0526b0 2024-04-25T16:09:49.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.859+0000 7fc155ff7740 10 client.17304 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:49.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.859+0000 7fc155ff7740 10 client.17304 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:09:49.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.859+0000 7fc155ff7740 20 client.17304 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:49.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.859+0000 7fc155ff7740 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x560d2c031f60 0x560d2c43dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.859+0000 7fc155ff7740 4 client.17304 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:49.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.859+0000 7fc155ff7740 1 -- 172.21.15.31:0/3777639766 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d2c387990 con 0x560d2c031f60 2024-04-25T16:09:49.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.861+0000 7fc14cee2640 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x560d2c031f60 0x560d2c43dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.869+0000 7fc14d6e3640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1300afd00 0x7fc1300b2210 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x560d2c41f070 tx=0x7fc1380046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:49.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.869+0000 7fc14cee2640 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x560d2c031f60 0x560d2c43dc80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e89959c9b54a54d server_cookie=19553d11f0e438d7 in_seq=0 out_seq=0 2024-04-25T16:09:49.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.869+0000 7fc127fff640 10 client.17304 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:09:49.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.871+0000 7fc127fff640 1 -- 172.21.15.31:0/3777639766 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560d2c387990 con 0x560d2c031f60 2024-04-25T16:09:49.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.871+0000 7fc127fff640 10 client.17304 handle_command_reply: tid=0 2024-04-25T16:09:49.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.871+0000 7fc155ff7740 1 client.17304 shutdown 2024-04-25T16:09:49.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc155ff7740 1 -- 172.21.15.31:0/3777639766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 msgr2=0x560d2c417070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc155ff7740 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c417070 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000c00 tx=0x7fc13c0c7660 comp rx=0 tx=0).stop 2024-04-25T16:09:49.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc155ff7740 1 -- 172.21.15.31:0/3777639766 shutdown_connections 2024-04-25T16:09:49.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc155ff7740 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c41bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc155ff7740 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x560d2c031f60 0x560d2c43dc80 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc155ff7740 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0526b0 0x560d2c417070 secure :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fc13c000c00 tx=0x7fc13c0c7660 comp rx=0 tx=0).stop 2024-04-25T16:09:49.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc155ff7740 1 --2- 172.21.15.31:0/3777639766 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c416b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc155ff7740 1 -- 172.21.15.31:0/3777639766 >> 172.21.15.31:0/3777639766 conn(0x560d2c0674e0 msgr2=0x560d2c410040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:49.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc127fff640 0 client.17304 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:09:49.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.872+0000 7fc155ff7740 1 -- 172.21.15.31:0/3777639766 shutdown_connections 2024-04-25T16:09:49.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.873+0000 7fc155ff7740 1 -- 172.21.15.31:0/3777639766 wait complete. 2024-04-25T16:09:49.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.873+0000 7fc155ff7740 20 client.17304 trim_cache size 0 max 16384 2024-04-25T16:09:49.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.920+0000 7fc155ff7740 1 Processor -- start 2024-04-25T16:09:49.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.920+0000 7fc155ff7740 1 -- start start 2024-04-25T16:09:49.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.924+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c40f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.924+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c40fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.925+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c410090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.926+0000 7fc14cee2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c40f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.926+0000 7fc14d6e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c410090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.926+0000 7fc14d6e3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c410090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60306/0 (socket says 172.21.15.31:60306) 2024-04-25T16:09:49.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.926+0000 7fc14d6e3640 1 -- 172.21.15.31:0/2702358991 learned_addr learned my addr 172.21.15.31:0/2702358991 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:49.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.926+0000 7fc155ff7740 1 -- 172.21.15.31:0/2702358991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc13400e4a0 con 0x560d2c0073a0 2024-04-25T16:09:49.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.926+0000 7fc155ff7740 1 -- 172.21.15.31:0/2702358991 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc13400ffe0 con 0x560d2c04a4d0 2024-04-25T16:09:49.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.926+0000 7fc155ff7740 1 -- 172.21.15.31:0/2702358991 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc13400eb70 con 0x560d2c073390 2024-04-25T16:09:49.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.927+0000 7fc147fff640 1 --2- 172.21.15.31:0/2702358991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c40fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.929+0000 7fc14d6e3640 1 -- 172.21.15.31:0/2702358991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 msgr2=0x560d2c40fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:49.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.929+0000 7fc14d6e3640 1 --2- 172.21.15.31:0/2702358991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c40fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.929+0000 7fc14d6e3640 1 -- 172.21.15.31:0/2702358991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 msgr2=0x560d2c40f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:49.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.929+0000 7fc14d6e3640 1 --2- 172.21.15.31:0/2702358991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c40f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.929+0000 7fc14d6e3640 1 -- 172.21.15.31:0/2702358991 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d2c417c70 con 0x560d2c0073a0 2024-04-25T16:09:49.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.930+0000 7fc14d6e3640 1 --2- 172.21.15.31:0/2702358991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c410090 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc1380116e0 tx=0x7fc138016070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afcbebac2aa4765 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:49.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.930+0000 7fc12e7fc640 1 -- 172.21.15.31:0/2702358991 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc138022030 con 0x560d2c0073a0 2024-04-25T16:09:49.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.930+0000 7fc12e7fc640 1 -- 172.21.15.31:0/2702358991 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc138018070 con 0x560d2c0073a0 2024-04-25T16:09:49.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.932+0000 7fc155ff7740 1 -- 172.21.15.31:0/2702358991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 msgr2=0x560d2c410090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.932+0000 7fc155ff7740 1 --2- 172.21.15.31:0/2702358991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c410090 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fc1380116e0 tx=0x7fc138016070 comp rx=0 tx=0).stop 2024-04-25T16:09:49.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.934+0000 7fc155ff7740 1 -- 172.21.15.31:0/2702358991 shutdown_connections 2024-04-25T16:09:49.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.934+0000 7fc155ff7740 1 --2- 172.21.15.31:0/2702358991 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c410090 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.934+0000 7fc155ff7740 1 --2- 172.21.15.31:0/2702358991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c40fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.934+0000 7fc155ff7740 1 --2- 172.21.15.31:0/2702358991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c40f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.934+0000 7fc155ff7740 1 -- 172.21.15.31:0/2702358991 >> 172.21.15.31:0/2702358991 conn(0x560d2c0674e0 msgr2=0x560d2c422be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:49.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.935+0000 7fc155ff7740 1 -- 172.21.15.31:0/2702358991 shutdown_connections 2024-04-25T16:09:49.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.935+0000 7fc155ff7740 1 -- 172.21.15.31:0/2702358991 wait complete. 2024-04-25T16:09:49.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.936+0000 7fc155ff7740 1 -- start start 2024-04-25T16:09:49.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.936+0000 7fc155ff7740 1 Processor -- start 2024-04-25T16:09:49.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.937+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c44cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.937+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c44d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.937+0000 7fc155ff7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c44df50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.938+0000 7fc155ff7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc13400f910 con 0x560d2c0073a0 2024-04-25T16:09:49.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.938+0000 7fc155ff7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc13400f240 con 0x560d2c04a4d0 2024-04-25T16:09:49.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.938+0000 7fc155ff7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc13400c480 con 0x560d2c073390 2024-04-25T16:09:49.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.938+0000 7fc155ff7740 10 client.0 _ll_register_callbacks cb 0x7fc13800b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:49.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.939+0000 7fc14cee2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c44cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.939+0000 7fc14cee2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c44cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60318/0 (socket says 172.21.15.31:60318) 2024-04-25T16:09:49.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.939+0000 7fc14cee2640 1 -- 172.21.15.31:0/1604428135 learned_addr learned my addr 172.21.15.31:0/1604428135 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:49.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.940+0000 7fc14cee2640 1 -- 172.21.15.31:0/1604428135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 msgr2=0x560d2c44df50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:49.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.940+0000 7fc14cee2640 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c44df50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.940+0000 7fc14cee2640 1 -- 172.21.15.31:0/1604428135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 msgr2=0x560d2c44d020 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:49.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.940+0000 7fc14cee2640 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c44d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.940+0000 7fc14cee2640 1 -- 172.21.15.31:0/1604428135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d2c44e510 con 0x560d2c0073a0 2024-04-25T16:09:49.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.940+0000 7fc14cee2640 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c44cae0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x560d2c43e390 tx=0x7fc14000bce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5886685e5ea206 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:49.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.941+0000 7fc127fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:49.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.941+0000 7fc127fff640 1 -- 172.21.15.31:0/1604428135 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc140008c00 con 0x560d2c0073a0 2024-04-25T16:09:49.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.942+0000 7fc155ff7740 1 -- 172.21.15.31:0/1604428135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc138018070 con 0x560d2c0073a0 2024-04-25T16:09:49.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.942+0000 7fc127fff640 1 -- 172.21.15.31:0/1604428135 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc140008da0 con 0x560d2c0073a0 2024-04-25T16:09:49.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.942+0000 7fc127fff640 1 -- 172.21.15.31:0/1604428135 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc140019430 con 0x560d2c0073a0 2024-04-25T16:09:49.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.943+0000 7fc155ff7740 10 client.17310 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:49.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.943+0000 7fc155ff7740 1 -- 172.21.15.31:0/1604428135 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560d2c4a5c40 con 0x560d2c0073a0 2024-04-25T16:09:49.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.946+0000 7fc127fff640 1 -- 172.21.15.31:0/1604428135 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc138018070 con 0x560d2c0073a0 2024-04-25T16:09:49.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.947+0000 7fc127fff640 1 -- 172.21.15.31:0/1604428135 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc140022ae0 con 0x560d2c0073a0 2024-04-25T16:09:49.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.952+0000 7fc155ff7740 10 client.17310 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:49.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.952+0000 7fc155ff7740 10 client.17310 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:09:49.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.952+0000 7fc155ff7740 20 client.17310 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:49.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.952+0000 7fc155ff7740 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x560d2c4a64b0 0x560d2c4a88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:49.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.952+0000 7fc155ff7740 4 client.17310 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:49.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.952+0000 7fc155ff7740 1 -- 172.21.15.31:0/1604428135 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d2c381800 con 0x560d2c4a64b0 2024-04-25T16:09:49.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.964+0000 7fc147fff640 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x560d2c4a64b0 0x560d2c4a88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:49.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.971+0000 7fc147fff640 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x560d2c4a64b0 0x560d2c4a88d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cb661d2c0c7553d8 server_cookie=a9a6fa89fc7bc51a in_seq=0 out_seq=0 2024-04-25T16:09:49.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.971+0000 7fc127fff640 10 client.17310 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:09:49.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.972+0000 7fc127fff640 1 -- 172.21.15.31:0/1604428135 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d2c381800 con 0x560d2c4a64b0 2024-04-25T16:09:49.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.972+0000 7fc127fff640 10 client.17310 handle_command_reply: tid=0 2024-04-25T16:09:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.974+0000 7fc155ff7740 1 client.17310 shutdown 2024-04-25T16:09:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.974+0000 7fc155ff7740 1 -- 172.21.15.31:0/1604428135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 msgr2=0x560d2c44cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.974+0000 7fc155ff7740 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c44cae0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x560d2c43e390 tx=0x7fc14000bce0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.975+0000 7fc155ff7740 1 -- 172.21.15.31:0/1604428135 shutdown_connections 2024-04-25T16:09:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.975+0000 7fc155ff7740 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560d2c073390 0x560d2c44df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.975+0000 7fc155ff7740 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560d2c04a4d0 0x560d2c44d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.975+0000 7fc155ff7740 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x560d2c4a64b0 0x560d2c4a88d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.975+0000 7fc155ff7740 1 --2- 172.21.15.31:0/1604428135 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560d2c0073a0 0x560d2c44cae0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.975+0000 7fc155ff7740 1 -- 172.21.15.31:0/1604428135 >> 172.21.15.31:0/1604428135 conn(0x560d2c0674e0 msgr2=0x560d2c413e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:49.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.975+0000 7fc127fff640 0 client.17310 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:09:49.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.975+0000 7fc155ff7740 1 -- 172.21.15.31:0/1604428135 shutdown_connections 2024-04-25T16:09:49.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.979+0000 7fc155ff7740 1 -- 172.21.15.31:0/1604428135 wait complete. 2024-04-25T16:09:49.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.979+0000 7fc155ff7740 20 client.17310 trim_cache size 0 max 16384 2024-04-25T16:09:49.982 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:49.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.984+0000 7fc14f146640 1 -- 172.21.15.31:0/3261096840 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1300afd00 msgr2=0x7fc1300b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.984+0000 7fc14f146640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1300afd00 0x7fc1300b2210 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x560d2c41f070 tx=0x7fc1380046c0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.984+0000 7fc14f146640 1 -- 172.21.15.31:0/3261096840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14810f760 msgr2=0x7fc14810b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:49.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.984+0000 7fc14f146640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14810f760 0x7fc14810b660 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fc13c035040 tx=0x7fc13c032cd0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.988+0000 7fc14f146640 1 -- 172.21.15.31:0/3261096840 shutdown_connections 2024-04-25T16:09:49.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.989+0000 7fc14f146640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc1300afd00 0x7fc1300b2210 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.989+0000 7fc14f146640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc1481120c0 0x7fc14810bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.989+0000 7fc14f146640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc14810f760 0x7fc14810b660 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.989+0000 7fc14f146640 1 --2- 172.21.15.31:0/3261096840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc148109fc0 0x7fc14810cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:49.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.989+0000 7fc14f146640 1 -- 172.21.15.31:0/3261096840 >> 172.21.15.31:0/3261096840 conn(0x7fc1480704e0 msgr2=0x7fc148110740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:49.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.991+0000 7fc14f146640 1 -- 172.21.15.31:0/3261096840 shutdown_connections 2024-04-25T16:09:49.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:49.995+0000 7fc14f146640 1 -- 172.21.15.31:0/3261096840 wait complete. 2024-04-25T16:09:50.041 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:09:50.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.412+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/4152531205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0b8109f40 msgr2=0x7ff0b810ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:50.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.412+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/4152531205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0b8109f40 0x7ff0b810ac40 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009860 tx=0x7ff0a802a340 comp rx=0 tx=0).stop 2024-04-25T16:09:50.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.412+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/4152531205 shutdown_connections 2024-04-25T16:09:50.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.412+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/4152531205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b810fe80 0x7ff0b81122a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.412+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/4152531205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0b810b180 0x7ff0b810b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.412+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/4152531205 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0b8109f40 0x7ff0b810ac40 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.412+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/4152531205 >> 172.21.15.31:0/4152531205 conn(0x7ff0b8078b70 msgr2=0x7ff0b81079c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:50.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.415+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/4152531205 shutdown_connections 2024-04-25T16:09:50.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.419+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/4152531205 wait complete. 2024-04-25T16:09:50.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.420+0000 7ff0bdbb1640 1 Processor -- start 2024-04-25T16:09:50.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.420+0000 7ff0bdbb1640 1 -- start start 2024-04-25T16:09:50.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.420+0000 7ff0bdbb1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0b8109f40 0x7ff0b80743a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.420+0000 7ff0bdbb1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0b810b180 0x7ff0b8072a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.420+0000 7ff0bdbb1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b810fe80 0x7ff0b8072f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.420+0000 7ff0bdbb1640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b806cde0 con 0x7ff0b8109f40 2024-04-25T16:09:50.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.420+0000 7ff0bdbb1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b806cc60 con 0x7ff0b810b180 2024-04-25T16:09:50.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.420+0000 7ff0bdbb1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b806cf60 con 0x7ff0b810fe80 2024-04-25T16:09:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.421+0000 7ff0bd3b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b810fe80 0x7ff0b8072f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.421+0000 7ff0bd3b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b810fe80 0x7ff0b8072f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47938/0 (socket says 172.21.15.31:47938) 2024-04-25T16:09:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.421+0000 7ff0bd3b0640 1 -- 172.21.15.31:0/1785418637 learned_addr learned my addr 172.21.15.31:0/1785418637 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.422+0000 7ff0bd3b0640 1 -- 172.21.15.31:0/1785418637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0b810b180 msgr2=0x7ff0b8072a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:09:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.422+0000 7ff0bd3b0640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0b810b180 0x7ff0b8072a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.422+0000 7ff0bd3b0640 1 -- 172.21.15.31:0/1785418637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0b8109f40 msgr2=0x7ff0b80743a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.422+0000 7ff0bd3b0640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0b8109f40 0x7ff0b80743a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.422+0000 7ff0bd3b0640 1 -- 172.21.15.31:0/1785418637 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b80734c0 con 0x7ff0b810fe80 2024-04-25T16:09:50.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.423+0000 7ff0bd3b0640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b810fe80 0x7ff0b8072f80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00d750 tx=0x7ff0ac00dc10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:50.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.428+0000 7ff0b5ffb640 1 -- 172.21.15.31:0/1785418637 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac017070 con 0x7ff0b810fe80 2024-04-25T16:09:50.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.428+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/1785418637 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b80737b0 con 0x7ff0b810fe80 2024-04-25T16:09:50.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.429+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/1785418637 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b807b630 con 0x7ff0b810fe80 2024-04-25T16:09:50.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.429+0000 7ff0b5ffb640 1 -- 172.21.15.31:0/1785418637 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0ac0054d0 con 0x7ff0b810fe80 2024-04-25T16:09:50.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.429+0000 7ff0b5ffb640 1 -- 172.21.15.31:0/1785418637 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac004cb0 con 0x7ff0b810fe80 2024-04-25T16:09:50.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.433+0000 7ff0c5da0740 1 Processor -- start 2024-04-25T16:09:50.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.435+0000 7ff0c5da0740 1 -- start start 2024-04-25T16:09:50.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.435+0000 7ff0b5ffb640 1 -- 172.21.15.31:0/1785418637 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff0ac010460 con 0x7ff0b810fe80 2024-04-25T16:09:50.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.435+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca36a29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.435+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32d74d0 0x560ca36a2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.435+0000 7ff0b7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca36a29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.435+0000 7ff0b7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca36a29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47954/0 (socket says 172.21.15.31:47954) 2024-04-25T16:09:50.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.436+0000 7ff0b7fff640 1 -- 172.21.15.31:0/391823872 learned_addr learned my addr 172.21.15.31:0/391823872 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:50.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.436+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 0x560ca36a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.436+0000 7ff0c5da0740 1 -- 172.21.15.31:0/391823872 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560ca3606d80 con 0x560ca32d74d0 2024-04-25T16:09:50.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.436+0000 7ff0b5ffb640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff0a00afcf0 0x7ff0a00b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.436+0000 7ff0b5ffb640 1 -- 172.21.15.31:0/1785418637 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff0ac0de640 con 0x7ff0b810fe80 2024-04-25T16:09:50.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.437+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32d74d0 0x560ca36a2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.437+0000 7ff0c5da0740 1 -- 172.21.15.31:0/391823872 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560ca35c0a20 con 0x560ca32df6b0 2024-04-25T16:09:50.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.437+0000 7ff0c5da0740 1 -- 172.21.15.31:0/391823872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560ca3602d40 con 0x560ca3300390 2024-04-25T16:09:50.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.437+0000 7ff0bd3b0640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff0a00afcf0 0x7ff0a00b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.437+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 0x560ca36a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.438+0000 7ff0b7fff640 1 -- 172.21.15.31:0/391823872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 msgr2=0x560ca36a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:50.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.438+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 0x560ca36a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.438+0000 7ff0b7fff640 1 -- 172.21.15.31:0/391823872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32d74d0 msgr2=0x560ca36a2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:50.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.438+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32d74d0 0x560ca36a2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.438+0000 7ff0b7fff640 1 -- 172.21.15.31:0/391823872 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ca35cbc30 con 0x560ca3300390 2024-04-25T16:09:50.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.438+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32d74d0 0x560ca36a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:09:50.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.438+0000 7ff0bd3b0640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff0a00afcf0 0x7ff0a00b2200 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0a3d40 tx=0x7ff0ac005f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:50.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.438+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca36a29a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0015690 tx=0x7ff0b00099d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea6d4034ab732f46 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:50.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.439+0000 7ff09effd640 1 -- 172.21.15.31:0/391823872 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0004b20 con 0x560ca3300390 2024-04-25T16:09:50.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.439+0000 7ff09effd640 1 -- 172.21.15.31:0/391823872 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0b0004cc0 con 0x560ca3300390 2024-04-25T16:09:50.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.439+0000 7ff09effd640 1 -- 172.21.15.31:0/391823872 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b000e6e0 con 0x560ca3300390 2024-04-25T16:09:50.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.440+0000 7ff0c5da0740 1 -- 172.21.15.31:0/391823872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 msgr2=0x560ca36a29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:50.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.440+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca36a29a0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0015690 tx=0x7ff0b00099d0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.440+0000 7ff0c5da0740 1 -- 172.21.15.31:0/391823872 shutdown_connections 2024-04-25T16:09:50.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.440+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 0x560ca36a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.440+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32d74d0 0x560ca36a2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.440+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/391823872 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca36a29a0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.440+0000 7ff0c5da0740 1 -- 172.21.15.31:0/391823872 >> 172.21.15.31:0/391823872 conn(0x560ca32f44e0 msgr2=0x560ca369e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:50.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.440+0000 7ff0c5da0740 1 -- 172.21.15.31:0/391823872 shutdown_connections 2024-04-25T16:09:50.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.441+0000 7ff0c5da0740 1 -- 172.21.15.31:0/391823872 wait complete. 2024-04-25T16:09:50.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.441+0000 7ff0c5da0740 1 -- start start 2024-04-25T16:09:50.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.441+0000 7ff0c5da0740 1 Processor -- start 2024-04-25T16:09:50.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.442+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca36ac4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.443+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 0x560ca36aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.443+0000 7ff0b7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca36ac4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.443+0000 7ff0b7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca36ac4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47958/0 (socket says 172.21.15.31:47958) 2024-04-25T16:09:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.443+0000 7ff0b7fff640 1 -- 172.21.15.31:0/3200206423 learned_addr learned my addr 172.21.15.31:0/3200206423 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.443+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 0x560ca36aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.443+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca3300390 0x560ca36b3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.443+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560ca36a46c0 con 0x560ca3300390 2024-04-25T16:09:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.443+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560ca358ddc0 con 0x560ca32df6b0 2024-04-25T16:09:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.444+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560ca36a4840 con 0x560ca32d74d0 2024-04-25T16:09:50.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.444+0000 7ff0b7fff640 1 -- 172.21.15.31:0/3200206423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 msgr2=0x560ca36aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:50.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.444+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 0x560ca36aca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.444+0000 7ff0b7fff640 1 -- 172.21.15.31:0/3200206423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca3300390 msgr2=0x560ca36b3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:50.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.444+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca3300390 0x560ca36b3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.444+0000 7ff0b7fff640 1 -- 172.21.15.31:0/3200206423 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ca33183f0 con 0x560ca32d74d0 2024-04-25T16:09:50.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.445+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca36ac4e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009500 tx=0x7ff0b0005c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7abc0528b0ff3efe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:50.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.445+0000 7ff0c5da0740 10 client.0 _ll_register_callbacks cb 0x560ca35a9600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:50.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.445+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0b0004cc0 con 0x560ca32d74d0 2024-04-25T16:09:50.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.446+0000 7ff09c7f8640 10 client.26977 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:50.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.446+0000 7ff09c7f8640 1 -- 172.21.15.31:0/3200206423 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0024020 con 0x560ca32d74d0 2024-04-25T16:09:50.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.446+0000 7ff09c7f8640 1 -- 172.21.15.31:0/3200206423 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff0b0004ef0 con 0x560ca32d74d0 2024-04-25T16:09:50.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.447+0000 7ff09c7f8640 1 -- 172.21.15.31:0/3200206423 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff0b00050f0 con 0x560ca32d74d0 2024-04-25T16:09:50.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.447+0000 7ff0c5da0740 10 client.26977 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:50.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.447+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=29}) -- 0x560ca33e27d0 con 0x560ca32d74d0 2024-04-25T16:09:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.447+0000 7ff09c7f8640 1 -- 172.21.15.31:0/3200206423 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ff0b000c070 con 0x560ca32d74d0 2024-04-25T16:09:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.447+0000 7ff0c5da0740 10 client.26977 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.448+0000 7ff0c5da0740 10 client.26977 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.448+0000 7ff0c5da0740 20 client.26977 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.448+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560ca32bef60 0x560ca36b9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.448+0000 7ff0c5da0740 4 client.26977 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:50.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.448+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff0b0004cc0 con 0x560ca32bef60 2024-04-25T16:09:50.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.448+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560ca32bef60 0x560ca36b9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.449+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560ca32bef60 0x560ca36b9bf0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=264716d6713d0811 server_cookie=f448d49961dbcd95 in_seq=0 out_seq=0 2024-04-25T16:09:50.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.449+0000 7ff09c7f8640 10 client.26977 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.453+0000 7ff09c7f8640 1 -- 172.21.15.31:0/3200206423 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7ff0b0004cc0 con 0x560ca32bef60 2024-04-25T16:09:50.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.453+0000 7ff09c7f8640 10 client.26977 handle_command_reply: tid=0 2024-04-25T16:09:50.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.455+0000 7ff0c5da0740 1 client.26977 shutdown 2024-04-25T16:09:50.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.455+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 msgr2=0x560ca36ac4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:50.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.455+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca36ac4e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009500 tx=0x7ff0b0005c70 comp rx=0 tx=0).stop 2024-04-25T16:09:50.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.455+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 shutdown_connections 2024-04-25T16:09:50.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.455+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560ca32bef60 0x560ca36b9bf0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.455+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca3300390 0x560ca36b3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.455+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32df6b0 0x560ca36aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.455+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/3200206423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca36ac4e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.455+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 >> 172.21.15.31:0/3200206423 conn(0x560ca32f44e0 msgr2=0x560ca369d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:50.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.462+0000 7ff09c7f8640 0 client.26977 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:50.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.462+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 shutdown_connections 2024-04-25T16:09:50.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.462+0000 7ff0c5da0740 1 -- 172.21.15.31:0/3200206423 wait complete. 2024-04-25T16:09:50.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.463+0000 7ff0c5da0740 20 client.26977 trim_cache size 0 max 16384 2024-04-25T16:09:50.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.503+0000 7ff0c5da0740 1 Processor -- start 2024-04-25T16:09:50.505 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.504+0000 7ff0c5da0740 1 -- start start 2024-04-25T16:09:50.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.504+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca369db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.505+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 0x560ca36b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.506 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.505+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca3300390 0x560ca369e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.505+0000 7ff0c5da0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b81c2740 con 0x560ca32df6b0 2024-04-25T16:09:50.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.505+0000 7ff0c5da0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b81c4280 con 0x560ca3300390 2024-04-25T16:09:50.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.505+0000 7ff0c5da0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b81c2e10 con 0x560ca32d74d0 2024-04-25T16:09:50.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.506+0000 7ff0b7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca369db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.508+0000 7ff0b7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca369db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47966/0 (socket says 172.21.15.31:47966) 2024-04-25T16:09:50.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.508+0000 7ff0b7fff640 1 -- 172.21.15.31:0/2163482324 learned_addr learned my addr 172.21.15.31:0/2163482324 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:50.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.508+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/2163482324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca3300390 0x560ca369e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.509+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/2163482324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 0x560ca36b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.510+0000 7ff0b7fff640 1 -- 172.21.15.31:0/2163482324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca3300390 msgr2=0x560ca369e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:50.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.510+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/2163482324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca3300390 0x560ca369e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.510+0000 7ff0b7fff640 1 -- 172.21.15.31:0/2163482324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 msgr2=0x560ca36b3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:50.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.510+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/2163482324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 0x560ca36b3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.511 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.510+0000 7ff0b7fff640 1 -- 172.21.15.31:0/2163482324 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ca36a4060 con 0x560ca32d74d0 2024-04-25T16:09:50.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.513+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/2163482324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca369db20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0005d70 tx=0x7ff0b0020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89382ee96a41644b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:50.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.516+0000 7ff09e7fc640 1 -- 172.21.15.31:0/2163482324 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0005940 con 0x560ca32d74d0 2024-04-25T16:09:50.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.516+0000 7ff09e7fc640 1 -- 172.21.15.31:0/2163482324 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0b0005400 con 0x560ca32d74d0 2024-04-25T16:09:50.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.516+0000 7ff09e7fc640 1 -- 172.21.15.31:0/2163482324 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00047e0 con 0x560ca32d74d0 2024-04-25T16:09:50.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.518+0000 7ff0c5da0740 1 -- 172.21.15.31:0/2163482324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 msgr2=0x560ca369db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:50.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.518+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/2163482324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca369db20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0005d70 tx=0x7ff0b0020000 comp rx=0 tx=0).stop 2024-04-25T16:09:50.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.521+0000 7ff0c5da0740 1 -- 172.21.15.31:0/2163482324 shutdown_connections 2024-04-25T16:09:50.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.521+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/2163482324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca3300390 0x560ca369e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.521+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/2163482324 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 0x560ca36b3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.521+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/2163482324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca32d74d0 0x560ca369db20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.521+0000 7ff0c5da0740 1 -- 172.21.15.31:0/2163482324 >> 172.21.15.31:0/2163482324 conn(0x560ca32f44e0 msgr2=0x560ca36a2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:50.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.527+0000 7ff0c5da0740 1 -- 172.21.15.31:0/2163482324 shutdown_connections 2024-04-25T16:09:50.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.527+0000 7ff0c5da0740 1 -- 172.21.15.31:0/2163482324 wait complete. 2024-04-25T16:09:50.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.528+0000 7ff0c5da0740 1 -- start start 2024-04-25T16:09:50.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.528+0000 7ff0c5da0740 1 Processor -- start 2024-04-25T16:09:50.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.529+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32d74d0 0x560ca36b0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.530+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 0x560ca36b0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.530+0000 7ff0c5da0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca371fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.530+0000 7ff0c5da0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b81c3bb0 con 0x560ca32df6b0 2024-04-25T16:09:50.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.530+0000 7ff0c5da0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b81c34e0 con 0x560ca32d74d0 2024-04-25T16:09:50.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.530+0000 7ff0c5da0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b8082330 con 0x560ca3300390 2024-04-25T16:09:50.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.531+0000 7ff0c5da0740 10 client.0 _ll_register_callbacks cb 0x7ff0a8035070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.533+0000 7ff0b7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca371fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.533+0000 7ff0b7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32d74d0 0x560ca36b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.533+0000 7ff0b7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32d74d0 0x560ca36b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58404/0 (socket says 172.21.15.31:58404) 2024-04-25T16:09:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.533+0000 7ff0b7fff640 1 -- 172.21.15.31:0/1391711963 learned_addr learned my addr 172.21.15.31:0/1391711963 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:50.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.535+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 0x560ca36b0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.537+0000 7ff0bcbaf640 1 -- 172.21.15.31:0/1391711963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 msgr2=0x560ca371fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:50.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.537+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca371fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.537+0000 7ff0bcbaf640 1 -- 172.21.15.31:0/1391711963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32d74d0 msgr2=0x560ca36b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.537+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32d74d0 0x560ca36b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.537+0000 7ff0bcbaf640 1 -- 172.21.15.31:0/1391711963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ca36b1400 con 0x560ca32df6b0 2024-04-25T16:09:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.538+0000 7ff0bcbaf640 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 0x560ca36b0d90 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009780 tx=0x7ff0a8030e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=def5afecac0e96b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:50.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.538+0000 7ff09c7f8640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.539+0000 7ff09c7f8640 1 -- 172.21.15.31:0/1391711963 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8037040 con 0x560ca32df6b0 2024-04-25T16:09:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.539+0000 7ff0c5da0740 1 -- 172.21.15.31:0/1391711963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0b0005400 con 0x560ca32df6b0 2024-04-25T16:09:50.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.540+0000 7ff09c7f8640 1 -- 172.21.15.31:0/1391711963 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ff0a8030750 con 0x560ca32df6b0 2024-04-25T16:09:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.540+0000 7ff09c7f8640 1 -- 172.21.15.31:0/1391711963 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff0a8030990 con 0x560ca32df6b0 2024-04-25T16:09:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.541+0000 7ff0c5da0740 10 client.17322 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:50.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.541+0000 7ff0c5da0740 1 -- 172.21.15.31:0/1391711963 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560ca3721570 con 0x560ca32df6b0 2024-04-25T16:09:50.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.542+0000 7ff09c7f8640 1 -- 172.21.15.31:0/1391711963 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0b0005400 con 0x560ca32df6b0 2024-04-25T16:09:50.545 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.544+0000 7ff09c7f8640 1 -- 172.21.15.31:0/1391711963 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ff0a8003cc0 con 0x560ca32df6b0 2024-04-25T16:09:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.545+0000 7ff0c5da0740 10 client.17322 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.545+0000 7ff0c5da0740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.545+0000 7ff0c5da0740 20 client.17322 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.545+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560ca32943a0 0x560ca3723e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:50.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.545+0000 7ff0c5da0740 4 client.17322 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:09:50.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.545+0000 7ff0c5da0740 1 -- 172.21.15.31:0/1391711963 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560ca3614990 con 0x560ca32943a0 2024-04-25T16:09:50.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.551+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560ca32943a0 0x560ca3723e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.563+0000 7ff0b7fff640 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560ca32943a0 0x560ca3723e80 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=407f3de04172cb81 server_cookie=16d5383eb56ad849 in_seq=0 out_seq=0 2024-04-25T16:09:50.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.563+0000 7ff09c7f8640 10 client.17322 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:50.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.567+0000 7ff09c7f8640 1 -- 172.21.15.31:0/1391711963 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560ca3614990 con 0x560ca32943a0 2024-04-25T16:09:50.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.567+0000 7ff09c7f8640 10 client.17322 handle_command_reply: tid=0 2024-04-25T16:09:50.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.567+0000 7ff0c5da0740 1 client.17322 shutdown 2024-04-25T16:09:50.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.569+0000 7ff0c5da0740 1 -- 172.21.15.31:0/1391711963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 msgr2=0x560ca36b0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:50.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.569+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 0x560ca36b0d90 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009780 tx=0x7ff0a8030e00 comp rx=0 tx=0).stop 2024-04-25T16:09:50.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff0c5da0740 1 -- 172.21.15.31:0/1391711963 shutdown_connections 2024-04-25T16:09:50.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x560ca32943a0 0x560ca3723e80 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x560ca3300390 0x560ca371fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x560ca32df6b0 0x560ca36b0d90 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff0c5da0740 1 --2- 172.21.15.31:0/1391711963 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x560ca32d74d0 0x560ca36b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff0c5da0740 1 -- 172.21.15.31:0/1391711963 >> 172.21.15.31:0/1391711963 conn(0x560ca32f44e0 msgr2=0x560ca369db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff09c7f8640 0 client.17322 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff0c5da0740 1 -- 172.21.15.31:0/1391711963 shutdown_connections 2024-04-25T16:09:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff0c5da0740 1 -- 172.21.15.31:0/1391711963 wait complete. 2024-04-25T16:09:50.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.575+0000 7ff0c5da0740 20 client.17322 trim_cache size 0 max 16384 2024-04-25T16:09:50.578 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:09:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.578+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/1785418637 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff0a00afcf0 msgr2=0x7ff0a00b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.578+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff0a00afcf0 0x7ff0a00b2200 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac0a3d40 tx=0x7ff0ac005f10 comp rx=0 tx=0).stop 2024-04-25T16:09:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.579+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/1785418637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b810fe80 msgr2=0x7ff0b8072f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:50.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.579+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b810fe80 0x7ff0b8072f80 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac00d750 tx=0x7ff0ac00dc10 comp rx=0 tx=0).stop 2024-04-25T16:09:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.583+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/1785418637 shutdown_connections 2024-04-25T16:09:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.583+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ff0a00afcf0 0x7ff0a00b2200 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.583+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0b810fe80 0x7ff0b8072f80 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.583+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff0b810b180 0x7ff0b8072a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.583+0000 7ff0bdbb1640 1 --2- 172.21.15.31:0/1785418637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ff0b8109f40 0x7ff0b80743a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:50.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.583+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/1785418637 >> 172.21.15.31:0/1785418637 conn(0x7ff0b8078b70 msgr2=0x7ff0b810be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:50.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.589+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/1785418637 shutdown_connections 2024-04-25T16:09:50.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:50.590+0000 7ff0bdbb1640 1 -- 172.21.15.31:0/1785418637 wait complete. 2024-04-25T16:09:51.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:51 smithi063 ceph-mon[32413]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 796 MiB/s wr, 2.04k op/s 2024-04-25T16:09:51.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:51 smithi186 ceph-mon[33258]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 796 MiB/s wr, 2.04k op/s 2024-04-25T16:09:51.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:51 smithi031 ceph-mon[29140]: pgmap v607: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 796 MiB/s wr, 2.04k op/s 2024-04-25T16:09:51.662 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:09:51.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:09:51.663 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:09:52.150 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:52 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:09:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.224+0000 7fc63bfff640 1 -- 172.21.15.31:0/580654162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc63c10fef0 msgr2=0x7fc63c110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.224+0000 7fc63bfff640 1 --2- 172.21.15.31:0/580654162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc63c10fef0 0x7fc63c110300 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009900 tx=0x7fc62c02a410 comp rx=0 tx=0).stop 2024-04-25T16:09:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.226+0000 7fc63bfff640 1 -- 172.21.15.31:0/580654162 shutdown_connections 2024-04-25T16:09:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.226+0000 7fc63bfff640 1 --2- 172.21.15.31:0/580654162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc63c10fef0 0x7fc63c110300 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.226+0000 7fc63bfff640 1 --2- 172.21.15.31:0/580654162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc63c10b1d0 0x7fc63c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.226+0000 7fc63bfff640 1 --2- 172.21.15.31:0/580654162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c10a350 0x7fc63c10ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.226+0000 7fc63bfff640 1 -- 172.21.15.31:0/580654162 >> 172.21.15.31:0/580654162 conn(0x7fc63c070400 msgr2=0x7fc63c072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:52.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.229+0000 7fc63bfff640 1 -- 172.21.15.31:0/580654162 shutdown_connections 2024-04-25T16:09:52.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.233+0000 7fc63bfff640 1 -- 172.21.15.31:0/580654162 wait complete. 2024-04-25T16:09:52.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.235+0000 7fc63bfff640 1 Processor -- start 2024-04-25T16:09:52.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.235+0000 7fc63bfff640 1 -- start start 2024-04-25T16:09:52.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.235+0000 7fc63bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc63c10a350 0x7fc63c119030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.236+0000 7fc63bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc63c10b1d0 0x7fc63c119570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.236+0000 7fc63bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c10fef0 0x7fc63c11a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.236+0000 7fc63bfff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c11e240 con 0x7fc63c10b1d0 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.236+0000 7fc63bfff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c11e0c0 con 0x7fc63c10a350 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.236+0000 7fc63bfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc63c11e3c0 con 0x7fc63c10fef0 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.236+0000 7fc63b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c10fef0 0x7fc63c11a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.236+0000 7fc63b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c10fef0 0x7fc63c11a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:47988/0 (socket says 172.21.15.31:47988) 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.236+0000 7fc63b7fe640 1 -- 172.21.15.31:0/2875540375 learned_addr learned my addr 172.21.15.31:0/2875540375 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.239+0000 7fc63b7fe640 1 -- 172.21.15.31:0/2875540375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc63c10a350 msgr2=0x7fc63c119030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.239+0000 7fc63b7fe640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc63c10a350 0x7fc63c119030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.239+0000 7fc63b7fe640 1 -- 172.21.15.31:0/2875540375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc63c10b1d0 msgr2=0x7fc63c119570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.239+0000 7fc63b7fe640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc63c10b1d0 0x7fc63c119570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.239+0000 7fc63b7fe640 1 -- 172.21.15.31:0/2875540375 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc63c11ab40 con 0x7fc63c10fef0 2024-04-25T16:09:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.240+0000 7fc63b7fe640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c10fef0 0x7fc63c11a4d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009850 tx=0x7fc62c0053c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.241+0000 7fc640984640 1 -- 172.21.15.31:0/2875540375 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c040030 con 0x7fc63c10fef0 2024-04-25T16:09:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.241+0000 7fc63bfff640 1 -- 172.21.15.31:0/2875540375 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc63c11add0 con 0x7fc63c10fef0 2024-04-25T16:09:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.241+0000 7fc63bfff640 1 -- 172.21.15.31:0/2875540375 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc63c11b340 con 0x7fc63c10fef0 2024-04-25T16:09:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.241+0000 7fc640984640 1 -- 172.21.15.31:0/2875540375 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc62c0055f0 con 0x7fc63c10fef0 2024-04-25T16:09:52.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.241+0000 7fc640984640 1 -- 172.21.15.31:0/2875540375 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0319a0 con 0x7fc63c10fef0 2024-04-25T16:09:52.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.246+0000 7fc640984640 1 -- 172.21.15.31:0/2875540375 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc62c043030 con 0x7fc63c10fef0 2024-04-25T16:09:52.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.249+0000 7fc640984640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6240afd00 0x7fc6240b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.250+0000 7fc640984640 1 -- 172.21.15.31:0/2875540375 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc62c100300 con 0x7fc63c10fef0 2024-04-25T16:09:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.253+0000 7fc63affd640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6240afd00 0x7fc6240b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.256+0000 7fc64883f740 1 Processor -- start 2024-04-25T16:09:52.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.257+0000 7fc64883f740 1 -- start start 2024-04-25T16:09:52.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.257+0000 7fc64883f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1518b410 0x55cd1552c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.257+0000 7fc64883f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd15162550 0x55cd1552cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.257+0000 7fc64883f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd15533b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.257+0000 7fc64883f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cd15491b00 con 0x55cd1516a730 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.257+0000 7fc64883f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cd1542d920 con 0x55cd1518b410 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.257+0000 7fc64883f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cd1548dac0 con 0x55cd15162550 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.257+0000 7fc63affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1518b410 0x55cd1552c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.258+0000 7fc63affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1518b410 0x55cd1552c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58444/0 (socket says 172.21.15.31:58444) 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.258+0000 7fc63affd640 1 -- 172.21.15.31:0/3947596983 learned_addr learned my addr 172.21.15.31:0/3947596983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.258+0000 7fc63affd640 1 -- 172.21.15.31:0/3947596983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd15162550 msgr2=0x55cd1552cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.258+0000 7fc63affd640 1 --2- 172.21.15.31:0/3947596983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd15162550 0x55cd1552cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.258+0000 7fc63affd640 1 -- 172.21.15.31:0/3947596983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 msgr2=0x55cd15533b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.258+0000 7fc63affd640 1 --2- 172.21.15.31:0/3947596983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd15533b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.258+0000 7fc63affd640 1 -- 172.21.15.31:0/3947596983 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd1526b7e0 con 0x55cd1518b410 2024-04-25T16:09:52.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.259+0000 7fc63affd640 1 --2- 172.21.15.31:0/3947596983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1518b410 0x55cd1552c6a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc63400efd0 tx=0x7fc634004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee7b53a90b79709c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.266+0000 7fc621ffb640 1 -- 172.21.15.31:0/3947596983 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63401f070 con 0x55cd1518b410 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.266+0000 7fc621ffb640 1 -- 172.21.15.31:0/3947596983 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc63401bab0 con 0x55cd1518b410 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.266+0000 7fc621ffb640 1 -- 172.21.15.31:0/3947596983 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc63400f9d0 con 0x55cd1518b410 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.267+0000 7fc64883f740 1 -- 172.21.15.31:0/3947596983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1518b410 msgr2=0x55cd1552c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.267+0000 7fc64883f740 1 --2- 172.21.15.31:0/3947596983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1518b410 0x55cd1552c6a0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fc63400efd0 tx=0x7fc634004a40 comp rx=0 tx=0).stop 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.267+0000 7fc63affd640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6240afd00 0x7fc6240b2210 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc634005c40 tx=0x7fc634005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.267+0000 7fc64883f740 1 -- 172.21.15.31:0/3947596983 shutdown_connections 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.267+0000 7fc64883f740 1 --2- 172.21.15.31:0/3947596983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd15533b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.267+0000 7fc64883f740 1 --2- 172.21.15.31:0/3947596983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd15162550 0x55cd1552cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.267+0000 7fc64883f740 1 --2- 172.21.15.31:0/3947596983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1518b410 0x55cd1552c6a0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.267+0000 7fc64883f740 1 -- 172.21.15.31:0/3947596983 >> 172.21.15.31:0/3947596983 conn(0x55cd1517f560 msgr2=0x55cd15528030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:52.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.268+0000 7fc64883f740 1 -- 172.21.15.31:0/3947596983 shutdown_connections 2024-04-25T16:09:52.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.268+0000 7fc64883f740 1 -- 172.21.15.31:0/3947596983 wait complete. 2024-04-25T16:09:52.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.268+0000 7fc64883f740 1 -- start start 2024-04-25T16:09:52.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.269+0000 7fc64883f740 1 Processor -- start 2024-04-25T16:09:52.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.269+0000 7fc64883f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15536550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.269+0000 7fc64883f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd15536a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.269+0000 7fc64883f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.269+0000 7fc64883f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cd15534570 con 0x55cd1516a730 2024-04-25T16:09:52.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.269+0000 7fc64883f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cd1542b5d0 con 0x55cd15162550 2024-04-25T16:09:52.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.269+0000 7fc64883f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cd155346f0 con 0x55cd1518b410 2024-04-25T16:09:52.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.269+0000 7fc64883f740 10 client.0 _ll_register_callbacks cb 0x55cd15419080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:52.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.270+0000 7fc63b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.270+0000 7fc63b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48018/0 (socket says 172.21.15.31:48018) 2024-04-25T16:09:52.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.270+0000 7fc63b7fe640 1 -- 172.21.15.31:0/2032848511 learned_addr learned my addr 172.21.15.31:0/2032848511 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:52.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.271+0000 7fc63affd640 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15536550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.271+0000 7fc63a7fc640 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd15536a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.271+0000 7fc63b7fe640 1 -- 172.21.15.31:0/2032848511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 msgr2=0x55cd15536550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.271+0000 7fc63b7fe640 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15536550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.271+0000 7fc63b7fe640 1 -- 172.21.15.31:0/2032848511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 msgr2=0x55cd15536a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.271+0000 7fc63b7fe640 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd15536a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.271+0000 7fc63b7fe640 1 -- 172.21.15.31:0/2032848511 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd152cbf70 con 0x55cd1518b410 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.272+0000 7fc63b7fe640 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553d9f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0c4d30 tx=0x7fc62c0c81a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d80f71825ba6d81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.272+0000 7fc61affd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.272+0000 7fc61affd640 1 -- 172.21.15.31:0/2032848511 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c0e2060 con 0x55cd1518b410 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.272+0000 7fc64883f740 1 -- 172.21.15.31:0/2032848511 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc63401bab0 con 0x55cd1518b410 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.272+0000 7fc61affd640 1 -- 172.21.15.31:0/2032848511 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc62c0ddbe0 con 0x55cd1518b410 2024-04-25T16:09:52.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.273+0000 7fc61affd640 1 -- 172.21.15.31:0/2032848511 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc62c04a040 con 0x55cd1518b410 2024-04-25T16:09:52.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.274+0000 7fc64883f740 10 client.26992 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.274+0000 7fc64883f740 1 -- 172.21.15.31:0/2032848511 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cd15163170 con 0x55cd1518b410 2024-04-25T16:09:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.274+0000 7fc61affd640 1 -- 172.21.15.31:0/2032848511 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc63401bab0 con 0x55cd1518b410 2024-04-25T16:09:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.276+0000 7fc61affd640 1 -- 172.21.15.31:0/2032848511 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc62c0d1160 con 0x55cd1518b410 2024-04-25T16:09:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.278+0000 7fc64883f740 10 client.26992 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.278+0000 7fc64883f740 10 client.26992 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.278+0000 7fc64883f740 20 client.26992 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.278+0000 7fc64883f740 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55cd1514a1f0 0x55cd15543ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.278+0000 7fc64883f740 4 client.26992 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:52.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.278+0000 7fc64883f740 1 -- 172.21.15.31:0/2032848511 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd1549f710 con 0x55cd1514a1f0 2024-04-25T16:09:52.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.282+0000 7fc63affd640 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55cd1514a1f0 0x55cd15543ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.287+0000 7fc61affd640 10 client.26992 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:52.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.287+0000 7fc63affd640 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55cd1514a1f0 0x55cd15543ee0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8b36202af05af1c server_cookie=732c85ab93472475 in_seq=0 out_seq=0 2024-04-25T16:09:52.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.290+0000 7fc61affd640 1 -- 172.21.15.31:0/2032848511 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cd1549f710 con 0x55cd1514a1f0 2024-04-25T16:09:52.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.290+0000 7fc61affd640 10 client.26992 handle_command_reply: tid=0 2024-04-25T16:09:52.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.290+0000 7fc64883f740 1 client.26992 shutdown 2024-04-25T16:09:52.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.290+0000 7fc64883f740 1 -- 172.21.15.31:0/2032848511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 msgr2=0x55cd1553d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.290+0000 7fc64883f740 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553d9f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc62c0c4d30 tx=0x7fc62c0c81a0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.291+0000 7fc64883f740 1 -- 172.21.15.31:0/2032848511 shutdown_connections 2024-04-25T16:09:52.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.291+0000 7fc64883f740 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55cd1514a1f0 0x55cd15543ee0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.291+0000 7fc64883f740 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553d9f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.291+0000 7fc64883f740 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd15536a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.291+0000 7fc64883f740 1 --2- 172.21.15.31:0/2032848511 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15536550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.291+0000 7fc64883f740 1 -- 172.21.15.31:0/2032848511 >> 172.21.15.31:0/2032848511 conn(0x55cd1517f560 msgr2=0x55cd15527050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:52.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.291+0000 7fc61affd640 0 client.26992 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:52.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.295+0000 7fc64883f740 1 -- 172.21.15.31:0/2032848511 shutdown_connections 2024-04-25T16:09:52.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.295+0000 7fc64883f740 1 -- 172.21.15.31:0/2032848511 wait complete. 2024-04-25T16:09:52.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.296+0000 7fc64883f740 20 client.26992 trim_cache size 0 max 16384 2024-04-25T16:09:52.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.335+0000 7fc64883f740 1 Processor -- start 2024-04-25T16:09:52.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.335+0000 7fc64883f740 1 -- start start 2024-04-25T16:09:52.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.336+0000 7fc64883f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15530fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.336+0000 7fc64883f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd1553d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.336+0000 7fc64883f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd15526d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.336+0000 7fc63affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15530fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.336+0000 7fc63a7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd1553d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.336+0000 7fc63affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15530fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58474/0 (socket says 172.21.15.31:58474) 2024-04-25T16:09:52.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.336+0000 7fc63affd640 1 -- 172.21.15.31:0/3231305334 learned_addr learned my addr 172.21.15.31:0/3231305334 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:52.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.337+0000 7fc64883f740 1 -- 172.21.15.31:0/3231305334 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc62800e500 con 0x55cd1516a730 2024-04-25T16:09:52.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.337+0000 7fc64883f740 1 -- 172.21.15.31:0/3231305334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc628010040 con 0x55cd15162550 2024-04-25T16:09:52.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.337+0000 7fc64883f740 1 -- 172.21.15.31:0/3231305334 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc62800ebd0 con 0x55cd1518b410 2024-04-25T16:09:52.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.337+0000 7fc63b7fe640 1 --2- 172.21.15.31:0/3231305334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd15526d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.339+0000 7fc63affd640 1 -- 172.21.15.31:0/3231305334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 msgr2=0x55cd15526d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:52.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.339+0000 7fc63affd640 1 --2- 172.21.15.31:0/3231305334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd15526d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.339+0000 7fc63affd640 1 -- 172.21.15.31:0/3231305334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 msgr2=0x55cd1553d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:52.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.339+0000 7fc63affd640 1 --2- 172.21.15.31:0/3231305334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd1553d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.339+0000 7fc63affd640 1 -- 172.21.15.31:0/3231305334 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd1514a420 con 0x55cd15162550 2024-04-25T16:09:52.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.339+0000 7fc63affd640 1 --2- 172.21.15.31:0/3231305334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15530fa0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55cd1552d870 tx=0x7fc634026000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e4762910814bc56 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:52.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.340+0000 7fc6217fa640 1 -- 172.21.15.31:0/3231305334 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc634039070 con 0x55cd15162550 2024-04-25T16:09:52.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.340+0000 7fc6217fa640 1 -- 172.21.15.31:0/3231305334 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc634014040 con 0x55cd15162550 2024-04-25T16:09:52.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.341+0000 7fc64883f740 1 -- 172.21.15.31:0/3231305334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 msgr2=0x55cd15530fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.341+0000 7fc64883f740 1 --2- 172.21.15.31:0/3231305334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15530fa0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x55cd1552d870 tx=0x7fc634026000 comp rx=0 tx=0).stop 2024-04-25T16:09:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.344+0000 7fc64883f740 1 -- 172.21.15.31:0/3231305334 shutdown_connections 2024-04-25T16:09:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.344+0000 7fc64883f740 1 --2- 172.21.15.31:0/3231305334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd15526d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.344+0000 7fc64883f740 1 --2- 172.21.15.31:0/3231305334 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd1516a730 0x55cd1553d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.344+0000 7fc64883f740 1 --2- 172.21.15.31:0/3231305334 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd15162550 0x55cd15530fa0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.344+0000 7fc64883f740 1 -- 172.21.15.31:0/3231305334 >> 172.21.15.31:0/3231305334 conn(0x55cd1517f560 msgr2=0x55cd15526620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.344+0000 7fc64883f740 1 -- 172.21.15.31:0/3231305334 shutdown_connections 2024-04-25T16:09:52.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.344+0000 7fc64883f740 1 -- 172.21.15.31:0/3231305334 wait complete. 2024-04-25T16:09:52.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.345+0000 7fc64883f740 1 -- start start 2024-04-25T16:09:52.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.345+0000 7fc64883f740 1 Processor -- start 2024-04-25T16:09:52.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.346+0000 7fc64883f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd15162550 0x55cd1553aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.346+0000 7fc64883f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1516a730 0x55cd1553b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.346+0000 7fc64883f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.346+0000 7fc64883f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc62800f970 con 0x55cd15162550 2024-04-25T16:09:52.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.346+0000 7fc64883f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc62800f2a0 con 0x55cd1516a730 2024-04-25T16:09:52.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.346+0000 7fc64883f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc62800c4e0 con 0x55cd1518b410 2024-04-25T16:09:52.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.346+0000 7fc64883f740 10 client.0 _ll_register_callbacks cb 0x7fc62c0c46e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:52.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.347+0000 7fc63a7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1516a730 0x55cd1553b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.348 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.347+0000 7fc63a7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1516a730 0x55cd1553b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58488/0 (socket says 172.21.15.31:58488) 2024-04-25T16:09:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.347+0000 7fc63a7fc640 1 -- 172.21.15.31:0/1017837506 learned_addr learned my addr 172.21.15.31:0/1017837506 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:52.349 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.347+0000 7fc63b7fe640 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.351+0000 7fc63affd640 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd15162550 0x55cd1553aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.351+0000 7fc63a7fc640 1 -- 172.21.15.31:0/1017837506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 msgr2=0x55cd1553bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:52.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.351+0000 7fc63a7fc640 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.351+0000 7fc63a7fc640 1 -- 172.21.15.31:0/1017837506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd15162550 msgr2=0x55cd1553aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:52.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.351+0000 7fc63a7fc640 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd15162550 0x55cd1553aae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.351+0000 7fc63a7fc640 1 -- 172.21.15.31:0/1017837506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd1553c510 con 0x55cd1516a730 2024-04-25T16:09:52.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.351+0000 7fc63a7fc640 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1516a730 0x55cd1553b020 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x55cd1552da80 tx=0x7fc63000f5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4a5fccd085bb6ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.351+0000 7fc61affd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.352+0000 7fc61affd640 1 -- 172.21.15.31:0/1017837506 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc630018020 con 0x55cd1516a730 2024-04-25T16:09:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.352+0000 7fc64883f740 1 -- 172.21.15.31:0/1017837506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc634014040 con 0x55cd1516a730 2024-04-25T16:09:52.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.353+0000 7fc61affd640 1 -- 172.21.15.31:0/1017837506 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc630014660 con 0x55cd1516a730 2024-04-25T16:09:52.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.354+0000 7fc61affd640 1 -- 172.21.15.31:0/1017837506 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc63000fea0 con 0x55cd1516a730 2024-04-25T16:09:52.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.354+0000 7fc64883f740 10 client.27014 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:52.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.354+0000 7fc64883f740 1 -- 172.21.15.31:0/1017837506 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cd1553d8d0 con 0x55cd1516a730 2024-04-25T16:09:52.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.354+0000 7fc61affd640 1 -- 172.21.15.31:0/1017837506 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc634014040 con 0x55cd1516a730 2024-04-25T16:09:52.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.355+0000 7fc61affd640 1 -- 172.21.15.31:0/1017837506 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc630013920 con 0x55cd1516a730 2024-04-25T16:09:52.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.355+0000 7fc64883f740 10 client.27014 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:52.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.355+0000 7fc64883f740 10 client.27014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:52.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.355+0000 7fc64883f740 20 client.27014 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:52.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.355+0000 7fc64883f740 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55cd1511f630 0x55cd155ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:52.356 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.355+0000 7fc64883f740 4 client.27014 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:09:52.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.355+0000 7fc64883f740 1 -- 172.21.15.31:0/1017837506 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55cd15499580 con 0x55cd1511f630 2024-04-25T16:09:52.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.357+0000 7fc63affd640 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55cd1511f630 0x55cd155ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:52.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.360+0000 7fc63affd640 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55cd1511f630 0x55cd155ae6d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17f4bdf7a9f52bea server_cookie=f0f6e1e7fcface5e in_seq=0 out_seq=0 2024-04-25T16:09:52.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.360+0000 7fc61affd640 10 client.27014 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:52.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.362+0000 7fc61affd640 1 -- 172.21.15.31:0/1017837506 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55cd15499580 con 0x55cd1511f630 2024-04-25T16:09:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.362+0000 7fc61affd640 10 client.27014 handle_command_reply: tid=0 2024-04-25T16:09:52.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.362+0000 7fc64883f740 1 client.27014 shutdown 2024-04-25T16:09:52.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.368+0000 7fc64883f740 1 -- 172.21.15.31:0/1017837506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1516a730 msgr2=0x55cd1553b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.368+0000 7fc64883f740 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1516a730 0x55cd1553b020 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x55cd1552da80 tx=0x7fc63000f5b0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.372+0000 7fc64883f740 1 -- 172.21.15.31:0/1017837506 shutdown_connections 2024-04-25T16:09:52.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.372+0000 7fc64883f740 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55cd1511f630 0x55cd155ae6d0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.372+0000 7fc64883f740 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd1518b410 0x55cd1553bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.372+0000 7fc64883f740 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd1516a730 0x55cd1553b020 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.372+0000 7fc64883f740 1 --2- 172.21.15.31:0/1017837506 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd15162550 0x55cd1553aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.372+0000 7fc64883f740 1 -- 172.21.15.31:0/1017837506 >> 172.21.15.31:0/1017837506 conn(0x55cd1517f560 msgr2=0x55cd15526620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:52.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.372+0000 7fc61affd640 0 client.27014 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:52.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.372+0000 7fc64883f740 1 -- 172.21.15.31:0/1017837506 shutdown_connections 2024-04-25T16:09:52.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.372+0000 7fc64883f740 1 -- 172.21.15.31:0/1017837506 wait complete. 2024-04-25T16:09:52.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.373+0000 7fc64883f740 20 client.27014 trim_cache size 0 max 16384 2024-04-25T16:09:52.375 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:09:52.376 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:09:52.376 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "513074d0-518d-4d16-b1a4-3c415943f55c", 2024-04-25T16:09:52.376 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:09:52.376 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:09:52.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 -- 172.21.15.31:0/2875540375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6240afd00 msgr2=0x7fc6240b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6240afd00 0x7fc6240b2210 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc634005c40 tx=0x7fc634005bd0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 -- 172.21.15.31:0/2875540375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c10fef0 msgr2=0x7fc63c11a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c10fef0 0x7fc63c11a4d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009850 tx=0x7fc62c0053c0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 -- 172.21.15.31:0/2875540375 shutdown_connections 2024-04-25T16:09:52.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc6240afd00 0x7fc6240b2210 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc63c10fef0 0x7fc63c11a4d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc63c10b1d0 0x7fc63c119570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 --2- 172.21.15.31:0/2875540375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc63c10a350 0x7fc63c119030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.398+0000 7fc63bfff640 1 -- 172.21.15.31:0/2875540375 >> 172.21.15.31:0/2875540375 conn(0x7fc63c070400 msgr2=0x7fc63c10dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:52.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.400+0000 7fc63bfff640 1 -- 172.21.15.31:0/2875540375 shutdown_connections 2024-04-25T16:09:52.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.400+0000 7fc63bfff640 1 -- 172.21.15.31:0/2875540375 wait complete. 2024-04-25T16:09:52.446 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:52 smithi031 ceph-mon[29140]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 733 MiB/s wr, 1.83k op/s 2024-04-25T16:09:52.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:09:52.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:09:52.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:52 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:09:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:52 smithi063 ceph-mon[32413]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 733 MiB/s wr, 1.83k op/s 2024-04-25T16:09:52.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:52 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:09:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:52 smithi186 ceph-mon[33258]: pgmap v608: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 733 MiB/s wr, 1.83k op/s 2024-04-25T16:09:52.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.989+0000 7fc811a27640 1 -- 172.21.15.31:0/2540094960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040ab780 msgr2=0x7fc8040abb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:52.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.989+0000 7fc811a27640 1 --2- 172.21.15.31:0/2540094960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040ab780 0x7fc8040abb90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fc800009900 tx=0x7fc80002a420 comp rx=0 tx=0).stop 2024-04-25T16:09:52.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.992+0000 7fc811a27640 1 -- 172.21.15.31:0/2540094960 shutdown_connections 2024-04-25T16:09:52.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.992+0000 7fc811a27640 1 --2- 172.21.15.31:0/2540094960 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040ab780 0x7fc8040abb90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.992+0000 7fc811a27640 1 --2- 172.21.15.31:0/2540094960 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8040a6a60 0x7fc8040a6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.992+0000 7fc811a27640 1 --2- 172.21.15.31:0/2540094960 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8040a5960 0x7fc8040a6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:52.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:52.992+0000 7fc811a27640 1 -- 172.21.15.31:0/2540094960 >> 172.21.15.31:0/2540094960 conn(0x7fc8040a09d0 msgr2=0x7fc8040a2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:53.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.002+0000 7fc811a27640 1 -- 172.21.15.31:0/2540094960 shutdown_connections 2024-04-25T16:09:53.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.002+0000 7fc811a27640 1 -- 172.21.15.31:0/2540094960 wait complete. 2024-04-25T16:09:53.007 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.003+0000 7fc811a27640 1 Processor -- start 2024-04-25T16:09:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.005+0000 7fc811a27640 1 -- start start 2024-04-25T16:09:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.006+0000 7fc811a27640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8040a5960 0x7fc804144630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.006+0000 7fc811a27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040a6a60 0x7fc804144b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.006+0000 7fc811a27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8040ab780 0x7fc80414bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.006+0000 7fc811a27640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc8040b9be0 con 0x7fc8040ab780 2024-04-25T16:09:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.006+0000 7fc811a27640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc8040b9a60 con 0x7fc8040a5960 2024-04-25T16:09:53.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.006+0000 7fc811a27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc8040b9d60 con 0x7fc8040a6a60 2024-04-25T16:09:53.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.011+0000 7fc80affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8040a5960 0x7fc804144630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.012+0000 7fc80a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040a6a60 0x7fc804144b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.012+0000 7fc80a7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040a6a60 0x7fc804144b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48064/0 (socket says 172.21.15.31:48064) 2024-04-25T16:09:53.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.012+0000 7fc80a7fc640 1 -- 172.21.15.31:0/1314831058 learned_addr learned my addr 172.21.15.31:0/1314831058 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:53.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.012+0000 7fc80b7fe640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8040ab780 0x7fc80414bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.015+0000 7fc80a7fc640 1 -- 172.21.15.31:0/1314831058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8040a5960 msgr2=0x7fc804144630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.015+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8040a5960 0x7fc804144630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.015+0000 7fc80a7fc640 1 -- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8040ab780 msgr2=0x7fc80414bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.015+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8040ab780 0x7fc80414bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.015+0000 7fc80a7fc640 1 -- 172.21.15.31:0/1314831058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8040ad260 con 0x7fc8040a6a60 2024-04-25T16:09:53.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.017+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040a6a60 0x7fc804144b70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc80c000b60 tx=0x7fc80c06fe10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:53.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.026+0000 7fc810a25640 1 -- 172.21.15.31:0/1314831058 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c071300 con 0x7fc8040a6a60 2024-04-25T16:09:53.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.026+0000 7fc810a25640 1 -- 172.21.15.31:0/1314831058 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc80c06bc90 con 0x7fc8040a6a60 2024-04-25T16:09:53.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.026+0000 7fc810a25640 1 -- 172.21.15.31:0/1314831058 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c0709e0 con 0x7fc8040a6a60 2024-04-25T16:09:53.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.026+0000 7fc811a27640 1 -- 172.21.15.31:0/1314831058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8040ad550 con 0x7fc8040a6a60 2024-04-25T16:09:53.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.026+0000 7fc811a27640 1 -- 172.21.15.31:0/1314831058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8040ada20 con 0x7fc8040a6a60 2024-04-25T16:09:53.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.034+0000 7fc8189b8740 1 Processor -- start 2024-04-25T16:09:53.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.036+0000 7fc8189b8740 1 -- start start 2024-04-25T16:09:53.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.036+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 0x55bd49a44c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.036+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd4967a4d0 0x55bd49a451d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.037+0000 7fc810a25640 1 -- 172.21.15.31:0/1314831058 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc80c070b80 con 0x7fc8040a6a60 2024-04-25T16:09:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.037+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a45e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.037+0000 7fc8189b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bd499a9d80 con 0x55bd4967a4d0 2024-04-25T16:09:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.037+0000 7fc8189b8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bd499638b0 con 0x55bd496826b0 2024-04-25T16:09:53.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.037+0000 7fc8189b8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bd499a5d40 con 0x55bd496a3390 2024-04-25T16:09:53.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.038+0000 7fc810a25640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7f40afcf0 0x7fc7f40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.039+0000 7fc80b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 0x55bd49a44c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.039+0000 7fc80b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 0x55bd49a44c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48070/0 (socket says 172.21.15.31:48070) 2024-04-25T16:09:53.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.039+0000 7fc80b7fe640 1 -- 172.21.15.31:0/2609372699 learned_addr learned my addr 172.21.15.31:0/2609372699 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:53.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.039+0000 7fc810a25640 1 -- 172.21.15.31:0/1314831058 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc80c143320 con 0x7fc8040a6a60 2024-04-25T16:09:53.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.039+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/2609372699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a45e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.041+0000 7fc80affd640 1 --2- 172.21.15.31:0/2609372699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd4967a4d0 0x55bd49a451d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.041+0000 7fc80a7fc640 1 -- 172.21.15.31:0/2609372699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 msgr2=0x55bd49a44c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:53.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.041+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/2609372699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 0x55bd49a44c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.041+0000 7fc80a7fc640 1 -- 172.21.15.31:0/2609372699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd4967a4d0 msgr2=0x55bd49a451d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:53.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.041+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/2609372699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd4967a4d0 0x55bd49a451d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.041+0000 7fc80a7fc640 1 -- 172.21.15.31:0/2609372699 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd49934a40 con 0x55bd496826b0 2024-04-25T16:09:53.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.043+0000 7fc80b7fe640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7f40afcf0 0x7fc7f40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.043+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/2609372699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a45e60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0690d0 tx=0x7fc80c071f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ffc2c4b0992428 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:53.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.045+0000 7fc7f1ffb640 1 -- 172.21.15.31:0/2609372699 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c06be40 con 0x55bd496826b0 2024-04-25T16:09:53.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.045+0000 7fc7f1ffb640 1 -- 172.21.15.31:0/2609372699 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc80c10b2c0 con 0x55bd496826b0 2024-04-25T16:09:53.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.045+0000 7fc7f1ffb640 1 -- 172.21.15.31:0/2609372699 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c070450 con 0x55bd496826b0 2024-04-25T16:09:53.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.048+0000 7fc8189b8740 1 -- 172.21.15.31:0/2609372699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 msgr2=0x55bd49a45e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:53.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.048+0000 7fc8189b8740 1 --2- 172.21.15.31:0/2609372699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a45e60 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0690d0 tx=0x7fc80c071f50 comp rx=0 tx=0).stop 2024-04-25T16:09:53.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.051+0000 7fc8189b8740 1 -- 172.21.15.31:0/2609372699 shutdown_connections 2024-04-25T16:09:53.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.051+0000 7fc8189b8740 1 --2- 172.21.15.31:0/2609372699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a45e60 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.051+0000 7fc8189b8740 1 --2- 172.21.15.31:0/2609372699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd4967a4d0 0x55bd49a451d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.051+0000 7fc8189b8740 1 --2- 172.21.15.31:0/2609372699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 0x55bd49a44c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.051+0000 7fc8189b8740 1 -- 172.21.15.31:0/2609372699 >> 172.21.15.31:0/2609372699 conn(0x55bd496974e0 msgr2=0x55bd49a40460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.056+0000 7fc8189b8740 1 -- 172.21.15.31:0/2609372699 shutdown_connections 2024-04-25T16:09:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.058+0000 7fc80b7fe640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7f40afcf0 0x7fc7f40b2200 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x55bd49a45ca0 tx=0x7fc8000056d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.059+0000 7fc8189b8740 1 -- 172.21.15.31:0/2609372699 wait complete. 2024-04-25T16:09:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.059+0000 7fc8189b8740 1 -- start start 2024-04-25T16:09:53.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.060+0000 7fc8189b8740 1 Processor -- start 2024-04-25T16:09:53.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.066+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4ea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.066+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a4efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.066+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496a3390 0x55bd49a55f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.066+0000 7fc8189b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55bd49a46910 con 0x55bd496a3390 2024-04-25T16:09:53.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.066+0000 7fc8189b8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55bd49930d10 con 0x55bd496826b0 2024-04-25T16:09:53.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.066+0000 7fc8189b8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55bd49a46a90 con 0x55bd4967a4d0 2024-04-25T16:09:53.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.067+0000 7fc8189b8740 10 client.0 _ll_register_callbacks cb 0x55bd4995d210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:53.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.068+0000 7fc80b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496a3390 0x55bd49a55f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.068+0000 7fc80b7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496a3390 0x55bd49a55f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:60470/0 (socket says 172.21.15.31:60470) 2024-04-25T16:09:53.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.068+0000 7fc80b7fe640 1 -- 172.21.15.31:0/3051974824 learned_addr learned my addr 172.21.15.31:0/3051974824 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:53.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.068+0000 7fc80affd640 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.068+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a4efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.069+0000 7fc80affd640 1 -- 172.21.15.31:0/3051974824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 msgr2=0x55bd49a4efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:53.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.069+0000 7fc80affd640 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a4efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.069+0000 7fc80affd640 1 -- 172.21.15.31:0/3051974824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496a3390 msgr2=0x55bd49a55f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:53.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.069+0000 7fc80affd640 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496a3390 0x55bd49a55f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.069+0000 7fc80affd640 1 -- 172.21.15.31:0/3051974824 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd497857d0 con 0x55bd4967a4d0 2024-04-25T16:09:53.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.071+0000 7fc80affd640 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4ea90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55bd49a44970 tx=0x7fc7fc011630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bceec306caf82e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:53.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.074+0000 7fc7eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:53.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.074+0000 7fc7eaffd640 1 -- 172.21.15.31:0/3051974824 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc019070 con 0x55bd4967a4d0 2024-04-25T16:09:53.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.074+0000 7fc8189b8740 1 -- 172.21.15.31:0/3051974824 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc80c10b2c0 con 0x55bd4967a4d0 2024-04-25T16:09:53.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.074+0000 7fc7eaffd640 1 -- 172.21.15.31:0/3051974824 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc7fc015660 con 0x55bd4967a4d0 2024-04-25T16:09:53.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.077+0000 7fc7eaffd640 1 -- 172.21.15.31:0/3051974824 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc7fc025050 con 0x55bd4967a4d0 2024-04-25T16:09:53.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.077+0000 7fc8189b8740 10 client.27010 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:53.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.077+0000 7fc8189b8740 1 -- 172.21.15.31:0/3051974824 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bd496bb3f0 con 0x55bd4967a4d0 2024-04-25T16:09:53.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.078+0000 7fc7eaffd640 1 -- 172.21.15.31:0/3051974824 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc80c10b2c0 con 0x55bd4967a4d0 2024-04-25T16:09:53.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.078+0000 7fc7eaffd640 1 -- 172.21.15.31:0/3051974824 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc7fc014940 con 0x55bd4967a4d0 2024-04-25T16:09:53.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.078+0000 7fc8189b8740 10 client.27010 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:53.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.078+0000 7fc8189b8740 10 client.27010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:53.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.078+0000 7fc8189b8740 20 client.27010 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:53.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.078+0000 7fc8189b8740 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bd49661f60 0x55bd49a5c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.078+0000 7fc8189b8740 4 client.27010 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:53.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.078+0000 7fc8189b8740 1 -- 172.21.15.31:0/3051974824 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd499b7990 con 0x55bd49661f60 2024-04-25T16:09:53.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.081+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bd49661f60 0x55bd49a5c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.092+0000 7fc7eaffd640 10 client.27010 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:53.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.092+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bd49661f60 0x55bd49a5c8a0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf86c180d1fbc3f0 server_cookie=a7ea841b0b66ba3c in_seq=0 out_seq=0 2024-04-25T16:09:53.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.097+0000 7fc7eaffd640 1 -- 172.21.15.31:0/3051974824 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55bd499b7990 con 0x55bd49661f60 2024-04-25T16:09:53.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.097+0000 7fc7eaffd640 10 client.27010 handle_command_reply: tid=0 2024-04-25T16:09:53.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.097+0000 7fc8189b8740 1 client.27010 shutdown 2024-04-25T16:09:53.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.101+0000 7fc8189b8740 1 -- 172.21.15.31:0/3051974824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 msgr2=0x55bd49a4ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:53.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.101+0000 7fc8189b8740 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4ea90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55bd49a44970 tx=0x7fc7fc011630 comp rx=0 tx=0).stop 2024-04-25T16:09:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.106+0000 7fc8189b8740 1 -- 172.21.15.31:0/3051974824 shutdown_connections 2024-04-25T16:09:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.106+0000 7fc8189b8740 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bd49661f60 0x55bd49a5c8a0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.106+0000 7fc8189b8740 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496a3390 0x55bd49a55f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.106+0000 7fc8189b8740 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a4efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.106+0000 7fc8189b8740 1 --2- 172.21.15.31:0/3051974824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4ea90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.106+0000 7fc8189b8740 1 -- 172.21.15.31:0/3051974824 >> 172.21.15.31:0/3051974824 conn(0x55bd496974e0 msgr2=0x55bd49a3f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.106+0000 7fc7eaffd640 0 client.27010 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:53.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.106+0000 7fc8189b8740 1 -- 172.21.15.31:0/3051974824 shutdown_connections 2024-04-25T16:09:53.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.113+0000 7fc8189b8740 1 -- 172.21.15.31:0/3051974824 wait complete. 2024-04-25T16:09:53.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.120+0000 7fc8189b8740 20 client.27010 trim_cache size 0 max 16384 2024-04-25T16:09:53.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.181+0000 7fc8189b8740 1 Processor -- start 2024-04-25T16:09:53.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.181+0000 7fc8189b8740 1 -- start start 2024-04-25T16:09:53.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.181+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.189 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496826b0 0x55bd49a4eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496a3390 0x55bd49a55e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc8189b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc80415df40 con 0x55bd496826b0 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc80affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc8189b8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc80415fa80 con 0x55bd496a3390 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc80affd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48088/0 (socket says 172.21.15.31:48088) 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc80affd640 1 -- 172.21.15.31:0/1473212220 learned_addr learned my addr 172.21.15.31:0/1473212220 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc8189b8740 1 -- 172.21.15.31:0/1473212220 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc80415e610 con 0x55bd4967a4d0 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc80b7fe640 1 --2- 172.21.15.31:0/1473212220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496a3390 0x55bd49a55e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc80affd640 1 -- 172.21.15.31:0/1473212220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496a3390 msgr2=0x55bd49a55e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc80affd640 1 --2- 172.21.15.31:0/1473212220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496a3390 0x55bd49a55e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc80affd640 1 -- 172.21.15.31:0/1473212220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496826b0 msgr2=0x55bd49a4eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:53.190 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc80affd640 1 --2- 172.21.15.31:0/1473212220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496826b0 0x55bd49a4eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.182+0000 7fc80affd640 1 -- 172.21.15.31:0/1473212220 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd4967b0f0 con 0x55bd4967a4d0 2024-04-25T16:09:53.191 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.183+0000 7fc80affd640 1 --2- 172.21.15.31:0/1473212220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4e660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc014e80 tx=0x7fc7fc006b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a226c4a0b5f84a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:53.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.192+0000 7fc7f17fa640 1 -- 172.21.15.31:0/1473212220 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc019040 con 0x55bd4967a4d0 2024-04-25T16:09:53.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.192+0000 7fc7f17fa640 1 -- 172.21.15.31:0/1473212220 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7fc02e070 con 0x55bd4967a4d0 2024-04-25T16:09:53.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.192+0000 7fc7f17fa640 1 -- 172.21.15.31:0/1473212220 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc014b00 con 0x55bd4967a4d0 2024-04-25T16:09:53.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.195+0000 7fc8189b8740 1 -- 172.21.15.31:0/1473212220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 msgr2=0x55bd49a4e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:53.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.195+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1473212220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4e660 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc014e80 tx=0x7fc7fc006b30 comp rx=0 tx=0).stop 2024-04-25T16:09:53.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.196+0000 7fc8189b8740 1 -- 172.21.15.31:0/1473212220 shutdown_connections 2024-04-25T16:09:53.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.196+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1473212220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496a3390 0x55bd49a55e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.196+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1473212220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd496826b0 0x55bd49a4eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.196+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1473212220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd4967a4d0 0x55bd49a4e660 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.196+0000 7fc8189b8740 1 -- 172.21.15.31:0/1473212220 >> 172.21.15.31:0/1473212220 conn(0x55bd496974e0 msgr2=0x55bd49a497f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.199+0000 7fc8189b8740 1 -- 172.21.15.31:0/1473212220 shutdown_connections 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.199+0000 7fc8189b8740 1 -- 172.21.15.31:0/1473212220 wait complete. 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.200+0000 7fc8189b8740 1 -- start start 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.201+0000 7fc8189b8740 1 Processor -- start 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.201+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd4967a4d0 0x55bd49a534d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a53a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc8189b8740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 0x55bd49ac2c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc8189b8740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc80415f3b0 con 0x55bd4967a4d0 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc8189b8740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc80415ece0 con 0x55bd496826b0 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc8189b8740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc8040b48d0 con 0x55bd496a3390 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc80a7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a53a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.204 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc80a7fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a53a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58546/0 (socket says 172.21.15.31:58546) 2024-04-25T16:09:53.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc80a7fc640 1 -- 172.21.15.31:0/1810546977 learned_addr learned my addr 172.21.15.31:0/1810546977 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:53.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc8189b8740 10 client.0 _ll_register_callbacks cb 0x7fc7fc009a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:53.205 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.202+0000 7fc80b7fe640 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 0x55bd49ac2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.203+0000 7fc80a7fc640 1 -- 172.21.15.31:0/1810546977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 msgr2=0x55bd49ac2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:53.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.203+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 0x55bd49ac2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.203+0000 7fc80a7fc640 1 -- 172.21.15.31:0/1810546977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd4967a4d0 msgr2=0x55bd49a534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:53.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.203+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd4967a4d0 0x55bd49a534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.203+0000 7fc80a7fc640 1 -- 172.21.15.31:0/1810546977 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd4996a4b0 con 0x55bd496826b0 2024-04-25T16:09:53.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.203+0000 7fc80a7fc640 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a53a10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc80c10b4b0 tx=0x7fc80c107670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1202de60acca8f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:53.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.212+0000 7fc7eaffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:09:53.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.212+0000 7fc7eaffd640 1 -- 172.21.15.31:0/1810546977 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc80c06be40 con 0x55bd496826b0 2024-04-25T16:09:53.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.212+0000 7fc8189b8740 1 -- 172.21.15.31:0/1810546977 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc7fc02e070 con 0x55bd496826b0 2024-04-25T16:09:53.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.212+0000 7fc7eaffd640 1 -- 172.21.15.31:0/1810546977 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc80c12e0b0 con 0x55bd496826b0 2024-04-25T16:09:53.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.213+0000 7fc7eaffd640 1 -- 172.21.15.31:0/1810546977 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc80c10fb60 con 0x55bd496826b0 2024-04-25T16:09:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.213+0000 7fc8189b8740 10 client.27029 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.213+0000 7fc8189b8740 1 -- 172.21.15.31:0/1810546977 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55bd49ac4010 con 0x55bd496826b0 2024-04-25T16:09:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.214+0000 7fc7eaffd640 1 -- 172.21.15.31:0/1810546977 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7fc02e070 con 0x55bd496826b0 2024-04-25T16:09:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.216+0000 7fc7eaffd640 1 -- 172.21.15.31:0/1810546977 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc80c107a40 con 0x55bd496826b0 2024-04-25T16:09:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.216+0000 7fc8189b8740 10 client.27029 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.216+0000 7fc8189b8740 10 client.27029 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:53.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.216+0000 7fc8189b8740 20 client.27029 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:53.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.216+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bd496373a0 0x55bd49ac6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:53.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.216+0000 7fc8189b8740 4 client.27029 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:09:53.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.216+0000 7fc8189b8740 1 -- 172.21.15.31:0/1810546977 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bd499b1800 con 0x55bd496373a0 2024-04-25T16:09:53.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.217+0000 7fc80affd640 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bd496373a0 0x55bd49ac6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:53.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.222+0000 7fc7eaffd640 10 client.27029 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:53.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.222+0000 7fc80affd640 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bd496373a0 0x55bd49ac6a30 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bb868aae0de2084 server_cookie=f798a96030d12781 in_seq=0 out_seq=0 2024-04-25T16:09:53.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.225+0000 7fc7eaffd640 1 -- 172.21.15.31:0/1810546977 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55bd499b1800 con 0x55bd496373a0 2024-04-25T16:09:53.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.225+0000 7fc7eaffd640 10 client.27029 handle_command_reply: tid=0 2024-04-25T16:09:53.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.225+0000 7fc8189b8740 1 client.27029 shutdown 2024-04-25T16:09:53.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.229+0000 7fc8189b8740 1 -- 172.21.15.31:0/1810546977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 msgr2=0x55bd49a53a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:53.230 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.229+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a53a10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc80c10b4b0 tx=0x7fc80c107670 comp rx=0 tx=0).stop 2024-04-25T16:09:53.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.229+0000 7fc8189b8740 1 -- 172.21.15.31:0/1810546977 shutdown_connections 2024-04-25T16:09:53.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.229+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55bd496373a0 0x55bd49ac6a30 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.229+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55bd496a3390 0x55bd49ac2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.229+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55bd496826b0 0x55bd49a53a10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.229+0000 7fc8189b8740 1 --2- 172.21.15.31:0/1810546977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55bd4967a4d0 0x55bd49a534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.229+0000 7fc7eaffd640 0 client.27029 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:53.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.229+0000 7fc8189b8740 1 -- 172.21.15.31:0/1810546977 >> 172.21.15.31:0/1810546977 conn(0x55bd496974e0 msgr2=0x55bd49a4e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:53.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.230+0000 7fc8189b8740 1 -- 172.21.15.31:0/1810546977 shutdown_connections 2024-04-25T16:09:53.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.232+0000 7fc8189b8740 1 -- 172.21.15.31:0/1810546977 wait complete. 2024-04-25T16:09:53.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.234+0000 7fc8189b8740 20 client.27029 trim_cache size 0 max 16384 2024-04-25T16:09:53.246 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:09:53.246 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (0 inodes in the stack)", 2024-04-25T16:09:53.246 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:09:53.246 INFO:teuthology.orchestra.run.smithi031.stdout: "513074d0-518d-4d16-b1a4-3c415943f55c": { 2024-04-25T16:09:53.246 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:09:53.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "513074d0-518d-4d16-b1a4-3c415943f55c", 2024-04-25T16:09:53.246 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:09:53.246 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:09:53.247 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:09:53.247 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:09:53.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.243+0000 7fc811a27640 1 -- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7f40afcf0 msgr2=0x7fc7f40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:53.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.243+0000 7fc811a27640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7f40afcf0 0x7fc7f40b2200 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x55bd49a45ca0 tx=0x7fc8000056d0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.244+0000 7fc811a27640 1 -- 172.21.15.31:0/1314831058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040a6a60 msgr2=0x7fc804144b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:53.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.244+0000 7fc811a27640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040a6a60 0x7fc804144b70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc80c000b60 tx=0x7fc80c06fe10 comp rx=0 tx=0).stop 2024-04-25T16:09:53.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.245+0000 7fc811a27640 1 -- 172.21.15.31:0/1314831058 shutdown_connections 2024-04-25T16:09:53.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.245+0000 7fc811a27640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc7f40afcf0 0x7fc7f40b2200 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.245+0000 7fc811a27640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc8040ab780 0x7fc80414bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.245+0000 7fc811a27640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc8040a6a60 0x7fc804144b70 secure :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fc80c000b60 tx=0x7fc80c06fe10 comp rx=0 tx=0).stop 2024-04-25T16:09:53.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.245+0000 7fc811a27640 1 --2- 172.21.15.31:0/1314831058 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8040a5960 0x7fc804144630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:53.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.245+0000 7fc811a27640 1 -- 172.21.15.31:0/1314831058 >> 172.21.15.31:0/1314831058 conn(0x7fc8040a09d0 msgr2=0x7fc8040a2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:53.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.246+0000 7fc811a27640 1 -- 172.21.15.31:0/1314831058 shutdown_connections 2024-04-25T16:09:53.248 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:09:53.246+0000 7fc811a27640 1 -- 172.21.15.31:0/1314831058 wait complete. 2024-04-25T16:09:53.301 INFO:tasks.cephfs.filesystem:scrub status for tag:513074d0-518d-4d16-b1a4-3c415943f55c - {'path': '/', 'tag': '513074d0-518d-4d16-b1a4-3c415943f55c', 'options': 'recursive,force'} 2024-04-25T16:09:53.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:53 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:09:53.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:53 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:53.452 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:53 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:09:53.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:53 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:09:53.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:53 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:53.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:53 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:09:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:53 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:09:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:53 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:53 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:09:54.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:54 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:09:54.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:54 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:54.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:54 smithi031 ceph-mon[29140]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 879 MiB/s wr, 2.27k op/s 2024-04-25T16:09:54.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:54 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:09:54.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:54 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:54.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:54 smithi063 ceph-mon[32413]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 879 MiB/s wr, 2.27k op/s 2024-04-25T16:09:54.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:54 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:09:54.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:54 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:54.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:54 smithi186 ceph-mon[33258]: pgmap v609: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 879 MiB/s wr, 2.27k op/s 2024-04-25T16:09:55.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:09:55.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:09:55.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:09:55.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:09:55.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:09:55.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:09:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:09:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:09:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:09:56.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:56 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:56.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:56 smithi031 ceph-mon[29140]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 MiB/s wr, 1.90k op/s 2024-04-25T16:09:56.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:56 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:56.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:56 smithi063 ceph-mon[32413]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 MiB/s wr, 1.90k op/s 2024-04-25T16:09:56.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:56 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:09:56.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:56 smithi186 ceph-mon[33258]: pgmap v610: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 MiB/s wr, 1.90k op/s 2024-04-25T16:09:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.947+0000 7fc5a9def640 1 -- 172.21.15.31:0/671365191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a4078df0 msgr2=0x7fc5a407b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:58.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.947+0000 7fc5a9def640 1 --2- 172.21.15.31:0/671365191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a4078df0 0x7fc5a407b210 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc598009cb0 tx=0x7fc59802a7a0 comp rx=0 tx=0).stop 2024-04-25T16:09:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.956+0000 7fc5a9def640 1 -- 172.21.15.31:0/671365191 shutdown_connections 2024-04-25T16:09:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.956+0000 7fc5a9def640 1 --2- 172.21.15.31:0/671365191 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a4078df0 0x7fc5a407b210 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.956+0000 7fc5a9def640 1 --2- 172.21.15.31:0/671365191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a410a0e0 0x7fc5a410a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.956+0000 7fc5a9def640 1 --2- 172.21.15.31:0/671365191 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5a410b450 0x7fc5a4109ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.956+0000 7fc5a9def640 1 -- 172.21.15.31:0/671365191 >> 172.21.15.31:0/671365191 conn(0x7fc5a4070330 msgr2=0x7fc5a40727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.962+0000 7fc5a9def640 1 -- 172.21.15.31:0/671365191 shutdown_connections 2024-04-25T16:09:58.967 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.965+0000 7fc5a9def640 1 -- 172.21.15.31:0/671365191 wait complete. 2024-04-25T16:09:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.966+0000 7fc5a9def640 1 Processor -- start 2024-04-25T16:09:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.967+0000 7fc5a9def640 1 -- start start 2024-04-25T16:09:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.967+0000 7fc5a9def640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a4078df0 0x7fc5a407fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.967+0000 7fc5a9def640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a410a0e0 0x7fc5a4080400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.967+0000 7fc5a9def640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5a410b450 0x7fc5a41b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.967+0000 7fc5a9def640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a407ea20 con 0x7fc5a410b450 2024-04-25T16:09:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.967+0000 7fc5a9def640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a407e8a0 con 0x7fc5a4078df0 2024-04-25T16:09:58.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.967+0000 7fc5a9def640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a407eba0 con 0x7fc5a410a0e0 2024-04-25T16:09:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.968+0000 7fc5a3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a410a0e0 0x7fc5a4080400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.968+0000 7fc5a3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a410a0e0 0x7fc5a4080400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:57182/0 (socket says 172.21.15.31:57182) 2024-04-25T16:09:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.968+0000 7fc5a3fff640 1 -- 172.21.15.31:0/4080287227 learned_addr learned my addr 172.21.15.31:0/4080287227 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.971+0000 7fc5a3fff640 1 -- 172.21.15.31:0/4080287227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a4078df0 msgr2=0x7fc5a407fec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:09:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.971+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a4078df0 0x7fc5a407fec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.971+0000 7fc5a3fff640 1 -- 172.21.15.31:0/4080287227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5a410b450 msgr2=0x7fc5a41b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.971+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5a410b450 0x7fc5a41b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.971+0000 7fc5a3fff640 1 -- 172.21.15.31:0/4080287227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a41b9170 con 0x7fc5a410a0e0 2024-04-25T16:09:58.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.972+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a410a0e0 0x7fc5a4080400 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc59400e9c0 tx=0x7fc59400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.983+0000 7fc5a1ffb640 1 -- 172.21.15.31:0/4080287227 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59400ccc0 con 0x7fc5a410a0e0 2024-04-25T16:09:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.983+0000 7fc5a1ffb640 1 -- 172.21.15.31:0/4080287227 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5940054d0 con 0x7fc5a410a0e0 2024-04-25T16:09:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.983+0000 7fc5a1ffb640 1 -- 172.21.15.31:0/4080287227 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc594010430 con 0x7fc5a410a0e0 2024-04-25T16:09:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.983+0000 7fc5a9def640 1 -- 172.21.15.31:0/4080287227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a41b9460 con 0x7fc5a410a0e0 2024-04-25T16:09:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.984+0000 7fc5a9def640 1 -- 172.21.15.31:0/4080287227 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a41b9960 con 0x7fc5a410a0e0 2024-04-25T16:09:58.996 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.989+0000 7fc5b1fe7740 1 Processor -- start 2024-04-25T16:09:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.995+0000 7fc5b1fe7740 1 -- start start 2024-04-25T16:09:58.999 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.995+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625593101b0 0x5625596b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.995+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625592e72f0 0x5625596b1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.995+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ef4d0 0x5625596b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.995+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x562559616460 con 0x5625592ef4d0 2024-04-25T16:09:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.995+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5625595d0590 con 0x5625592e72f0 2024-04-25T16:09:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.995+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x562559612420 con 0x5625593101b0 2024-04-25T16:09:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.998+0000 7fc5a95ee640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625593101b0 0x5625596b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.999+0000 7fc5a95ee640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625593101b0 0x5625596b0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:57184/0 (socket says 172.21.15.31:57184) 2024-04-25T16:09:59.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.999+0000 7fc5a95ee640 1 -- 172.21.15.31:0/4262608892 learned_addr learned my addr 172.21.15.31:0/4262608892 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:59.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:58.999+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/4262608892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ef4d0 0x5625596b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.003+0000 7fc5a1ffb640 1 -- 172.21.15.31:0/4080287227 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc5940046c0 con 0x7fc5a410a0e0 2024-04-25T16:09:59.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.004+0000 7fc5a8ded640 1 --2- 172.21.15.31:0/4262608892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625592e72f0 0x5625596b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.004+0000 7fc5a3fff640 1 -- 172.21.15.31:0/4262608892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625593101b0 msgr2=0x5625596b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.004+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/4262608892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625593101b0 0x5625596b0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.004+0000 7fc5a3fff640 1 -- 172.21.15.31:0/4262608892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625592e72f0 msgr2=0x5625596b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.004+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/4262608892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625592e72f0 0x5625596b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.004+0000 7fc5a3fff640 1 -- 172.21.15.31:0/4262608892 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625595dd0d0 con 0x5625592ef4d0 2024-04-25T16:09:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.005+0000 7fc5a1ffb640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc58c0afcf0 0x7fc58c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.006+0000 7fc5a1ffb640 1 -- 172.21.15.31:0/4080287227 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc5940e0fd0 con 0x7fc5a410a0e0 2024-04-25T16:09:59.008 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.007+0000 7fc5a95ee640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc58c0afcf0 0x7fc58c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.009 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.008+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/4262608892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ef4d0 0x5625596b7f90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc5940dea20 tx=0x7fc5940e2ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb93b0cacc4d1fa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.009+0000 7fc5a95ee640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc58c0afcf0 0x7fc58c0b2200 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc59802ad90 tx=0x7fc598009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.009+0000 7fc58affd640 1 -- 172.21.15.31:0/4262608892 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940a5320 con 0x5625592ef4d0 2024-04-25T16:09:59.010 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.009+0000 7fc58affd640 1 -- 172.21.15.31:0/4262608892 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5940a8f70 con 0x5625592ef4d0 2024-04-25T16:09:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.012+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/4262608892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ef4d0 msgr2=0x5625596b7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.012+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/4262608892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ef4d0 0x5625596b7f90 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fc5940dea20 tx=0x7fc5940e2ea0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.012+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/4262608892 shutdown_connections 2024-04-25T16:09:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.012+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/4262608892 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ef4d0 0x5625596b7f90 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.012+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/4262608892 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625592e72f0 0x5625596b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.012+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/4262608892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625593101b0 0x5625596b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.012+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/4262608892 >> 172.21.15.31:0/4262608892 conn(0x562559304300 msgr2=0x5625596ac4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.025+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/4262608892 shutdown_connections 2024-04-25T16:09:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.025+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/4262608892 wait complete. 2024-04-25T16:09:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.026+0000 7fc5b1fe7740 1 -- start start 2024-04-25T16:09:59.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.026+0000 7fc5b1fe7740 1 Processor -- start 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.027+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596ba6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.027+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x5625596bac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.027+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.027+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5625596b8a30 con 0x5625592ced80 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.027+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56255959db60 con 0x5625593101b0 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.027+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5625596b8bb0 con 0x5625592e72f0 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.027+0000 7fc5b1fe7740 10 client.0 _ll_register_callbacks cb 0x5625595c22b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.028+0000 7fc5a95ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.028+0000 7fc5a95ee640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33192/0 (socket says 172.21.15.31:33192) 2024-04-25T16:09:59.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.028+0000 7fc5a95ee640 1 -- 172.21.15.31:0/1974428795 learned_addr learned my addr 172.21.15.31:0/1974428795 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.029+0000 7fc5a95ee640 1 -- 172.21.15.31:0/1974428795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 msgr2=0x5625596ba6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.029+0000 7fc5a95ee640 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596ba6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.029+0000 7fc5a95ee640 1 -- 172.21.15.31:0/1974428795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 msgr2=0x5625596bac20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:09:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.029+0000 7fc5a95ee640 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x5625596bac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.029+0000 7fc5a95ee640 1 -- 172.21.15.31:0/1974428795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562559328210 con 0x5625592ced80 2024-04-25T16:09:59.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.029+0000 7fc5a95ee640 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c1b80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc598030ef0 tx=0x7fc5980309a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b046a7051be69873 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:59.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.033+0000 7fc583fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:09:59.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.033+0000 7fc583fff640 1 -- 172.21.15.31:0/1974428795 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59804c070 con 0x5625592ced80 2024-04-25T16:09:59.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.034+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/1974428795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562559624070 con 0x5625592ced80 2024-04-25T16:09:59.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.034+0000 7fc583fff640 1 -- 172.21.15.31:0/1974428795 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc598005460 con 0x5625592ced80 2024-04-25T16:09:59.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.049+0000 7fc583fff640 1 -- 172.21.15.31:0/1974428795 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc598058050 con 0x5625592ced80 2024-04-25T16:09:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.050+0000 7fc5b1fe7740 10 client.17352 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:59.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.050+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/1974428795 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625593f4620 con 0x5625592ced80 2024-04-25T16:09:59.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.056+0000 7fc583fff640 1 -- 172.21.15.31:0/1974428795 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x562559624070 con 0x5625592ced80 2024-04-25T16:09:59.057 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.056+0000 7fc583fff640 1 -- 172.21.15.31:0/1974428795 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc598044070 con 0x5625592ced80 2024-04-25T16:09:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.056+0000 7fc5b1fe7740 10 client.17352 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.056+0000 7fc5b1fe7740 10 client.17352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.056+0000 7fc5b1fe7740 20 client.17352 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.056+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5625592a41c0 0x5625596c8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.056+0000 7fc5b1fe7740 4 client.17352 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:09:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.056+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/1974428795 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56255961dee0 con 0x5625592a41c0 2024-04-25T16:09:59.061 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.060+0000 7fc5a8ded640 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5625592a41c0 0x5625596c8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.069+0000 7fc5a8ded640 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5625592a41c0 0x5625596c8610 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e54e9c5882be386e server_cookie=537e91e9e744552 in_seq=0 out_seq=0 2024-04-25T16:09:59.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.069+0000 7fc583fff640 10 client.17352 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:59.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.072+0000 7fc583fff640 1 -- 172.21.15.31:0/1974428795 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56255961dee0 con 0x5625592a41c0 2024-04-25T16:09:59.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.072+0000 7fc583fff640 10 client.17352 handle_command_reply: tid=0 2024-04-25T16:09:59.076 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.072+0000 7fc5b1fe7740 1 client.17352 shutdown 2024-04-25T16:09:59.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.081+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/1974428795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 msgr2=0x5625596c1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:59.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.082+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c1b80 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fc598030ef0 tx=0x7fc5980309a0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.082+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/1974428795 shutdown_connections 2024-04-25T16:09:59.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.082+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5625592a41c0 0x5625596c8610 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.082+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c1b80 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.082+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x5625596bac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.082+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/1974428795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596ba6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.082+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/1974428795 >> 172.21.15.31:0/1974428795 conn(0x562559304300 msgr2=0x5625596ab500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:59.084 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.083+0000 7fc583fff640 0 client.17352 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:59.090 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.087+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/1974428795 shutdown_connections 2024-04-25T16:09:59.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.091+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/1974428795 wait complete. 2024-04-25T16:09:59.096 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.095+0000 7fc5b1fe7740 20 client.17352 trim_cache size 0 max 16384 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.139+0000 7fc5b1fe7740 1 Processor -- start 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.139+0000 7fc5b1fe7740 1 -- start start 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.139+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596b9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.139+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.139+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x5625596cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.139+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc59000e590 con 0x5625592ced80 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.139+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc5900100d0 con 0x5625593101b0 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.139+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc59000ec60 con 0x5625592e72f0 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.140+0000 7fc5a95ee640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x5625596cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.140+0000 7fc5a95ee640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x5625596cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:47574/0 (socket says 172.21.15.31:47574) 2024-04-25T16:09:59.145 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.140+0000 7fc5a95ee640 1 -- 172.21.15.31:0/258657682 learned_addr learned my addr 172.21.15.31:0/258657682 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.141+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/258657682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.143+0000 7fc5a8ded640 1 --2- 172.21.15.31:0/258657682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596b9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.147+0000 7fc5a95ee640 1 -- 172.21.15.31:0/258657682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 msgr2=0x5625596c1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.147+0000 7fc5a95ee640 1 --2- 172.21.15.31:0/258657682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.147+0000 7fc5a95ee640 1 -- 172.21.15.31:0/258657682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 msgr2=0x5625596b9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.147+0000 7fc5a95ee640 1 --2- 172.21.15.31:0/258657682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596b9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.147+0000 7fc5a95ee640 1 -- 172.21.15.31:0/258657682 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5625592ef710 con 0x5625593101b0 2024-04-25T16:09:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.152+0000 7fc5a95ee640 1 --2- 172.21.15.31:0/258657682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x5625596cce00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc5980095f0 tx=0x7fc598030e20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cab0f04c221fba5e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.154+0000 7fc58a7fc640 1 -- 172.21.15.31:0/258657682 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc598061070 con 0x5625593101b0 2024-04-25T16:09:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.154+0000 7fc58a7fc640 1 -- 172.21.15.31:0/258657682 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc598005460 con 0x5625593101b0 2024-04-25T16:09:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.155+0000 7fc58a7fc640 1 -- 172.21.15.31:0/258657682 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc59804c070 con 0x5625593101b0 2024-04-25T16:09:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.155+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/258657682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 msgr2=0x5625596cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:09:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.155+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/258657682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x5625596cce00 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc5980095f0 tx=0x7fc598030e20 comp rx=0 tx=0).stop 2024-04-25T16:09:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.155+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/258657682 shutdown_connections 2024-04-25T16:09:59.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.155+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/258657682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x5625596cce00 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.155+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/258657682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.155+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/258657682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596b9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.155+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/258657682 >> 172.21.15.31:0/258657682 conn(0x562559304300 msgr2=0x5625596b4a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.156+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/258657682 shutdown_connections 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.156+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/258657682 wait complete. 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.156+0000 7fc5b1fe7740 1 -- start start 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.156+0000 7fc5b1fe7740 1 Processor -- start 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.157+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.157+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.157+0000 7fc5b1fe7740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x56255972e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.157+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc59000fa00 con 0x5625592ced80 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.157+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc59000f330 con 0x5625593101b0 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.157+0000 7fc5b1fe7740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc59000c570 con 0x5625592e72f0 2024-04-25T16:09:59.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.157+0000 7fc5b1fe7740 10 client.0 _ll_register_callbacks cb 0x7fc594005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:09:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.157+0000 7fc5a3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.158+0000 7fc5a3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:57200/0 (socket says 172.21.15.31:57200) 2024-04-25T16:09:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.158+0000 7fc5a3fff640 1 -- 172.21.15.31:0/2452014294 learned_addr learned my addr 172.21.15.31:0/2452014294 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:09:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.158+0000 7fc5a8ded640 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.158+0000 7fc5a95ee640 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x56255972e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.158+0000 7fc5a3fff640 1 -- 172.21.15.31:0/2452014294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 msgr2=0x56255972e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.158+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x56255972e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.158+0000 7fc5a3fff640 1 -- 172.21.15.31:0/2452014294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 msgr2=0x5625596c0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.158+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.158+0000 7fc5a3fff640 1 -- 172.21.15.31:0/2452014294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56255972eef0 con 0x5625592e72f0 2024-04-25T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.159+0000 7fc5a3fff640 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc5940a54e0 tx=0x7fc5940a98d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=785e6a7a9bc93a9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:09:59.162 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.159+0000 7fc583fff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:09:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.159+0000 7fc583fff640 1 -- 172.21.15.31:0/2452014294 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5940a8bd0 con 0x5625592e72f0 2024-04-25T16:09:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.159+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/2452014294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc598005460 con 0x5625592e72f0 2024-04-25T16:09:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.160+0000 7fc583fff640 1 -- 172.21.15.31:0/2452014294 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc5940e2910 con 0x5625592e72f0 2024-04-25T16:09:59.163 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.160+0000 7fc583fff640 1 -- 172.21.15.31:0/2452014294 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc5940a8db0 con 0x5625592e72f0 2024-04-25T16:09:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.166+0000 7fc5b1fe7740 10 client.27031 fetch_fsmap learned FSMap version 29 2024-04-25T16:09:59.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.166+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/2452014294 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5625597300d0 con 0x5625592e72f0 2024-04-25T16:09:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.170+0000 7fc583fff640 1 -- 172.21.15.31:0/2452014294 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc598005460 con 0x5625592e72f0 2024-04-25T16:09:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.170+0000 7fc583fff640 1 -- 172.21.15.31:0/2452014294 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc5940b0b90 con 0x5625592e72f0 2024-04-25T16:09:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.170+0000 7fc5b1fe7740 10 client.27031 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:09:59.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.170+0000 7fc5b1fe7740 10 client.27031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:09:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.170+0000 7fc5b1fe7740 20 client.27031 populate_metadata read hostname 'smithi031' 2024-04-25T16:09:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.170+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5625597309c0 0x562559732de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:09:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.170+0000 7fc5b1fe7740 4 client.27031 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:09:59.172 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.170+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/2452014294 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5625595995f0 con 0x5625597309c0 2024-04-25T16:09:59.175 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.174+0000 7fc5a8ded640 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5625597309c0 0x562559732de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:09:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.181+0000 7fc5a8ded640 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5625597309c0 0x562559732de0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3afbaf42d770b22b server_cookie=861fdb1512536831 in_seq=0 out_seq=0 2024-04-25T16:09:59.183 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:09:59.182+0000 7fc583fff640 10 client.27031 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:09:59.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:59 smithi063 ceph-mon[32413]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 903 MiB/s wr, 2.15k op/s 2024-04-25T16:09:59.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:09:59 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:09:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:59 smithi186 ceph-mon[33258]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 903 MiB/s wr, 2.15k op/s 2024-04-25T16:09:59.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:09:59 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:09:59.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:59 smithi031 ceph-mon[29140]: pgmap v611: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 903 MiB/s wr, 2.15k op/s 2024-04-25T16:09:59.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:09:59 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:10:00.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:00 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:10:00.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:00 smithi063 ceph-mon[32413]: overall HEALTH_OK 2024-04-25T16:10:00.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:00 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:10:00.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:00 smithi186 ceph-mon[33258]: overall HEALTH_OK 2024-04-25T16:10:00.462 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:00 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:10:00.462 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:00 smithi031 ceph-mon[29140]: overall HEALTH_OK 2024-04-25T16:10:01.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:01 smithi063 ceph-mon[32413]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 777 MiB/s wr, 1.84k op/s 2024-04-25T16:10:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.370+0000 7fc583fff640 1 -- 172.21.15.31:0/2452014294 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5625595995f0 con 0x5625597309c0 2024-04-25T16:10:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.370+0000 7fc583fff640 10 client.27031 handle_command_reply: tid=0 2024-04-25T16:10:01.372 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.370+0000 7fc5b1fe7740 1 client.27031 shutdown 2024-04-25T16:10:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.376+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/2452014294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 msgr2=0x5625596c1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.376+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1000 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc5940a54e0 tx=0x7fc5940a98d0 comp rx=0 tx=0).stop 2024-04-25T16:10:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.376+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/2452014294 shutdown_connections 2024-04-25T16:10:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.376+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5625597309c0 0x562559732de0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.376+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5625593101b0 0x56255972e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.376+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5625592e72f0 0x5625596c1000 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.376+0000 7fc5b1fe7740 1 --2- 172.21.15.31:0/2452014294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5625592ced80 0x5625596c0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.376+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/2452014294 >> 172.21.15.31:0/2452014294 conn(0x562559304300 msgr2=0x5625596af2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:01.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.376+0000 7fc583fff640 0 client.27031 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.377+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/2452014294 shutdown_connections 2024-04-25T16:10:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.377+0000 7fc5b1fe7740 1 -- 172.21.15.31:0/2452014294 wait complete. 2024-04-25T16:10:01.379 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.377+0000 7fc5b1fe7740 20 client.27031 trim_cache size 0 max 16384 2024-04-25T16:10:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:10:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:10:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:10:01.380 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:10:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 -- 172.21.15.31:0/4080287227 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc58c0afcf0 msgr2=0x7fc58c0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc58c0afcf0 0x7fc58c0b2200 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc59802ad90 tx=0x7fc598009520 comp rx=0 tx=0).stop 2024-04-25T16:10:01.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 -- 172.21.15.31:0/4080287227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a410a0e0 msgr2=0x7fc5a4080400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a410a0e0 0x7fc5a4080400 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc59400e9c0 tx=0x7fc59400ee80 comp rx=0 tx=0).stop 2024-04-25T16:10:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 -- 172.21.15.31:0/4080287227 shutdown_connections 2024-04-25T16:10:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc58c0afcf0 0x7fc58c0b2200 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc5a410b450 0x7fc5a41b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5a410a0e0 0x7fc5a4080400 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 --2- 172.21.15.31:0/4080287227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a4078df0 0x7fc5a407fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.381+0000 7fc5a9def640 1 -- 172.21.15.31:0/4080287227 >> 172.21.15.31:0/4080287227 conn(0x7fc5a4070330 msgr2=0x7fc5a4072750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.387+0000 7fc5a9def640 1 -- 172.21.15.31:0/4080287227 shutdown_connections 2024-04-25T16:10:01.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:01.387+0000 7fc5a9def640 1 -- 172.21.15.31:0/4080287227 wait complete. 2024-04-25T16:10:01.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:01 smithi186 ceph-mon[33258]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 777 MiB/s wr, 1.84k op/s 2024-04-25T16:10:01.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:01 smithi031 ceph-mon[29140]: pgmap v612: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 777 MiB/s wr, 1.84k op/s 2024-04-25T16:10:02.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:02 smithi031 ceph-mon[29140]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 717 MiB/s wr, 1.77k op/s 2024-04-25T16:10:02.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:02 smithi063 ceph-mon[32413]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 717 MiB/s wr, 1.77k op/s 2024-04-25T16:10:02.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:02 smithi186 ceph-mon[33258]: pgmap v613: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 717 MiB/s wr, 1.77k op/s 2024-04-25T16:10:05.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:05 smithi063 ceph-mon[32413]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 867 MiB/s wr, 2.18k op/s 2024-04-25T16:10:05.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:05 smithi186 ceph-mon[33258]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 867 MiB/s wr, 2.18k op/s 2024-04-25T16:10:05.461 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:05 smithi031 ceph-mon[29140]: pgmap v614: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 867 MiB/s wr, 2.18k op/s 2024-04-25T16:10:07.315 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:07 smithi063 ceph-mon[32413]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 748 MiB/s wr, 1.72k op/s 2024-04-25T16:10:07.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:07 smithi186 ceph-mon[33258]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 748 MiB/s wr, 1.72k op/s 2024-04-25T16:10:07.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:07 smithi031 ceph-mon[29140]: pgmap v615: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 748 MiB/s wr, 1.72k op/s 2024-04-25T16:10:09.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:09 smithi063 ceph-mon[32413]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 939 MiB/s wr, 1.94k op/s 2024-04-25T16:10:09.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:09 smithi186 ceph-mon[33258]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 939 MiB/s wr, 1.94k op/s 2024-04-25T16:10:09.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:09 smithi031 ceph-mon[29140]: pgmap v616: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 939 MiB/s wr, 1.94k op/s 2024-04-25T16:10:10.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:10 smithi186 ceph-mon[33258]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 796 MiB/s wr, 1.75k op/s 2024-04-25T16:10:10.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:10 smithi031 ceph-mon[29140]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 796 MiB/s wr, 1.75k op/s 2024-04-25T16:10:10.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:10 smithi063 ceph-mon[32413]: pgmap v617: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 796 MiB/s wr, 1.75k op/s 2024-04-25T16:10:13.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:13 smithi063 ceph-mon[32413]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 742 MiB/s wr, 1.65k op/s 2024-04-25T16:10:13.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:13 smithi186 ceph-mon[33258]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 742 MiB/s wr, 1.65k op/s 2024-04-25T16:10:13.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:13 smithi031 ceph-mon[29140]: pgmap v618: 97 pgs: 97 active+clean; 10 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 742 MiB/s wr, 1.65k op/s 2024-04-25T16:10:14.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:14 smithi063 ceph-mon[32413]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 908 MiB/s wr, 2.27k op/s 2024-04-25T16:10:14.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:14 smithi186 ceph-mon[33258]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 908 MiB/s wr, 2.27k op/s 2024-04-25T16:10:14.457 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:14 smithi031 ceph-mon[29140]: pgmap v619: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 908 MiB/s wr, 2.27k op/s 2024-04-25T16:10:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.844+0000 7f9f13484640 1 -- 172.21.15.31:0/39887309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c110070 msgr2=0x7f9f0c110480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:14.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.844+0000 7f9f13484640 1 --2- 172.21.15.31:0/39887309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c110070 0x7f9f0c110480 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f9efc009900 tx=0x7f9efc02a440 comp rx=0 tx=0).stop 2024-04-25T16:10:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.856+0000 7f9f13484640 1 -- 172.21.15.31:0/39887309 shutdown_connections 2024-04-25T16:10:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.856+0000 7f9f13484640 1 --2- 172.21.15.31:0/39887309 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c110070 0x7f9f0c110480 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.856+0000 7f9f13484640 1 --2- 172.21.15.31:0/39887309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f0c10b350 0x7f9f0c10b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.856+0000 7f9f13484640 1 --2- 172.21.15.31:0/39887309 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f0c10a420 0x7f9f0c10ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.858 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.856+0000 7f9f13484640 1 -- 172.21.15.31:0/39887309 >> 172.21.15.31:0/39887309 conn(0x7f9f0c070540 msgr2=0x7f9f0c0729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:14.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.861+0000 7f9f13484640 1 -- 172.21.15.31:0/39887309 shutdown_connections 2024-04-25T16:10:14.863 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.861+0000 7f9f13484640 1 -- 172.21.15.31:0/39887309 wait complete. 2024-04-25T16:10:14.865 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.864+0000 7f9f13484640 1 Processor -- start 2024-04-25T16:10:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.867+0000 7f9f13484640 1 -- start start 2024-04-25T16:10:14.869 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.867+0000 7f9f13484640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f0c10a420 0x7f9f0c1a90f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.867+0000 7f9f13484640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c10b350 0x7f9f0c1a9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.867+0000 7f9f13484640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f0c110070 0x7f9f0c1b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.867+0000 7f9f13484640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c11e550 con 0x7f9f0c10b350 2024-04-25T16:10:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.867+0000 7f9f13484640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c11e3d0 con 0x7f9f0c110070 2024-04-25T16:10:14.870 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.867+0000 7f9f13484640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9f0c11e6d0 con 0x7f9f0c10a420 2024-04-25T16:10:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.872+0000 7f9f11a21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f0c110070 0x7f9f0c1b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.872+0000 7f9f10a1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c10b350 0x7f9f0c1a9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.872+0000 7f9f10a1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c10b350 0x7f9f0c1a9630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57752/0 (socket says 172.21.15.31:57752) 2024-04-25T16:10:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.872+0000 7f9f10a1f640 1 -- 172.21.15.31:0/2460089975 learned_addr learned my addr 172.21.15.31:0/2460089975 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:14.874 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.872+0000 7f9f11220640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f0c10a420 0x7f9f0c1a90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.880+0000 7f9f10a1f640 1 -- 172.21.15.31:0/2460089975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f0c10a420 msgr2=0x7f9f0c1a90f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:14.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.880+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f0c10a420 0x7f9f0c1a90f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.880+0000 7f9f10a1f640 1 -- 172.21.15.31:0/2460089975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f0c110070 msgr2=0x7f9f0c1b05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:14.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.880+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f0c110070 0x7f9f0c1b05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.882 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.880+0000 7f9f10a1f640 1 -- 172.21.15.31:0/2460089975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f0c111c90 con 0x7f9f0c10b350 2024-04-25T16:10:14.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.882+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c10b350 0x7f9f0c1a9630 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400ba50 tx=0x7f9f0400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.885+0000 7f9f027fc640 1 -- 172.21.15.31:0/2460089975 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04019070 con 0x7f9f0c10b350 2024-04-25T16:10:14.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.886+0000 7f9f13484640 1 -- 172.21.15.31:0/2460089975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f0c111f80 con 0x7f9f0c10b350 2024-04-25T16:10:14.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.886+0000 7f9f13484640 1 -- 172.21.15.31:0/2460089975 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f0c1124d0 con 0x7f9f0c10b350 2024-04-25T16:10:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.889+0000 7f9f027fc640 1 -- 172.21.15.31:0/2460089975 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f0400cbe0 con 0x7f9f0c10b350 2024-04-25T16:10:14.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.889+0000 7f9f027fc640 1 -- 172.21.15.31:0/2460089975 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f040075d0 con 0x7f9f0c10b350 2024-04-25T16:10:14.905 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.904+0000 7f9f027fc640 1 -- 172.21.15.31:0/2460089975 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9f0401c070 con 0x7f9f0c10b350 2024-04-25T16:10:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.908+0000 7f9f027fc640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9ef80afc70 0x7f9ef80b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.909 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.908+0000 7f9f027fc640 1 -- 172.21.15.31:0/2460089975 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9f040ddfd0 con 0x7f9f0c10b350 2024-04-25T16:10:14.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.912+0000 7f9f11220640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9ef80afc70 0x7f9ef80b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.912+0000 7f9f1a40c740 1 Processor -- start 2024-04-25T16:10:14.916 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.914+0000 7f9f1a40c740 1 -- start start 2024-04-25T16:10:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.915+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2ece970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.915+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2eceef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.915+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2ed5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.915+0000 7f9f1a40c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a5f2e34460 con 0x55a5f2b0d4d0 2024-04-25T16:10:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.915+0000 7f9f1a40c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a5f2dee590 con 0x55a5f2b2e1b0 2024-04-25T16:10:14.917 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.915+0000 7f9f1a40c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a5f2e30420 con 0x55a5f2b052f0 2024-04-25T16:10:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.917+0000 7f9f10a1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2eceef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.917+0000 7f9f10a1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2eceef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:58000/0 (socket says 172.21.15.31:58000) 2024-04-25T16:10:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.917+0000 7f9f10a1f640 1 -- 172.21.15.31:0/1923488762 learned_addr learned my addr 172.21.15.31:0/1923488762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.920+0000 7f9f11a21640 1 --2- 172.21.15.31:0/1923488762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2ed5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.921+0000 7f9f11220640 1 --2- 172.21.15.31:0/1923488762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2ece970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.921+0000 7f9f10a1f640 1 -- 172.21.15.31:0/1923488762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 msgr2=0x55a5f2ece970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:14.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.922+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/1923488762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2ece970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.922+0000 7f9f10a1f640 1 -- 172.21.15.31:0/1923488762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b0d4d0 msgr2=0x55a5f2ed5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:14.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.922+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/1923488762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2ed5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.922+0000 7f9f10a1f640 1 -- 172.21.15.31:0/1923488762 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5f2dfb0d0 con 0x55a5f2b052f0 2024-04-25T16:10:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.925+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/1923488762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2eceef0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9f04007800 tx=0x7f9f040099b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e71a7e1a5c3380b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.927+0000 7f9ef77fe640 1 -- 172.21.15.31:0/1923488762 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f040c0070 con 0x55a5f2b052f0 2024-04-25T16:10:14.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.927+0000 7f9ef77fe640 1 -- 172.21.15.31:0/1923488762 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f040a6610 con 0x55a5f2b052f0 2024-04-25T16:10:14.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.929+0000 7f9f11220640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9ef80afc70 0x7f9ef80b2180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9f080058c0 tx=0x7f9f080127c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:14.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.931+0000 7f9f1a40c740 1 -- 172.21.15.31:0/1923488762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 msgr2=0x55a5f2eceef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:14.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.931+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/1923488762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2eceef0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f9f04007800 tx=0x7f9f040099b0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.937+0000 7f9f1a40c740 1 -- 172.21.15.31:0/1923488762 shutdown_connections 2024-04-25T16:10:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.937+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/1923488762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2ed5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.937+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/1923488762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2eceef0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.937+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/1923488762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2ece970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.937+0000 7f9f1a40c740 1 -- 172.21.15.31:0/1923488762 >> 172.21.15.31:0/1923488762 conn(0x55a5f2b22300 msgr2=0x55a5f2eca300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.940+0000 7f9f1a40c740 1 -- 172.21.15.31:0/1923488762 shutdown_connections 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.940+0000 7f9f1a40c740 1 -- 172.21.15.31:0/1923488762 wait complete. 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.940+0000 7f9f1a40c740 1 -- start start 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.941+0000 7f9f1a40c740 1 Processor -- start 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.941+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2ed8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.941+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2ed8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.941+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2edfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.941+0000 7f9f1a40c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a5f2ed6890 con 0x55a5f2b2e1b0 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.941+0000 7f9f1a40c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a5f2dbbb60 con 0x55a5f2b0d4d0 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.941+0000 7f9f1a40c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a5f2ed6a10 con 0x55a5f2b052f0 2024-04-25T16:10:14.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.941+0000 7f9f1a40c740 10 client.0 _ll_register_callbacks cb 0x55a5f2de02b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.944+0000 7f9f11a21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2edfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.945+0000 7f9f11a21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2edfb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57770/0 (socket says 172.21.15.31:57770) 2024-04-25T16:10:14.946 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.945+0000 7f9f11a21640 1 -- 172.21.15.31:0/2987850321 learned_addr learned my addr 172.21.15.31:0/2987850321 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.945+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2ed8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.945+0000 7f9f11220640 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2ed8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.947+0000 7f9f11220640 1 -- 172.21.15.31:0/2987850321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b0d4d0 msgr2=0x55a5f2ed8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:14.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.947+0000 7f9f11220640 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2ed8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.947+0000 7f9f11220640 1 -- 172.21.15.31:0/2987850321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b2e1b0 msgr2=0x55a5f2edfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:14.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.948+0000 7f9f11220640 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2edfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.948+0000 7f9f11220640 1 -- 172.21.15.31:0/2987850321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5f2b46210 con 0x55a5f2b052f0 2024-04-25T16:10:14.951 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.948+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2ed8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:10:14.952 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.951+0000 7f9f11220640 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2ed8660 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c109c60 tx=0x7f9f08004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2add0fd383007ce7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.952+0000 7f9ef4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:10:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.952+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/2987850321 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f08022030 con 0x55a5f2b052f0 2024-04-25T16:10:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.952+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2987850321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f040a6610 con 0x55a5f2b052f0 2024-04-25T16:10:14.953 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.952+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/2987850321 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9f0800f840 con 0x55a5f2b052f0 2024-04-25T16:10:14.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.953+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/2987850321 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f0800fa40 con 0x55a5f2b052f0 2024-04-25T16:10:14.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.955+0000 7f9f1a40c740 10 client.27040 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:14.956 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.955+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2987850321 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a5f2c12620 con 0x55a5f2b052f0 2024-04-25T16:10:14.957 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.956+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/2987850321 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f040a6610 con 0x55a5f2b052f0 2024-04-25T16:10:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.958+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/2987850321 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f9f080313a0 con 0x55a5f2b052f0 2024-04-25T16:10:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.958+0000 7f9f1a40c740 10 client.27040 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:14.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.958+0000 7f9f1a40c740 10 client.27040 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.958+0000 7f9f1a40c740 20 client.27040 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.958+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a5f2aecd80 0x55a5f2ee68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.958+0000 7f9f1a40c740 4 client.27040 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:14.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.958+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2987850321 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a5f2e42070 con 0x55a5f2aecd80 2024-04-25T16:10:14.961 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.960+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a5f2aecd80 0x55a5f2ee68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:14.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.962+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a5f2aecd80 0x55a5f2ee68e0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5956113d5db4d683 server_cookie=daf14acb43093265 in_seq=0 out_seq=0 2024-04-25T16:10:14.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.962+0000 7f9ef4ff9640 10 client.27040 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.967+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/2987850321 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a5f2e42070 con 0x55a5f2aecd80 2024-04-25T16:10:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.967+0000 7f9ef4ff9640 10 client.27040 handle_command_reply: tid=0 2024-04-25T16:10:14.969 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.968+0000 7f9f1a40c740 1 client.27040 shutdown 2024-04-25T16:10:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.978+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2987850321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 msgr2=0x55a5f2ed8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:14.979 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.978+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2ed8660 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c109c60 tx=0x7f9f08004730 comp rx=0 tx=0).stop 2024-04-25T16:10:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.986+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2987850321 shutdown_connections 2024-04-25T16:10:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.986+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a5f2aecd80 0x55a5f2ee68e0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.987+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2edfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.987+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2ed8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.987+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2987850321 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b052f0 0x55a5f2ed8660 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.987+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2987850321 >> 172.21.15.31:0/2987850321 conn(0x55a5f2b22300 msgr2=0x55a5f2ecd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:14.988 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.987+0000 7f9ef4ff9640 0 client.27040 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:14.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.987+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2987850321 shutdown_connections 2024-04-25T16:10:14.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.987+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2987850321 wait complete. 2024-04-25T16:10:14.994 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:14.987+0000 7f9f1a40c740 20 client.27040 trim_cache size 0 max 16384 2024-04-25T16:10:15.029 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.028+0000 7f9f1a40c740 1 Processor -- start 2024-04-25T16:10:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.028+0000 7f9f1a40c740 1 -- start start 2024-04-25T16:10:15.030 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.029+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b052f0 0x55a5f2ecd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.029+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.030+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2ecd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.030+0000 7f9f1a40c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef000e400 con 0x55a5f2b052f0 2024-04-25T16:10:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.030+0000 7f9f1a40c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef000ff40 con 0x55a5f2b2e1b0 2024-04-25T16:10:15.031 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.030+0000 7f9f1a40c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef000ead0 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.030+0000 7f9f11a21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2ecd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.030+0000 7f9f11a21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2ecd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:60606/0 (socket says 172.21.15.31:60606) 2024-04-25T16:10:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.030+0000 7f9f11a21640 1 -- 172.21.15.31:0/2438808328 learned_addr learned my addr 172.21.15.31:0/2438808328 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:15.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.030+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2438808328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.030+0000 7f9f11220640 1 --2- 172.21.15.31:0/2438808328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b052f0 0x55a5f2ecd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.031+0000 7f9f10a1f640 1 -- 172.21.15.31:0/2438808328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 msgr2=0x55a5f2ecd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.031+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2438808328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2ecd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.031+0000 7f9f10a1f640 1 -- 172.21.15.31:0/2438808328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b052f0 msgr2=0x55a5f2ecd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.031+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2438808328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b052f0 0x55a5f2ecd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.031+0000 7f9f10a1f640 1 -- 172.21.15.31:0/2438808328 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a5f2aecfb0 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.031+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/2438808328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edfa90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f04007800 tx=0x7f9f040dd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bd6ab64e5c4a597 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.032+0000 7f9ef6ffd640 1 -- 172.21.15.31:0/2438808328 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f040c0040 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.032+0000 7f9ef6ffd640 1 -- 172.21.15.31:0/2438808328 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f040b9070 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.032+0000 7f9ef6ffd640 1 -- 172.21.15.31:0/2438808328 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0400cd90 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.033+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2438808328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 msgr2=0x55a5f2edfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.034+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2438808328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edfa90 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9f04007800 tx=0x7f9f040dd040 comp rx=0 tx=0).stop 2024-04-25T16:10:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.034+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2438808328 shutdown_connections 2024-04-25T16:10:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.034+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2438808328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2ecd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.034+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2438808328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edfa90 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.034+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/2438808328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b052f0 0x55a5f2ecd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.034+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2438808328 >> 172.21.15.31:0/2438808328 conn(0x55a5f2b22300 msgr2=0x55a5f2ed2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.034+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2438808328 shutdown_connections 2024-04-25T16:10:15.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.035+0000 7f9f1a40c740 1 -- 172.21.15.31:0/2438808328 wait complete. 2024-04-25T16:10:15.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.035+0000 7f9f1a40c740 1 -- start start 2024-04-25T16:10:15.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.036+0000 7f9f1a40c740 1 Processor -- start 2024-04-25T16:10:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.037+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b052f0 0x55a5f2edcc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.037+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:15.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.037+0000 7f9f1a40c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2f4cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.038+0000 7f9f10a1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:15.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.038+0000 7f9f10a1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:58018/0 (socket says 172.21.15.31:58018) 2024-04-25T16:10:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.038+0000 7f9f11a21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2f4cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.038+0000 7f9f11a21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2f4cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:60612/0 (socket says 172.21.15.31:60612) 2024-04-25T16:10:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.038+0000 7f9f10a1f640 1 -- 172.21.15.31:0/59292325 learned_addr learned my addr 172.21.15.31:0/59292325 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.038+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef000f870 con 0x55a5f2b052f0 2024-04-25T16:10:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.038+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef000f1a0 con 0x55a5f2b2e1b0 2024-04-25T16:10:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.038+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef000c490 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.039+0000 7f9f1a40c740 10 client.0 _ll_register_callbacks cb 0x7f9f0800a810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.039+0000 7f9f10a1f640 1 -- 172.21.15.31:0/59292325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 msgr2=0x55a5f2f4cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.039+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2f4cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.039+0000 7f9f10a1f640 1 -- 172.21.15.31:0/59292325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b052f0 msgr2=0x55a5f2edcc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.039+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b052f0 0x55a5f2edcc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:15.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.039+0000 7f9f10a1f640 1 -- 172.21.15.31:0/59292325 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a5f2f4d240 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.042+0000 7f9f10a1f640 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edd180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400ba20 tx=0x7f9f040a5e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=844a538ec64065ee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.043+0000 7f9ef4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:10:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.043+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/59292325 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f040c0040 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.043+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9f040b9070 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.043+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/59292325 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9f040a2c00 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.044+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/59292325 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f9f04088050 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.047+0000 7f9f1a40c740 10 client.27046 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:15.048 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.047+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a5f2f4e490 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.051 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.050+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/59292325 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9f040b9070 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.050+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/59292325 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f9f04014440 con 0x55a5f2b0d4d0 2024-04-25T16:10:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.052+0000 7f9f1a40c740 10 client.27046 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.052+0000 7f9f1a40c740 10 client.27046 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.052+0000 7f9f1a40c740 20 client.27046 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.052+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a5f2ac21c0 0x55a5f2f508c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:15.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.052+0000 7f9f1a40c740 4 client.27046 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:10:15.054 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.052+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a5f2ee6ed0 con 0x55a5f2ac21c0 2024-04-25T16:10:15.055 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.053+0000 7f9f11220640 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a5f2ac21c0 0x55a5f2f508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:15.068 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.065+0000 7f9f11220640 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a5f2ac21c0 0x55a5f2f508c0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b1587e2d8c3d4a5 server_cookie=1390c361bbd3f60d in_seq=0 out_seq=0 2024-04-25T16:10:15.072 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:15.068+0000 7f9ef4ff9640 10 client.27046 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:17.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:17 smithi063 ceph-mon[32413]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 789 MiB/s wr, 1.93k op/s 2024-04-25T16:10:17.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:17 smithi186 ceph-mon[33258]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 789 MiB/s wr, 1.93k op/s 2024-04-25T16:10:17.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:17 smithi031 ceph-mon[29140]: pgmap v620: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 789 MiB/s wr, 1.93k op/s 2024-04-25T16:10:18.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:18 smithi031 ceph-mon[29140]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 968 MiB/s wr, 2.18k op/s 2024-04-25T16:10:18.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:18 smithi063 ceph-mon[32413]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 968 MiB/s wr, 2.18k op/s 2024-04-25T16:10:18.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:18 smithi186 ceph-mon[33258]: pgmap v621: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 968 MiB/s wr, 2.18k op/s 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.292+0000 7f9ef4ff9640 1 -- 172.21.15.31:0/59292325 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a5f2ee6ed0 con 0x55a5f2ac21c0 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.292+0000 7f9ef4ff9640 10 client.27046 handle_command_reply: tid=0 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.292+0000 7f9f1a40c740 1 client.27046 shutdown 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.292+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 msgr2=0x55a5f2edd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.292+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edd180 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400ba20 tx=0x7f9f040a5e00 comp rx=0 tx=0).stop 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 shutdown_connections 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55a5f2ac21c0 0x55a5f2f508c0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a5f2b2e1b0 0x55a5f2f4cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a5f2b0d4d0 0x55a5f2edd180 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9f1a40c740 1 --2- 172.21.15.31:0/59292325 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a5f2b052f0 0x55a5f2edcc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 >> 172.21.15.31:0/59292325 conn(0x55a5f2b22300 msgr2=0x55a5f2ed2a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9ef4ff9640 0 client.27046 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:20.295 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 shutdown_connections 2024-04-25T16:10:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9f1a40c740 1 -- 172.21.15.31:0/59292325 wait complete. 2024-04-25T16:10:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.293+0000 7f9f1a40c740 20 client.27046 trim_cache size 0 max 16384 2024-04-25T16:10:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:10:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:10:20.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:10:20.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.297+0000 7f9f13484640 1 -- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9ef80afc70 msgr2=0x7f9ef80b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.298+0000 7f9f13484640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9ef80afc70 0x7f9ef80b2180 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9f080058c0 tx=0x7f9f080127c0 comp rx=0 tx=0).stop 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.298+0000 7f9f13484640 1 -- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c10b350 msgr2=0x7f9f0c1a9630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.298+0000 7f9f13484640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c10b350 0x7f9f0c1a9630 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f9f0400ba50 tx=0x7f9f0400bf10 comp rx=0 tx=0).stop 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.298+0000 7f9f13484640 1 -- 172.21.15.31:0/2460089975 shutdown_connections 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.298+0000 7f9f13484640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9ef80afc70 0x7f9ef80b2180 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.298+0000 7f9f13484640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9f0c110070 0x7f9f0c1b05c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.298+0000 7f9f13484640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9f0c10b350 0x7f9f0c1a9630 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.298+0000 7f9f13484640 1 --2- 172.21.15.31:0/2460089975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f0c10a420 0x7f9f0c1a90f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:20.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.298+0000 7f9f13484640 1 -- 172.21.15.31:0/2460089975 >> 172.21.15.31:0/2460089975 conn(0x7f9f0c070540 msgr2=0x7f9f0c10c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.307+0000 7f9f13484640 1 -- 172.21.15.31:0/2460089975 shutdown_connections 2024-04-25T16:10:20.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:20.307+0000 7f9f13484640 1 -- 172.21.15.31:0/2460089975 wait complete. 2024-04-25T16:10:21.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:21 smithi063 ceph-mon[32413]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 772 MiB/s wr, 1.86k op/s 2024-04-25T16:10:21.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:21 smithi186 ceph-mon[33258]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 772 MiB/s wr, 1.86k op/s 2024-04-25T16:10:21.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:21 smithi031 ceph-mon[29140]: pgmap v622: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 772 MiB/s wr, 1.86k op/s 2024-04-25T16:10:23.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:10:23.303 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:10:23.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:23 smithi063 ceph-mon[32413]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 750 MiB/s wr, 1.82k op/s 2024-04-25T16:10:23.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:23 smithi186 ceph-mon[33258]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 750 MiB/s wr, 1.82k op/s 2024-04-25T16:10:23.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:23 smithi031 ceph-mon[29140]: pgmap v623: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 750 MiB/s wr, 1.82k op/s 2024-04-25T16:10:23.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.894+0000 7f28f9714640 1 -- 172.21.15.31:0/2943578685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f41102b0 msgr2=0x7f28f41126d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:23.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.894+0000 7f28f9714640 1 --2- 172.21.15.31:0/2943578685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f41102b0 0x7f28f41126d0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f28ec009be0 tx=0x7f28ec02a710 comp rx=0 tx=0).stop 2024-04-25T16:10:23.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.909+0000 7f28f9714640 1 -- 172.21.15.31:0/2943578685 shutdown_connections 2024-04-25T16:10:23.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.909+0000 7f28f9714640 1 --2- 172.21.15.31:0/2943578685 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f41102b0 0x7f28f41126d0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.909+0000 7f28f9714640 1 --2- 172.21.15.31:0/2943578685 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28f410b520 0x7f28f410b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.909+0000 7f28f9714640 1 --2- 172.21.15.31:0/2943578685 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f410a320 0x7f28f410afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.909+0000 7f28f9714640 1 -- 172.21.15.31:0/2943578685 >> 172.21.15.31:0/2943578685 conn(0x7f28f4070350 msgr2=0x7f28f4072800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:23.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.912+0000 7f28f9714640 1 -- 172.21.15.31:0/2943578685 shutdown_connections 2024-04-25T16:10:23.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.915+0000 7f28f9714640 1 -- 172.21.15.31:0/2943578685 wait complete. 2024-04-25T16:10:23.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.916+0000 7f28f9714640 1 Processor -- start 2024-04-25T16:10:23.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.917+0000 7f28f9714640 1 -- start start 2024-04-25T16:10:23.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.917+0000 7f28f9714640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f410a320 0x7f28f41a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.917+0000 7f28f9714640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28f410b520 0x7f28f41a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.917+0000 7f28f9714640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f41102b0 0x7f28f41b8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.917+0000 7f28f9714640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28f4115b20 con 0x7f28f41102b0 2024-04-25T16:10:23.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.917+0000 7f28f9714640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41159a0 con 0x7f28f410a320 2024-04-25T16:10:23.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.918+0000 7f28f9714640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28f4115ca0 con 0x7f28f410b520 2024-04-25T16:10:23.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.919+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f41102b0 0x7f28f41b8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:23.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.919+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28f410b520 0x7f28f41a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:23.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.919+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f41102b0 0x7f28f41b8e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35364/0 (socket says 172.21.15.31:35364) 2024-04-25T16:10:23.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.919+0000 7f28f37fe640 1 -- 172.21.15.31:0/2902919844 learned_addr learned my addr 172.21.15.31:0/2902919844 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:23.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.922+0000 7f28f27fc640 1 -- 172.21.15.31:0/2902919844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f410a320 msgr2=0x7f28f41a0690 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:10:23.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.922+0000 7f28f27fc640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f410a320 0x7f28f41a0690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.922+0000 7f28f27fc640 1 -- 172.21.15.31:0/2902919844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f41102b0 msgr2=0x7f28f41b8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:23.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.922+0000 7f28f27fc640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f41102b0 0x7f28f41b8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.922+0000 7f28f27fc640 1 -- 172.21.15.31:0/2902919844 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f41b9370 con 0x7f28f410b520 2024-04-25T16:10:23.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.923+0000 7f28f27fc640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28f410b520 0x7f28f41a0bd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f28e4007860 tx=0x7f28e4007d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:23.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.924+0000 7f28e3fff640 1 -- 172.21.15.31:0/2902919844 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4017020 con 0x7f28f410b520 2024-04-25T16:10:23.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.924+0000 7f28e3fff640 1 -- 172.21.15.31:0/2902919844 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e4007e50 con 0x7f28f410b520 2024-04-25T16:10:23.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.925+0000 7f28e3fff640 1 -- 172.21.15.31:0/2902919844 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e4013e20 con 0x7f28f410b520 2024-04-25T16:10:23.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.934+0000 7f28f9714640 1 -- 172.21.15.31:0/2902919844 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f41b9540 con 0x7f28f410b520 2024-04-25T16:10:23.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.934+0000 7f28f9714640 1 -- 172.21.15.31:0/2902919844 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f41b99b0 con 0x7f28f410b520 2024-04-25T16:10:23.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.946+0000 7f28e3fff640 1 -- 172.21.15.31:0/2902919844 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f28e401a020 con 0x7f28f410b520 2024-04-25T16:10:23.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.947+0000 7f28e3fff640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28dc0afcf0 0x7f28dc0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.948+0000 7f28e3fff640 1 -- 172.21.15.31:0/2902919844 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f28e40de0e0 con 0x7f28f410b520 2024-04-25T16:10:23.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.954+0000 7f29005c9740 1 Processor -- start 2024-04-25T16:10:23.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.954+0000 7f29005c9740 1 -- start start 2024-04-25T16:10:23.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.955+0000 7f29005c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6757c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.955+0000 7f29005c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff67581d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.955+0000 7f29005c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 0x557ff6758e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.955+0000 7f29005c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557ff66bcd80 con 0x557ff63b6390 2024-04-25T16:10:23.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.955+0000 7f29005c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557ff66768b0 con 0x557ff63956b0 2024-04-25T16:10:23.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.955+0000 7f29005c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ff66b8d40 con 0x557ff638d4d0 2024-04-25T16:10:23.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.956+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff67581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:23.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.956+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff67581d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53880/0 (socket says 172.21.15.31:53880) 2024-04-25T16:10:23.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.956+0000 7f28f27fc640 1 -- 172.21.15.31:0/614766000 learned_addr learned my addr 172.21.15.31:0/614766000 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:23.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.956+0000 7f28f37fe640 1 --2- 172.21.15.31:0/614766000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 0x557ff6758e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.957+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/614766000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6757c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.957+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28dc0afcf0 0x7f28dc0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.958+0000 7f28f27fc640 1 -- 172.21.15.31:0/614766000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 msgr2=0x557ff6758e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.958+0000 7f28f27fc640 1 --2- 172.21.15.31:0/614766000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 0x557ff6758e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.958+0000 7f28f27fc640 1 -- 172.21.15.31:0/614766000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 msgr2=0x557ff6757c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.958+0000 7f28f27fc640 1 --2- 172.21.15.31:0/614766000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6757c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.958+0000 7f28f27fc640 1 -- 172.21.15.31:0/614766000 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ff6647a40 con 0x557ff638d4d0 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.959+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28dc0afcf0 0x7f28dc0b2200 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f28e8008fd0 tx=0x7f28e800b290 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.959+0000 7f28f27fc640 1 --2- 172.21.15.31:0/614766000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff67581d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f28e40a25f0 tx=0x7f28e40a5ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16aa4e72b35fdf25 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.960+0000 7f28e17fa640 1 -- 172.21.15.31:0/614766000 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e40c0e60 con 0x557ff638d4d0 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.960+0000 7f28e17fa640 1 -- 172.21.15.31:0/614766000 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28e40a6570 con 0x557ff638d4d0 2024-04-25T16:10:23.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.960+0000 7f28e17fa640 1 -- 172.21.15.31:0/614766000 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e40bf070 con 0x557ff638d4d0 2024-04-25T16:10:23.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.965+0000 7f29005c9740 1 -- 172.21.15.31:0/614766000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 msgr2=0x557ff67581d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:23.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.965+0000 7f29005c9740 1 --2- 172.21.15.31:0/614766000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff67581d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f28e40a25f0 tx=0x7f28e40a5ce0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.966+0000 7f29005c9740 1 -- 172.21.15.31:0/614766000 shutdown_connections 2024-04-25T16:10:23.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.966+0000 7f29005c9740 1 --2- 172.21.15.31:0/614766000 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 0x557ff6758e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.966+0000 7f29005c9740 1 --2- 172.21.15.31:0/614766000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff67581d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.966+0000 7f29005c9740 1 --2- 172.21.15.31:0/614766000 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6757c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.966+0000 7f29005c9740 1 -- 172.21.15.31:0/614766000 >> 172.21.15.31:0/614766000 conn(0x557ff63aa4e0 msgr2=0x557ff6753460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:23.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.973+0000 7f29005c9740 1 -- 172.21.15.31:0/614766000 shutdown_connections 2024-04-25T16:10:23.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.973+0000 7f29005c9740 1 -- 172.21.15.31:0/614766000 wait complete. 2024-04-25T16:10:23.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.974+0000 7f29005c9740 1 -- start start 2024-04-25T16:10:23.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.974+0000 7f29005c9740 1 Processor -- start 2024-04-25T16:10:23.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.975+0000 7f29005c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff6761b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.975+0000 7f29005c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 0x557ff6762090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.976+0000 7f29005c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6768ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.976+0000 7f29005c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557ff6759970 con 0x557ff63b6390 2024-04-25T16:10:23.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.976+0000 7f29005c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557ff6643d10 con 0x557ff63956b0 2024-04-25T16:10:23.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.976+0000 7f29005c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ff6759af0 con 0x557ff638d4d0 2024-04-25T16:10:23.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.981+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6768ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:23.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.981+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6768ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35382/0 (socket says 172.21.15.31:35382) 2024-04-25T16:10:23.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.981+0000 7f28f37fe640 1 -- 172.21.15.31:0/1608350318 learned_addr learned my addr 172.21.15.31:0/1608350318 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:23.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.981+0000 7f28f27fc640 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 0x557ff6762090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:23.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.981+0000 7f29005c9740 10 client.0 _ll_register_callbacks cb 0x557ff6670210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:23.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.983+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff6761b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:23.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.984+0000 7f28f37fe640 1 -- 172.21.15.31:0/1608350318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 msgr2=0x557ff6761b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:23.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.984+0000 7f28f37fe640 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff6761b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.984+0000 7f28f37fe640 1 -- 172.21.15.31:0/1608350318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 msgr2=0x557ff6762090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.984+0000 7f28f37fe640 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 0x557ff6762090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.985+0000 7f28f37fe640 1 -- 172.21.15.31:0/1608350318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ff64987d0 con 0x557ff63b6390 2024-04-25T16:10:23.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.986+0000 7f28f37fe640 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6768ff0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f28ec030950 tx=0x7f28ec030ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8010f0612fe96df2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:23.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.989+0000 7f28d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:10:23.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.989+0000 7f28d67fc640 1 -- 172.21.15.31:0/1608350318 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec042070 con 0x557ff63b6390 2024-04-25T16:10:23.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.990+0000 7f28d67fc640 1 -- 172.21.15.31:0/1608350318 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f28ec003970 con 0x557ff63b6390 2024-04-25T16:10:23.991 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.990+0000 7f29005c9740 1 -- 172.21.15.31:0/1608350318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28e40a6570 con 0x557ff63b6390 2024-04-25T16:10:23.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.992+0000 7f28d67fc640 1 -- 172.21.15.31:0/1608350318 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f28ec036040 con 0x557ff63b6390 2024-04-25T16:10:23.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.992+0000 7f29005c9740 10 client.17385 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:23.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.992+0000 7f29005c9740 1 -- 172.21.15.31:0/1608350318 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557ff63ce3f0 con 0x557ff63b6390 2024-04-25T16:10:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.995+0000 7f28d67fc640 1 -- 172.21.15.31:0/1608350318 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28ec003b70 con 0x557ff63b6390 2024-04-25T16:10:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.995+0000 7f28d67fc640 1 -- 172.21.15.31:0/1608350318 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f28ec031440 con 0x557ff63b6390 2024-04-25T16:10:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.995+0000 7f29005c9740 10 client.17385 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.995+0000 7f29005c9740 10 client.17385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.995+0000 7f29005c9740 20 client.17385 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.995+0000 7f29005c9740 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557ff6374f60 0x557ff676f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.995+0000 7f29005c9740 4 client.17385 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.995+0000 7f29005c9740 1 -- 172.21.15.31:0/1608350318 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f28e40a6570 con 0x557ff6374f60 2024-04-25T16:10:23.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:23.998+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557ff6374f60 0x557ff676f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.007+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557ff6374f60 0x557ff676f480 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4a9159c3ef21b3e server_cookie=ef5da0ba9a53ad13 in_seq=0 out_seq=0 2024-04-25T16:10:24.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.007+0000 7f28d67fc640 10 client.17385 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.009+0000 7f28d67fc640 1 -- 172.21.15.31:0/1608350318 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f28e40a6570 con 0x557ff6374f60 2024-04-25T16:10:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.010+0000 7f28d67fc640 10 client.17385 handle_command_reply: tid=0 2024-04-25T16:10:24.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.010+0000 7f29005c9740 1 client.17385 shutdown 2024-04-25T16:10:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.016+0000 7f29005c9740 1 -- 172.21.15.31:0/1608350318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 msgr2=0x557ff6768ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.016+0000 7f29005c9740 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6768ff0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f28ec030950 tx=0x7f28ec030ca0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.016+0000 7f29005c9740 1 -- 172.21.15.31:0/1608350318 shutdown_connections 2024-04-25T16:10:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.016+0000 7f29005c9740 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557ff6374f60 0x557ff676f480 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.016+0000 7f29005c9740 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6768ff0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.016+0000 7f29005c9740 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff63956b0 0x557ff6762090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.016+0000 7f29005c9740 1 --2- 172.21.15.31:0/1608350318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff638d4d0 0x557ff6761b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.016+0000 7f29005c9740 1 -- 172.21.15.31:0/1608350318 >> 172.21.15.31:0/1608350318 conn(0x557ff63aa4e0 msgr2=0x557ff6756370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:24.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.017+0000 7f28d67fc640 0 client.17385 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:24.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.018+0000 7f29005c9740 1 -- 172.21.15.31:0/1608350318 shutdown_connections 2024-04-25T16:10:24.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.018+0000 7f29005c9740 1 -- 172.21.15.31:0/1608350318 wait complete. 2024-04-25T16:10:24.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.018+0000 7f29005c9740 20 client.17385 trim_cache size 0 max 16384 2024-04-25T16:10:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.084+0000 7f29005c9740 1 Processor -- start 2024-04-25T16:10:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.085+0000 7f29005c9740 1 -- start start 2024-04-25T16:10:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.085+0000 7f29005c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 0x557ff67600f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.085+0000 7f29005c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6768f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.085+0000 7f29005c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6760630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.085+0000 7f29005c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41c27d0 con 0x557ff63b6390 2024-04-25T16:10:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.085+0000 7f29005c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41c4310 con 0x557ff638d4d0 2024-04-25T16:10:24.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.085+0000 7f29005c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41c2ea0 con 0x557ff63956b0 2024-04-25T16:10:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.087+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6768f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.087+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6760630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.087+0000 7f28f37fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6760630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35384/0 (socket says 172.21.15.31:35384) 2024-04-25T16:10:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.087+0000 7f28f37fe640 1 -- 172.21.15.31:0/3975860362 learned_addr learned my addr 172.21.15.31:0/3975860362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:24.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.087+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/3975860362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 0x557ff67600f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:24.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.088+0000 7f28f37fe640 1 -- 172.21.15.31:0/3975860362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 msgr2=0x557ff6768f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:24.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.088+0000 7f28f37fe640 1 --2- 172.21.15.31:0/3975860362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6768f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.088+0000 7f28f37fe640 1 -- 172.21.15.31:0/3975860362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 msgr2=0x557ff67600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:24.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.088+0000 7f28f37fe640 1 --2- 172.21.15.31:0/3975860362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 0x557ff67600f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.088+0000 7f28f37fe640 1 -- 172.21.15.31:0/3975860362 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ff6761900 con 0x557ff63b6390 2024-04-25T16:10:24.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.088+0000 7f28f37fe640 1 --2- 172.21.15.31:0/3975860362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6760630 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f28ec034040 tx=0x7f28ec000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1916081f86c566 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:24.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.107+0000 7f28e0ff9640 1 -- 172.21.15.31:0/3975860362 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28ec042070 con 0x557ff63b6390 2024-04-25T16:10:24.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.107+0000 7f28e0ff9640 1 -- 172.21.15.31:0/3975860362 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28ec031640 con 0x557ff63b6390 2024-04-25T16:10:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.109+0000 7f29005c9740 1 -- 172.21.15.31:0/3975860362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 msgr2=0x557ff6760630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.109+0000 7f29005c9740 1 --2- 172.21.15.31:0/3975860362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6760630 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f28ec034040 tx=0x7f28ec000f40 comp rx=0 tx=0).stop 2024-04-25T16:10:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.111+0000 7f29005c9740 1 -- 172.21.15.31:0/3975860362 shutdown_connections 2024-04-25T16:10:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.111+0000 7f29005c9740 1 --2- 172.21.15.31:0/3975860362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6760630 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.111+0000 7f29005c9740 1 --2- 172.21.15.31:0/3975860362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6768f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.111+0000 7f29005c9740 1 --2- 172.21.15.31:0/3975860362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 0x557ff67600f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.111+0000 7f29005c9740 1 -- 172.21.15.31:0/3975860362 >> 172.21.15.31:0/3975860362 conn(0x557ff63aa4e0 msgr2=0x557ff675c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:24.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.121+0000 7f29005c9740 1 -- 172.21.15.31:0/3975860362 shutdown_connections 2024-04-25T16:10:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.121+0000 7f29005c9740 1 -- 172.21.15.31:0/3975860362 wait complete. 2024-04-25T16:10:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.122+0000 7f29005c9740 1 -- start start 2024-04-25T16:10:24.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.122+0000 7f29005c9740 1 Processor -- start 2024-04-25T16:10:24.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.122+0000 7f29005c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 0x557ff6768cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.122+0000 7f29005c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6765840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.122+0000 7f29005c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6765d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.122+0000 7f29005c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41c3c40 con 0x557ff63b6390 2024-04-25T16:10:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.122+0000 7f29005c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41c3570 con 0x557ff638d4d0 2024-04-25T16:10:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.122+0000 7f29005c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f28f41c07b0 con 0x557ff63956b0 2024-04-25T16:10:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.122+0000 7f29005c9740 10 client.0 _ll_register_callbacks cb 0x7f28ec009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.123+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6765840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:24.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.123+0000 7f28f27fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6765840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53894/0 (socket says 172.21.15.31:53894) 2024-04-25T16:10:24.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.123+0000 7f28f27fc640 1 -- 172.21.15.31:0/1022891457 learned_addr learned my addr 172.21.15.31:0/1022891457 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:24.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.123+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 0x557ff6768cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:24.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.123+0000 7f28f27fc640 1 -- 172.21.15.31:0/1022891457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 msgr2=0x557ff6768cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.123+0000 7f28f27fc640 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 0x557ff6768cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.123+0000 7f28f27fc640 1 -- 172.21.15.31:0/1022891457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 msgr2=0x557ff6765d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.123+0000 7f28f27fc640 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6765d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.124+0000 7f28f27fc640 1 -- 172.21.15.31:0/1022891457 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ff638e0f0 con 0x557ff63956b0 2024-04-25T16:10:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.125+0000 7f28f27fc640 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6765840 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f28e400d490 tx=0x7f28e4013750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1af8a98b8aa3c224 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.125+0000 7f28d67fc640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:10:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.125+0000 7f28d67fc640 1 -- 172.21.15.31:0/1022891457 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e40b4060 con 0x557ff63956b0 2024-04-25T16:10:24.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.125+0000 7f28d67fc640 1 -- 172.21.15.31:0/1022891457 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f28e40a59e0 con 0x557ff63956b0 2024-04-25T16:10:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.128+0000 7f29005c9740 1 -- 172.21.15.31:0/1022891457 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28ec031640 con 0x557ff63956b0 2024-04-25T16:10:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.128+0000 7f28d67fc640 1 -- 172.21.15.31:0/1022891457 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f28e40c0e60 con 0x557ff63956b0 2024-04-25T16:10:24.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.129+0000 7f29005c9740 10 client.27061 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:24.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.129+0000 7f29005c9740 1 -- 172.21.15.31:0/1022891457 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557ff667d4b0 con 0x557ff63956b0 2024-04-25T16:10:24.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.130+0000 7f28d67fc640 1 -- 172.21.15.31:0/1022891457 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f28ec031640 con 0x557ff63956b0 2024-04-25T16:10:24.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.132+0000 7f28d67fc640 1 -- 172.21.15.31:0/1022891457 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f28e400dab0 con 0x557ff63956b0 2024-04-25T16:10:24.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.132+0000 7f29005c9740 10 client.27061 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:24.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.132+0000 7f29005c9740 10 client.27061 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:24.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.132+0000 7f29005c9740 20 client.27061 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:24.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.132+0000 7f29005c9740 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557ff634a3a0 0x557ff67d8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.132+0000 7f29005c9740 4 client.27061 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:10:24.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.132+0000 7f29005c9740 1 -- 172.21.15.31:0/1022891457 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557ff66ca990 con 0x557ff634a3a0 2024-04-25T16:10:24.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.141+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557ff634a3a0 0x557ff67d8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:24.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.149+0000 7f28f2ffd640 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557ff634a3a0 0x557ff67d8e00 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17d53bd6014227f5 server_cookie=6c7bf665caece15c in_seq=0 out_seq=0 2024-04-25T16:10:24.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.149+0000 7f28d67fc640 10 client.27061 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:24.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.152+0000 7f28d67fc640 1 -- 172.21.15.31:0/1022891457 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557ff66ca990 con 0x557ff634a3a0 2024-04-25T16:10:24.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.152+0000 7f28d67fc640 10 client.27061 handle_command_reply: tid=0 2024-04-25T16:10:24.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.152+0000 7f29005c9740 1 client.27061 shutdown 2024-04-25T16:10:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.160+0000 7f29005c9740 1 -- 172.21.15.31:0/1022891457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 msgr2=0x557ff6765840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:24.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.160+0000 7f29005c9740 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6765840 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f28e400d490 tx=0x7f28e4013750 comp rx=0 tx=0).stop 2024-04-25T16:10:24.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.161+0000 7f29005c9740 1 -- 172.21.15.31:0/1022891457 shutdown_connections 2024-04-25T16:10:24.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.161+0000 7f29005c9740 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557ff634a3a0 0x557ff67d8e00 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.161+0000 7f29005c9740 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557ff63b6390 0x557ff6765d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.161+0000 7f29005c9740 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557ff63956b0 0x557ff6765840 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.161+0000 7f29005c9740 1 --2- 172.21.15.31:0/1022891457 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557ff638d4d0 0x557ff6768cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.161+0000 7f29005c9740 1 -- 172.21.15.31:0/1022891457 >> 172.21.15.31:0/1022891457 conn(0x557ff63aa4e0 msgr2=0x557ff67600f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:24.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.161+0000 7f28d67fc640 0 client.27061 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.162+0000 7f29005c9740 1 -- 172.21.15.31:0/1022891457 shutdown_connections 2024-04-25T16:10:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.163+0000 7f29005c9740 1 -- 172.21.15.31:0/1022891457 wait complete. 2024-04-25T16:10:24.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.163+0000 7f29005c9740 20 client.27061 trim_cache size 0 max 16384 2024-04-25T16:10:24.165 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:10:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:10:24.166 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:10:24.166 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:10:24.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.166+0000 7f28f9714640 1 -- 172.21.15.31:0/2902919844 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28dc0afcf0 msgr2=0x7f28dc0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:24.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.166+0000 7f28f9714640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28dc0afcf0 0x7f28dc0b2200 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f28e8008fd0 tx=0x7f28e800b290 comp rx=0 tx=0).stop 2024-04-25T16:10:24.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.166+0000 7f28f9714640 1 -- 172.21.15.31:0/2902919844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28f410b520 msgr2=0x7f28f41a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:24.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.166+0000 7f28f9714640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28f410b520 0x7f28f41a0bd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f28e4007860 tx=0x7f28e4007d20 comp rx=0 tx=0).stop 2024-04-25T16:10:24.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.169+0000 7f28f9714640 1 -- 172.21.15.31:0/2902919844 shutdown_connections 2024-04-25T16:10:24.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.169+0000 7f28f9714640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f28dc0afcf0 0x7f28dc0b2200 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.169+0000 7f28f9714640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f28f41102b0 0x7f28f41b8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.169+0000 7f28f9714640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28f410b520 0x7f28f41a0bd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.169+0000 7f28f9714640 1 --2- 172.21.15.31:0/2902919844 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f28f410a320 0x7f28f41a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.169+0000 7f28f9714640 1 -- 172.21.15.31:0/2902919844 >> 172.21.15.31:0/2902919844 conn(0x7f28f4070350 msgr2=0x7f28f410c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:24.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.169+0000 7f28f9714640 1 -- 172.21.15.31:0/2902919844 shutdown_connections 2024-04-25T16:10:24.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.169+0000 7f28f9714640 1 -- 172.21.15.31:0/2902919844 wait complete. 2024-04-25T16:10:24.222 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:10:24.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:10:24.457 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:24 smithi031 ceph-mon[29140]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 921 MiB/s wr, 2.16k op/s 2024-04-25T16:10:24.515 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:24 smithi063 ceph-mon[32413]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 921 MiB/s wr, 2.16k op/s 2024-04-25T16:10:24.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.584+0000 7fc226ffd640 1 -- 172.21.15.31:0/2478227271 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22002f320 con 0x7fc228110030 2024-04-25T16:10:24.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.586+0000 7fc22e4e0640 1 -- 172.21.15.31:0/2478227271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc228110030 msgr2=0x7fc228110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:24.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.586+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/2478227271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc228110030 0x7fc228110440 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc220009ca0 tx=0x7fc22002a8d0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.586+0000 7fc22e4e0640 1 -- 172.21.15.31:0/2478227271 shutdown_connections 2024-04-25T16:10:24.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.586+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/2478227271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc228110030 0x7fc228110440 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.586+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/2478227271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc22810b310 0x7fc22810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.586+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/2478227271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc22810a420 0x7fc22810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.586+0000 7fc22e4e0640 1 -- 172.21.15.31:0/2478227271 >> 172.21.15.31:0/2478227271 conn(0x7fc2280704e0 msgr2=0x7fc228072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:24.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.589+0000 7fc22e4e0640 1 -- 172.21.15.31:0/2478227271 shutdown_connections 2024-04-25T16:10:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.592+0000 7fc22e4e0640 1 -- 172.21.15.31:0/2478227271 wait complete. 2024-04-25T16:10:24.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.595+0000 7fc22e4e0640 1 Processor -- start 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.595+0000 7fc22e4e0640 1 -- start start 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.596+0000 7fc22e4e0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc22810a420 0x7fc2281a9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.596+0000 7fc22e4e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc22810b310 0x7fc2281a9580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.596+0000 7fc22e4e0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc228110030 0x7fc22807ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.596+0000 7fc22e4e0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc22811e560 con 0x7fc22810a420 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.596+0000 7fc22e4e0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc22811e3e0 con 0x7fc228110030 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.596+0000 7fc22e4e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc22811e6e0 con 0x7fc22810b310 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.598+0000 7fc227fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc22810a420 0x7fc2281a9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.598+0000 7fc227fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc22810a420 0x7fc2281a9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35404/0 (socket says 172.21.15.31:35404) 2024-04-25T16:10:24.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.598+0000 7fc227fff640 1 -- 172.21.15.31:0/3192852432 learned_addr learned my addr 172.21.15.31:0/3192852432 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:24.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.599+0000 7fc227fff640 1 -- 172.21.15.31:0/3192852432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc22810b310 msgr2=0x7fc2281a9580 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:10:24.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.599+0000 7fc227fff640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc22810b310 0x7fc2281a9580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.599+0000 7fc227fff640 1 -- 172.21.15.31:0/3192852432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc228110030 msgr2=0x7fc22807ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:24.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.599+0000 7fc227fff640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc228110030 0x7fc22807ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:24.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.599+0000 7fc227fff640 1 -- 172.21.15.31:0/3192852432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22807b470 con 0x7fc22810a420 2024-04-25T16:10:24.603 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.601+0000 7fc227fff640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc22810a420 0x7fc2281a9000 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc21800d770 tx=0x7fc21800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:24.604 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.602+0000 7fc2257fa640 1 -- 172.21.15.31:0/3192852432 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc218017070 con 0x7fc22810a420 2024-04-25T16:10:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.602+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc22807b760 con 0x7fc22810a420 2024-04-25T16:10:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.602+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22807bd30 con 0x7fc22810a420 2024-04-25T16:10:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.603+0000 7fc2257fa640 1 -- 172.21.15.31:0/3192852432 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc218009c30 con 0x7fc22810a420 2024-04-25T16:10:24.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.603+0000 7fc2257fa640 1 -- 172.21.15.31:0/3192852432 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21800f620 con 0x7fc22810a420 2024-04-25T16:10:24.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.608+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2140052c0 con 0x7fc22810a420 2024-04-25T16:10:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.616+0000 7fc2257fa640 1 -- 172.21.15.31:0/3192852432 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc21801a070 con 0x7fc22810a420 2024-04-25T16:10:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.618+0000 7fc2257fa640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2100afd20 0x7fc2100b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:24.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.618+0000 7fc2257fa640 1 -- 172.21.15.31:0/3192852432 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc2180de620 con 0x7fc22810a420 2024-04-25T16:10:24.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.627+0000 7fc2257fa640 1 -- 172.21.15.31:0/3192852432 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc2180a72b0 con 0x7fc22810a420 2024-04-25T16:10:24.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.628+0000 7fc2277fe640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2100afd20 0x7fc2100b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:24.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:24 smithi186 ceph-mon[33258]: pgmap v624: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 921 MiB/s wr, 2.16k op/s 2024-04-25T16:10:24.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:24.655+0000 7fc2277fe640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2100afd20 0x7fc2100b2230 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc2281aa120 tx=0x7fc21c006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:25.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.345+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc2140055b0 con 0x7fc22810a420 2024-04-25T16:10:25.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.349+0000 7fc2257fa640 1 -- 172.21.15.31:0/3192852432 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7fc2180a6bb0 con 0x7fc22810a420 2024-04-25T16:10:25.352 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:10:25.352 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:10:25.352 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:10:25.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.355+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2100afd20 msgr2=0x7fc2100b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:25.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.355+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2100afd20 0x7fc2100b2230 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fc2281aa120 tx=0x7fc21c006bd0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.355+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc22810a420 msgr2=0x7fc2281a9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:25.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.355+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc22810a420 0x7fc2281a9000 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fc21800d770 tx=0x7fc21800dc30 comp rx=0 tx=0).stop 2024-04-25T16:10:25.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.357+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 shutdown_connections 2024-04-25T16:10:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.357+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc2100afd20 0x7fc2100b2230 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.357+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc228110030 0x7fc22807ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.357+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc22810b310 0x7fc2281a9580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.357+0000 7fc22e4e0640 1 --2- 172.21.15.31:0/3192852432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc22810a420 0x7fc2281a9000 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.357+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 >> 172.21.15.31:0/3192852432 conn(0x7fc2280704e0 msgr2=0x7fc22810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.358+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 shutdown_connections 2024-04-25T16:10:25.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.358+0000 7fc22e4e0640 1 -- 172.21.15.31:0/3192852432 wait complete. 2024-04-25T16:10:25.404 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:10:25.704 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:25 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3192852432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:10:25.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:25 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3192852432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:10:25.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:25 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3192852432' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:10:25.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.912+0000 7fa3c3205640 1 -- 172.21.15.31:0/3392160227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3bc10a420 msgr2=0x7fa3bc10add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:25.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.912+0000 7fa3c3205640 1 --2- 172.21.15.31:0/3392160227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3bc10a420 0x7fa3bc10add0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0009860 tx=0x7fa3b002a380 comp rx=0 tx=0).stop 2024-04-25T16:10:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.915+0000 7fa3c3205640 1 -- 172.21.15.31:0/3392160227 shutdown_connections 2024-04-25T16:10:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.915+0000 7fa3c3205640 1 --2- 172.21.15.31:0/3392160227 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3bc110030 0x7fa3bc110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.915+0000 7fa3c3205640 1 --2- 172.21.15.31:0/3392160227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3bc10b310 0x7fa3bc10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.915+0000 7fa3c3205640 1 --2- 172.21.15.31:0/3392160227 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3bc10a420 0x7fa3bc10add0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.915+0000 7fa3c3205640 1 -- 172.21.15.31:0/3392160227 >> 172.21.15.31:0/3392160227 conn(0x7fa3bc0704e0 msgr2=0x7fa3bc072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:25.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.924+0000 7fa3c3205640 1 -- 172.21.15.31:0/3392160227 shutdown_connections 2024-04-25T16:10:25.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.928+0000 7fa3c3205640 1 -- 172.21.15.31:0/3392160227 wait complete. 2024-04-25T16:10:25.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.929+0000 7fa3c3205640 1 Processor -- start 2024-04-25T16:10:25.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.935+0000 7fa3c3205640 1 -- start start 2024-04-25T16:10:25.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.938+0000 7fa3c3205640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3bc10a420 0x7fa3bc1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:25.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.938+0000 7fa3c3205640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3bc10b310 0x7fa3bc1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:25.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.938+0000 7fa3c3205640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3bc110030 0x7fa3bc1a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:25.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.938+0000 7fa3c3205640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc113d00 con 0x7fa3bc10b310 2024-04-25T16:10:25.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.938+0000 7fa3c3205640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc113b80 con 0x7fa3bc10a420 2024-04-25T16:10:25.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.938+0000 7fa3c3205640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa3bc113e80 con 0x7fa3bc110030 2024-04-25T16:10:25.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.940+0000 7fa3c17a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3bc110030 0x7fa3bc1a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:25.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.941+0000 7fa3c17a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3bc110030 0x7fa3bc1a7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53918/0 (socket says 172.21.15.31:53918) 2024-04-25T16:10:25.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.942+0000 7fa3c17a2640 1 -- 172.21.15.31:0/2560333492 learned_addr learned my addr 172.21.15.31:0/2560333492 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:25.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.942+0000 7fa3c0fa1640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3bc10a420 0x7fa3bc1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:25.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.942+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3bc10b310 0x7fa3bc1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:25.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.943+0000 7fa3c17a2640 1 -- 172.21.15.31:0/2560333492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3bc10a420 msgr2=0x7fa3bc1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:25.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.943+0000 7fa3c17a2640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3bc10a420 0x7fa3bc1a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.943+0000 7fa3c17a2640 1 -- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3bc10b310 msgr2=0x7fa3bc1a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:25.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.943+0000 7fa3c17a2640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3bc10b310 0x7fa3bc1a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.943+0000 7fa3c17a2640 1 -- 172.21.15.31:0/2560333492 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3bc1a83e0 con 0x7fa3bc110030 2024-04-25T16:10:25.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.950+0000 7fa3c17a2640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3bc110030 0x7fa3bc1a7d70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa3b401fd90 tx=0x7fa3b401d750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:25.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.952+0000 7fa3b9ffb640 1 -- 172.21.15.31:0/2560333492 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b401de60 con 0x7fa3bc110030 2024-04-25T16:10:25.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.952+0000 7fa3c3205640 1 -- 172.21.15.31:0/2560333492 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3bc1a86d0 con 0x7fa3bc110030 2024-04-25T16:10:25.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.952+0000 7fa3c3205640 1 -- 172.21.15.31:0/2560333492 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3bc1a8c20 con 0x7fa3bc110030 2024-04-25T16:10:25.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.953+0000 7fa3b9ffb640 1 -- 172.21.15.31:0/2560333492 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3b4003970 con 0x7fa3bc110030 2024-04-25T16:10:25.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.953+0000 7fa3b9ffb640 1 -- 172.21.15.31:0/2560333492 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b4004690 con 0x7fa3bc110030 2024-04-25T16:10:25.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.965+0000 7fa3ca194740 1 Processor -- start 2024-04-25T16:10:25.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.966+0000 7fa3b9ffb640 1 -- 172.21.15.31:0/2560333492 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa3b402e020 con 0x7fa3bc110030 2024-04-25T16:10:25.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.967+0000 7fa3b9ffb640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa3a40afcf0 0x7fa3a40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:25.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.968+0000 7fa3b9ffb640 1 -- 172.21.15.31:0/2560333492 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa3b40ef9c0 con 0x7fa3bc110030 2024-04-25T16:10:25.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.969+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa3a40afcf0 0x7fa3a40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:25.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.969+0000 7fa3ca194740 1 -- start start 2024-04-25T16:10:25.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.969+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891e09390 0x5638921ab810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:25.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.969+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891de04d0 0x5638921abd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:25.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.969+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de86b0 0x5638921aca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:25.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.969+0000 7fa3ca194740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56389210fd80 con 0x563891e09390 2024-04-25T16:10:25.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.969+0000 7fa3ca194740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5638920c9a20 con 0x563891de86b0 2024-04-25T16:10:25.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.970+0000 7fa3ca194740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56389210bd40 con 0x563891de04d0 2024-04-25T16:10:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.973+0000 7fa3bbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891de04d0 0x5638921abd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.973+0000 7fa3c0fa1640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891e09390 0x5638921ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:25.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.973+0000 7fa3bbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891de04d0 0x5638921abd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53934/0 (socket says 172.21.15.31:53934) 2024-04-25T16:10:25.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.973+0000 7fa3bbfff640 1 -- 172.21.15.31:0/2382251240 learned_addr learned my addr 172.21.15.31:0/2382251240 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:25.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.976+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa3a40afcf0 0x7fa3a40b2200 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc1a1940 tx=0x7fa3ac009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.978+0000 7fa3c17a2640 1 --2- 172.21.15.31:0/2382251240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de86b0 0x5638921aca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:25.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.978+0000 7fa3bbfff640 1 -- 172.21.15.31:0/2382251240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de86b0 msgr2=0x5638921aca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.978+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/2382251240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de86b0 0x5638921aca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.978+0000 7fa3bbfff640 1 -- 172.21.15.31:0/2382251240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891e09390 msgr2=0x5638921ab810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.978+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/2382251240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891e09390 0x5638921ab810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.978+0000 7fa3bbfff640 1 -- 172.21.15.31:0/2382251240 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638920d4c30 con 0x563891de04d0 2024-04-25T16:10:25.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.983+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/2382251240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891de04d0 0x5638921abd90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac006dd0 tx=0x7fa3ac019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ac7da1f826388ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.987+0000 7fa3a2ffd640 1 -- 172.21.15.31:0/2382251240 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac015dd0 con 0x563891de04d0 2024-04-25T16:10:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.987+0000 7fa3a2ffd640 1 -- 172.21.15.31:0/2382251240 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3ac023070 con 0x563891de04d0 2024-04-25T16:10:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.987+0000 7fa3a2ffd640 1 -- 172.21.15.31:0/2382251240 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac01e670 con 0x563891de04d0 2024-04-25T16:10:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.989+0000 7fa3ca194740 1 -- 172.21.15.31:0/2382251240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891de04d0 msgr2=0x5638921abd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.989+0000 7fa3ca194740 1 --2- 172.21.15.31:0/2382251240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891de04d0 0x5638921abd90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac006dd0 tx=0x7fa3ac019000 comp rx=0 tx=0).stop 2024-04-25T16:10:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.995+0000 7fa3ca194740 1 -- 172.21.15.31:0/2382251240 shutdown_connections 2024-04-25T16:10:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.995+0000 7fa3ca194740 1 --2- 172.21.15.31:0/2382251240 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de86b0 0x5638921aca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.995+0000 7fa3ca194740 1 --2- 172.21.15.31:0/2382251240 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891de04d0 0x5638921abd90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.995+0000 7fa3ca194740 1 --2- 172.21.15.31:0/2382251240 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891e09390 0x5638921ab810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:25.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:25.995+0000 7fa3ca194740 1 -- 172.21.15.31:0/2382251240 >> 172.21.15.31:0/2382251240 conn(0x563891dfd4e0 msgr2=0x5638921a7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.001+0000 7fa3ca194740 1 -- 172.21.15.31:0/2382251240 shutdown_connections 2024-04-25T16:10:26.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.002+0000 7fa3ca194740 1 -- 172.21.15.31:0/2382251240 wait complete. 2024-04-25T16:10:26.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.002+0000 7fa3ca194740 1 -- start start 2024-04-25T16:10:26.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.002+0000 7fa3ca194740 1 Processor -- start 2024-04-25T16:10:26.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.006+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.006+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.006+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921bc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.007+0000 7fa3ca194740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5638921ad4d0 con 0x563891de86b0 2024-04-25T16:10:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.007+0000 7fa3ca194740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563892096dc0 con 0x563891de04d0 2024-04-25T16:10:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.007+0000 7fa3ca194740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5638921ad650 con 0x563891e09390 2024-04-25T16:10:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.007+0000 7fa3ca194740 10 client.0 _ll_register_callbacks cb 0x5638920b2600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.007+0000 7fa3c0fa1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.007+0000 7fa3c0fa1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58034/0 (socket says 172.21.15.31:58034) 2024-04-25T16:10:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.007+0000 7fa3c0fa1640 1 -- 172.21.15.31:0/487787716 learned_addr learned my addr 172.21.15.31:0/487787716 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:26.011 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.009+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.012+0000 7fa3c17a2640 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921bc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.012+0000 7fa3c0fa1640 1 -- 172.21.15.31:0/487787716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 msgr2=0x5638921bc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.012+0000 7fa3c0fa1640 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921bc9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.012+0000 7fa3c0fa1640 1 -- 172.21.15.31:0/487787716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 msgr2=0x5638921b5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.012+0000 7fa3c0fa1640 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921b5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.015 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.012+0000 7fa3c0fa1640 1 -- 172.21.15.31:0/487787716 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563891e213f0 con 0x563891de04d0 2024-04-25T16:10:26.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.014+0000 7fa3c0fa1640 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b5540 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0000c00 tx=0x7fa3b0030d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e728dac4b9c89d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:26.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.015+0000 7fa39bfff640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:10:26.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.018+0000 7fa39bfff640 1 -- 172.21.15.31:0/487787716 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0041070 con 0x563891de04d0 2024-04-25T16:10:26.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.019+0000 7fa39bfff640 1 -- 172.21.15.31:0/487787716 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa3b0004240 con 0x563891de04d0 2024-04-25T16:10:26.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.021+0000 7fa3ca194740 1 -- 172.21.15.31:0/487787716 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3ac023070 con 0x563891de04d0 2024-04-25T16:10:26.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.023+0000 7fa39bfff640 1 -- 172.21.15.31:0/487787716 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa3b0037040 con 0x563891de04d0 2024-04-25T16:10:26.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.026+0000 7fa3ca194740 10 client.27074 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:26.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.026+0000 7fa3ca194740 1 -- 172.21.15.31:0/487787716 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563891eeb7d0 con 0x563891de04d0 2024-04-25T16:10:26.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.029+0000 7fa39bfff640 1 -- 172.21.15.31:0/487787716 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3ac023070 con 0x563891de04d0 2024-04-25T16:10:26.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.029+0000 7fa39bfff640 1 -- 172.21.15.31:0/487787716 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa3b00049e0 con 0x563891de04d0 2024-04-25T16:10:26.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.030+0000 7fa3ca194740 10 client.27074 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:26.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.030+0000 7fa3ca194740 10 client.27074 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:10:26.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.030+0000 7fa3ca194740 20 client.27074 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:26.032 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.030+0000 7fa3ca194740 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x563891dc7f60 0x5638921c32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.030+0000 7fa3ca194740 4 client.27074 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:26.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.030+0000 7fa3ca194740 1 -- 172.21.15.31:0/487787716 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56389211d990 con 0x563891dc7f60 2024-04-25T16:10:26.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.031+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x563891dc7f60 0x5638921c32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.033+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x563891dc7f60 0x5638921c32a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7b11b69823909811 server_cookie=197947c96ccb3887 in_seq=0 out_seq=0 2024-04-25T16:10:26.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.035+0000 7fa39bfff640 10 client.27074 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:10:26.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.037+0000 7fa39bfff640 1 -- 172.21.15.31:0/487787716 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56389211d990 con 0x563891dc7f60 2024-04-25T16:10:26.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.037+0000 7fa39bfff640 10 client.27074 handle_command_reply: tid=0 2024-04-25T16:10:26.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.038+0000 7fa3ca194740 1 client.27074 shutdown 2024-04-25T16:10:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.044+0000 7fa3ca194740 1 -- 172.21.15.31:0/487787716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 msgr2=0x5638921b5540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.044+0000 7fa3ca194740 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b5540 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0000c00 tx=0x7fa3b0030d70 comp rx=0 tx=0).stop 2024-04-25T16:10:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.044+0000 7fa3ca194740 1 -- 172.21.15.31:0/487787716 shutdown_connections 2024-04-25T16:10:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.044+0000 7fa3ca194740 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x563891dc7f60 0x5638921c32a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.044+0000 7fa3ca194740 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921bc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.044+0000 7fa3ca194740 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921b5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.044+0000 7fa3ca194740 1 --2- 172.21.15.31:0/487787716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b5540 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.044+0000 7fa3ca194740 1 -- 172.21.15.31:0/487787716 >> 172.21.15.31:0/487787716 conn(0x563891dfd4e0 msgr2=0x5638921a9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.045+0000 7fa39bfff640 0 client.27074 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:10:26.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.045+0000 7fa3ca194740 1 -- 172.21.15.31:0/487787716 shutdown_connections 2024-04-25T16:10:26.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.045+0000 7fa3ca194740 1 -- 172.21.15.31:0/487787716 wait complete. 2024-04-25T16:10:26.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.045+0000 7fa3ca194740 20 client.27074 trim_cache size 0 max 16384 2024-04-25T16:10:26.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.086+0000 7fa3ca194740 1 Processor -- start 2024-04-25T16:10:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.089+0000 7fa3ca194740 1 -- start start 2024-04-25T16:10:26.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.089+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921a70c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.091+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921bc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.091+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921b5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.092 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.091+0000 7fa3ca194740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a800e550 con 0x563891de86b0 2024-04-25T16:10:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.091+0000 7fa3c0fa1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921a70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.091+0000 7fa3c0fa1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921a70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58046/0 (socket says 172.21.15.31:58046) 2024-04-25T16:10:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.091+0000 7fa3c0fa1640 1 -- 172.21.15.31:0/3864930955 learned_addr learned my addr 172.21.15.31:0/3864930955 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.093+0000 7fa3c17a2640 1 --2- 172.21.15.31:0/3864930955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921b5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.093+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/3864930955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921bc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.091+0000 7fa3ca194740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a8010090 con 0x563891de04d0 2024-04-25T16:10:26.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.096+0000 7fa3ca194740 1 -- 172.21.15.31:0/3864930955 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a800ec20 con 0x563891e09390 2024-04-25T16:10:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.097+0000 7fa3c0fa1640 1 -- 172.21.15.31:0/3864930955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 msgr2=0x5638921b5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.097+0000 7fa3c0fa1640 1 --2- 172.21.15.31:0/3864930955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921b5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.097+0000 7fa3c0fa1640 1 -- 172.21.15.31:0/3864930955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 msgr2=0x5638921bc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.097+0000 7fa3c0fa1640 1 --2- 172.21.15.31:0/3864930955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921bc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.097+0000 7fa3c0fa1640 1 -- 172.21.15.31:0/3864930955 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638921bce40 con 0x563891de04d0 2024-04-25T16:10:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.097+0000 7fa3c0fa1640 1 --2- 172.21.15.31:0/3864930955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921a70c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0048040 tx=0x7fa3b0009720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=575bd6333f29a598 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.098+0000 7fa3a27fc640 1 -- 172.21.15.31:0/3864930955 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0053070 con 0x563891de04d0 2024-04-25T16:10:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.098+0000 7fa3a27fc640 1 -- 172.21.15.31:0/3864930955 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3b0003840 con 0x563891de04d0 2024-04-25T16:10:26.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.098+0000 7fa3a27fc640 1 -- 172.21.15.31:0/3864930955 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0041070 con 0x563891de04d0 2024-04-25T16:10:26.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.100+0000 7fa3ca194740 1 -- 172.21.15.31:0/3864930955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 msgr2=0x5638921a70c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.101+0000 7fa3ca194740 1 --2- 172.21.15.31:0/3864930955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921a70c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0048040 tx=0x7fa3b0009720 comp rx=0 tx=0).stop 2024-04-25T16:10:26.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.102+0000 7fa3ca194740 1 -- 172.21.15.31:0/3864930955 shutdown_connections 2024-04-25T16:10:26.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.102+0000 7fa3ca194740 1 --2- 172.21.15.31:0/3864930955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921b5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.102+0000 7fa3ca194740 1 --2- 172.21.15.31:0/3864930955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921bc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.102+0000 7fa3ca194740 1 --2- 172.21.15.31:0/3864930955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921a70c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.102+0000 7fa3ca194740 1 -- 172.21.15.31:0/3864930955 >> 172.21.15.31:0/3864930955 conn(0x563891dfd4e0 msgr2=0x5638921b5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.107+0000 7fa3ca194740 1 -- 172.21.15.31:0/3864930955 shutdown_connections 2024-04-25T16:10:26.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.109+0000 7fa3ca194740 1 -- 172.21.15.31:0/3864930955 wait complete. 2024-04-25T16:10:26.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.110+0000 7fa3ca194740 1 -- start start 2024-04-25T16:10:26.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.110+0000 7fa3ca194740 1 Processor -- start 2024-04-25T16:10:26.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.111+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b9ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.111+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921ba010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3ca194740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921baef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3c0fa1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b9ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3c0fa1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b9ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58048/0 (socket says 172.21.15.31:58048) 2024-04-25T16:10:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3bbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921ba010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3bbfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921ba010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35454/0 (socket says 172.21.15.31:35454) 2024-04-25T16:10:26.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3ca194740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a800f9c0 con 0x563891de86b0 2024-04-25T16:10:26.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3ca194740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a800f2f0 con 0x563891de04d0 2024-04-25T16:10:26.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3ca194740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa3a800c530 con 0x563891e09390 2024-04-25T16:10:26.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3c0fa1640 1 -- 172.21.15.31:0/4205372658 learned_addr learned my addr 172.21.15.31:0/4205372658 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:26.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.112+0000 7fa3ca194740 10 client.0 _ll_register_callbacks cb 0x7fa3b401a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.113+0000 7fa3bbfff640 1 -- 172.21.15.31:0/4205372658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 msgr2=0x5638921baef0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.113+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921baef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.113+0000 7fa3bbfff640 1 -- 172.21.15.31:0/4205372658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 msgr2=0x5638921b9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.113+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b9ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.113+0000 7fa3bbfff640 1 -- 172.21.15.31:0/4205372658 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638921bb4b0 con 0x563891de86b0 2024-04-25T16:10:26.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.114+0000 7fa3bbfff640 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921ba010 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac006da0 tx=0x7fa3ac0093c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11eb1e387e6df9dd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.123+0000 7fa39bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:10:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.123+0000 7fa39bfff640 1 -- 172.21.15.31:0/4205372658 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3ac0315f0 con 0x563891de86b0 2024-04-25T16:10:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.123+0000 7fa39bfff640 1 -- 172.21.15.31:0/4205372658 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa3ac031790 con 0x563891de86b0 2024-04-25T16:10:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.124+0000 7fa3ca194740 1 -- 172.21.15.31:0/4205372658 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa3b0003840 con 0x563891de86b0 2024-04-25T16:10:26.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.126+0000 7fa39bfff640 1 -- 172.21.15.31:0/4205372658 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa3ac031990 con 0x563891de86b0 2024-04-25T16:10:26.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.126+0000 7fa3ca194740 10 client.17409 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:26.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.126+0000 7fa3ca194740 1 -- 172.21.15.31:0/4205372658 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5638921bc7e0 con 0x563891de86b0 2024-04-25T16:10:26.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.139+0000 7fa39bfff640 1 -- 172.21.15.31:0/4205372658 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3ac023040 con 0x563891de86b0 2024-04-25T16:10:26.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.140+0000 7fa39bfff640 1 -- 172.21.15.31:0/4205372658 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa3ac03a470 con 0x563891de86b0 2024-04-25T16:10:26.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.141+0000 7fa3ca194740 10 client.17409 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:26.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.141+0000 7fa3ca194740 10 client.17409 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:10:26.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.141+0000 7fa3ca194740 20 client.17409 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:26.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.141+0000 7fa3ca194740 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x563891d9d3a0 0x56389222d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.141+0000 7fa3ca194740 4 client.17409 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:10:26.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.141+0000 7fa3ca194740 1 -- 172.21.15.31:0/4205372658 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fa3b0003840 con 0x563891d9d3a0 2024-04-25T16:10:26.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.143+0000 7fa3c0fa1640 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x563891d9d3a0 0x56389222d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.148+0000 7fa3c0fa1640 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x563891d9d3a0 0x56389222d9a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=330541446feba2ef server_cookie=b8af15d6facc1fc7 in_seq=0 out_seq=0 2024-04-25T16:10:26.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.148+0000 7fa39bfff640 10 client.17409 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:10:26.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.151+0000 7fa39bfff640 1 -- 172.21.15.31:0/4205372658 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fa3b0003840 con 0x563891d9d3a0 2024-04-25T16:10:26.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.151+0000 7fa39bfff640 10 client.17409 handle_command_reply: tid=0 2024-04-25T16:10:26.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.151+0000 7fa3ca194740 1 client.17409 shutdown 2024-04-25T16:10:26.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.152+0000 7fa3ca194740 1 -- 172.21.15.31:0/4205372658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 msgr2=0x5638921ba010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.152+0000 7fa3ca194740 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921ba010 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa3ac006da0 tx=0x7fa3ac0093c0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.152+0000 7fa3ca194740 1 -- 172.21.15.31:0/4205372658 shutdown_connections 2024-04-25T16:10:26.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.152+0000 7fa3ca194740 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x563891d9d3a0 0x56389222d9a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.152+0000 7fa3ca194740 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563891e09390 0x5638921baef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.152+0000 7fa3ca194740 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563891de86b0 0x5638921ba010 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.152+0000 7fa3ca194740 1 --2- 172.21.15.31:0/4205372658 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563891de04d0 0x5638921b9ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.152+0000 7fa3ca194740 1 -- 172.21.15.31:0/4205372658 >> 172.21.15.31:0/4205372658 conn(0x563891dfd4e0 msgr2=0x5638921aa5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.153+0000 7fa39bfff640 0 client.17409 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:10:26.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.157+0000 7fa3ca194740 1 -- 172.21.15.31:0/4205372658 shutdown_connections 2024-04-25T16:10:26.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.157+0000 7fa3ca194740 1 -- 172.21.15.31:0/4205372658 wait complete. 2024-04-25T16:10:26.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.157+0000 7fa3ca194740 20 client.17409 trim_cache size 0 max 16384 2024-04-25T16:10:26.163 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 -- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa3a40afcf0 msgr2=0x7fa3a40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa3a40afcf0 0x7fa3a40b2200 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa3bc1a1940 tx=0x7fa3ac009110 comp rx=0 tx=0).stop 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 -- 172.21.15.31:0/2560333492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3bc110030 msgr2=0x7fa3bc1a7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3bc110030 0x7fa3bc1a7d70 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa3b401fd90 tx=0x7fa3b401d750 comp rx=0 tx=0).stop 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 -- 172.21.15.31:0/2560333492 shutdown_connections 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa3a40afcf0 0x7fa3a40b2200 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3bc110030 0x7fa3bc1a7d70 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa3bc10b310 0x7fa3bc1a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 --2- 172.21.15.31:0/2560333492 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa3bc10a420 0x7fa3bc1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.169+0000 7fa3c3205640 1 -- 172.21.15.31:0/2560333492 >> 172.21.15.31:0/2560333492 conn(0x7fa3bc0704e0 msgr2=0x7fa3bc10e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.173+0000 7fa3c3205640 1 -- 172.21.15.31:0/2560333492 shutdown_connections 2024-04-25T16:10:26.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.176+0000 7fa3c3205640 1 -- 172.21.15.31:0/2560333492 wait complete. 2024-04-25T16:10:26.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:10:26.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:26 smithi031 ceph-mon[29140]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 780 MiB/s wr, 1.55k op/s 2024-04-25T16:10:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.771+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/441810264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410b190 msgr2=0x7f8ae410b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.771+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/441810264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410b190 0x7f8ae410b5a0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f8adc009ca0 tx=0x7f8adc02a8e0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.772+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/441810264 shutdown_connections 2024-04-25T16:10:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.772+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/441810264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ae410fe90 0x7f8ae41102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.772+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/441810264 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410b190 0x7f8ae410b5a0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.772+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/441810264 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ae410a340 0x7f8ae410ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.772+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/441810264 >> 172.21.15.31:0/441810264 conn(0x7f8ae4070320 msgr2=0x7f8ae40727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.773+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/441810264 shutdown_connections 2024-04-25T16:10:26.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.775+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/441810264 wait complete. 2024-04-25T16:10:26.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.777+0000 7f8ae8a2f640 1 Processor -- start 2024-04-25T16:10:26.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.780+0000 7f8ae8a2f640 1 -- start start 2024-04-25T16:10:26.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.781+0000 7f8ae8a2f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ae410a340 0x7f8ae4117280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.781+0000 7f8ae8a2f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410fe90 0x7f8ae41177c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.781+0000 7f8ae8a2f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ae41a5970 0x7f8ae41a7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.781+0000 7f8ae8a2f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae411e280 con 0x7f8ae410fe90 2024-04-25T16:10:26.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.781+0000 7f8ae8a2f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae411e100 con 0x7f8ae41a5970 2024-04-25T16:10:26.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.781+0000 7f8ae8a2f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae411e400 con 0x7f8ae410a340 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.781+0000 7f8ae259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410fe90 0x7f8ae41177c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.782+0000 7f8ae259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410fe90 0x7f8ae41177c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35480/0 (socket says 172.21.15.31:35480) 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.782+0000 7f8ae259c640 1 -- 172.21.15.31:0/168244170 learned_addr learned my addr 172.21.15.31:0/168244170 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.782+0000 7f8ae359e640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ae41a5970 0x7f8ae41a7d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.783+0000 7f8ae259c640 1 -- 172.21.15.31:0/168244170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ae410a340 msgr2=0x7f8ae4117280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.783+0000 7f8ae259c640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ae410a340 0x7f8ae4117280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.783+0000 7f8ae259c640 1 -- 172.21.15.31:0/168244170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ae41a5970 msgr2=0x7f8ae41a7d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.783+0000 7f8ae259c640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ae41a5970 0x7f8ae41a7d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.783+0000 7f8ae259c640 1 -- 172.21.15.31:0/168244170 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ae41a82d0 con 0x7f8ae410fe90 2024-04-25T16:10:26.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.784+0000 7f8ae259c640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410fe90 0x7f8ae41177c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8adc009dc0 tx=0x7f8adc003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:26.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.785+0000 7f8acbfff640 1 -- 172.21.15.31:0/168244170 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8adc004040 con 0x7f8ae410fe90 2024-04-25T16:10:26.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.785+0000 7f8acbfff640 1 -- 172.21.15.31:0/168244170 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8adc004690 con 0x7f8ae410fe90 2024-04-25T16:10:26.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.787+0000 7f8acbfff640 1 -- 172.21.15.31:0/168244170 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8adc03a400 con 0x7f8ae410fe90 2024-04-25T16:10:26.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.790+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/168244170 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ae41a8560 con 0x7f8ae410fe90 2024-04-25T16:10:26.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.790+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/168244170 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ae41a8a30 con 0x7f8ae410fe90 2024-04-25T16:10:26.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.796+0000 7f8af01b4740 1 Processor -- start 2024-04-25T16:10:26.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.801+0000 7f8af01b4740 1 -- start start 2024-04-25T16:10:26.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.801+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.801+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a6f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.801+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 0x55d80b3a7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.801+0000 7f8af01b4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d80b30ad80 con 0x55d80afdb4d0 2024-04-25T16:10:26.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.801+0000 7f8af01b4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d80b2c4a20 con 0x55d80afe36b0 2024-04-25T16:10:26.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.801+0000 7f8af01b4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d80b306d40 con 0x55d80b004390 2024-04-25T16:10:26.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.806+0000 7f8ae359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.807+0000 7f8ae359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a69e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:53980/0 (socket says 172.21.15.31:53980) 2024-04-25T16:10:26.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.807+0000 7f8ae359e640 1 -- 172.21.15.31:0/448173107 learned_addr learned my addr 172.21.15.31:0/448173107 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:26.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.807+0000 7f8acbfff640 1 -- 172.21.15.31:0/168244170 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8adc0305e0 con 0x7f8ae410fe90 2024-04-25T16:10:26.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.808+0000 7f8ae2d9d640 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.809+0000 7f8acbfff640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8acc0afb50 0x7f8acc0b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.810+0000 7f8acbfff640 1 -- 172.21.15.31:0/168244170 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8adc0ffd60 con 0x7f8ae410fe90 2024-04-25T16:10:26.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.811+0000 7f8ae259c640 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 0x55d80b3a7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.815+0000 7f8ae359e640 1 -- 172.21.15.31:0/448173107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 msgr2=0x55d80b3a7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.815+0000 7f8ae359e640 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 0x55d80b3a7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.815+0000 7f8ae359e640 1 -- 172.21.15.31:0/448173107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 msgr2=0x55d80b3a6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.815+0000 7f8ae359e640 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.815+0000 7f8ae359e640 1 -- 172.21.15.31:0/448173107 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d80b2cfc30 con 0x55d80b004390 2024-04-25T16:10:26.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.822+0000 7f8ae359e640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8acc0afb50 0x7f8acc0b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.822+0000 7f8ae359e640 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a69e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8ad400acc0 tx=0x7f8ad400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ac4603059fe5548 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:26.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.823+0000 7f8ae259c640 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 0x55d80b3a7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:10:26.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.825+0000 7f8ac97fa640 1 -- 172.21.15.31:0/448173107 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad4012a20 con 0x55d80b004390 2024-04-25T16:10:26.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.825+0000 7f8ac97fa640 1 -- 172.21.15.31:0/448173107 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ad4012c40 con 0x55d80b004390 2024-04-25T16:10:26.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.825+0000 7f8ac97fa640 1 -- 172.21.15.31:0/448173107 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40196d0 con 0x55d80b004390 2024-04-25T16:10:26.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.826+0000 7f8af01b4740 1 -- 172.21.15.31:0/448173107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 msgr2=0x55d80b3a69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.826+0000 7f8af01b4740 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a69e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8ad400acc0 tx=0x7f8ad400f790 comp rx=0 tx=0).stop 2024-04-25T16:10:26.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.829+0000 7f8af01b4740 1 -- 172.21.15.31:0/448173107 shutdown_connections 2024-04-25T16:10:26.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.829+0000 7f8af01b4740 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 0x55d80b3a7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.829+0000 7f8af01b4740 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.829+0000 7f8af01b4740 1 --2- 172.21.15.31:0/448173107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a69e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.829+0000 7f8af01b4740 1 -- 172.21.15.31:0/448173107 >> 172.21.15.31:0/448173107 conn(0x55d80aff84e0 msgr2=0x55d80b3a21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.832+0000 7f8ae359e640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8acc0afb50 0x7f8acc0b2060 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8ad401f000 tx=0x7f8ad401c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:26.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.835+0000 7f8af01b4740 1 -- 172.21.15.31:0/448173107 shutdown_connections 2024-04-25T16:10:26.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.836+0000 7f8af01b4740 1 -- 172.21.15.31:0/448173107 wait complete. 2024-04-25T16:10:26.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.837+0000 7f8af01b4740 1 -- start start 2024-04-25T16:10:26.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.837+0000 7f8af01b4740 1 Processor -- start 2024-04-25T16:10:26.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.838+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afdb4d0 0x55d80b3b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.838+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.838+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.838+0000 7f8af01b4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d80b3a8700 con 0x55d80afe36b0 2024-04-25T16:10:26.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.838+0000 7f8af01b4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d80b291dc0 con 0x55d80afdb4d0 2024-04-25T16:10:26.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.838+0000 7f8af01b4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d80b3a8880 con 0x55d80b004390 2024-04-25T16:10:26.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.838+0000 7f8af01b4740 10 client.0 _ll_register_callbacks cb 0x55d80b2ad600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:26.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.839+0000 7f8ae259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.839+0000 7f8ae259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35486/0 (socket says 172.21.15.31:35486) 2024-04-25T16:10:26.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.839+0000 7f8ae259c640 1 -- 172.21.15.31:0/2800132677 learned_addr learned my addr 172.21.15.31:0/2800132677 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:26.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.839+0000 7f8ae259c640 1 -- 172.21.15.31:0/2800132677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 msgr2=0x55d80b3b7b60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:26.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.839+0000 7f8ae259c640 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3b7b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.839+0000 7f8ae259c640 1 -- 172.21.15.31:0/2800132677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afdb4d0 msgr2=0x55d80b3b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.839+0000 7f8ae259c640 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afdb4d0 0x55d80b3b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.839+0000 7f8ae259c640 1 -- 172.21.15.31:0/2800132677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d80b01c3f0 con 0x55d80afe36b0 2024-04-25T16:10:26.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.841+0000 7f8ae259c640 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b0c00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8adc03a020 tx=0x7f8adc0c7c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67d5e16a7666427 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:26.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.841+0000 7f8ac6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:10:26.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.842+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/2800132677 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0c7d00 con 0x55d80afe36b0 2024-04-25T16:10:26.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.843+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/2800132677 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8adc0c6140 con 0x55d80afe36b0 2024-04-25T16:10:26.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.843+0000 7f8af01b4740 1 -- 172.21.15.31:0/2800132677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ad4012c40 con 0x55d80afe36b0 2024-04-25T16:10:26.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.843+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/2800132677 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8adc0c88f0 con 0x55d80afe36b0 2024-04-25T16:10:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.851+0000 7f8af01b4740 10 client.17421 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:26.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.851+0000 7f8af01b4740 1 -- 172.21.15.31:0/2800132677 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d80b0e67d0 con 0x55d80afe36b0 2024-04-25T16:10:26.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.856+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/2800132677 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ad4012c40 con 0x55d80afe36b0 2024-04-25T16:10:26.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.856+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/2800132677 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8adc0c7450 con 0x55d80afe36b0 2024-04-25T16:10:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.860+0000 7f8af01b4740 10 client.17421 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.860+0000 7f8af01b4740 10 client.17421 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:10:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.860+0000 7f8af01b4740 20 client.17421 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.860+0000 7f8af01b4740 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55d80afc2f60 0x55d80b3be320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.860+0000 7f8af01b4740 4 client.17421 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:26.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.860+0000 7f8af01b4740 1 -- 172.21.15.31:0/2800132677 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d80b318990 con 0x55d80afc2f60 2024-04-25T16:10:26.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.860+0000 7f8ae2d9d640 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55d80afc2f60 0x55d80b3be320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.865+0000 7f8ae2d9d640 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55d80afc2f60 0x55d80b3be320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c51d4606f5513af server_cookie=a0b608de9b2b1ddb in_seq=0 out_seq=0 2024-04-25T16:10:26.867 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.865+0000 7f8ac6ff5640 10 client.17421 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:10:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.867+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/2800132677 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d80b318990 con 0x55d80afc2f60 2024-04-25T16:10:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.867+0000 7f8ac6ff5640 10 client.17421 handle_command_reply: tid=0 2024-04-25T16:10:26.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8af01b4740 1 client.17421 shutdown 2024-04-25T16:10:26.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:26 smithi063 ceph-mon[32413]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 780 MiB/s wr, 1.55k op/s 2024-04-25T16:10:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8af01b4740 1 -- 172.21.15.31:0/2800132677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 msgr2=0x55d80b3b0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8af01b4740 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b0c00 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f8adc03a020 tx=0x7f8adc0c7c00 comp rx=0 tx=0).stop 2024-04-25T16:10:26.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8af01b4740 1 -- 172.21.15.31:0/2800132677 shutdown_connections 2024-04-25T16:10:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8af01b4740 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55d80afc2f60 0x55d80b3be320 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8af01b4740 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8af01b4740 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b0c00 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8af01b4740 1 --2- 172.21.15.31:0/2800132677 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afdb4d0 0x55d80b3b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8af01b4740 1 -- 172.21.15.31:0/2800132677 >> 172.21.15.31:0/2800132677 conn(0x55d80aff84e0 msgr2=0x55d80b3a13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.868+0000 7f8ac6ff5640 0 client.17421 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:10:26.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.874+0000 7f8af01b4740 1 -- 172.21.15.31:0/2800132677 shutdown_connections 2024-04-25T16:10:26.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.874+0000 7f8af01b4740 1 -- 172.21.15.31:0/2800132677 wait complete. 2024-04-25T16:10:26.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.876+0000 7f8af01b4740 20 client.17421 trim_cache size 0 max 16384 2024-04-25T16:10:26.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:26 smithi186 ceph-mon[33258]: pgmap v625: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 780 MiB/s wr, 1.55k op/s 2024-04-25T16:10:26.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.917+0000 7f8af01b4740 1 Processor -- start 2024-04-25T16:10:26.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.917+0000 7f8af01b4740 1 -- start start 2024-04-25T16:10:26.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.917+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.918+0000 7f8ae2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.918+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 0x55d80b3a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.918+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.918+0000 7f8af01b4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae41b1680 con 0x55d80afdb4d0 2024-04-25T16:10:26.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.918+0000 7f8af01b4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae41b31c0 con 0x55d80afe36b0 2024-04-25T16:10:26.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.918+0000 7f8af01b4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae41b1d50 con 0x55d80b004390 2024-04-25T16:10:26.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.919+0000 7f8ae2d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35490/0 (socket says 172.21.15.31:35490) 2024-04-25T16:10:26.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.919+0000 7f8ae2d9d640 1 -- 172.21.15.31:0/605090813 learned_addr learned my addr 172.21.15.31:0/605090813 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:26.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.919+0000 7f8ae359e640 1 --2- 172.21.15.31:0/605090813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.919+0000 7f8ae259c640 1 --2- 172.21.15.31:0/605090813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 0x55d80b3a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.919+0000 7f8ae2d9d640 1 -- 172.21.15.31:0/605090813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 msgr2=0x55d80b3a1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.919+0000 7f8ae2d9d640 1 --2- 172.21.15.31:0/605090813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.919+0000 7f8ae2d9d640 1 -- 172.21.15.31:0/605090813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 msgr2=0x55d80b3a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:26.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.919+0000 7f8ae2d9d640 1 --2- 172.21.15.31:0/605090813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 0x55d80b3a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.919+0000 7f8ae2d9d640 1 -- 172.21.15.31:0/605090813 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d80b393ba0 con 0x55d80afdb4d0 2024-04-25T16:10:26.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.920+0000 7f8ae2d9d640 1 --2- 172.21.15.31:0/605090813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a09c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800ab60 tx=0x7f8ad8005db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4581761158d986 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:26.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.922+0000 7f8ac8ff9640 1 -- 172.21.15.31:0/605090813 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8010d70 con 0x55d80afdb4d0 2024-04-25T16:10:26.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.923+0000 7f8ac8ff9640 1 -- 172.21.15.31:0/605090813 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ad8005020 con 0x55d80afdb4d0 2024-04-25T16:10:26.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.923+0000 7f8ac8ff9640 1 -- 172.21.15.31:0/605090813 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ad801a840 con 0x55d80afdb4d0 2024-04-25T16:10:26.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.928+0000 7f8af01b4740 1 -- 172.21.15.31:0/605090813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 msgr2=0x55d80b3a09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.928+0000 7f8af01b4740 1 --2- 172.21.15.31:0/605090813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a09c0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800ab60 tx=0x7f8ad8005db0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.934+0000 7f8af01b4740 1 -- 172.21.15.31:0/605090813 shutdown_connections 2024-04-25T16:10:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.934+0000 7f8af01b4740 1 --2- 172.21.15.31:0/605090813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3a1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.934+0000 7f8af01b4740 1 --2- 172.21.15.31:0/605090813 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afe36b0 0x55d80b3a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.934+0000 7f8af01b4740 1 --2- 172.21.15.31:0/605090813 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afdb4d0 0x55d80b3a09c0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.934+0000 7f8af01b4740 1 -- 172.21.15.31:0/605090813 >> 172.21.15.31:0/605090813 conn(0x55d80aff84e0 msgr2=0x55d80b3a6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.934+0000 7f8af01b4740 1 -- 172.21.15.31:0/605090813 shutdown_connections 2024-04-25T16:10:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.935+0000 7f8af01b4740 1 -- 172.21.15.31:0/605090813 wait complete. 2024-04-25T16:10:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.935+0000 7f8af01b4740 1 -- start start 2024-04-25T16:10:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.935+0000 7f8af01b4740 1 Processor -- start 2024-04-25T16:10:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.936+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afdb4d0 0x55d80b3b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.936+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.937+0000 7f8af01b4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3b6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.937+0000 7f8af01b4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae41b2af0 con 0x55d80afe36b0 2024-04-25T16:10:26.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.937+0000 7f8af01b4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae41b2420 con 0x55d80afdb4d0 2024-04-25T16:10:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.937+0000 7f8af01b4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8ae41af660 con 0x55d80b004390 2024-04-25T16:10:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.937+0000 7f8af01b4740 10 client.0 _ll_register_callbacks cb 0x7f8adc0c7fc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.938+0000 7f8ae259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.938+0000 7f8ae259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35504/0 (socket says 172.21.15.31:35504) 2024-04-25T16:10:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.938+0000 7f8ae259c640 1 -- 172.21.15.31:0/339444967 learned_addr learned my addr 172.21.15.31:0/339444967 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:26.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.939+0000 7f8ae259c640 1 -- 172.21.15.31:0/339444967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 msgr2=0x55d80b3b6160 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:26.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.939+0000 7f8ae259c640 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3b6160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.939+0000 7f8ae259c640 1 -- 172.21.15.31:0/339444967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afdb4d0 msgr2=0x55d80b3b4cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:26.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.939+0000 7f8ae259c640 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afdb4d0 0x55d80b3b4cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.939+0000 7f8ae259c640 1 -- 172.21.15.31:0/339444967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d80b3b6720 con 0x55d80afe36b0 2024-04-25T16:10:26.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.940+0000 7f8ae259c640 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b5230 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0c7ef0 tx=0x7f8adc0317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b446ecfa6b24d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:26.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.942+0000 7f8ac6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:10:26.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.942+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/339444967 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8adc0cc360 con 0x55d80afe36b0 2024-04-25T16:10:26.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.942+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/339444967 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8adc0041e0 con 0x55d80afe36b0 2024-04-25T16:10:26.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.943+0000 7f8af01b4740 1 -- 172.21.15.31:0/339444967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ad8005020 con 0x55d80afe36b0 2024-04-25T16:10:26.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.946+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/339444967 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8adc0c81f0 con 0x55d80afe36b0 2024-04-25T16:10:26.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.946+0000 7f8af01b4740 10 client.17427 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:26.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.946+0000 7f8af01b4740 1 -- 172.21.15.31:0/339444967 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d80b4269b0 con 0x55d80afe36b0 2024-04-25T16:10:26.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.949+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/339444967 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ad8005020 con 0x55d80afe36b0 2024-04-25T16:10:26.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.949+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/339444967 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f8adc0c3ad0 con 0x55d80afe36b0 2024-04-25T16:10:26.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.949+0000 7f8af01b4740 10 client.17427 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:26.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.949+0000 7f8af01b4740 10 client.17427 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:10:26.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.949+0000 7f8af01b4740 20 client.17427 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:26.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.949+0000 7f8af01b4740 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55d80af983a0 0x55d80b429240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:26.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.949+0000 7f8af01b4740 4 client.17427 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:10:26.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.949+0000 7f8af01b4740 1 -- 172.21.15.31:0/339444967 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d80b312800 con 0x55d80af983a0 2024-04-25T16:10:26.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.951+0000 7f8ae2d9d640 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55d80af983a0 0x55d80b429240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:26.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.969+0000 7f8ae2d9d640 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55d80af983a0 0x55d80b429240 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fe4836d674f37478 server_cookie=2fc40d53a17e48ee in_seq=0 out_seq=0 2024-04-25T16:10:26.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.969+0000 7f8ac6ff5640 10 client.17427 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:10:26.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.977+0000 7f8ac6ff5640 1 -- 172.21.15.31:0/339444967 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d80b312800 con 0x55d80af983a0 2024-04-25T16:10:26.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.977+0000 7f8ac6ff5640 10 client.17427 handle_command_reply: tid=0 2024-04-25T16:10:26.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.979+0000 7f8af01b4740 1 client.17427 shutdown 2024-04-25T16:10:26.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.983+0000 7f8af01b4740 1 -- 172.21.15.31:0/339444967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 msgr2=0x55d80b3b5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:26.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.983+0000 7f8af01b4740 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b5230 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0c7ef0 tx=0x7f8adc0317e0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.983+0000 7f8af01b4740 1 -- 172.21.15.31:0/339444967 shutdown_connections 2024-04-25T16:10:26.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.983+0000 7f8af01b4740 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55d80af983a0 0x55d80b429240 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.983+0000 7f8af01b4740 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d80b004390 0x55d80b3b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.983+0000 7f8af01b4740 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d80afe36b0 0x55d80b3b5230 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.983+0000 7f8af01b4740 1 --2- 172.21.15.31:0/339444967 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d80afdb4d0 0x55d80b3b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:26.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.983+0000 7f8af01b4740 1 -- 172.21.15.31:0/339444967 >> 172.21.15.31:0/339444967 conn(0x55d80aff84e0 msgr2=0x55d80b3a4ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:26.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.984+0000 7f8ac6ff5640 0 client.17427 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:10:26.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.992+0000 7f8af01b4740 1 -- 172.21.15.31:0/339444967 shutdown_connections 2024-04-25T16:10:26.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.993+0000 7f8af01b4740 1 -- 172.21.15.31:0/339444967 wait complete. 2024-04-25T16:10:26.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:26.997+0000 7f8af01b4740 20 client.17427 trim_cache size 0 max 16384 2024-04-25T16:10:27.002 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.001+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8acc0afb50 msgr2=0x7f8acc0b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.001+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8acc0afb50 0x7f8acc0b2060 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8ad401f000 tx=0x7f8ad401c000 comp rx=0 tx=0).stop 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.001+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410fe90 msgr2=0x7f8ae41177c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.001+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410fe90 0x7f8ae41177c0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f8adc009dc0 tx=0x7f8adc003ba0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.003+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/168244170 shutdown_connections 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.003+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8acc0afb50 0x7f8acc0b2060 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.003+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8ae41a5970 0x7f8ae41a7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.003+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8ae410fe90 0x7f8ae41177c0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.003+0000 7f8ae8a2f640 1 --2- 172.21.15.31:0/168244170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ae410a340 0x7f8ae4117280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.003+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/168244170 >> 172.21.15.31:0/168244170 conn(0x7f8ae4070320 msgr2=0x7f8ae410be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:27.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.011+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/168244170 shutdown_connections 2024-04-25T16:10:27.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.013+0000 7f8ae8a2f640 1 -- 172.21.15.31:0/168244170 wait complete. 2024-04-25T16:10:27.092 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:10:27.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.606+0000 7f693fde3640 1 -- 172.21.15.31:0/200979373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f693810b310 msgr2=0x7f693810b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.606+0000 7f693fde3640 1 --2- 172.21.15.31:0/200979373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f693810b310 0x7f693810b720 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f6928009860 tx=0x7f692802a380 comp rx=0 tx=0).stop 2024-04-25T16:10:27.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.610+0000 7f693fde3640 1 -- 172.21.15.31:0/200979373 shutdown_connections 2024-04-25T16:10:27.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.610+0000 7f693fde3640 1 --2- 172.21.15.31:0/200979373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6938110030 0x7f6938110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.610+0000 7f693fde3640 1 --2- 172.21.15.31:0/200979373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f693810b310 0x7f693810b720 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.610+0000 7f693fde3640 1 --2- 172.21.15.31:0/200979373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f693810a420 0x7f693810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.610+0000 7f693fde3640 1 -- 172.21.15.31:0/200979373 >> 172.21.15.31:0/200979373 conn(0x7f69380704e0 msgr2=0x7f6938072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:27.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.610+0000 7f693fde3640 1 -- 172.21.15.31:0/200979373 shutdown_connections 2024-04-25T16:10:27.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.610+0000 7f693fde3640 1 -- 172.21.15.31:0/200979373 wait complete. 2024-04-25T16:10:27.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.611+0000 7f693fde3640 1 Processor -- start 2024-04-25T16:10:27.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.612+0000 7f693fde3640 1 -- start start 2024-04-25T16:10:27.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.612+0000 7f693fde3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f693810a420 0x7f6938117010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.612+0000 7f693fde3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f693810b310 0x7f6938117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.612+0000 7f693fde3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6938110030 0x7f69381184b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.612+0000 7f693fde3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f693811e540 con 0x7f6938110030 2024-04-25T16:10:27.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.612+0000 7f693fde3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f693811e3c0 con 0x7f693810a420 2024-04-25T16:10:27.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.612+0000 7f693fde3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f693811e6c0 con 0x7f693810b310 2024-04-25T16:10:27.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.613+0000 7f693db7f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f693810a420 0x7f6938117010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.613+0000 7f693db7f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f693810a420 0x7f6938117010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58110/0 (socket says 172.21.15.31:58110) 2024-04-25T16:10:27.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.613+0000 7f693db7f640 1 -- 172.21.15.31:0/669469990 learned_addr learned my addr 172.21.15.31:0/669469990 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:27.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.613+0000 7f693e380640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6938110030 0x7f69381184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.613+0000 7f693d37e640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f693810b310 0x7f6938117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.615 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.614+0000 7f693e380640 1 -- 172.21.15.31:0/669469990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f693810b310 msgr2=0x7f6938117550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.614+0000 7f693e380640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f693810b310 0x7f6938117550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.614+0000 7f693e380640 1 -- 172.21.15.31:0/669469990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f693810a420 msgr2=0x7f6938117010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.614+0000 7f693e380640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f693810a420 0x7f6938117010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.614+0000 7f693e380640 1 -- 172.21.15.31:0/669469990 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6938118b20 con 0x7f6938110030 2024-04-25T16:10:27.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.615+0000 7f693e380640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6938110030 0x7f69381184b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f693000ef00 tx=0x7f693000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:27.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.616+0000 7f692effd640 1 -- 172.21.15.31:0/669469990 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930004b20 con 0x7f6938110030 2024-04-25T16:10:27.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.616+0000 7f693fde3640 1 -- 172.21.15.31:0/669469990 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6938118e10 con 0x7f6938110030 2024-04-25T16:10:27.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.616+0000 7f693fde3640 1 -- 172.21.15.31:0/669469990 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6938119360 con 0x7f6938110030 2024-04-25T16:10:27.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.617+0000 7f692effd640 1 -- 172.21.15.31:0/669469990 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6930003e20 con 0x7f6938110030 2024-04-25T16:10:27.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.617+0000 7f692effd640 1 -- 172.21.15.31:0/669469990 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693000b570 con 0x7f6938110030 2024-04-25T16:10:27.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.627+0000 7f6946d6e740 1 Processor -- start 2024-04-25T16:10:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.658+0000 7f6946d6e740 1 -- start start 2024-04-25T16:10:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.659+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a75c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.659+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3914d0 0x56492a75cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.659+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a75da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.659+0000 7f6946d6e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56492a6c0d80 con 0x56492a3ba390 2024-04-25T16:10:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.659+0000 7f6946d6e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56492a67aa20 con 0x56492a3914d0 2024-04-25T16:10:27.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.659+0000 7f6946d6e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56492a6bcd40 con 0x56492a3996b0 2024-04-25T16:10:27.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.661+0000 7f692effd640 1 -- 172.21.15.31:0/669469990 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f693001d020 con 0x7f6938110030 2024-04-25T16:10:27.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.662+0000 7f692effd640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69240afcf0 0x7f69240b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.663+0000 7f692effd640 1 -- 172.21.15.31:0/669469990 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f69300de8f0 con 0x7f6938110030 2024-04-25T16:10:27.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.666+0000 7f693d37e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a75da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.666+0000 7f693d37e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a75da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54032/0 (socket says 172.21.15.31:54032) 2024-04-25T16:10:27.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.666+0000 7f693d37e640 1 -- 172.21.15.31:0/2311702289 learned_addr learned my addr 172.21.15.31:0/2311702289 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:27.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.666+0000 7f693db7f640 1 --2- 172.21.15.31:0/2311702289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3914d0 0x56492a75cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.667+0000 7f693d37e640 1 --2- 172.21.15.31:0/2311702289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a75c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.674+0000 7f693d37e640 1 -- 172.21.15.31:0/2311702289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 msgr2=0x56492a75da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:27.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.674+0000 7f693d37e640 1 --2- 172.21.15.31:0/2311702289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a75da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.674+0000 7f693d37e640 1 -- 172.21.15.31:0/2311702289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3914d0 msgr2=0x56492a75cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:27.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.674+0000 7f693d37e640 1 --2- 172.21.15.31:0/2311702289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3914d0 0x56492a75cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.674+0000 7f693d37e640 1 -- 172.21.15.31:0/2311702289 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56492a685c30 con 0x56492a3ba390 2024-04-25T16:10:27.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.676+0000 7f693e380640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69240afcf0 0x7f69240b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.683 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.682+0000 7f693d37e640 1 --2- 172.21.15.31:0/2311702289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a75c810 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6928009830 tx=0x7f692802a380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62ff45ba3e7a038e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:27.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.686+0000 7f6923fff640 1 -- 172.21.15.31:0/2311702289 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928037040 con 0x56492a3ba390 2024-04-25T16:10:27.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.686+0000 7f6923fff640 1 -- 172.21.15.31:0/2311702289 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6928038040 con 0x56492a3ba390 2024-04-25T16:10:27.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.686+0000 7f6923fff640 1 -- 172.21.15.31:0/2311702289 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928003c70 con 0x56492a3ba390 2024-04-25T16:10:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.687+0000 7f6946d6e740 1 -- 172.21.15.31:0/2311702289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 msgr2=0x56492a75c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.687+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2311702289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a75c810 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6928009830 tx=0x7f692802a380 comp rx=0 tx=0).stop 2024-04-25T16:10:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.695+0000 7f6946d6e740 1 -- 172.21.15.31:0/2311702289 shutdown_connections 2024-04-25T16:10:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.695+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2311702289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a75da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.695+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2311702289 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3914d0 0x56492a75cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.695+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2311702289 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a75c810 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.695+0000 7f6946d6e740 1 -- 172.21.15.31:0/2311702289 >> 172.21.15.31:0/2311702289 conn(0x56492a3ae4e0 msgr2=0x56492a758020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:27.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.701+0000 7f6946d6e740 1 -- 172.21.15.31:0/2311702289 shutdown_connections 2024-04-25T16:10:27.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.701+0000 7f6946d6e740 1 -- 172.21.15.31:0/2311702289 wait complete. 2024-04-25T16:10:27.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.702+0000 7f6946d6e740 1 -- start start 2024-04-25T16:10:27.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.702+0000 7f6946d6e740 1 Processor -- start 2024-04-25T16:10:27.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.704+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3914d0 0x56492a766610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.704+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3996b0 0x56492a766b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.704+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a76dab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.704+0000 7f6946d6e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x56492a75e520 con 0x56492a3ba390 2024-04-25T16:10:27.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.704+0000 7f6946d6e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x56492a647dc0 con 0x56492a3996b0 2024-04-25T16:10:27.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.704+0000 7f6946d6e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x56492a75e6a0 con 0x56492a3914d0 2024-04-25T16:10:27.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.704+0000 7f6946d6e740 10 client.0 _ll_register_callbacks cb 0x56492a663600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.705+0000 7f693db7f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3996b0 0x56492a766b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.705+0000 7f693db7f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3996b0 0x56492a766b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58130/0 (socket says 172.21.15.31:58130) 2024-04-25T16:10:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.705+0000 7f693db7f640 1 -- 172.21.15.31:0/2835465055 learned_addr learned my addr 172.21.15.31:0/2835465055 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.705+0000 7f693d37e640 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a76dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.705+0000 7f693d37e640 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3914d0 0x56492a766610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.706+0000 7f693db7f640 1 -- 172.21.15.31:0/2835465055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3914d0 msgr2=0x56492a766610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.706+0000 7f693db7f640 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3914d0 0x56492a766610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.706+0000 7f693db7f640 1 -- 172.21.15.31:0/2835465055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 msgr2=0x56492a76dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:27.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.706+0000 7f693db7f640 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a76dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.706+0000 7f693db7f640 1 -- 172.21.15.31:0/2835465055 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56492a3d23f0 con 0x56492a3996b0 2024-04-25T16:10:27.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.708+0000 7f693db7f640 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3996b0 0x56492a766b50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f69340115e0 tx=0x7f6934011aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=805977dc623488ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.723+0000 7f69217fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:10:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.723+0000 7f69217fa640 1 -- 172.21.15.31:0/2835465055 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693400f6f0 con 0x56492a3996b0 2024-04-25T16:10:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.724+0000 7f6946d6e740 1 -- 172.21.15.31:0/2835465055 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6928038040 con 0x56492a3996b0 2024-04-25T16:10:27.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.724+0000 7f69217fa640 1 -- 172.21.15.31:0/2835465055 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f693400f890 con 0x56492a3996b0 2024-04-25T16:10:27.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.734+0000 7f69217fa640 1 -- 172.21.15.31:0/2835465055 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f693400fa90 con 0x56492a3996b0 2024-04-25T16:10:27.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.734+0000 7f6946d6e740 10 client.27098 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:27.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.734+0000 7f6946d6e740 1 -- 172.21.15.31:0/2835465055 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56492a49c7d0 con 0x56492a3996b0 2024-04-25T16:10:27.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.738+0000 7f69217fa640 1 -- 172.21.15.31:0/2835465055 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6928038040 con 0x56492a3996b0 2024-04-25T16:10:27.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.740+0000 7f69217fa640 1 -- 172.21.15.31:0/2835465055 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f693401d400 con 0x56492a3996b0 2024-04-25T16:10:27.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.740+0000 7f6946d6e740 10 client.27098 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:27.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.741+0000 7f6946d6e740 10 client.27098 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:10:27.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.741+0000 7f6946d6e740 20 client.27098 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:27.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.741+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56492a378f60 0x56492a7747e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.741+0000 7f6946d6e740 4 client.27098 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:27.742 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.741+0000 7f6946d6e740 1 -- 172.21.15.31:0/2835465055 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56492a6ce990 con 0x56492a378f60 2024-04-25T16:10:27.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.744+0000 7f693d37e640 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56492a378f60 0x56492a7747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.750+0000 7f693d37e640 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56492a378f60 0x56492a7747e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8ee924ad9d2bc0c9 server_cookie=e9c84a8d121247e in_seq=0 out_seq=0 2024-04-25T16:10:27.752 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.750+0000 7f69217fa640 10 client.27098 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:10:27.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.755+0000 7f69217fa640 1 -- 172.21.15.31:0/2835465055 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56492a6ce990 con 0x56492a378f60 2024-04-25T16:10:27.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.755+0000 7f69217fa640 10 client.27098 handle_command_reply: tid=0 2024-04-25T16:10:27.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.755+0000 7f6946d6e740 1 client.27098 shutdown 2024-04-25T16:10:27.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.762+0000 7f6946d6e740 1 -- 172.21.15.31:0/2835465055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3996b0 msgr2=0x56492a766b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.762+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3996b0 0x56492a766b50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f69340115e0 tx=0x7f6934011aa0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.763 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.762+0000 7f693e380640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69240afcf0 0x7f69240b2200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f69300a6a80 tx=0x7f693000cf70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:27.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.762+0000 7f6946d6e740 1 -- 172.21.15.31:0/2835465055 shutdown_connections 2024-04-25T16:10:27.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.763+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a76dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.763+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56492a378f60 0x56492a7747e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.763+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3996b0 0x56492a766b50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.763+0000 7f6946d6e740 1 --2- 172.21.15.31:0/2835465055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3914d0 0x56492a766610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.763+0000 7f6946d6e740 1 -- 172.21.15.31:0/2835465055 >> 172.21.15.31:0/2835465055 conn(0x56492a3ae4e0 msgr2=0x56492a757040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:27.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.763+0000 7f69217fa640 0 client.27098 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:10:27.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.775+0000 7f6946d6e740 1 -- 172.21.15.31:0/2835465055 shutdown_connections 2024-04-25T16:10:27.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.775+0000 7f6946d6e740 1 -- 172.21.15.31:0/2835465055 wait complete. 2024-04-25T16:10:27.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.776+0000 7f6946d6e740 20 client.27098 trim_cache size 0 max 16384 2024-04-25T16:10:27.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.851+0000 7f6946d6e740 1 Processor -- start 2024-04-25T16:10:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.853+0000 7f6946d6e740 1 -- start start 2024-04-25T16:10:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.854+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3914d0 0x56492a75b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.854+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.854+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3ba390 0x56492a764d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.854+0000 7f6946d6e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f691c00e400 con 0x56492a3914d0 2024-04-25T16:10:27.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.854+0000 7f6946d6e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f691c00ff40 con 0x56492a3ba390 2024-04-25T16:10:27.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.854+0000 7f6946d6e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f691c00ead0 con 0x56492a3996b0 2024-04-25T16:10:27.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.856+0000 7f693d37e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3914d0 0x56492a75b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.856+0000 7f693d37e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3914d0 0x56492a75b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35546/0 (socket says 172.21.15.31:35546) 2024-04-25T16:10:27.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.856+0000 7f693d37e640 1 -- 172.21.15.31:0/358190302 learned_addr learned my addr 172.21.15.31:0/358190302 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:27.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.858+0000 7f693d37e640 1 --2- 172.21.15.31:0/358190302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3ba390 0x56492a764d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.859+0000 7f693db7f640 1 --2- 172.21.15.31:0/358190302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.859+0000 7f693d37e640 1 -- 172.21.15.31:0/358190302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 msgr2=0x56492a76d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.859+0000 7f693d37e640 1 --2- 172.21.15.31:0/358190302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.859+0000 7f693d37e640 1 -- 172.21.15.31:0/358190302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3ba390 msgr2=0x56492a764d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.859+0000 7f693d37e640 1 --2- 172.21.15.31:0/358190302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3ba390 0x56492a764d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.861 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.859+0000 7f693d37e640 1 -- 172.21.15.31:0/358190302 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56492a761ce0 con 0x56492a3914d0 2024-04-25T16:10:27.866 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.864+0000 7f693d37e640 1 --2- 172.21.15.31:0/358190302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3914d0 0x56492a75b530 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6928009860 tx=0x7f6928000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd84153ddf45461 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.867+0000 7f69237fe640 1 -- 172.21.15.31:0/358190302 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928037030 con 0x56492a3914d0 2024-04-25T16:10:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.867+0000 7f69237fe640 1 -- 172.21.15.31:0/358190302 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6928030e70 con 0x56492a3914d0 2024-04-25T16:10:27.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.867+0000 7f69237fe640 1 -- 172.21.15.31:0/358190302 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6928056040 con 0x56492a3914d0 2024-04-25T16:10:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.869+0000 7f6946d6e740 1 -- 172.21.15.31:0/358190302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3914d0 msgr2=0x56492a75b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.870 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.869+0000 7f6946d6e740 1 --2- 172.21.15.31:0/358190302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3914d0 0x56492a75b530 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6928009860 tx=0x7f6928000bc0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.872+0000 7f6946d6e740 1 -- 172.21.15.31:0/358190302 shutdown_connections 2024-04-25T16:10:27.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.872+0000 7f6946d6e740 1 --2- 172.21.15.31:0/358190302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3ba390 0x56492a764d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.872+0000 7f6946d6e740 1 --2- 172.21.15.31:0/358190302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.872+0000 7f6946d6e740 1 --2- 172.21.15.31:0/358190302 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3914d0 0x56492a75b530 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.872+0000 7f6946d6e740 1 -- 172.21.15.31:0/358190302 >> 172.21.15.31:0/358190302 conn(0x56492a3ae4e0 msgr2=0x56492a75ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:27.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.872+0000 7f6946d6e740 1 -- 172.21.15.31:0/358190302 shutdown_connections 2024-04-25T16:10:27.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.873+0000 7f6946d6e740 1 -- 172.21.15.31:0/358190302 wait complete. 2024-04-25T16:10:27.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.873+0000 7f6946d6e740 1 -- start start 2024-04-25T16:10:27.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.874+0000 7f6946d6e740 1 Processor -- start 2024-04-25T16:10:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.875+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3914d0 0x56492a76b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.875+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.875+0000 7f6946d6e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a76c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.875+0000 7f6946d6e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f691c00f870 con 0x56492a3ba390 2024-04-25T16:10:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.875+0000 7f6946d6e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f691c00f1a0 con 0x56492a3914d0 2024-04-25T16:10:27.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.875+0000 7f6946d6e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f691c00c490 con 0x56492a3996b0 2024-04-25T16:10:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.876+0000 7f693db7f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.876+0000 7f693db7f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54060/0 (socket says 172.21.15.31:54060) 2024-04-25T16:10:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.876+0000 7f693db7f640 1 -- 172.21.15.31:0/1657470065 learned_addr learned my addr 172.21.15.31:0/1657470065 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.876+0000 7f6946d6e740 10 client.0 _ll_register_callbacks cb 0x7f69340095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.876+0000 7f693db7f640 1 -- 172.21.15.31:0/1657470065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3914d0 msgr2=0x56492a76b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.876+0000 7f693db7f640 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3914d0 0x56492a76b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.876+0000 7f693db7f640 1 -- 172.21.15.31:0/1657470065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 msgr2=0x56492a76c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.876+0000 7f693db7f640 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a76c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.876+0000 7f693db7f640 1 -- 172.21.15.31:0/1657470065 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56492a76df10 con 0x56492a3996b0 2024-04-25T16:10:27.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.881+0000 7f693db7f640 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76b560 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6934004810 tx=0x7f6934009b80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=250b11e05f59740b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:27.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.881+0000 7f69217fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:10:27.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.882+0000 7f69217fa640 1 -- 172.21.15.31:0/1657470065 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693400fe30 con 0x56492a3996b0 2024-04-25T16:10:27.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.882+0000 7f6946d6e740 1 -- 172.21.15.31:0/1657470065 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x56492a6c8800 con 0x56492a3996b0 2024-04-25T16:10:27.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.882+0000 7f69217fa640 1 -- 172.21.15.31:0/1657470065 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f693401da10 con 0x56492a3996b0 2024-04-25T16:10:27.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.883+0000 7f69217fa640 1 -- 172.21.15.31:0/1657470065 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6934037050 con 0x56492a3996b0 2024-04-25T16:10:27.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.883+0000 7f6946d6e740 10 client.27094 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:27.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.883+0000 7f6946d6e740 1 -- 172.21.15.31:0/1657470065 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56492a76db00 con 0x56492a3996b0 2024-04-25T16:10:27.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.885+0000 7f69217fa640 1 -- 172.21.15.31:0/1657470065 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x56492a6c8800 con 0x56492a3996b0 2024-04-25T16:10:27.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.885+0000 7f69217fa640 1 -- 172.21.15.31:0/1657470065 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f6934014ab0 con 0x56492a3996b0 2024-04-25T16:10:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.885+0000 7f6946d6e740 10 client.27094 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.885+0000 7f6946d6e740 10 client.27094 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:10:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.885+0000 7f6946d6e740 20 client.27094 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.885+0000 7f6946d6e740 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56492a34e3a0 0x56492a7df1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.885+0000 7f6946d6e740 4 client.27094 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:10:27.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.885+0000 7f6946d6e740 1 -- 172.21.15.31:0/1657470065 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56492a643610 con 0x56492a34e3a0 2024-04-25T16:10:27.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.887+0000 7f693d37e640 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56492a34e3a0 0x56492a7df1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.888+0000 7f693d37e640 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56492a34e3a0 0x56492a7df1a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f996e81a5ea2f2 server_cookie=b6a56ca1f26b155e in_seq=0 out_seq=0 2024-04-25T16:10:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.889+0000 7f69217fa640 10 client.27094 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:10:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.889+0000 7f69217fa640 1 -- 172.21.15.31:0/1657470065 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56492a643610 con 0x56492a34e3a0 2024-04-25T16:10:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.889+0000 7f69217fa640 10 client.27094 handle_command_reply: tid=0 2024-04-25T16:10:27.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.889+0000 7f6946d6e740 1 client.27094 shutdown 2024-04-25T16:10:27.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.897+0000 7f6946d6e740 1 -- 172.21.15.31:0/1657470065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 msgr2=0x56492a76b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.897+0000 7f6946d6e740 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76b560 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f6934004810 tx=0x7f6934009b80 comp rx=0 tx=0).stop 2024-04-25T16:10:27.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.898+0000 7f6946d6e740 1 -- 172.21.15.31:0/1657470065 shutdown_connections 2024-04-25T16:10:27.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.898+0000 7f6946d6e740 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x56492a3ba390 0x56492a76c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.898+0000 7f6946d6e740 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56492a3996b0 0x56492a76b560 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.898+0000 7f6946d6e740 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x56492a34e3a0 0x56492a7df1a0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.898+0000 7f6946d6e740 1 --2- 172.21.15.31:0/1657470065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x56492a3914d0 0x56492a76b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.898+0000 7f69217fa640 0 client.27094 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:10:27.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.898+0000 7f6946d6e740 1 -- 172.21.15.31:0/1657470065 >> 172.21.15.31:0/1657470065 conn(0x56492a3ae4e0 msgr2=0x56492a75b850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:27.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.899+0000 7f6946d6e740 1 -- 172.21.15.31:0/1657470065 shutdown_connections 2024-04-25T16:10:27.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.899+0000 7f6946d6e740 1 -- 172.21.15.31:0/1657470065 wait complete. 2024-04-25T16:10:27.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.899+0000 7f6946d6e740 20 client.27094 trim_cache size 0 max 16384 2024-04-25T16:10:27.901 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:10:27.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.910+0000 7f693fde3640 1 -- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69240afcf0 msgr2=0x7f69240b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.910+0000 7f693fde3640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69240afcf0 0x7f69240b2200 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f69300a6a80 tx=0x7f693000cf70 comp rx=0 tx=0).stop 2024-04-25T16:10:27.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.910+0000 7f693fde3640 1 -- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6938110030 msgr2=0x7f69381184b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:27.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.910+0000 7f693fde3640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6938110030 0x7f69381184b0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f693000ef00 tx=0x7f693000c740 comp rx=0 tx=0).stop 2024-04-25T16:10:27.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.914+0000 7f693fde3640 1 -- 172.21.15.31:0/669469990 shutdown_connections 2024-04-25T16:10:27.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.914+0000 7f693fde3640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f69240afcf0 0x7f69240b2200 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.914+0000 7f693fde3640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f6938110030 0x7f69381184b0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.914+0000 7f693fde3640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f693810b310 0x7f6938117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.914+0000 7f693fde3640 1 --2- 172.21.15.31:0/669469990 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f693810a420 0x7f6938117010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:27.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.914+0000 7f693fde3640 1 -- 172.21.15.31:0/669469990 >> 172.21.15.31:0/669469990 conn(0x7f69380704e0 msgr2=0x7f693810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:27.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.916+0000 7f693fde3640 1 -- 172.21.15.31:0/669469990 shutdown_connections 2024-04-25T16:10:27.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:27.916+0000 7f693fde3640 1 -- 172.21.15.31:0/669469990 wait complete. 2024-04-25T16:10:28.013 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:10:28.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.412+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1264753737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb98101b90 msgr2=0x7fcb98102890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:28.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.412+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1264753737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb98101b90 0x7fcb98102890 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c01a910 tx=0x7fcb8c03b470 comp rx=0 tx=0).stop 2024-04-25T16:10:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.417+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1264753737 shutdown_connections 2024-04-25T16:10:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.417+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1264753737 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb98107ad0 0x7fcb98109ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.417+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1264753737 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb98102dd0 0x7fcb981031e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.417+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1264753737 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb98101b90 0x7fcb98102890 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.417+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1264753737 >> 172.21.15.31:0/1264753737 conn(0x7fcb980fcda0 msgr2=0x7fcb980ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:28.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.420+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1264753737 shutdown_connections 2024-04-25T16:10:28.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.420+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1264753737 wait complete. 2024-04-25T16:10:28.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.421+0000 7fcb9dbc5640 1 Processor -- start 2024-04-25T16:10:28.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.421+0000 7fcb9dbc5640 1 -- start start 2024-04-25T16:10:28.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.421+0000 7fcb9dbc5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb98101b90 0x7fcb9810aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.422+0000 7fcb9dbc5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb98102dd0 0x7fcb9810b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.422+0000 7fcb9dbc5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb98107ad0 0x7fcb98111f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.422+0000 7fcb9dbc5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9806ec90 con 0x7fcb98101b90 2024-04-25T16:10:28.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.422+0000 7fcb9dbc5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9806eb10 con 0x7fcb98107ad0 2024-04-25T16:10:28.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.422+0000 7fcb9dbc5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb9806ee10 con 0x7fcb98102dd0 2024-04-25T16:10:28.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.422+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb98102dd0 0x7fcb9810b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.424+0000 7fcb9d3c4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb98107ad0 0x7fcb98111f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.424+0000 7fcb9cbc3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb98101b90 0x7fcb9810aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.424+0000 7fcb9cbc3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb98101b90 0x7fcb9810aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35556/0 (socket says 172.21.15.31:35556) 2024-04-25T16:10:28.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.424+0000 7fcb9cbc3640 1 -- 172.21.15.31:0/1404497389 learned_addr learned my addr 172.21.15.31:0/1404497389 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:28.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.426+0000 7fcb9cbc3640 1 -- 172.21.15.31:0/1404497389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb98102dd0 msgr2=0x7fcb9810b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:28.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.426+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb98102dd0 0x7fcb9810b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.426+0000 7fcb9cbc3640 1 -- 172.21.15.31:0/1404497389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb98107ad0 msgr2=0x7fcb98111f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:28.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.426+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb98107ad0 0x7fcb98111f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.426+0000 7fcb9cbc3640 1 -- 172.21.15.31:0/1404497389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb981125d0 con 0x7fcb98101b90 2024-04-25T16:10:28.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.428+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb98101b90 0x7fcb9810aed0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c004dc0 tx=0x7fcb8c004ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:28.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.431+0000 7fcb95ffb640 1 -- 172.21.15.31:0/1404497389 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c04f070 con 0x7fcb98101b90 2024-04-25T16:10:28.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.432+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1404497389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb98112860 con 0x7fcb98101b90 2024-04-25T16:10:28.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.432+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1404497389 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb98112dd0 con 0x7fcb98101b90 2024-04-25T16:10:28.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.432+0000 7fcb95ffb640 1 -- 172.21.15.31:0/1404497389 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb8c040720 con 0x7fcb98101b90 2024-04-25T16:10:28.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.432+0000 7fcb95ffb640 1 -- 172.21.15.31:0/1404497389 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c046030 con 0x7fcb98101b90 2024-04-25T16:10:28.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.440+0000 7fcba5db9740 1 Processor -- start 2024-04-25T16:10:28.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.445+0000 7fcba5db9740 1 -- start start 2024-04-25T16:10:28.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.445+0000 7fcb95ffb640 1 -- 172.21.15.31:0/1404497389 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcb8c003850 con 0x7fcb98101b90 2024-04-25T16:10:28.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.447+0000 7fcb95ffb640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb800afcf0 0x7fcb800b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbafc390 0x55d3bbe9ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbe9ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcba5db9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d3bbe02d80 con 0x55d3bbadb6b0 2024-04-25T16:10:28.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcba5db9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d3bbdbca20 con 0x55d3bbafc390 2024-04-25T16:10:28.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcb9cbc3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbafc390 0x55d3bbe9ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcba5db9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d3bbdfed40 con 0x55d3bbad34d0 2024-04-25T16:10:28.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcb9cbc3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbafc390 0x55d3bbe9ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58178/0 (socket says 172.21.15.31:58178) 2024-04-25T16:10:28.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcb9cbc3640 1 -- 172.21.15.31:0/2941668486 learned_addr learned my addr 172.21.15.31:0/2941668486 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:28.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.448+0000 7fcb97fff640 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbe9ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.449+0000 7fcb95ffb640 1 -- 172.21.15.31:0/1404497389 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcb8c004080 con 0x7fcb98101b90 2024-04-25T16:10:28.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.449+0000 7fcb9d3c4640 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.450+0000 7fcb97fff640 1 -- 172.21.15.31:0/2941668486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbafc390 msgr2=0x55d3bbe9ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:28.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.450+0000 7fcb97fff640 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbafc390 0x55d3bbe9ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.450+0000 7fcb97fff640 1 -- 172.21.15.31:0/2941668486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 msgr2=0x55d3bbe9fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:28.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.450+0000 7fcb97fff640 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.450+0000 7fcb97fff640 1 -- 172.21.15.31:0/2941668486 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3bbdc7c30 con 0x55d3bbad34d0 2024-04-25T16:10:28.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.450+0000 7fcb9d3c4640 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:10:28.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.450+0000 7fcb9d3c4640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb800afcf0 0x7fcb800b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.450+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbafc390 0x55d3bbe9ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:10:28.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.461+0000 7fcb97fff640 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbe9ef80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000a400 tx=0x7fcb900096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f99a37d7112afb50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:28.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.465+0000 7fcb9d3c4640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb800afcf0 0x7fcb800b2200 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcb88006a50 tx=0x7fcb88008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.466+0000 7fcb7effd640 1 -- 172.21.15.31:0/2941668486 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90013790 con 0x55d3bbad34d0 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.466+0000 7fcb7effd640 1 -- 172.21.15.31:0/2941668486 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb90019070 con 0x55d3bbad34d0 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.466+0000 7fcb7effd640 1 -- 172.21.15.31:0/2941668486 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90014cb0 con 0x55d3bbad34d0 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.469+0000 7fcba5db9740 1 -- 172.21.15.31:0/2941668486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 msgr2=0x55d3bbe9ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.469+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbe9ef80 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcb9000a400 tx=0x7fcb900096d0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.476+0000 7fcba5db9740 1 -- 172.21.15.31:0/2941668486 shutdown_connections 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.476+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.476+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbe9ef80 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.476+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2941668486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbafc390 0x55d3bbe9ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.476+0000 7fcba5db9740 1 -- 172.21.15.31:0/2941668486 >> 172.21.15.31:0/2941668486 conn(0x55d3bbaf04e0 msgr2=0x55d3bbe9a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.477+0000 7fcba5db9740 1 -- 172.21.15.31:0/2941668486 shutdown_connections 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.477+0000 7fcba5db9740 1 -- 172.21.15.31:0/2941668486 wait complete. 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.477+0000 7fcba5db9740 1 -- start start 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.478+0000 7fcba5db9740 1 Processor -- start 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.478+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 0x55d3bbea8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.478+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbea8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.478+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.478+0000 7fcba5db9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d3bbea0720 con 0x55d3bbadb6b0 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.478+0000 7fcba5db9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d3bbd89dc0 con 0x55d3bbad34d0 2024-04-25T16:10:28.480 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.478+0000 7fcba5db9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d3bbea08a0 con 0x55d3bbafc390 2024-04-25T16:10:28.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.479+0000 7fcba5db9740 10 client.0 _ll_register_callbacks cb 0x55d3bbda5600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:28.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.479+0000 7fcb9d3c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.479+0000 7fcb9d3c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54090/0 (socket says 172.21.15.31:54090) 2024-04-25T16:10:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.479+0000 7fcb9d3c4640 1 -- 172.21.15.31:0/284722633 learned_addr learned my addr 172.21.15.31:0/284722633 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.479+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 0x55d3bbea8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.480+0000 7fcb97fff640 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbea8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.480+0000 7fcb9d3c4640 1 -- 172.21.15.31:0/284722633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 msgr2=0x55d3bbea8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.480+0000 7fcb9d3c4640 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 0x55d3bbea8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.480+0000 7fcb9d3c4640 1 -- 172.21.15.31:0/284722633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 msgr2=0x55d3bbea8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.480+0000 7fcb9d3c4640 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbea8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.480+0000 7fcb9d3c4640 1 -- 172.21.15.31:0/284722633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3bbb143f0 con 0x55d3bbafc390 2024-04-25T16:10:28.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.484+0000 7fcb9d3c4640 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafbd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcb880156d0 tx=0x7fcb88004700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c375b66c3c635833 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:28.487 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.486+0000 7fcb77fff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:10:28.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.489+0000 7fcb77fff640 1 -- 172.21.15.31:0/284722633 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb88013600 con 0x55d3bbafc390 2024-04-25T16:10:28.490 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.489+0000 7fcba5db9740 1 -- 172.21.15.31:0/284722633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb90019070 con 0x55d3bbafc390 2024-04-25T16:10:28.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.490+0000 7fcb77fff640 1 -- 172.21.15.31:0/284722633 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcb880137a0 con 0x55d3bbafc390 2024-04-25T16:10:28.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.490+0000 7fcb77fff640 1 -- 172.21.15.31:0/284722633 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcb880139a0 con 0x55d3bbafc390 2024-04-25T16:10:28.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.491+0000 7fcba5db9740 10 client.27106 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:28.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.491+0000 7fcba5db9740 1 -- 172.21.15.31:0/284722633 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d3bbbde7d0 con 0x55d3bbafc390 2024-04-25T16:10:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.491+0000 7fcb77fff640 1 -- 172.21.15.31:0/284722633 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb90019070 con 0x55d3bbafc390 2024-04-25T16:10:28.493 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.491+0000 7fcb77fff640 1 -- 172.21.15.31:0/284722633 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcb88012030 con 0x55d3bbafc390 2024-04-25T16:10:28.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.492+0000 7fcba5db9740 10 client.27106 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.492+0000 7fcba5db9740 10 client.27106 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:10:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.492+0000 7fcba5db9740 20 client.27106 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.492+0000 7fcba5db9740 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3bbabaf60 0x55d3bbeb6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.492+0000 7fcba5db9740 4 client.27106 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.492+0000 7fcba5db9740 1 -- 172.21.15.31:0/284722633 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d3bbe10990 con 0x55d3bbabaf60 2024-04-25T16:10:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.492+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3bbabaf60 0x55d3bbeb6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.493+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3bbabaf60 0x55d3bbeb6a20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f70c3bceac280662 server_cookie=b77585c62b72da7 in_seq=0 out_seq=0 2024-04-25T16:10:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.493+0000 7fcb77fff640 10 client.27106 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:10:28.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.494+0000 7fcb77fff640 1 -- 172.21.15.31:0/284722633 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d3bbe10990 con 0x55d3bbabaf60 2024-04-25T16:10:28.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.494+0000 7fcb77fff640 10 client.27106 handle_command_reply: tid=0 2024-04-25T16:10:28.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.494+0000 7fcba5db9740 1 client.27106 shutdown 2024-04-25T16:10:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcba5db9740 1 -- 172.21.15.31:0/284722633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 msgr2=0x55d3bbeafbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcba5db9740 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafbd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcb880156d0 tx=0x7fcb88004700 comp rx=0 tx=0).stop 2024-04-25T16:10:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcba5db9740 1 -- 172.21.15.31:0/284722633 shutdown_connections 2024-04-25T16:10:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcba5db9740 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafbd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcba5db9740 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3bbabaf60 0x55d3bbeb6a20 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcba5db9740 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbea8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcba5db9740 1 --2- 172.21.15.31:0/284722633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 0x55d3bbea8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcba5db9740 1 -- 172.21.15.31:0/284722633 >> 172.21.15.31:0/284722633 conn(0x55d3bbaf04e0 msgr2=0x55d3bbe9d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:28.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcb77fff640 0 client.27106 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:10:28.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.498+0000 7fcba5db9740 1 -- 172.21.15.31:0/284722633 shutdown_connections 2024-04-25T16:10:28.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.500+0000 7fcba5db9740 1 -- 172.21.15.31:0/284722633 wait complete. 2024-04-25T16:10:28.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.500+0000 7fcba5db9740 20 client.27106 trim_cache size 0 max 16384 2024-04-25T16:10:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.539+0000 7fcba5db9740 1 Processor -- start 2024-04-25T16:10:28.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.539+0000 7fcba5db9740 1 -- start start 2024-04-25T16:10:28.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.542+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 0x55d3bbe9e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.543+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.544 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.543+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.543+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.543+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35582/0 (socket says 172.21.15.31:35582) 2024-04-25T16:10:28.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.543+0000 7fcb97fff640 1 -- 172.21.15.31:0/570605070 learned_addr learned my addr 172.21.15.31:0/570605070 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:28.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.543+0000 7fcba5db9740 1 -- 172.21.15.31:0/570605070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400e540 con 0x55d3bbadb6b0 2024-04-25T16:10:28.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.543+0000 7fcba5db9740 1 -- 172.21.15.31:0/570605070 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcb84010080 con 0x55d3bbad34d0 2024-04-25T16:10:28.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.544+0000 7fcba5db9740 1 -- 172.21.15.31:0/570605070 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400ec10 con 0x55d3bbafc390 2024-04-25T16:10:28.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.544+0000 7fcb9d3c4640 1 --2- 172.21.15.31:0/570605070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.544+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/570605070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 0x55d3bbe9e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.545+0000 7fcb97fff640 1 -- 172.21.15.31:0/570605070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 msgr2=0x55d3bbeafaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:28.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.545+0000 7fcb97fff640 1 --2- 172.21.15.31:0/570605070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.545+0000 7fcb97fff640 1 -- 172.21.15.31:0/570605070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 msgr2=0x55d3bbe9e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:28.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.545+0000 7fcb97fff640 1 --2- 172.21.15.31:0/570605070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 0x55d3bbe9e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.545+0000 7fcb97fff640 1 -- 172.21.15.31:0/570605070 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3bbeb0030 con 0x55d3bbadb6b0 2024-04-25T16:10:28.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.546+0000 7fcb97fff640 1 --2- 172.21.15.31:0/570605070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9eec0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x55d3bbe9e720 tx=0x7fcb90004a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37df835686d1f71d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:28.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.548+0000 7fcb7e7fc640 1 -- 172.21.15.31:0/570605070 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90013810 con 0x55d3bbadb6b0 2024-04-25T16:10:28.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.548+0000 7fcb7e7fc640 1 -- 172.21.15.31:0/570605070 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb90019070 con 0x55d3bbadb6b0 2024-04-25T16:10:28.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.548+0000 7fcb7e7fc640 1 -- 172.21.15.31:0/570605070 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90014d60 con 0x55d3bbadb6b0 2024-04-25T16:10:28.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.550+0000 7fcba5db9740 1 -- 172.21.15.31:0/570605070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 msgr2=0x55d3bbe9eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:28.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.550+0000 7fcba5db9740 1 --2- 172.21.15.31:0/570605070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9eec0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x55d3bbe9e720 tx=0x7fcb90004a20 comp rx=0 tx=0).stop 2024-04-25T16:10:28.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.553+0000 7fcba5db9740 1 -- 172.21.15.31:0/570605070 shutdown_connections 2024-04-25T16:10:28.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.553+0000 7fcba5db9740 1 --2- 172.21.15.31:0/570605070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbafc390 0x55d3bbeafaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.553+0000 7fcba5db9740 1 --2- 172.21.15.31:0/570605070 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbadb6b0 0x55d3bbe9eec0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.553+0000 7fcba5db9740 1 --2- 172.21.15.31:0/570605070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbad34d0 0x55d3bbe9e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.553+0000 7fcba5db9740 1 -- 172.21.15.31:0/570605070 >> 172.21.15.31:0/570605070 conn(0x55d3bbaf04e0 msgr2=0x55d3bbea8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:28.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.556+0000 7fcba5db9740 1 -- 172.21.15.31:0/570605070 shutdown_connections 2024-04-25T16:10:28.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.556+0000 7fcba5db9740 1 -- 172.21.15.31:0/570605070 wait complete. 2024-04-25T16:10:28.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.557+0000 7fcba5db9740 1 -- start start 2024-04-25T16:10:28.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.557+0000 7fcba5db9740 1 Processor -- start 2024-04-25T16:10:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.565+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbeaf4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.565+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbadb6b0 0x55d3bbeafa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.565+0000 7fcba5db9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbafc390 0x55d3bbeac120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.565+0000 7fcba5db9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400f9b0 con 0x55d3bbafc390 2024-04-25T16:10:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.565+0000 7fcba5db9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400f2e0 con 0x55d3bbadb6b0 2024-04-25T16:10:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.565+0000 7fcba5db9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8400c520 con 0x55d3bbad34d0 2024-04-25T16:10:28.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.565+0000 7fcba5db9740 10 client.0 _ll_register_callbacks cb 0x7fcb8800c560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.566+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbadb6b0 0x55d3bbeafa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.566+0000 7fcb97fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbadb6b0 0x55d3bbeafa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58210/0 (socket says 172.21.15.31:58210) 2024-04-25T16:10:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.566+0000 7fcb97fff640 1 -- 172.21.15.31:0/2116222471 learned_addr learned my addr 172.21.15.31:0/2116222471 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.567+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbeaf4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.567+0000 7fcb97fff640 1 -- 172.21.15.31:0/2116222471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 msgr2=0x55d3bbeaf4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.567+0000 7fcb97fff640 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbeaf4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.567+0000 7fcb97fff640 1 -- 172.21.15.31:0/2116222471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbafc390 msgr2=0x55d3bbeac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:28.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.567+0000 7fcb97fff640 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbafc390 0x55d3bbeac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.567+0000 7fcb97fff640 1 -- 172.21.15.31:0/2116222471 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3bbeac850 con 0x55d3bbadb6b0 2024-04-25T16:10:28.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.567+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbeaf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:10:28.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.577+0000 7fcb97fff640 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbadb6b0 0x55d3bbeafa20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009710 tx=0x7fcb900144b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aec0020c448f7be8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:28.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.577+0000 7fcb77fff640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:10:28.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.581+0000 7fcb77fff640 1 -- 172.21.15.31:0/2116222471 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb90014cb0 con 0x55d3bbadb6b0 2024-04-25T16:10:28.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.581+0000 7fcb77fff640 1 -- 172.21.15.31:0/2116222471 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcb90013610 con 0x55d3bbadb6b0 2024-04-25T16:10:28.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.585+0000 7fcba5db9740 1 -- 172.21.15.31:0/2116222471 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb90019070 con 0x55d3bbadb6b0 2024-04-25T16:10:28.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.588+0000 7fcb77fff640 1 -- 172.21.15.31:0/2116222471 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcb90014e50 con 0x55d3bbadb6b0 2024-04-25T16:10:28.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.588+0000 7fcba5db9740 10 client.27116 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:28.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.588+0000 7fcba5db9740 1 -- 172.21.15.31:0/2116222471 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d3bbf1dab0 con 0x55d3bbadb6b0 2024-04-25T16:10:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.590+0000 7fcb77fff640 1 -- 172.21.15.31:0/2116222471 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcb90019070 con 0x55d3bbadb6b0 2024-04-25T16:10:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.590+0000 7fcb77fff640 1 -- 172.21.15.31:0/2116222471 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcb90030890 con 0x55d3bbadb6b0 2024-04-25T16:10:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.590+0000 7fcba5db9740 10 client.27116 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.590+0000 7fcba5db9740 10 client.27116 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:10:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.590+0000 7fcba5db9740 20 client.27116 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.590+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3bba903a0 0x55d3bbf203b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.590+0000 7fcba5db9740 4 client.27116 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:10:28.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.590+0000 7fcba5db9740 1 -- 172.21.15.31:0/2116222471 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d3bbe0a800 con 0x55d3bba903a0 2024-04-25T16:10:28.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.595+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3bba903a0 0x55d3bbf203b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:28.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.615+0000 7fcb9cbc3640 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3bba903a0 0x55d3bbf203b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1124ab19b827d51b server_cookie=3aad59352ac2e974 in_seq=0 out_seq=0 2024-04-25T16:10:28.617 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.615+0000 7fcb77fff640 10 client.27116 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:10:28.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.623+0000 7fcb77fff640 1 -- 172.21.15.31:0/2116222471 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d3bbe0a800 con 0x55d3bba903a0 2024-04-25T16:10:28.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.623+0000 7fcb77fff640 10 client.27116 handle_command_reply: tid=0 2024-04-25T16:10:28.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.627+0000 7fcba5db9740 1 client.27116 shutdown 2024-04-25T16:10:28.631 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcba5db9740 1 -- 172.21.15.31:0/2116222471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbadb6b0 msgr2=0x55d3bbeafa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbadb6b0 0x55d3bbeafa20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009710 tx=0x7fcb900144b0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcba5db9740 1 -- 172.21.15.31:0/2116222471 shutdown_connections 2024-04-25T16:10:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55d3bba903a0 0x55d3bbf203b0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d3bbafc390 0x55d3bbeac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d3bbadb6b0 0x55d3bbeafa20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcba5db9740 1 --2- 172.21.15.31:0/2116222471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d3bbad34d0 0x55d3bbeaf4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcba5db9740 1 -- 172.21.15.31:0/2116222471 >> 172.21.15.31:0/2116222471 conn(0x55d3bbaf04e0 msgr2=0x55d3bbea3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcb77fff640 0 client.27116 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:10:28.632 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.630+0000 7fcba5db9740 1 -- 172.21.15.31:0/2116222471 shutdown_connections 2024-04-25T16:10:28.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.632+0000 7fcba5db9740 1 -- 172.21.15.31:0/2116222471 wait complete. 2024-04-25T16:10:28.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.635+0000 7fcba5db9740 20 client.27116 trim_cache size 0 max 16384 2024-04-25T16:10:28.641 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:10:28.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.642+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb800afcf0 msgr2=0x7fcb800b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:28.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.643+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb800afcf0 0x7fcb800b2200 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fcb88006a50 tx=0x7fcb88008040 comp rx=0 tx=0).stop 2024-04-25T16:10:28.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.643+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb98101b90 msgr2=0x7fcb9810aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:28.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.643+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb98101b90 0x7fcb9810aed0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c004dc0 tx=0x7fcb8c004ae0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.643+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1404497389 shutdown_connections 2024-04-25T16:10:28.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.643+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcb800afcf0 0x7fcb800b2200 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.645 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.643+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcb98107ad0 0x7fcb98111f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.643+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb98102dd0 0x7fcb9810b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.643+0000 7fcb9dbc5640 1 --2- 172.21.15.31:0/1404497389 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcb98101b90 0x7fcb9810aed0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:28.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.644+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1404497389 >> 172.21.15.31:0/1404497389 conn(0x7fcb980fcda0 msgr2=0x7fcb98103ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:28.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.647+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1404497389 shutdown_connections 2024-04-25T16:10:28.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:28.647+0000 7fcb9dbc5640 1 -- 172.21.15.31:0/1404497389 wait complete. 2024-04-25T16:10:28.687 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:10:29.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.116+0000 7f0d42ffd640 1 -- 172.21.15.31:0/4178861047 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c02f320 con 0x7f0d4410b190 2024-04-25T16:10:29.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.117+0000 7f0d49502640 1 -- 172.21.15.31:0/4178861047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d4410b190 msgr2=0x7f0d4410b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:29.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.117+0000 7f0d49502640 1 --2- 172.21.15.31:0/4178861047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d4410b190 0x7f0d4410b5a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c009ca0 tx=0x7f0d3c02a8d0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.117+0000 7f0d49502640 1 -- 172.21.15.31:0/4178861047 shutdown_connections 2024-04-25T16:10:29.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.117+0000 7f0d49502640 1 --2- 172.21.15.31:0/4178861047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d4410fe90 0x7f0d441102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.117+0000 7f0d49502640 1 --2- 172.21.15.31:0/4178861047 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d4410b190 0x7f0d4410b5a0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.117+0000 7f0d49502640 1 --2- 172.21.15.31:0/4178861047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d4410a340 0x7f0d4410ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.117+0000 7f0d49502640 1 -- 172.21.15.31:0/4178861047 >> 172.21.15.31:0/4178861047 conn(0x7f0d44070320 msgr2=0x7f0d440727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:29.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.124+0000 7f0d49502640 1 -- 172.21.15.31:0/4178861047 shutdown_connections 2024-04-25T16:10:29.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.125+0000 7f0d49502640 1 -- 172.21.15.31:0/4178861047 wait complete. 2024-04-25T16:10:29.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.127+0000 7f0d49502640 1 Processor -- start 2024-04-25T16:10:29.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.127+0000 7f0d49502640 1 -- start start 2024-04-25T16:10:29.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.128+0000 7f0d49502640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d4410a340 0x7f0d441a8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.128+0000 7f0d49502640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d4410fe90 0x7f0d441a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.128+0000 7f0d49502640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441ae4b0 0x7f0d441b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.128+0000 7f0d49502640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4411e2e0 con 0x7f0d4410a340 2024-04-25T16:10:29.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.128+0000 7f0d49502640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4411e160 con 0x7f0d4410fe90 2024-04-25T16:10:29.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.128+0000 7f0d49502640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d4411e460 con 0x7f0d441ae4b0 2024-04-25T16:10:29.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.131+0000 7f0d437fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d4410fe90 0x7f0d441a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.131+0000 7f0d437fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d4410fe90 0x7f0d441a95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58232/0 (socket says 172.21.15.31:58232) 2024-04-25T16:10:29.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.131+0000 7f0d437fe640 1 -- 172.21.15.31:0/595662874 learned_addr learned my addr 172.21.15.31:0/595662874 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:29.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.132+0000 7f0d48d01640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441ae4b0 0x7f0d441b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.132+0000 7f0d43fff640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d4410a340 0x7f0d441a8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.137 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.135+0000 7f0d437fe640 1 -- 172.21.15.31:0/595662874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441ae4b0 msgr2=0x7f0d441b08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:29.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.135+0000 7f0d437fe640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441ae4b0 0x7f0d441b08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.135+0000 7f0d437fe640 1 -- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d4410a340 msgr2=0x7f0d441a8ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:29.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.135+0000 7f0d437fe640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d4410a340 0x7f0d441a8ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.135+0000 7f0d437fe640 1 -- 172.21.15.31:0/595662874 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d44111c90 con 0x7f0d4410fe90 2024-04-25T16:10:29.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.140+0000 7f0d437fe640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d4410fe90 0x7f0d441a95b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c02aec0 tx=0x7f0d3c003ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:29.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.141+0000 7f0d417fa640 1 -- 172.21.15.31:0/595662874 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c004240 con 0x7f0d4410fe90 2024-04-25T16:10:29.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.142+0000 7f0d417fa640 1 -- 172.21.15.31:0/595662874 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d3c0305d0 con 0x7f0d4410fe90 2024-04-25T16:10:29.143 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.142+0000 7f0d417fa640 1 -- 172.21.15.31:0/595662874 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3c03b670 con 0x7f0d4410fe90 2024-04-25T16:10:29.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.143+0000 7f0d49502640 1 -- 172.21.15.31:0/595662874 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d44111f20 con 0x7f0d4410fe90 2024-04-25T16:10:29.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.143+0000 7f0d49502640 1 -- 172.21.15.31:0/595662874 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d441124a0 con 0x7f0d4410fe90 2024-04-25T16:10:29.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.148+0000 7f0d51614740 1 Processor -- start 2024-04-25T16:10:29.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.152+0000 7f0d51614740 1 -- start start 2024-04-25T16:10:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.153+0000 7f0d51614740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe8d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.153+0000 7f0d51614740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 0x563d3fe8df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.153+0000 7f0d51614740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 0x563d3fe8ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.153+0000 7f0d51614740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563d3fdf1d80 con 0x563d3faca6b0 2024-04-25T16:10:29.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.153+0000 7f0d51614740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563d3fdaba20 con 0x563d3faeb390 2024-04-25T16:10:29.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.153+0000 7f0d51614740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563d3fdedd40 con 0x563d3fac24d0 2024-04-25T16:10:29.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.157+0000 7f0d48d01640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe8d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.157+0000 7f0d417fa640 1 -- 172.21.15.31:0/595662874 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0d3c030a80 con 0x7f0d4410fe90 2024-04-25T16:10:29.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.157+0000 7f0d48d01640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe8d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58236/0 (socket says 172.21.15.31:58236) 2024-04-25T16:10:29.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.157+0000 7f0d48d01640 1 -- 172.21.15.31:0/395135432 learned_addr learned my addr 172.21.15.31:0/395135432 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:29.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.158+0000 7f0d48d01640 1 -- 172.21.15.31:0/395135432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 msgr2=0x563d3fe8df60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:29.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.158+0000 7f0d48d01640 1 --2- 172.21.15.31:0/395135432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 0x563d3fe8df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.158+0000 7f0d48d01640 1 -- 172.21.15.31:0/395135432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 msgr2=0x563d3fe8ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:29.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.158+0000 7f0d48d01640 1 --2- 172.21.15.31:0/395135432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 0x563d3fe8ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.158+0000 7f0d48d01640 1 -- 172.21.15.31:0/395135432 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d3fdb6c30 con 0x563d3faeb390 2024-04-25T16:10:29.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.159+0000 7f0d48d01640 1 --2- 172.21.15.31:0/395135432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe8d9e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800ab80 tx=0x7f0d3800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3837366300cb1e94 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:29.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.160+0000 7f0d417fa640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0d2c0afcf0 0x7f0d2c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.161+0000 7f0d2a7fc640 1 -- 172.21.15.31:0/395135432 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3801d070 con 0x563d3faeb390 2024-04-25T16:10:29.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.161+0000 7f0d2a7fc640 1 -- 172.21.15.31:0/395135432 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d38018af0 con 0x563d3faeb390 2024-04-25T16:10:29.162 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.161+0000 7f0d2a7fc640 1 -- 172.21.15.31:0/395135432 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38010a70 con 0x563d3faeb390 2024-04-25T16:10:29.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.161+0000 7f0d417fa640 1 -- 172.21.15.31:0/595662874 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0d3c100260 con 0x7f0d4410fe90 2024-04-25T16:10:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.163+0000 7f0d51614740 1 -- 172.21.15.31:0/395135432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 msgr2=0x563d3fe8d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:29.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.163+0000 7f0d51614740 1 --2- 172.21.15.31:0/395135432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe8d9e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0d3800ab80 tx=0x7f0d3800f790 comp rx=0 tx=0).stop 2024-04-25T16:10:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.163+0000 7f0d51614740 1 -- 172.21.15.31:0/395135432 shutdown_connections 2024-04-25T16:10:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.163+0000 7f0d51614740 1 --2- 172.21.15.31:0/395135432 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 0x563d3fe8ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.163+0000 7f0d51614740 1 --2- 172.21.15.31:0/395135432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 0x563d3fe8df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.163+0000 7f0d51614740 1 --2- 172.21.15.31:0/395135432 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe8d9e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.163+0000 7f0d51614740 1 -- 172.21.15.31:0/395135432 >> 172.21.15.31:0/395135432 conn(0x563d3fadf4e0 msgr2=0x563d3fe891f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:29.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.164+0000 7f0d43fff640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0d2c0afcf0 0x7f0d2c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.166+0000 7f0d51614740 1 -- 172.21.15.31:0/395135432 shutdown_connections 2024-04-25T16:10:29.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.170+0000 7f0d43fff640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0d2c0afcf0 0x7f0d2c0b2200 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x563d3fe89f80 tx=0x7f0d34008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:29.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.173+0000 7f0d51614740 1 -- 172.21.15.31:0/395135432 wait complete. 2024-04-25T16:10:29.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.174+0000 7f0d51614740 1 -- start start 2024-04-25T16:10:29.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.174+0000 7f0d51614740 1 Processor -- start 2024-04-25T16:10:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.175+0000 7f0d51614740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3fac24d0 0x563d3fe978b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.175+0000 7f0d51614740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faca6b0 0x563d3fe97df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.176+0000 7f0d51614740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 0x563d3fe9ed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.176+0000 7f0d437fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faca6b0 0x563d3fe97df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.176+0000 7f0d437fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faca6b0 0x563d3fe97df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58252/0 (socket says 172.21.15.31:58252) 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.176+0000 7f0d437fe640 1 -- 172.21.15.31:0/1350782328 learned_addr learned my addr 172.21.15.31:0/1350782328 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.176+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563d3fe8f700 con 0x563d3fac24d0 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.176+0000 7f0d48d01640 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 0x563d3fe9ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.176+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563d3fd78dc0 con 0x563d3faca6b0 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.176+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563d3fe8f880 con 0x563d3faeb390 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.177+0000 7f0d51614740 10 client.0 _ll_register_callbacks cb 0x563d3fd94600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.177+0000 7f0d48d01640 1 -- 172.21.15.31:0/1350782328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faca6b0 msgr2=0x563d3fe97df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.177+0000 7f0d48d01640 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faca6b0 0x563d3fe97df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.177+0000 7f0d48d01640 1 -- 172.21.15.31:0/1350782328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3fac24d0 msgr2=0x563d3fe978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:29.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.177+0000 7f0d48d01640 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3fac24d0 0x563d3fe978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.177+0000 7f0d48d01640 1 -- 172.21.15.31:0/1350782328 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d3fb033f0 con 0x563d3faeb390 2024-04-25T16:10:29.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.177+0000 7f0d48d01640 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 0x563d3fe9ed50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0d38006010 tx=0x7f0d38018ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b21abcad2a49d4e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:29.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.177+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d38018af0 con 0x563d3faeb390 2024-04-25T16:10:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.181+0000 7f0d237fe640 10 client.27115 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:10:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.181+0000 7f0d237fe640 1 -- 172.21.15.31:0/1350782328 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d38027020 con 0x563d3faeb390 2024-04-25T16:10:29.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.182+0000 7f0d237fe640 1 -- 172.21.15.31:0/1350782328 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0d38005450 con 0x563d3faeb390 2024-04-25T16:10:29.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.182+0000 7f0d237fe640 1 -- 172.21.15.31:0/1350782328 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0d3801d070 con 0x563d3faeb390 2024-04-25T16:10:29.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.186+0000 7f0d51614740 10 client.27115 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:29.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.186+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({fsmap=29}) -- 0x563d3fbcd7d0 con 0x563d3faeb390 2024-04-25T16:10:29.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.194+0000 7f0d237fe640 1 -- 172.21.15.31:0/1350782328 <== mon.1 v2:172.21.15.186:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0d38010770 con 0x563d3faeb390 2024-04-25T16:10:29.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.195+0000 7f0d51614740 10 client.27115 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:29.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.195+0000 7f0d51614740 10 client.27115 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:29.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.195+0000 7f0d51614740 20 client.27115 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:29.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.195+0000 7f0d51614740 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563d3faa9f60 0x563d3fea5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.195+0000 7f0d51614740 4 client.27115 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:29.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.195+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f0d38018af0 con 0x563d3faa9f60 2024-04-25T16:10:29.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.198+0000 7f0d43fff640 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563d3faa9f60 0x563d3fea5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.211+0000 7f0d43fff640 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563d3faa9f60 0x563d3fea5090 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72d1c4e188640bb4 server_cookie=77cb76afbe8e2200 in_seq=0 out_seq=0 2024-04-25T16:10:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.212+0000 7f0d237fe640 10 client.27115 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:29.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.217+0000 7f0d237fe640 1 -- 172.21.15.31:0/1350782328 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f0d38018af0 con 0x563d3faa9f60 2024-04-25T16:10:29.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.217+0000 7f0d237fe640 10 client.27115 handle_command_reply: tid=0 2024-04-25T16:10:29.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.218+0000 7f0d51614740 1 client.27115 shutdown 2024-04-25T16:10:29.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.219+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 msgr2=0x563d3fe9ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:29.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.219+0000 7f0d51614740 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 0x563d3fe9ed50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f0d38006010 tx=0x7f0d38018ed0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.221+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 shutdown_connections 2024-04-25T16:10:29.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.221+0000 7f0d51614740 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563d3faa9f60 0x563d3fea5090 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.221+0000 7f0d51614740 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 0x563d3fe9ed50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.221+0000 7f0d51614740 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faca6b0 0x563d3fe97df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.221+0000 7f0d51614740 1 --2- 172.21.15.31:0/1350782328 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3fac24d0 0x563d3fe978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.221+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 >> 172.21.15.31:0/1350782328 conn(0x563d3fadf4e0 msgr2=0x563d3fe883f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:29.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.222+0000 7f0d237fe640 0 client.27115 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:29.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.222+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 shutdown_connections 2024-04-25T16:10:29.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.223+0000 7f0d51614740 1 -- 172.21.15.31:0/1350782328 wait complete. 2024-04-25T16:10:29.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.223+0000 7f0d51614740 20 client.27115 trim_cache size 0 max 16384 2024-04-25T16:10:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.260+0000 7f0d51614740 1 Processor -- start 2024-04-25T16:10:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.260+0000 7f0d51614740 1 -- start start 2024-04-25T16:10:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.261+0000 7f0d51614740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 0x563d3fe97fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.261+0000 7f0d51614740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 0x563d3fe96b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.261+0000 7f0d51614740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe97080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.261+0000 7f0d51614740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0d441c2ed0 con 0x563d3faca6b0 2024-04-25T16:10:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.261+0000 7f0d51614740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0d441c4a10 con 0x563d3faeb390 2024-04-25T16:10:29.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.261+0000 7f0d51614740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d441c35a0 con 0x563d3fac24d0 2024-04-25T16:10:29.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.263+0000 7f0d43fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 0x563d3fe97fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.263+0000 7f0d43fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 0x563d3fe97fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54136/0 (socket says 172.21.15.31:54136) 2024-04-25T16:10:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.263+0000 7f0d43fff640 1 -- 172.21.15.31:0/3404454246 learned_addr learned my addr 172.21.15.31:0/3404454246 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.263+0000 7f0d48d01640 1 --2- 172.21.15.31:0/3404454246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe97080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.264+0000 7f0d43fff640 1 -- 172.21.15.31:0/3404454246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 msgr2=0x563d3fe97080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.264+0000 7f0d43fff640 1 --2- 172.21.15.31:0/3404454246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe97080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.264+0000 7f0d43fff640 1 -- 172.21.15.31:0/3404454246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 msgr2=0x563d3fe96b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.264+0000 7f0d43fff640 1 --2- 172.21.15.31:0/3404454246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 0x563d3fe96b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.264+0000 7f0d43fff640 1 -- 172.21.15.31:0/3404454246 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d3fe7aba0 con 0x563d3fac24d0 2024-04-25T16:10:29.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.265+0000 7f0d43fff640 1 --2- 172.21.15.31:0/3404454246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 0x563d3fe97fc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0d34000c00 tx=0x7f0d340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1f3c2bbed1fe631 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:29.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.268+0000 7f0d29ffb640 1 -- 172.21.15.31:0/3404454246 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d3400dd60 con 0x563d3fac24d0 2024-04-25T16:10:29.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.268+0000 7f0d29ffb640 1 -- 172.21.15.31:0/3404454246 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d34016dd0 con 0x563d3fac24d0 2024-04-25T16:10:29.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.269+0000 7f0d51614740 1 -- 172.21.15.31:0/3404454246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 msgr2=0x563d3fe97fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:29.274 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.269+0000 7f0d51614740 1 --2- 172.21.15.31:0/3404454246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 0x563d3fe97fc0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0d34000c00 tx=0x7f0d340046c0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.273+0000 7f0d51614740 1 -- 172.21.15.31:0/3404454246 shutdown_connections 2024-04-25T16:10:29.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.273+0000 7f0d51614740 1 --2- 172.21.15.31:0/3404454246 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3faeb390 0x563d3fe97080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.273+0000 7f0d51614740 1 --2- 172.21.15.31:0/3404454246 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 0x563d3fe96b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.273+0000 7f0d51614740 1 --2- 172.21.15.31:0/3404454246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3fac24d0 0x563d3fe97fc0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.273+0000 7f0d51614740 1 -- 172.21.15.31:0/3404454246 >> 172.21.15.31:0/3404454246 conn(0x563d3fadf4e0 msgr2=0x563d3fe978b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.278+0000 7f0d51614740 1 -- 172.21.15.31:0/3404454246 shutdown_connections 2024-04-25T16:10:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.278+0000 7f0d51614740 1 -- 172.21.15.31:0/3404454246 wait complete. 2024-04-25T16:10:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.278+0000 7f0d51614740 1 -- start start 2024-04-25T16:10:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.279+0000 7f0d51614740 1 Processor -- start 2024-04-25T16:10:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d51614740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3fac24d0 0x563d3fe9b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d51614740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 0x563d3fe9bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d51614740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 0x563d3fe9ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d51614740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0d441c4340 con 0x563d3faca6b0 2024-04-25T16:10:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d51614740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0d441c3c70 con 0x563d3fac24d0 2024-04-25T16:10:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d51614740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0d44119340 con 0x563d3faeb390 2024-04-25T16:10:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d51614740 10 client.0 _ll_register_callbacks cb 0x7f0d3801b070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d43fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3fac24d0 0x563d3fe9b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d43fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3fac24d0 0x563d3fe9b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58258/0 (socket says 172.21.15.31:58258) 2024-04-25T16:10:29.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.280+0000 7f0d43fff640 1 -- 172.21.15.31:0/2963130245 learned_addr learned my addr 172.21.15.31:0/2963130245 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:29.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.283+0000 7f0d48d01640 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 0x563d3fe9ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.285+0000 7f0d43fff640 1 -- 172.21.15.31:0/2963130245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 msgr2=0x563d3fe9ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.285+0000 7f0d43fff640 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 0x563d3fe9ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.285+0000 7f0d43fff640 1 -- 172.21.15.31:0/2963130245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 msgr2=0x563d3fe9bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:29.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.285+0000 7f0d43fff640 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 0x563d3fe9bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.287 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.285+0000 7f0d43fff640 1 -- 172.21.15.31:0/2963130245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d3fe9d290 con 0x563d3fac24d0 2024-04-25T16:10:29.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.286+0000 7f0d43fff640 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3fac24d0 0x563d3fe9b860 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400d820 tx=0x7f0d34023bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74c711a79f9590f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:29.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.286+0000 7f0d237fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:10:29.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.286+0000 7f0d237fe640 1 -- 172.21.15.31:0/2963130245 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d34030070 con 0x563d3fac24d0 2024-04-25T16:10:29.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.286+0000 7f0d51614740 1 -- 172.21.15.31:0/2963130245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0d34016dd0 con 0x563d3fac24d0 2024-04-25T16:10:29.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.286+0000 7f0d237fe640 1 -- 172.21.15.31:0/2963130245 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0d34016b20 con 0x563d3fac24d0 2024-04-25T16:10:29.288 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.287+0000 7f0d237fe640 1 -- 172.21.15.31:0/2963130245 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0d3403c050 con 0x563d3fac24d0 2024-04-25T16:10:29.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.292+0000 7f0d51614740 10 client.27131 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:29.293 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.292+0000 7f0d51614740 1 -- 172.21.15.31:0/2963130245 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563d3fe9e4f0 con 0x563d3fac24d0 2024-04-25T16:10:29.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.294+0000 7f0d237fe640 1 -- 172.21.15.31:0/2963130245 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0d34016dd0 con 0x563d3fac24d0 2024-04-25T16:10:29.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.295+0000 7f0d237fe640 1 -- 172.21.15.31:0/2963130245 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0d34013070 con 0x563d3fac24d0 2024-04-25T16:10:29.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.295+0000 7f0d51614740 10 client.27131 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:29.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.295+0000 7f0d51614740 10 client.27131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:29.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.295+0000 7f0d51614740 20 client.27131 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:29.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.295+0000 7f0d51614740 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563d3fa7f3a0 0x563d3ff31810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:29.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.295+0000 7f0d51614740 4 client.27131 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:10:29.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.295+0000 7f0d51614740 1 -- 172.21.15.31:0/2963130245 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563d3fdff990 con 0x563d3fa7f3a0 2024-04-25T16:10:29.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.295+0000 7f0d437fe640 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563d3fa7f3a0 0x563d3ff31810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:29.301 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.296+0000 7f0d437fe640 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563d3fa7f3a0 0x563d3ff31810 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b897108e29267f3a server_cookie=63558af9fcd9c867 in_seq=0 out_seq=0 2024-04-25T16:10:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.300+0000 7f0d237fe640 10 client.27131 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.300+0000 7f0d237fe640 1 -- 172.21.15.31:0/2963130245 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563d3fdff990 con 0x563d3fa7f3a0 2024-04-25T16:10:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.300+0000 7f0d237fe640 10 client.27131 handle_command_reply: tid=0 2024-04-25T16:10:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.300+0000 7f0d51614740 1 client.27131 shutdown 2024-04-25T16:10:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.300+0000 7f0d51614740 1 -- 172.21.15.31:0/2963130245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3fac24d0 msgr2=0x563d3fe9b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:29.302 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.300+0000 7f0d51614740 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3fac24d0 0x563d3fe9b860 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400d820 tx=0x7f0d34023bc0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.301+0000 7f0d51614740 1 -- 172.21.15.31:0/2963130245 shutdown_connections 2024-04-25T16:10:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.301+0000 7f0d51614740 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563d3fa7f3a0 0x563d3ff31810 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.301+0000 7f0d51614740 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563d3faeb390 0x563d3fe9ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.301+0000 7f0d51614740 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563d3faca6b0 0x563d3fe9bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.301+0000 7f0d51614740 1 --2- 172.21.15.31:0/2963130245 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563d3fac24d0 0x563d3fe9b860 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.301+0000 7f0d51614740 1 -- 172.21.15.31:0/2963130245 >> 172.21.15.31:0/2963130245 conn(0x563d3fadf4e0 msgr2=0x563d3fe8ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.301+0000 7f0d237fe640 0 client.27131 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.301+0000 7f0d51614740 1 -- 172.21.15.31:0/2963130245 shutdown_connections 2024-04-25T16:10:29.303 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.301+0000 7f0d51614740 1 -- 172.21.15.31:0/2963130245 wait complete. 2024-04-25T16:10:29.305 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.304+0000 7f0d51614740 20 client.27131 trim_cache size 0 max 16384 2024-04-25T16:10:29.309 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:10:29.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 -- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0d2c0afcf0 msgr2=0x7f0d2c0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:29.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0d2c0afcf0 0x7f0d2c0b2200 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x563d3fe89f80 tx=0x7f0d34008040 comp rx=0 tx=0).stop 2024-04-25T16:10:29.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 -- 172.21.15.31:0/595662874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d4410fe90 msgr2=0x7f0d441a95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:29.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d4410fe90 0x7f0d441a95b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c02aec0 tx=0x7f0d3c003ce0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 -- 172.21.15.31:0/595662874 shutdown_connections 2024-04-25T16:10:29.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0d2c0afcf0 0x7f0d2c0b2200 secure :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0x563d3fe89f80 tx=0x7f0d34008040 comp rx=0 tx=0).stop 2024-04-25T16:10:29.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d441ae4b0 0x7f0d441b08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0d4410fe90 0x7f0d441a95b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 --2- 172.21.15.31:0/595662874 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0d4410a340 0x7f0d441a8ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:29.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 -- 172.21.15.31:0/595662874 >> 172.21.15.31:0/595662874 conn(0x7f0d44070320 msgr2=0x7f0d4410be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:29.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.308+0000 7f0d49502640 1 -- 172.21.15.31:0/595662874 shutdown_connections 2024-04-25T16:10:29.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:29.309+0000 7f0d49502640 1 -- 172.21.15.31:0/595662874 wait complete. 2024-04-25T16:10:29.347 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:29 smithi031 ceph-mon[29140]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 951 MiB/s wr, 1.88k op/s 2024-04-25T16:10:29.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:29 smithi063 ceph-mon[32413]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 951 MiB/s wr, 1.88k op/s 2024-04-25T16:10:29.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:29 smithi186 ceph-mon[33258]: pgmap v626: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 951 MiB/s wr, 1.88k op/s 2024-04-25T16:10:30.348 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:10:30.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:10:30.348 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:10:30.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.873+0000 7fe822a96640 1 -- 172.21.15.31:0/3082372824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81c10b760 msgr2=0x7fe81c10bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:30.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.873+0000 7fe822a96640 1 --2- 172.21.15.31:0/3082372824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81c10b760 0x7fe81c10bb70 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fe814005fd0 tx=0x7fe81402e6e0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.879+0000 7fe822a96640 1 -- 172.21.15.31:0/3082372824 shutdown_connections 2024-04-25T16:10:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.879+0000 7fe822a96640 1 --2- 172.21.15.31:0/3082372824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe81c110460 0x7fe81c112880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.879+0000 7fe822a96640 1 --2- 172.21.15.31:0/3082372824 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81c10b760 0x7fe81c10bb70 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.879+0000 7fe822a96640 1 --2- 172.21.15.31:0/3082372824 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe81c10a520 0x7fe81c10b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.879+0000 7fe822a96640 1 -- 172.21.15.31:0/3082372824 >> 172.21.15.31:0/3082372824 conn(0x7fe81c070940 msgr2=0x7fe81c072df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:30.882 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.879+0000 7fe822a96640 1 -- 172.21.15.31:0/3082372824 shutdown_connections 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.886+0000 7fe822a96640 1 -- 172.21.15.31:0/3082372824 wait complete. 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.887+0000 7fe822a96640 1 Processor -- start 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.887+0000 7fe822a96640 1 -- start start 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.887+0000 7fe822a96640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe81c10a520 0x7fe81c1a8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.887+0000 7fe822a96640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81c10b760 0x7fe81c1a9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.887+0000 7fe822a96640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe81c110460 0x7fe81c1b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.887+0000 7fe822a96640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c11e460 con 0x7fe81c10b760 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.887+0000 7fe822a96640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c11e2e0 con 0x7fe81c110460 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.887+0000 7fe822a96640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe81c11e5e0 con 0x7fe81c10a520 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.888+0000 7fe821033640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe81c110460 0x7fe81c1b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.888+0000 7fe821033640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe81c110460 0x7fe81c1b04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58270/0 (socket says 172.21.15.31:58270) 2024-04-25T16:10:30.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.888+0000 7fe821033640 1 -- 172.21.15.31:0/2257632423 learned_addr learned my addr 172.21.15.31:0/2257632423 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:30.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.888+0000 7fe820832640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe81c10a520 0x7fe81c1a8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:30.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.889+0000 7fe821033640 1 -- 172.21.15.31:0/2257632423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe81c10a520 msgr2=0x7fe81c1a8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:30.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.889+0000 7fe821033640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe81c10a520 0x7fe81c1a8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.889+0000 7fe821033640 1 -- 172.21.15.31:0/2257632423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81c10b760 msgr2=0x7fe81c1a9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:30.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.889+0000 7fe821033640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81c10b760 0x7fe81c1a9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.889+0000 7fe821033640 1 -- 172.21.15.31:0/2257632423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe81c114d20 con 0x7fe81c110460 2024-04-25T16:10:30.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.889+0000 7fe821033640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe81c110460 0x7fe81c1b04b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c970 tx=0x7fe80c00ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:30.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.889+0000 7fe819ffb640 1 -- 172.21.15.31:0/2257632423 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c007bb0 con 0x7fe81c110460 2024-04-25T16:10:30.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.893+0000 7fe819ffb640 1 -- 172.21.15.31:0/2257632423 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe80c007d50 con 0x7fe81c110460 2024-04-25T16:10:30.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.893+0000 7fe819ffb640 1 -- 172.21.15.31:0/2257632423 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c018730 con 0x7fe81c110460 2024-04-25T16:10:30.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.895+0000 7fe822a96640 1 -- 172.21.15.31:0/2257632423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe81c115010 con 0x7fe81c110460 2024-04-25T16:10:30.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.895+0000 7fe822a96640 1 -- 172.21.15.31:0/2257632423 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe81c115510 con 0x7fe81c110460 2024-04-25T16:10:30.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.911+0000 7fe819ffb640 1 -- 172.21.15.31:0/2257632423 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe80c0213f0 con 0x7fe81c110460 2024-04-25T16:10:30.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.913+0000 7fe819ffb640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8040afc20 0x7fe8040b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.914+0000 7fe819ffb640 1 -- 172.21.15.31:0/2257632423 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe80c0dfc40 con 0x7fe81c110460 2024-04-25T16:10:30.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.920+0000 7fe829a24740 1 Processor -- start 2024-04-25T16:10:30.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.920+0000 7fe829a24740 1 -- start start 2024-04-25T16:10:30.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.920+0000 7fe829a24740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 0x55f27c9464a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.920+0000 7fe829a24740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c57c550 0x55f27c946a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.920+0000 7fe829a24740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c94d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.921+0000 7fe820832640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8040afc20 0x7fe8040b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.921+0000 7fe829a24740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f27c8abb00 con 0x55f27c57c550 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.921+0000 7fe829a24740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f27c847920 con 0x55f27c584730 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.921+0000 7fe829a24740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f27c8a7ac0 con 0x55f27c5a5410 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.922+0000 7fe821033640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c94d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.922+0000 7fe821033640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c94d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58276/0 (socket says 172.21.15.31:58276) 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.922+0000 7fe821033640 1 -- 172.21.15.31:0/1174921772 learned_addr learned my addr 172.21.15.31:0/1174921772 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.923+0000 7fe821033640 1 -- 172.21.15.31:0/1174921772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 msgr2=0x55f27c9464a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.923+0000 7fe821033640 1 --2- 172.21.15.31:0/1174921772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 0x55f27c9464a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.923+0000 7fe821033640 1 -- 172.21.15.31:0/1174921772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c57c550 msgr2=0x55f27c946a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.923+0000 7fe821033640 1 --2- 172.21.15.31:0/1174921772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c57c550 0x55f27c946a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.923+0000 7fe821033640 1 -- 172.21.15.31:0/1174921772 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f27c6857e0 con 0x55f27c584730 2024-04-25T16:10:30.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.924+0000 7fe821033640 1 --2- 172.21.15.31:0/1174921772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c94d920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00afd0 tx=0x7fe80c0a7900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5ab0fdd0c947b03 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:30.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.931+0000 7fe820832640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8040afc20 0x7fe8040b2130 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55f27c947670 tx=0x7fe81000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:30.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.931+0000 7fe802ffd640 1 -- 172.21.15.31:0/1174921772 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c0c2070 con 0x55f27c584730 2024-04-25T16:10:30.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.932+0000 7fe802ffd640 1 -- 172.21.15.31:0/1174921772 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe80c0ac480 con 0x55f27c584730 2024-04-25T16:10:30.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.932+0000 7fe802ffd640 1 -- 172.21.15.31:0/1174921772 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c0b0aa0 con 0x55f27c584730 2024-04-25T16:10:30.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.934+0000 7fe829a24740 1 -- 172.21.15.31:0/1174921772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 msgr2=0x55f27c94d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:30.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.934+0000 7fe829a24740 1 --2- 172.21.15.31:0/1174921772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c94d920 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00afd0 tx=0x7fe80c0a7900 comp rx=0 tx=0).stop 2024-04-25T16:10:30.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.935+0000 7fe829a24740 1 -- 172.21.15.31:0/1174921772 shutdown_connections 2024-04-25T16:10:30.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.935+0000 7fe829a24740 1 --2- 172.21.15.31:0/1174921772 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c94d920 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.935+0000 7fe829a24740 1 --2- 172.21.15.31:0/1174921772 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c57c550 0x55f27c946a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.935+0000 7fe829a24740 1 --2- 172.21.15.31:0/1174921772 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 0x55f27c9464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.935+0000 7fe829a24740 1 -- 172.21.15.31:0/1174921772 >> 172.21.15.31:0/1174921772 conn(0x55f27c599560 msgr2=0x55f27c941e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:30.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.935+0000 7fe829a24740 1 -- 172.21.15.31:0/1174921772 shutdown_connections 2024-04-25T16:10:30.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.935+0000 7fe829a24740 1 -- 172.21.15.31:0/1174921772 wait complete. 2024-04-25T16:10:30.939 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.936+0000 7fe829a24740 1 -- start start 2024-04-25T16:10:30.940 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.938+0000 7fe829a24740 1 Processor -- start 2024-04-25T16:10:30.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.939+0000 7fe829a24740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c950370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.939+0000 7fe829a24740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c9508b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.939+0000 7fe829a24740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c5a5410 0x55f27c957810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.939+0000 7fe829a24740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f27c94e370 con 0x55f27c5a5410 2024-04-25T16:10:30.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.939+0000 7fe829a24740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f27c8455d0 con 0x55f27c584730 2024-04-25T16:10:30.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.939+0000 7fe829a24740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f27c94e4f0 con 0x55f27c57c550 2024-04-25T16:10:30.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.939+0000 7fe829a24740 10 client.0 _ll_register_callbacks cb 0x55f27c833080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:30.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.940+0000 7fe81bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c9508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:30.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.940+0000 7fe81bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c9508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58282/0 (socket says 172.21.15.31:58282) 2024-04-25T16:10:30.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.940+0000 7fe81bfff640 1 -- 172.21.15.31:0/3906709572 learned_addr learned my addr 172.21.15.31:0/3906709572 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:30.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.941+0000 7fe820832640 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c950370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:30.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.942+0000 7fe821033640 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c5a5410 0x55f27c957810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:30.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.943+0000 7fe820832640 1 -- 172.21.15.31:0/3906709572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 msgr2=0x55f27c9508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:30.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.943+0000 7fe820832640 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c9508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.943+0000 7fe820832640 1 -- 172.21.15.31:0/3906709572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c5a5410 msgr2=0x55f27c957810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:30.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.943+0000 7fe820832640 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c5a5410 0x55f27c957810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.943+0000 7fe820832640 1 -- 172.21.15.31:0/3906709572 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f27c6e5f70 con 0x55f27c57c550 2024-04-25T16:10:30.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.945+0000 7fe820832640 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c950370 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe81c10b120 tx=0x7fe8100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=163633a8a81ac24a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:30.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.946+0000 7fe7fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:10:30.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.946+0000 7fe7fbfff640 1 -- 172.21.15.31:0/3906709572 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe810025020 con 0x55f27c57c550 2024-04-25T16:10:30.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.946+0000 7fe7fbfff640 1 -- 172.21.15.31:0/3906709572 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe810015910 con 0x55f27c57c550 2024-04-25T16:10:30.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.950+0000 7fe829a24740 1 -- 172.21.15.31:0/3906709572 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe80c0ac480 con 0x55f27c57c550 2024-04-25T16:10:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.951+0000 7fe7fbfff640 1 -- 172.21.15.31:0/3906709572 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe81001d040 con 0x55f27c57c550 2024-04-25T16:10:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.952+0000 7fe829a24740 10 client.27127 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:30.953 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.952+0000 7fe829a24740 1 -- 172.21.15.31:0/3906709572 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f27c57d170 con 0x55f27c57c550 2024-04-25T16:10:30.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.953+0000 7fe7fbfff640 1 -- 172.21.15.31:0/3906709572 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe80c0ac480 con 0x55f27c57c550 2024-04-25T16:10:30.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.953+0000 7fe7fbfff640 1 -- 172.21.15.31:0/3906709572 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe810032020 con 0x55f27c57c550 2024-04-25T16:10:30.954 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.953+0000 7fe829a24740 10 client.27127 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:30.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.953+0000 7fe829a24740 10 client.27127 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:30.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.953+0000 7fe829a24740 20 client.27127 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:30.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.953+0000 7fe829a24740 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f27c5641f0 0x55f27c95de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:30.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.953+0000 7fe829a24740 4 client.27127 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:30.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.953+0000 7fe829a24740 1 -- 172.21.15.31:0/3906709572 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f27c8b9710 con 0x55f27c5641f0 2024-04-25T16:10:30.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.955+0000 7fe81bfff640 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f27c5641f0 0x55f27c95de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:30.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.958+0000 7fe81bfff640 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f27c5641f0 0x55f27c95de40 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a06121000ff93bb3 server_cookie=c272a8b0617a2ad1 in_seq=0 out_seq=0 2024-04-25T16:10:30.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.958+0000 7fe7fbfff640 10 client.27127 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:30.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.960+0000 7fe7fbfff640 1 -- 172.21.15.31:0/3906709572 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f27c8b9710 con 0x55f27c5641f0 2024-04-25T16:10:30.961 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.960+0000 7fe7fbfff640 10 client.27127 handle_command_reply: tid=0 2024-04-25T16:10:30.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.960+0000 7fe829a24740 1 client.27127 shutdown 2024-04-25T16:10:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.964+0000 7fe829a24740 1 -- 172.21.15.31:0/3906709572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 msgr2=0x55f27c950370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:30.965 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.964+0000 7fe829a24740 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c950370 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fe81c10b120 tx=0x7fe8100046c0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.966+0000 7fe829a24740 1 -- 172.21.15.31:0/3906709572 shutdown_connections 2024-04-25T16:10:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.966+0000 7fe829a24740 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f27c5641f0 0x55f27c95de40 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.966+0000 7fe829a24740 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c5a5410 0x55f27c957810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.966+0000 7fe829a24740 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c584730 0x55f27c9508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.966+0000 7fe829a24740 1 --2- 172.21.15.31:0/3906709572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c950370 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:30.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.966+0000 7fe829a24740 1 -- 172.21.15.31:0/3906709572 >> 172.21.15.31:0/3906709572 conn(0x55f27c599560 msgr2=0x55f27c940ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:30.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.967+0000 7fe7fbfff640 0 client.27127 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:30.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.967+0000 7fe829a24740 1 -- 172.21.15.31:0/3906709572 shutdown_connections 2024-04-25T16:10:30.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.973+0000 7fe829a24740 1 -- 172.21.15.31:0/3906709572 wait complete. 2024-04-25T16:10:30.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:30.977+0000 7fe829a24740 20 client.27127 trim_cache size 0 max 16384 2024-04-25T16:10:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.067+0000 7fe829a24740 1 Processor -- start 2024-04-25T16:10:31.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.070+0000 7fe829a24740 1 -- start start 2024-04-25T16:10:31.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.071+0000 7fe829a24740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c57c550 0x55f27c942050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.071+0000 7fe829a24740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c584730 0x55f27c940bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.072+0000 7fe829a24740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 0x55f27c9577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.072+0000 7fe829a24740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe80800e550 con 0x55f27c584730 2024-04-25T16:10:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.072+0000 7fe829a24740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe808010090 con 0x55f27c57c550 2024-04-25T16:10:31.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.072+0000 7fe829a24740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe80800ec20 con 0x55f27c5a5410 2024-04-25T16:10:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.074+0000 7fe821033640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 0x55f27c9577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.074+0000 7fe821033640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 0x55f27c9577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54204/0 (socket says 172.21.15.31:54204) 2024-04-25T16:10:31.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.074+0000 7fe821033640 1 -- 172.21.15.31:0/3305837460 learned_addr learned my addr 172.21.15.31:0/3305837460 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.075+0000 7fe821033640 1 -- 172.21.15.31:0/3305837460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c57c550 msgr2=0x55f27c942050 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.075+0000 7fe821033640 1 --2- 172.21.15.31:0/3305837460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c57c550 0x55f27c942050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.075+0000 7fe821033640 1 -- 172.21.15.31:0/3305837460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c584730 msgr2=0x55f27c940bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.075+0000 7fe821033640 1 --2- 172.21.15.31:0/3305837460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c584730 0x55f27c940bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.076+0000 7fe821033640 1 -- 172.21.15.31:0/3305837460 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f27c564420 con 0x55f27c5a5410 2024-04-25T16:10:31.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.132+0000 7fe821033640 1 --2- 172.21.15.31:0/3305837460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 0x55f27c9577a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe80c007590 tx=0x7fe80c0a8910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3444dd58592cfdff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:31.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.142+0000 7fe8027fc640 1 -- 172.21.15.31:0/3305837460 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c0c7020 con 0x55f27c5a5410 2024-04-25T16:10:31.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.142+0000 7fe8027fc640 1 -- 172.21.15.31:0/3305837460 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe80c0b0f30 con 0x55f27c5a5410 2024-04-25T16:10:31.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.142+0000 7fe8027fc640 1 -- 172.21.15.31:0/3305837460 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe80c0c2040 con 0x55f27c5a5410 2024-04-25T16:10:31.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.145+0000 7fe829a24740 1 -- 172.21.15.31:0/3305837460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 msgr2=0x55f27c9577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.145+0000 7fe829a24740 1 --2- 172.21.15.31:0/3305837460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 0x55f27c9577a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe80c007590 tx=0x7fe80c0a8910 comp rx=0 tx=0).stop 2024-04-25T16:10:31.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.145+0000 7fe829a24740 1 -- 172.21.15.31:0/3305837460 shutdown_connections 2024-04-25T16:10:31.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.145+0000 7fe829a24740 1 --2- 172.21.15.31:0/3305837460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c5a5410 0x55f27c9577a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.145+0000 7fe829a24740 1 --2- 172.21.15.31:0/3305837460 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c584730 0x55f27c940bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.145+0000 7fe829a24740 1 --2- 172.21.15.31:0/3305837460 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c57c550 0x55f27c942050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.145+0000 7fe829a24740 1 -- 172.21.15.31:0/3305837460 >> 172.21.15.31:0/3305837460 conn(0x55f27c599560 msgr2=0x55f27c9404c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:31.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.145+0000 7fe829a24740 1 -- 172.21.15.31:0/3305837460 shutdown_connections 2024-04-25T16:10:31.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.145+0000 7fe829a24740 1 -- 172.21.15.31:0/3305837460 wait complete. 2024-04-25T16:10:31.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.146+0000 7fe829a24740 1 -- start start 2024-04-25T16:10:31.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.146+0000 7fe829a24740 1 Processor -- start 2024-04-25T16:10:31.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.147+0000 7fe829a24740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c956b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.147+0000 7fe829a24740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c584730 0x55f27c957050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.147+0000 7fe829a24740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c5a5410 0x55f27c9c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.147+0000 7fe829a24740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe80800f9c0 con 0x55f27c584730 2024-04-25T16:10:31.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.147+0000 7fe829a24740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe80800f2f0 con 0x55f27c5a5410 2024-04-25T16:10:31.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.147+0000 7fe829a24740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe80800c530 con 0x55f27c57c550 2024-04-25T16:10:31.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.148+0000 7fe821033640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c5a5410 0x55f27c9c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.148+0000 7fe821033640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c5a5410 0x55f27c9c41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58306/0 (socket says 172.21.15.31:58306) 2024-04-25T16:10:31.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.148+0000 7fe821033640 1 -- 172.21.15.31:0/643301840 learned_addr learned my addr 172.21.15.31:0/643301840 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:31.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.148+0000 7fe820832640 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c956b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.148+0000 7fe829a24740 10 client.0 _ll_register_callbacks cb 0x7fe810017620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:31.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.149+0000 7fe820832640 1 -- 172.21.15.31:0/643301840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c5a5410 msgr2=0x55f27c9c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:31.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.149+0000 7fe820832640 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c5a5410 0x55f27c9c41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.149+0000 7fe820832640 1 -- 172.21.15.31:0/643301840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c584730 msgr2=0x55f27c957050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:31.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.149+0000 7fe820832640 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c584730 0x55f27c957050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.149+0000 7fe820832640 1 -- 172.21.15.31:0/643301840 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f27c9578a0 con 0x55f27c57c550 2024-04-25T16:10:31.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.149+0000 7fe820832640 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c956b10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe81002adb0 tx=0x7fe8100052d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=284be115fa1dfefe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:31.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.152+0000 7fe7fbfff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:10:31.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.153+0000 7fe7fbfff640 1 -- 172.21.15.31:0/643301840 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe810025040 con 0x55f27c57c550 2024-04-25T16:10:31.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.153+0000 7fe829a24740 1 -- 172.21.15.31:0/643301840 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe80c0b0f30 con 0x55f27c57c550 2024-04-25T16:10:31.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.153+0000 7fe7fbfff640 1 -- 172.21.15.31:0/643301840 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe810015890 con 0x55f27c57c550 2024-04-25T16:10:31.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.156+0000 7fe7fbfff640 1 -- 172.21.15.31:0/643301840 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe810015ab0 con 0x55f27c57c550 2024-04-25T16:10:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.159+0000 7fe829a24740 10 client.27133 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:31.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.159+0000 7fe829a24740 1 -- 172.21.15.31:0/643301840 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f27c9c5780 con 0x55f27c57c550 2024-04-25T16:10:31.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.160+0000 7fe7fbfff640 1 -- 172.21.15.31:0/643301840 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe80c0b0f30 con 0x55f27c57c550 2024-04-25T16:10:31.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.163+0000 7fe7fbfff640 1 -- 172.21.15.31:0/643301840 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe810032020 con 0x55f27c57c550 2024-04-25T16:10:31.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.164+0000 7fe829a24740 10 client.27133 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:31.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.164+0000 7fe829a24740 10 client.27133 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:31.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.164+0000 7fe829a24740 20 client.27133 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:31.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.164+0000 7fe829a24740 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f27c539630 0x55f27c9c8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.164+0000 7fe829a24740 4 client.27133 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:10:31.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.164+0000 7fe829a24740 1 -- 172.21.15.31:0/643301840 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f27c8b3580 con 0x55f27c539630 2024-04-25T16:10:31.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.167+0000 7fe81bfff640 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f27c539630 0x55f27c9c8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.176+0000 7fe81bfff640 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f27c539630 0x55f27c9c8170 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c78ea8912f824a19 server_cookie=fb107f2d83dcdf51 in_seq=0 out_seq=0 2024-04-25T16:10:31.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.176+0000 7fe7fbfff640 10 client.27133 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:31.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.177+0000 7fe7fbfff640 1 -- 172.21.15.31:0/643301840 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f27c8b3580 con 0x55f27c539630 2024-04-25T16:10:31.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.177+0000 7fe7fbfff640 10 client.27133 handle_command_reply: tid=0 2024-04-25T16:10:31.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.180+0000 7fe829a24740 1 client.27133 shutdown 2024-04-25T16:10:31.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.181+0000 7fe829a24740 1 -- 172.21.15.31:0/643301840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 msgr2=0x55f27c956b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.182+0000 7fe829a24740 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c956b10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe81002adb0 tx=0x7fe8100052d0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.186+0000 7fe829a24740 1 -- 172.21.15.31:0/643301840 shutdown_connections 2024-04-25T16:10:31.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.186+0000 7fe829a24740 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f27c539630 0x55f27c9c8170 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.186+0000 7fe829a24740 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f27c5a5410 0x55f27c9c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.186+0000 7fe829a24740 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f27c584730 0x55f27c957050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.186+0000 7fe829a24740 1 --2- 172.21.15.31:0/643301840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f27c57c550 0x55f27c956b10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.188 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.186+0000 7fe829a24740 1 -- 172.21.15.31:0/643301840 >> 172.21.15.31:0/643301840 conn(0x55f27c599560 msgr2=0x55f27c940ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:31.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.194+0000 7fe7fbfff640 0 client.27133 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:31.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.194+0000 7fe829a24740 1 -- 172.21.15.31:0/643301840 shutdown_connections 2024-04-25T16:10:31.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.194+0000 7fe829a24740 1 -- 172.21.15.31:0/643301840 wait complete. 2024-04-25T16:10:31.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.197+0000 7fe829a24740 20 client.27133 trim_cache size 0 max 16384 2024-04-25T16:10:31.202 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:10:31.202 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:10:31.202 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "965e1a2d-0a42-4aeb-a84d-1a5e532b5995", 2024-04-25T16:10:31.202 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:10:31.202 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.218+0000 7fe822a96640 1 -- 172.21.15.31:0/2257632423 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8040afc20 msgr2=0x7fe8040b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.218+0000 7fe822a96640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8040afc20 0x7fe8040b2130 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x55f27c947670 tx=0x7fe81000d040 comp rx=0 tx=0).stop 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.218+0000 7fe822a96640 1 -- 172.21.15.31:0/2257632423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe81c110460 msgr2=0x7fe81c1b04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.218+0000 7fe822a96640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe81c110460 0x7fe81c1b04b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe80c00c970 tx=0x7fe80c00ce30 comp rx=0 tx=0).stop 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.218+0000 7fe822a96640 1 -- 172.21.15.31:0/2257632423 shutdown_connections 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.219+0000 7fe822a96640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8040afc20 0x7fe8040b2130 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.219+0000 7fe822a96640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe81c110460 0x7fe81c1b04b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.219+0000 7fe822a96640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe81c10b760 0x7fe81c1a9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.219+0000 7fe822a96640 1 --2- 172.21.15.31:0/2257632423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe81c10a520 0x7fe81c1a8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.219+0000 7fe822a96640 1 -- 172.21.15.31:0/2257632423 >> 172.21.15.31:0/2257632423 conn(0x7fe81c070940 msgr2=0x7fe81c10c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:31.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.222+0000 7fe822a96640 1 -- 172.21.15.31:0/2257632423 shutdown_connections 2024-04-25T16:10:31.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.222+0000 7fe822a96640 1 -- 172.21.15.31:0/2257632423 wait complete. 2024-04-25T16:10:31.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:10:31.304 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:10:31.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:31 smithi063 ceph-mon[32413]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 762 MiB/s wr, 1.67k op/s 2024-04-25T16:10:31.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:31 smithi186 ceph-mon[33258]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 762 MiB/s wr, 1.67k op/s 2024-04-25T16:10:31.429 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:31 smithi031 ceph-mon[29140]: pgmap v627: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 762 MiB/s wr, 1.67k op/s 2024-04-25T16:10:31.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.691+0000 7fb012746640 1 -- 172.21.15.31:0/285000977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00c10a340 msgr2=0x7fb00c10ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.691+0000 7fb012746640 1 --2- 172.21.15.31:0/285000977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00c10a340 0x7fb00c10ac50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7faffc009ca0 tx=0x7faffc02a8c0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.695+0000 7fb012746640 1 -- 172.21.15.31:0/285000977 shutdown_connections 2024-04-25T16:10:31.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.695+0000 7fb012746640 1 --2- 172.21.15.31:0/285000977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb00c10fe90 0x7fb00c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.695+0000 7fb012746640 1 --2- 172.21.15.31:0/285000977 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb00c10b190 0x7fb00c10b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.695+0000 7fb012746640 1 --2- 172.21.15.31:0/285000977 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00c10a340 0x7fb00c10ac50 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.697 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.695+0000 7fb012746640 1 -- 172.21.15.31:0/285000977 >> 172.21.15.31:0/285000977 conn(0x7fb00c070320 msgr2=0x7fb00c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:31.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.703+0000 7fb012746640 1 -- 172.21.15.31:0/285000977 shutdown_connections 2024-04-25T16:10:31.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.704+0000 7fb012746640 1 -- 172.21.15.31:0/285000977 wait complete. 2024-04-25T16:10:31.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.705+0000 7fb012746640 1 Processor -- start 2024-04-25T16:10:31.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.705+0000 7fb012746640 1 -- start start 2024-04-25T16:10:31.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb012746640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00c10a340 0x7fb00c117110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb012746640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb00c10b190 0x7fb00c117650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb012746640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb00c10fe90 0x7fb00c1185b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb012746640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c11e2c0 con 0x7fb00c10fe90 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb012746640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c11e140 con 0x7fb00c10a340 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb012746640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb00c11e440 con 0x7fb00c10b190 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb010f43640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb00c10b190 0x7fb00c117650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb011744640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00c10a340 0x7fb00c117110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb010f43640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb00c10b190 0x7fb00c117650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54234/0 (socket says 172.21.15.31:54234) 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb011744640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00c10a340 0x7fb00c117110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:58326/0 (socket says 172.21.15.31:58326) 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb010f43640 1 -- 172.21.15.31:0/2783952569 learned_addr learned my addr 172.21.15.31:0/2783952569 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb010f43640 1 -- 172.21.15.31:0/2783952569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00c10a340 msgr2=0x7fb00c117110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb010f43640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00c10a340 0x7fb00c117110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb010f43640 1 -- 172.21.15.31:0/2783952569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb00c10fe90 msgr2=0x7fb00c1185b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.706+0000 7fb010f43640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb00c10fe90 0x7fb00c1185b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.707+0000 7fb010f43640 1 -- 172.21.15.31:0/2783952569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb00c118c20 con 0x7fb00c10b190 2024-04-25T16:10:31.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.707+0000 7fb010f43640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb00c10b190 0x7fb00c117650 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb00000b4a0 tx=0x7fb00000b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:31.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.713+0000 7fb00a7fc640 1 -- 172.21.15.31:0/2783952569 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000014070 con 0x7fb00c10b190 2024-04-25T16:10:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.713+0000 7fb00a7fc640 1 -- 172.21.15.31:0/2783952569 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb000010ac0 con 0x7fb00c10b190 2024-04-25T16:10:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.713+0000 7fb00a7fc640 1 -- 172.21.15.31:0/2783952569 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb000017c30 con 0x7fb00c10b190 2024-04-25T16:10:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.715+0000 7fb012746640 1 -- 172.21.15.31:0/2783952569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb00c118f10 con 0x7fb00c10b190 2024-04-25T16:10:31.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.715+0000 7fb012746640 1 -- 172.21.15.31:0/2783952569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb00c119490 con 0x7fb00c10b190 2024-04-25T16:10:31.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.719+0000 7fb01a85b740 1 Processor -- start 2024-04-25T16:10:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.720+0000 7fb00a7fc640 1 -- 172.21.15.31:0/2783952569 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb000017070 con 0x7fb00c10b190 2024-04-25T16:10:31.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.720+0000 7fb01a85b740 1 -- start start 2024-04-25T16:10:31.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.722+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559248161390 0x559248502c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.722+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x559248503200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.722+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x559248503e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.722+0000 7fb01a85b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x559248467d80 con 0x5592481384d0 2024-04-25T16:10:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.722+0000 7fb01a85b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5592484218b0 con 0x559248161390 2024-04-25T16:10:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.722+0000 7fb01a85b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559248463d40 con 0x5592481406b0 2024-04-25T16:10:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.723+0000 7fb011744640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x559248503200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.723+0000 7fb011744640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x559248503200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35750/0 (socket says 172.21.15.31:35750) 2024-04-25T16:10:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.723+0000 7fb011744640 1 -- 172.21.15.31:0/1105517098 learned_addr learned my addr 172.21.15.31:0/1105517098 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.723+0000 7fb00a7fc640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faff40afcf0 0x7faff40b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.725 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.723+0000 7fb00a7fc640 1 -- 172.21.15.31:0/2783952569 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fb0000df1d0 con 0x7fb00c10b190 2024-04-25T16:10:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.724+0000 7fb011744640 1 -- 172.21.15.31:0/1105517098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 msgr2=0x559248503e90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.724+0000 7fb011744640 1 --2- 172.21.15.31:0/1105517098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x559248503e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.724+0000 7fb011744640 1 -- 172.21.15.31:0/1105517098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559248161390 msgr2=0x559248502c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.724+0000 7fb011744640 1 --2- 172.21.15.31:0/1105517098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559248161390 0x559248502c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.724+0000 7fb011744640 1 -- 172.21.15.31:0/1105517098 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592483f2a40 con 0x5592481384d0 2024-04-25T16:10:31.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.726+0000 7fb011f45640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faff40afcf0 0x7faff40b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.726+0000 7fb011744640 1 --2- 172.21.15.31:0/1105517098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x559248503200 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7faffc009ca0 tx=0x7faffc0098b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a440feb3bd22300f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.732+0000 7fb012746640 1 -- 172.21.15.31:0/1105517098 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc031710 con 0x5592481384d0 2024-04-25T16:10:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.733+0000 7fb012746640 1 -- 172.21.15.31:0/1105517098 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faffc036030 con 0x5592481384d0 2024-04-25T16:10:31.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.733+0000 7fb012746640 1 -- 172.21.15.31:0/1105517098 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc004820 con 0x5592481384d0 2024-04-25T16:10:31.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.733+0000 7fb011f45640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faff40afcf0 0x7faff40b2200 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x559248503cd0 tx=0x7fb004005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:31.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.735+0000 7fb01a85b740 1 -- 172.21.15.31:0/1105517098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 msgr2=0x559248503200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.735+0000 7fb01a85b740 1 --2- 172.21.15.31:0/1105517098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x559248503200 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7faffc009ca0 tx=0x7faffc0098b0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.735+0000 7fb01a85b740 1 -- 172.21.15.31:0/1105517098 shutdown_connections 2024-04-25T16:10:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.735+0000 7fb01a85b740 1 --2- 172.21.15.31:0/1105517098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x559248503e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.735+0000 7fb01a85b740 1 --2- 172.21.15.31:0/1105517098 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x559248503200 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.735+0000 7fb01a85b740 1 --2- 172.21.15.31:0/1105517098 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559248161390 0x559248502c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.735+0000 7fb01a85b740 1 -- 172.21.15.31:0/1105517098 >> 172.21.15.31:0/1105517098 conn(0x5592481554e0 msgr2=0x5592484fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:31.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.736+0000 7fb01a85b740 1 -- 172.21.15.31:0/1105517098 shutdown_connections 2024-04-25T16:10:31.739 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.738+0000 7fb01a85b740 1 -- 172.21.15.31:0/1105517098 wait complete. 2024-04-25T16:10:31.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.738+0000 7fb01a85b740 1 -- start start 2024-04-25T16:10:31.740 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.738+0000 7fb01a85b740 1 Processor -- start 2024-04-25T16:10:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.742+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481384d0 0x55924850ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.742+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x55924850cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.742+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 0x559248513ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.742+0000 7fb01a85b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5592485047c0 con 0x559248161390 2024-04-25T16:10:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.742+0000 7fb01a85b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5592483eed10 con 0x5592481384d0 2024-04-25T16:10:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.742+0000 7fb01a85b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559248504940 con 0x5592481406b0 2024-04-25T16:10:31.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.742+0000 7fb01a85b740 10 client.0 _ll_register_callbacks cb 0x55924841b210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.745+0000 7fb010f43640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x55924850cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.745+0000 7fb010f43640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x55924850cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54250/0 (socket says 172.21.15.31:54250) 2024-04-25T16:10:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.745+0000 7fb010f43640 1 -- 172.21.15.31:0/4049522747 learned_addr learned my addr 172.21.15.31:0/4049522747 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.746+0000 7fb011744640 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481384d0 0x55924850ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.746+0000 7fb011f45640 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 0x559248513ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.746+0000 7fb011744640 1 -- 172.21.15.31:0/4049522747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 msgr2=0x55924850cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.746+0000 7fb011744640 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x55924850cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.746+0000 7fb011744640 1 -- 172.21.15.31:0/4049522747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 msgr2=0x559248513ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.746+0000 7fb011744640 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 0x559248513ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.746+0000 7fb011744640 1 -- 172.21.15.31:0/4049522747 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592482437d0 con 0x5592481384d0 2024-04-25T16:10:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.749+0000 7fb011744640 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481384d0 0x55924850ca20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faffc02aeb0 tx=0x7faffc0095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f01841c7d0b0243a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.749+0000 7faff0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:10:31.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.749+0000 7faff0ff9640 1 -- 172.21.15.31:0/4049522747 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc04a040 con 0x5592481384d0 2024-04-25T16:10:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.749+0000 7fb01a85b740 1 -- 172.21.15.31:0/4049522747 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faffc036030 con 0x5592481384d0 2024-04-25T16:10:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.750+0000 7faff0ff9640 1 -- 172.21.15.31:0/4049522747 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faffc003b70 con 0x5592481384d0 2024-04-25T16:10:31.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.750+0000 7faff0ff9640 1 -- 172.21.15.31:0/4049522747 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faffc003d70 con 0x5592481384d0 2024-04-25T16:10:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.752+0000 7fb01a85b740 10 client.27155 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:31.754 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.752+0000 7fb01a85b740 1 -- 172.21.15.31:0/4049522747 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5592481793f0 con 0x5592481384d0 2024-04-25T16:10:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.754+0000 7faff0ff9640 1 -- 172.21.15.31:0/4049522747 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faffc036030 con 0x5592481384d0 2024-04-25T16:10:31.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.755+0000 7faff0ff9640 1 -- 172.21.15.31:0/4049522747 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7faffc04d030 con 0x5592481384d0 2024-04-25T16:10:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.759+0000 7fb01a85b740 10 client.27155 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.759+0000 7fb01a85b740 10 client.27155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.759+0000 7fb01a85b740 20 client.27155 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.759+0000 7fb01a85b740 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55924811ff60 0x559248519f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.759+0000 7fb01a85b740 4 client.27155 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:31.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.759+0000 7fb01a85b740 1 -- 172.21.15.31:0/4049522747 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559248475990 con 0x55924811ff60 2024-04-25T16:10:31.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.760+0000 7fb010f43640 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55924811ff60 0x559248519f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.772+0000 7fb010f43640 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55924811ff60 0x559248519f80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f45affc2b3328427 server_cookie=f3d354b41f726445 in_seq=0 out_seq=0 2024-04-25T16:10:31.774 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.773+0000 7faff0ff9640 10 client.27155 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:31.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7faff0ff9640 1 -- 172.21.15.31:0/4049522747 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559248475990 con 0x55924811ff60 2024-04-25T16:10:31.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7faff0ff9640 10 client.27155 handle_command_reply: tid=0 2024-04-25T16:10:31.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7fb01a85b740 1 client.27155 shutdown 2024-04-25T16:10:31.777 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7fb01a85b740 1 -- 172.21.15.31:0/4049522747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481384d0 msgr2=0x55924850ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7fb01a85b740 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481384d0 0x55924850ca20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7faffc02aeb0 tx=0x7faffc0095c0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7fb01a85b740 1 -- 172.21.15.31:0/4049522747 shutdown_connections 2024-04-25T16:10:31.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7fb01a85b740 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55924811ff60 0x559248519f80 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7fb01a85b740 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 0x559248513ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7fb01a85b740 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x55924850cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7fb01a85b740 1 --2- 172.21.15.31:0/4049522747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481384d0 0x55924850ca20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.776+0000 7fb01a85b740 1 -- 172.21.15.31:0/4049522747 >> 172.21.15.31:0/4049522747 conn(0x5592481554e0 msgr2=0x559248501690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:31.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.778+0000 7faff0ff9640 0 client.27155 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.783+0000 7fb01a85b740 1 -- 172.21.15.31:0/4049522747 shutdown_connections 2024-04-25T16:10:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.783+0000 7fb01a85b740 1 -- 172.21.15.31:0/4049522747 wait complete. 2024-04-25T16:10:31.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.784+0000 7fb01a85b740 20 client.27155 trim_cache size 0 max 16384 2024-04-25T16:10:31.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.827+0000 7fb01a85b740 1 Processor -- start 2024-04-25T16:10:31.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.827+0000 7fb01a85b740 1 -- start start 2024-04-25T16:10:31.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.828+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481384d0 0x5592484fdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.828+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481406b0 0x559248507f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.828+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 0x559248501c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.828+0000 7fb01a85b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faff800e550 con 0x559248161390 2024-04-25T16:10:31.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.829+0000 7fb01a85b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faff8010090 con 0x5592481406b0 2024-04-25T16:10:31.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.829+0000 7fb01a85b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faff800ec20 con 0x5592481384d0 2024-04-25T16:10:31.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.830+0000 7fb010f43640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481406b0 0x559248507f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.830+0000 7fb011744640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481384d0 0x5592484fdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.830+0000 7fb011744640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481384d0 0x5592484fdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54266/0 (socket says 172.21.15.31:54266) 2024-04-25T16:10:31.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.830+0000 7fb011744640 1 -- 172.21.15.31:0/2014787306 learned_addr learned my addr 172.21.15.31:0/2014787306 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:31.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.831+0000 7fb011f45640 1 --2- 172.21.15.31:0/2014787306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 0x559248501c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.831+0000 7fb011744640 1 -- 172.21.15.31:0/2014787306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481406b0 msgr2=0x559248507f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.831+0000 7fb011744640 1 --2- 172.21.15.31:0/2014787306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481406b0 0x559248507f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.831+0000 7fb011744640 1 -- 172.21.15.31:0/2014787306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 msgr2=0x559248501c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.831+0000 7fb011744640 1 --2- 172.21.15.31:0/2014787306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 0x559248501c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.831+0000 7fb011744640 1 -- 172.21.15.31:0/2014787306 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592481390f0 con 0x5592481384d0 2024-04-25T16:10:31.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.832+0000 7fb011744640 1 --2- 172.21.15.31:0/2014787306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481384d0 0x5592484fdba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7faffc030ac0 tx=0x7faffc0098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db32edb96c621574 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:31.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.842+0000 7faff2ffd640 1 -- 172.21.15.31:0/2014787306 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc003b70 con 0x5592481384d0 2024-04-25T16:10:31.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.842+0000 7faff2ffd640 1 -- 172.21.15.31:0/2014787306 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faffc036070 con 0x5592481384d0 2024-04-25T16:10:31.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.842+0000 7faff2ffd640 1 -- 172.21.15.31:0/2014787306 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc004330 con 0x5592481384d0 2024-04-25T16:10:31.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.843+0000 7fb01a85b740 1 -- 172.21.15.31:0/2014787306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481384d0 msgr2=0x5592484fdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.843+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2014787306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481384d0 0x5592484fdba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7faffc030ac0 tx=0x7faffc0098b0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.846+0000 7fb01a85b740 1 -- 172.21.15.31:0/2014787306 shutdown_connections 2024-04-25T16:10:31.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.846+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2014787306 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x559248161390 0x559248501c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.846+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2014787306 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5592481406b0 0x559248507f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.846+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2014787306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481384d0 0x5592484fdba0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.846+0000 7fb01a85b740 1 -- 172.21.15.31:0/2014787306 >> 172.21.15.31:0/2014787306 conn(0x5592481554e0 msgr2=0x559248501580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:31.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.854+0000 7fb01a85b740 1 -- 172.21.15.31:0/2014787306 shutdown_connections 2024-04-25T16:10:31.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.858+0000 7fb01a85b740 1 -- 172.21.15.31:0/2014787306 wait complete. 2024-04-25T16:10:31.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.858+0000 7fb01a85b740 1 -- start start 2024-04-25T16:10:31.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.858+0000 7fb01a85b740 1 Processor -- start 2024-04-25T16:10:31.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.872+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x55924850e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.872+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x55924850ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.872+0000 7fb01a85b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559248161390 0x559248511bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.873+0000 7fb01a85b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faff800f9c0 con 0x5592481384d0 2024-04-25T16:10:31.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.873+0000 7fb01a85b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faff800f2f0 con 0x559248161390 2024-04-25T16:10:31.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.873+0000 7fb01a85b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faff800c530 con 0x5592481406b0 2024-04-25T16:10:31.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.873+0000 7fb01a85b740 10 client.0 _ll_register_callbacks cb 0x7faffc009800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.874+0000 7fb011744640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x55924850e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.874+0000 7fb011744640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x55924850e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:35782/0 (socket says 172.21.15.31:35782) 2024-04-25T16:10:31.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.875+0000 7fb011744640 1 -- 172.21.15.31:0/2036973682 learned_addr learned my addr 172.21.15.31:0/2036973682 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:31.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.875+0000 7fb010f43640 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x55924850ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.876+0000 7fb011744640 1 -- 172.21.15.31:0/2036973682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 msgr2=0x55924850ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:31.877 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.876+0000 7fb011744640 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x55924850ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.876+0000 7fb011744640 1 -- 172.21.15.31:0/2036973682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559248161390 msgr2=0x559248511bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:10:31.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.876+0000 7fb011744640 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559248161390 0x559248511bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.876+0000 7fb011744640 1 -- 172.21.15.31:0/2036973682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592484284b0 con 0x5592481384d0 2024-04-25T16:10:31.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.877+0000 7fb011744640 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x55924850e7f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7faffc009ca0 tx=0x7faffc0308e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36b3d6e0d6f1fb73 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:31.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.877+0000 7faff0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:10:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.877+0000 7faff0ff9640 1 -- 172.21.15.31:0/2036973682 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faffc02f310 con 0x5592481384d0 2024-04-25T16:10:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.878+0000 7faff0ff9640 1 -- 172.21.15.31:0/2036973682 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faffc030b00 con 0x5592481384d0 2024-04-25T16:10:31.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.878+0000 7fb01a85b740 1 -- 172.21.15.31:0/2036973682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faffc036070 con 0x5592481384d0 2024-04-25T16:10:31.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.880+0000 7faff0ff9640 1 -- 172.21.15.31:0/2036973682 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faffc073050 con 0x5592481384d0 2024-04-25T16:10:31.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.880+0000 7fb01a85b740 10 client.17472 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:31.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.880+0000 7fb01a85b740 1 -- 172.21.15.31:0/2036973682 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559248513230 con 0x5592481384d0 2024-04-25T16:10:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.881+0000 7faff0ff9640 1 -- 172.21.15.31:0/2036973682 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faffc055070 con 0x5592481384d0 2024-04-25T16:10:31.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.881+0000 7faff0ff9640 1 -- 172.21.15.31:0/2036973682 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7faffc063040 con 0x5592481384d0 2024-04-25T16:10:31.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.884+0000 7fb01a85b740 10 client.17472 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.885+0000 7fb01a85b740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.885+0000 7fb01a85b740 20 client.17472 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.885+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5592480f53a0 0x559248584370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.885+0000 7fb01a85b740 4 client.17472 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:10:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.885+0000 7fb01a85b740 1 -- 172.21.15.31:0/2036973682 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7faffc036070 con 0x5592480f53a0 2024-04-25T16:10:31.887 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.885+0000 7fb010f43640 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5592480f53a0 0x559248584370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.886+0000 7fb010f43640 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5592480f53a0 0x559248584370 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=871c8f80abb292c7 server_cookie=224a1a2bd4d0798b in_seq=0 out_seq=0 2024-04-25T16:10:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.886+0000 7faff0ff9640 10 client.17472 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.887+0000 7faff0ff9640 1 -- 172.21.15.31:0/2036973682 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x7faffc036070 con 0x5592480f53a0 2024-04-25T16:10:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.887+0000 7faff0ff9640 10 client.17472 handle_command_reply: tid=0 2024-04-25T16:10:31.888 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.887+0000 7fb01a85b740 1 client.17472 shutdown 2024-04-25T16:10:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.893+0000 7fb01a85b740 1 -- 172.21.15.31:0/2036973682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 msgr2=0x55924850e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.893+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x55924850e7f0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7faffc009ca0 tx=0x7faffc0308e0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.898+0000 7fb01a85b740 1 -- 172.21.15.31:0/2036973682 shutdown_connections 2024-04-25T16:10:31.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.898+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5592480f53a0 0x559248584370 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.898+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x559248161390 0x559248511bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.898+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5592481406b0 0x55924850ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.898+0000 7fb01a85b740 1 --2- 172.21.15.31:0/2036973682 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5592481384d0 0x55924850e7f0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.898+0000 7fb01a85b740 1 -- 172.21.15.31:0/2036973682 >> 172.21.15.31:0/2036973682 conn(0x5592481554e0 msgr2=0x5592484fcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.898+0000 7faff0ff9640 0 client.17472 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.899+0000 7fb01a85b740 1 -- 172.21.15.31:0/2036973682 shutdown_connections 2024-04-25T16:10:31.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.899+0000 7fb01a85b740 1 -- 172.21.15.31:0/2036973682 wait complete. 2024-04-25T16:10:31.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.900+0000 7fb01a85b740 20 client.17472 trim_cache size 0 max 16384 2024-04-25T16:10:31.903 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:10:31.903 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (11 inodes in the stack)", 2024-04-25T16:10:31.903 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:10:31.903 INFO:teuthology.orchestra.run.smithi031.stdout: "965e1a2d-0a42-4aeb-a84d-1a5e532b5995": { 2024-04-25T16:10:31.903 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:10:31.903 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "965e1a2d-0a42-4aeb-a84d-1a5e532b5995", 2024-04-25T16:10:31.903 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:10:31.903 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:10:31.904 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:10:31.904 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:10:31.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 -- 172.21.15.31:0/2783952569 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faff40afcf0 msgr2=0x7faff40b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faff40afcf0 0x7faff40b2200 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x559248503cd0 tx=0x7fb004005bf0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 -- 172.21.15.31:0/2783952569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb00c10b190 msgr2=0x7fb00c117650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:31.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb00c10b190 0x7fb00c117650 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb00000b4a0 tx=0x7fb00000b960 comp rx=0 tx=0).stop 2024-04-25T16:10:31.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 -- 172.21.15.31:0/2783952569 shutdown_connections 2024-04-25T16:10:31.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faff40afcf0 0x7faff40b2200 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fb00c10fe90 0x7fb00c1185b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb00c10b190 0x7fb00c117650 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 --2- 172.21.15.31:0/2783952569 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb00c10a340 0x7fb00c117110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:31.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.902+0000 7faff3fff640 1 -- 172.21.15.31:0/2783952569 >> 172.21.15.31:0/2783952569 conn(0x7fb00c070320 msgr2=0x7fb00c10be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:31.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.903+0000 7faff3fff640 1 -- 172.21.15.31:0/2783952569 shutdown_connections 2024-04-25T16:10:31.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:10:31.905+0000 7faff3fff640 1 -- 172.21.15.31:0/2783952569 wait complete. 2024-04-25T16:10:31.956 INFO:tasks.cephfs.filesystem:scrub status for tag:965e1a2d-0a42-4aeb-a84d-1a5e532b5995 - {'path': '/', 'tag': '965e1a2d-0a42-4aeb-a84d-1a5e532b5995', 'options': 'recursive,force'} 2024-04-25T16:10:32.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:32 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:10:32.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:32 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:10:32.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:32 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:10:32.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:32 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:10:32.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:32 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:10:32.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:32 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:10:32.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:32 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:10:32.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:32 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:10:32.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:32 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:10:33.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:33 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:10:33.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:33 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:10:33.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:33 smithi031 ceph-mon[29140]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 781 MiB/s wr, 1.77k op/s 2024-04-25T16:10:33.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:33 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:10:33.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:33 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:10:33.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:33 smithi063 ceph-mon[32413]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 781 MiB/s wr, 1.77k op/s 2024-04-25T16:10:33.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:33 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:10:33.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:33 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:10:33.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:33 smithi186 ceph-mon[33258]: pgmap v628: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 781 MiB/s wr, 1.77k op/s 2024-04-25T16:10:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.769+0000 7f472e222640 1 -- 172.21.15.31:0/1260431817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f472810b450 msgr2=0x7f4728109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.769+0000 7f472e222640 1 --2- 172.21.15.31:0/1260431817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f472810b450 0x7f4728109ba0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f472000b2e0 tx=0x7f472002b7f0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.770+0000 7f472e222640 1 -- 172.21.15.31:0/1260431817 shutdown_connections 2024-04-25T16:10:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.770+0000 7f472e222640 1 --2- 172.21.15.31:0/1260431817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4728078df0 0x7f472807b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.770+0000 7f472e222640 1 --2- 172.21.15.31:0/1260431817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f472810a0e0 0x7f472810a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.770+0000 7f472e222640 1 --2- 172.21.15.31:0/1260431817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f472810b450 0x7f4728109ba0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.770+0000 7f472e222640 1 -- 172.21.15.31:0/1260431817 >> 172.21.15.31:0/1260431817 conn(0x7f4728070330 msgr2=0x7f47280727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.775+0000 7f472e222640 1 -- 172.21.15.31:0/1260431817 shutdown_connections 2024-04-25T16:10:33.780 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.779+0000 7f472e222640 1 -- 172.21.15.31:0/1260431817 wait complete. 2024-04-25T16:10:33.781 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.780+0000 7f472e222640 1 Processor -- start 2024-04-25T16:10:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.780+0000 7f472e222640 1 -- start start 2024-04-25T16:10:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.781+0000 7f472e222640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4728078df0 0x7f47281b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.781+0000 7f472e222640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f472810a0e0 0x7f47281b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.781+0000 7f472e222640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f472810b450 0x7f47281bad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.781+0000 7f472e222640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f472807ea30 con 0x7f472810b450 2024-04-25T16:10:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.781+0000 7f472e222640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f472807e8b0 con 0x7f472810a0e0 2024-04-25T16:10:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.781+0000 7f472e222640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f472807ebb0 con 0x7f4728078df0 2024-04-25T16:10:33.785 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.782+0000 7f472ca1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f472810a0e0 0x7f47281b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.782+0000 7f472ca1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f472810a0e0 0x7f47281b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:41002/0 (socket says 172.21.15.31:41002) 2024-04-25T16:10:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.782+0000 7f472ca1f640 1 -- 172.21.15.31:0/1836262434 learned_addr learned my addr 172.21.15.31:0/1836262434 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.782+0000 7f472d220640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4728078df0 0x7f47281b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.782+0000 7f472ca1f640 1 -- 172.21.15.31:0/1836262434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4728078df0 msgr2=0x7f47281b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.782+0000 7f472ca1f640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4728078df0 0x7f47281b3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.783+0000 7f472ca1f640 1 -- 172.21.15.31:0/1836262434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f472810b450 msgr2=0x7f47281bad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.783+0000 7f472ca1f640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f472810b450 0x7f47281bad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.786 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.783+0000 7f472ca1f640 1 -- 172.21.15.31:0/1836262434 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47281bb3c0 con 0x7f472810a0e0 2024-04-25T16:10:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.783+0000 7f472ca1f640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f472810a0e0 0x7f47281b3dc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4724009e60 tx=0x7f472400ed90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.783+0000 7f471e7fc640 1 -- 172.21.15.31:0/1836262434 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f472400c8b0 con 0x7f472810a0e0 2024-04-25T16:10:33.787 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.784+0000 7f472e222640 1 -- 172.21.15.31:0/1836262434 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4728079260 con 0x7f472810a0e0 2024-04-25T16:10:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.784+0000 7f472e222640 1 -- 172.21.15.31:0/1836262434 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47280797b0 con 0x7f472810a0e0 2024-04-25T16:10:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.784+0000 7f471e7fc640 1 -- 172.21.15.31:0/1836262434 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4724005020 con 0x7f472810a0e0 2024-04-25T16:10:33.788 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.784+0000 7f471e7fc640 1 -- 172.21.15.31:0/1836262434 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4724004c20 con 0x7f472810a0e0 2024-04-25T16:10:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.788+0000 7f471e7fc640 1 -- 172.21.15.31:0/1836262434 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f472401f440 con 0x7f472810a0e0 2024-04-25T16:10:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.790+0000 7f471e7fc640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f47140afbe0 0x7f47140b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.791+0000 7f471e7fc640 1 -- 172.21.15.31:0/1836262434 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f47240e7380 con 0x7f472810a0e0 2024-04-25T16:10:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.792+0000 7f472d220640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f47140afbe0 0x7f47140b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:33.795 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.793+0000 7f472d220640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f47140afbe0 0x7f47140b20f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4720000f80 tx=0x7f4720005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:33.814 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.812+0000 7f4736409740 1 Processor -- start 2024-04-25T16:10:33.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.825+0000 7f4736409740 1 -- start start 2024-04-25T16:10:33.830 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.826+0000 7f4736409740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d939b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.826+0000 7f4736409740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 0x557a8d93a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.826+0000 7f4736409740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d940f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.826+0000 7f4736409740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557a8d89f460 con 0x557a8d5702f0 2024-04-25T16:10:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.827+0000 7f4736409740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557a8d859590 con 0x557a8d5784d0 2024-04-25T16:10:33.831 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.827+0000 7f4736409740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557a8d89b420 con 0x557a8d5991b0 2024-04-25T16:10:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.832+0000 7f472d220640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d939b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.832+0000 7f472ca1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 0x557a8d93a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.832+0000 7f472ca1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 0x557a8d93a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52810/0 (socket says 172.21.15.31:52810) 2024-04-25T16:10:33.837 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.832+0000 7f472ca1f640 1 -- 172.21.15.31:0/141011649 learned_addr learned my addr 172.21.15.31:0/141011649 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:33.838 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.835+0000 7f472da21640 1 --2- 172.21.15.31:0/141011649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d940f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.840+0000 7f472da21640 1 -- 172.21.15.31:0/141011649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 msgr2=0x557a8d939b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:33.842 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.840+0000 7f472da21640 1 --2- 172.21.15.31:0/141011649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d939b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.840+0000 7f472da21640 1 -- 172.21.15.31:0/141011649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 msgr2=0x557a8d93a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:33.843 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.840+0000 7f472da21640 1 --2- 172.21.15.31:0/141011649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 0x557a8d93a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.844 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.840+0000 7f472da21640 1 -- 172.21.15.31:0/141011649 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a8d8660d0 con 0x557a8d5784d0 2024-04-25T16:10:33.846 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.845+0000 7f472da21640 1 --2- 172.21.15.31:0/141011649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d940f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f47180096d0 tx=0x7f4718009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e136736676ce46a1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.853+0000 7f47137fe640 1 -- 172.21.15.31:0/141011649 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4718015020 con 0x557a8d5784d0 2024-04-25T16:10:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.853+0000 7f47137fe640 1 -- 172.21.15.31:0/141011649 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f471800bd70 con 0x557a8d5784d0 2024-04-25T16:10:33.857 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.853+0000 7f47137fe640 1 -- 172.21.15.31:0/141011649 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4718010380 con 0x557a8d5784d0 2024-04-25T16:10:33.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.868+0000 7f4736409740 1 -- 172.21.15.31:0/141011649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 msgr2=0x557a8d940f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:33.873 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.868+0000 7f4736409740 1 --2- 172.21.15.31:0/141011649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d940f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f47180096d0 tx=0x7f4718009b90 comp rx=0 tx=0).stop 2024-04-25T16:10:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.872+0000 7f4736409740 1 -- 172.21.15.31:0/141011649 shutdown_connections 2024-04-25T16:10:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.872+0000 7f4736409740 1 --2- 172.21.15.31:0/141011649 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d940f90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.872+0000 7f4736409740 1 --2- 172.21.15.31:0/141011649 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 0x557a8d93a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.872+0000 7f4736409740 1 --2- 172.21.15.31:0/141011649 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d939b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.872+0000 7f4736409740 1 -- 172.21.15.31:0/141011649 >> 172.21.15.31:0/141011649 conn(0x557a8d58d300 msgr2=0x557a8d9354a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:33.883 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.879+0000 7f4736409740 1 -- 172.21.15.31:0/141011649 shutdown_connections 2024-04-25T16:10:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.882+0000 7f4736409740 1 -- 172.21.15.31:0/141011649 wait complete. 2024-04-25T16:10:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.884+0000 7f4736409740 1 -- start start 2024-04-25T16:10:33.886 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.884+0000 7f4736409740 1 Processor -- start 2024-04-25T16:10:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.885+0000 7f4736409740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 0x557a8d943870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.887 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.886+0000 7f4736409740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d943db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.886+0000 7f4736409740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d94ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.886+0000 7f4736409740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557a8d9419e0 con 0x557a8d5702f0 2024-04-25T16:10:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.886+0000 7f4736409740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557a8d826b60 con 0x557a8d5784d0 2024-04-25T16:10:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.886+0000 7f4736409740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557a8d941b60 con 0x557a8d5991b0 2024-04-25T16:10:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.886+0000 7f4736409740 10 client.0 _ll_register_callbacks cb 0x557a8d84b2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.887+0000 7f472ca1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d943db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:33.888 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.887+0000 7f472ca1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d943db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:41018/0 (socket says 172.21.15.31:41018) 2024-04-25T16:10:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.887+0000 7f472ca1f640 1 -- 172.21.15.31:0/4122835134 learned_addr learned my addr 172.21.15.31:0/4122835134 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.888+0000 7f472d220640 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 0x557a8d943870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.888+0000 7f472ca1f640 1 -- 172.21.15.31:0/4122835134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 msgr2=0x557a8d94ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.888+0000 7f472ca1f640 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d94ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.888+0000 7f472ca1f640 1 -- 172.21.15.31:0/4122835134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 msgr2=0x557a8d943870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.888+0000 7f472ca1f640 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 0x557a8d943870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.888+0000 7f472ca1f640 1 -- 172.21.15.31:0/4122835134 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a8d5b1210 con 0x557a8d5784d0 2024-04-25T16:10:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.889+0000 7f472ca1f640 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d943db0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f47240b0770 tx=0x7f47240094a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d55bc73ba3a73c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.890+0000 7f4710ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:10:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.890+0000 7f4710ff9640 1 -- 172.21.15.31:0/4122835134 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240b07e0 con 0x557a8d5784d0 2024-04-25T16:10:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.890+0000 7f4736409740 1 -- 172.21.15.31:0/4122835134 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f471800bd70 con 0x557a8d5784d0 2024-04-25T16:10:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.890+0000 7f4710ff9640 1 -- 172.21.15.31:0/4122835134 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f47240054d0 con 0x557a8d5784d0 2024-04-25T16:10:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.891+0000 7f4710ff9640 1 -- 172.21.15.31:0/4122835134 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f47240b3c80 con 0x557a8d5784d0 2024-04-25T16:10:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.891+0000 7f4736409740 10 client.27170 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.891+0000 7f4736409740 1 -- 172.21.15.31:0/4122835134 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557a8d67d620 con 0x557a8d5784d0 2024-04-25T16:10:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.892+0000 7f4710ff9640 1 -- 172.21.15.31:0/4122835134 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f471800bd70 con 0x557a8d5784d0 2024-04-25T16:10:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.892+0000 7f4710ff9640 1 -- 172.21.15.31:0/4122835134 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f47240e6040 con 0x557a8d5784d0 2024-04-25T16:10:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.893+0000 7f4736409740 10 client.27170 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.893+0000 7f4736409740 10 client.27170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.893+0000 7f4736409740 20 client.27170 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.893+0000 7f4736409740 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557a8d557d80 0x557a8d962ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.893+0000 7f4736409740 4 client.27170 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:33.896 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.893+0000 7f4736409740 1 -- 172.21.15.31:0/4122835134 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a8d8ad070 con 0x557a8d557d80 2024-04-25T16:10:33.897 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.895+0000 7f472d220640 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557a8d557d80 0x557a8d962ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:33.912 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.910+0000 7f472d220640 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557a8d557d80 0x557a8d962ba0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6698057172bc58ca server_cookie=d8863ae846f56f4c in_seq=0 out_seq=0 2024-04-25T16:10:33.913 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.910+0000 7f4710ff9640 10 client.27170 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.917+0000 7f4710ff9640 1 -- 172.21.15.31:0/4122835134 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557a8d8ad070 con 0x557a8d557d80 2024-04-25T16:10:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.917+0000 7f4710ff9640 10 client.27170 handle_command_reply: tid=0 2024-04-25T16:10:33.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.918+0000 7f4736409740 1 client.27170 shutdown 2024-04-25T16:10:33.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.926+0000 7f4736409740 1 -- 172.21.15.31:0/4122835134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 msgr2=0x557a8d943db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.926+0000 7f4736409740 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d943db0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f47240b0770 tx=0x7f47240094a0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.926+0000 7f4736409740 1 -- 172.21.15.31:0/4122835134 shutdown_connections 2024-04-25T16:10:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.926+0000 7f4736409740 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557a8d557d80 0x557a8d962ba0 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.927+0000 7f4736409740 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d94ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.927+0000 7f4736409740 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5784d0 0x557a8d943db0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.927+0000 7f4736409740 1 --2- 172.21.15.31:0/4122835134 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5702f0 0x557a8d943870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:33.928 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.927+0000 7f4736409740 1 -- 172.21.15.31:0/4122835134 >> 172.21.15.31:0/4122835134 conn(0x557a8d58d300 msgr2=0x557a8d934500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.929+0000 7f4710ff9640 0 client.27170 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:33.930 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.929+0000 7f4736409740 1 -- 172.21.15.31:0/4122835134 shutdown_connections 2024-04-25T16:10:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.929+0000 7f4736409740 1 -- 172.21.15.31:0/4122835134 wait complete. 2024-04-25T16:10:33.931 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:33.930+0000 7f4736409740 20 client.27170 trim_cache size 0 max 16384 2024-04-25T16:10:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.011+0000 7f4736409740 1 Processor -- start 2024-04-25T16:10:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.012+0000 7f4736409740 1 -- start start 2024-04-25T16:10:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.012+0000 7f4736409740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 0x557a8d93cc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:34.014 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.012+0000 7f4736409740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5784d0 0x557a8d94ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.012+0000 7f4736409740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5991b0 0x557a8d93d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.013+0000 7f4736409740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f470c00e550 con 0x557a8d5991b0 2024-04-25T16:10:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.013+0000 7f4736409740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f470c010090 con 0x557a8d5702f0 2024-04-25T16:10:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.013+0000 7f4736409740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f470c00ec20 con 0x557a8d5784d0 2024-04-25T16:10:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.013+0000 7f472da21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5991b0 0x557a8d93d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.013+0000 7f472da21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5991b0 0x557a8d93d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:52818/0 (socket says 172.21.15.31:52818) 2024-04-25T16:10:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.013+0000 7f472da21640 1 -- 172.21.15.31:0/1585337747 learned_addr learned my addr 172.21.15.31:0/1585337747 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.014+0000 7f472da21640 1 -- 172.21.15.31:0/1585337747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5784d0 msgr2=0x557a8d94ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.014+0000 7f472da21640 1 --2- 172.21.15.31:0/1585337747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5784d0 0x557a8d94ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.014+0000 7f472da21640 1 -- 172.21.15.31:0/1585337747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 msgr2=0x557a8d93cc80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:10:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.014+0000 7f472da21640 1 --2- 172.21.15.31:0/1585337747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 0x557a8d93cc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.014+0000 7f472da21640 1 -- 172.21.15.31:0/1585337747 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a8d557fb0 con 0x557a8d5991b0 2024-04-25T16:10:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.015+0000 7f472da21640 1 --2- 172.21.15.31:0/1585337747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5991b0 0x557a8d93d1c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f47180095f0 tx=0x7f471800b6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a8afb232aef24c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.023+0000 7f4712ffd640 1 -- 172.21.15.31:0/1585337747 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4718015070 con 0x557a8d5991b0 2024-04-25T16:10:34.025 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.024+0000 7f4712ffd640 1 -- 172.21.15.31:0/1585337747 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f471801f030 con 0x557a8d5991b0 2024-04-25T16:10:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.025+0000 7f4712ffd640 1 -- 172.21.15.31:0/1585337747 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f471801b430 con 0x557a8d5991b0 2024-04-25T16:10:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.025+0000 7f4736409740 1 -- 172.21.15.31:0/1585337747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5991b0 msgr2=0x557a8d93d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:34.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.025+0000 7f4736409740 1 --2- 172.21.15.31:0/1585337747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5991b0 0x557a8d93d1c0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f47180095f0 tx=0x7f471800b6c0 comp rx=0 tx=0).stop 2024-04-25T16:10:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.026+0000 7f4736409740 1 -- 172.21.15.31:0/1585337747 shutdown_connections 2024-04-25T16:10:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.026+0000 7f4736409740 1 --2- 172.21.15.31:0/1585337747 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5991b0 0x557a8d93d1c0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:34.027 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.026+0000 7f4736409740 1 --2- 172.21.15.31:0/1585337747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5784d0 0x557a8d94ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:34.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.026+0000 7f4736409740 1 --2- 172.21.15.31:0/1585337747 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 0x557a8d93cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:34.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.026+0000 7f4736409740 1 -- 172.21.15.31:0/1585337747 >> 172.21.15.31:0/1585337747 conn(0x557a8d58d300 msgr2=0x557a8d93dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.031+0000 7f4736409740 1 -- 172.21.15.31:0/1585337747 shutdown_connections 2024-04-25T16:10:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.031+0000 7f4736409740 1 -- 172.21.15.31:0/1585337747 wait complete. 2024-04-25T16:10:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.031+0000 7f4736409740 1 -- start start 2024-04-25T16:10:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.032+0000 7f4736409740 1 Processor -- start 2024-04-25T16:10:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.032+0000 7f4736409740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 0x557a8d94ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.032+0000 7f4736409740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5784d0 0x557a8d9476b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.032+0000 7f4736409740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d947c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.032+0000 7f4736409740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f470c00f9c0 con 0x557a8d5784d0 2024-04-25T16:10:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.032+0000 7f4736409740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f470c00f2f0 con 0x557a8d5702f0 2024-04-25T16:10:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.032+0000 7f4736409740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f470c00c530 con 0x557a8d5991b0 2024-04-25T16:10:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.032+0000 7f4736409740 10 client.0 _ll_register_callbacks cb 0x7f47240b0000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472d220640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 0x557a8d94ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472d220640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 0x557a8d94ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:41034/0 (socket says 172.21.15.31:41034) 2024-04-25T16:10:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472ca1f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5784d0 0x557a8d9476b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472d220640 1 -- 172.21.15.31:0/3336824202 learned_addr learned my addr 172.21.15.31:0/3336824202 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472da21640 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d947c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472ca1f640 1 -- 172.21.15.31:0/3336824202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 msgr2=0x557a8d947c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472ca1f640 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d947c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472ca1f640 1 -- 172.21.15.31:0/3336824202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 msgr2=0x557a8d94ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472ca1f640 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 0x557a8d94ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.033+0000 7f472ca1f640 1 -- 172.21.15.31:0/3336824202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a8d9c8f00 con 0x557a8d5784d0 2024-04-25T16:10:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.035+0000 7f472ca1f640 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5784d0 0x557a8d9476b0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f47240b3e40 tx=0x7f4724005700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5173bc0644ad0787 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.035+0000 7f4710ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:10:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.035+0000 7f4710ff9640 1 -- 172.21.15.31:0/3336824202 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47240b8730 con 0x557a8d5784d0 2024-04-25T16:10:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.036+0000 7f4710ff9640 1 -- 172.21.15.31:0/3336824202 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f47240b82a0 con 0x557a8d5784d0 2024-04-25T16:10:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.036+0000 7f4736409740 1 -- 172.21.15.31:0/3336824202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f471801f030 con 0x557a8d5784d0 2024-04-25T16:10:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.036+0000 7f4710ff9640 1 -- 172.21.15.31:0/3336824202 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f47240b84a0 con 0x557a8d5784d0 2024-04-25T16:10:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.036+0000 7f4736409740 10 client.17487 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:34.038 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.036+0000 7f4736409740 1 -- 172.21.15.31:0/3336824202 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557a8d9ca190 con 0x557a8d5784d0 2024-04-25T16:10:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.038+0000 7f4710ff9640 1 -- 172.21.15.31:0/3336824202 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f471801f030 con 0x557a8d5784d0 2024-04-25T16:10:34.039 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.038+0000 7f4710ff9640 1 -- 172.21.15.31:0/3336824202 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f47240e6070 con 0x557a8d5784d0 2024-04-25T16:10:34.040 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.038+0000 7f4736409740 10 client.17487 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.038+0000 7f4736409740 10 client.17487 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.038+0000 7f4736409740 20 client.17487 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.038+0000 7f4736409740 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557a8d52d1c0 0x557a8d9cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.038+0000 7f4736409740 4 client.17487 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:10:34.041 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.038+0000 7f4736409740 1 -- 172.21.15.31:0/3336824202 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557a8d8a6ee0 con 0x557a8d52d1c0 2024-04-25T16:10:34.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.042+0000 7f472d220640 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557a8d52d1c0 0x557a8d9cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:34.044 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.043+0000 7f472d220640 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557a8d52d1c0 0x557a8d9cc9a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4297ca4aec65e3f0 server_cookie=f91a9c309fa20ad2 in_seq=0 out_seq=0 2024-04-25T16:10:34.045 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:34.043+0000 7f4710ff9640 10 client.17487 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:34.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:34 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:10:34.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:34 smithi031 ceph-mon[29140]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 897 MiB/s wr, 2.24k op/s 2024-04-25T16:10:34.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:34 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:10:34.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:34 smithi063 ceph-mon[32413]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 897 MiB/s wr, 2.24k op/s 2024-04-25T16:10:34.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:34 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:10:34.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:34 smithi186 ceph-mon[33258]: pgmap v629: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 897 MiB/s wr, 2.24k op/s 2024-04-25T16:10:37.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:37 smithi063 ceph-mon[32413]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 745 MiB/s wr, 2.00k op/s 2024-04-25T16:10:37.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:37 smithi186 ceph-mon[33258]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 745 MiB/s wr, 2.00k op/s 2024-04-25T16:10:37.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:37 smithi031 ceph-mon[29140]: pgmap v630: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 745 MiB/s wr, 2.00k op/s 2024-04-25T16:10:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.932+0000 7f4710ff9640 1 -- 172.21.15.31:0/3336824202 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557a8d8a6ee0 con 0x557a8d52d1c0 2024-04-25T16:10:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.932+0000 7f4710ff9640 10 client.17487 handle_command_reply: tid=0 2024-04-25T16:10:38.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.932+0000 7f4736409740 1 client.17487 shutdown 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 -- 172.21.15.31:0/3336824202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5784d0 msgr2=0x557a8d9476b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5784d0 0x557a8d9476b0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f47240b3e40 tx=0x7f4724005700 comp rx=0 tx=0).stop 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 -- 172.21.15.31:0/3336824202 shutdown_connections 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557a8d52d1c0 0x557a8d9cc9a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557a8d5991b0 0x557a8d947c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557a8d5784d0 0x557a8d9476b0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 --2- 172.21.15.31:0/3336824202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557a8d5702f0 0x557a8d94ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 -- 172.21.15.31:0/3336824202 >> 172.21.15.31:0/3336824202 conn(0x557a8d58d300 msgr2=0x557a8d93cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4710ff9640 0 client.17487 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 -- 172.21.15.31:0/3336824202 shutdown_connections 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 1 -- 172.21.15.31:0/3336824202 wait complete. 2024-04-25T16:10:38.936 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.933+0000 7f4736409740 20 client.17487 trim_cache size 0 max 16384 2024-04-25T16:10:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:10:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:10:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:10:38.937 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:10:38.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.935+0000 7f472e222640 1 -- 172.21.15.31:0/1836262434 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f47140afbe0 msgr2=0x7f47140b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.935+0000 7f472e222640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f47140afbe0 0x7f47140b20f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f4720000f80 tx=0x7f4720005fd0 comp rx=0 tx=0).stop 2024-04-25T16:10:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.935+0000 7f472e222640 1 -- 172.21.15.31:0/1836262434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f472810a0e0 msgr2=0x7f47281b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.935+0000 7f472e222640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f472810a0e0 0x7f47281b3dc0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4724009e60 tx=0x7f472400ed90 comp rx=0 tx=0).stop 2024-04-25T16:10:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.936+0000 7f472e222640 1 -- 172.21.15.31:0/1836262434 shutdown_connections 2024-04-25T16:10:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.936+0000 7f472e222640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f47140afbe0 0x7f47140b20f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.936+0000 7f472e222640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f472810b450 0x7f47281bad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.936+0000 7f472e222640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f472810a0e0 0x7f47281b3dc0 secure :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4724009e60 tx=0x7f472400ed90 comp rx=0 tx=0).stop 2024-04-25T16:10:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.936+0000 7f472e222640 1 --2- 172.21.15.31:0/1836262434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4728078df0 0x7f47281b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:38.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.936+0000 7f472e222640 1 -- 172.21.15.31:0/1836262434 >> 172.21.15.31:0/1836262434 conn(0x7f4728070330 msgr2=0x7f4728076de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.936+0000 7f472e222640 1 -- 172.21.15.31:0/1836262434 shutdown_connections 2024-04-25T16:10:38.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:38.941+0000 7f472e222640 1 -- 172.21.15.31:0/1836262434 wait complete. 2024-04-25T16:10:39.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:39 smithi063 ceph-mon[32413]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 908 MiB/s wr, 2.47k op/s 2024-04-25T16:10:39.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:39 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:10:39.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:39 smithi186 ceph-mon[33258]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 908 MiB/s wr, 2.47k op/s 2024-04-25T16:10:39.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:39 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:10:39.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:39 smithi031 ceph-mon[29140]: pgmap v631: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 908 MiB/s wr, 2.47k op/s 2024-04-25T16:10:39.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:39 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:10:40.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:40 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:10:40.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:40 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:10:40.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:40 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:10:41.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:41 smithi031 ceph-mon[29140]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 683 MiB/s wr, 1.99k op/s 2024-04-25T16:10:41.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:41 smithi063 ceph-mon[32413]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 683 MiB/s wr, 1.99k op/s 2024-04-25T16:10:41.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:41 smithi186 ceph-mon[33258]: pgmap v632: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 683 MiB/s wr, 1.99k op/s 2024-04-25T16:10:42.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:42 smithi063 ceph-mon[32413]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 725 MiB/s wr, 1.99k op/s 2024-04-25T16:10:42.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:42 smithi186 ceph-mon[33258]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 725 MiB/s wr, 1.99k op/s 2024-04-25T16:10:42.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:42 smithi031 ceph-mon[29140]: pgmap v633: 97 pgs: 97 active+clean; 10 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 725 MiB/s wr, 1.99k op/s 2024-04-25T16:10:45.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:45 smithi186 ceph-mon[33258]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 882 MiB/s wr, 2.21k op/s 2024-04-25T16:10:45.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:45 smithi031 ceph-mon[29140]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 882 MiB/s wr, 2.21k op/s 2024-04-25T16:10:45.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:45 smithi063 ceph-mon[32413]: pgmap v634: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 882 MiB/s wr, 2.21k op/s 2024-04-25T16:10:47.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:47 smithi031 ceph-mon[29140]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 757 MiB/s wr, 1.73k op/s 2024-04-25T16:10:47.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:47 smithi063 ceph-mon[32413]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 757 MiB/s wr, 1.73k op/s 2024-04-25T16:10:47.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:47 smithi186 ceph-mon[33258]: pgmap v635: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 757 MiB/s wr, 1.73k op/s 2024-04-25T16:10:48.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:48 smithi031 ceph-mon[29140]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 942 MiB/s wr, 2.25k op/s 2024-04-25T16:10:48.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:48 smithi063 ceph-mon[32413]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 942 MiB/s wr, 2.25k op/s 2024-04-25T16:10:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:48 smithi186 ceph-mon[33258]: pgmap v636: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 942 MiB/s wr, 2.25k op/s 2024-04-25T16:10:51.398 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:51 smithi186 ceph-mon[33258]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 732 MiB/s wr, 1.72k op/s 2024-04-25T16:10:51.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:51 smithi031 ceph-mon[29140]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 732 MiB/s wr, 1.72k op/s 2024-04-25T16:10:51.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:51 smithi063 ceph-mon[32413]: pgmap v637: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 732 MiB/s wr, 1.72k op/s 2024-04-25T16:10:52.428 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.426+0000 7fabc3e05640 1 -- 172.21.15.31:0/4070687463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabbc10a420 msgr2=0x7fabbc10ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.426+0000 7fabc3e05640 1 --2- 172.21.15.31:0/4070687463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabbc10a420 0x7fabbc10ae10 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fabac009900 tx=0x7fabac02a440 comp rx=0 tx=0).stop 2024-04-25T16:10:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.427+0000 7fabc3e05640 1 -- 172.21.15.31:0/4070687463 shutdown_connections 2024-04-25T16:10:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.427+0000 7fabc3e05640 1 --2- 172.21.15.31:0/4070687463 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc110070 0x7fabbc110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.427+0000 7fabc3e05640 1 --2- 172.21.15.31:0/4070687463 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabbc10b350 0x7fabbc10b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.427+0000 7fabc3e05640 1 --2- 172.21.15.31:0/4070687463 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabbc10a420 0x7fabbc10ae10 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.429 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.427+0000 7fabc3e05640 1 -- 172.21.15.31:0/4070687463 >> 172.21.15.31:0/4070687463 conn(0x7fabbc070540 msgr2=0x7fabbc0729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:52.441 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.440+0000 7fabc3e05640 1 -- 172.21.15.31:0/4070687463 shutdown_connections 2024-04-25T16:10:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.448+0000 7fabc3e05640 1 -- 172.21.15.31:0/4070687463 wait complete. 2024-04-25T16:10:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.449+0000 7fabc3e05640 1 Processor -- start 2024-04-25T16:10:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.449+0000 7fabc3e05640 1 -- start start 2024-04-25T16:10:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.450+0000 7fabc3e05640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabbc10b350 0x7fabbc116c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.450+0000 7fabc3e05640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabbc110070 0x7fabbc1171c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.450+0000 7fabc3e05640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc1a5d10 0x7fabbc1a8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.450+0000 7fabc3e05640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fabbc11e620 con 0x7fabbc110070 2024-04-25T16:10:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.450+0000 7fabc3e05640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fabbc11e4a0 con 0x7fabbc10b350 2024-04-25T16:10:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.450+0000 7fabc3e05640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fabbc11e7a0 con 0x7fabbc1a5d10 2024-04-25T16:10:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.450+0000 7fabc13a0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabbc110070 0x7fabbc1171c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.450+0000 7fabc13a0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabbc110070 0x7fabbc1171c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42722/0 (socket says 172.21.15.31:42722) 2024-04-25T16:10:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.450+0000 7fabc13a0640 1 -- 172.21.15.31:0/2175349634 learned_addr learned my addr 172.21.15.31:0/2175349634 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.451+0000 7fabc1ba1640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabbc10b350 0x7fabbc116c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.451+0000 7fabc23a2640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc1a5d10 0x7fabbc1a8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.452+0000 7fabc23a2640 1 -- 172.21.15.31:0/2175349634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabbc10b350 msgr2=0x7fabbc116c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.452+0000 7fabc23a2640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabbc10b350 0x7fabbc116c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.452+0000 7fabc23a2640 1 -- 172.21.15.31:0/2175349634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabbc110070 msgr2=0x7fabbc1171c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.452+0000 7fabc23a2640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabbc110070 0x7fabbc1171c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.452+0000 7fabc23a2640 1 -- 172.21.15.31:0/2175349634 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabbc1a8670 con 0x7fabbc1a5d10 2024-04-25T16:10:52.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.455+0000 7fabc23a2640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc1a5d10 0x7fabbc1a8130 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fabb800b3e0 tx=0x7fabb800b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.457+0000 7fabb2ffd640 1 -- 172.21.15.31:0/2175349634 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb8015020 con 0x7fabbc1a5d10 2024-04-25T16:10:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.457+0000 7fabb2ffd640 1 -- 172.21.15.31:0/2175349634 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb8011ce0 con 0x7fabbc1a5d10 2024-04-25T16:10:52.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.457+0000 7fabb2ffd640 1 -- 172.21.15.31:0/2175349634 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb8010e00 con 0x7fabbc1a5d10 2024-04-25T16:10:52.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.459+0000 7fabc3e05640 1 -- 172.21.15.31:0/2175349634 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabbc1a8960 con 0x7fabbc1a5d10 2024-04-25T16:10:52.461 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.459+0000 7fabc3e05640 1 -- 172.21.15.31:0/2175349634 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabbc1a8eb0 con 0x7fabbc1a5d10 2024-04-25T16:10:52.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.464+0000 7fabcad91740 1 Processor -- start 2024-04-25T16:10:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.464+0000 7fabcad91740 1 -- start start 2024-04-25T16:10:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.465+0000 7fabcad91740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 0x55e765683970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.465+0000 7fabcad91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765683ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.465+0000 7fabcad91740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 0x55e76568adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.465+0000 7fabcad91740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e7655e9460 con 0x55e7652c24d0 2024-04-25T16:10:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.465+0000 7fabcad91740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e7655a3590 con 0x55e7652e31b0 2024-04-25T16:10:52.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.465+0000 7fabcad91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e7655e5420 con 0x55e7652ba2f0 2024-04-25T16:10:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.469+0000 7fabc1ba1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765683ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.469+0000 7fabc1ba1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765683ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:59660/0 (socket says 172.21.15.31:59660) 2024-04-25T16:10:52.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.469+0000 7fabc1ba1640 1 -- 172.21.15.31:0/3026858268 learned_addr learned my addr 172.21.15.31:0/3026858268 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.471+0000 7fabc13a0640 1 --2- 172.21.15.31:0/3026858268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 0x55e765683970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.471+0000 7fabb2ffd640 1 -- 172.21.15.31:0/2175349634 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fabb8018020 con 0x7fabbc1a5d10 2024-04-25T16:10:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.471+0000 7fabc13a0640 1 --2- 172.21.15.31:0/3026858268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 0x55e76568adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.471+0000 7fabc1ba1640 1 -- 172.21.15.31:0/3026858268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 msgr2=0x55e765683970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.471+0000 7fabc1ba1640 1 --2- 172.21.15.31:0/3026858268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 0x55e765683970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.471+0000 7fabc1ba1640 1 -- 172.21.15.31:0/3026858268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 msgr2=0x55e76568adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.471+0000 7fabc1ba1640 1 --2- 172.21.15.31:0/3026858268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 0x55e76568adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.473 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.471+0000 7fabc1ba1640 1 -- 172.21.15.31:0/3026858268 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7655b00d0 con 0x55e7652ba2f0 2024-04-25T16:10:52.474 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.473+0000 7fabb2ffd640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faba80afcf0 0x7faba80b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.476 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.474+0000 7fabb2ffd640 1 -- 172.21.15.31:0/2175349634 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fabb80df3c0 con 0x7fabbc1a5d10 2024-04-25T16:10:52.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.476+0000 7fabc13a0640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faba80afcf0 0x7faba80b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.480+0000 7fabc1ba1640 1 --2- 172.21.15.31:0/3026858268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765683ef0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fabac000c00 tx=0x7fabac009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d6812c501b204d3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.483+0000 7faba7fff640 1 -- 172.21.15.31:0/3026858268 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabac040030 con 0x55e7652ba2f0 2024-04-25T16:10:52.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.484+0000 7faba7fff640 1 -- 172.21.15.31:0/3026858268 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabac004200 con 0x55e7652ba2f0 2024-04-25T16:10:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.485+0000 7fabcad91740 1 -- 172.21.15.31:0/3026858268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 msgr2=0x55e765683ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.485+0000 7fabcad91740 1 --2- 172.21.15.31:0/3026858268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765683ef0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fabac000c00 tx=0x7fabac009680 comp rx=0 tx=0).stop 2024-04-25T16:10:52.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.485+0000 7fabc13a0640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faba80afcf0 0x7faba80b2200 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009500 tx=0x7fabb4009490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:52.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.490+0000 7fabcad91740 1 -- 172.21.15.31:0/3026858268 shutdown_connections 2024-04-25T16:10:52.491 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.490+0000 7fabcad91740 1 --2- 172.21.15.31:0/3026858268 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 0x55e76568adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.490+0000 7fabcad91740 1 --2- 172.21.15.31:0/3026858268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765683ef0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.490+0000 7fabcad91740 1 --2- 172.21.15.31:0/3026858268 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 0x55e765683970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.490+0000 7fabcad91740 1 -- 172.21.15.31:0/3026858268 >> 172.21.15.31:0/3026858268 conn(0x55e7652d7300 msgr2=0x55e76567f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.490+0000 7fabcad91740 1 -- 172.21.15.31:0/3026858268 shutdown_connections 2024-04-25T16:10:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.490+0000 7fabcad91740 1 -- 172.21.15.31:0/3026858268 wait complete. 2024-04-25T16:10:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.491+0000 7fabcad91740 1 -- start start 2024-04-25T16:10:52.492 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.491+0000 7fabcad91740 1 Processor -- start 2024-04-25T16:10:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.492+0000 7fabcad91740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652ba2f0 0x55e76568d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.492+0000 7fabcad91740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 0x55e76568db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.492+0000 7fabcad91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652e31b0 0x55e765694a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.492+0000 7fabcad91740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e76568b830 con 0x55e7652c24d0 2024-04-25T16:10:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.492+0000 7fabcad91740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e765570b60 con 0x55e7652ba2f0 2024-04-25T16:10:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.492+0000 7fabcad91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e76568b9b0 con 0x55e7652e31b0 2024-04-25T16:10:52.493 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.492+0000 7fabcad91740 10 client.0 _ll_register_callbacks cb 0x55e7655952b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:52.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.494+0000 7fabc23a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652e31b0 0x55e765694a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.494+0000 7fabc1ba1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652ba2f0 0x55e76568d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.494+0000 7fabc23a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652e31b0 0x55e765694a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:59662/0 (socket says 172.21.15.31:59662) 2024-04-25T16:10:52.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.494+0000 7fabc23a2640 1 -- 172.21.15.31:0/3197677817 learned_addr learned my addr 172.21.15.31:0/3197677817 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.496+0000 7fabc13a0640 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 0x55e76568db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.499+0000 7fabc13a0640 1 -- 172.21.15.31:0/3197677817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652e31b0 msgr2=0x55e765694a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:52.502 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.499+0000 7fabc13a0640 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652e31b0 0x55e765694a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.499+0000 7fabc13a0640 1 -- 172.21.15.31:0/3197677817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652ba2f0 msgr2=0x55e76568d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.499+0000 7fabc13a0640 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652ba2f0 0x55e76568d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.499+0000 7fabc13a0640 1 -- 172.21.15.31:0/3197677817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7652fb210 con 0x55e7652c24d0 2024-04-25T16:10:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.503+0000 7fabc13a0640 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 0x55e76568db00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x55e765684b40 tx=0x7fabb4004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca5a02128d4f16a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:52.505 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.503+0000 7faba57fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:10:52.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.512+0000 7faba57fa640 1 -- 172.21.15.31:0/3197677817 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb402a060 con 0x55e7652c24d0 2024-04-25T16:10:52.513 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.512+0000 7faba57fa640 1 -- 172.21.15.31:0/3197677817 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fabb4019ca0 con 0x55e7652c24d0 2024-04-25T16:10:52.514 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.512+0000 7fabcad91740 1 -- 172.21.15.31:0/3197677817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabac004200 con 0x55e7652c24d0 2024-04-25T16:10:52.515 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.513+0000 7faba57fa640 1 -- 172.21.15.31:0/3197677817 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fabb4025400 con 0x55e7652c24d0 2024-04-25T16:10:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.514+0000 7fabcad91740 10 client.17499 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:52.516 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.515+0000 7fabcad91740 1 -- 172.21.15.31:0/3197677817 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e7653c7620 con 0x55e7652c24d0 2024-04-25T16:10:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.515+0000 7faba57fa640 1 -- 172.21.15.31:0/3197677817 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabac004200 con 0x55e7652c24d0 2024-04-25T16:10:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.516+0000 7faba57fa640 1 -- 172.21.15.31:0/3197677817 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fabb401a590 con 0x55e7652c24d0 2024-04-25T16:10:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.516+0000 7fabcad91740 10 client.17499 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.516+0000 7fabcad91740 10 client.17499 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.516+0000 7fabcad91740 20 client.17499 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.516+0000 7fabcad91740 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7652a1d80 0x55e76569b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.516+0000 7fabcad91740 4 client.17499 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:10:52.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.516+0000 7fabcad91740 1 -- 172.21.15.31:0/3197677817 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7655f7070 con 0x55e7652a1d80 2024-04-25T16:10:52.518 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.516+0000 7fabc1ba1640 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7652a1d80 0x55e76569b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.527+0000 7fabc1ba1640 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7652a1d80 0x55e76569b880 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f76f0a2a58507fba server_cookie=dfb1f1c3382a62f5 in_seq=0 out_seq=0 2024-04-25T16:10:52.528 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.527+0000 7faba57fa640 10 client.17499 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.531+0000 7faba57fa640 1 -- 172.21.15.31:0/3197677817 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e7655f7070 con 0x55e7652a1d80 2024-04-25T16:10:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.531+0000 7faba57fa640 10 client.17499 handle_command_reply: tid=0 2024-04-25T16:10:52.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.531+0000 7fabcad91740 1 client.17499 shutdown 2024-04-25T16:10:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.533+0000 7fabcad91740 1 -- 172.21.15.31:0/3197677817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 msgr2=0x55e76568db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.533+0000 7fabcad91740 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 0x55e76568db00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x55e765684b40 tx=0x7fabb4004a60 comp rx=0 tx=0).stop 2024-04-25T16:10:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.533+0000 7fabcad91740 1 -- 172.21.15.31:0/3197677817 shutdown_connections 2024-04-25T16:10:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.533+0000 7fabcad91740 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7652a1d80 0x55e76569b880 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.533+0000 7fabcad91740 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652e31b0 0x55e765694a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.533+0000 7fabcad91740 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652c24d0 0x55e76568db00 secure :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0x55e765684b40 tx=0x7fabb4004a60 comp rx=0 tx=0).stop 2024-04-25T16:10:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.533+0000 7fabcad91740 1 --2- 172.21.15.31:0/3197677817 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652ba2f0 0x55e76568d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.533+0000 7fabcad91740 1 -- 172.21.15.31:0/3197677817 >> 172.21.15.31:0/3197677817 conn(0x55e7652d7300 msgr2=0x55e765682210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.533+0000 7faba57fa640 0 client.17499 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.534+0000 7fabcad91740 1 -- 172.21.15.31:0/3197677817 shutdown_connections 2024-04-25T16:10:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.534+0000 7fabcad91740 1 -- 172.21.15.31:0/3197677817 wait complete. 2024-04-25T16:10:52.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.534+0000 7fabcad91740 20 client.17499 trim_cache size 0 max 16384 2024-04-25T16:10:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.572+0000 7fabcad91740 1 Processor -- start 2024-04-25T16:10:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.572+0000 7fabcad91740 1 -- start start 2024-04-25T16:10:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.572+0000 7fabcad91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e76568cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.572+0000 7fabcad91740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652e31b0 0x55e765694a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.573+0000 7fabcad91740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652771c0 0x55e76567eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.573+0000 7fabcad91740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faba000e540 con 0x55e7652e31b0 2024-04-25T16:10:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.573+0000 7fabcad91740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faba0010080 con 0x55e7652771c0 2024-04-25T16:10:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.573+0000 7fabcad91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faba000ec10 con 0x55e7652ba2f0 2024-04-25T16:10:52.580 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.573+0000 7fabc13a0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652e31b0 0x55e765694a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.573+0000 7fabc13a0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652e31b0 0x55e765694a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:42740/0 (socket says 172.21.15.31:42740) 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.573+0000 7fabc13a0640 1 -- 172.21.15.31:0/2360007852 learned_addr learned my addr 172.21.15.31:0/2360007852 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.573+0000 7fabc23a2640 1 --2- 172.21.15.31:0/2360007852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652771c0 0x55e76567eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.574+0000 7fabc23a2640 1 -- 172.21.15.31:0/2360007852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 msgr2=0x55e76568cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.574+0000 7fabc23a2640 1 --2- 172.21.15.31:0/2360007852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e76568cf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.574+0000 7fabc23a2640 1 -- 172.21.15.31:0/2360007852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652e31b0 msgr2=0x55e765694a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.574+0000 7fabc23a2640 1 --2- 172.21.15.31:0/2360007852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652e31b0 0x55e765694a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.574+0000 7fabc23a2640 1 -- 172.21.15.31:0/2360007852 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7656883a0 con 0x55e7652771c0 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.576+0000 7fabc23a2640 1 --2- 172.21.15.31:0/2360007852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652771c0 0x55e76567eab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fabb8006fd0 tx=0x7fabb80b0040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a28ff3b6a7558a31 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.576+0000 7faba77fe640 1 -- 172.21.15.31:0/2360007852 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb80c0020 con 0x55e7652771c0 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.577+0000 7faba77fe640 1 -- 172.21.15.31:0/2360007852 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb80a7f50 con 0x55e7652771c0 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.578+0000 7fabcad91740 1 -- 172.21.15.31:0/2360007852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652771c0 msgr2=0x55e76567eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:52.581 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.578+0000 7fabcad91740 1 --2- 172.21.15.31:0/2360007852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652771c0 0x55e76567eab0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fabb8006fd0 tx=0x7fabb80b0040 comp rx=0 tx=0).stop 2024-04-25T16:10:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.584+0000 7fabcad91740 1 -- 172.21.15.31:0/2360007852 shutdown_connections 2024-04-25T16:10:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.584+0000 7fabcad91740 1 --2- 172.21.15.31:0/2360007852 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652771c0 0x55e76567eab0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.584+0000 7fabcad91740 1 --2- 172.21.15.31:0/2360007852 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652e31b0 0x55e765694a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.584+0000 7fabcad91740 1 --2- 172.21.15.31:0/2360007852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e76568cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.584+0000 7fabcad91740 1 -- 172.21.15.31:0/2360007852 >> 172.21.15.31:0/2360007852 conn(0x55e7652d7300 msgr2=0x55e76568c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.589+0000 7fabcad91740 1 -- 172.21.15.31:0/2360007852 shutdown_connections 2024-04-25T16:10:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.589+0000 7fabcad91740 1 -- 172.21.15.31:0/2360007852 wait complete. 2024-04-25T16:10:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.590+0000 7fabcad91740 1 -- start start 2024-04-25T16:10:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.590+0000 7fabcad91740 1 Processor -- start 2024-04-25T16:10:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.591+0000 7fabcad91740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652771c0 0x55e765691dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.591+0000 7fabcad91740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765692310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.591+0000 7fabcad91740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 0x55e7656931f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.591+0000 7fabcad91740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faba000f9b0 con 0x55e7652771c0 2024-04-25T16:10:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.591+0000 7fabcad91740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faba000f2e0 con 0x55e7652e31b0 2024-04-25T16:10:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.591+0000 7fabcad91740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faba000c520 con 0x55e7652ba2f0 2024-04-25T16:10:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.591+0000 7fabcad91740 10 client.0 _ll_register_callbacks cb 0x7fabb4007000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:10:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc13a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765692310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc13a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765692310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:59676/0 (socket says 172.21.15.31:59676) 2024-04-25T16:10:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc13a0640 1 -- 172.21.15.31:0/3945389330 learned_addr learned my addr 172.21.15.31:0/3945389330 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:10:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc23a2640 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 0x55e7656931f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc1ba1640 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652771c0 0x55e765691dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc23a2640 1 -- 172.21.15.31:0/3945389330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 msgr2=0x55e765692310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc23a2640 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765692310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc23a2640 1 -- 172.21.15.31:0/3945389330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652771c0 msgr2=0x55e765691dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:10:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc23a2640 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652771c0 0x55e765691dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.594+0000 7fabc23a2640 1 -- 172.21.15.31:0/3945389330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7656937b0 con 0x55e7652e31b0 2024-04-25T16:10:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.595+0000 7fabc23a2640 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 0x55e7656931f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fabb80a7130 tx=0x7fabb80a30b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5957b807b6c42b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:10:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.596+0000 7faba57fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:10:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.596+0000 7faba57fa640 1 -- 172.21.15.31:0/3945389330 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabb80c0030 con 0x55e7652e31b0 2024-04-25T16:10:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.596+0000 7fabcad91740 1 -- 172.21.15.31:0/3945389330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabb80a7f50 con 0x55e7652e31b0 2024-04-25T16:10:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.596+0000 7faba57fa640 1 -- 172.21.15.31:0/3945389330 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fabb80a8b80 con 0x55e7652e31b0 2024-04-25T16:10:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.596+0000 7faba57fa640 1 -- 172.21.15.31:0/3945389330 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fabb80abc00 con 0x55e7652e31b0 2024-04-25T16:10:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.597+0000 7fabcad91740 10 client.27188 fetch_fsmap learned FSMap version 29 2024-04-25T16:10:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.597+0000 7fabcad91740 1 -- 172.21.15.31:0/3945389330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e765694a40 con 0x55e7652e31b0 2024-04-25T16:10:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.597+0000 7faba57fa640 1 -- 172.21.15.31:0/3945389330 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabb80a7f50 con 0x55e7652e31b0 2024-04-25T16:10:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.599+0000 7faba57fa640 1 -- 172.21.15.31:0/3945389330 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fabb80a6bd0 con 0x55e7652e31b0 2024-04-25T16:10:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.599+0000 7fabcad91740 10 client.27188 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:10:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.599+0000 7fabcad91740 10 client.27188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:10:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.599+0000 7fabcad91740 20 client.27188 populate_metadata read hostname 'smithi031' 2024-04-25T16:10:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.599+0000 7fabcad91740 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7657040d0 0x55e7657064f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:10:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.599+0000 7fabcad91740 4 client.27188 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:10:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.599+0000 7fabcad91740 1 -- 172.21.15.31:0/3945389330 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e7655f0ee0 con 0x55e7657040d0 2024-04-25T16:10:52.616 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.615+0000 7fabc1ba1640 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7657040d0 0x55e7657064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:10:52.617 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.616+0000 7fabc1ba1640 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7657040d0 0x55e7657064f0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dae9504aaba6b9c server_cookie=dba29cbf333abfa3 in_seq=0 out_seq=0 2024-04-25T16:10:52.628 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:52.620+0000 7faba57fa640 10 client.27188 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:53.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:53 smithi031 ceph-mon[29140]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 810 MiB/s wr, 1.88k op/s 2024-04-25T16:10:53.577 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:53 smithi186 ceph-mon[33258]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 810 MiB/s wr, 1.88k op/s 2024-04-25T16:10:53.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:53 smithi063 ceph-mon[32413]: pgmap v638: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 810 MiB/s wr, 1.88k op/s 2024-04-25T16:10:55.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:55 smithi031 ceph-mon[29140]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 931 MiB/s wr, 2.05k op/s 2024-04-25T16:10:55.454 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:55 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:10:55.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:55 smithi063 ceph-mon[32413]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 931 MiB/s wr, 2.05k op/s 2024-04-25T16:10:55.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:55 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:10:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:55 smithi186 ceph-mon[33258]: pgmap v639: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 931 MiB/s wr, 2.05k op/s 2024-04-25T16:10:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:55 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:10:56.453 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:56 smithi031 ceph-mon[29140]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 763 MiB/s wr, 1.79k op/s 2024-04-25T16:10:56.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:56 smithi063 ceph-mon[32413]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 763 MiB/s wr, 1.79k op/s 2024-04-25T16:10:56.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:56 smithi186 ceph-mon[33258]: pgmap v640: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 763 MiB/s wr, 1.79k op/s 2024-04-25T16:10:57.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:10:57.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:10:57.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:57 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:10:57.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:10:57.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:10:57.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:57 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:10:57.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:10:57.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:10:57.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:57 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:10:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.725+0000 7faba57fa640 1 -- 172.21.15.31:0/3945389330 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e7655f0ee0 con 0x55e7657040d0 2024-04-25T16:10:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.725+0000 7faba57fa640 10 client.27188 handle_command_reply: tid=0 2024-04-25T16:10:58.727 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.725+0000 7fabcad91740 1 client.27188 shutdown 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.729+0000 7fabcad91740 1 -- 172.21.15.31:0/3945389330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 msgr2=0x55e7656931f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.729+0000 7fabcad91740 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 0x55e7656931f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fabb80a7130 tx=0x7fabb80a30b0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.729+0000 7fabcad91740 1 -- 172.21.15.31:0/3945389330 shutdown_connections 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.729+0000 7fabcad91740 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55e7657040d0 0x55e7657064f0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.729+0000 7fabcad91740 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e7652e31b0 0x55e7656931f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.729+0000 7fabcad91740 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e7652ba2f0 0x55e765692310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.729+0000 7fabcad91740 1 --2- 172.21.15.31:0/3945389330 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e7652771c0 0x55e765691dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.729+0000 7fabcad91740 1 -- 172.21.15.31:0/3945389330 >> 172.21.15.31:0/3945389330 conn(0x55e7652d7300 msgr2=0x55e765682100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.729+0000 7faba57fa640 0 client.27188 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.730+0000 7fabcad91740 1 -- 172.21.15.31:0/3945389330 shutdown_connections 2024-04-25T16:10:58.732 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.730+0000 7fabcad91740 1 -- 172.21.15.31:0/3945389330 wait complete. 2024-04-25T16:10:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.730+0000 7fabcad91740 20 client.27188 trim_cache size 0 max 16384 2024-04-25T16:10:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:10:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:10:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:10:58.733 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:10:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.737+0000 7fabc3e05640 1 -- 172.21.15.31:0/2175349634 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faba80afcf0 msgr2=0x7faba80b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.738+0000 7fabc3e05640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faba80afcf0 0x7faba80b2200 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009500 tx=0x7fabb4009490 comp rx=0 tx=0).stop 2024-04-25T16:10:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.738+0000 7fabc3e05640 1 -- 172.21.15.31:0/2175349634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc1a5d10 msgr2=0x7fabbc1a8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:10:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.738+0000 7fabc3e05640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc1a5d10 0x7fabbc1a8130 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fabb800b3e0 tx=0x7fabb800b8a0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.743+0000 7fabc3e05640 1 -- 172.21.15.31:0/2175349634 shutdown_connections 2024-04-25T16:10:58.744 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.743+0000 7fabc3e05640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faba80afcf0 0x7faba80b2200 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.743+0000 7fabc3e05640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabbc1a5d10 0x7fabbc1a8130 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.743+0000 7fabc3e05640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fabbc110070 0x7fabbc1171c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.743+0000 7fabc3e05640 1 --2- 172.21.15.31:0/2175349634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fabbc10b350 0x7fabbc116c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:10:58.745 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.743+0000 7fabc3e05640 1 -- 172.21.15.31:0/2175349634 >> 172.21.15.31:0/2175349634 conn(0x7fabbc070540 msgr2=0x7fabbc072960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:10:58.748 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.745+0000 7fabc3e05640 1 -- 172.21.15.31:0/2175349634 shutdown_connections 2024-04-25T16:10:58.752 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:10:58.750+0000 7fabc3e05640 1 -- 172.21.15.31:0/2175349634 wait complete. 2024-04-25T16:10:58.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:10:58 smithi063 ceph-mon[32413]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 915 MiB/s wr, 2.34k op/s 2024-04-25T16:10:58.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:10:58 smithi186 ceph-mon[33258]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 915 MiB/s wr, 2.34k op/s 2024-04-25T16:10:58.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:10:58 smithi031 ceph-mon[29140]: pgmap v641: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 915 MiB/s wr, 2.34k op/s 2024-04-25T16:11:01.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:01 smithi031 ceph-mon[29140]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 707 MiB/s wr, 1.66k op/s 2024-04-25T16:11:01.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:01 smithi063 ceph-mon[32413]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 707 MiB/s wr, 1.66k op/s 2024-04-25T16:11:01.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:01 smithi186 ceph-mon[33258]: pgmap v642: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 707 MiB/s wr, 1.66k op/s 2024-04-25T16:11:01.958 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:11:01.958 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:11:02.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.511+0000 7f8238f6c640 1 -- 172.21.15.31:0/771469294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f823410a420 msgr2=0x7f823410add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:02.514 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.511+0000 7f8238f6c640 1 --2- 172.21.15.31:0/771469294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f823410a420 0x7f823410add0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f8224009ca0 tx=0x7f822402a8e0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.515+0000 7f8238f6c640 1 -- 172.21.15.31:0/771469294 shutdown_connections 2024-04-25T16:11:02.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.515+0000 7f8238f6c640 1 --2- 172.21.15.31:0/771469294 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8234110030 0x7f8234110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.517 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.515+0000 7f8238f6c640 1 --2- 172.21.15.31:0/771469294 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f823410b310 0x7f823410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.515+0000 7f8238f6c640 1 --2- 172.21.15.31:0/771469294 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f823410a420 0x7f823410add0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.518 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.515+0000 7f8238f6c640 1 -- 172.21.15.31:0/771469294 >> 172.21.15.31:0/771469294 conn(0x7f82340704e0 msgr2=0x7f8234072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:02.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.520+0000 7f8238f6c640 1 -- 172.21.15.31:0/771469294 shutdown_connections 2024-04-25T16:11:02.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.524+0000 7f8238f6c640 1 -- 172.21.15.31:0/771469294 wait complete. 2024-04-25T16:11:02.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.525+0000 7f8238f6c640 1 Processor -- start 2024-04-25T16:11:02.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.527+0000 7f8238f6c640 1 -- start start 2024-04-25T16:11:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.527+0000 7f8238f6c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f823410a420 0x7f82341a9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.527+0000 7f8238f6c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823410b310 0x7f82341a9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.530 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.527+0000 7f8238f6c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234110030 0x7f82341b0600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.527+0000 7f8238f6c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f823411e520 con 0x7f8234110030 2024-04-25T16:11:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.527+0000 7f8238f6c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f823411e3a0 con 0x7f823410a420 2024-04-25T16:11:02.531 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.527+0000 7f8238f6c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f823411e6a0 con 0x7f823410b310 2024-04-25T16:11:02.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.529+0000 7f8232d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234110030 0x7f82341b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.529+0000 7f8232d9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234110030 0x7f82341b0600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33782/0 (socket says 172.21.15.31:33782) 2024-04-25T16:11:02.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.529+0000 7f8232d9d640 1 -- 172.21.15.31:0/1456892011 learned_addr learned my addr 172.21.15.31:0/1456892011 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:02.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.530+0000 7f8231d9b640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823410b310 0x7f82341a9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.532+0000 7f8231d9b640 1 -- 172.21.15.31:0/1456892011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f823410a420 msgr2=0x7f82341a9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:02.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.532+0000 7f8231d9b640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f823410a420 0x7f82341a9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.532+0000 7f8231d9b640 1 -- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234110030 msgr2=0x7f82341b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:02.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.532+0000 7f8231d9b640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234110030 0x7f82341b0600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.532+0000 7f8231d9b640 1 -- 172.21.15.31:0/1456892011 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82341b0c70 con 0x7f823410b310 2024-04-25T16:11:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.535+0000 7f8231d9b640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823410b310 0x7f82341a9670 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f821c00b3f0 tx=0x7f821c00b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.538+0000 7f8232d9d640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234110030 0x7f82341b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:11:02.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.540+0000 7f822b7fe640 1 -- 172.21.15.31:0/1456892011 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c015020 con 0x7f823410b310 2024-04-25T16:11:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.541+0000 7f8238f6c640 1 -- 172.21.15.31:0/1456892011 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82341b0f60 con 0x7f823410b310 2024-04-25T16:11:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.541+0000 7f8238f6c640 1 -- 172.21.15.31:0/1456892011 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82341b1450 con 0x7f823410b310 2024-04-25T16:11:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.542+0000 7f822b7fe640 1 -- 172.21.15.31:0/1456892011 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f821c011ce0 con 0x7f823410b310 2024-04-25T16:11:02.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.544+0000 7f822b7fe640 1 -- 172.21.15.31:0/1456892011 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c010e10 con 0x7f823410b310 2024-04-25T16:11:02.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.549+0000 7f823fe19740 1 Processor -- start 2024-04-25T16:11:02.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.553+0000 7f822b7fe640 1 -- 172.21.15.31:0/1456892011 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f821c018020 con 0x7f823410b310 2024-04-25T16:11:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.553+0000 7f823fe19740 1 -- start start 2024-04-25T16:11:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.554+0000 7f823fe19740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 0x55952be68a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.554+0000 7f823fe19740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be68fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.554+0000 7f823fe19740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952baa66b0 0x55952be69c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.554+0000 7f823fe19740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55952bdcdd80 con 0x55952ba9e4d0 2024-04-25T16:11:02.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.554+0000 7f823fe19740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55952bd878b0 con 0x55952bac7390 2024-04-25T16:11:02.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.554+0000 7f823fe19740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55952bdc9d40 con 0x55952baa66b0 2024-04-25T16:11:02.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.556+0000 7f822b7fe640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82200afd00 0x7f82200b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.557+0000 7f822b7fe640 1 -- 172.21.15.31:0/1456892011 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f821c0de870 con 0x7f823410b310 2024-04-25T16:11:02.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.558+0000 7f8232d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 0x55952be68a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.559+0000 7f8232d9d640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82200afd00 0x7f82200b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.559+0000 7f823259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be68fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.559+0000 7f823259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be68fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33788/0 (socket says 172.21.15.31:33788) 2024-04-25T16:11:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.559+0000 7f823259c640 1 -- 172.21.15.31:0/922952733 learned_addr learned my addr 172.21.15.31:0/922952733 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:02.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.560+0000 7f8231d9b640 1 --2- 172.21.15.31:0/922952733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952baa66b0 0x55952be69c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.562+0000 7f823259c640 1 -- 172.21.15.31:0/922952733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952baa66b0 msgr2=0x55952be69c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.562+0000 7f823259c640 1 --2- 172.21.15.31:0/922952733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952baa66b0 0x55952be69c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.562+0000 7f823259c640 1 -- 172.21.15.31:0/922952733 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 msgr2=0x55952be68a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.562+0000 7f823259c640 1 --2- 172.21.15.31:0/922952733 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 0x55952be68a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.562+0000 7f823259c640 1 -- 172.21.15.31:0/922952733 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55952bd58a40 con 0x55952ba9e4d0 2024-04-25T16:11:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.562+0000 7f8231d9b640 1 --2- 172.21.15.31:0/922952733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952baa66b0 0x55952be69c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:11:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.564+0000 7f8232d9d640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82200afd00 0x7f82200b2210 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f822c013970 tx=0x7f822c009770 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:02.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.564+0000 7f823259c640 1 --2- 172.21.15.31:0/922952733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be68fe0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f82240099a0 tx=0x7f82240099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de936ef6c869ed32 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.564+0000 7f8228ff9640 1 -- 172.21.15.31:0/922952733 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8224031a20 con 0x55952ba9e4d0 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.565+0000 7f8228ff9640 1 -- 172.21.15.31:0/922952733 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8224036040 con 0x55952ba9e4d0 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.566+0000 7f8228ff9640 1 -- 172.21.15.31:0/922952733 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82240043c0 con 0x55952ba9e4d0 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.566+0000 7f823fe19740 1 -- 172.21.15.31:0/922952733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 msgr2=0x55952be68fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.566+0000 7f823fe19740 1 --2- 172.21.15.31:0/922952733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be68fe0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f82240099a0 tx=0x7f82240099d0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.566+0000 7f823fe19740 1 -- 172.21.15.31:0/922952733 shutdown_connections 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.566+0000 7f823fe19740 1 --2- 172.21.15.31:0/922952733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952baa66b0 0x55952be69c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.566+0000 7f823fe19740 1 --2- 172.21.15.31:0/922952733 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be68fe0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.566+0000 7f823fe19740 1 --2- 172.21.15.31:0/922952733 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 0x55952be68a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.566+0000 7f823fe19740 1 -- 172.21.15.31:0/922952733 >> 172.21.15.31:0/922952733 conn(0x55952babb4e0 msgr2=0x55952be64270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.567+0000 7f823fe19740 1 -- 172.21.15.31:0/922952733 shutdown_connections 2024-04-25T16:11:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.567+0000 7f823fe19740 1 -- 172.21.15.31:0/922952733 wait complete. 2024-04-25T16:11:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.568+0000 7f823fe19740 1 -- start start 2024-04-25T16:11:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.568+0000 7f823fe19740 1 Processor -- start 2024-04-25T16:11:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.568+0000 7f823fe19740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952ba9e4d0 0x55952be839a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.568+0000 7f823fe19740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952baa66b0 0x55952be83ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.568+0000 7f823fe19740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 0x55952be8ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.568+0000 7f823fe19740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55952be6a770 con 0x55952baa66b0 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.568+0000 7f823fe19740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55952bd54d10 con 0x55952bac7390 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.568+0000 7f823fe19740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55952be6a8f0 con 0x55952ba9e4d0 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.569+0000 7f823fe19740 10 client.0 _ll_register_callbacks cb 0x55952bd81210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.569+0000 7f8231d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952baa66b0 0x55952be83ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.569+0000 7f8231d9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952baa66b0 0x55952be83ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33798/0 (socket says 172.21.15.31:33798) 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.569+0000 7f8231d9b640 1 -- 172.21.15.31:0/1456081601 learned_addr learned my addr 172.21.15.31:0/1456081601 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.570+0000 7f8232d9d640 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 0x55952be8ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.571+0000 7f8231d9b640 1 -- 172.21.15.31:0/1456081601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952ba9e4d0 msgr2=0x55952be839a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.571+0000 7f8231d9b640 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952ba9e4d0 0x55952be839a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.571+0000 7f8231d9b640 1 -- 172.21.15.31:0/1456081601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 msgr2=0x55952be8ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.571+0000 7f8231d9b640 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 0x55952be8ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.571+0000 7f8231d9b640 1 -- 172.21.15.31:0/1456081601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55952bba97d0 con 0x55952baa66b0 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.571+0000 7f8231d9b640 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952baa66b0 0x55952be83ee0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f821c000c00 tx=0x7f821c0a65a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d21d47f4b84c93d6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.571+0000 7f8219ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.572+0000 7f8219ffb640 1 -- 172.21.15.31:0/1456081601 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c0a6d00 con 0x55952baa66b0 2024-04-25T16:11:02.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.572+0000 7f823fe19740 1 -- 172.21.15.31:0/1456081601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8224036040 con 0x55952baa66b0 2024-04-25T16:11:02.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.572+0000 7f8219ffb640 1 -- 172.21.15.31:0/1456081601 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f821c011830 con 0x55952baa66b0 2024-04-25T16:11:02.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.573+0000 7f8219ffb640 1 -- 172.21.15.31:0/1456081601 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f821c0ab0b0 con 0x55952baa66b0 2024-04-25T16:11:02.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.573+0000 7f823fe19740 10 client.17514 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:02.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.573+0000 7f823fe19740 1 -- 172.21.15.31:0/1456081601 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55952badf3f0 con 0x55952baa66b0 2024-04-25T16:11:02.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.574+0000 7f8219ffb640 1 -- 172.21.15.31:0/1456081601 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8224036040 con 0x55952baa66b0 2024-04-25T16:11:02.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.576+0000 7f8219ffb640 1 -- 172.21.15.31:0/1456081601 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f821c08fb10 con 0x55952baa66b0 2024-04-25T16:11:02.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.577+0000 7f823fe19740 10 client.17514 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:02.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.577+0000 7f823fe19740 10 client.17514 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:02.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.577+0000 7f823fe19740 20 client.17514 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:02.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.577+0000 7f823fe19740 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55952ba85f60 0x55952beb3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.577+0000 7f823fe19740 4 client.17514 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:02.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.577+0000 7f823fe19740 1 -- 172.21.15.31:0/1456081601 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55952bddb990 con 0x55952ba85f60 2024-04-25T16:11:02.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.579+0000 7f823259c640 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55952ba85f60 0x55952beb3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.586+0000 7f823259c640 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55952ba85f60 0x55952beb3250 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77dc5a2ee7d9e1c7 server_cookie=149789377f056751 in_seq=0 out_seq=0 2024-04-25T16:11:02.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.586+0000 7f8219ffb640 10 client.17514 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.591+0000 7f8219ffb640 1 -- 172.21.15.31:0/1456081601 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55952bddb990 con 0x55952ba85f60 2024-04-25T16:11:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.592+0000 7f8219ffb640 10 client.17514 handle_command_reply: tid=0 2024-04-25T16:11:02.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.592+0000 7f823fe19740 1 client.17514 shutdown 2024-04-25T16:11:02.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.595+0000 7f823fe19740 1 -- 172.21.15.31:0/1456081601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952baa66b0 msgr2=0x55952be83ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:02.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.595+0000 7f823fe19740 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952baa66b0 0x55952be83ee0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f821c000c00 tx=0x7f821c0a65a0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.595+0000 7f823fe19740 1 -- 172.21.15.31:0/1456081601 shutdown_connections 2024-04-25T16:11:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.595+0000 7f823fe19740 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55952ba85f60 0x55952beb3250 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.595+0000 7f823fe19740 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952bac7390 0x55952be8ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.595+0000 7f823fe19740 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952baa66b0 0x55952be83ee0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.595+0000 7f823fe19740 1 --2- 172.21.15.31:0/1456081601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952ba9e4d0 0x55952be839a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.595+0000 7f8219ffb640 0 client.17514 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:02.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.595+0000 7f823fe19740 1 -- 172.21.15.31:0/1456081601 >> 172.21.15.31:0/1456081601 conn(0x55952babb4e0 msgr2=0x55952be67180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.598+0000 7f823fe19740 1 -- 172.21.15.31:0/1456081601 shutdown_connections 2024-04-25T16:11:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.601+0000 7f823fe19740 1 -- 172.21.15.31:0/1456081601 wait complete. 2024-04-25T16:11:02.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.601+0000 7f823fe19740 20 client.17514 trim_cache size 0 max 16384 2024-04-25T16:11:02.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.651+0000 7f823fe19740 1 Processor -- start 2024-04-25T16:11:02.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.651+0000 7f823fe19740 1 -- start start 2024-04-25T16:11:02.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.652+0000 7f823fe19740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be67aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.653+0000 7f823fe19740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 0x55952be630d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.653+0000 7f823fe19740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 0x55952be82f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.653+0000 7f823fe19740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f821400e320 con 0x55952ba9e4d0 2024-04-25T16:11:02.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.653+0000 7f823fe19740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f821400fe60 con 0x55952baa66b0 2024-04-25T16:11:02.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.654+0000 7f823fe19740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f821400e9f0 con 0x55952bac7390 2024-04-25T16:11:02.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.656+0000 7f8232d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 0x55952be82f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.656+0000 7f8232d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 0x55952be82f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48264/0 (socket says 172.21.15.31:48264) 2024-04-25T16:11:02.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.656+0000 7f8232d9d640 1 -- 172.21.15.31:0/2145441077 learned_addr learned my addr 172.21.15.31:0/2145441077 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:02.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.658+0000 7f823259c640 1 --2- 172.21.15.31:0/2145441077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be67aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.658+0000 7f8231d9b640 1 --2- 172.21.15.31:0/2145441077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 0x55952be630d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.659+0000 7f823259c640 1 -- 172.21.15.31:0/2145441077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 msgr2=0x55952be82f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.659+0000 7f823259c640 1 --2- 172.21.15.31:0/2145441077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 0x55952be82f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.659+0000 7f823259c640 1 -- 172.21.15.31:0/2145441077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 msgr2=0x55952be630d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.659+0000 7f823259c640 1 --2- 172.21.15.31:0/2145441077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 0x55952be630d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.659+0000 7f823259c640 1 -- 172.21.15.31:0/2145441077 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55952ba9f0f0 con 0x55952ba9e4d0 2024-04-25T16:11:02.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.660+0000 7f823259c640 1 --2- 172.21.15.31:0/2145441077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be67aa0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8224009bf0 tx=0x7f8224000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5c3c9871ac3c55 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:02.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.666+0000 7f821bfff640 1 -- 172.21.15.31:0/2145441077 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8224005bd0 con 0x55952ba9e4d0 2024-04-25T16:11:02.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.667+0000 7f821bfff640 1 -- 172.21.15.31:0/2145441077 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8224036040 con 0x55952ba9e4d0 2024-04-25T16:11:02.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.667+0000 7f821bfff640 1 -- 172.21.15.31:0/2145441077 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8224050a50 con 0x55952ba9e4d0 2024-04-25T16:11:02.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.669+0000 7f823fe19740 1 -- 172.21.15.31:0/2145441077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 msgr2=0x55952be67aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:02.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.669+0000 7f823fe19740 1 --2- 172.21.15.31:0/2145441077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be67aa0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f8224009bf0 tx=0x7f8224000f40 comp rx=0 tx=0).stop 2024-04-25T16:11:02.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.671+0000 7f823fe19740 1 -- 172.21.15.31:0/2145441077 shutdown_connections 2024-04-25T16:11:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.671+0000 7f823fe19740 1 --2- 172.21.15.31:0/2145441077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 0x55952be82f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.671+0000 7f823fe19740 1 --2- 172.21.15.31:0/2145441077 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 0x55952be630d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.671+0000 7f823fe19740 1 --2- 172.21.15.31:0/2145441077 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be67aa0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.671+0000 7f823fe19740 1 -- 172.21.15.31:0/2145441077 >> 172.21.15.31:0/2145441077 conn(0x55952babb4e0 msgr2=0x55952be629c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.676+0000 7f823fe19740 1 -- 172.21.15.31:0/2145441077 shutdown_connections 2024-04-25T16:11:02.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.676+0000 7f823fe19740 1 -- 172.21.15.31:0/2145441077 wait complete. 2024-04-25T16:11:02.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.677+0000 7f823fe19740 1 -- start start 2024-04-25T16:11:02.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.677+0000 7f823fe19740 1 Processor -- start 2024-04-25T16:11:02.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.678+0000 7f823fe19740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be88780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.678+0000 7f823fe19740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 0x55952be88cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.679+0000 7f823fe19740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 0x55952bf195f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.679+0000 7f823fe19740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f821400f790 con 0x55952ba9e4d0 2024-04-25T16:11:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.679+0000 7f823fe19740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f821400f0c0 con 0x55952baa66b0 2024-04-25T16:11:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.679+0000 7f823fe19740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f821400c3b0 con 0x55952bac7390 2024-04-25T16:11:02.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.679+0000 7f823fe19740 10 client.0 _ll_register_callbacks cb 0x7f821c0a66c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:02.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.680+0000 7f823259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be88780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.680+0000 7f8232d9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 0x55952bf195f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.680+0000 7f823259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be88780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:33806/0 (socket says 172.21.15.31:33806) 2024-04-25T16:11:02.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.680+0000 7f823259c640 1 -- 172.21.15.31:0/4134733983 learned_addr learned my addr 172.21.15.31:0/4134733983 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:02.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.680+0000 7f8231d9b640 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 0x55952be88cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.682+0000 7f8231d9b640 1 -- 172.21.15.31:0/4134733983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 msgr2=0x55952bf195f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.682+0000 7f8231d9b640 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 0x55952bf195f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.682+0000 7f8231d9b640 1 -- 172.21.15.31:0/4134733983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 msgr2=0x55952be88780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.682+0000 7f8231d9b640 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be88780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.682+0000 7f8231d9b640 1 -- 172.21.15.31:0/4134733983 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55952bd8e4b0 con 0x55952baa66b0 2024-04-25T16:11:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.682+0000 7f8231d9b640 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 0x55952be88cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f821c00bdb0 tx=0x7f821c00d130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c08c78ff8a8ad9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.682+0000 7f8219ffb640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:02.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.682+0000 7f8219ffb640 1 -- 172.21.15.31:0/4134733983 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f821c0a6d00 con 0x55952baa66b0 2024-04-25T16:11:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.686+0000 7f8219ffb640 1 -- 172.21.15.31:0/4134733983 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f821c0a2990 con 0x55952baa66b0 2024-04-25T16:11:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.687+0000 7f823fe19740 1 -- 172.21.15.31:0/4134733983 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8224036040 con 0x55952baa66b0 2024-04-25T16:11:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.687+0000 7f8219ffb640 1 -- 172.21.15.31:0/4134733983 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f821c0a2670 con 0x55952baa66b0 2024-04-25T16:11:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.687+0000 7f823fe19740 10 client.27203 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:02.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.687+0000 7f823fe19740 1 -- 172.21.15.31:0/4134733983 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55952bf1ac50 con 0x55952baa66b0 2024-04-25T16:11:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.688+0000 7f8219ffb640 1 -- 172.21.15.31:0/4134733983 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f821c0a5f60 con 0x55952baa66b0 2024-04-25T16:11:02.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.688+0000 7f8219ffb640 1 -- 172.21.15.31:0/4134733983 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f821c0ac240 con 0x55952baa66b0 2024-04-25T16:11:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.693+0000 7f823fe19740 10 client.27203 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.693+0000 7f823fe19740 10 client.27203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.693+0000 7f823fe19740 20 client.27203 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.693+0000 7f823fe19740 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55952ba5b3a0 0x55952bf1d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.693+0000 7f823fe19740 4 client.27203 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:11:02.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.693+0000 7f823fe19740 1 -- 172.21.15.31:0/4134733983 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f8224036040 con 0x55952ba5b3a0 2024-04-25T16:11:02.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.694+0000 7f823259c640 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55952ba5b3a0 0x55952bf1d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:02.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.695+0000 7f8219ffb640 10 client.27203 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:02.699 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.695+0000 7f823259c640 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55952ba5b3a0 0x55952bf1d480 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e28980e0d8b5dce6 server_cookie=eb3c23cfc5b94be8 in_seq=0 out_seq=0 2024-04-25T16:11:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.701+0000 7f8219ffb640 1 -- 172.21.15.31:0/4134733983 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f8224036040 con 0x55952ba5b3a0 2024-04-25T16:11:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.701+0000 7f8219ffb640 10 client.27203 handle_command_reply: tid=0 2024-04-25T16:11:02.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.701+0000 7f823fe19740 1 client.27203 shutdown 2024-04-25T16:11:02.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.701+0000 7f823fe19740 1 -- 172.21.15.31:0/4134733983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 msgr2=0x55952be88cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:02.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.701+0000 7f823fe19740 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 0x55952be88cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f821c00bdb0 tx=0x7f821c00d130 comp rx=0 tx=0).stop 2024-04-25T16:11:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.705+0000 7f823fe19740 1 -- 172.21.15.31:0/4134733983 shutdown_connections 2024-04-25T16:11:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.706+0000 7f823fe19740 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55952ba5b3a0 0x55952bf1d480 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.706+0000 7f823fe19740 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55952bac7390 0x55952bf195f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.706+0000 7f823fe19740 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55952baa66b0 0x55952be88cc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.706+0000 7f8219ffb640 0 client.27203 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:02.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.706+0000 7f823fe19740 1 --2- 172.21.15.31:0/4134733983 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55952ba9e4d0 0x55952be88780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.706+0000 7f823fe19740 1 -- 172.21.15.31:0/4134733983 >> 172.21.15.31:0/4134733983 conn(0x55952babb4e0 msgr2=0x55952be67800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.707+0000 7f823fe19740 1 -- 172.21.15.31:0/4134733983 shutdown_connections 2024-04-25T16:11:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.708+0000 7f823fe19740 1 -- 172.21.15.31:0/4134733983 wait complete. 2024-04-25T16:11:02.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.708+0000 7f823fe19740 20 client.27203 trim_cache size 0 max 16384 2024-04-25T16:11:02.713 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:11:02.713 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:11:02.713 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:11:02.713 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:11:02.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.716+0000 7f8238f6c640 1 -- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82200afd00 msgr2=0x7f82200b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:02.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.716+0000 7f8238f6c640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82200afd00 0x7f82200b2210 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f822c013970 tx=0x7f822c009770 comp rx=0 tx=0).stop 2024-04-25T16:11:02.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.716+0000 7f8238f6c640 1 -- 172.21.15.31:0/1456892011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823410b310 msgr2=0x7f82341a9670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:02.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.716+0000 7f8238f6c640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823410b310 0x7f82341a9670 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f821c00b3f0 tx=0x7f821c00b8b0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.719+0000 7f8238f6c640 1 -- 172.21.15.31:0/1456892011 shutdown_connections 2024-04-25T16:11:02.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.719+0000 7f8238f6c640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f82200afd00 0x7f82200b2210 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.719+0000 7f8238f6c640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8234110030 0x7f82341b0600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.719+0000 7f8238f6c640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f823410b310 0x7f82341a9670 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.719+0000 7f8238f6c640 1 --2- 172.21.15.31:0/1456892011 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f823410a420 0x7f82341a9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:02.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.719+0000 7f8238f6c640 1 -- 172.21.15.31:0/1456892011 >> 172.21.15.31:0/1456892011 conn(0x7f82340704e0 msgr2=0x7f823410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:02.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.731+0000 7f8238f6c640 1 -- 172.21.15.31:0/1456892011 shutdown_connections 2024-04-25T16:11:02.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:02.731+0000 7f8238f6c640 1 -- 172.21.15.31:0/1456892011 wait complete. 2024-04-25T16:11:02.789 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:11:02.789 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:11:03.318 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.316+0000 7fcd20ae7640 1 -- 172.21.15.31:0/82571099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd1c103c00 msgr2=0x7fcd1c104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:03.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.316+0000 7fcd20ae7640 1 --2- 172.21.15.31:0/82571099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd1c103c00 0x7fcd1c104010 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fcd14009ca0 tx=0x7fcd1402e9d0 comp rx=0 tx=0).stop 2024-04-25T16:11:03.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.317+0000 7fcd20ae7640 1 -- 172.21.15.31:0/82571099 shutdown_connections 2024-04-25T16:11:03.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.317+0000 7fcd20ae7640 1 --2- 172.21.15.31:0/82571099 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd1c108900 0x7fcd1c10ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:03.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.317+0000 7fcd20ae7640 1 --2- 172.21.15.31:0/82571099 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd1c103c00 0x7fcd1c104010 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:03.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.317+0000 7fcd20ae7640 1 --2- 172.21.15.31:0/82571099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c1029c0 0x7fcd1c1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:03.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.317+0000 7fcd20ae7640 1 -- 172.21.15.31:0/82571099 >> 172.21.15.31:0/82571099 conn(0x7fcd1c0fdbd0 msgr2=0x7fcd1c100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:03.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.318+0000 7fcd20ae7640 1 -- 172.21.15.31:0/82571099 shutdown_connections 2024-04-25T16:11:03.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.320+0000 7fcd20ae7640 1 -- 172.21.15.31:0/82571099 wait complete. 2024-04-25T16:11:03.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.322+0000 7fcd20ae7640 1 Processor -- start 2024-04-25T16:11:03.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd20ae7640 1 -- start start 2024-04-25T16:11:03.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd20ae7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd1c1029c0 0x7fcd1c1a91d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:03.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd20ae7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c103c00 0x7fcd1c1a9710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:03.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd20ae7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd1c108900 0x7fcd1c1b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:03.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd20ae7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c10d530 con 0x7fcd1c1029c0 2024-04-25T16:11:03.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd20ae7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c10d3b0 con 0x7fcd1c108900 2024-04-25T16:11:03.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd20ae7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcd1c10d6b0 con 0x7fcd1c103c00 2024-04-25T16:11:03.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd19d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c103c00 0x7fcd1c1a9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:03.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd19d9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c103c00 0x7fcd1c1a9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48286/0 (socket says 172.21.15.31:48286) 2024-04-25T16:11:03.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.324+0000 7fcd19d9b640 1 -- 172.21.15.31:0/232545997 learned_addr learned my addr 172.21.15.31:0/232545997 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:03.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.325+0000 7fcd1a59c640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd1c1029c0 0x7fcd1c1a91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:03.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.325+0000 7fcd19d9b640 1 -- 172.21.15.31:0/232545997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd1c108900 msgr2=0x7fcd1c1b06a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:11:03.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.325+0000 7fcd19d9b640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd1c108900 0x7fcd1c1b06a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:03.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.325+0000 7fcd19d9b640 1 -- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd1c1029c0 msgr2=0x7fcd1c1a91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:03.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.325+0000 7fcd19d9b640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd1c1029c0 0x7fcd1c1a91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:03.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.325+0000 7fcd19d9b640 1 -- 172.21.15.31:0/232545997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd1c1b0d10 con 0x7fcd1c103c00 2024-04-25T16:11:03.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.326+0000 7fcd19d9b640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c103c00 0x7fcd1c1a9710 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcd14000f80 tx=0x7fcd14032620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:03.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.326+0000 7fcd0b7fe640 1 -- 172.21.15.31:0/232545997 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14040030 con 0x7fcd1c103c00 2024-04-25T16:11:03.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.327+0000 7fcd20ae7640 1 -- 172.21.15.31:0/232545997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd1c1b0fa0 con 0x7fcd1c103c00 2024-04-25T16:11:03.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.327+0000 7fcd20ae7640 1 -- 172.21.15.31:0/232545997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd1c1b1480 con 0x7fcd1c103c00 2024-04-25T16:11:03.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.327+0000 7fcd0b7fe640 1 -- 172.21.15.31:0/232545997 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd14003e20 con 0x7fcd1c103c00 2024-04-25T16:11:03.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.327+0000 7fcd0b7fe640 1 -- 172.21.15.31:0/232545997 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd14037050 con 0x7fcd1c103c00 2024-04-25T16:11:03.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.332+0000 7fcd20ae7640 1 -- 172.21.15.31:0/232545997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccfc0052c0 con 0x7fcd1c103c00 2024-04-25T16:11:03.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.336+0000 7fcd0b7fe640 1 -- 172.21.15.31:0/232545997 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcd1403a070 con 0x7fcd1c103c00 2024-04-25T16:11:03.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.337+0000 7fcd0b7fe640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcd040afcf0 0x7fcd040b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:03.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.338+0000 7fcd1a59c640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcd040afcf0 0x7fcd040b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:03.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.338+0000 7fcd0b7fe640 1 -- 172.21.15.31:0/232545997 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcd140fef80 con 0x7fcd1c103c00 2024-04-25T16:11:03.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.338+0000 7fcd1a59c640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcd040afcf0 0x7fcd040b2200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:03.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:03.346+0000 7fcd0b7fe640 1 -- 172.21.15.31:0/232545997 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcd140c7b10 con 0x7fcd1c103c00 2024-04-25T16:11:03.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:03 smithi031 ceph-mon[29140]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 740 MiB/s wr, 1.66k op/s 2024-04-25T16:11:03.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:03 smithi063 ceph-mon[32413]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 740 MiB/s wr, 1.66k op/s 2024-04-25T16:11:03.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:03 smithi186 ceph-mon[33258]: pgmap v643: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 740 MiB/s wr, 1.66k op/s 2024-04-25T16:11:04.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.094+0000 7fcd20ae7640 1 -- 172.21.15.31:0/232545997 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fccfc0055b0 con 0x7fcd1c103c00 2024-04-25T16:11:04.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.098+0000 7fcd0b7fe640 1 -- 172.21.15.31:0/232545997 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7fcd140c7410 con 0x7fcd1c103c00 2024-04-25T16:11:04.100 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:11:04.100 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:11:04.104 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:11:04.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.109+0000 7fcd097fa640 1 -- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcd040afcf0 msgr2=0x7fcd040b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.109+0000 7fcd097fa640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcd040afcf0 0x7fcd040b2200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c006fd0 tx=0x7fcd0c008040 comp rx=0 tx=0).stop 2024-04-25T16:11:04.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.109+0000 7fcd097fa640 1 -- 172.21.15.31:0/232545997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c103c00 msgr2=0x7fcd1c1a9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.109+0000 7fcd097fa640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c103c00 0x7fcd1c1a9710 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fcd14000f80 tx=0x7fcd14032620 comp rx=0 tx=0).stop 2024-04-25T16:11:04.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.113+0000 7fcd097fa640 1 -- 172.21.15.31:0/232545997 shutdown_connections 2024-04-25T16:11:04.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.113+0000 7fcd097fa640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcd040afcf0 0x7fcd040b2200 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.113+0000 7fcd097fa640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcd1c108900 0x7fcd1c1b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.113+0000 7fcd097fa640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd1c103c00 0x7fcd1c1a9710 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.113+0000 7fcd097fa640 1 --2- 172.21.15.31:0/232545997 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcd1c1029c0 0x7fcd1c1a91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.113+0000 7fcd097fa640 1 -- 172.21.15.31:0/232545997 >> 172.21.15.31:0/232545997 conn(0x7fcd1c0fdbd0 msgr2=0x7fcd1c1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:04.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.113+0000 7fcd097fa640 1 -- 172.21.15.31:0/232545997 shutdown_connections 2024-04-25T16:11:04.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.113+0000 7fcd097fa640 1 -- 172.21.15.31:0/232545997 wait complete. 2024-04-25T16:11:04.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:11:04.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:04 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/232545997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:11:04.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:04 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/232545997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:11:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:04 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/232545997' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:11:04.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.668+0000 7f5a2f613640 1 -- 172.21.15.31:0/246525924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a28110030 msgr2=0x7f5a28110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.670 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.668+0000 7f5a2f613640 1 --2- 172.21.15.31:0/246525924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a28110030 0x7f5a28110440 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009be0 tx=0x7f5a2002a6f0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.670+0000 7f5a2f613640 1 -- 172.21.15.31:0/246525924 shutdown_connections 2024-04-25T16:11:04.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.670+0000 7f5a2f613640 1 --2- 172.21.15.31:0/246525924 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a28110030 0x7f5a28110440 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.670+0000 7f5a2f613640 1 --2- 172.21.15.31:0/246525924 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a2810b310 0x7f5a2810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.670+0000 7f5a2f613640 1 --2- 172.21.15.31:0/246525924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a2810a420 0x7f5a2810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.670+0000 7f5a2f613640 1 -- 172.21.15.31:0/246525924 >> 172.21.15.31:0/246525924 conn(0x7f5a280704e0 msgr2=0x7f5a28072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.677+0000 7f5a2f613640 1 -- 172.21.15.31:0/246525924 shutdown_connections 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.677+0000 7f5a2f613640 1 -- 172.21.15.31:0/246525924 wait complete. 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.678+0000 7f5a2f613640 1 Processor -- start 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.678+0000 7f5a2f613640 1 -- start start 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.679+0000 7f5a2f613640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a2810a420 0x7f5a281a90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.679+0000 7f5a2f613640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a2810b310 0x7f5a281a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.679+0000 7f5a2f613640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a28110030 0x7f5a281b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.679+0000 7f5a2f613640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2811e490 con 0x7f5a2810b310 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.679+0000 7f5a2f613640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2811e310 con 0x7f5a2810a420 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.679+0000 7f5a2f613640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5a2811e610 con 0x7f5a28110030 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.680+0000 7f5a2dbb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a28110030 0x7f5a281b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.680+0000 7f5a2dbb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a28110030 0x7f5a281b0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34106/0 (socket says 172.21.15.31:34106) 2024-04-25T16:11:04.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.680+0000 7f5a2dbb0640 1 -- 172.21.15.31:0/1752617676 learned_addr learned my addr 172.21.15.31:0/1752617676 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:04.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.680+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a2810a420 0x7f5a281a90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.680+0000 7f5a2dbb0640 1 -- 172.21.15.31:0/1752617676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a2810a420 msgr2=0x7f5a281a90a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.680+0000 7f5a2dbb0640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a2810a420 0x7f5a281a90a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.680+0000 7f5a2dbb0640 1 -- 172.21.15.31:0/1752617676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a2810b310 msgr2=0x7f5a281a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.680+0000 7f5a2dbb0640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a2810b310 0x7f5a281a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.681+0000 7f5a2dbb0640 1 -- 172.21.15.31:0/1752617676 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a28111c90 con 0x7f5a28110030 2024-04-25T16:11:04.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.681+0000 7f5a2dbb0640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a28110030 0x7f5a281b0570 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5a20034040 tx=0x7f5a20030ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:04.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.684+0000 7f5a1e7fc640 1 -- 172.21.15.31:0/1752617676 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20041020 con 0x7f5a28110030 2024-04-25T16:11:04.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.684+0000 7f5a2f613640 1 -- 172.21.15.31:0/1752617676 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a28111f20 con 0x7f5a28110030 2024-04-25T16:11:04.687 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.684+0000 7f5a2f613640 1 -- 172.21.15.31:0/1752617676 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a28112410 con 0x7f5a28110030 2024-04-25T16:11:04.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.684+0000 7f5a1e7fc640 1 -- 172.21.15.31:0/1752617676 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a20003e20 con 0x7f5a28110030 2024-04-25T16:11:04.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.686+0000 7f5a1e7fc640 1 -- 172.21.15.31:0/1752617676 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a20036040 con 0x7f5a28110030 2024-04-25T16:11:04.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.689+0000 7f5a1e7fc640 1 -- 172.21.15.31:0/1752617676 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5a20044020 con 0x7f5a28110030 2024-04-25T16:11:04.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.690+0000 7f5a1e7fc640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a140afd00 0x7f5a140b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.691+0000 7f5a1e7fc640 1 -- 172.21.15.31:0/1752617676 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5a20100230 con 0x7f5a28110030 2024-04-25T16:11:04.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.694+0000 7f5a3659f740 1 Processor -- start 2024-04-25T16:11:04.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.694+0000 7f5a3659f740 1 -- start start 2024-04-25T16:11:04.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.695+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf343810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.695+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf343d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.695+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf344a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.695+0000 7f5a3659f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d5bf2a7d80 con 0x55d5bef806b0 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.695+0000 7f5a3659f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d5bf261a20 con 0x55d5bef784d0 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.695+0000 7f5a3659f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d5bf2a3d40 con 0x55d5befa1390 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.695+0000 7f5a2dbb0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf344a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.696+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a140afd00 0x7f5a140b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.696+0000 7f5a2d3af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf343810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.696+0000 7f5a2d3af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf343810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34110/0 (socket says 172.21.15.31:34110) 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.696+0000 7f5a2d3af640 1 -- 172.21.15.31:0/4164028348 learned_addr learned my addr 172.21.15.31:0/4164028348 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.697+0000 7f5a2cbae640 1 --2- 172.21.15.31:0/4164028348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf343d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.697+0000 7f5a2d3af640 1 -- 172.21.15.31:0/4164028348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 msgr2=0x55d5bf343d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.697+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/4164028348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf343d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.697+0000 7f5a2d3af640 1 -- 172.21.15.31:0/4164028348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 msgr2=0x55d5bf344a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:04.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.697+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/4164028348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf344a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.698+0000 7f5a2d3af640 1 -- 172.21.15.31:0/4164028348 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5bf26cc30 con 0x55d5befa1390 2024-04-25T16:11:04.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.699+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a140afd00 0x7f5a140b2210 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5a18013570 tx=0x7f5a18013970 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:04.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.699+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/4164028348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf343810 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5a18013e70 tx=0x7f5a18020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bd62e5925a19094 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:04.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.713+0000 7f5a137fe640 1 -- 172.21.15.31:0/4164028348 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a180120d0 con 0x55d5befa1390 2024-04-25T16:11:04.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.714+0000 7f5a137fe640 1 -- 172.21.15.31:0/4164028348 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a18024020 con 0x55d5befa1390 2024-04-25T16:11:04.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.716+0000 7f5a3659f740 1 -- 172.21.15.31:0/4164028348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 msgr2=0x55d5bf343810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.716+0000 7f5a3659f740 1 --2- 172.21.15.31:0/4164028348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf343810 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f5a18013e70 tx=0x7f5a18020000 comp rx=0 tx=0).stop 2024-04-25T16:11:04.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.720+0000 7f5a3659f740 1 -- 172.21.15.31:0/4164028348 shutdown_connections 2024-04-25T16:11:04.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.720+0000 7f5a3659f740 1 --2- 172.21.15.31:0/4164028348 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf344a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.720+0000 7f5a3659f740 1 --2- 172.21.15.31:0/4164028348 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf343d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.720+0000 7f5a3659f740 1 --2- 172.21.15.31:0/4164028348 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf343810 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.720+0000 7f5a3659f740 1 -- 172.21.15.31:0/4164028348 >> 172.21.15.31:0/4164028348 conn(0x55d5bef954e0 msgr2=0x55d5bf33f020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:04.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.725+0000 7f5a3659f740 1 -- 172.21.15.31:0/4164028348 shutdown_connections 2024-04-25T16:11:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.725+0000 7f5a3659f740 1 -- 172.21.15.31:0/4164028348 wait complete. 2024-04-25T16:11:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.726+0000 7f5a3659f740 1 -- start start 2024-04-25T16:11:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.726+0000 7f5a3659f740 1 Processor -- start 2024-04-25T16:11:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.730+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf34d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.730+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf34da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.730+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf3549f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.730+0000 7f5a3659f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d5bf345520 con 0x55d5bef806b0 2024-04-25T16:11:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.730+0000 7f5a3659f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d5bf22edc0 con 0x55d5bef784d0 2024-04-25T16:11:04.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.730+0000 7f5a3659f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d5bf3456a0 con 0x55d5befa1390 2024-04-25T16:11:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.730+0000 7f5a3659f740 10 client.0 _ll_register_callbacks cb 0x55d5bf24a600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.730+0000 7f5a2cbae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf34da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.731+0000 7f5a2cbae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf34da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53376/0 (socket says 172.21.15.31:53376) 2024-04-25T16:11:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.731+0000 7f5a2cbae640 1 -- 172.21.15.31:0/2642596079 learned_addr learned my addr 172.21.15.31:0/2642596079 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:04.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.731+0000 7f5a2dbb0640 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf3549f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.732+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf34d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.732+0000 7f5a2cbae640 1 -- 172.21.15.31:0/2642596079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 msgr2=0x55d5bf3549f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.732+0000 7f5a2cbae640 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf3549f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.732+0000 7f5a2cbae640 1 -- 172.21.15.31:0/2642596079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 msgr2=0x55d5bf34d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.732+0000 7f5a2cbae640 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf34d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.732+0000 7f5a2cbae640 1 -- 172.21.15.31:0/2642596079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5befb93f0 con 0x55d5bef806b0 2024-04-25T16:11:04.738 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.736+0000 7f5a2cbae640 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf34da90 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5a240116f0 tx=0x7f5a24011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fafa2b6c9097286 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:04.741 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.738+0000 7f5a10ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:04.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.743+0000 7f5a10ff9640 1 -- 172.21.15.31:0/2642596079 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a24004440 con 0x55d5bef806b0 2024-04-25T16:11:04.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.743+0000 7f5a10ff9640 1 -- 172.21.15.31:0/2642596079 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5a2400f920 con 0x55d5bef806b0 2024-04-25T16:11:04.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.746+0000 7f5a3659f740 1 -- 172.21.15.31:0/2642596079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a18024020 con 0x55d5bef806b0 2024-04-25T16:11:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.754+0000 7f5a10ff9640 1 -- 172.21.15.31:0/2642596079 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5a2400fb20 con 0x55d5bef806b0 2024-04-25T16:11:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.754+0000 7f5a3659f740 10 client.17532 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:04.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.754+0000 7f5a3659f740 1 -- 172.21.15.31:0/2642596079 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5bf0837d0 con 0x55d5bef806b0 2024-04-25T16:11:04.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.756+0000 7f5a10ff9640 1 -- 172.21.15.31:0/2642596079 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a18024020 con 0x55d5bef806b0 2024-04-25T16:11:04.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.756+0000 7f5a10ff9640 1 -- 172.21.15.31:0/2642596079 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5a24018070 con 0x55d5bef806b0 2024-04-25T16:11:04.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.756+0000 7f5a3659f740 10 client.17532 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:04.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.756+0000 7f5a3659f740 10 client.17532 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:11:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.756+0000 7f5a3659f740 20 client.17532 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.756+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d5bef5ff60 0x55d5bf35b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.756+0000 7f5a3659f740 4 client.17532 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:04.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.756+0000 7f5a3659f740 1 -- 172.21.15.31:0/2642596079 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d5bf2b5990 con 0x55d5bef5ff60 2024-04-25T16:11:04.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.761+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d5bef5ff60 0x55d5bf35b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.768+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d5bef5ff60 0x55d5bf35b710 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b71056d5472b232f server_cookie=7a31feba67ced6ff in_seq=0 out_seq=0 2024-04-25T16:11:04.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.770+0000 7f5a10ff9640 10 client.17532 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:11:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.777+0000 7f5a10ff9640 1 -- 172.21.15.31:0/2642596079 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d5bf2b5990 con 0x55d5bef5ff60 2024-04-25T16:11:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.777+0000 7f5a10ff9640 10 client.17532 handle_command_reply: tid=0 2024-04-25T16:11:04.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.777+0000 7f5a3659f740 1 client.17532 shutdown 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a3659f740 1 -- 172.21.15.31:0/2642596079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 msgr2=0x55d5bf34da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf34da90 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f5a240116f0 tx=0x7f5a24011bb0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a3659f740 1 -- 172.21.15.31:0/2642596079 shutdown_connections 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d5bef5ff60 0x55d5bf35b710 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf3549f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf34da90 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2642596079 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf34d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a3659f740 1 -- 172.21.15.31:0/2642596079 >> 172.21.15.31:0/2642596079 conn(0x55d5bef954e0 msgr2=0x55d5bf33e040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a10ff9640 0 client.17532 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.778+0000 7f5a3659f740 1 -- 172.21.15.31:0/2642596079 shutdown_connections 2024-04-25T16:11:04.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.779+0000 7f5a3659f740 1 -- 172.21.15.31:0/2642596079 wait complete. 2024-04-25T16:11:04.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.779+0000 7f5a3659f740 20 client.17532 trim_cache size 0 max 16384 2024-04-25T16:11:04.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.815+0000 7f5a3659f740 1 Processor -- start 2024-04-25T16:11:04.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.815+0000 7f5a3659f740 1 -- start start 2024-04-25T16:11:04.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.816+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf345410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.816+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf354910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.818 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.817+0000 7f5a2d3af640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf345410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.817+0000 7f5a2d3af640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf345410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53400/0 (socket says 172.21.15.31:53400) 2024-04-25T16:11:04.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.817+0000 7f5a2d3af640 1 -- 172.21.15.31:0/2990302132 learned_addr learned my addr 172.21.15.31:0/2990302132 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:04.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.817+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf34bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.817+0000 7f5a3659f740 1 -- 172.21.15.31:0/2990302132 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c00e590 con 0x55d5bef806b0 2024-04-25T16:11:04.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.817+0000 7f5a3659f740 1 -- 172.21.15.31:0/2990302132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c0100d0 con 0x55d5bef784d0 2024-04-25T16:11:04.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.817+0000 7f5a3659f740 1 -- 172.21.15.31:0/2990302132 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c00ec60 con 0x55d5befa1390 2024-04-25T16:11:04.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.817+0000 7f5a2dbb0640 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf34bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.818+0000 7f5a2cbae640 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf354910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.818+0000 7f5a2d3af640 1 -- 172.21.15.31:0/2990302132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 msgr2=0x55d5bf34bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:04.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.818+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf34bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.818+0000 7f5a2d3af640 1 -- 172.21.15.31:0/2990302132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 msgr2=0x55d5bf354910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:04.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.818+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf354910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.818+0000 7f5a2d3af640 1 -- 172.21.15.31:0/2990302132 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d5bf33f980 con 0x55d5bef784d0 2024-04-25T16:11:04.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.818+0000 7f5a2cbae640 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf354910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:11:04.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.818+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf345410 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5a180058c0 tx=0x7f5a18021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9e7718e8c0680c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:04.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.819+0000 7f5a12ffd640 1 -- 172.21.15.31:0/2990302132 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a180109b0 con 0x55d5bef784d0 2024-04-25T16:11:04.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.819+0000 7f5a12ffd640 1 -- 172.21.15.31:0/2990302132 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a18024040 con 0x55d5bef784d0 2024-04-25T16:11:04.823 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.822+0000 7f5a12ffd640 1 -- 172.21.15.31:0/2990302132 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18005610 con 0x55d5bef784d0 2024-04-25T16:11:04.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.833+0000 7f5a3659f740 1 -- 172.21.15.31:0/2990302132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 msgr2=0x55d5bf345410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.835 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.833+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf345410 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f5a180058c0 tx=0x7f5a18021000 comp rx=0 tx=0).stop 2024-04-25T16:11:04.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.834+0000 7f5a3659f740 1 -- 172.21.15.31:0/2990302132 shutdown_connections 2024-04-25T16:11:04.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.834+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5befa1390 0x55d5bf34bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.834+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5bef806b0 0x55d5bf354910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.834+0000 7f5a3659f740 1 --2- 172.21.15.31:0/2990302132 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf345410 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.836 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.834+0000 7f5a3659f740 1 -- 172.21.15.31:0/2990302132 >> 172.21.15.31:0/2990302132 conn(0x55d5bef954e0 msgr2=0x55d5bf348270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:04.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.838+0000 7f5a3659f740 1 -- 172.21.15.31:0/2990302132 shutdown_connections 2024-04-25T16:11:04.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.838+0000 7f5a3659f740 1 -- 172.21.15.31:0/2990302132 wait complete. 2024-04-25T16:11:04.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.839+0000 7f5a3659f740 1 -- start start 2024-04-25T16:11:04.841 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.839+0000 7f5a3659f740 1 Processor -- start 2024-04-25T16:11:04.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.841+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf354470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.841+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bef806b0 0x55d5bf350ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.841+0000 7f5a3659f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5befa1390 0x55d5bf351530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.844 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.841+0000 7f5a3659f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c00fa00 con 0x55d5befa1390 2024-04-25T16:11:04.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.841+0000 7f5a3659f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c00f330 con 0x55d5bef784d0 2024-04-25T16:11:04.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.841+0000 7f5a3659f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0c00c570 con 0x55d5bef806b0 2024-04-25T16:11:04.845 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.842+0000 7f5a3659f740 10 client.0 _ll_register_callbacks cb 0x7f5a24009710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:04.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.845+0000 7f5a2d3af640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf354470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.846+0000 7f5a2d3af640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf354470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53402/0 (socket says 172.21.15.31:53402) 2024-04-25T16:11:04.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.846+0000 7f5a2dbb0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5befa1390 0x55d5bf351530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.847 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.846+0000 7f5a2d3af640 1 -- 172.21.15.31:0/1810523959 learned_addr learned my addr 172.21.15.31:0/1810523959 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:04.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.846+0000 7f5a2d3af640 1 -- 172.21.15.31:0/1810523959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bef806b0 msgr2=0x55d5bf350ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:04.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.846+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bef806b0 0x55d5bf350ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.847+0000 7f5a2d3af640 1 -- 172.21.15.31:0/1810523959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5befa1390 msgr2=0x55d5bf351530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:04.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.847+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5befa1390 0x55d5bf351530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.848 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.847+0000 7f5a2d3af640 1 -- 172.21.15.31:0/1810523959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d5bf354b10 con 0x55d5bef784d0 2024-04-25T16:11:04.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.848+0000 7f5a2d3af640 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf354470 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800f490 tx=0x7f5a180104c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2037eb1ced7a3015 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:04.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.850+0000 7f5a10ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:04.851 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.850+0000 7f5a10ff9640 1 -- 172.21.15.31:0/1810523959 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18005390 con 0x55d5bef784d0 2024-04-25T16:11:04.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.850+0000 7f5a10ff9640 1 -- 172.21.15.31:0/1810523959 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5a18005530 con 0x55d5bef784d0 2024-04-25T16:11:04.852 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.850+0000 7f5a3659f740 1 -- 172.21.15.31:0/1810523959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5a18024040 con 0x55d5bef784d0 2024-04-25T16:11:04.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.852+0000 7f5a10ff9640 1 -- 172.21.15.31:0/1810523959 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5a180109b0 con 0x55d5bef784d0 2024-04-25T16:11:04.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.857+0000 7f5a3659f740 10 client.27218 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:04.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.857+0000 7f5a3659f740 1 -- 172.21.15.31:0/1810523959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d5bf3c2890 con 0x55d5bef784d0 2024-04-25T16:11:04.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.868+0000 7f5a10ff9640 1 -- 172.21.15.31:0/1810523959 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5a18024040 con 0x55d5bef784d0 2024-04-25T16:11:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.868+0000 7f5a10ff9640 1 -- 172.21.15.31:0/1810523959 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5a1801f070 con 0x55d5bef784d0 2024-04-25T16:11:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.874+0000 7f5a3659f740 10 client.27218 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.874+0000 7f5a3659f740 10 client.27218 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:11:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.874+0000 7f5a3659f740 20 client.27218 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.874+0000 7f5a3659f740 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d5bef353a0 0x55d5bf3c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.874+0000 7f5a3659f740 4 client.27218 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:04.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.874+0000 7f5a3659f740 1 -- 172.21.15.31:0/1810523959 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d5bf2af800 con 0x55d5bef353a0 2024-04-25T16:11:04.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.885+0000 7f5a2cbae640 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d5bef353a0 0x55d5bf3c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:04.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.890+0000 7f5a2cbae640 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d5bef353a0 0x55d5bf3c4f10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7edbab19140f4c3e server_cookie=c0f7c1b98fc3428c in_seq=0 out_seq=0 2024-04-25T16:11:04.891 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.890+0000 7f5a10ff9640 10 client.27218 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:11:04.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.902+0000 7f5a10ff9640 1 -- 172.21.15.31:0/1810523959 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d5bf2af800 con 0x55d5bef353a0 2024-04-25T16:11:04.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.902+0000 7f5a10ff9640 10 client.27218 handle_command_reply: tid=0 2024-04-25T16:11:04.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.902+0000 7f5a3659f740 1 client.27218 shutdown 2024-04-25T16:11:04.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 -- 172.21.15.31:0/1810523959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 msgr2=0x55d5bf354470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf354470 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800f490 tx=0x7f5a180104c0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 -- 172.21.15.31:0/1810523959 shutdown_connections 2024-04-25T16:11:04.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55d5bef353a0 0x55d5bf3c4f10 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d5befa1390 0x55d5bf351530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d5bef806b0 0x55d5bf350ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 --2- 172.21.15.31:0/1810523959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d5bef784d0 0x55d5bf354470 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 -- 172.21.15.31:0/1810523959 >> 172.21.15.31:0/1810523959 conn(0x55d5bef954e0 msgr2=0x55d5bf348270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a10ff9640 0 client.27218 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:11:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 -- 172.21.15.31:0/1810523959 shutdown_connections 2024-04-25T16:11:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.903+0000 7f5a3659f740 1 -- 172.21.15.31:0/1810523959 wait complete. 2024-04-25T16:11:04.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.905+0000 7f5a3659f740 20 client.27218 trim_cache size 0 max 16384 2024-04-25T16:11:04.909 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 -- 172.21.15.31:0/1752617676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a140afd00 msgr2=0x7f5a140b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a140afd00 0x7f5a140b2210 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5a18013570 tx=0x7f5a18013970 comp rx=0 tx=0).stop 2024-04-25T16:11:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 -- 172.21.15.31:0/1752617676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a28110030 msgr2=0x7f5a281b0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a28110030 0x7f5a281b0570 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f5a20034040 tx=0x7f5a20030ce0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 -- 172.21.15.31:0/1752617676 shutdown_connections 2024-04-25T16:11:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a140afd00 0x7f5a140b2210 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a28110030 0x7f5a281b0570 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5a2810b310 0x7f5a281a95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 --2- 172.21.15.31:0/1752617676 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5a2810a420 0x7f5a281a90a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:04.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 -- 172.21.15.31:0/1752617676 >> 172.21.15.31:0/1752617676 conn(0x7f5a280704e0 msgr2=0x7f5a2810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:04.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 -- 172.21.15.31:0/1752617676 shutdown_connections 2024-04-25T16:11:04.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:04.911+0000 7f5a2f613640 1 -- 172.21.15.31:0/1752617676 wait complete. 2024-04-25T16:11:04.979 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:11:05.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.402+0000 7f33effff640 1 -- 172.21.15.31:0/1386286260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f010b3f0 msgr2=0x7f33f0109ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:05.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.402+0000 7f33edffb640 1 -- 172.21.15.31:0/1386286260 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e803a070 con 0x7f33f010b3f0 2024-04-25T16:11:05.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.402+0000 7f33effff640 1 --2- 172.21.15.31:0/1386286260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f010b3f0 0x7f33f0109ba0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f33e800bc00 tx=0x7f33e8009260 comp rx=0 tx=0).stop 2024-04-25T16:11:05.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.402+0000 7f33effff640 1 -- 172.21.15.31:0/1386286260 shutdown_connections 2024-04-25T16:11:05.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.402+0000 7f33effff640 1 --2- 172.21.15.31:0/1386286260 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33f0078da0 0x7f33f007b1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.402+0000 7f33effff640 1 --2- 172.21.15.31:0/1386286260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33f010a0e0 0x7f33f010a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.402+0000 7f33effff640 1 --2- 172.21.15.31:0/1386286260 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f010b3f0 0x7f33f0109ba0 secure :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f33e800bc00 tx=0x7f33e8009260 comp rx=0 tx=0).stop 2024-04-25T16:11:05.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.402+0000 7f33effff640 1 -- 172.21.15.31:0/1386286260 >> 172.21.15.31:0/1386286260 conn(0x7f33f0070320 msgr2=0x7f33f00727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:05.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.403+0000 7f33effff640 1 -- 172.21.15.31:0/1386286260 shutdown_connections 2024-04-25T16:11:05.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.403+0000 7f33effff640 1 -- 172.21.15.31:0/1386286260 wait complete. 2024-04-25T16:11:05.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.404+0000 7f33effff640 1 Processor -- start 2024-04-25T16:11:05.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.405+0000 7f33effff640 1 -- start start 2024-04-25T16:11:05.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.406+0000 7f33effff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0078da0 0x7f33f0084990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.406+0000 7f33effff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33f010a0e0 0x7f33f0084f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.406+0000 7f33effff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33f01b6bb0 0x7f33f01b8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.406+0000 7f33effff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33f007ead0 con 0x7f33f01b6bb0 2024-04-25T16:11:05.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.406+0000 7f33effff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f33f007e950 con 0x7f33f0078da0 2024-04-25T16:11:05.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.406+0000 7f33effff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33f007ec50 con 0x7f33f010a0e0 2024-04-25T16:11:05.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.407+0000 7f33ee7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33f010a0e0 0x7f33f0084f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.407+0000 7f33ee7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33f010a0e0 0x7f33f0084f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34156/0 (socket says 172.21.15.31:34156) 2024-04-25T16:11:05.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.407+0000 7f33ee7fc640 1 -- 172.21.15.31:0/872777731 learned_addr learned my addr 172.21.15.31:0/872777731 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:05.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.407+0000 7f33eeffd640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0078da0 0x7f33f0084990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.409+0000 7f33eeffd640 1 -- 172.21.15.31:0/872777731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33f010a0e0 msgr2=0x7f33f0084f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:05.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.409+0000 7f33eeffd640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33f010a0e0 0x7f33f0084f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.409+0000 7f33eeffd640 1 -- 172.21.15.31:0/872777731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33f01b6bb0 msgr2=0x7f33f01b8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:05.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.409+0000 7f33eeffd640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33f01b6bb0 0x7f33f01b8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.409+0000 7f33eeffd640 1 -- 172.21.15.31:0/872777731 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33f01b9750 con 0x7f33f0078da0 2024-04-25T16:11:05.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.414+0000 7f33eeffd640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0078da0 0x7f33f0084990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f33f00748c0 tx=0x7f33e8037d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:05.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.417+0000 7f33f492c640 1 -- 172.21.15.31:0/872777731 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e804e070 con 0x7f33f0078da0 2024-04-25T16:11:05.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.418+0000 7f33effff640 1 -- 172.21.15.31:0/872777731 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33f01b99e0 con 0x7f33f0078da0 2024-04-25T16:11:05.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.418+0000 7f33effff640 1 -- 172.21.15.31:0/872777731 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33f01b9f70 con 0x7f33f0078da0 2024-04-25T16:11:05.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.418+0000 7f33f492c640 1 -- 172.21.15.31:0/872777731 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e800acb0 con 0x7f33f0078da0 2024-04-25T16:11:05.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.420+0000 7f33f492c640 1 -- 172.21.15.31:0/872777731 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e804e070 con 0x7f33f0078da0 2024-04-25T16:11:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.431+0000 7f33fc7da740 1 Processor -- start 2024-04-25T16:11:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.431+0000 7f33fc7da740 1 -- start start 2024-04-25T16:11:05.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.431+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa3d390 0x55cd3addf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.431+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa144d0 0x55cd3addff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.431+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.431+0000 7f33fc7da740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cd3ad43d80 con 0x55cd3aa144d0 2024-04-25T16:11:05.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.431+0000 7f33fc7da740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cd3acfda20 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.431+0000 7f33fc7da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cd3ad3fd40 con 0x55cd3aa3d390 2024-04-25T16:11:05.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ef7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa3d390 0x55cd3addf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ef7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa3d390 0x55cd3addf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34170/0 (socket says 172.21.15.31:34170) 2024-04-25T16:11:05.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ef7fe640 1 -- 172.21.15.31:0/3860830919 learned_addr learned my addr 172.21.15.31:0/3860830919 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:05.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ee7fc640 1 --2- 172.21.15.31:0/3860830919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ef7fe640 1 -- 172.21.15.31:0/3860830919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa1c6b0 msgr2=0x55cd3ade0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/3860830919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ef7fe640 1 -- 172.21.15.31:0/3860830919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa144d0 msgr2=0x55cd3addff60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/3860830919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa144d0 0x55cd3addff60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ef7fe640 1 -- 172.21.15.31:0/3860830919 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd3ad08c30 con 0x55cd3aa3d390 2024-04-25T16:11:05.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.432+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/3860830919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa3d390 0x55cd3addf9e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f33e000a6b0 tx=0x7f33e00071c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=866f5c81d889575f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:05.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.433+0000 7f33f492c640 1 -- 172.21.15.31:0/872777731 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f33e803d070 con 0x7f33f0078da0 2024-04-25T16:11:05.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.435+0000 7f33f492c640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f33d80aff20 0x7f33d80b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.436+0000 7f33f492c640 1 -- 172.21.15.31:0/872777731 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f33e8045030 con 0x7f33f0078da0 2024-04-25T16:11:05.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.436+0000 7f33d5ffb640 1 -- 172.21.15.31:0/3860830919 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e0007d80 con 0x55cd3aa3d390 2024-04-25T16:11:05.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.437+0000 7f33d5ffb640 1 -- 172.21.15.31:0/3860830919 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e0013070 con 0x55cd3aa3d390 2024-04-25T16:11:05.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.438+0000 7f33fc7da740 1 -- 172.21.15.31:0/3860830919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa3d390 msgr2=0x55cd3addf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:05.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.438+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3860830919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa3d390 0x55cd3addf9e0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f33e000a6b0 tx=0x7f33e00071c0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.441+0000 7f33eeffd640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f33d80aff20 0x7f33d80b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.446+0000 7f33fc7da740 1 -- 172.21.15.31:0/3860830919 shutdown_connections 2024-04-25T16:11:05.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.446+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3860830919 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.446+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3860830919 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa144d0 0x55cd3addff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.446+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3860830919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa3d390 0x55cd3addf9e0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.446+0000 7f33fc7da740 1 -- 172.21.15.31:0/3860830919 >> 172.21.15.31:0/3860830919 conn(0x55cd3aa314e0 msgr2=0x55cd3addb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:05.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.447+0000 7f33fc7da740 1 -- 172.21.15.31:0/3860830919 shutdown_connections 2024-04-25T16:11:05.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:05 smithi031 ceph-mon[29140]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 895 MiB/s wr, 1.85k op/s 2024-04-25T16:11:05.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.447+0000 7f33fc7da740 1 -- 172.21.15.31:0/3860830919 wait complete. 2024-04-25T16:11:05.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.448+0000 7f33fc7da740 1 -- start start 2024-04-25T16:11:05.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.448+0000 7f33fc7da740 1 Processor -- start 2024-04-25T16:11:05.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.450+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa144d0 0x55cd3ade9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.450+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade9b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.450+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3a9fbf60 0x55cd3adf0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.451+0000 7f33ee7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.451+0000 7f33ee7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34180/0 (socket says 172.21.15.31:34180) 2024-04-25T16:11:05.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.451+0000 7f33fc7da740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55cd3ade1750 con 0x55cd3aa144d0 2024-04-25T16:11:05.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.451+0000 7f33fc7da740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55cd3accadc0 con 0x55cd3a9fbf60 2024-04-25T16:11:05.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.451+0000 7f33fc7da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55cd3ade18d0 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.451+0000 7f33fc7da740 10 client.0 _ll_register_callbacks cb 0x55cd3ace6600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:05.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.451+0000 7f33ee7fc640 1 -- 172.21.15.31:0/3154911055 learned_addr learned my addr 172.21.15.31:0/3154911055 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:05.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.451+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3a9fbf60 0x55cd3adf0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.452+0000 7f33ef7fe640 1 -- 172.21.15.31:0/3154911055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 msgr2=0x55cd3ade9b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:05.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.452+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade9b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.452+0000 7f33ef7fe640 1 -- 172.21.15.31:0/3154911055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa144d0 msgr2=0x55cd3ade9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:05.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.452+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa144d0 0x55cd3ade9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.452+0000 7f33ef7fe640 1 -- 172.21.15.31:0/3154911055 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd3aa553f0 con 0x55cd3a9fbf60 2024-04-25T16:11:05.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.452+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3a9fbf60 0x55cd3adf0ac0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f33e000c010 tx=0x7f33e00169a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f057191e0b0b6556 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:05.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.453+0000 7f33ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:05.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.453+0000 7f33ceffd640 1 -- 172.21.15.31:0/3154911055 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e0024020 con 0x55cd3a9fbf60 2024-04-25T16:11:05.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.453+0000 7f33ceffd640 1 -- 172.21.15.31:0/3154911055 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f33e0017620 con 0x55cd3a9fbf60 2024-04-25T16:11:05.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.453+0000 7f33fc7da740 1 -- 172.21.15.31:0/3154911055 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cd3ad51990 con 0x55cd3a9fbf60 2024-04-25T16:11:05.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.457+0000 7f33ceffd640 1 -- 172.21.15.31:0/3154911055 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f33e0017820 con 0x55cd3a9fbf60 2024-04-25T16:11:05.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.457+0000 7f33eeffd640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f33d80aff20 0x7f33d80b2430 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f33e80cbe60 tx=0x7f33e80d40f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:05.461 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.460+0000 7f33fc7da740 10 client.27227 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:05.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.460+0000 7f33fc7da740 1 -- 172.21.15.31:0/3154911055 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cd3ab1f7d0 con 0x55cd3a9fbf60 2024-04-25T16:11:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.462+0000 7f33ceffd640 1 -- 172.21.15.31:0/3154911055 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cd3ad51990 con 0x55cd3a9fbf60 2024-04-25T16:11:05.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.462+0000 7f33ceffd640 1 -- 172.21.15.31:0/3154911055 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f33e0020ac0 con 0x55cd3a9fbf60 2024-04-25T16:11:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.464+0000 7f33fc7da740 10 client.27227 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.464+0000 7f33fc7da740 10 client.27227 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:11:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.464+0000 7f33fc7da740 20 client.27227 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.464+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55cd3a9d13a0 0x55cd3adf7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.464+0000 7f33fc7da740 4 client.27227 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:05.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.464+0000 7f33fc7da740 1 -- 172.21.15.31:0/3154911055 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd3ad4b800 con 0x55cd3a9d13a0 2024-04-25T16:11:05.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.468+0000 7f33ee7fc640 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55cd3a9d13a0 0x55cd3adf7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.498+0000 7f33ee7fc640 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55cd3a9d13a0 0x55cd3adf7060 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=735e39bd90ca67dc server_cookie=51a32769165579b0 in_seq=0 out_seq=0 2024-04-25T16:11:05.504 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.503+0000 7f33ceffd640 10 client.27227 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:11:05.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.503+0000 7f33ceffd640 1 -- 172.21.15.31:0/3154911055 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55cd3ad4b800 con 0x55cd3a9d13a0 2024-04-25T16:11:05.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.503+0000 7f33ceffd640 10 client.27227 handle_command_reply: tid=0 2024-04-25T16:11:05.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.505+0000 7f33fc7da740 1 client.27227 shutdown 2024-04-25T16:11:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.508+0000 7f33fc7da740 1 -- 172.21.15.31:0/3154911055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3a9fbf60 msgr2=0x55cd3adf0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:05.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.508+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3a9fbf60 0x55cd3adf0ac0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f33e000c010 tx=0x7f33e00169a0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.510+0000 7f33fc7da740 1 -- 172.21.15.31:0/3154911055 shutdown_connections 2024-04-25T16:11:05.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.510+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3a9fbf60 0x55cd3adf0ac0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.510+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55cd3a9d13a0 0x55cd3adf7060 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.510+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade9b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.510+0000 7f33fc7da740 1 --2- 172.21.15.31:0/3154911055 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa144d0 0x55cd3ade9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.510+0000 7f33fc7da740 1 -- 172.21.15.31:0/3154911055 >> 172.21.15.31:0/3154911055 conn(0x55cd3aa314e0 msgr2=0x55cd3adda3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:05.512 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.511+0000 7f33ceffd640 0 client.27227 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:11:05.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.519+0000 7f33fc7da740 1 -- 172.21.15.31:0/3154911055 shutdown_connections 2024-04-25T16:11:05.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.520+0000 7f33fc7da740 1 -- 172.21.15.31:0/3154911055 wait complete. 2024-04-25T16:11:05.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.521+0000 7f33fc7da740 20 client.27227 trim_cache size 0 max 16384 2024-04-25T16:11:05.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.565+0000 7f33fc7da740 1 Processor -- start 2024-04-25T16:11:05.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.566+0000 7f33fc7da740 1 -- start start 2024-04-25T16:11:05.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.566+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3a9fbf60 0x55cd3ade0a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.567+0000 7f33ef7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3a9fbf60 0x55cd3ade0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.567+0000 7f33ef7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3a9fbf60 0x55cd3ade0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53428/0 (socket says 172.21.15.31:53428) 2024-04-25T16:11:05.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.567+0000 7f33ef7fe640 1 -- 172.21.15.31:0/1500525316 learned_addr learned my addr 172.21.15.31:0/1500525316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:05.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.567+0000 7f33fc7da740 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa144d0 0x55cd3adf09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.567+0000 7f33fc7da740 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.568+0000 7f33fc7da740 1 -- 172.21.15.31:0/1500525316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc00e590 con 0x55cd3a9fbf60 2024-04-25T16:11:05.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.568+0000 7f33fc7da740 1 -- 172.21.15.31:0/1500525316 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc0100d0 con 0x55cd3aa144d0 2024-04-25T16:11:05.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.568+0000 7f33fc7da740 1 -- 172.21.15.31:0/1500525316 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc00ec60 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.568+0000 7f33ee7fc640 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa144d0 0x55cd3adf09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.569+0000 7f33ef7fe640 1 -- 172.21.15.31:0/1500525316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 msgr2=0x55cd3ade4280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:05.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.569+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade4280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.569+0000 7f33ef7fe640 1 -- 172.21.15.31:0/1500525316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa144d0 msgr2=0x55cd3adf09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:05.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.569+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa144d0 0x55cd3adf09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.569+0000 7f33ef7fe640 1 -- 172.21.15.31:0/1500525316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd3adccba0 con 0x55cd3a9fbf60 2024-04-25T16:11:05.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.569+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3a9fbf60 0x55cd3ade0a30 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f33e0005f30 tx=0x7f33e000ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce8cb1a7b3aab0f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:05.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.572+0000 7f33d57fa640 1 -- 172.21.15.31:0/1500525316 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e0024070 con 0x55cd3a9fbf60 2024-04-25T16:11:05.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.572+0000 7f33d57fa640 1 -- 172.21.15.31:0/1500525316 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33e0013030 con 0x55cd3a9fbf60 2024-04-25T16:11:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.574+0000 7f33fc7da740 1 -- 172.21.15.31:0/1500525316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3a9fbf60 msgr2=0x55cd3ade0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.574+0000 7f33fc7da740 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3a9fbf60 0x55cd3ade0a30 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f33e0005f30 tx=0x7f33e000ac00 comp rx=0 tx=0).stop 2024-04-25T16:11:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.574+0000 7f33fc7da740 1 -- 172.21.15.31:0/1500525316 shutdown_connections 2024-04-25T16:11:05.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.574+0000 7f33fc7da740 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa1c6b0 0x55cd3ade4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.574+0000 7f33fc7da740 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3aa144d0 0x55cd3adf09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.574+0000 7f33fc7da740 1 --2- 172.21.15.31:0/1500525316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3a9fbf60 0x55cd3ade0a30 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.574+0000 7f33fc7da740 1 -- 172.21.15.31:0/1500525316 >> 172.21.15.31:0/1500525316 conn(0x55cd3aa314e0 msgr2=0x55cd3ade9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:05.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.577+0000 7f33fc7da740 1 -- 172.21.15.31:0/1500525316 shutdown_connections 2024-04-25T16:11:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.577+0000 7f33fc7da740 1 -- 172.21.15.31:0/1500525316 wait complete. 2024-04-25T16:11:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.577+0000 7f33fc7da740 1 -- start start 2024-04-25T16:11:05.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.578+0000 7f33fc7da740 1 Processor -- start 2024-04-25T16:11:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.578+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa144d0 0x55cd3adf09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.578+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa1c6b0 0x55cd3aded570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.578+0000 7f33fc7da740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3adedab0 0x55cd3ae5d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.578+0000 7f33fc7da740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc00fa00 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.578+0000 7f33fc7da740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc00f330 con 0x55cd3adedab0 2024-04-25T16:11:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.578+0000 7f33fc7da740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f33dc00c570 con 0x55cd3aa144d0 2024-04-25T16:11:05.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.578+0000 7f33fc7da740 10 client.0 _ll_register_callbacks cb 0x7f33e0004600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.580+0000 7f33ee7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa1c6b0 0x55cd3aded570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.580+0000 7f33ee7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa1c6b0 0x55cd3aded570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53434/0 (socket says 172.21.15.31:53434) 2024-04-25T16:11:05.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.581+0000 7f33ee7fc640 1 -- 172.21.15.31:0/2367340286 learned_addr learned my addr 172.21.15.31:0/2367340286 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.582+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa144d0 0x55cd3adf09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.582+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3adedab0 0x55cd3ae5d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.582+0000 7f33ee7fc640 1 -- 172.21.15.31:0/2367340286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa144d0 msgr2=0x55cd3adf09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.582+0000 7f33ee7fc640 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa144d0 0x55cd3adf09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.582+0000 7f33ee7fc640 1 -- 172.21.15.31:0/2367340286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3adedab0 msgr2=0x55cd3ae5d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.582+0000 7f33ee7fc640 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3adedab0 0x55cd3ae5d260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.582+0000 7f33ee7fc640 1 -- 172.21.15.31:0/2367340286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd3ae5d7a0 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.582+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa144d0 0x55cd3adf09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:11:05.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.592+0000 7f33ee7fc640 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa1c6b0 0x55cd3aded570 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f33e4006a20 tx=0x7f33e4009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54fab9d3dd92910c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:05.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.601+0000 7f33ceffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.610+0000 7f33ceffd640 1 -- 172.21.15.31:0/2367340286 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33e400f040 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.610+0000 7f33ceffd640 1 -- 172.21.15.31:0/2367340286 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f33e4027b70 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.610+0000 7f33fc7da740 1 -- 172.21.15.31:0/2367340286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55cd3acc6610 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.614+0000 7f33ceffd640 1 -- 172.21.15.31:0/2367340286 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f33e402eac0 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.615+0000 7f33fc7da740 10 client.17544 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:05.616 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.615+0000 7f33fc7da740 1 -- 172.21.15.31:0/2367340286 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55cd3ae5ea50 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:05 smithi063 ceph-mon[32413]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 895 MiB/s wr, 1.85k op/s 2024-04-25T16:11:05.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.623+0000 7f33ceffd640 1 -- 172.21.15.31:0/2367340286 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55cd3acc6610 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.624 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.623+0000 7f33ceffd640 1 -- 172.21.15.31:0/2367340286 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f33e4035070 con 0x55cd3aa1c6b0 2024-04-25T16:11:05.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.624+0000 7f33fc7da740 10 client.17544 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:05.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.624+0000 7f33fc7da740 10 client.17544 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:11:05.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.624+0000 7f33fc7da740 20 client.17544 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:05.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.624+0000 7f33fc7da740 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55cd3ae5f3e0 0x55cd3ae61800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:05.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.624+0000 7f33fc7da740 4 client.17544 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:05.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.624+0000 7f33fc7da740 1 -- 172.21.15.31:0/2367340286 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd3ad14240 con 0x55cd3ae5f3e0 2024-04-25T16:11:05.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.626+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55cd3ae5f3e0 0x55cd3ae61800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.646+0000 7f33ceffd640 10 client.17544 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:11:05.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.646+0000 7f33ef7fe640 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55cd3ae5f3e0 0x55cd3ae61800 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=515ebbc47f792e5d server_cookie=3900e3c4eed063b7 in_seq=0 out_seq=0 2024-04-25T16:11:05.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:05 smithi186 ceph-mon[33258]: pgmap v644: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 895 MiB/s wr, 1.85k op/s 2024-04-25T16:11:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.650+0000 7f33ceffd640 1 -- 172.21.15.31:0/2367340286 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd3ad14240 con 0x55cd3ae5f3e0 2024-04-25T16:11:05.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.650+0000 7f33ceffd640 10 client.17544 handle_command_reply: tid=0 2024-04-25T16:11:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.651+0000 7f33fc7da740 1 client.17544 shutdown 2024-04-25T16:11:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.651+0000 7f33fc7da740 1 -- 172.21.15.31:0/2367340286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa1c6b0 msgr2=0x55cd3aded570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:05.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.651+0000 7f33fc7da740 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa1c6b0 0x55cd3aded570 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f33e4006a20 tx=0x7f33e4009190 comp rx=0 tx=0).stop 2024-04-25T16:11:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.655+0000 7f33fc7da740 1 -- 172.21.15.31:0/2367340286 shutdown_connections 2024-04-25T16:11:05.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.655+0000 7f33fc7da740 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55cd3adedab0 0x55cd3ae5d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.655+0000 7f33fc7da740 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55cd3aa1c6b0 0x55cd3aded570 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.655+0000 7f33fc7da740 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55cd3ae5f3e0 0x55cd3ae61800 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.655+0000 7f33fc7da740 1 --2- 172.21.15.31:0/2367340286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55cd3aa144d0 0x55cd3adf09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.655+0000 7f33fc7da740 1 -- 172.21.15.31:0/2367340286 >> 172.21.15.31:0/2367340286 conn(0x55cd3aa314e0 msgr2=0x55cd3ade95a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.656+0000 7f33ceffd640 0 client.17544 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:11:05.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.656+0000 7f33fc7da740 1 -- 172.21.15.31:0/2367340286 shutdown_connections 2024-04-25T16:11:05.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.656+0000 7f33fc7da740 1 -- 172.21.15.31:0/2367340286 wait complete. 2024-04-25T16:11:05.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.657+0000 7f33fc7da740 20 client.17544 trim_cache size 0 max 16384 2024-04-25T16:11:05.659 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:05.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.663+0000 7f33effff640 1 -- 172.21.15.31:0/872777731 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f33d80aff20 msgr2=0x7f33d80b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:05.664 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.663+0000 7f33effff640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f33d80aff20 0x7f33d80b2430 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f33e80cbe60 tx=0x7f33e80d40f0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.663+0000 7f33effff640 1 -- 172.21.15.31:0/872777731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0078da0 msgr2=0x7f33f0084990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:05.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.663+0000 7f33effff640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0078da0 0x7f33f0084990 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f33f00748c0 tx=0x7f33e8037d40 comp rx=0 tx=0).stop 2024-04-25T16:11:05.665 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.664+0000 7f33effff640 1 -- 172.21.15.31:0/872777731 shutdown_connections 2024-04-25T16:11:05.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.664+0000 7f33effff640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f33d80aff20 0x7f33d80b2430 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.664+0000 7f33effff640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f33f01b6bb0 0x7f33f01b8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.664+0000 7f33effff640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33f010a0e0 0x7f33f0084f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.664+0000 7f33effff640 1 --2- 172.21.15.31:0/872777731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0078da0 0x7f33f0084990 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:05.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.664+0000 7f33effff640 1 -- 172.21.15.31:0/872777731 >> 172.21.15.31:0/872777731 conn(0x7f33f0070320 msgr2=0x7f33f0074d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:05.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.665+0000 7f33effff640 1 -- 172.21.15.31:0/872777731 shutdown_connections 2024-04-25T16:11:05.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:05.665+0000 7f33effff640 1 -- 172.21.15.31:0/872777731 wait complete. 2024-04-25T16:11:05.736 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:11:06.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.275+0000 7f13aa848640 1 -- 172.21.15.31:0/793029031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4103910 msgr2=0x7f13a4104610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.275+0000 7f13aa848640 1 --2- 172.21.15.31:0/793029031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4103910 0x7f13a4104610 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f139c009ca0 tx=0x7f139c02a8f0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.279+0000 7f13aa848640 1 -- 172.21.15.31:0/793029031 shutdown_connections 2024-04-25T16:11:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.279+0000 7f13aa848640 1 --2- 172.21.15.31:0/793029031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a4109850 0x7f13a410bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.279+0000 7f13aa848640 1 --2- 172.21.15.31:0/793029031 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13a4104b50 0x7f13a4104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.279+0000 7f13aa848640 1 --2- 172.21.15.31:0/793029031 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4103910 0x7f13a4104610 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.279+0000 7f13aa848640 1 -- 172.21.15.31:0/793029031 >> 172.21.15.31:0/793029031 conn(0x7f13a40feb20 msgr2=0x7f13a4100fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:06.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.281+0000 7f13aa848640 1 -- 172.21.15.31:0/793029031 shutdown_connections 2024-04-25T16:11:06.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.281+0000 7f13aa848640 1 -- 172.21.15.31:0/793029031 wait complete. 2024-04-25T16:11:06.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.283+0000 7f13aa848640 1 Processor -- start 2024-04-25T16:11:06.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.283+0000 7f13aa848640 1 -- start start 2024-04-25T16:11:06.286 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.285+0000 7f13aa848640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a4103910 0x7f13a419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.289 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.287+0000 7f13aa848640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4104b50 0x7f13a419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.287+0000 7f13aa848640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13a4109850 0x7f13a41a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.287+0000 7f13a9846640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a4103910 0x7f13a419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.287+0000 7f13a9846640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a4103910 0x7f13a419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34206/0 (socket says 172.21.15.31:34206) 2024-04-25T16:11:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.287+0000 7f13a9846640 1 -- 172.21.15.31:0/310719758 learned_addr learned my addr 172.21.15.31:0/310719758 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.288+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f13a410f580 con 0x7f13a4104b50 2024-04-25T16:11:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.288+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f13a410f400 con 0x7f13a4109850 2024-04-25T16:11:06.290 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.288+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f13a410f700 con 0x7f13a4103910 2024-04-25T16:11:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.288+0000 7f13aa047640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13a4109850 0x7f13a41a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.291 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.289+0000 7f13a9045640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4104b50 0x7f13a419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.292+0000 7f13aa047640 1 -- 172.21.15.31:0/310719758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a4103910 msgr2=0x7f13a419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.292+0000 7f13aa047640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a4103910 0x7f13a419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.292+0000 7f13aa047640 1 -- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4104b50 msgr2=0x7f13a419a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.294 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.292+0000 7f13aa047640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4104b50 0x7f13a419a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.293+0000 7f13aa047640 1 -- 172.21.15.31:0/310719758 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13a41a1d10 con 0x7f13a4109850 2024-04-25T16:11:06.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.295+0000 7f13aa047640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13a4109850 0x7f13a41a16a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f139400b3e0 tx=0x7f139400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.296+0000 7f139affd640 1 -- 172.21.15.31:0/310719758 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394015020 con 0x7f13a4109850 2024-04-25T16:11:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.297+0000 7f139affd640 1 -- 172.21.15.31:0/310719758 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1394011ce0 con 0x7f13a4109850 2024-04-25T16:11:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.297+0000 7f139affd640 1 -- 172.21.15.31:0/310719758 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1394010de0 con 0x7f13a4109850 2024-04-25T16:11:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.297+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13a4074e30 con 0x7f13a4109850 2024-04-25T16:11:06.298 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.297+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13a40753b0 con 0x7f13a4109850 2024-04-25T16:11:06.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.302+0000 7f13b2a33740 1 Processor -- start 2024-04-25T16:11:06.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.306+0000 7f13b2a33740 1 -- start start 2024-04-25T16:11:06.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.306+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c61390 0x557139003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.306+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c384d0 0x557139003f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.306+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 0x557139004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.306+0000 7f13b2a33740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557138f67d80 con 0x557138c406b0 2024-04-25T16:11:06.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.306+0000 7f13b2a33740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557138f21a20 con 0x557138c384d0 2024-04-25T16:11:06.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.306+0000 7f13b2a33740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557138f63d40 con 0x557138c61390 2024-04-25T16:11:06.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.307+0000 7f13a9846640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c384d0 0x557139003f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.307+0000 7f13a9846640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c384d0 0x557139003f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53484/0 (socket says 172.21.15.31:53484) 2024-04-25T16:11:06.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.307+0000 7f13a9846640 1 -- 172.21.15.31:0/2144028438 learned_addr learned my addr 172.21.15.31:0/2144028438 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:06.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.308+0000 7f13a9045640 1 --2- 172.21.15.31:0/2144028438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c61390 0x557139003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.308+0000 7f13a9846640 1 -- 172.21.15.31:0/2144028438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c61390 msgr2=0x557139003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.308+0000 7f13a9846640 1 --2- 172.21.15.31:0/2144028438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c61390 0x557139003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.308+0000 7f13a9846640 1 -- 172.21.15.31:0/2144028438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 msgr2=0x557139004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.310 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.308+0000 7f13a9846640 1 --2- 172.21.15.31:0/2144028438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 0x557139004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.308+0000 7f13a9846640 1 -- 172.21.15.31:0/2144028438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557138f2cc30 con 0x557138c384d0 2024-04-25T16:11:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.309+0000 7f13a9846640 1 --2- 172.21.15.31:0/2144028438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c384d0 0x557139003f80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f139c02f570 tx=0x7f139c009990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1775ce2c10c7a61 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.310+0000 7f138ffff640 1 -- 172.21.15.31:0/2144028438 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0317c0 con 0x557138c384d0 2024-04-25T16:11:06.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.313+0000 7f138ffff640 1 -- 172.21.15.31:0/2144028438 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139c036030 con 0x557138c384d0 2024-04-25T16:11:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.315+0000 7f13b2a33740 1 -- 172.21.15.31:0/2144028438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c384d0 msgr2=0x557139003f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.315+0000 7f13b2a33740 1 --2- 172.21.15.31:0/2144028438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c384d0 0x557139003f80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f139c02f570 tx=0x7f139c009990 comp rx=0 tx=0).stop 2024-04-25T16:11:06.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.315+0000 7f139affd640 1 -- 172.21.15.31:0/310719758 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1394018020 con 0x7f13a4109850 2024-04-25T16:11:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.321+0000 7f13b2a33740 1 -- 172.21.15.31:0/2144028438 shutdown_connections 2024-04-25T16:11:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.321+0000 7f13b2a33740 1 --2- 172.21.15.31:0/2144028438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 0x557139004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.321+0000 7f13b2a33740 1 --2- 172.21.15.31:0/2144028438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c384d0 0x557139003f80 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.321+0000 7f13b2a33740 1 --2- 172.21.15.31:0/2144028438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c61390 0x557139003a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.321+0000 7f13b2a33740 1 -- 172.21.15.31:0/2144028438 >> 172.21.15.31:0/2144028438 conn(0x557138c554e0 msgr2=0x557138fff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:06.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.321+0000 7f13b2a33740 1 -- 172.21.15.31:0/2144028438 shutdown_connections 2024-04-25T16:11:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.321+0000 7f13b2a33740 1 -- 172.21.15.31:0/2144028438 wait complete. 2024-04-25T16:11:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.322+0000 7f13b2a33740 1 -- start start 2024-04-25T16:11:06.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.322+0000 7f13b2a33740 1 Processor -- start 2024-04-25T16:11:06.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.323+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c384d0 0x55713900d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.323+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 0x55713900dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.323+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139014b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.323+0000 7f13b2a33740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557139005760 con 0x557138c406b0 2024-04-25T16:11:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.323+0000 7f13b2a33740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557138eeedc0 con 0x557138c61390 2024-04-25T16:11:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.323+0000 7f13b2a33740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5571390058e0 con 0x557138c384d0 2024-04-25T16:11:06.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.323+0000 7f13b2a33740 10 client.0 _ll_register_callbacks cb 0x557138f0a600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:06.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.324+0000 7f13a9045640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c384d0 0x55713900d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.324+0000 7f13a9045640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c384d0 0x55713900d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34224/0 (socket says 172.21.15.31:34224) 2024-04-25T16:11:06.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.324+0000 7f13a9045640 1 -- 172.21.15.31:0/240151618 learned_addr learned my addr 172.21.15.31:0/240151618 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:06.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.325+0000 7f13a9846640 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 0x55713900dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.325+0000 7f13a9045640 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139014b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.330+0000 7f13a9045640 1 -- 172.21.15.31:0/240151618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 msgr2=0x557139014b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.330+0000 7f13a9045640 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139014b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.330+0000 7f13a9045640 1 -- 172.21.15.31:0/240151618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 msgr2=0x55713900dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.330+0000 7f13a9045640 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 0x55713900dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.330+0000 7f13a9045640 1 -- 172.21.15.31:0/240151618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557138c793f0 con 0x557138c384d0 2024-04-25T16:11:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.330+0000 7f13a9846640 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 0x55713900dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:11:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.330+0000 7f139affd640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13900afaf0 0x7f13900b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.331+0000 7f13a9045640 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c384d0 0x55713900d6e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f13a0018c70 tx=0x7f13a0008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e42ec7ea65d643d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:06.332 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.331+0000 7f138d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:06.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.331+0000 7f139affd640 1 -- 172.21.15.31:0/310719758 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f13940de2e0 con 0x7f13a4109850 2024-04-25T16:11:06.335 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.333+0000 7f138d7fa640 1 -- 172.21.15.31:0/240151618 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0016c00 con 0x557138c384d0 2024-04-25T16:11:06.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.333+0000 7f138d7fa640 1 -- 172.21.15.31:0/240151618 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f13a0016da0 con 0x557138c384d0 2024-04-25T16:11:06.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.334+0000 7f13a9846640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13900afaf0 0x7f13900b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.334+0000 7f13b2a33740 1 -- 172.21.15.31:0/240151618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f139c036030 con 0x557138c384d0 2024-04-25T16:11:06.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.337+0000 7f138d7fa640 1 -- 172.21.15.31:0/240151618 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f13a0026050 con 0x557138c384d0 2024-04-25T16:11:06.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.337+0000 7f13b2a33740 10 client.27208 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:06.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.337+0000 7f13b2a33740 1 -- 172.21.15.31:0/240151618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557138d437d0 con 0x557138c384d0 2024-04-25T16:11:06.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.338+0000 7f138d7fa640 1 -- 172.21.15.31:0/240151618 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139c036030 con 0x557138c384d0 2024-04-25T16:11:06.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.338+0000 7f138d7fa640 1 -- 172.21.15.31:0/240151618 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f13a0015040 con 0x557138c384d0 2024-04-25T16:11:06.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.339+0000 7f13b2a33740 10 client.27208 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:06.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.339+0000 7f13b2a33740 10 client.27208 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:11:06.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.339+0000 7f13b2a33740 20 client.27208 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:06.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.339+0000 7f13b2a33740 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x557138c1ff60 0x55713902c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.339+0000 7f13b2a33740 4 client.27208 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:06.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.339+0000 7f13b2a33740 1 -- 172.21.15.31:0/240151618 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557138f75990 con 0x557138c1ff60 2024-04-25T16:11:06.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.340+0000 7f13a9045640 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x557138c1ff60 0x55713902c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.351+0000 7f13a9846640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13900afaf0 0x7f13900b2000 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55713900e750 tx=0x7f139c031a50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:06.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.351+0000 7f13a9045640 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x557138c1ff60 0x55713902c4d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=829b68f7c7a54f8e server_cookie=e9ad6f6acc29761d in_seq=0 out_seq=0 2024-04-25T16:11:06.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.352+0000 7f138d7fa640 10 client.27208 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:11:06.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.353+0000 7f138d7fa640 1 -- 172.21.15.31:0/240151618 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557138f75990 con 0x557138c1ff60 2024-04-25T16:11:06.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.353+0000 7f138d7fa640 10 client.27208 handle_command_reply: tid=0 2024-04-25T16:11:06.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.353+0000 7f13b2a33740 1 client.27208 shutdown 2024-04-25T16:11:06.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.362+0000 7f13b2a33740 1 -- 172.21.15.31:0/240151618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c384d0 msgr2=0x55713900d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.362+0000 7f13b2a33740 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c384d0 0x55713900d6e0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f13a0018c70 tx=0x7f13a0008040 comp rx=0 tx=0).stop 2024-04-25T16:11:06.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.366+0000 7f13b2a33740 1 -- 172.21.15.31:0/240151618 shutdown_connections 2024-04-25T16:11:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.366+0000 7f13b2a33740 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x557138c1ff60 0x55713902c4d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.366+0000 7f13b2a33740 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139014b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.366+0000 7f13b2a33740 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c406b0 0x55713900dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.366+0000 7f13b2a33740 1 --2- 172.21.15.31:0/240151618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c384d0 0x55713900d6e0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.366+0000 7f13b2a33740 1 -- 172.21.15.31:0/240151618 >> 172.21.15.31:0/240151618 conn(0x557138c554e0 msgr2=0x557139002120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.366+0000 7f138d7fa640 0 client.27208 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:11:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.367+0000 7f13b2a33740 1 -- 172.21.15.31:0/240151618 shutdown_connections 2024-04-25T16:11:06.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.367+0000 7f13b2a33740 1 -- 172.21.15.31:0/240151618 wait complete. 2024-04-25T16:11:06.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.368+0000 7f13b2a33740 20 client.27208 trim_cache size 0 max 16384 2024-04-25T16:11:06.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.405+0000 7f13b2a33740 1 Processor -- start 2024-04-25T16:11:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.406+0000 7f13b2a33740 1 -- start start 2024-04-25T16:11:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.406+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x55713900d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.407+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139002720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.407+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139002c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.407+0000 7f13b2a33740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f138800e540 con 0x557138c384d0 2024-04-25T16:11:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.407+0000 7f13b2a33740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1388010080 con 0x557138c61390 2024-04-25T16:11:06.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.407+0000 7f13b2a33740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f138800ec10 con 0x557138c406b0 2024-04-25T16:11:06.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.407+0000 7f13aa047640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139002c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.407+0000 7f13a9846640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x55713900d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.408+0000 7f13a9846640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x55713900d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53462/0 (socket says 172.21.15.31:53462) 2024-04-25T16:11:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.408+0000 7f13a9846640 1 -- 172.21.15.31:0/393715137 learned_addr learned my addr 172.21.15.31:0/393715137 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:06.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.408+0000 7f13a9045640 1 --2- 172.21.15.31:0/393715137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139002720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.409+0000 7f13a9846640 1 -- 172.21.15.31:0/393715137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 msgr2=0x557139002720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.409+0000 7f13a9846640 1 --2- 172.21.15.31:0/393715137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139002720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.409+0000 7f13a9846640 1 -- 172.21.15.31:0/393715137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 msgr2=0x557139002c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.409+0000 7f13a9846640 1 --2- 172.21.15.31:0/393715137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139002c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.409+0000 7f13a9846640 1 -- 172.21.15.31:0/393715137 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557139003550 con 0x557138c384d0 2024-04-25T16:11:06.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.410+0000 7f13a9846640 1 --2- 172.21.15.31:0/393715137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x55713900d9a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f139c009a00 tx=0x7f139c009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c92c5484d6139aa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:06.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.413+0000 7f138f7fe640 1 -- 172.21.15.31:0/393715137 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c05e070 con 0x557138c384d0 2024-04-25T16:11:06.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.413+0000 7f138f7fe640 1 -- 172.21.15.31:0/393715137 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139c036050 con 0x557138c384d0 2024-04-25T16:11:06.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.413+0000 7f138f7fe640 1 -- 172.21.15.31:0/393715137 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f139c0516c0 con 0x557138c384d0 2024-04-25T16:11:06.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.415+0000 7f13b2a33740 1 -- 172.21.15.31:0/393715137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 msgr2=0x55713900d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.415+0000 7f13b2a33740 1 --2- 172.21.15.31:0/393715137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x55713900d9a0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f139c009a00 tx=0x7f139c009870 comp rx=0 tx=0).stop 2024-04-25T16:11:06.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.422+0000 7f13b2a33740 1 -- 172.21.15.31:0/393715137 shutdown_connections 2024-04-25T16:11:06.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.422+0000 7f13b2a33740 1 --2- 172.21.15.31:0/393715137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139002c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.422+0000 7f13b2a33740 1 --2- 172.21.15.31:0/393715137 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139002720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.422+0000 7f13b2a33740 1 --2- 172.21.15.31:0/393715137 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x55713900d9a0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.422+0000 7f13b2a33740 1 -- 172.21.15.31:0/393715137 >> 172.21.15.31:0/393715137 conn(0x557138c554e0 msgr2=0x557139002010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:06.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.426+0000 7f13b2a33740 1 -- 172.21.15.31:0/393715137 shutdown_connections 2024-04-25T16:11:06.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.426+0000 7f13b2a33740 1 -- 172.21.15.31:0/393715137 wait complete. 2024-04-25T16:11:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.427+0000 7f13b2a33740 1 -- start start 2024-04-25T16:11:06.428 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.427+0000 7f13b2a33740 1 Processor -- start 2024-04-25T16:11:06.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.428+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x5571390112c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.429 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.428+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139011800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.429+0000 7f13b2a33740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139092830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.429+0000 7f13a9045640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139011800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.429+0000 7f13a9045640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139011800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34238/0 (socket says 172.21.15.31:34238) 2024-04-25T16:11:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.429+0000 7f13a9045640 1 -- 172.21.15.31:0/3868282183 learned_addr learned my addr 172.21.15.31:0/3868282183 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.430+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f138800f9b0 con 0x557138c384d0 2024-04-25T16:11:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.430+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f138800f2e0 con 0x557138c61390 2024-04-25T16:11:06.431 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.430+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f138800c520 con 0x557138c406b0 2024-04-25T16:11:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.430+0000 7f13a9846640 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x5571390112c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.430+0000 7f13aa047640 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139092830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.430+0000 7f13b2a33740 10 client.0 _ll_register_callbacks cb 0x7f13a000d7e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.431+0000 7f13a9045640 1 -- 172.21.15.31:0/3868282183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 msgr2=0x557139092830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.431+0000 7f13a9045640 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139092830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.431+0000 7f13a9045640 1 -- 172.21.15.31:0/3868282183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 msgr2=0x5571390112c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.431+0000 7f13a9045640 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x5571390112c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.431+0000 7f13a9045640 1 -- 172.21.15.31:0/3868282183 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557139092da0 con 0x557138c406b0 2024-04-25T16:11:06.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.432+0000 7f13a9045640 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139011800 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f13a0006dc0 tx=0x7f13a0016fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9543e99d662f84ae server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:06.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.434+0000 7f138d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:06.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.434+0000 7f138d7fa640 1 -- 172.21.15.31:0/3868282183 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13a0021cd0 con 0x557138c406b0 2024-04-25T16:11:06.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.434+0000 7f138d7fa640 1 -- 172.21.15.31:0/3868282183 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f13a00145c0 con 0x557138c406b0 2024-04-25T16:11:06.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.436+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f139c036050 con 0x557138c406b0 2024-04-25T16:11:06.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.438+0000 7f138d7fa640 1 -- 172.21.15.31:0/3868282183 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f13a0018790 con 0x557138c406b0 2024-04-25T16:11:06.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.439+0000 7f13b2a33740 10 client.27214 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:06.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.439+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557139093f80 con 0x557138c406b0 2024-04-25T16:11:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.441+0000 7f138d7fa640 1 -- 172.21.15.31:0/3868282183 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13a0005e20 con 0x557138c406b0 2024-04-25T16:11:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.441+0000 7f138d7fa640 1 -- 172.21.15.31:0/3868282183 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f13a0011510 con 0x557138c406b0 2024-04-25T16:11:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.441+0000 7f13b2a33740 10 client.27214 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.441+0000 7f13b2a33740 10 client.27214 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:11:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.441+0000 7f13b2a33740 20 client.27214 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.441+0000 7f13b2a33740 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x557138bf53a0 0x5571390968f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.441+0000 7f13b2a33740 4 client.27214 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.441+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f139c036050 con 0x557138bf53a0 2024-04-25T16:11:06.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.442+0000 7f13a9846640 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x557138bf53a0 0x5571390968f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.444+0000 7f13a9846640 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x557138bf53a0 0x5571390968f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=246918344dc5cda server_cookie=74841b044804dc90 in_seq=0 out_seq=0 2024-04-25T16:11:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.450+0000 7f138d7fa640 10 client.27214 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:11:06.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.458+0000 7f138d7fa640 1 -- 172.21.15.31:0/3868282183 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f139c036050 con 0x557138bf53a0 2024-04-25T16:11:06.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.458+0000 7f138d7fa640 10 client.27214 handle_command_reply: tid=0 2024-04-25T16:11:06.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.458+0000 7f13b2a33740 1 client.27214 shutdown 2024-04-25T16:11:06.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.462+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 msgr2=0x557139011800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.462+0000 7f13b2a33740 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139011800 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f13a0006dc0 tx=0x7f13a0016fa0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.462+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 shutdown_connections 2024-04-25T16:11:06.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.462+0000 7f13b2a33740 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x557138bf53a0 0x5571390968f0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.462+0000 7f13b2a33740 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557138c61390 0x557139092830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.462+0000 7f13b2a33740 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557138c406b0 0x557139011800 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.462+0000 7f13b2a33740 1 --2- 172.21.15.31:0/3868282183 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557138c384d0 0x5571390112c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.462+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 >> 172.21.15.31:0/3868282183 conn(0x557138c554e0 msgr2=0x557139002a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:06.466 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.464+0000 7f138d7fa640 0 client.27214 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:11:06.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.466+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 shutdown_connections 2024-04-25T16:11:06.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.466+0000 7f13b2a33740 1 -- 172.21.15.31:0/3868282183 wait complete. 2024-04-25T16:11:06.468 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:06.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.467+0000 7f13b2a33740 20 client.27214 trim_cache size 0 max 16384 2024-04-25T16:11:06.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.467+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13900afaf0 msgr2=0x7f13900b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.467+0000 7f13aa848640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13900afaf0 0x7f13900b2000 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x55713900e750 tx=0x7f139c031a50 comp rx=0 tx=0).stop 2024-04-25T16:11:06.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.468+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13a4109850 msgr2=0x7f13a41a16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.469 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.468+0000 7f13aa848640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13a4109850 0x7f13a41a16a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f139400b3e0 tx=0x7f139400b8a0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.469+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 shutdown_connections 2024-04-25T16:11:06.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.470+0000 7f13aa848640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13900afaf0 0x7f13900b2000 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.470+0000 7f13aa848640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f13a4109850 0x7f13a41a16a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.470+0000 7f13aa848640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f13a4104b50 0x7f13a419a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.470+0000 7f13aa848640 1 --2- 172.21.15.31:0/310719758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13a4103910 0x7f13a419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.470+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 >> 172.21.15.31:0/310719758 conn(0x7f13a40feb20 msgr2=0x7f13a4105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:06.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.470+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 shutdown_connections 2024-04-25T16:11:06.472 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.470+0000 7f13aa848640 1 -- 172.21.15.31:0/310719758 wait complete. 2024-04-25T16:11:06.509 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:06 smithi031 ceph-mon[29140]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 743 MiB/s wr, 1.70k op/s 2024-04-25T16:11:06.510 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:11:06.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:06 smithi063 ceph-mon[32413]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 743 MiB/s wr, 1.70k op/s 2024-04-25T16:11:06.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:06 smithi186 ceph-mon[33258]: pgmap v645: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 743 MiB/s wr, 1.70k op/s 2024-04-25T16:11:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.935+0000 7fd23e3c5640 1 -- 172.21.15.31:0/972785050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd238110030 msgr2=0x7fd238110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.935+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/972785050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd238110030 0x7fd238110440 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fd230009ca0 tx=0x7fd23002a8e0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.936+0000 7fd23e3c5640 1 -- 172.21.15.31:0/972785050 shutdown_connections 2024-04-25T16:11:06.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.936+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/972785050 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd238110030 0x7fd238110440 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.936+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/972785050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd23810b310 0x7fd23810b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.936+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/972785050 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd23810a420 0x7fd23810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.936+0000 7fd23e3c5640 1 -- 172.21.15.31:0/972785050 >> 172.21.15.31:0/972785050 conn(0x7fd2380704e0 msgr2=0x7fd238072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.952+0000 7fd23e3c5640 1 -- 172.21.15.31:0/972785050 shutdown_connections 2024-04-25T16:11:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.952+0000 7fd23e3c5640 1 -- 172.21.15.31:0/972785050 wait complete. 2024-04-25T16:11:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.953+0000 7fd23e3c5640 1 Processor -- start 2024-04-25T16:11:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.953+0000 7fd23e3c5640 1 -- start start 2024-04-25T16:11:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.954+0000 7fd23e3c5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd23810a420 0x7fd238117050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.954+0000 7fd23e3c5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd23810b310 0x7fd238117590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.954+0000 7fd23e3c5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd238110030 0x7fd2381184f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.954+0000 7fd23e3c5640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd23811e490 con 0x7fd238110030 2024-04-25T16:11:06.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.954+0000 7fd23e3c5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd23811e310 con 0x7fd23810b310 2024-04-25T16:11:06.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.954+0000 7fd23e3c5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd23811e610 con 0x7fd23810a420 2024-04-25T16:11:06.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.956+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd23810b310 0x7fd238117590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.956+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd23810b310 0x7fd238117590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53516/0 (socket says 172.21.15.31:53516) 2024-04-25T16:11:06.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.956+0000 7fd2377fe640 1 -- 172.21.15.31:0/1857456407 learned_addr learned my addr 172.21.15.31:0/1857456407 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:06.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.956+0000 7fd237fff640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd23810a420 0x7fd238117050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.956+0000 7fd23c962640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd238110030 0x7fd2381184f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.957+0000 7fd23c962640 1 -- 172.21.15.31:0/1857456407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd23810a420 msgr2=0x7fd238117050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.957+0000 7fd23c962640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd23810a420 0x7fd238117050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.957+0000 7fd23c962640 1 -- 172.21.15.31:0/1857456407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd23810b310 msgr2=0x7fd238117590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.957+0000 7fd23c962640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd23810b310 0x7fd238117590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.957+0000 7fd23c962640 1 -- 172.21.15.31:0/1857456407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd238118b60 con 0x7fd238110030 2024-04-25T16:11:06.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.958+0000 7fd23c962640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd238110030 0x7fd2381184f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd230009bf0 tx=0x7fd230003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.968+0000 7fd2357fa640 1 -- 172.21.15.31:0/1857456407 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd230003c90 con 0x7fd238110030 2024-04-25T16:11:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.968+0000 7fd2357fa640 1 -- 172.21.15.31:0/1857456407 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd230003e30 con 0x7fd238110030 2024-04-25T16:11:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.968+0000 7fd23e3c5640 1 -- 172.21.15.31:0/1857456407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd238118df0 con 0x7fd238110030 2024-04-25T16:11:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.968+0000 7fd23e3c5640 1 -- 172.21.15.31:0/1857456407 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd238119340 con 0x7fd238110030 2024-04-25T16:11:06.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.969+0000 7fd2357fa640 1 -- 172.21.15.31:0/1857456407 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2300045a0 con 0x7fd238110030 2024-04-25T16:11:06.975 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.973+0000 7fd24527b740 1 Processor -- start 2024-04-25T16:11:06.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.976+0000 7fd24527b740 1 -- start start 2024-04-25T16:11:06.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.976+0000 7fd24527b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d42390 0x5654a50e4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd24527b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 0x5654a50e4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd24527b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d216b0 0x5654a50e5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd24527b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5654a5048d80 con 0x5654a4d42390 2024-04-25T16:11:06.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd24527b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5654a5002a20 con 0x5654a4d194d0 2024-04-25T16:11:06.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd24527b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5654a5044d40 con 0x5654a4d216b0 2024-04-25T16:11:06.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd2357fa640 1 -- 172.21.15.31:0/1857456407 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd230030880 con 0x7fd238110030 2024-04-25T16:11:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d216b0 0x5654a50e5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d216b0 0x5654a50e5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34264/0 (socket says 172.21.15.31:34264) 2024-04-25T16:11:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd2377fe640 1 -- 172.21.15.31:0/1828013973 learned_addr learned my addr 172.21.15.31:0/1828013973 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:06.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd2377fe640 1 -- 172.21.15.31:0/1828013973 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 msgr2=0x5654a50e4d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.977+0000 7fd2377fe640 1 --2- 172.21.15.31:0/1828013973 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 0x5654a50e4d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.978+0000 7fd2377fe640 1 -- 172.21.15.31:0/1828013973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d42390 msgr2=0x5654a50e4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.978+0000 7fd2377fe640 1 --2- 172.21.15.31:0/1828013973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d42390 0x5654a50e4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.978+0000 7fd2377fe640 1 -- 172.21.15.31:0/1828013973 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654a500dc30 con 0x5654a4d216b0 2024-04-25T16:11:06.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.978+0000 7fd2377fe640 1 --2- 172.21.15.31:0/1828013973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d216b0 0x5654a50e5a20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd228009d80 tx=0x7fd2280127c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b945c970c443d8e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:06.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.979+0000 7fd2357fa640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2200afd00 0x7fd2200b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.980+0000 7fd2357fa640 1 -- 172.21.15.31:0/1857456407 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd230100100 con 0x7fd238110030 2024-04-25T16:11:06.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.980+0000 7fd237fff640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2200afd00 0x7fd2200b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.982+0000 7fd237fff640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2200afd00 0x7fd2200b2210 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x5654a50e4530 tx=0x7fd22c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.985+0000 7fd21e7fc640 1 -- 172.21.15.31:0/1828013973 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd228010650 con 0x5654a4d216b0 2024-04-25T16:11:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.986+0000 7fd21e7fc640 1 -- 172.21.15.31:0/1828013973 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd228010ca0 con 0x5654a4d216b0 2024-04-25T16:11:06.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.986+0000 7fd21e7fc640 1 -- 172.21.15.31:0/1828013973 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22801bce0 con 0x5654a4d216b0 2024-04-25T16:11:06.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.987+0000 7fd24527b740 1 -- 172.21.15.31:0/1828013973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d216b0 msgr2=0x5654a50e5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:06.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.987+0000 7fd24527b740 1 --2- 172.21.15.31:0/1828013973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d216b0 0x5654a50e5a20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd228009d80 tx=0x7fd2280127c0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.988+0000 7fd24527b740 1 -- 172.21.15.31:0/1828013973 shutdown_connections 2024-04-25T16:11:06.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.988+0000 7fd24527b740 1 --2- 172.21.15.31:0/1828013973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d216b0 0x5654a50e5a20 secure :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd228009d80 tx=0x7fd2280127c0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.988+0000 7fd24527b740 1 --2- 172.21.15.31:0/1828013973 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 0x5654a50e4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.988+0000 7fd24527b740 1 --2- 172.21.15.31:0/1828013973 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d42390 0x5654a50e4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.988+0000 7fd24527b740 1 -- 172.21.15.31:0/1828013973 >> 172.21.15.31:0/1828013973 conn(0x5654a4d364e0 msgr2=0x5654a50e0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:06.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.993+0000 7fd24527b740 1 -- 172.21.15.31:0/1828013973 shutdown_connections 2024-04-25T16:11:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.993+0000 7fd24527b740 1 -- 172.21.15.31:0/1828013973 wait complete. 2024-04-25T16:11:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.994+0000 7fd24527b740 1 -- start start 2024-04-25T16:11:06.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.994+0000 7fd24527b740 1 Processor -- start 2024-04-25T16:11:06.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.994+0000 7fd24527b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 0x5654a50ee750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.994+0000 7fd24527b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d42390 0x5654a50eec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.994+0000 7fd24527b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d00f60 0x5654a50f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:06.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.994+0000 7fd24527b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5654a50e6530 con 0x5654a4d00f60 2024-04-25T16:11:06.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.994+0000 7fd24527b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5654a4fcfdc0 con 0x5654a4d194d0 2024-04-25T16:11:06.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.994+0000 7fd24527b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5654a50e66b0 con 0x5654a4d42390 2024-04-25T16:11:06.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.994+0000 7fd24527b740 10 client.0 _ll_register_callbacks cb 0x5654a4feb600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:06.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.995+0000 7fd237fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 0x5654a50ee750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.995+0000 7fd237fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 0x5654a50ee750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53520/0 (socket says 172.21.15.31:53520) 2024-04-25T16:11:06.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.995+0000 7fd237fff640 1 -- 172.21.15.31:0/3198965848 learned_addr learned my addr 172.21.15.31:0/3198965848 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:06.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.996+0000 7fd2377fe640 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d42390 0x5654a50eec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.996+0000 7fd23c962640 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d00f60 0x5654a50f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.996+0000 7fd237fff640 1 -- 172.21.15.31:0/3198965848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d42390 msgr2=0x5654a50eec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.996+0000 7fd237fff640 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d42390 0x5654a50eec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.996+0000 7fd237fff640 1 -- 172.21.15.31:0/3198965848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d00f60 msgr2=0x5654a50f5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:06.998 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.996+0000 7fd237fff640 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d00f60 0x5654a50f5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:06.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.996+0000 7fd237fff640 1 -- 172.21.15.31:0/3198965848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654a4d5a3f0 con 0x5654a4d194d0 2024-04-25T16:11:07.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.998+0000 7fd237fff640 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 0x5654a50ee750 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0127d0 tx=0x7fd22c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c107ae4cba6cde1a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:07.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.999+0000 7fd2177fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:07.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:06.999+0000 7fd2177fe640 1 -- 172.21.15.31:0/3198965848 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd22c00fad0 con 0x5654a4d194d0 2024-04-25T16:11:07.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.017+0000 7fd24527b740 1 -- 172.21.15.31:0/3198965848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd228010ca0 con 0x5654a4d194d0 2024-04-25T16:11:07.019 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.017+0000 7fd2177fe640 1 -- 172.21.15.31:0/3198965848 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd22c00fc70 con 0x5654a4d194d0 2024-04-25T16:11:07.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.019+0000 7fd2177fe640 1 -- 172.21.15.31:0/3198965848 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd22c00fe70 con 0x5654a4d194d0 2024-04-25T16:11:07.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.019+0000 7fd24527b740 10 client.27254 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:07.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.019+0000 7fd24527b740 1 -- 172.21.15.31:0/3198965848 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5654a4e247d0 con 0x5654a4d194d0 2024-04-25T16:11:07.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.021+0000 7fd2177fe640 1 -- 172.21.15.31:0/3198965848 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd228010ca0 con 0x5654a4d194d0 2024-04-25T16:11:07.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.021+0000 7fd2177fe640 1 -- 172.21.15.31:0/3198965848 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd22c01f6c0 con 0x5654a4d194d0 2024-04-25T16:11:07.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.022+0000 7fd24527b740 10 client.27254 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:07.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.022+0000 7fd24527b740 10 client.27254 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:11:07.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.022+0000 7fd24527b740 20 client.27254 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:07.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.022+0000 7fd24527b740 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5654a4cd63a0 0x5654a50fc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.022+0000 7fd24527b740 4 client.27254 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:07.025 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.022+0000 7fd24527b740 1 -- 172.21.15.31:0/3198965848 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5654a5056990 con 0x5654a4cd63a0 2024-04-25T16:11:07.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.024+0000 7fd2377fe640 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5654a4cd63a0 0x5654a50fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.026+0000 7fd2377fe640 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5654a4cd63a0 0x5654a50fc360 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7043b51d99422e6d server_cookie=172cf32cb36eab6f in_seq=0 out_seq=0 2024-04-25T16:11:07.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.026+0000 7fd2177fe640 10 client.27254 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:11:07.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.028+0000 7fd2177fe640 1 -- 172.21.15.31:0/3198965848 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5654a5056990 con 0x5654a4cd63a0 2024-04-25T16:11:07.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.028+0000 7fd2177fe640 10 client.27254 handle_command_reply: tid=0 2024-04-25T16:11:07.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.029+0000 7fd24527b740 1 client.27254 shutdown 2024-04-25T16:11:07.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.037+0000 7fd24527b740 1 -- 172.21.15.31:0/3198965848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 msgr2=0x5654a50ee750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.037+0000 7fd24527b740 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 0x5654a50ee750 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0127d0 tx=0x7fd22c004730 comp rx=0 tx=0).stop 2024-04-25T16:11:07.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.038+0000 7fd24527b740 1 -- 172.21.15.31:0/3198965848 shutdown_connections 2024-04-25T16:11:07.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.038+0000 7fd24527b740 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5654a4cd63a0 0x5654a50fc360 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.038+0000 7fd24527b740 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d00f60 0x5654a50f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.038+0000 7fd24527b740 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d42390 0x5654a50eec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.038+0000 7fd24527b740 1 --2- 172.21.15.31:0/3198965848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d194d0 0x5654a50ee750 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.038+0000 7fd24527b740 1 -- 172.21.15.31:0/3198965848 >> 172.21.15.31:0/3198965848 conn(0x5654a4d364e0 msgr2=0x5654a50df040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.044+0000 7fd2177fe640 0 client.27254 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:11:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.045+0000 7fd24527b740 1 -- 172.21.15.31:0/3198965848 shutdown_connections 2024-04-25T16:11:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.045+0000 7fd24527b740 1 -- 172.21.15.31:0/3198965848 wait complete. 2024-04-25T16:11:07.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.045+0000 7fd24527b740 20 client.27254 trim_cache size 0 max 16384 2024-04-25T16:11:07.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.094+0000 7fd24527b740 1 Processor -- start 2024-04-25T16:11:07.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.094+0000 7fd24527b740 1 -- start start 2024-04-25T16:11:07.096 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.094+0000 7fd24527b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d00f60 0x5654a50dfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.095+0000 7fd24527b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.095+0000 7fd24527b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d42390 0x5654a50dff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.095+0000 7fd24527b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd2381b18b0 con 0x5654a4d42390 2024-04-25T16:11:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.095+0000 7fd24527b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd2381b33f0 con 0x5654a4d00f60 2024-04-25T16:11:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.095+0000 7fd24527b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd2381b1f80 con 0x5654a4d194d0 2024-04-25T16:11:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.095+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.095+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34286/0 (socket says 172.21.15.31:34286) 2024-04-25T16:11:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.095+0000 7fd2377fe640 1 -- 172.21.15.31:0/4155097087 learned_addr learned my addr 172.21.15.31:0/4155097087 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:07.097 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.095+0000 7fd237fff640 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d00f60 0x5654a50dfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.096+0000 7fd23c962640 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d42390 0x5654a50dff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.096+0000 7fd2377fe640 1 -- 172.21.15.31:0/4155097087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d00f60 msgr2=0x5654a50dfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.096+0000 7fd2377fe640 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d00f60 0x5654a50dfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.096+0000 7fd2377fe640 1 -- 172.21.15.31:0/4155097087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d42390 msgr2=0x5654a50dff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.096+0000 7fd2377fe640 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d42390 0x5654a50dff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.096+0000 7fd2377fe640 1 -- 172.21.15.31:0/4155097087 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5654a50e9ee0 con 0x5654a4d194d0 2024-04-25T16:11:07.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.096+0000 7fd237fff640 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d00f60 0x5654a50dfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:11:07.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.097+0000 7fd2377fe640 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f5bb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x5654a50e5890 tx=0x7fd2280098e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56623554b9af48c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:07.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.098+0000 7fd21dffb640 1 -- 172.21.15.31:0/4155097087 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd228020040 con 0x5654a4d194d0 2024-04-25T16:11:07.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.104+0000 7fd21dffb640 1 -- 172.21.15.31:0/4155097087 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd228010450 con 0x5654a4d194d0 2024-04-25T16:11:07.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.104+0000 7fd21dffb640 1 -- 172.21.15.31:0/4155097087 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd228003850 con 0x5654a4d194d0 2024-04-25T16:11:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.106+0000 7fd24527b740 1 -- 172.21.15.31:0/4155097087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 msgr2=0x5654a50f5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.107+0000 7fd24527b740 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f5bb0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x5654a50e5890 tx=0x7fd2280098e0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.107+0000 7fd24527b740 1 -- 172.21.15.31:0/4155097087 shutdown_connections 2024-04-25T16:11:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.107+0000 7fd24527b740 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d42390 0x5654a50dff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.107+0000 7fd24527b740 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f5bb0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.107+0000 7fd24527b740 1 --2- 172.21.15.31:0/4155097087 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d00f60 0x5654a50dfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.107+0000 7fd24527b740 1 -- 172.21.15.31:0/4155097087 >> 172.21.15.31:0/4155097087 conn(0x5654a4d364e0 msgr2=0x5654a50dad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.107+0000 7fd24527b740 1 -- 172.21.15.31:0/4155097087 shutdown_connections 2024-04-25T16:11:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.107+0000 7fd24527b740 1 -- 172.21.15.31:0/4155097087 wait complete. 2024-04-25T16:11:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.107+0000 7fd24527b740 1 -- start start 2024-04-25T16:11:07.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.108+0000 7fd24527b740 1 Processor -- start 2024-04-25T16:11:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.108+0000 7fd24527b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d00f60 0x5654a50f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.108+0000 7fd24527b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.108+0000 7fd24527b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d42390 0x5654a50f4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.108+0000 7fd24527b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd2381b2d20 con 0x5654a4d00f60 2024-04-25T16:11:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.108+0000 7fd24527b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd2381b2650 con 0x5654a4d42390 2024-04-25T16:11:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.108+0000 7fd24527b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd2381af890 con 0x5654a4d194d0 2024-04-25T16:11:07.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.108+0000 7fd24527b740 10 client.0 _ll_register_callbacks cb 0x7fd22c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.109+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.109+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34300/0 (socket says 172.21.15.31:34300) 2024-04-25T16:11:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.109+0000 7fd2377fe640 1 -- 172.21.15.31:0/3201836035 learned_addr learned my addr 172.21.15.31:0/3201836035 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:07.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.109+0000 7fd23c962640 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d42390 0x5654a50f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.110+0000 7fd2377fe640 1 -- 172.21.15.31:0/3201836035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d42390 msgr2=0x5654a50f4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.110+0000 7fd2377fe640 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d42390 0x5654a50f4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.110+0000 7fd2377fe640 1 -- 172.21.15.31:0/3201836035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d00f60 msgr2=0x5654a50f0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.110+0000 7fd2377fe640 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d00f60 0x5654a50f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.110+0000 7fd2377fe640 1 -- 172.21.15.31:0/3201836035 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5654a50f4810 con 0x5654a4d194d0 2024-04-25T16:11:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.111+0000 7fd2377fe640 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f1310 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd228009800 tx=0x7fd228009550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7519dde09605c4c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.111+0000 7fd2177fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.112+0000 7fd2177fe640 1 -- 172.21.15.31:0/3201836035 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd228020070 con 0x5654a4d194d0 2024-04-25T16:11:07.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.112+0000 7fd24527b740 1 -- 172.21.15.31:0/3201836035 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd228010450 con 0x5654a4d194d0 2024-04-25T16:11:07.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.113+0000 7fd2177fe640 1 -- 172.21.15.31:0/3201836035 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd228010700 con 0x5654a4d194d0 2024-04-25T16:11:07.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.113+0000 7fd2177fe640 1 -- 172.21.15.31:0/3201836035 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd228010920 con 0x5654a4d194d0 2024-04-25T16:11:07.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.113+0000 7fd24527b740 10 client.27232 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:07.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.114+0000 7fd24527b740 1 -- 172.21.15.31:0/3201836035 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5654a50f5a30 con 0x5654a4d194d0 2024-04-25T16:11:07.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.114+0000 7fd2177fe640 1 -- 172.21.15.31:0/3201836035 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd228010450 con 0x5654a4d194d0 2024-04-25T16:11:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.115+0000 7fd2177fe640 1 -- 172.21.15.31:0/3201836035 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd228019020 con 0x5654a4d194d0 2024-04-25T16:11:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.115+0000 7fd24527b740 10 client.27232 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.115+0000 7fd24527b740 10 client.27232 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:11:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.115+0000 7fd24527b740 20 client.27232 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.115+0000 7fd24527b740 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5654a51648e0 0x5654a5166d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.115+0000 7fd24527b740 4 client.27232 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.115+0000 7fd24527b740 1 -- 172.21.15.31:0/3201836035 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5654a5050800 con 0x5654a51648e0 2024-04-25T16:11:07.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.115+0000 7fd237fff640 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5654a51648e0 0x5654a5166d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.117+0000 7fd237fff640 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5654a51648e0 0x5654a5166d00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86bec5741231c54a server_cookie=8f862b7ea50cacd4 in_seq=0 out_seq=0 2024-04-25T16:11:07.119 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.118+0000 7fd2177fe640 10 client.27232 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:11:07.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.119+0000 7fd2177fe640 1 -- 172.21.15.31:0/3201836035 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5654a5050800 con 0x5654a51648e0 2024-04-25T16:11:07.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.119+0000 7fd2177fe640 10 client.27232 handle_command_reply: tid=0 2024-04-25T16:11:07.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.119+0000 7fd24527b740 1 client.27232 shutdown 2024-04-25T16:11:07.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.145+0000 7fd24527b740 1 -- 172.21.15.31:0/3201836035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 msgr2=0x5654a50f1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.145+0000 7fd24527b740 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f1310 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd228009800 tx=0x7fd228009550 comp rx=0 tx=0).stop 2024-04-25T16:11:07.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.146+0000 7fd24527b740 1 -- 172.21.15.31:0/3201836035 shutdown_connections 2024-04-25T16:11:07.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.146+0000 7fd24527b740 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5654a4d42390 0x5654a50f4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.146+0000 7fd24527b740 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5654a4d194d0 0x5654a50f1310 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.146+0000 7fd24527b740 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5654a51648e0 0x5654a5166d00 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.146+0000 7fd24527b740 1 --2- 172.21.15.31:0/3201836035 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5654a4d00f60 0x5654a50f0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.146+0000 7fd24527b740 1 -- 172.21.15.31:0/3201836035 >> 172.21.15.31:0/3201836035 conn(0x5654a4d364e0 msgr2=0x5654a50ee290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.146+0000 7fd2177fe640 0 client.27232 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:11:07.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.150+0000 7fd24527b740 1 -- 172.21.15.31:0/3201836035 shutdown_connections 2024-04-25T16:11:07.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.150+0000 7fd24527b740 1 -- 172.21.15.31:0/3201836035 wait complete. 2024-04-25T16:11:07.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.150+0000 7fd24527b740 20 client.27232 trim_cache size 0 max 16384 2024-04-25T16:11:07.163 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:07.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.163+0000 7fd23e3c5640 1 -- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2200afd00 msgr2=0x7fd2200b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.163+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2200afd00 0x7fd2200b2210 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x5654a50e4530 tx=0x7fd22c005f90 comp rx=0 tx=0).stop 2024-04-25T16:11:07.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.163+0000 7fd23e3c5640 1 -- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd238110030 msgr2=0x7fd2381184f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.163+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd238110030 0x7fd2381184f0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd230009bf0 tx=0x7fd230003ba0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.170+0000 7fd23e3c5640 1 -- 172.21.15.31:0/1857456407 shutdown_connections 2024-04-25T16:11:07.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.170+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2200afd00 0x7fd2200b2210 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.170+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd238110030 0x7fd2381184f0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.170+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd23810b310 0x7fd238117590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.170+0000 7fd23e3c5640 1 --2- 172.21.15.31:0/1857456407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd23810a420 0x7fd238117050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.170+0000 7fd23e3c5640 1 -- 172.21.15.31:0/1857456407 >> 172.21.15.31:0/1857456407 conn(0x7fd2380704e0 msgr2=0x7fd23810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.176+0000 7fd23e3c5640 1 -- 172.21.15.31:0/1857456407 shutdown_connections 2024-04-25T16:11:07.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.177+0000 7fd23e3c5640 1 -- 172.21.15.31:0/1857456407 wait complete. 2024-04-25T16:11:07.244 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:11:07.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.641+0000 7f5f532d8640 1 -- 172.21.15.31:0/2560078459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4c10fef0 msgr2=0x7f5f4c110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.641+0000 7f5f532d8640 1 --2- 172.21.15.31:0/2560078459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4c10fef0 0x7f5f4c110300 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5f48009900 tx=0x7f5f4802a440 comp rx=0 tx=0).stop 2024-04-25T16:11:07.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.647+0000 7f5f532d8640 1 -- 172.21.15.31:0/2560078459 shutdown_connections 2024-04-25T16:11:07.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.647+0000 7f5f532d8640 1 --2- 172.21.15.31:0/2560078459 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4c10fef0 0x7f5f4c110300 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.647+0000 7f5f532d8640 1 --2- 172.21.15.31:0/2560078459 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c10b1d0 0x7f5f4c10b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.647+0000 7f5f532d8640 1 --2- 172.21.15.31:0/2560078459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f4c10a3e0 0x7f5f4c10ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.647+0000 7f5f532d8640 1 -- 172.21.15.31:0/2560078459 >> 172.21.15.31:0/2560078459 conn(0x7f5f4c070400 msgr2=0x7f5f4c072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.648+0000 7f5f532d8640 1 -- 172.21.15.31:0/2560078459 shutdown_connections 2024-04-25T16:11:07.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.648+0000 7f5f532d8640 1 -- 172.21.15.31:0/2560078459 wait complete. 2024-04-25T16:11:07.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.650+0000 7f5f532d8640 1 Processor -- start 2024-04-25T16:11:07.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.650+0000 7f5f532d8640 1 -- start start 2024-04-25T16:11:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.651+0000 7f5f532d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f4c10a3e0 0x7f5f4c1a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.651+0000 7f5f532d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c10b1d0 0x7f5f4c1a93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.651+0000 7f5f532d8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4c10fef0 0x7f5f4c1b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.651+0000 7f5f532d8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c11e300 con 0x7f5f4c10b1d0 2024-04-25T16:11:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.651+0000 7f5f532d8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c11e180 con 0x7f5f4c10a3e0 2024-04-25T16:11:07.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.651+0000 7f5f532d8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c11e480 con 0x7f5f4c10fef0 2024-04-25T16:11:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.652+0000 7f5f51074640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f4c10a3e0 0x7f5f4c1a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.652+0000 7f5f51074640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f4c10a3e0 0x7f5f4c1a8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53548/0 (socket says 172.21.15.31:53548) 2024-04-25T16:11:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.652+0000 7f5f51074640 1 -- 172.21.15.31:0/694984190 learned_addr learned my addr 172.21.15.31:0/694984190 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:07.654 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.652+0000 7f5f51875640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4c10fef0 0x7f5f4c1b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.653+0000 7f5f51074640 1 -- 172.21.15.31:0/694984190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4c10fef0 msgr2=0x7f5f4c1b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.653+0000 7f5f51074640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4c10fef0 0x7f5f4c1b0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.653+0000 7f5f51074640 1 -- 172.21.15.31:0/694984190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c10b1d0 msgr2=0x7f5f4c1a93b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:11:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.653+0000 7f5f51074640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c10b1d0 0x7f5f4c1a93b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.654+0000 7f5f51074640 1 -- 172.21.15.31:0/694984190 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f4c1b09b0 con 0x7f5f4c10a3e0 2024-04-25T16:11:07.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.655+0000 7f5f51074640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f4c10a3e0 0x7f5f4c1a8e70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400ebc0 tx=0x7f5f4400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:07.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.660+0000 7f5f427fc640 1 -- 172.21.15.31:0/694984190 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f44010630 con 0x7f5f4c10a3e0 2024-04-25T16:11:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.661+0000 7f5f427fc640 1 -- 172.21.15.31:0/694984190 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f44010c80 con 0x7f5f4c10a3e0 2024-04-25T16:11:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.661+0000 7f5f427fc640 1 -- 172.21.15.31:0/694984190 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f44017b30 con 0x7f5f4c10a3e0 2024-04-25T16:11:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.661+0000 7f5f532d8640 1 -- 172.21.15.31:0/694984190 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f4c1b0ca0 con 0x7f5f4c10a3e0 2024-04-25T16:11:07.663 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.661+0000 7f5f532d8640 1 -- 172.21.15.31:0/694984190 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f4c1b11a0 con 0x7f5f4c10a3e0 2024-04-25T16:11:07.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.667+0000 7f5f5a189740 1 Processor -- start 2024-04-25T16:11:07.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.667+0000 7f5f5a189740 1 -- start start 2024-04-25T16:11:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.668+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 0x55abbda9b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.668+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 0x55abbda9bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.668+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 0x55abbda9cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.668+0000 7f5f5a189740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55abbd9ffd80 con 0x55abbd6d86b0 2024-04-25T16:11:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.668+0000 7f5f5a189740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55abbd9b9a20 con 0x55abbd6f9390 2024-04-25T16:11:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.668+0000 7f5f5a189740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55abbd9fbd40 con 0x55abbd6d04d0 2024-04-25T16:11:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.668+0000 7f5f51875640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 0x55abbda9b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.668+0000 7f5f51875640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 0x55abbda9b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53564/0 (socket says 172.21.15.31:53564) 2024-04-25T16:11:07.671 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.669+0000 7f5f51875640 1 -- 172.21.15.31:0/4031963488 learned_addr learned my addr 172.21.15.31:0/4031963488 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:07.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.670+0000 7f5f427fc640 1 -- 172.21.15.31:0/694984190 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5f44017cd0 con 0x7f5f4c10a3e0 2024-04-25T16:11:07.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.670+0000 7f5f51074640 1 --2- 172.21.15.31:0/4031963488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 0x55abbda9bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.672 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.671+0000 7f5f427fc640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f380afcf0 0x7f5f380b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.672+0000 7f5f427fc640 1 -- 172.21.15.31:0/694984190 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5f440df540 con 0x7f5f4c10a3e0 2024-04-25T16:11:07.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.672+0000 7f5f51074640 1 -- 172.21.15.31:0/4031963488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 msgr2=0x55abbda9b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.672+0000 7f5f51074640 1 --2- 172.21.15.31:0/4031963488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 0x55abbda9b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.672+0000 7f5f51074640 1 -- 172.21.15.31:0/4031963488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 msgr2=0x55abbda9cbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:07.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.672+0000 7f5f51074640 1 --2- 172.21.15.31:0/4031963488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 0x55abbda9cbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.672+0000 7f5f51074640 1 -- 172.21.15.31:0/4031963488 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abbd9c4c30 con 0x55abbd6d04d0 2024-04-25T16:11:07.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.674+0000 7f5f51074640 1 --2- 172.21.15.31:0/4031963488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 0x55abbda9bf60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f440dcf90 tx=0x7f5f44017fc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=793f9dfa9a3c9e86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.675+0000 7f5f51875640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f380afcf0 0x7f5f380b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.676+0000 7f5f377fe640 1 -- 172.21.15.31:0/4031963488 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f440a3890 con 0x55abbd6d04d0 2024-04-25T16:11:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.676+0000 7f5f377fe640 1 -- 172.21.15.31:0/4031963488 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f440e2b20 con 0x55abbd6d04d0 2024-04-25T16:11:07.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.676+0000 7f5f377fe640 1 -- 172.21.15.31:0/4031963488 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f440a7140 con 0x55abbd6d04d0 2024-04-25T16:11:07.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.678+0000 7f5f5a189740 1 -- 172.21.15.31:0/4031963488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 msgr2=0x55abbda9bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.678+0000 7f5f5a189740 1 --2- 172.21.15.31:0/4031963488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 0x55abbda9bf60 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5f440dcf90 tx=0x7f5f44017fc0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.678+0000 7f5f51875640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f380afcf0 0x7f5f380b2200 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5f480096c0 tx=0x7f5f48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:07.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.678+0000 7f5f5a189740 1 -- 172.21.15.31:0/4031963488 shutdown_connections 2024-04-25T16:11:07.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.678+0000 7f5f5a189740 1 --2- 172.21.15.31:0/4031963488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 0x55abbda9cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.678+0000 7f5f5a189740 1 --2- 172.21.15.31:0/4031963488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 0x55abbda9bf60 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.678+0000 7f5f5a189740 1 --2- 172.21.15.31:0/4031963488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 0x55abbda9b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.678+0000 7f5f5a189740 1 -- 172.21.15.31:0/4031963488 >> 172.21.15.31:0/4031963488 conn(0x55abbd6ed4e0 msgr2=0x55abbda971f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.687+0000 7f5f5a189740 1 -- 172.21.15.31:0/4031963488 shutdown_connections 2024-04-25T16:11:07.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.689+0000 7f5f5a189740 1 -- 172.21.15.31:0/4031963488 wait complete. 2024-04-25T16:11:07.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.690+0000 7f5f5a189740 1 -- start start 2024-04-25T16:11:07.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.690+0000 7f5f5a189740 1 Processor -- start 2024-04-25T16:11:07.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.691+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 0x55abbdaa5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.691+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 0x55abbdaa5c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.691+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 0x55abbdaacbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.691+0000 7f5f5a189740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55abbda9d6a0 con 0x55abbd6d86b0 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.691+0000 7f5f5a189740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55abbd986dc0 con 0x55abbd6f9390 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.691+0000 7f5f5a189740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55abbda9d820 con 0x55abbd6d04d0 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.692+0000 7f5f5a189740 10 client.0 _ll_register_callbacks cb 0x55abbd9a2600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.692+0000 7f5f50873640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 0x55abbdaa5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.692+0000 7f5f50873640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 0x55abbdaa5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53546/0 (socket says 172.21.15.31:53546) 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.692+0000 7f5f50873640 1 -- 172.21.15.31:0/255815230 learned_addr learned my addr 172.21.15.31:0/255815230 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:07.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.693+0000 7f5f51074640 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 0x55abbdaa5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.696+0000 7f5f50873640 1 -- 172.21.15.31:0/255815230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 msgr2=0x55abbdaa5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.696+0000 7f5f50873640 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 0x55abbdaa5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.696+0000 7f5f50873640 1 -- 172.21.15.31:0/255815230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 msgr2=0x55abbdaacbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.696+0000 7f5f50873640 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 0x55abbdaacbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.698 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.696+0000 7f5f50873640 1 -- 172.21.15.31:0/255815230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abbd7113f0 con 0x55abbd6d86b0 2024-04-25T16:11:07.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.698+0000 7f5f50873640 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 0x55abbdaa5c70 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c009dd0 tx=0x7f5f3c00b4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea190752f9ff5941 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.701+0000 7f5f34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.701+0000 7f5f34ff9640 1 -- 172.21.15.31:0/255815230 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f3c015020 con 0x55abbd6d86b0 2024-04-25T16:11:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.701+0000 7f5f5a189740 1 -- 172.21.15.31:0/255815230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55abbda0d990 con 0x55abbd6d86b0 2024-04-25T16:11:07.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.702+0000 7f5f34ff9640 1 -- 172.21.15.31:0/255815230 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5f3c011420 con 0x55abbd6d86b0 2024-04-25T16:11:07.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.703+0000 7f5f34ff9640 1 -- 172.21.15.31:0/255815230 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5f3c011640 con 0x55abbd6d86b0 2024-04-25T16:11:07.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.703+0000 7f5f5a189740 10 client.17574 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:07.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.703+0000 7f5f5a189740 1 -- 172.21.15.31:0/255815230 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55abbd7db7d0 con 0x55abbd6d86b0 2024-04-25T16:11:07.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.705+0000 7f5f34ff9640 1 -- 172.21.15.31:0/255815230 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55abbda0d990 con 0x55abbd6d86b0 2024-04-25T16:11:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.705+0000 7f5f34ff9640 1 -- 172.21.15.31:0/255815230 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5f3c010a90 con 0x55abbd6d86b0 2024-04-25T16:11:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.705+0000 7f5f5a189740 10 client.17574 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.705+0000 7f5f5a189740 10 client.17574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.706+0000 7f5f5a189740 20 client.17574 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.706+0000 7f5f5a189740 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55abbd6b7f60 0x55abbdac4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.706+0000 7f5f5a189740 4 client.17574 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:07.707 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.706+0000 7f5f5a189740 1 -- 172.21.15.31:0/255815230 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abbda07800 con 0x55abbd6b7f60 2024-04-25T16:11:07.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.708+0000 7f5f51074640 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55abbd6b7f60 0x55abbdac4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.717+0000 7f5f34ff9640 10 client.17574 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:07.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.717+0000 7f5f51074640 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55abbd6b7f60 0x55abbdac4a20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=308b9aa0f87bcaa2 server_cookie=df3fc671aaff2871 in_seq=0 out_seq=0 2024-04-25T16:11:07.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.721+0000 7f5f34ff9640 1 -- 172.21.15.31:0/255815230 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55abbda07800 con 0x55abbd6b7f60 2024-04-25T16:11:07.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.721+0000 7f5f34ff9640 10 client.17574 handle_command_reply: tid=0 2024-04-25T16:11:07.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.721+0000 7f5f5a189740 1 client.17574 shutdown 2024-04-25T16:11:07.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.724+0000 7f5f5a189740 1 -- 172.21.15.31:0/255815230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 msgr2=0x55abbdaa5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.724+0000 7f5f5a189740 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 0x55abbdaa5c70 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c009dd0 tx=0x7f5f3c00b4b0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.731+0000 7f5f5a189740 1 -- 172.21.15.31:0/255815230 shutdown_connections 2024-04-25T16:11:07.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.731+0000 7f5f5a189740 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55abbd6b7f60 0x55abbdac4a20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.731+0000 7f5f5a189740 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6f9390 0x55abbdaacbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.731+0000 7f5f5a189740 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d86b0 0x55abbdaa5c70 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.731+0000 7f5f5a189740 1 --2- 172.21.15.31:0/255815230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d04d0 0x55abbdaa5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.731+0000 7f5f5a189740 1 -- 172.21.15.31:0/255815230 >> 172.21.15.31:0/255815230 conn(0x55abbd6ed4e0 msgr2=0x55abbda959c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.731+0000 7f5f34ff9640 0 client.17574 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:07.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.731+0000 7f5f5a189740 1 -- 172.21.15.31:0/255815230 shutdown_connections 2024-04-25T16:11:07.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.731+0000 7f5f5a189740 1 -- 172.21.15.31:0/255815230 wait complete. 2024-04-25T16:11:07.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.732+0000 7f5f5a189740 20 client.17574 trim_cache size 0 max 16384 2024-04-25T16:11:07.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.776+0000 7f5f5a189740 1 Processor -- start 2024-04-25T16:11:07.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.776+0000 7f5f5a189740 1 -- start start 2024-04-25T16:11:07.778 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.777+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d04d0 0x55abbdac4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.777+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d86b0 0x55abbdaacaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.777+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6f9390 0x55abbdaa04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.778+0000 7f5f5a189740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c1b17f0 con 0x55abbd6f9390 2024-04-25T16:11:07.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.778+0000 7f5f5a189740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c1b3330 con 0x55abbd6d04d0 2024-04-25T16:11:07.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.778+0000 7f5f5a189740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c1b1ec0 con 0x55abbd6d86b0 2024-04-25T16:11:07.779 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.778+0000 7f5f51074640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d04d0 0x55abbdac4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.778+0000 7f5f51074640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d04d0 0x55abbdac4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53582/0 (socket says 172.21.15.31:53582) 2024-04-25T16:11:07.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.778+0000 7f5f51074640 1 -- 172.21.15.31:0/4192890307 learned_addr learned my addr 172.21.15.31:0/4192890307 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:07.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.778+0000 7f5f50873640 1 --2- 172.21.15.31:0/4192890307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d86b0 0x55abbdaacaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.778+0000 7f5f51875640 1 --2- 172.21.15.31:0/4192890307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6f9390 0x55abbdaa04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.779+0000 7f5f51074640 1 -- 172.21.15.31:0/4192890307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d86b0 msgr2=0x55abbdaacaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.779+0000 7f5f51074640 1 --2- 172.21.15.31:0/4192890307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d86b0 0x55abbdaacaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.779+0000 7f5f51074640 1 -- 172.21.15.31:0/4192890307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6f9390 msgr2=0x55abbdaa04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.779+0000 7f5f51074640 1 --2- 172.21.15.31:0/4192890307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6f9390 0x55abbdaa04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.781 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.779+0000 7f5f51074640 1 -- 172.21.15.31:0/4192890307 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abbda88ba0 con 0x55abbd6d04d0 2024-04-25T16:11:07.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.781+0000 7f5f51074640 1 --2- 172.21.15.31:0/4192890307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d04d0 0x55abbdac4930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5f440a3a50 tx=0x7f5f440a7b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d39008f5dc3be29c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:07.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.787+0000 7f5f36ffd640 1 -- 172.21.15.31:0/4192890307 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f440a7320 con 0x55abbd6d04d0 2024-04-25T16:11:07.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.787+0000 7f5f36ffd640 1 -- 172.21.15.31:0/4192890307 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f440a74e0 con 0x55abbd6d04d0 2024-04-25T16:11:07.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.787+0000 7f5f36ffd640 1 -- 172.21.15.31:0/4192890307 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f440a77a0 con 0x55abbd6d04d0 2024-04-25T16:11:07.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.788+0000 7f5f5a189740 1 -- 172.21.15.31:0/4192890307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d04d0 msgr2=0x55abbdac4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.788+0000 7f5f5a189740 1 --2- 172.21.15.31:0/4192890307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d04d0 0x55abbdac4930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f5f440a3a50 tx=0x7f5f440a7b40 comp rx=0 tx=0).stop 2024-04-25T16:11:07.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.789+0000 7f5f5a189740 1 -- 172.21.15.31:0/4192890307 shutdown_connections 2024-04-25T16:11:07.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.789+0000 7f5f5a189740 1 --2- 172.21.15.31:0/4192890307 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6f9390 0x55abbdaa04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.789+0000 7f5f5a189740 1 --2- 172.21.15.31:0/4192890307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6d86b0 0x55abbdaacaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.789+0000 7f5f5a189740 1 --2- 172.21.15.31:0/4192890307 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d04d0 0x55abbdac4930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.789+0000 7f5f5a189740 1 -- 172.21.15.31:0/4192890307 >> 172.21.15.31:0/4192890307 conn(0x55abbd6ed4e0 msgr2=0x55abbdaa5730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.789+0000 7f5f5a189740 1 -- 172.21.15.31:0/4192890307 shutdown_connections 2024-04-25T16:11:07.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.789+0000 7f5f5a189740 1 -- 172.21.15.31:0/4192890307 wait complete. 2024-04-25T16:11:07.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.790+0000 7f5f5a189740 1 -- start start 2024-04-25T16:11:07.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.791+0000 7f5f5a189740 1 Processor -- start 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d04d0 0x55abbdaacaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d86b0 0x55abbdaa9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f5a189740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6f9390 0x55abbdaa9d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f5a189740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c1b2c60 con 0x55abbd6d04d0 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f5a189740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c1b2590 con 0x55abbd6d86b0 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f5a189740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5f4c118700 con 0x55abbd6f9390 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f5a189740 10 client.0 _ll_register_callbacks cb 0x7f5f3c006ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f51074640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d04d0 0x55abbdaacaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f51074640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d04d0 0x55abbdaacaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53572/0 (socket says 172.21.15.31:53572) 2024-04-25T16:11:07.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.792+0000 7f5f51074640 1 -- 172.21.15.31:0/2849164266 learned_addr learned my addr 172.21.15.31:0/2849164266 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:07.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.793+0000 7f5f50873640 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d86b0 0x55abbdaa9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.793+0000 7f5f51074640 1 -- 172.21.15.31:0/2849164266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6f9390 msgr2=0x55abbdaa9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.793+0000 7f5f51074640 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6f9390 0x55abbdaa9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.793+0000 7f5f51074640 1 -- 172.21.15.31:0/2849164266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d86b0 msgr2=0x55abbdaa9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:07.795 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.793+0000 7f5f51074640 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d86b0 0x55abbdaa9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.793+0000 7f5f51074640 1 -- 172.21.15.31:0/2849164266 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abbdb2ae40 con 0x55abbd6d04d0 2024-04-25T16:11:07.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.794+0000 7f5f51074640 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d04d0 0x55abbdaacaf0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5f44017470 tx=0x7f5f440a3e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e656b386f2fab22 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:07.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.794+0000 7f5f34ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:07.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.794+0000 7f5f34ff9640 1 -- 172.21.15.31:0/2849164266 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f440a77a0 con 0x55abbd6d04d0 2024-04-25T16:11:07.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.795+0000 7f5f34ff9640 1 -- 172.21.15.31:0/2849164266 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5f440df0c0 con 0x55abbd6d04d0 2024-04-25T16:11:07.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.795+0000 7f5f5a189740 1 -- 172.21.15.31:0/2849164266 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5f440a74e0 con 0x55abbd6d04d0 2024-04-25T16:11:07.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.795+0000 7f5f34ff9640 1 -- 172.21.15.31:0/2849164266 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5f440e2de0 con 0x55abbd6d04d0 2024-04-25T16:11:07.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.795+0000 7f5f5a189740 10 client.17580 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.795+0000 7f5f5a189740 1 -- 172.21.15.31:0/2849164266 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55abbdb2c0c0 con 0x55abbd6d04d0 2024-04-25T16:11:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.796+0000 7f5f34ff9640 1 -- 172.21.15.31:0/2849164266 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5f440a74e0 con 0x55abbd6d04d0 2024-04-25T16:11:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.796+0000 7f5f34ff9640 1 -- 172.21.15.31:0/2849164266 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5f440a3340 con 0x55abbd6d04d0 2024-04-25T16:11:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.796+0000 7f5f5a189740 10 client.17580 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.796+0000 7f5f5a189740 10 client.17580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.796+0000 7f5f5a189740 20 client.17580 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.796+0000 7f5f5a189740 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55abbd68d3a0 0x55abbdb2e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.796+0000 7f5f5a189740 4 client.17580 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:07.798 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.797+0000 7f5f5a189740 1 -- 172.21.15.31:0/2849164266 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55abbdac5010 con 0x55abbd68d3a0 2024-04-25T16:11:07.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.797+0000 7f5f50873640 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55abbd68d3a0 0x55abbdb2e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:07.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.805+0000 7f5f50873640 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55abbd68d3a0 0x55abbdb2e4c0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fed3e8a4196ae97 server_cookie=7e285e5b9ff6243f in_seq=0 out_seq=0 2024-04-25T16:11:07.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.805+0000 7f5f34ff9640 10 client.17580 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.807+0000 7f5f34ff9640 1 -- 172.21.15.31:0/2849164266 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55abbdac5010 con 0x55abbd68d3a0 2024-04-25T16:11:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.808+0000 7f5f34ff9640 10 client.17580 handle_command_reply: tid=0 2024-04-25T16:11:07.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.808+0000 7f5f5a189740 1 client.17580 shutdown 2024-04-25T16:11:07.810 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.808+0000 7f5f5a189740 1 -- 172.21.15.31:0/2849164266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d04d0 msgr2=0x55abbdaacaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.808+0000 7f5f5a189740 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d04d0 0x55abbdaacaf0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5f44017470 tx=0x7f5f440a3e90 comp rx=0 tx=0).stop 2024-04-25T16:11:07.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.809+0000 7f5f5a189740 1 -- 172.21.15.31:0/2849164266 shutdown_connections 2024-04-25T16:11:07.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.809+0000 7f5f5a189740 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55abbd68d3a0 0x55abbdb2e4c0 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.809+0000 7f5f5a189740 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55abbd6f9390 0x55abbdaa9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.809+0000 7f5f5a189740 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55abbd6d86b0 0x55abbdaa9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.809+0000 7f5f5a189740 1 --2- 172.21.15.31:0/2849164266 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55abbd6d04d0 0x55abbdaacaf0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.809+0000 7f5f5a189740 1 -- 172.21.15.31:0/2849164266 >> 172.21.15.31:0/2849164266 conn(0x55abbd6ed4e0 msgr2=0x55abbdaa04c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.809+0000 7f5f34ff9640 0 client.17580 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:07.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.814+0000 7f5f5a189740 1 -- 172.21.15.31:0/2849164266 shutdown_connections 2024-04-25T16:11:07.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.814+0000 7f5f5a189740 1 -- 172.21.15.31:0/2849164266 wait complete. 2024-04-25T16:11:07.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.814+0000 7f5f5a189740 20 client.17580 trim_cache size 0 max 16384 2024-04-25T16:11:07.816 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:07.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.816+0000 7f5f532d8640 1 -- 172.21.15.31:0/694984190 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f380afcf0 msgr2=0x7f5f380b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.817 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.816+0000 7f5f532d8640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f380afcf0 0x7f5f380b2200 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5f480096c0 tx=0x7f5f48009260 comp rx=0 tx=0).stop 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.816+0000 7f5f532d8640 1 -- 172.21.15.31:0/694984190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f4c10a3e0 msgr2=0x7f5f4c1a8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.816+0000 7f5f532d8640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f4c10a3e0 0x7f5f4c1a8e70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400ebc0 tx=0x7f5f4400c6a0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.817+0000 7f5f532d8640 1 -- 172.21.15.31:0/694984190 shutdown_connections 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.817+0000 7f5f532d8640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5f380afcf0 0x7f5f380b2200 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.817+0000 7f5f532d8640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f4c10fef0 0x7f5f4c1b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.817+0000 7f5f532d8640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5f4c10b1d0 0x7f5f4c1a93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.817+0000 7f5f532d8640 1 --2- 172.21.15.31:0/694984190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5f4c10a3e0 0x7f5f4c1a8e70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.817+0000 7f5f532d8640 1 -- 172.21.15.31:0/694984190 >> 172.21.15.31:0/694984190 conn(0x7f5f4c070400 msgr2=0x7f5f4c10bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.817+0000 7f5f532d8640 1 -- 172.21.15.31:0/694984190 shutdown_connections 2024-04-25T16:11:07.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:07.818+0000 7f5f532d8640 1 -- 172.21.15.31:0/694984190 wait complete. 2024-04-25T16:11:08.860 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:11:08.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:11:08.860 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:11:09.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.304+0000 7fba1f59e640 1 -- 172.21.15.31:0/1543865698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 msgr2=0x7fba2010b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:09.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.304+0000 7fba1f59e640 1 --2- 172.21.15.31:0/1543865698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 0x7fba2010b5a0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fba18009ca0 tx=0x7fba1802a8f0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.304+0000 7fba1f59e640 1 -- 172.21.15.31:0/1543865698 shutdown_connections 2024-04-25T16:11:09.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.304+0000 7fba1f59e640 1 --2- 172.21.15.31:0/1543865698 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2010fe90 0x7fba201102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.304+0000 7fba1f59e640 1 --2- 172.21.15.31:0/1543865698 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 0x7fba2010b5a0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.304+0000 7fba1f59e640 1 --2- 172.21.15.31:0/1543865698 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba2010a340 0x7fba2010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.304+0000 7fba1f59e640 1 -- 172.21.15.31:0/1543865698 >> 172.21.15.31:0/1543865698 conn(0x7fba20070320 msgr2=0x7fba200727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.323+0000 7fba1f59e640 1 -- 172.21.15.31:0/1543865698 shutdown_connections 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.323+0000 7fba1f59e640 1 -- 172.21.15.31:0/1543865698 wait complete. 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.324+0000 7fba1f59e640 1 Processor -- start 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.324+0000 7fba1f59e640 1 -- start start 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.325+0000 7fba1f59e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba2010a340 0x7fba201a8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.325+0000 7fba1f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 0x7fba201a9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.325+0000 7fba1f59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2010fe90 0x7fba201b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.325+0000 7fba1f59e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fba2011e2e0 con 0x7fba2010fe90 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.325+0000 7fba1f59e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fba2011e160 con 0x7fba2010a340 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.325+0000 7fba1f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fba2011e460 con 0x7fba2010b190 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.327+0000 7fba1dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 0x7fba201a9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.327+0000 7fba1dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 0x7fba201a9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34404/0 (socket says 172.21.15.31:34404) 2024-04-25T16:11:09.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.327+0000 7fba1dd9b640 1 -- 172.21.15.31:0/875195248 learned_addr learned my addr 172.21.15.31:0/875195248 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:09.330 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.327+0000 7fba1ed9d640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2010fe90 0x7fba201b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.328+0000 7fba1dd9b640 1 -- 172.21.15.31:0/875195248 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba2010a340 msgr2=0x7fba201a8f00 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:11:09.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.328+0000 7fba1dd9b640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba2010a340 0x7fba201a8f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.328+0000 7fba1dd9b640 1 -- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2010fe90 msgr2=0x7fba201b03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:09.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.328+0000 7fba1dd9b640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2010fe90 0x7fba201b03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.328+0000 7fba1dd9b640 1 -- 172.21.15.31:0/875195248 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba20111c90 con 0x7fba2010b190 2024-04-25T16:11:09.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.329+0000 7fba1dd9b640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 0x7fba201a9440 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fba1802aee0 tx=0x7fba180317f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:09.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.330+0000 7fba0f7fe640 1 -- 172.21.15.31:0/875195248 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1802fb10 con 0x7fba2010b190 2024-04-25T16:11:09.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.330+0000 7fba1f59e640 1 -- 172.21.15.31:0/875195248 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba20111f20 con 0x7fba2010b190 2024-04-25T16:11:09.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.330+0000 7fba1f59e640 1 -- 172.21.15.31:0/875195248 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba201124f0 con 0x7fba2010b190 2024-04-25T16:11:09.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.330+0000 7fba0f7fe640 1 -- 172.21.15.31:0/875195248 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba1802d070 con 0x7fba2010b190 2024-04-25T16:11:09.334 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.330+0000 7fba0f7fe640 1 -- 172.21.15.31:0/875195248 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba180466e0 con 0x7fba2010b190 2024-04-25T16:11:09.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.334+0000 7fba2b97b740 1 Processor -- start 2024-04-25T16:11:09.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.334+0000 7fba2b97b740 1 -- start start 2024-04-25T16:11:09.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.334+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a889d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.336+0000 7fba0f7fe640 1 -- 172.21.15.31:0/875195248 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fba18046880 con 0x7fba2010b190 2024-04-25T16:11:09.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.337+0000 7fba0f7fe640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fba080afcf0 0x7fba080b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.337+0000 7fba0f7fe640 1 -- 172.21.15.31:0/875195248 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fba1802fcb0 con 0x7fba2010b190 2024-04-25T16:11:09.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.338+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a889dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.338+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 0x5648a88a4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.338+0000 7fba2b97b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5648a8802b00 con 0x5648a84fc410 2024-04-25T16:11:09.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.338+0000 7fba2b97b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5648a879e920 con 0x5648a84db730 2024-04-25T16:11:09.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.338+0000 7fba2b97b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5648a87feac0 con 0x5648a84d3550 2024-04-25T16:11:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.342+0000 7fba1ed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 0x5648a88a4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.342+0000 7fba1dd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a889dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.342+0000 7fba1ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a889d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.342+0000 7fba1ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a889d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53598/0 (socket says 172.21.15.31:53598) 2024-04-25T16:11:09.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.342+0000 7fba1ed9d640 1 -- 172.21.15.31:0/1005239201 learned_addr learned my addr 172.21.15.31:0/1005239201 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:09.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.344+0000 7fba1e59c640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fba080afcf0 0x7fba080b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.346+0000 7fba1e59c640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fba080afcf0 0x7fba080b2200 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fba20109c10 tx=0x7fba14006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.355+0000 7fba1ed9d640 1 -- 172.21.15.31:0/1005239201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 msgr2=0x5648a889dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.355+0000 7fba1ed9d640 1 --2- 172.21.15.31:0/1005239201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a889dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.355+0000 7fba1ed9d640 1 -- 172.21.15.31:0/1005239201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 msgr2=0x5648a88a4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:09.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.355+0000 7fba1ed9d640 1 --2- 172.21.15.31:0/1005239201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 0x5648a88a4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.355+0000 7fba1ed9d640 1 -- 172.21.15.31:0/1005239201 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648a85dc7e0 con 0x5648a84fc410 2024-04-25T16:11:09.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.367+0000 7fba1ed9d640 1 --2- 172.21.15.31:0/1005239201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a889d660 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fba10010970 tx=0x7fba10005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57a5ac3c9d081f7d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:09.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.373+0000 7fba0cff9640 1 -- 172.21.15.31:0/1005239201 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10016030 con 0x5648a84fc410 2024-04-25T16:11:09.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.373+0000 7fba0cff9640 1 -- 172.21.15.31:0/1005239201 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba1001e3b0 con 0x5648a84fc410 2024-04-25T16:11:09.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.373+0000 7fba0cff9640 1 -- 172.21.15.31:0/1005239201 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba10019040 con 0x5648a84fc410 2024-04-25T16:11:09.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.376+0000 7fba2b97b740 1 -- 172.21.15.31:0/1005239201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 msgr2=0x5648a889d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:09.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.376+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1005239201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a889d660 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fba10010970 tx=0x7fba10005f90 comp rx=0 tx=0).stop 2024-04-25T16:11:09.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.381+0000 7fba2b97b740 1 -- 172.21.15.31:0/1005239201 shutdown_connections 2024-04-25T16:11:09.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.381+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1005239201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 0x5648a88a4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.381+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1005239201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a889dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.381+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1005239201 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a889d660 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.381+0000 7fba2b97b740 1 -- 172.21.15.31:0/1005239201 >> 172.21.15.31:0/1005239201 conn(0x5648a84f0560 msgr2=0x5648a8898ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:09.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.390+0000 7fba2b97b740 1 -- 172.21.15.31:0/1005239201 shutdown_connections 2024-04-25T16:11:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.394+0000 7fba2b97b740 1 -- 172.21.15.31:0/1005239201 wait complete. 2024-04-25T16:11:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.395+0000 7fba2b97b740 1 -- start start 2024-04-25T16:11:09.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.395+0000 7fba2b97b740 1 Processor -- start 2024-04-25T16:11:09.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.396+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88a7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.396+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 0x5648a88a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.396+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a88aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.396+0000 7fba2b97b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5648a88a5530 con 0x5648a84fc410 2024-04-25T16:11:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.396+0000 7fba2b97b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5648a879c5d0 con 0x5648a84db730 2024-04-25T16:11:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.396+0000 7fba2b97b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5648a88a56b0 con 0x5648a84d3550 2024-04-25T16:11:09.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.396+0000 7fba2b97b740 10 client.0 _ll_register_callbacks cb 0x5648a878a080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:09.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.403+0000 7fba1e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88a7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.403+0000 7fba1ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a88aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.403+0000 7fba1ed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a88aeab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53600/0 (socket says 172.21.15.31:53600) 2024-04-25T16:11:09.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.403+0000 7fba1ed9d640 1 -- 172.21.15.31:0/1715043903 learned_addr learned my addr 172.21.15.31:0/1715043903 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:09.405 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.403+0000 7fba1dd9b640 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 0x5648a88a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.414+0000 7fba1e59c640 1 -- 172.21.15.31:0/1715043903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 msgr2=0x5648a88a7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:09.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.414+0000 7fba1e59c640 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 0x5648a88a7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.414+0000 7fba1e59c640 1 -- 172.21.15.31:0/1715043903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 msgr2=0x5648a88aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:09.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.414+0000 7fba1e59c640 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a88aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.414+0000 7fba1e59c640 1 -- 172.21.15.31:0/1715043903 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648a863cf70 con 0x5648a84d3550 2024-04-25T16:11:09.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.421+0000 7fba1e59c640 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88a7610 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fba20074a30 tx=0x7fba140046e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a615e78737f623b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:09.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.422+0000 7fba05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:09.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.428+0000 7fba05ffb640 1 -- 172.21.15.31:0/1715043903 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1401d5f0 con 0x5648a84d3550 2024-04-25T16:11:09.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.429+0000 7fba2b97b740 1 -- 172.21.15.31:0/1715043903 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba1001e3b0 con 0x5648a84d3550 2024-04-25T16:11:09.430 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.429+0000 7fba05ffb640 1 -- 172.21.15.31:0/1715043903 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fba1401d790 con 0x5648a84d3550 2024-04-25T16:11:09.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.431+0000 7fba05ffb640 1 -- 172.21.15.31:0/1715043903 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba1401d990 con 0x5648a84d3550 2024-04-25T16:11:09.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.432+0000 7fba2b97b740 10 client.27259 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:09.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.432+0000 7fba2b97b740 1 -- 172.21.15.31:0/1715043903 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5648a84d4170 con 0x5648a84d3550 2024-04-25T16:11:09.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.435+0000 7fba05ffb640 1 -- 172.21.15.31:0/1715043903 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba1001e3b0 con 0x5648a84d3550 2024-04-25T16:11:09.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.437+0000 7fba05ffb640 1 -- 172.21.15.31:0/1715043903 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fba14012070 con 0x5648a84d3550 2024-04-25T16:11:09.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.437+0000 7fba2b97b740 10 client.27259 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:09.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.437+0000 7fba2b97b740 10 client.27259 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:09.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.437+0000 7fba2b97b740 20 client.27259 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:09.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.437+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5648a84bb1f0 0x5648a88b5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.438+0000 7fba2b97b740 4 client.27259 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:09.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.438+0000 7fba2b97b740 1 -- 172.21.15.31:0/1715043903 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5648a8810710 con 0x5648a84bb1f0 2024-04-25T16:11:09.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.442+0000 7fba1dd9b640 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5648a84bb1f0 0x5648a88b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.455 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:09 smithi031 ceph-mon[29140]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 917 MiB/s wr, 2.05k op/s 2024-04-25T16:11:09.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.462+0000 7fba1dd9b640 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5648a84bb1f0 0x5648a88b5480 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae59824ab38e8910 server_cookie=14804af493711edb in_seq=0 out_seq=0 2024-04-25T16:11:09.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.462+0000 7fba05ffb640 10 client.27259 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:09.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.472+0000 7fba05ffb640 1 -- 172.21.15.31:0/1715043903 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5648a8810710 con 0x5648a84bb1f0 2024-04-25T16:11:09.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.472+0000 7fba05ffb640 10 client.27259 handle_command_reply: tid=0 2024-04-25T16:11:09.474 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.473+0000 7fba2b97b740 1 client.27259 shutdown 2024-04-25T16:11:09.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.474+0000 7fba2b97b740 1 -- 172.21.15.31:0/1715043903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 msgr2=0x5648a88a7610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:09.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.474+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88a7610 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fba20074a30 tx=0x7fba140046e0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.481+0000 7fba2b97b740 1 -- 172.21.15.31:0/1715043903 shutdown_connections 2024-04-25T16:11:09.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.481+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5648a84bb1f0 0x5648a88b5480 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.481+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a88aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.481+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84db730 0x5648a88a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.481+0000 7fba2b97b740 1 --2- 172.21.15.31:0/1715043903 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88a7610 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.481+0000 7fba2b97b740 1 -- 172.21.15.31:0/1715043903 >> 172.21.15.31:0/1715043903 conn(0x5648a84f0560 msgr2=0x5648a8897f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:09.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.481+0000 7fba05ffb640 0 client.27259 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:09.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.483+0000 7fba2b97b740 1 -- 172.21.15.31:0/1715043903 shutdown_connections 2024-04-25T16:11:09.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.485+0000 7fba2b97b740 1 -- 172.21.15.31:0/1715043903 wait complete. 2024-04-25T16:11:09.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.490+0000 7fba2b97b740 20 client.27259 trim_cache size 0 max 16384 2024-04-25T16:11:09.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.569+0000 7fba2b97b740 1 Processor -- start 2024-04-25T16:11:09.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.570+0000 7fba2b97b740 1 -- start start 2024-04-25T16:11:09.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.570+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84d3550 0x5648a88a2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.571+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84db730 0x5648a8897c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.571+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a88aea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.574 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.573+0000 7fba1e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84d3550 0x5648a88a2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.573+0000 7fba1e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84d3550 0x5648a88a2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53616/0 (socket says 172.21.15.31:53616) 2024-04-25T16:11:09.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.573+0000 7fba1e59c640 1 -- 172.21.15.31:0/701084278 learned_addr learned my addr 172.21.15.31:0/701084278 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:09.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.573+0000 7fba2b97b740 1 -- 172.21.15.31:0/701084278 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fba0000e3c0 con 0x5648a84fc410 2024-04-25T16:11:09.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.573+0000 7fba2b97b740 1 -- 172.21.15.31:0/701084278 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fba0000ff00 con 0x5648a84d3550 2024-04-25T16:11:09.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.573+0000 7fba2b97b740 1 -- 172.21.15.31:0/701084278 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fba0000ea90 con 0x5648a84db730 2024-04-25T16:11:09.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.574+0000 7fba1ed9d640 1 --2- 172.21.15.31:0/701084278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a88aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.575+0000 7fba1dd9b640 1 --2- 172.21.15.31:0/701084278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84db730 0x5648a8897c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.575+0000 7fba1e59c640 1 -- 172.21.15.31:0/701084278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84db730 msgr2=0x5648a8897c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.575+0000 7fba1e59c640 1 --2- 172.21.15.31:0/701084278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84db730 0x5648a8897c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.575+0000 7fba1e59c640 1 -- 172.21.15.31:0/701084278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 msgr2=0x5648a88aea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.575+0000 7fba1e59c640 1 --2- 172.21.15.31:0/701084278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a88aea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.577 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.575+0000 7fba1e59c640 1 -- 172.21.15.31:0/701084278 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5648a8899b80 con 0x5648a84d3550 2024-04-25T16:11:09.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.576+0000 7fba1e59c640 1 --2- 172.21.15.31:0/701084278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84d3550 0x5648a88a2090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fba14009f40 tx=0x7fba14012fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ea7c4c2d62f5c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:09.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.577+0000 7fba07fff640 1 -- 172.21.15.31:0/701084278 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba1401db30 con 0x5648a84d3550 2024-04-25T16:11:09.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.578+0000 7fba07fff640 1 -- 172.21.15.31:0/701084278 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba14022070 con 0x5648a84d3550 2024-04-25T16:11:09.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.579+0000 7fba2b97b740 1 -- 172.21.15.31:0/701084278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84d3550 msgr2=0x5648a88a2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:09.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.579+0000 7fba2b97b740 1 --2- 172.21.15.31:0/701084278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84d3550 0x5648a88a2090 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fba14009f40 tx=0x7fba14012fe0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.594+0000 7fba2b97b740 1 -- 172.21.15.31:0/701084278 shutdown_connections 2024-04-25T16:11:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.594+0000 7fba2b97b740 1 --2- 172.21.15.31:0/701084278 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84fc410 0x5648a88aea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.594+0000 7fba2b97b740 1 --2- 172.21.15.31:0/701084278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84db730 0x5648a8897c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.594+0000 7fba2b97b740 1 --2- 172.21.15.31:0/701084278 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84d3550 0x5648a88a2090 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.594+0000 7fba2b97b740 1 -- 172.21.15.31:0/701084278 >> 172.21.15.31:0/701084278 conn(0x5648a84f0560 msgr2=0x5648a8897560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.595+0000 7fba2b97b740 1 -- 172.21.15.31:0/701084278 shutdown_connections 2024-04-25T16:11:09.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.595+0000 7fba2b97b740 1 -- 172.21.15.31:0/701084278 wait complete. 2024-04-25T16:11:09.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.596+0000 7fba2b97b740 1 -- start start 2024-04-25T16:11:09.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.597+0000 7fba2b97b740 1 Processor -- start 2024-04-25T16:11:09.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.598+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.598+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84db730 0x5648a88ac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.598+0000 7fba2b97b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84fc410 0x5648a891b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.598+0000 7fba2b97b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fba0000f830 con 0x5648a84db730 2024-04-25T16:11:09.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.599+0000 7fba2b97b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fba0000f160 con 0x5648a84fc410 2024-04-25T16:11:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.599+0000 7fba2b97b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fba0000c450 con 0x5648a84d3550 2024-04-25T16:11:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.599+0000 7fba1e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.599+0000 7fba1e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88ac2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34424/0 (socket says 172.21.15.31:34424) 2024-04-25T16:11:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.599+0000 7fba1e59c640 1 -- 172.21.15.31:0/3119013204 learned_addr learned my addr 172.21.15.31:0/3119013204 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.599+0000 7fba2b97b740 10 client.0 _ll_register_callbacks cb 0x7fba14013ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.599+0000 7fba1ed9d640 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84fc410 0x5648a891b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.600+0000 7fba1e59c640 1 -- 172.21.15.31:0/3119013204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84fc410 msgr2=0x5648a891b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.600+0000 7fba1e59c640 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84fc410 0x5648a891b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.601 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.600+0000 7fba1e59c640 1 -- 172.21.15.31:0/3119013204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84db730 msgr2=0x5648a88ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:09.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.600+0000 7fba1e59c640 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84db730 0x5648a88ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.602 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.600+0000 7fba1e59c640 1 -- 172.21.15.31:0/3119013204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5648a891bd90 con 0x5648a84d3550 2024-04-25T16:11:09.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.611+0000 7fba1e59c640 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88ac2f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba14034040 tx=0x7fba1401d620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e76fd3bf9fb1f363 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:09.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.611+0000 7fba05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.617+0000 7fba05ffb640 1 -- 172.21.15.31:0/3119013204 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba14004240 con 0x5648a84d3550 2024-04-25T16:11:09.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.617+0000 7fba05ffb640 1 -- 172.21.15.31:0/3119013204 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fba140043e0 con 0x5648a84d3550 2024-04-25T16:11:09.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:09 smithi063 ceph-mon[32413]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 917 MiB/s wr, 2.05k op/s 2024-04-25T16:11:09.620 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.619+0000 7fba2b97b740 1 -- 172.21.15.31:0/3119013204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba14022070 con 0x5648a84d3550 2024-04-25T16:11:09.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.620+0000 7fba05ffb640 1 -- 172.21.15.31:0/3119013204 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fba14004620 con 0x5648a84d3550 2024-04-25T16:11:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.624+0000 7fba2b97b740 10 client.27262 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:09.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.624+0000 7fba2b97b740 1 -- 172.21.15.31:0/3119013204 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5648a891d050 con 0x5648a84d3550 2024-04-25T16:11:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.627+0000 7fba05ffb640 1 -- 172.21.15.31:0/3119013204 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fba14022070 con 0x5648a84d3550 2024-04-25T16:11:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.627+0000 7fba05ffb640 1 -- 172.21.15.31:0/3119013204 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fba14012040 con 0x5648a84d3550 2024-04-25T16:11:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.628+0000 7fba2b97b740 10 client.27262 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.628+0000 7fba2b97b740 10 client.27262 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.628+0000 7fba2b97b740 20 client.27262 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.628+0000 7fba2b97b740 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5648a8490630 0x5648a891fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:09.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.628+0000 7fba2b97b740 4 client.27262 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:11:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.628+0000 7fba2b97b740 1 -- 172.21.15.31:0/3119013204 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5648a880a580 con 0x5648a8490630 2024-04-25T16:11:09.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.628+0000 7fba1dd9b640 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5648a8490630 0x5648a891fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.635+0000 7fba1dd9b640 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5648a8490630 0x5648a891fa30 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=474703abf169636e server_cookie=57bcc8fc776caf15 in_seq=0 out_seq=0 2024-04-25T16:11:09.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.636+0000 7fba05ffb640 10 client.27262 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:09.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:09 smithi186 ceph-mon[33258]: pgmap v646: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 917 MiB/s wr, 2.05k op/s 2024-04-25T16:11:09.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.703+0000 7fba05ffb640 1 -- 172.21.15.31:0/3119013204 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5648a880a580 con 0x5648a8490630 2024-04-25T16:11:09.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.703+0000 7fba05ffb640 10 client.27262 handle_command_reply: tid=0 2024-04-25T16:11:09.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.704+0000 7fba2b97b740 1 client.27262 shutdown 2024-04-25T16:11:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.709+0000 7fba2b97b740 1 -- 172.21.15.31:0/3119013204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 msgr2=0x5648a88ac2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:09.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.709+0000 7fba2b97b740 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88ac2f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fba14034040 tx=0x7fba1401d620 comp rx=0 tx=0).stop 2024-04-25T16:11:09.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.709+0000 7fba2b97b740 1 -- 172.21.15.31:0/3119013204 shutdown_connections 2024-04-25T16:11:09.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.709+0000 7fba2b97b740 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5648a8490630 0x5648a891fa30 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.710+0000 7fba05ffb640 0 client.27262 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:09.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.710+0000 7fba2b97b740 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5648a84fc410 0x5648a891b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.710+0000 7fba2b97b740 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5648a84db730 0x5648a88ac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.710+0000 7fba2b97b740 1 --2- 172.21.15.31:0/3119013204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5648a84d3550 0x5648a88ac2f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.710+0000 7fba2b97b740 1 -- 172.21.15.31:0/3119013204 >> 172.21.15.31:0/3119013204 conn(0x5648a84f0560 msgr2=0x5648a8897560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:09.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.715+0000 7fba2b97b740 1 -- 172.21.15.31:0/3119013204 shutdown_connections 2024-04-25T16:11:09.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.715+0000 7fba2b97b740 1 -- 172.21.15.31:0/3119013204 wait complete. 2024-04-25T16:11:09.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.720+0000 7fba2b97b740 20 client.27262 trim_cache size 0 max 16384 2024-04-25T16:11:09.722 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:11:09.722 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:11:09.722 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "27f9b6c8-512a-4488-8565-54eadfad4bf4", 2024-04-25T16:11:09.723 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:11:09.723 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.721+0000 7fba1f59e640 1 -- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fba080afcf0 msgr2=0x7fba080b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.721+0000 7fba1f59e640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fba080afcf0 0x7fba080b2200 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fba20109c10 tx=0x7fba14006bd0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.721+0000 7fba1f59e640 1 -- 172.21.15.31:0/875195248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 msgr2=0x7fba201a9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.721+0000 7fba1f59e640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 0x7fba201a9440 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fba1802aee0 tx=0x7fba180317f0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.722+0000 7fba1f59e640 1 -- 172.21.15.31:0/875195248 shutdown_connections 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.722+0000 7fba1f59e640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fba080afcf0 0x7fba080b2200 secure :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fba20109c10 tx=0x7fba14006bd0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.722+0000 7fba1f59e640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fba2010fe90 0x7fba201b03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.722+0000 7fba1f59e640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fba2010b190 0x7fba201a9440 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.722+0000 7fba1f59e640 1 --2- 172.21.15.31:0/875195248 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba2010a340 0x7fba201a8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:09.726 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.722+0000 7fba1f59e640 1 -- 172.21.15.31:0/875195248 >> 172.21.15.31:0/875195248 conn(0x7fba20070320 msgr2=0x7fba2010be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:09.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.723+0000 7fba1f59e640 1 -- 172.21.15.31:0/875195248 shutdown_connections 2024-04-25T16:11:09.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:09.723+0000 7fba1f59e640 1 -- 172.21.15.31:0/875195248 wait complete. 2024-04-25T16:11:09.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:11:09.782 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:11:10.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.243+0000 7f72bf813640 1 -- 172.21.15.31:0/4001249955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72b810b310 msgr2=0x7f72b810b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:10.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.243+0000 7f72bf813640 1 --2- 172.21.15.31:0/4001249955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72b810b310 0x7f72b810b720 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009ca0 tx=0x7f72b002a8d0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.244+0000 7f72bf813640 1 -- 172.21.15.31:0/4001249955 shutdown_connections 2024-04-25T16:11:10.246 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.244+0000 7f72bf813640 1 --2- 172.21.15.31:0/4001249955 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72b8110030 0x7f72b8110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.244+0000 7f72bf813640 1 --2- 172.21.15.31:0/4001249955 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72b810b310 0x7f72b810b720 secure :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009ca0 tx=0x7f72b002a8d0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.244+0000 7f72bf813640 1 --2- 172.21.15.31:0/4001249955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b810a420 0x7f72b810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.244+0000 7f72bf813640 1 -- 172.21.15.31:0/4001249955 >> 172.21.15.31:0/4001249955 conn(0x7f72b80704e0 msgr2=0x7f72b8072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:10.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.249+0000 7f72bf813640 1 -- 172.21.15.31:0/4001249955 shutdown_connections 2024-04-25T16:11:10.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.249+0000 7f72bf813640 1 -- 172.21.15.31:0/4001249955 wait complete. 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.250+0000 7f72bf813640 1 Processor -- start 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.250+0000 7f72bf813640 1 -- start start 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.251+0000 7f72bf813640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72b810a420 0x7f72b81a9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.251+0000 7f72bf813640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72b8110030 0x7f72b81a9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.251+0000 7f72bf813640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b81ae640 0x7f72b81b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.251+0000 7f72bf813640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f72b811e540 con 0x7f72b8110030 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.251+0000 7f72bf813640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f72b811e3c0 con 0x7f72b810a420 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.251+0000 7f72bf813640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72b811e6c0 con 0x7f72b81ae640 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bddb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b81ae640 0x7f72b81b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bddb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b81ae640 0x7f72b81b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34446/0 (socket says 172.21.15.31:34446) 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bddb0640 1 -- 172.21.15.31:0/3414171425 learned_addr learned my addr 172.21.15.31:0/3414171425 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bcdae640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72b8110030 0x7f72b81a9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bddb0640 1 -- 172.21.15.31:0/3414171425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72b810a420 msgr2=0x7f72b81a9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bd5af640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72b810a420 0x7f72b81a9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bddb0640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72b810a420 0x7f72b81a9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bddb0640 1 -- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72b8110030 msgr2=0x7f72b81a9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bddb0640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72b8110030 0x7f72b81a9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.253+0000 7f72bddb0640 1 -- 172.21.15.31:0/3414171425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b81b1190 con 0x7f72b81ae640 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.254+0000 7f72bcdae640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72b8110030 0x7f72b81a9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.254+0000 7f72bd5af640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72b810a420 0x7f72b81a9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:11:10.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.254+0000 7f72bddb0640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b81ae640 0x7f72b81b0a60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f72a802f710 tx=0x7f72a802fbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.258+0000 7f72ae7fc640 1 -- 172.21.15.31:0/3414171425 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8039070 con 0x7f72b81ae640 2024-04-25T16:11:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.258+0000 7f72ae7fc640 1 -- 172.21.15.31:0/3414171425 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72a802fe20 con 0x7f72b81ae640 2024-04-25T16:11:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.260+0000 7f72ae7fc640 1 -- 172.21.15.31:0/3414171425 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8004c60 con 0x7f72b81ae640 2024-04-25T16:11:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.262+0000 7f72bf813640 1 -- 172.21.15.31:0/3414171425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b8074e30 con 0x7f72b81ae640 2024-04-25T16:11:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.262+0000 7f72bf813640 1 -- 172.21.15.31:0/3414171425 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b8075380 con 0x7f72b81ae640 2024-04-25T16:11:10.269 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.267+0000 7f72c67a0740 1 Processor -- start 2024-04-25T16:11:10.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.275+0000 7f72ae7fc640 1 -- 172.21.15.31:0/3414171425 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f72a8004e00 con 0x7f72b81ae640 2024-04-25T16:11:10.277 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.275+0000 7f72c67a0740 1 -- start start 2024-04-25T16:11:10.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.275+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 0x55712623ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.275+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e724d0 0x55712623cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.275+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 0x55712623dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.275+0000 7f72c67a0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5571261a1d80 con 0x557125e9b390 2024-04-25T16:11:10.278 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.276+0000 7f72c67a0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55712615b8b0 con 0x557125e7a6b0 2024-04-25T16:11:10.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.276+0000 7f72c67a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55712619dd40 con 0x557125e724d0 2024-04-25T16:11:10.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.276+0000 7f72bcdae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 0x55712623ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.276+0000 7f72bcdae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 0x55712623ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53644/0 (socket says 172.21.15.31:53644) 2024-04-25T16:11:10.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.276+0000 7f72bcdae640 1 -- 172.21.15.31:0/607281731 learned_addr learned my addr 172.21.15.31:0/607281731 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:10.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.276+0000 7f72bcdae640 1 --2- 172.21.15.31:0/607281731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 0x55712623dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.277+0000 7f72ae7fc640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72a40afa20 0x7f72a40b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.277+0000 7f72bcdae640 1 -- 172.21.15.31:0/607281731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e724d0 msgr2=0x55712623cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:10.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.277+0000 7f72bcdae640 1 --2- 172.21.15.31:0/607281731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e724d0 0x55712623cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.277+0000 7f72bcdae640 1 -- 172.21.15.31:0/607281731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 msgr2=0x55712623dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:10.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.277+0000 7f72bcdae640 1 --2- 172.21.15.31:0/607281731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 0x55712623dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.277+0000 7f72bcdae640 1 -- 172.21.15.31:0/607281731 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55712612ca40 con 0x557125e9b390 2024-04-25T16:11:10.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.278+0000 7f72ae7fc640 1 -- 172.21.15.31:0/3414171425 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f72a8100940 con 0x7f72b81ae640 2024-04-25T16:11:10.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.278+0000 7f72bcdae640 1 --2- 172.21.15.31:0/607281731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 0x55712623ca60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f72b0000f80 tx=0x7f72b0009450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c93584870c4cc2e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:10.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.279+0000 7f72a37fe640 1 -- 172.21.15.31:0/607281731 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b0031a30 con 0x557125e9b390 2024-04-25T16:11:10.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.279+0000 7f72bddb0640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72a40afa20 0x7f72a40b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.281 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.279+0000 7f72a37fe640 1 -- 172.21.15.31:0/607281731 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72b0036040 con 0x557125e9b390 2024-04-25T16:11:10.282 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.279+0000 7f72a37fe640 1 -- 172.21.15.31:0/607281731 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b0003c60 con 0x557125e9b390 2024-04-25T16:11:10.283 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.281+0000 7f72c67a0740 1 -- 172.21.15.31:0/607281731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 msgr2=0x55712623ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:10.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.281+0000 7f72c67a0740 1 --2- 172.21.15.31:0/607281731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 0x55712623ca60 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f72b0000f80 tx=0x7f72b0009450 comp rx=0 tx=0).stop 2024-04-25T16:11:10.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.281+0000 7f72bddb0640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72a40afa20 0x7f72a40b1f30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f72a80c60a0 tx=0x7f72a8009cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:10.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.281+0000 7f72c67a0740 1 -- 172.21.15.31:0/607281731 shutdown_connections 2024-04-25T16:11:10.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.281+0000 7f72c67a0740 1 --2- 172.21.15.31:0/607281731 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 0x55712623dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.281+0000 7f72c67a0740 1 --2- 172.21.15.31:0/607281731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e724d0 0x55712623cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.281+0000 7f72c67a0740 1 --2- 172.21.15.31:0/607281731 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 0x55712623ca60 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.281+0000 7f72c67a0740 1 -- 172.21.15.31:0/607281731 >> 172.21.15.31:0/607281731 conn(0x557125e8f4e0 msgr2=0x557126238270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:10.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.290+0000 7f72c67a0740 1 -- 172.21.15.31:0/607281731 shutdown_connections 2024-04-25T16:11:10.292 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.291+0000 7f72c67a0740 1 -- 172.21.15.31:0/607281731 wait complete. 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.291+0000 7f72c67a0740 1 -- start start 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.292+0000 7f72c67a0740 1 Processor -- start 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e7a6b0 0x557126257ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 0x557126258020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e59f60 0x55712625ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72c67a0740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55712623e720 con 0x557125e9b390 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72c67a0740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557126128d10 con 0x557125e59f60 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72c67a0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55712623e8a0 con 0x557125e7a6b0 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72c67a0740 10 client.0 _ll_register_callbacks cb 0x557126155210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72bcdae640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e7a6b0 0x557126257ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72bcdae640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e59f60 0x55712625ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72bcdae640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e7a6b0 0x557126257ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34454/0 (socket says 172.21.15.31:34454) 2024-04-25T16:11:10.296 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.293+0000 7f72bcdae640 1 -- 172.21.15.31:0/1778283352 learned_addr learned my addr 172.21.15.31:0/1778283352 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:10.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.294+0000 7f72bcdae640 1 -- 172.21.15.31:0/1778283352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e59f60 msgr2=0x55712625ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:10.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.294+0000 7f72bcdae640 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e59f60 0x55712625ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.294+0000 7f72bcdae640 1 -- 172.21.15.31:0/1778283352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 msgr2=0x557126258020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:10.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.294+0000 7f72bcdae640 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 0x557126258020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.294+0000 7f72bcdae640 1 -- 172.21.15.31:0/1778283352 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557125f7d7d0 con 0x557125e7a6b0 2024-04-25T16:11:10.297 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.294+0000 7f72bcdae640 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e7a6b0 0x557126257ae0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f72b00307c0 tx=0x7f72b00093e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dde99f90b232c708 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.295+0000 7f72a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.295+0000 7f72a0ff9640 1 -- 172.21.15.31:0/1778283352 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b0054060 con 0x557125e7a6b0 2024-04-25T16:11:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.295+0000 7f72a0ff9640 1 -- 172.21.15.31:0/1778283352 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f72b003e630 con 0x557125e7a6b0 2024-04-25T16:11:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.295+0000 7f72c67a0740 1 -- 172.21.15.31:0/1778283352 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b0036040 con 0x557125e7a6b0 2024-04-25T16:11:10.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.296+0000 7f72a0ff9640 1 -- 172.21.15.31:0/1778283352 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f72b0030ec0 con 0x557125e7a6b0 2024-04-25T16:11:10.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.300+0000 7f72c67a0740 10 client.27268 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:10.304 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.300+0000 7f72c67a0740 1 -- 172.21.15.31:0/1778283352 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557125eb33f0 con 0x557125e7a6b0 2024-04-25T16:11:10.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.302+0000 7f72a0ff9640 1 -- 172.21.15.31:0/1778283352 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72b0036040 con 0x557125e7a6b0 2024-04-25T16:11:10.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.304+0000 7f72a0ff9640 1 -- 172.21.15.31:0/1778283352 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f72b0057050 con 0x557125e7a6b0 2024-04-25T16:11:10.306 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.304+0000 7f72c67a0740 10 client.27268 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:10.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.304+0000 7f72c67a0740 10 client.27268 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:10.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.304+0000 7f72c67a0740 20 client.27268 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:10.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.304+0000 7f72c67a0740 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557125e2f3a0 0x5571262652e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.304+0000 7f72c67a0740 4 client.27268 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:10.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.304+0000 7f72c67a0740 1 -- 172.21.15.31:0/1778283352 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5571261af990 con 0x557125e2f3a0 2024-04-25T16:11:10.307 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.306+0000 7f72bd5af640 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557125e2f3a0 0x5571262652e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.312+0000 7f72bd5af640 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557125e2f3a0 0x5571262652e0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c4a6feac4c9f99a server_cookie=564a5ad8fdfcacd6 in_seq=0 out_seq=0 2024-04-25T16:11:10.314 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.312+0000 7f72a0ff9640 10 client.27268 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.315+0000 7f72a0ff9640 1 -- 172.21.15.31:0/1778283352 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5571261af990 con 0x557125e2f3a0 2024-04-25T16:11:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.315+0000 7f72a0ff9640 10 client.27268 handle_command_reply: tid=0 2024-04-25T16:11:10.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.315+0000 7f72c67a0740 1 client.27268 shutdown 2024-04-25T16:11:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.319+0000 7f72c67a0740 1 -- 172.21.15.31:0/1778283352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e7a6b0 msgr2=0x557126257ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.319+0000 7f72c67a0740 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e7a6b0 0x557126257ae0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f72b00307c0 tx=0x7f72b00093e0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.320+0000 7f72c67a0740 1 -- 172.21.15.31:0/1778283352 shutdown_connections 2024-04-25T16:11:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.320+0000 7f72c67a0740 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x557125e2f3a0 0x5571262652e0 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.320+0000 7f72c67a0740 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e59f60 0x55712625ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.320+0000 7f72c67a0740 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e9b390 0x557126258020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.320+0000 7f72c67a0740 1 --2- 172.21.15.31:0/1778283352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e7a6b0 0x557126257ae0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.320+0000 7f72c67a0740 1 -- 172.21.15.31:0/1778283352 >> 172.21.15.31:0/1778283352 conn(0x557125e8f4e0 msgr2=0x55712623b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.321+0000 7f72a0ff9640 0 client.27268 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.321+0000 7f72c67a0740 1 -- 172.21.15.31:0/1778283352 shutdown_connections 2024-04-25T16:11:10.322 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.321+0000 7f72c67a0740 1 -- 172.21.15.31:0/1778283352 wait complete. 2024-04-25T16:11:10.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.322+0000 7f72c67a0740 20 client.27268 trim_cache size 0 max 16384 2024-04-25T16:11:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.367+0000 7f72c67a0740 1 Processor -- start 2024-04-25T16:11:10.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.367+0000 7f72c67a0740 1 -- start start 2024-04-25T16:11:10.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.367+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e59f60 0x55712623bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.368+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e7a6b0 0x55712625eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.368+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e9b390 0x55712625f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.368+0000 7f72bcdae640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e59f60 0x55712623bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.368+0000 7f72bcdae640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e59f60 0x55712623bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34462/0 (socket says 172.21.15.31:34462) 2024-04-25T16:11:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.368+0000 7f72bcdae640 1 -- 172.21.15.31:0/3405845640 learned_addr learned my addr 172.21.15.31:0/3405845640 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.368+0000 7f72c67a0740 1 -- 172.21.15.31:0/3405845640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f72b80756a0 con 0x557125e7a6b0 2024-04-25T16:11:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.368+0000 7f72c67a0740 1 -- 172.21.15.31:0/3405845640 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f72b80771e0 con 0x557125e9b390 2024-04-25T16:11:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.368+0000 7f72c67a0740 1 -- 172.21.15.31:0/3405845640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72b8075d70 con 0x557125e59f60 2024-04-25T16:11:10.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.369+0000 7f72bcdae640 1 --2- 172.21.15.31:0/3405845640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e9b390 0x55712625f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.369+0000 7f72bd5af640 1 --2- 172.21.15.31:0/3405845640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e7a6b0 0x55712625eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.369+0000 7f72bd5af640 1 -- 172.21.15.31:0/3405845640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e59f60 msgr2=0x55712623bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.369+0000 7f72bd5af640 1 --2- 172.21.15.31:0/3405845640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e59f60 0x55712623bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.369+0000 7f72bd5af640 1 -- 172.21.15.31:0/3405845640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e9b390 msgr2=0x55712625f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.369+0000 7f72bd5af640 1 --2- 172.21.15.31:0/3405845640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e9b390 0x55712625f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.371 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.369+0000 7f72bd5af640 1 -- 172.21.15.31:0/3405845640 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557125e730f0 con 0x557125e7a6b0 2024-04-25T16:11:10.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.372+0000 7f72bd5af640 1 --2- 172.21.15.31:0/3405845640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e7a6b0 0x55712625eed0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x55712623d6e0 tx=0x7f72b400aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52cc74aa8ef71a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.375+0000 7f72a2ffd640 1 -- 172.21.15.31:0/3405845640 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4013860 con 0x557125e7a6b0 2024-04-25T16:11:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.375+0000 7f72a2ffd640 1 -- 172.21.15.31:0/3405845640 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72b40160e0 con 0x557125e7a6b0 2024-04-25T16:11:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.375+0000 7f72a2ffd640 1 -- 172.21.15.31:0/3405845640 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b4014670 con 0x557125e7a6b0 2024-04-25T16:11:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.376+0000 7f72c67a0740 1 -- 172.21.15.31:0/3405845640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e7a6b0 msgr2=0x55712625eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.376+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3405845640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e7a6b0 0x55712625eed0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x55712623d6e0 tx=0x7f72b400aa30 comp rx=0 tx=0).stop 2024-04-25T16:11:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.377+0000 7f72c67a0740 1 -- 172.21.15.31:0/3405845640 shutdown_connections 2024-04-25T16:11:10.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.377+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3405845640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e9b390 0x55712625f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.377+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3405845640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e7a6b0 0x55712625eed0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.377+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3405845640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e59f60 0x55712623bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.377+0000 7f72c67a0740 1 -- 172.21.15.31:0/3405845640 >> 172.21.15.31:0/3405845640 conn(0x557125e8f4e0 msgr2=0x557126256770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.377+0000 7f72c67a0740 1 -- 172.21.15.31:0/3405845640 shutdown_connections 2024-04-25T16:11:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.377+0000 7f72c67a0740 1 -- 172.21.15.31:0/3405845640 wait complete. 2024-04-25T16:11:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.377+0000 7f72c67a0740 1 -- start start 2024-04-25T16:11:10.380 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.378+0000 7f72c67a0740 1 Processor -- start 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.378+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e59f60 0x55712625efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 0x55712625f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72bcdae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e59f60 0x55712625efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72c67a0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e9b390 0x55712625d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72bcdae640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e59f60 0x55712625efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53684/0 (socket says 172.21.15.31:53684) 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72bcdae640 1 -- 172.21.15.31:0/3489912646 learned_addr learned my addr 172.21.15.31:0/3489912646 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f72b8076b10 con 0x557125e59f60 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f72b8076440 con 0x557125e7a6b0 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f72b8118320 con 0x557125e9b390 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72c67a0740 10 client.0 _ll_register_callbacks cb 0x7f72b00308a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:10.381 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72bcdae640 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e9b390 0x55712625d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.379+0000 7f72bd5af640 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 0x55712625f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.380+0000 7f72bcdae640 1 -- 172.21.15.31:0/3489912646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 msgr2=0x55712625f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:10.382 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.380+0000 7f72bcdae640 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 0x55712625f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.381+0000 7f72bcdae640 1 -- 172.21.15.31:0/3489912646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e59f60 msgr2=0x55712625efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.381+0000 7f72bcdae640 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e59f60 0x55712625efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.381+0000 7f72bcdae640 1 -- 172.21.15.31:0/3489912646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5571261624b0 con 0x557125e9b390 2024-04-25T16:11:10.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.384+0000 7f72bcdae640 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e9b390 0x55712625d7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f72b0030a50 tx=0x7f72b002ae80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76b83bb997d2f224 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.385+0000 7f72a0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.385+0000 7f72a0ff9640 1 -- 172.21.15.31:0/3489912646 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b003e640 con 0x557125e9b390 2024-04-25T16:11:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.385+0000 7f72a0ff9640 1 -- 172.21.15.31:0/3489912646 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f72b003e7e0 con 0x557125e9b390 2024-04-25T16:11:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.385+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b40160e0 con 0x557125e9b390 2024-04-25T16:11:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.386+0000 7f72a0ff9640 1 -- 172.21.15.31:0/3489912646 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f72b003e9e0 con 0x557125e9b390 2024-04-25T16:11:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.386+0000 7f72c67a0740 10 client.27274 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:10.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.387+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5571262cc0a0 con 0x557125e9b390 2024-04-25T16:11:10.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.387+0000 7f72a0ff9640 1 -- 172.21.15.31:0/3489912646 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f72b40160e0 con 0x557125e9b390 2024-04-25T16:11:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.389+0000 7f72a0ff9640 1 -- 172.21.15.31:0/3489912646 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f72b0057070 con 0x557125e9b390 2024-04-25T16:11:10.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.390+0000 7f72c67a0740 10 client.27274 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.390+0000 7f72c67a0740 10 client.27274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.390+0000 7f72c67a0740 20 client.27274 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.390+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571262cca00 0x5571262cee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.390+0000 7f72c67a0740 4 client.27274 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:11:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.390+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557126242360 con 0x5571262cca00 2024-04-25T16:11:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.390+0000 7f72bd5af640 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571262cca00 0x5571262cee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:10.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.391+0000 7f72bd5af640 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571262cca00 0x5571262cee20 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb3cc529b8794ffc server_cookie=2dd2be9a9674da76 in_seq=0 out_seq=0 2024-04-25T16:11:10.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.391+0000 7f72a0ff9640 10 client.27274 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.392+0000 7f72a0ff9640 1 -- 172.21.15.31:0/3489912646 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x557126242360 con 0x5571262cca00 2024-04-25T16:11:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.392+0000 7f72a0ff9640 10 client.27274 handle_command_reply: tid=0 2024-04-25T16:11:10.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.393+0000 7f72c67a0740 1 client.27274 shutdown 2024-04-25T16:11:10.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.394+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e9b390 msgr2=0x55712625d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.394+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e9b390 0x55712625d7a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f72b0030a50 tx=0x7f72b002ae80 comp rx=0 tx=0).stop 2024-04-25T16:11:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.394+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 shutdown_connections 2024-04-25T16:11:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.394+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5571262cca00 0x5571262cee20 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.394+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557125e9b390 0x55712625d7a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.394+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557125e7a6b0 0x55712625f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.394+0000 7f72c67a0740 1 --2- 172.21.15.31:0/3489912646 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557125e59f60 0x55712625efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.396 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.394+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 >> 172.21.15.31:0/3489912646 conn(0x557125e8f4e0 msgr2=0x55712623b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.395+0000 7f72a0ff9640 0 client.27274 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.395+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 shutdown_connections 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.395+0000 7f72c67a0740 1 -- 172.21.15.31:0/3489912646 wait complete. 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.395+0000 7f72c67a0740 20 client.27274 trim_cache size 0 max 16384 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (5 inodes in the stack)", 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stdout: "27f9b6c8-512a-4488-8565-54eadfad4bf4": { 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:11:10.397 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "27f9b6c8-512a-4488-8565-54eadfad4bf4", 2024-04-25T16:11:10.398 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:11:10.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:11:10.398 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:11:10.398 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:11:10.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 -- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72a40afa20 msgr2=0x7f72a40b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72a40afa20 0x7f72a40b1f30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f72a80c60a0 tx=0x7f72a8009cb0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 -- 172.21.15.31:0/3414171425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b81ae640 msgr2=0x7f72b81b0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b81ae640 0x7f72b81b0a60 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f72a802f710 tx=0x7f72a802fbd0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 -- 172.21.15.31:0/3414171425 shutdown_connections 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f72a40afa20 0x7f72a40b1f30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72b81ae640 0x7f72b81b0a60 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f72b8110030 0x7f72b81a9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 --2- 172.21.15.31:0/3414171425 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f72b810a420 0x7f72b81a9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.396+0000 7f72bf813640 1 -- 172.21.15.31:0/3414171425 >> 172.21.15.31:0/3414171425 conn(0x7f72b80704e0 msgr2=0x7f72b810c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.397+0000 7f72bf813640 1 -- 172.21.15.31:0/3414171425 shutdown_connections 2024-04-25T16:11:10.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:10.397+0000 7f72bf813640 1 -- 172.21.15.31:0/3414171425 wait complete. 2024-04-25T16:11:10.449 INFO:tasks.cephfs.filesystem:scrub status for tag:27f9b6c8-512a-4488-8565-54eadfad4bf4 - {'path': '/', 'tag': '27f9b6c8-512a-4488-8565-54eadfad4bf4', 'options': 'recursive,force'} 2024-04-25T16:11:10.521 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:10 smithi063 ceph-mon[32413]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 742 MiB/s wr, 1.44k op/s 2024-04-25T16:11:10.550 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:10 smithi186 ceph-mon[33258]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 742 MiB/s wr, 1.44k op/s 2024-04-25T16:11:10.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:10 smithi031 ceph-mon[29140]: pgmap v647: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 742 MiB/s wr, 1.44k op/s 2024-04-25T16:11:11.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:11 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:11:11.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:11 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:11.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:11 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:11:11.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:11 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:11:11.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:11 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:11.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:11 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:11:11.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:11 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:11.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:11 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:11:11.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:11 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:11:11.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:11 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:11.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:11 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:11:11.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:11 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:11.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:11 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:11:11.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:11 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:11:11.705 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:11 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.079+0000 7f0fb58e0640 1 -- 172.21.15.31:0/3422973787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb0110070 msgr2=0x7f0fb0110480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:12.081 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.079+0000 7f0fb58e0640 1 --2- 172.21.15.31:0/3422973787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb0110070 0x7f0fb0110480 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009ca0 tx=0x7f0fa802a8d0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.086+0000 7f0fb58e0640 1 -- 172.21.15.31:0/3422973787 shutdown_connections 2024-04-25T16:11:12.087 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.086+0000 7f0fb58e0640 1 --2- 172.21.15.31:0/3422973787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb0110070 0x7f0fb0110480 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.086+0000 7f0fb58e0640 1 --2- 172.21.15.31:0/3422973787 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fb010b350 0x7f0fb010b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.086+0000 7f0fb58e0640 1 --2- 172.21.15.31:0/3422973787 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb010a420 0x7f0fb010ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.086+0000 7f0fb58e0640 1 -- 172.21.15.31:0/3422973787 >> 172.21.15.31:0/3422973787 conn(0x7f0fb0070540 msgr2=0x7f0fb00729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:12.088 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.086+0000 7f0fb58e0640 1 -- 172.21.15.31:0/3422973787 shutdown_connections 2024-04-25T16:11:12.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.086+0000 7f0fb58e0640 1 -- 172.21.15.31:0/3422973787 wait complete. 2024-04-25T16:11:12.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.087+0000 7f0fb58e0640 1 Processor -- start 2024-04-25T16:11:12.089 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.088+0000 7f0fb58e0640 1 -- start start 2024-04-25T16:11:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.089+0000 7f0fb58e0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fb010a420 0x7f0fb01a9020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.089+0000 7f0fb58e0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb010b350 0x7f0fb01a9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.090+0000 7f0fb58e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb0110070 0x7f0fb01b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.090+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fb010a420 0x7f0fb01a9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.090+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fb010a420 0x7f0fb01a9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:53704/0 (socket says 172.21.15.31:53704) 2024-04-25T16:11:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.090+0000 7f0faeffd640 1 -- 172.21.15.31:0/1412305745 learned_addr learned my addr 172.21.15.31:0/1412305745 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.090+0000 7f0fb58e0640 1 -- 172.21.15.31:0/1412305745 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb011e5d0 con 0x7f0fb010b350 2024-04-25T16:11:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.090+0000 7f0faf7fe640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb0110070 0x7f0fb01b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.090+0000 7f0fb58e0640 1 -- 172.21.15.31:0/1412305745 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb011e450 con 0x7f0fb010a420 2024-04-25T16:11:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.090+0000 7f0fb58e0640 1 -- 172.21.15.31:0/1412305745 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb011e750 con 0x7f0fb0110070 2024-04-25T16:11:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.091+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb010b350 0x7f0fb01a9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.092+0000 7f0faf7fe640 1 -- 172.21.15.31:0/1412305745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fb010a420 msgr2=0x7f0fb01a9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.092+0000 7f0faf7fe640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fb010a420 0x7f0fb01a9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.092+0000 7f0faf7fe640 1 -- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb010b350 msgr2=0x7f0fb01a9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.092+0000 7f0faf7fe640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb010b350 0x7f0fb01a9560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.092+0000 7f0faf7fe640 1 -- 172.21.15.31:0/1412305745 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb01b0b60 con 0x7f0fb0110070 2024-04-25T16:11:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.092+0000 7f0faf7fe640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb0110070 0x7f0fb01b04f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009bf0 tx=0x7f0fa8003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.092+0000 7f0faeffd640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fb010a420 0x7f0fb01a9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:11:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.093+0000 7f0fb48de640 1 -- 172.21.15.31:0/1412305745 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8004040 con 0x7f0fb0110070 2024-04-25T16:11:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.093+0000 7f0fb48de640 1 -- 172.21.15.31:0/1412305745 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa8004690 con 0x7f0fb0110070 2024-04-25T16:11:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.093+0000 7f0fb58e0640 1 -- 172.21.15.31:0/1412305745 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb01b0df0 con 0x7f0fb0110070 2024-04-25T16:11:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.094+0000 7f0fb58e0640 1 -- 172.21.15.31:0/1412305745 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb01b1220 con 0x7f0fb0110070 2024-04-25T16:11:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.099+0000 7f0fbc78b740 1 Processor -- start 2024-04-25T16:11:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.103+0000 7f0fbc78b740 1 -- start start 2024-04-25T16:11:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.104+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b59b31b0 0x55f4b5d53c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.104+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5d541c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.104+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d5b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.104+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f4b5cb9460 con 0x55f4b598a2f0 2024-04-25T16:11:12.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.104+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f4b5c73590 con 0x55f4b59b31b0 2024-04-25T16:11:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.104+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f4b5cb5420 con 0x55f4b59924d0 2024-04-25T16:11:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.105+0000 7f0fae7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d5b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.105+0000 7f0fae7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d5b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34478/0 (socket says 172.21.15.31:34478) 2024-04-25T16:11:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.105+0000 7f0fae7fc640 1 -- 172.21.15.31:0/2137887712 learned_addr learned my addr 172.21.15.31:0/2137887712 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.106+0000 7f0fae7fc640 1 -- 172.21.15.31:0/2137887712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b59b31b0 msgr2=0x55f4b5d53c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.106+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/2137887712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b59b31b0 0x55f4b5d53c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.106+0000 7f0fae7fc640 1 -- 172.21.15.31:0/2137887712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 msgr2=0x55f4b5d541c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.106+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/2137887712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5d541c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.106+0000 7f0fae7fc640 1 -- 172.21.15.31:0/2137887712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4b5c800d0 con 0x55f4b59924d0 2024-04-25T16:11:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.107+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/2137887712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d5b0c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009a80 tx=0x7f0fa4009f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60c96a8e8bc16437 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.107+0000 7f0fb48de640 1 -- 172.21.15.31:0/1412305745 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80305d0 con 0x7f0fb0110070 2024-04-25T16:11:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.108+0000 7f0fb48de640 1 -- 172.21.15.31:0/1412305745 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0fa80307b0 con 0x7f0fb0110070 2024-04-25T16:11:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.109+0000 7f0fb48de640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f980afd20 0x7f0f980b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.110+0000 7f0fb48de640 1 -- 172.21.15.31:0/1412305745 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0fa8101540 con 0x7f0fb0110070 2024-04-25T16:11:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.111+0000 7f0f957fa640 1 -- 172.21.15.31:0/2137887712 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4018020 con 0x55f4b59924d0 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.112+0000 7f0f957fa640 1 -- 172.21.15.31:0/2137887712 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa4010dc0 con 0x55f4b59924d0 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.112+0000 7f0f957fa640 1 -- 172.21.15.31:0/2137887712 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa40136b0 con 0x55f4b59924d0 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.115+0000 7f0fbc78b740 1 -- 172.21.15.31:0/2137887712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 msgr2=0x55f4b5d5b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.115+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/2137887712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d5b0c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009a80 tx=0x7f0fa4009f40 comp rx=0 tx=0).stop 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.115+0000 7f0fbc78b740 1 -- 172.21.15.31:0/2137887712 shutdown_connections 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.115+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/2137887712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d5b0c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.115+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/2137887712 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5d541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.115+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/2137887712 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b59b31b0 0x55f4b5d53c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.115+0000 7f0fbc78b740 1 -- 172.21.15.31:0/2137887712 >> 172.21.15.31:0/2137887712 conn(0x55f4b59a7300 msgr2=0x55f4b5d4f5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:12.117 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.115+0000 7f0faeffd640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f980afd20 0x7f0f980b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.120+0000 7f0fbc78b740 1 -- 172.21.15.31:0/2137887712 shutdown_connections 2024-04-25T16:11:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.121+0000 7f0fbc78b740 1 -- 172.21.15.31:0/2137887712 wait complete. 2024-04-25T16:11:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.122+0000 7f0fbc78b740 1 -- start start 2024-04-25T16:11:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.122+0000 7f0fbc78b740 1 Processor -- start 2024-04-25T16:11:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.123+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b598a2f0 0x55f4b5d5d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.123+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d5de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.123+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b59b31b0 0x55f4b5d64db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.123+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f4b5d5ba20 con 0x55f4b59b31b0 2024-04-25T16:11:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.123+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f4b5c40b60 con 0x55f4b598a2f0 2024-04-25T16:11:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.123+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f4b5d5bba0 con 0x55f4b59924d0 2024-04-25T16:11:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.123+0000 7f0fbc78b740 10 client.0 _ll_register_callbacks cb 0x55f4b5c652b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.124+0000 7f0faf7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b59b31b0 0x55f4b5d64db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.124+0000 7f0faf7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b59b31b0 0x55f4b5d64db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53708/0 (socket says 172.21.15.31:53708) 2024-04-25T16:11:12.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.124+0000 7f0faf7fe640 1 -- 172.21.15.31:0/461787588 learned_addr learned my addr 172.21.15.31:0/461787588 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:12.127 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.126+0000 7f0faeffd640 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b598a2f0 0x55f4b5d5d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.127+0000 7f0faf7fe640 1 -- 172.21.15.31:0/461787588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 msgr2=0x55f4b5d5de50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.127+0000 7f0faf7fe640 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d5de50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.127+0000 7f0faf7fe640 1 -- 172.21.15.31:0/461787588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b598a2f0 msgr2=0x55f4b5d5d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.127+0000 7f0faf7fe640 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b598a2f0 0x55f4b5d5d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.127+0000 7f0faf7fe640 1 -- 172.21.15.31:0/461787588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4b59cb210 con 0x55f4b59b31b0 2024-04-25T16:11:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.128+0000 7f0faeffd640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f980afd20 0x7f0f980b2230 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x55f4b5d50360 tx=0x7f0fa0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.140+0000 7f0faf7fe640 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b59b31b0 0x55f4b5d64db0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0fa802f6d0 tx=0x7f0fa80c93a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20326db41cb7765b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.140+0000 7f0f937f6640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.145+0000 7f0f937f6640 1 -- 172.21.15.31:0/461787588 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa80c9560 con 0x55f4b59b31b0 2024-04-25T16:11:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.145+0000 7f0f937f6640 1 -- 172.21.15.31:0/461787588 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0fa802f720 con 0x55f4b59b31b0 2024-04-25T16:11:12.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.146+0000 7f0fbc78b740 1 -- 172.21.15.31:0/461787588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fa4010dc0 con 0x55f4b59b31b0 2024-04-25T16:11:12.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.147+0000 7f0f937f6640 1 -- 172.21.15.31:0/461787588 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0fa80ca150 con 0x55f4b59b31b0 2024-04-25T16:11:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.154+0000 7f0fbc78b740 10 client.17616 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.154+0000 7f0fbc78b740 1 -- 172.21.15.31:0/461787588 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f4b5a97620 con 0x55f4b59b31b0 2024-04-25T16:11:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.158+0000 7f0f937f6640 1 -- 172.21.15.31:0/461787588 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa4010dc0 con 0x55f4b59b31b0 2024-04-25T16:11:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.158+0000 7f0f937f6640 1 -- 172.21.15.31:0/461787588 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0fa81010d0 con 0x55f4b59b31b0 2024-04-25T16:11:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.158+0000 7f0fbc78b740 10 client.17616 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:12.159 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.158+0000 7f0fbc78b740 10 client.17616 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:12.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.158+0000 7f0fbc78b740 20 client.17616 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:12.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.158+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f4b5971d80 0x55f4b5d6b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.158+0000 7f0fbc78b740 4 client.17616 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:12.160 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.158+0000 7f0fbc78b740 1 -- 172.21.15.31:0/461787588 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f4b5cc7070 con 0x55f4b5971d80 2024-04-25T16:11:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.161+0000 7f0faeffd640 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f4b5971d80 0x55f4b5d6b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.161+0000 7f0faeffd640 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f4b5971d80 0x55f4b5d6b6f0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb08e0b53b90f412 server_cookie=2fbc7280bb118ee8 in_seq=0 out_seq=0 2024-04-25T16:11:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.161+0000 7f0f937f6640 10 client.17616 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.163+0000 7f0f937f6640 1 -- 172.21.15.31:0/461787588 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f4b5cc7070 con 0x55f4b5971d80 2024-04-25T16:11:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.163+0000 7f0f937f6640 10 client.17616 handle_command_reply: tid=0 2024-04-25T16:11:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.163+0000 7f0fbc78b740 1 client.17616 shutdown 2024-04-25T16:11:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.166+0000 7f0fbc78b740 1 -- 172.21.15.31:0/461787588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b59b31b0 msgr2=0x55f4b5d64db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.166+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b59b31b0 0x55f4b5d64db0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0fa802f6d0 tx=0x7f0fa80c93a0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.178+0000 7f0fbc78b740 1 -- 172.21.15.31:0/461787588 shutdown_connections 2024-04-25T16:11:12.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.178+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f4b5971d80 0x55f4b5d6b6f0 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.178+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b59b31b0 0x55f4b5d64db0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.178+0000 7f0f937f6640 0 client.17616 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.178+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d5de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.178+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/461787588 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b598a2f0 0x55f4b5d5d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.178+0000 7f0fbc78b740 1 -- 172.21.15.31:0/461787588 >> 172.21.15.31:0/461787588 conn(0x55f4b59a7300 msgr2=0x55f4b5d4e410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.178+0000 7f0fbc78b740 1 -- 172.21.15.31:0/461787588 shutdown_connections 2024-04-25T16:11:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.178+0000 7f0fbc78b740 1 -- 172.21.15.31:0/461787588 wait complete. 2024-04-25T16:11:12.180 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.179+0000 7f0fbc78b740 20 client.17616 trim_cache size 0 max 16384 2024-04-25T16:11:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.255+0000 7f0fbc78b740 1 Processor -- start 2024-04-25T16:11:12.256 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.255+0000 7f0fbc78b740 1 -- start start 2024-04-25T16:11:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.256+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b5971d80 0x55f4b5d52ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.257 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.256+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5d53020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.256+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d64cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.256+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb01ba090 con 0x55f4b598a2f0 2024-04-25T16:11:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.256+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb01bbbd0 con 0x55f4b5971d80 2024-04-25T16:11:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.256+0000 7f0fae7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5d53020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.256+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb01ba760 con 0x55f4b59924d0 2024-04-25T16:11:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.256+0000 7f0fae7fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5d53020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53720/0 (socket says 172.21.15.31:53720) 2024-04-25T16:11:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.256+0000 7f0fae7fc640 1 -- 172.21.15.31:0/997408482 learned_addr learned my addr 172.21.15.31:0/997408482 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.257+0000 7f0fae7fc640 1 -- 172.21.15.31:0/997408482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 msgr2=0x55f4b5d64cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:12.258 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.257+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/997408482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d64cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.257+0000 7f0fae7fc640 1 -- 172.21.15.31:0/997408482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b5971d80 msgr2=0x55f4b5d52ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.257+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/997408482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b5971d80 0x55f4b5d52ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.257+0000 7f0fae7fc640 1 -- 172.21.15.31:0/997408482 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4b5d58a40 con 0x55f4b598a2f0 2024-04-25T16:11:12.264 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.261+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/997408482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5d53020 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009a50 tx=0x7f0fa4009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8005859bd520fe3 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.284+0000 7f0f94ff9640 1 -- 172.21.15.31:0/997408482 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4018050 con 0x55f4b598a2f0 2024-04-25T16:11:12.285 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.284+0000 7f0f94ff9640 1 -- 172.21.15.31:0/997408482 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa4028930 con 0x55f4b598a2f0 2024-04-25T16:11:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.285+0000 7f0fbc78b740 1 -- 172.21.15.31:0/997408482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 msgr2=0x55f4b5d53020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:12.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.286+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/997408482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5d53020 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4009a50 tx=0x7f0fa4009770 comp rx=0 tx=0).stop 2024-04-25T16:11:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.288+0000 7f0fbc78b740 1 -- 172.21.15.31:0/997408482 shutdown_connections 2024-04-25T16:11:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.288+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/997408482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b59924d0 0x55f4b5d64cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.288+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/997408482 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5d53020 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.289+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/997408482 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b5971d80 0x55f4b5d52ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.289+0000 7f0fbc78b740 1 -- 172.21.15.31:0/997408482 >> 172.21.15.31:0/997408482 conn(0x55f4b59a7300 msgr2=0x55f4b5d523d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.291+0000 7f0fbc78b740 1 -- 172.21.15.31:0/997408482 shutdown_connections 2024-04-25T16:11:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.292+0000 7f0fbc78b740 1 -- 172.21.15.31:0/997408482 wait complete. 2024-04-25T16:11:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.293+0000 7f0fbc78b740 1 -- start start 2024-04-25T16:11:12.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.293+0000 7f0fbc78b740 1 Processor -- start 2024-04-25T16:11:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.296+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b5971d80 0x55f4b5dcda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.297+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5dcffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.297+0000 7f0fbc78b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b59924d0 0x55f4b5dd2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.297+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb01bb500 con 0x55f4b598a2f0 2024-04-25T16:11:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.297+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb01bae30 con 0x55f4b59924d0 2024-04-25T16:11:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.297+0000 7f0fbc78b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb01b8070 con 0x55f4b5971d80 2024-04-25T16:11:12.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.298+0000 7f0fbc78b740 10 client.0 _ll_register_callbacks cb 0x7f0fa802f620 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.301+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b5971d80 0x55f4b5dcda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.301+0000 7f0faeffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b5971d80 0x55f4b5dcda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:34512/0 (socket says 172.21.15.31:34512) 2024-04-25T16:11:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.301+0000 7f0faeffd640 1 -- 172.21.15.31:0/1767796715 learned_addr learned my addr 172.21.15.31:0/1767796715 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.302+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5dcffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.302+0000 7f0faf7fe640 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b59924d0 0x55f4b5dd2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.305+0000 7f0faeffd640 1 -- 172.21.15.31:0/1767796715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b59924d0 msgr2=0x55f4b5dd2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:12.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.305+0000 7f0faeffd640 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b59924d0 0x55f4b5dd2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.305+0000 7f0faeffd640 1 -- 172.21.15.31:0/1767796715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 msgr2=0x55f4b5dcffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.305+0000 7f0faeffd640 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5dcffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.305+0000 7f0faeffd640 1 -- 172.21.15.31:0/1767796715 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4b5dd2a50 con 0x55f4b5971d80 2024-04-25T16:11:12.307 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.306+0000 7f0faeffd640 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b5971d80 0x55f4b5dcda70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x55f4b5d6be90 tx=0x7f0fa00047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eecc744fa6a39e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:12.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.306+0000 7f0f92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.311+0000 7f0f92ff5640 1 -- 172.21.15.31:0/1767796715 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa000f040 con 0x55f4b5971d80 2024-04-25T16:11:12.314 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.311+0000 7f0f92ff5640 1 -- 172.21.15.31:0/1767796715 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f0fa0015c10 con 0x55f4b5971d80 2024-04-25T16:11:12.315 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.313+0000 7f0fbc78b740 1 -- 172.21.15.31:0/1767796715 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0fa4028930 con 0x55f4b5971d80 2024-04-25T16:11:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.318+0000 7f0f92ff5640 1 -- 172.21.15.31:0/1767796715 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f0fa000ee40 con 0x55f4b5971d80 2024-04-25T16:11:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.318+0000 7f0fbc78b740 10 client.27286 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:12.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.318+0000 7f0fbc78b740 1 -- 172.21.15.31:0/1767796715 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f4b5dd3c30 con 0x55f4b5971d80 2024-04-25T16:11:12.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.320+0000 7f0f92ff5640 1 -- 172.21.15.31:0/1767796715 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0fa4028930 con 0x55f4b5971d80 2024-04-25T16:11:12.322 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.320+0000 7f0f92ff5640 1 -- 172.21.15.31:0/1767796715 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f0fa0012040 con 0x55f4b5971d80 2024-04-25T16:11:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.321+0000 7f0fbc78b740 10 client.27286 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.321+0000 7f0fbc78b740 10 client.27286 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.321+0000 7f0fbc78b740 20 client.27286 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.321+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f4b59b31b0 0x55f4b5dd64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.321+0000 7f0fbc78b740 4 client.27286 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:11:12.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.321+0000 7f0fbc78b740 1 -- 172.21.15.31:0/1767796715 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f4b5cc0ee0 con 0x55f4b59b31b0 2024-04-25T16:11:12.337 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.336+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f4b59b31b0 0x55f4b5dd64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:12.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.350+0000 7f0fae7fc640 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f4b59b31b0 0x55f4b5dd64e0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d718ee31ef2181e server_cookie=1dc4dfaad95d889e in_seq=0 out_seq=0 2024-04-25T16:11:12.358 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:12.352+0000 7f0f92ff5640 10 client.27286 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:12.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:12 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:12.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:12 smithi063 ceph-mon[32413]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 752 MiB/s wr, 1.45k op/s 2024-04-25T16:11:12.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:12 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:12.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:12 smithi186 ceph-mon[33258]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 752 MiB/s wr, 1.45k op/s 2024-04-25T16:11:12.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:12 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:12.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:12 smithi031 ceph-mon[29140]: pgmap v648: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 752 MiB/s wr, 1.45k op/s 2024-04-25T16:11:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.916+0000 7f0f92ff5640 1 -- 172.21.15.31:0/1767796715 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f4b5cc0ee0 con 0x55f4b59b31b0 2024-04-25T16:11:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.916+0000 7f0f92ff5640 10 client.27286 handle_command_reply: tid=0 2024-04-25T16:11:14.918 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.916+0000 7f0fbc78b740 1 client.27286 shutdown 2024-04-25T16:11:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 -- 172.21.15.31:0/1767796715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b5971d80 msgr2=0x55f4b5dcda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:14.919 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b5971d80 0x55f4b5dcda70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x55f4b5d6be90 tx=0x7f0fa00047d0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 -- 172.21.15.31:0/1767796715 shutdown_connections 2024-04-25T16:11:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55f4b59b31b0 0x55f4b5dd64e0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f4b59924d0 0x55f4b5dd2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f4b598a2f0 0x55f4b5dcffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 --2- 172.21.15.31:0/1767796715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f4b5971d80 0x55f4b5dcda70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0f92ff5640 0 client.27286 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 -- 172.21.15.31:0/1767796715 >> 172.21.15.31:0/1767796715 conn(0x55f4b59a7300 msgr2=0x55f4b5d52e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:14.920 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 -- 172.21.15.31:0/1767796715 shutdown_connections 2024-04-25T16:11:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.918+0000 7f0fbc78b740 1 -- 172.21.15.31:0/1767796715 wait complete. 2024-04-25T16:11:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.919+0000 7f0fbc78b740 20 client.27286 trim_cache size 0 max 16384 2024-04-25T16:11:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:11:14.921 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:11:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:11:14.922 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:11:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.924+0000 7f0f967fc640 1 -- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f980afd20 msgr2=0x7f0f980b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.924+0000 7f0f967fc640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f980afd20 0x7f0f980b2230 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x55f4b5d50360 tx=0x7f0fa0008040 comp rx=0 tx=0).stop 2024-04-25T16:11:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.924+0000 7f0f967fc640 1 -- 172.21.15.31:0/1412305745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb0110070 msgr2=0x7f0fb01b04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:14.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.924+0000 7f0f967fc640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb0110070 0x7f0fb01b04f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f0fa8009bf0 tx=0x7f0fa8003ba0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.943+0000 7f0f967fc640 1 -- 172.21.15.31:0/1412305745 shutdown_connections 2024-04-25T16:11:14.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.943+0000 7f0f967fc640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f0f980afd20 0x7f0f980b2230 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.943+0000 7f0f967fc640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fb0110070 0x7f0fb01b04f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.943+0000 7f0f967fc640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f0fb010b350 0x7f0fb01a9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.943+0000 7f0f967fc640 1 --2- 172.21.15.31:0/1412305745 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0fb010a420 0x7f0fb01a9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:14.945 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.943+0000 7f0f967fc640 1 -- 172.21.15.31:0/1412305745 >> 172.21.15.31:0/1412305745 conn(0x7f0fb0070540 msgr2=0x7f0fb0072960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.946+0000 7f0f967fc640 1 -- 172.21.15.31:0/1412305745 shutdown_connections 2024-04-25T16:11:14.947 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:14.946+0000 7f0f967fc640 1 -- 172.21.15.31:0/1412305745 wait complete. 2024-04-25T16:11:15.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:15 smithi031 ceph-mon[29140]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 949 MiB/s wr, 2.06k op/s 2024-04-25T16:11:15.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:15 smithi063 ceph-mon[32413]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 949 MiB/s wr, 2.06k op/s 2024-04-25T16:11:15.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:15 smithi186 ceph-mon[33258]: pgmap v649: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 949 MiB/s wr, 2.06k op/s 2024-04-25T16:11:17.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:17 smithi031 ceph-mon[29140]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 764 MiB/s wr, 1.84k op/s 2024-04-25T16:11:17.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:17 smithi063 ceph-mon[32413]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 764 MiB/s wr, 1.84k op/s 2024-04-25T16:11:17.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:17 smithi186 ceph-mon[33258]: pgmap v650: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 764 MiB/s wr, 1.84k op/s 2024-04-25T16:11:19.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:19 smithi063 ceph-mon[32413]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 925 MiB/s wr, 2.24k op/s 2024-04-25T16:11:19.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:19 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:11:19.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:19 smithi186 ceph-mon[33258]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 925 MiB/s wr, 2.24k op/s 2024-04-25T16:11:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:19 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:11:19.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:19 smithi031 ceph-mon[29140]: pgmap v651: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 925 MiB/s wr, 2.24k op/s 2024-04-25T16:11:19.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:19 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:11:20.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:20 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:11:20.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:20 smithi063 ceph-mon[32413]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 740 MiB/s wr, 1.79k op/s 2024-04-25T16:11:20.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:20 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:11:20.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:20 smithi186 ceph-mon[33258]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 740 MiB/s wr, 1.79k op/s 2024-04-25T16:11:20.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:20 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:11:20.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:20 smithi031 ceph-mon[29140]: pgmap v652: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 740 MiB/s wr, 1.79k op/s 2024-04-25T16:11:23.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:23 smithi031 ceph-mon[29140]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 743 MiB/s wr, 1.78k op/s 2024-04-25T16:11:23.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:23 smithi063 ceph-mon[32413]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 743 MiB/s wr, 1.78k op/s 2024-04-25T16:11:23.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:23 smithi186 ceph-mon[33258]: pgmap v653: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 743 MiB/s wr, 1.78k op/s 2024-04-25T16:11:25.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:25 smithi031 ceph-mon[29140]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 947 MiB/s wr, 2.24k op/s 2024-04-25T16:11:25.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:25 smithi063 ceph-mon[32413]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 947 MiB/s wr, 2.24k op/s 2024-04-25T16:11:25.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:25 smithi186 ceph-mon[33258]: pgmap v654: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 947 MiB/s wr, 2.24k op/s 2024-04-25T16:11:27.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:27 smithi063 ceph-mon[32413]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 765 MiB/s wr, 1.81k op/s 2024-04-25T16:11:27.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:27 smithi186 ceph-mon[33258]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 765 MiB/s wr, 1.81k op/s 2024-04-25T16:11:27.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:27 smithi031 ceph-mon[29140]: pgmap v655: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 765 MiB/s wr, 1.81k op/s 2024-04-25T16:11:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.273+0000 7fef13bc0640 1 -- 172.21.15.31:0/4079876634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 msgr2=0x7fef0c106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.273+0000 7fef13bc0640 1 --2- 172.21.15.31:0/4079876634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 0x7fef0c106a40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fef04009ca0 tx=0x7fef0402a8f0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.281+0000 7fef13bc0640 1 -- 172.21.15.31:0/4079876634 shutdown_connections 2024-04-25T16:11:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.281+0000 7fef13bc0640 1 --2- 172.21.15.31:0/4079876634 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef0c10bc80 0x7fef0c10e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.281+0000 7fef13bc0640 1 --2- 172.21.15.31:0/4079876634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c106f80 0x7fef0c107390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.281+0000 7fef13bc0640 1 --2- 172.21.15.31:0/4079876634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 0x7fef0c106a40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.281+0000 7fef13bc0640 1 -- 172.21.15.31:0/4079876634 >> 172.21.15.31:0/4079876634 conn(0x7fef0c100f10 msgr2=0x7fef0c1033e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.289+0000 7fef13bc0640 1 -- 172.21.15.31:0/4079876634 shutdown_connections 2024-04-25T16:11:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.291+0000 7fef13bc0640 1 -- 172.21.15.31:0/4079876634 wait complete. 2024-04-25T16:11:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.292+0000 7fef13bc0640 1 Processor -- start 2024-04-25T16:11:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.294+0000 7fef13bc0640 1 -- start start 2024-04-25T16:11:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.295+0000 7fef13bc0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 0x7fef0c06fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.295+0000 7fef13bc0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef0c106f80 0x7fef0c06ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.295+0000 7fef13bc0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c10bc80 0x7fef0c072e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.295+0000 7fef13bc0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c079a60 con 0x7fef0c106f80 2024-04-25T16:11:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.295+0000 7fef13bc0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c0798e0 con 0x7fef0c10bc80 2024-04-25T16:11:28.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.295+0000 7fef13bc0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef0c079be0 con 0x7fef0c105d40 2024-04-25T16:11:28.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.295+0000 7fef1115b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef0c106f80 0x7fef0c06ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.296+0000 7fef1195c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 0x7fef0c06fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.296+0000 7fef1195c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 0x7fef0c06fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:59936/0 (socket says 172.21.15.31:59936) 2024-04-25T16:11:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.296+0000 7fef1195c640 1 -- 172.21.15.31:0/2658368111 learned_addr learned my addr 172.21.15.31:0/2658368111 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.297+0000 7fef1195c640 1 -- 172.21.15.31:0/2658368111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c10bc80 msgr2=0x7fef0c072e80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:11:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.297+0000 7fef1195c640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c10bc80 0x7fef0c072e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.299 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.297+0000 7fef1195c640 1 -- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef0c106f80 msgr2=0x7fef0c06ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.298+0000 7fef1195c640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef0c106f80 0x7fef0c06ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.298+0000 7fef1195c640 1 -- 172.21.15.31:0/2658368111 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef0c0734f0 con 0x7fef0c105d40 2024-04-25T16:11:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.298+0000 7fef1115b640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef0c106f80 0x7fef0c06ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:11:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.299+0000 7fef1195c640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 0x7fef0c06fa20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fef04006010 tx=0x7fef04030bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:28.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.299+0000 7fef02ffd640 1 -- 172.21.15.31:0/2658368111 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef04045620 con 0x7fef0c105d40 2024-04-25T16:11:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.299+0000 7fef13bc0640 1 -- 172.21.15.31:0/2658368111 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef0c073780 con 0x7fef0c105d40 2024-04-25T16:11:28.301 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.299+0000 7fef13bc0640 1 -- 172.21.15.31:0/2658368111 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef0c073d20 con 0x7fef0c105d40 2024-04-25T16:11:28.306 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.305+0000 7fef02ffd640 1 -- 172.21.15.31:0/2658368111 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef0402d070 con 0x7fef0c105d40 2024-04-25T16:11:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.307+0000 7fef02ffd640 1 -- 172.21.15.31:0/2658368111 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0403bb30 con 0x7fef0c105d40 2024-04-25T16:11:28.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.307+0000 7fef02ffd640 1 -- 172.21.15.31:0/2658368111 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fef0404e3f0 con 0x7fef0c105d40 2024-04-25T16:11:28.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.309+0000 7fef02ffd640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feef80afb00 0x7feef80b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.310 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.309+0000 7fef02ffd640 1 -- 172.21.15.31:0/2658368111 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fef04100cb0 con 0x7fef0c105d40 2024-04-25T16:11:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.311+0000 7fef1aa70740 1 Processor -- start 2024-04-25T16:11:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.311+0000 7fef1aa70740 1 -- start start 2024-04-25T16:11:28.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.318+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df8c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.319 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.318+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbc32f0 0x55b37df8cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.320 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.318+0000 7fef1195c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df8c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.319+0000 7fef1195c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df8c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49044/0 (socket says 172.21.15.31:49044) 2024-04-25T16:11:28.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.319+0000 7fef1195c640 1 -- 172.21.15.31:0/4076737602 learned_addr learned my addr 172.21.15.31:0/4076737602 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:28.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.319+0000 7fef1aa70740 1 --2- 172.21.15.31:0/4076737602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbcb4d0 0x55b37df93df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.319+0000 7fef1aa70740 1 -- 172.21.15.31:0/4076737602 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b37def2460 con 0x55b37dbcb4d0 2024-04-25T16:11:28.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.319+0000 7fef1aa70740 1 -- 172.21.15.31:0/4076737602 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b37deac590 con 0x55b37dbec1b0 2024-04-25T16:11:28.321 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.319+0000 7fef1aa70740 1 -- 172.21.15.31:0/4076737602 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b37deee420 con 0x55b37dbc32f0 2024-04-25T16:11:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.321+0000 7fef1195c640 1 -- 172.21.15.31:0/4076737602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbc32f0 msgr2=0x55b37df8cef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.321+0000 7fef1195c640 1 --2- 172.21.15.31:0/4076737602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbc32f0 0x55b37df8cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.321+0000 7fef1195c640 1 -- 172.21.15.31:0/4076737602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbcb4d0 msgr2=0x55b37df93df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.321+0000 7fef1195c640 1 --2- 172.21.15.31:0/4076737602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbcb4d0 0x55b37df93df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.321+0000 7fef1195c640 1 -- 172.21.15.31:0/4076737602 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b37deb90d0 con 0x55b37dbec1b0 2024-04-25T16:11:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.321+0000 7fef1195c640 1 --2- 172.21.15.31:0/4076737602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df8c970 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fef0402aee0 tx=0x7fef0402a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8c957f7923a727d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.322+0000 7feef7fff640 1 -- 172.21.15.31:0/4076737602 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef040c8cd0 con 0x55b37dbec1b0 2024-04-25T16:11:28.323 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.322+0000 7feef7fff640 1 -- 172.21.15.31:0/4076737602 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef04036040 con 0x55b37dbec1b0 2024-04-25T16:11:28.324 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.322+0000 7feef7fff640 1 -- 172.21.15.31:0/4076737602 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef0403bcd0 con 0x55b37dbec1b0 2024-04-25T16:11:28.325 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.324+0000 7fef1aa70740 1 -- 172.21.15.31:0/4076737602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 msgr2=0x55b37df8c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:28.326 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.324+0000 7fef1aa70740 1 --2- 172.21.15.31:0/4076737602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df8c970 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fef0402aee0 tx=0x7fef0402a8f0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.338 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.336+0000 7fef1115b640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feef80afb00 0x7feef80b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.341+0000 7fef1aa70740 1 -- 172.21.15.31:0/4076737602 shutdown_connections 2024-04-25T16:11:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.341+0000 7fef1aa70740 1 --2- 172.21.15.31:0/4076737602 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbcb4d0 0x55b37df93df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.341+0000 7fef1aa70740 1 --2- 172.21.15.31:0/4076737602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbc32f0 0x55b37df8cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.342+0000 7fef1aa70740 1 --2- 172.21.15.31:0/4076737602 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df8c970 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.343 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.342+0000 7fef1aa70740 1 -- 172.21.15.31:0/4076737602 >> 172.21.15.31:0/4076737602 conn(0x55b37dbe0300 msgr2=0x55b37df88300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.343+0000 7fef1aa70740 1 -- 172.21.15.31:0/4076737602 shutdown_connections 2024-04-25T16:11:28.344 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.343+0000 7fef1aa70740 1 -- 172.21.15.31:0/4076737602 wait complete. 2024-04-25T16:11:28.345 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.344+0000 7fef1aa70740 1 -- start start 2024-04-25T16:11:28.346 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.345+0000 7fef1aa70740 1 Processor -- start 2024-04-25T16:11:28.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.346+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbc32f0 0x55b37df963f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.346+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbcb4d0 0x55b37df96930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.347 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.346+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df9d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.346+0000 7fef1aa70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55b37df94600 con 0x55b37dbcb4d0 2024-04-25T16:11:28.348 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1aa70740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55b37de79b60 con 0x55b37dbec1b0 2024-04-25T16:11:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1215d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df9d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1215d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df9d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49048/0 (socket says 172.21.15.31:49048) 2024-04-25T16:11:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1215d640 1 -- 172.21.15.31:0/2569578580 learned_addr learned my addr 172.21.15.31:0/2569578580 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1aa70740 1 -- 172.21.15.31:0/2569578580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55b37df94780 con 0x55b37dbc32f0 2024-04-25T16:11:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1195c640 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbc32f0 0x55b37df963f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.349 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1aa70740 10 client.0 _ll_register_callbacks cb 0x55b37de9e2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1215d640 1 -- 172.21.15.31:0/2569578580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbc32f0 msgr2=0x55b37df963f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1215d640 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbc32f0 0x55b37df963f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1215d640 1 -- 172.21.15.31:0/2569578580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbcb4d0 msgr2=0x55b37df96930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.347+0000 7fef1215d640 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbcb4d0 0x55b37df96930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.348+0000 7fef1215d640 1 -- 172.21.15.31:0/2569578580 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b37dc04210 con 0x55b37dbec1b0 2024-04-25T16:11:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.348+0000 7fef1aa70740 1 -- 172.21.15.31:0/2569578580 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef04036040 con 0x55b37dbec1b0 2024-04-25T16:11:28.350 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.348+0000 7fef1215d640 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df9d890 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef0800e3c0 tx=0x7fef0800e880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=baa670b59fcc8342 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:28.351 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.348+0000 7feef57fa640 10 client.27317 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.353+0000 7feef57fa640 1 -- 172.21.15.31:0/2569578580 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08010840 con 0x55b37dbec1b0 2024-04-25T16:11:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.353+0000 7feef57fa640 1 -- 172.21.15.31:0/2569578580 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fef080109e0 con 0x55b37dbec1b0 2024-04-25T16:11:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.353+0000 7feef57fa640 1 -- 172.21.15.31:0/2569578580 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fef08010c80 con 0x55b37dbec1b0 2024-04-25T16:11:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.353+0000 7fef1aa70740 10 client.27317 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:28.355 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.354+0000 7fef1aa70740 1 -- 172.21.15.31:0/2569578580 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({fsmap=29}) -- 0x55b37dcd0620 con 0x55b37dbec1b0 2024-04-25T16:11:28.356 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.354+0000 7feef57fa640 1 -- 172.21.15.31:0/2569578580 <== mon.2 v2:172.21.15.63:3300/0 4 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fef08018420 con 0x55b37dbec1b0 2024-04-25T16:11:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.355+0000 7fef1aa70740 10 client.27317 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.355+0000 7fef1aa70740 10 client.27317 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.355+0000 7fef1aa70740 20 client.27317 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.355+0000 7fef1aa70740 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b37dbaad80 0x55b37dfa3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.355+0000 7fef1aa70740 4 client.27317 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.355+0000 7fef1aa70740 1 -- 172.21.15.31:0/2569578580 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b37df00070 con 0x55b37dbaad80 2024-04-25T16:11:28.357 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.356+0000 7fef1195c640 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b37dbaad80 0x55b37dfa3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.358+0000 7fef1195c640 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b37dbaad80 0x55b37dfa3dd0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66ffdc02b38c921e server_cookie=4c3fb1c8a2645244 in_seq=0 out_seq=0 2024-04-25T16:11:28.360 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.358+0000 7feef57fa640 10 client.27317 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:28.361 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.360+0000 7feef57fa640 1 -- 172.21.15.31:0/2569578580 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b37df00070 con 0x55b37dbaad80 2024-04-25T16:11:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.360+0000 7feef57fa640 10 client.27317 handle_command_reply: tid=0 2024-04-25T16:11:28.362 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.361+0000 7fef1aa70740 1 client.27317 shutdown 2024-04-25T16:11:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.362+0000 7fef1aa70740 1 -- 172.21.15.31:0/2569578580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 msgr2=0x55b37df9d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:28.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.362+0000 7fef1aa70740 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df9d890 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fef0800e3c0 tx=0x7fef0800e880 comp rx=0 tx=0).stop 2024-04-25T16:11:28.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.364+0000 7fef1115b640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feef80afb00 0x7feef80b2010 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55b37df97460 tx=0x7feefc005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.367+0000 7fef1aa70740 1 -- 172.21.15.31:0/2569578580 shutdown_connections 2024-04-25T16:11:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.367+0000 7fef1aa70740 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b37dbaad80 0x55b37dfa3dd0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.367+0000 7fef1aa70740 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df9d890 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.367+0000 7fef1aa70740 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbcb4d0 0x55b37df96930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.367+0000 7fef1aa70740 1 --2- 172.21.15.31:0/2569578580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbc32f0 0x55b37df963f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.367+0000 7fef1aa70740 1 -- 172.21.15.31:0/2569578580 >> 172.21.15.31:0/2569578580 conn(0x55b37dbe0300 msgr2=0x55b37df87250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:28.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.368+0000 7feef57fa640 0 client.27317 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:28.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.369+0000 7fef1aa70740 1 -- 172.21.15.31:0/2569578580 shutdown_connections 2024-04-25T16:11:28.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.369+0000 7fef1aa70740 1 -- 172.21.15.31:0/2569578580 wait complete. 2024-04-25T16:11:28.370 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.369+0000 7fef1aa70740 20 client.27317 trim_cache size 0 max 16384 2024-04-25T16:11:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.450+0000 7fef1aa70740 1 Processor -- start 2024-04-25T16:11:28.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.451+0000 7fef1aa70740 1 -- start start 2024-04-25T16:11:28.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.452+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 0x55b37df86f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.452+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbcb4d0 0x55b37df9d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.452+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df94ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.453+0000 7fef1aa70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feef000e4e0 con 0x55b37dbc32f0 2024-04-25T16:11:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.453+0000 7fef1aa70740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7feef0010020 con 0x55b37dbec1b0 2024-04-25T16:11:28.454 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.453+0000 7fef1aa70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feef000ebb0 con 0x55b37dbcb4d0 2024-04-25T16:11:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.453+0000 7fef1195c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 0x55b37df86f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.453+0000 7fef1195c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 0x55b37df86f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:38792/0 (socket says 172.21.15.31:38792) 2024-04-25T16:11:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.453+0000 7fef1195c640 1 -- 172.21.15.31:0/3392036315 learned_addr learned my addr 172.21.15.31:0/3392036315 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.453+0000 7fef1115b640 1 --2- 172.21.15.31:0/3392036315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbcb4d0 0x55b37df9d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.455 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.454+0000 7fef1215d640 1 --2- 172.21.15.31:0/3392036315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df94ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.454+0000 7fef1195c640 1 -- 172.21.15.31:0/3392036315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbcb4d0 msgr2=0x55b37df9d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.454+0000 7fef1195c640 1 --2- 172.21.15.31:0/3392036315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbcb4d0 0x55b37df9d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.454+0000 7fef1195c640 1 -- 172.21.15.31:0/3392036315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 msgr2=0x55b37df94ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.454+0000 7fef1195c640 1 --2- 172.21.15.31:0/3392036315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df94ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.454+0000 7fef1195c640 1 -- 172.21.15.31:0/3392036315 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b37df94840 con 0x55b37dbc32f0 2024-04-25T16:11:28.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.455+0000 7fef1195c640 1 --2- 172.21.15.31:0/3392036315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 0x55b37df86f30 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fef040c6430 tx=0x7fef04031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fdaf92a22d7180 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.455+0000 7feef77fe640 1 -- 172.21.15.31:0/3392036315 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef040e3070 con 0x55b37dbc32f0 2024-04-25T16:11:28.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.456+0000 7feef77fe640 1 -- 172.21.15.31:0/3392036315 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef04031a70 con 0x55b37dbc32f0 2024-04-25T16:11:28.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.457+0000 7fef1aa70740 1 -- 172.21.15.31:0/3392036315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 msgr2=0x55b37df86f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:28.459 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.457+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3392036315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 0x55b37df86f30 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fef040c6430 tx=0x7fef04031ee0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.462+0000 7fef1aa70740 1 -- 172.21.15.31:0/3392036315 shutdown_connections 2024-04-25T16:11:28.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.462+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3392036315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37df94ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.463 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.462+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3392036315 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbcb4d0 0x55b37df9d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.462+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3392036315 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 0x55b37df86f30 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.462+0000 7fef1aa70740 1 -- 172.21.15.31:0/3392036315 >> 172.21.15.31:0/3392036315 conn(0x55b37dbe0300 msgr2=0x55b37df86820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:28.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.464+0000 7fef1aa70740 1 -- 172.21.15.31:0/3392036315 shutdown_connections 2024-04-25T16:11:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.468+0000 7fef1aa70740 1 -- 172.21.15.31:0/3392036315 wait complete. 2024-04-25T16:11:28.470 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.468+0000 7fef1aa70740 1 -- start start 2024-04-25T16:11:28.472 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.469+0000 7fef1aa70740 1 Processor -- start 2024-04-25T16:11:28.479 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.473+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 0x55b37df9a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.473+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbcb4d0 0x55b37df9ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.473+0000 7fef1aa70740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37e00a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.473+0000 7fef1aa70740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7feef000f950 con 0x55b37dbc32f0 2024-04-25T16:11:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.473+0000 7fef1aa70740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7feef000f280 con 0x55b37dbec1b0 2024-04-25T16:11:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.473+0000 7fef1aa70740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feef000c4c0 con 0x55b37dbcb4d0 2024-04-25T16:11:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.474+0000 7fef1aa70740 10 client.0 _ll_register_callbacks cb 0x7fef08009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.477+0000 7fef1215d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37e00a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.478+0000 7fef1215d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37e00a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49078/0 (socket says 172.21.15.31:49078) 2024-04-25T16:11:28.480 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.478+0000 7fef1215d640 1 -- 172.21.15.31:0/3062337999 learned_addr learned my addr 172.21.15.31:0/3062337999 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.481+0000 7fef1215d640 1 -- 172.21.15.31:0/3062337999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbcb4d0 msgr2=0x55b37df9ab50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.481+0000 7fef1215d640 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbcb4d0 0x55b37df9ab50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.482 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.481+0000 7fef1215d640 1 -- 172.21.15.31:0/3062337999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 msgr2=0x55b37df9a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.481+0000 7fef1215d640 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 0x55b37df9a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:28.483 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.481+0000 7fef1215d640 1 -- 172.21.15.31:0/3062337999 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b37df9dac0 con 0x55b37dbec1b0 2024-04-25T16:11:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.483+0000 7fef1215d640 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37e00a170 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fef08009720 tx=0x7fef08005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b340dcc801bceedf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:28.485 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.484+0000 7feef57fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:28.486 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.485+0000 7feef57fa640 1 -- 172.21.15.31:0/3062337999 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef08030020 con 0x55b37dbec1b0 2024-04-25T16:11:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.485+0000 7feef57fa640 1 -- 172.21.15.31:0/3062337999 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fef08010840 con 0x55b37dbec1b0 2024-04-25T16:11:28.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.486+0000 7fef1aa70740 1 -- 172.21.15.31:0/3062337999 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fef04031a70 con 0x55b37dbec1b0 2024-04-25T16:11:28.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.487+0000 7feef57fa640 1 -- 172.21.15.31:0/3062337999 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fef08010a40 con 0x55b37dbec1b0 2024-04-25T16:11:28.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.488+0000 7fef1aa70740 10 client.27320 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:28.489 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.488+0000 7fef1aa70740 1 -- 172.21.15.31:0/3062337999 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b37e00b660 con 0x55b37dbec1b0 2024-04-25T16:11:28.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.496+0000 7feef57fa640 1 -- 172.21.15.31:0/3062337999 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef04031a70 con 0x55b37dbec1b0 2024-04-25T16:11:28.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.496+0000 7feef57fa640 1 -- 172.21.15.31:0/3062337999 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fef08003aa0 con 0x55b37dbec1b0 2024-04-25T16:11:28.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.496+0000 7fef1aa70740 10 client.27320 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:28.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.496+0000 7fef1aa70740 10 client.27320 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:28.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.496+0000 7fef1aa70740 20 client.27320 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:28.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.497+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b37db801c0 0x55b37e00e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:28.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.497+0000 7fef1aa70740 4 client.27320 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:11:28.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.497+0000 7fef1aa70740 1 -- 172.21.15.31:0/3062337999 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b37def9ee0 con 0x55b37db801c0 2024-04-25T16:11:28.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.499+0000 7fef1195c640 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b37db801c0 0x55b37e00e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:28.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.502+0000 7fef1195c640 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b37db801c0 0x55b37e00e030 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=977bbbf19af34f3c server_cookie=2ed07204b299f61e in_seq=0 out_seq=0 2024-04-25T16:11:28.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:28.503+0000 7feef57fa640 10 client.27320 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:29.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:29 smithi063 ceph-mon[32413]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 918 MiB/s wr, 2.50k op/s 2024-04-25T16:11:29.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:29 smithi186 ceph-mon[33258]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 918 MiB/s wr, 2.50k op/s 2024-04-25T16:11:29.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:29 smithi031 ceph-mon[29140]: pgmap v656: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 918 MiB/s wr, 2.50k op/s 2024-04-25T16:11:30.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:30 smithi063 ceph-mon[32413]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 737 MiB/s wr, 2.14k op/s 2024-04-25T16:11:30.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:30 smithi186 ceph-mon[33258]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 737 MiB/s wr, 2.14k op/s 2024-04-25T16:11:30.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:30 smithi031 ceph-mon[29140]: pgmap v657: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 737 MiB/s wr, 2.14k op/s 2024-04-25T16:11:33.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:33 smithi031 ceph-mon[29140]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 780 MiB/s wr, 2.30k op/s 2024-04-25T16:11:33.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:33 smithi063 ceph-mon[32413]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 780 MiB/s wr, 2.30k op/s 2024-04-25T16:11:33.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:33 smithi186 ceph-mon[33258]: pgmap v658: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 780 MiB/s wr, 2.30k op/s 2024-04-25T16:11:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.920+0000 7feef57fa640 1 -- 172.21.15.31:0/3062337999 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b37def9ee0 con 0x55b37db801c0 2024-04-25T16:11:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.920+0000 7feef57fa640 10 client.27320 handle_command_reply: tid=0 2024-04-25T16:11:34.923 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.920+0000 7fef1aa70740 1 client.27320 shutdown 2024-04-25T16:11:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.923+0000 7fef1aa70740 1 -- 172.21.15.31:0/3062337999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 msgr2=0x55b37e00a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:34.924 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.923+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37e00a170 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fef08009720 tx=0x7fef08005f90 comp rx=0 tx=0).stop 2024-04-25T16:11:34.925 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.923+0000 7fef1aa70740 1 -- 172.21.15.31:0/3062337999 shutdown_connections 2024-04-25T16:11:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.923+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55b37db801c0 0x55b37e00e030 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.923+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55b37dbec1b0 0x55b37e00a170 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.923+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b37dbcb4d0 0x55b37df9ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.923+0000 7fef1aa70740 1 --2- 172.21.15.31:0/3062337999 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55b37dbc32f0 0x55b37df9a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.923+0000 7fef1aa70740 1 -- 172.21.15.31:0/3062337999 >> 172.21.15.31:0/3062337999 conn(0x55b37dbe0300 msgr2=0x55b37df86820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.923+0000 7feef57fa640 0 client.27320 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:34.926 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.924+0000 7fef1aa70740 1 -- 172.21.15.31:0/3062337999 shutdown_connections 2024-04-25T16:11:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.924+0000 7fef1aa70740 1 -- 172.21.15.31:0/3062337999 wait complete. 2024-04-25T16:11:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.924+0000 7fef1aa70740 20 client.27320 trim_cache size 0 max 16384 2024-04-25T16:11:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:11:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:11:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:11:34.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:11:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.934+0000 7fef13bc0640 1 -- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feef80afb00 msgr2=0x7feef80b2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.934+0000 7fef13bc0640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feef80afb00 0x7feef80b2010 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x55b37df97460 tx=0x7feefc005e50 comp rx=0 tx=0).stop 2024-04-25T16:11:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.934+0000 7fef13bc0640 1 -- 172.21.15.31:0/2658368111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 msgr2=0x7fef0c06fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:34.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.934+0000 7fef13bc0640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 0x7fef0c06fa20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fef04006010 tx=0x7fef04030bb0 comp rx=0 tx=0).stop 2024-04-25T16:11:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.938+0000 7fef13bc0640 1 -- 172.21.15.31:0/2658368111 shutdown_connections 2024-04-25T16:11:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.938+0000 7fef13bc0640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7feef80afb00 0x7feef80b2010 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.938+0000 7fef13bc0640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fef0c10bc80 0x7fef0c072e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.938+0000 7fef13bc0640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fef0c106f80 0x7fef0c06ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:34.940 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.938+0000 7fef13bc0640 1 --2- 172.21.15.31:0/2658368111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef0c105d40 0x7fef0c06fa20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.938+0000 7fef13bc0640 1 -- 172.21.15.31:0/2658368111 >> 172.21.15.31:0/2658368111 conn(0x7fef0c100f10 msgr2=0x7fef0c103330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.940+0000 7fef13bc0640 1 -- 172.21.15.31:0/2658368111 shutdown_connections 2024-04-25T16:11:34.941 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:34.940+0000 7fef13bc0640 1 -- 172.21.15.31:0/2658368111 wait complete. 2024-04-25T16:11:35.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:35 smithi063 ceph-mon[32413]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 943 MiB/s wr, 2.60k op/s 2024-04-25T16:11:35.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:35 smithi186 ceph-mon[33258]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 943 MiB/s wr, 2.60k op/s 2024-04-25T16:11:35.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:35 smithi031 ceph-mon[29140]: pgmap v659: 97 pgs: 97 active+clean; 10 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 943 MiB/s wr, 2.60k op/s 2024-04-25T16:11:37.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:37 smithi063 ceph-mon[32413]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 751 MiB/s wr, 2.20k op/s 2024-04-25T16:11:37.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:37 smithi186 ceph-mon[33258]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 751 MiB/s wr, 2.20k op/s 2024-04-25T16:11:37.710 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:37 smithi031 ceph-mon[29140]: pgmap v660: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 751 MiB/s wr, 2.20k op/s 2024-04-25T16:11:38.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:38 smithi063 ceph-mon[32413]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 909 MiB/s wr, 2.66k op/s 2024-04-25T16:11:38.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:38 smithi186 ceph-mon[33258]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 909 MiB/s wr, 2.66k op/s 2024-04-25T16:11:38.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:38 smithi031 ceph-mon[29140]: pgmap v661: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 909 MiB/s wr, 2.66k op/s 2024-04-25T16:11:40.450 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:11:40.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:11:40.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.940+0000 7f13700bd640 1 -- 172.21.15.31:0/3922897235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f136810b310 msgr2=0x7f136810b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:40.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.940+0000 7f13700bd640 1 --2- 172.21.15.31:0/3922897235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f136810b310 0x7f136810b720 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1360009ca0 tx=0x7f136002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.940+0000 7f13700bd640 1 -- 172.21.15.31:0/3922897235 shutdown_connections 2024-04-25T16:11:40.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.940+0000 7f13700bd640 1 --2- 172.21.15.31:0/3922897235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1368110030 0x7f1368110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.940+0000 7f13700bd640 1 --2- 172.21.15.31:0/3922897235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f136810b310 0x7f136810b720 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.940+0000 7f13700bd640 1 --2- 172.21.15.31:0/3922897235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f136810a420 0x7f136810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.940+0000 7f13700bd640 1 -- 172.21.15.31:0/3922897235 >> 172.21.15.31:0/3922897235 conn(0x7f13680704e0 msgr2=0x7f1368072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:40.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.942+0000 7f13700bd640 1 -- 172.21.15.31:0/3922897235 shutdown_connections 2024-04-25T16:11:40.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.943+0000 7f13700bd640 1 -- 172.21.15.31:0/3922897235 wait complete. 2024-04-25T16:11:40.946 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.944+0000 7f13700bd640 1 Processor -- start 2024-04-25T16:11:40.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.944+0000 7f13700bd640 1 -- start start 2024-04-25T16:11:40.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.945+0000 7f13700bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f136810a420 0x7f1368119e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.945+0000 7f13700bd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f136810b310 0x7f136811a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.945+0000 7f13700bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1368110030 0x7f136811b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.945+0000 7f13700bd640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1368111f70 con 0x7f136810b310 2024-04-25T16:11:40.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.945+0000 7f13700bd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1368111df0 con 0x7f136810a420 2024-04-25T16:11:40.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.945+0000 7f13700bd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f13681120f0 con 0x7f1368110030 2024-04-25T16:11:40.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.946+0000 7f136d658640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f136810b310 0x7f136811a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:40.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.946+0000 7f136d658640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f136810b310 0x7f136811a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41552/0 (socket says 172.21.15.31:41552) 2024-04-25T16:11:40.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.946+0000 7f136d658640 1 -- 172.21.15.31:0/3461111148 learned_addr learned my addr 172.21.15.31:0/3461111148 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:40.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.946+0000 7f136e65a640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1368110030 0x7f136811b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:40.949 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.946+0000 7f136de59640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f136810a420 0x7f1368119e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:40.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.947+0000 7f136d658640 1 -- 172.21.15.31:0/3461111148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1368110030 msgr2=0x7f136811b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:40.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.947+0000 7f136d658640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1368110030 0x7f136811b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.947+0000 7f136d658640 1 -- 172.21.15.31:0/3461111148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f136810a420 msgr2=0x7f1368119e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:40.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.947+0000 7f136d658640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f136810a420 0x7f1368119e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.947+0000 7f136d658640 1 -- 172.21.15.31:0/3461111148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136811b910 con 0x7f136810b310 2024-04-25T16:11:40.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.949+0000 7f136d658640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f136810b310 0x7f136811a340 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f136002aee0 tx=0x7f1360030820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:40.952 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.950+0000 7f135effd640 1 -- 172.21.15.31:0/3461111148 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360030b90 con 0x7f136810b310 2024-04-25T16:11:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.950+0000 7f13700bd640 1 -- 172.21.15.31:0/3461111148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136811bba0 con 0x7f136810b310 2024-04-25T16:11:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.950+0000 7f13700bd640 1 -- 172.21.15.31:0/3461111148 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136811c0f0 con 0x7f136810b310 2024-04-25T16:11:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.955+0000 7f1376f75740 1 Processor -- start 2024-04-25T16:11:40.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.955+0000 7f1376f75740 1 -- start start 2024-04-25T16:11:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.956+0000 7f1376f75740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d71345e390 0x55d7137ffa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.956+0000 7f1376f75740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d7137fffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.957+0000 7f1376f75740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71343d6b0 0x55d713800c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.957+0000 7f1376f75740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d713764d80 con 0x55d71345e390 2024-04-25T16:11:40.959 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.957+0000 7f1376f75740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d71371e8b0 con 0x55d71343d6b0 2024-04-25T16:11:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.957+0000 7f1376f75740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d713760d40 con 0x55d7134354d0 2024-04-25T16:11:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.957+0000 7f135effd640 1 -- 172.21.15.31:0/3461111148 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f136002d070 con 0x7f136810b310 2024-04-25T16:11:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.957+0000 7f136de59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d7137fffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.957+0000 7f136de59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d7137fffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:46026/0 (socket says 172.21.15.31:46026) 2024-04-25T16:11:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.957+0000 7f136de59640 1 -- 172.21.15.31:0/2257903806 learned_addr learned my addr 172.21.15.31:0/2257903806 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:40.960 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.958+0000 7f136d658640 1 --2- 172.21.15.31:0/2257903806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71343d6b0 0x55d713800c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.960+0000 7f136de59640 1 -- 172.21.15.31:0/2257903806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71343d6b0 msgr2=0x55d713800c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.960+0000 7f136de59640 1 --2- 172.21.15.31:0/2257903806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71343d6b0 0x55d713800c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.960+0000 7f136de59640 1 -- 172.21.15.31:0/2257903806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d71345e390 msgr2=0x55d7137ffa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:40.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.960+0000 7f136de59640 1 --2- 172.21.15.31:0/2257903806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d71345e390 0x55d7137ffa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.960+0000 7f136de59640 1 -- 172.21.15.31:0/2257903806 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7136efa40 con 0x55d7134354d0 2024-04-25T16:11:40.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.969+0000 7f136de59640 1 --2- 172.21.15.31:0/2257903806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d7137fffe0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1358009c20 tx=0x7f13580117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11da188dc39c1703 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:40.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.969+0000 7f135effd640 1 -- 172.21.15.31:0/3461111148 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f136003a680 con 0x7f136810b310 2024-04-25T16:11:40.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.969+0000 7f135effd640 1 -- 172.21.15.31:0/3461111148 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1360004c00 con 0x7f136810b310 2024-04-25T16:11:40.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.970+0000 7f1353fff640 1 -- 172.21.15.31:0/2257903806 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1358012450 con 0x55d7134354d0 2024-04-25T16:11:40.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.972+0000 7f1353fff640 1 -- 172.21.15.31:0/2257903806 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1358012aa0 con 0x55d7134354d0 2024-04-25T16:11:40.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.973+0000 7f1376f75740 1 -- 172.21.15.31:0/2257903806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 msgr2=0x55d7137fffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:40.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.973+0000 7f1376f75740 1 --2- 172.21.15.31:0/2257903806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d7137fffe0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f1358009c20 tx=0x7f13580117b0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.974+0000 7f135effd640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13540afc50 0x7f13540b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.974+0000 7f1376f75740 1 -- 172.21.15.31:0/2257903806 shutdown_connections 2024-04-25T16:11:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.974+0000 7f1376f75740 1 --2- 172.21.15.31:0/2257903806 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71343d6b0 0x55d713800c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.974+0000 7f1376f75740 1 --2- 172.21.15.31:0/2257903806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d7137fffe0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.974+0000 7f1376f75740 1 --2- 172.21.15.31:0/2257903806 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d71345e390 0x55d7137ffa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.974+0000 7f1376f75740 1 -- 172.21.15.31:0/2257903806 >> 172.21.15.31:0/2257903806 conn(0x55d7134524e0 msgr2=0x55d7137fb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:40.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.975+0000 7f135effd640 1 -- 172.21.15.31:0/3461111148 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1360101100 con 0x7f136810b310 2024-04-25T16:11:40.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.979+0000 7f136e65a640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13540afc50 0x7f13540b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.981+0000 7f1376f75740 1 -- 172.21.15.31:0/2257903806 shutdown_connections 2024-04-25T16:11:40.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.984+0000 7f1376f75740 1 -- 172.21.15.31:0/2257903806 wait complete. 2024-04-25T16:11:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.985+0000 7f1376f75740 1 -- start start 2024-04-25T16:11:40.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.985+0000 7f1376f75740 1 Processor -- start 2024-04-25T16:11:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.987+0000 7f1376f75740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7134354d0 0x55d713809700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.987+0000 7f1376f75740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 0x55d713809c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.987+0000 7f1376f75740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d713810ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:40.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.987+0000 7f1376f75740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55d7138017d0 con 0x55d7134354d0 2024-04-25T16:11:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.987+0000 7f1376f75740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55d7136ebd10 con 0x55d71345e390 2024-04-25T16:11:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.987+0000 7f1376f75740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55d713801950 con 0x55d71343d6b0 2024-04-25T16:11:40.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.987+0000 7f1376f75740 10 client.0 _ll_register_callbacks cb 0x55d713718210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.990+0000 7f136e65a640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13540afc50 0x7f13540b2160 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55d713800ab0 tx=0x7f13640046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.990+0000 7f136e65a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d713810ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.990+0000 7f136d658640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 0x55d713809c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.990+0000 7f136d658640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 0x55d713809c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:46032/0 (socket says 172.21.15.31:46032) 2024-04-25T16:11:40.992 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.990+0000 7f136d658640 1 -- 172.21.15.31:0/514423897 learned_addr learned my addr 172.21.15.31:0/514423897 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.990+0000 7f136de59640 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7134354d0 0x55d713809700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.991+0000 7f136d658640 1 -- 172.21.15.31:0/514423897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 msgr2=0x55d713810ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.991+0000 7f136d658640 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d713810ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.991+0000 7f136d658640 1 -- 172.21.15.31:0/514423897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7134354d0 msgr2=0x55d713809700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.991+0000 7f136d658640 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7134354d0 0x55d713809700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:40.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.991+0000 7f136d658640 1 -- 172.21.15.31:0/514423897 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7135407d0 con 0x55d71343d6b0 2024-04-25T16:11:40.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.991+0000 7f136d658640 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 0x55d713809c40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1360009dc0 tx=0x7f13600c8e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=217bac8677091660 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:40.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.992+0000 7f13517fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:40.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.992+0000 7f13517fa640 1 -- 172.21.15.31:0/514423897 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13600c9da0 con 0x55d71343d6b0 2024-04-25T16:11:40.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.992+0000 7f1376f75740 1 -- 172.21.15.31:0/514423897 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1358012aa0 con 0x55d71343d6b0 2024-04-25T16:11:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.992+0000 7f13517fa640 1 -- 172.21.15.31:0/514423897 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f13600c5930 con 0x55d71343d6b0 2024-04-25T16:11:40.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.992+0000 7f13517fa640 1 -- 172.21.15.31:0/514423897 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f13600c96a0 con 0x55d71343d6b0 2024-04-25T16:11:40.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.993+0000 7f1376f75740 10 client.27310 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:40.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.993+0000 7f1376f75740 1 -- 172.21.15.31:0/514423897 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d7134763f0 con 0x55d71343d6b0 2024-04-25T16:11:41.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.997+0000 7f13517fa640 1 -- 172.21.15.31:0/514423897 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1358012aa0 con 0x55d71343d6b0 2024-04-25T16:11:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:40.999+0000 7f13517fa640 1 -- 172.21.15.31:0/514423897 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f13600c5010 con 0x55d71343d6b0 2024-04-25T16:11:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.000+0000 7f1376f75740 10 client.27310 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.000+0000 7f1376f75740 10 client.27310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.000+0000 7f1376f75740 20 client.27310 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.000+0000 7f1376f75740 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d71341cf60 0x55d713817060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.000+0000 7f1376f75740 4 client.27310 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:41.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.000+0000 7f1376f75740 1 -- 172.21.15.31:0/514423897 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d713772990 con 0x55d71341cf60 2024-04-25T16:11:41.010 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.006+0000 7f136de59640 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d71341cf60 0x55d713817060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:41.020 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.018+0000 7f13517fa640 10 client.27310 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:41.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.018+0000 7f136de59640 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d71341cf60 0x55d713817060 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f5b93f1852d994c5 server_cookie=a95a40c2233cb8fb in_seq=0 out_seq=0 2024-04-25T16:11:41.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.023+0000 7f13517fa640 1 -- 172.21.15.31:0/514423897 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d713772990 con 0x55d71341cf60 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.023+0000 7f13517fa640 10 client.27310 handle_command_reply: tid=0 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.023+0000 7f1376f75740 1 client.27310 shutdown 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.024+0000 7f1376f75740 1 -- 172.21.15.31:0/514423897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 msgr2=0x55d713809c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.024+0000 7f1376f75740 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 0x55d713809c40 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1360009dc0 tx=0x7f13600c8e10 comp rx=0 tx=0).stop 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.026+0000 7f1376f75740 1 -- 172.21.15.31:0/514423897 shutdown_connections 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.027+0000 7f1376f75740 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d71341cf60 0x55d713817060 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.027+0000 7f1376f75740 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d713810ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.027+0000 7f1376f75740 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 0x55d713809c40 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.027+0000 7f1376f75740 1 --2- 172.21.15.31:0/514423897 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7134354d0 0x55d713809700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.027+0000 7f1376f75740 1 -- 172.21.15.31:0/514423897 >> 172.21.15.31:0/514423897 conn(0x55d7134524e0 msgr2=0x55d7137fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.027+0000 7f13517fa640 0 client.27310 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:41.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.027+0000 7f1376f75740 1 -- 172.21.15.31:0/514423897 shutdown_connections 2024-04-25T16:11:41.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.031+0000 7f1376f75740 1 -- 172.21.15.31:0/514423897 wait complete. 2024-04-25T16:11:41.034 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.031+0000 7f1376f75740 20 client.27310 trim_cache size 0 max 16384 2024-04-25T16:11:41.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.116+0000 7f1376f75740 1 Processor -- start 2024-04-25T16:11:41.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.119+0000 7f1376f75740 1 -- start start 2024-04-25T16:11:41.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.119+0000 7f1376f75740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7134354d0 0x55d7137fe120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.120+0000 7f1376f75740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 0x55d713810ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.120+0000 7f1376f75740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d7137fe660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.120+0000 7f1376f75740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f134c00e400 con 0x55d7134354d0 2024-04-25T16:11:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.120+0000 7f1376f75740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f134c00ff40 con 0x55d71345e390 2024-04-25T16:11:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.120+0000 7f1376f75740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f134c00ead0 con 0x55d71343d6b0 2024-04-25T16:11:41.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.123+0000 7f136e65a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d7137fe660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:41.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.123+0000 7f136e65a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d7137fe660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:34228/0 (socket says 172.21.15.31:34228) 2024-04-25T16:11:41.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.123+0000 7f136e65a640 1 -- 172.21.15.31:0/2369597358 learned_addr learned my addr 172.21.15.31:0/2369597358 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.127+0000 7f136e65a640 1 -- 172.21.15.31:0/2369597358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 msgr2=0x55d713810ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.127+0000 7f136e65a640 1 --2- 172.21.15.31:0/2369597358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 0x55d713810ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.127+0000 7f136e65a640 1 -- 172.21.15.31:0/2369597358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7134354d0 msgr2=0x55d7137fe120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.127+0000 7f136e65a640 1 --2- 172.21.15.31:0/2369597358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7134354d0 0x55d7137fe120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.127+0000 7f136e65a640 1 -- 172.21.15.31:0/2369597358 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7134360f0 con 0x55d71345e390 2024-04-25T16:11:41.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.130+0000 7f136e65a640 1 --2- 172.21.15.31:0/2369597358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d7137fe660 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1364011ef0 tx=0x7f1364014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd01cc438e0ea6b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.134+0000 7f13537fe640 1 -- 172.21.15.31:0/2369597358 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364003d70 con 0x55d71345e390 2024-04-25T16:11:41.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.134+0000 7f13537fe640 1 -- 172.21.15.31:0/2369597358 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1364023070 con 0x55d71345e390 2024-04-25T16:11:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.136+0000 7f1376f75740 1 -- 172.21.15.31:0/2369597358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 msgr2=0x55d7137fe660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.136+0000 7f1376f75740 1 --2- 172.21.15.31:0/2369597358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d7137fe660 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1364011ef0 tx=0x7f1364014000 comp rx=0 tx=0).stop 2024-04-25T16:11:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.136+0000 7f1376f75740 1 -- 172.21.15.31:0/2369597358 shutdown_connections 2024-04-25T16:11:41.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.136+0000 7f1376f75740 1 --2- 172.21.15.31:0/2369597358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d71345e390 0x55d7137fe660 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.136+0000 7f1376f75740 1 --2- 172.21.15.31:0/2369597358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d71343d6b0 0x55d713810ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.136+0000 7f1376f75740 1 --2- 172.21.15.31:0/2369597358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d7134354d0 0x55d7137fe120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.136+0000 7f1376f75740 1 -- 172.21.15.31:0/2369597358 >> 172.21.15.31:0/2369597358 conn(0x55d7134524e0 msgr2=0x55d713804350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:41.144 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.142+0000 7f1376f75740 1 -- 172.21.15.31:0/2369597358 shutdown_connections 2024-04-25T16:11:41.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.142+0000 7f1376f75740 1 -- 172.21.15.31:0/2369597358 wait complete. 2024-04-25T16:11:41.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.143+0000 7f1376f75740 1 -- start start 2024-04-25T16:11:41.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.143+0000 7f1376f75740 1 Processor -- start 2024-04-25T16:11:41.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.144+0000 7f1376f75740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d71380d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.144+0000 7f1376f75740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d71343d6b0 0x55d71380dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.144+0000 7f1376f75740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d7133f23a0 0x55d71387d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.144+0000 7f1376f75740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f134c00f870 con 0x55d71343d6b0 2024-04-25T16:11:41.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.144+0000 7f1376f75740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f134c00f1a0 con 0x55d7133f23a0 2024-04-25T16:11:41.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.144+0000 7f1376f75740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f134c00c490 con 0x55d7134354d0 2024-04-25T16:11:41.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.145+0000 7f1376f75740 10 client.0 _ll_register_callbacks cb 0x7f13600c9470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:41.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.145+0000 7f136de59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d71380d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:41.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.145+0000 7f136de59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d71380d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:46048/0 (socket says 172.21.15.31:46048) 2024-04-25T16:11:41.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.145+0000 7f136de59640 1 -- 172.21.15.31:0/3501380957 learned_addr learned my addr 172.21.15.31:0/3501380957 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:41.147 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.145+0000 7f136e65a640 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d7133f23a0 0x55d71387d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:41.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.145+0000 7f136d658640 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d71343d6b0 0x55d71380dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:41.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.146+0000 7f136e65a640 1 -- 172.21.15.31:0/3501380957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 msgr2=0x55d71380d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:41.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.146+0000 7f136e65a640 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d71380d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.146+0000 7f136e65a640 1 -- 172.21.15.31:0/3501380957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d71343d6b0 msgr2=0x55d71380dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:41.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.146+0000 7f136e65a640 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d71343d6b0 0x55d71380dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.146+0000 7f136e65a640 1 -- 172.21.15.31:0/3501380957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7137254b0 con 0x55d7133f23a0 2024-04-25T16:11:41.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.147+0000 7f136e65a640 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d7133f23a0 0x55d71387d400 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f13640039c0 tx=0x7f1364003da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4aea0a3b6b173ff server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:41.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.147+0000 7f13517fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.147+0000 7f13517fa640 1 -- 172.21.15.31:0/3501380957 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1364035060 con 0x55d7133f23a0 2024-04-25T16:11:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.147+0000 7f1376f75740 1 -- 172.21.15.31:0/3501380957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d71376c800 con 0x55d7133f23a0 2024-04-25T16:11:41.150 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.148+0000 7f13517fa640 1 -- 172.21.15.31:0/3501380957 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f13640051b0 con 0x55d7133f23a0 2024-04-25T16:11:41.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.151+0000 7f13517fa640 1 -- 172.21.15.31:0/3501380957 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1364003e90 con 0x55d7133f23a0 2024-04-25T16:11:41.153 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.151+0000 7f1376f75740 10 client.27338 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:41.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.151+0000 7f1376f75740 1 -- 172.21.15.31:0/3501380957 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d71387e650 con 0x55d7133f23a0 2024-04-25T16:11:41.154 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.152+0000 7f13517fa640 1 -- 172.21.15.31:0/3501380957 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55d71376c800 con 0x55d7133f23a0 2024-04-25T16:11:41.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.154+0000 7f13517fa640 1 -- 172.21.15.31:0/3501380957 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f1364027420 con 0x55d7133f23a0 2024-04-25T16:11:41.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.154+0000 7f1376f75740 10 client.27338 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:41.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.154+0000 7f1376f75740 10 client.27338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:41.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.154+0000 7f1376f75740 20 client.27338 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:41.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.154+0000 7f1376f75740 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d71387ef00 0x55d713881320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.154+0000 7f1376f75740 4 client.27338 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:11:41.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.154+0000 7f1376f75740 1 -- 172.21.15.31:0/3501380957 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d7136e74a0 con 0x55d71387ef00 2024-04-25T16:11:41.160 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.158+0000 7f136de59640 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d71387ef00 0x55d713881320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:41.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.159+0000 7f136de59640 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d71387ef00 0x55d713881320 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30fa89ebdd0057c2 server_cookie=977fb131a2983566 in_seq=0 out_seq=0 2024-04-25T16:11:41.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.159+0000 7f13517fa640 10 client.27338 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:41.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.161+0000 7f13517fa640 1 -- 172.21.15.31:0/3501380957 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d7136e74a0 con 0x55d71387ef00 2024-04-25T16:11:41.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.161+0000 7f13517fa640 10 client.27338 handle_command_reply: tid=0 2024-04-25T16:11:41.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.161+0000 7f1376f75740 1 client.27338 shutdown 2024-04-25T16:11:41.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f1376f75740 1 -- 172.21.15.31:0/3501380957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d7133f23a0 msgr2=0x55d71387d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:41.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f1376f75740 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d7133f23a0 0x55d71387d400 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f13640039c0 tx=0x7f1364003da0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f1376f75740 1 -- 172.21.15.31:0/3501380957 shutdown_connections 2024-04-25T16:11:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f1376f75740 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55d71387ef00 0x55d713881320 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f1376f75740 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55d7133f23a0 0x55d71387d400 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f1376f75740 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55d71343d6b0 0x55d71380dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f1376f75740 1 --2- 172.21.15.31:0/3501380957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d7134354d0 0x55d71380d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f1376f75740 1 -- 172.21.15.31:0/3501380957 >> 172.21.15.31:0/3501380957 conn(0x55d7134524e0 msgr2=0x55d7137fac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f13517fa640 0 client.27338 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:41.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.166+0000 7f1376f75740 1 -- 172.21.15.31:0/3501380957 shutdown_connections 2024-04-25T16:11:41.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.167+0000 7f1376f75740 1 -- 172.21.15.31:0/3501380957 wait complete. 2024-04-25T16:11:41.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.167+0000 7f1376f75740 20 client.27338 trim_cache size 0 max 16384 2024-04-25T16:11:41.170 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:11:41.170 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:11:41.170 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:11:41.170 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:11:41.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.168+0000 7f13700bd640 1 -- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13540afc50 msgr2=0x7f13540b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:41.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.169+0000 7f13700bd640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13540afc50 0x7f13540b2160 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55d713800ab0 tx=0x7f13640046c0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.169+0000 7f13700bd640 1 -- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f136810b310 msgr2=0x7f136811a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:41.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.169+0000 7f13700bd640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f136810b310 0x7f136811a340 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f136002aee0 tx=0x7f1360030820 comp rx=0 tx=0).stop 2024-04-25T16:11:41.172 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.170+0000 7f13700bd640 1 -- 172.21.15.31:0/3461111148 shutdown_connections 2024-04-25T16:11:41.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.170+0000 7f13700bd640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f13540afc50 0x7f13540b2160 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.170+0000 7f13700bd640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1368110030 0x7f136811b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.170+0000 7f13700bd640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f136810b310 0x7f136811a340 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.170+0000 7f13700bd640 1 --2- 172.21.15.31:0/3461111148 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f136810a420 0x7f1368119e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.170+0000 7f13700bd640 1 -- 172.21.15.31:0/3461111148 >> 172.21.15.31:0/3461111148 conn(0x7f13680704e0 msgr2=0x7f136810e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:41.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.174+0000 7f13700bd640 1 -- 172.21.15.31:0/3461111148 shutdown_connections 2024-04-25T16:11:41.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.177+0000 7f13700bd640 1 -- 172.21.15.31:0/3461111148 wait complete. 2024-04-25T16:11:41.233 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:11:41.234 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:11:41.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.574+0000 7f1aaf59e640 1 -- 172.21.15.31:0/942651795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010b190 msgr2=0x7f1ab010b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:41.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.574+0000 7f1aaf59e640 1 --2- 172.21.15.31:0/942651795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010b190 0x7f1ab010b5a0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0009900 tx=0x7f1aa002a410 comp rx=0 tx=0).stop 2024-04-25T16:11:41.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.574+0000 7f1aaf59e640 1 -- 172.21.15.31:0/942651795 shutdown_connections 2024-04-25T16:11:41.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.574+0000 7f1aaf59e640 1 --2- 172.21.15.31:0/942651795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab010fe90 0x7f1ab01102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.574+0000 7f1aaf59e640 1 --2- 172.21.15.31:0/942651795 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010b190 0x7f1ab010b5a0 secure :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0009900 tx=0x7f1aa002a410 comp rx=0 tx=0).stop 2024-04-25T16:11:41.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.574+0000 7f1aaf59e640 1 --2- 172.21.15.31:0/942651795 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ab010a340 0x7f1ab010ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.574+0000 7f1aaf59e640 1 -- 172.21.15.31:0/942651795 >> 172.21.15.31:0/942651795 conn(0x7f1ab0070320 msgr2=0x7f1ab00727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:41.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.575+0000 7f1aaf59e640 1 -- 172.21.15.31:0/942651795 shutdown_connections 2024-04-25T16:11:41.579 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.575+0000 7f1aaf59e640 1 -- 172.21.15.31:0/942651795 wait complete. 2024-04-25T16:11:41.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.577+0000 7f1aaf59e640 1 Processor -- start 2024-04-25T16:11:41.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.578+0000 7f1aaf59e640 1 -- start start 2024-04-25T16:11:41.580 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.578+0000 7f1aaf59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab010a340 0x7f1ab0119170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.581 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.579+0000 7f1aaf59e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010fe90 0x7f1ab0119730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.579+0000 7f1aadd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010fe90 0x7f1ab0119730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.579+0000 7f1aadd9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010fe90 0x7f1ab0119730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41608/0 (socket says 172.21.15.31:41608) 2024-04-25T16:11:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.579+0000 7f1aadd9b640 1 -- 172.21.15.31:0/4176043048 learned_addr learned my addr 172.21.15.31:0/4176043048 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.579+0000 7f1aaf59e640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ab011a610 0x7f1ab011aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.580+0000 7f1aaf59e640 1 -- 172.21.15.31:0/4176043048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab011e280 con 0x7f1ab010fe90 2024-04-25T16:11:41.582 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.580+0000 7f1aaf59e640 1 -- 172.21.15.31:0/4176043048 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab011e100 con 0x7f1ab011a610 2024-04-25T16:11:41.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.580+0000 7f1aaed9d640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ab011a610 0x7f1ab011aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:41.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.580+0000 7f1aaf59e640 1 -- 172.21.15.31:0/4176043048 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab011e400 con 0x7f1ab010a340 2024-04-25T16:11:41.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.581+0000 7f1aadd9b640 1 -- 172.21.15.31:0/4176043048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab010a340 msgr2=0x7f1ab0119170 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:11:41.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.581+0000 7f1aadd9b640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab010a340 0x7f1ab0119170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.581+0000 7f1aadd9b640 1 -- 172.21.15.31:0/4176043048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ab011a610 msgr2=0x7f1ab011aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:41.583 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.581+0000 7f1aadd9b640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ab011a610 0x7f1ab011aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:41.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.581+0000 7f1aadd9b640 1 -- 172.21.15.31:0/4176043048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab011b150 con 0x7f1ab010fe90 2024-04-25T16:11:41.584 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.582+0000 7f1aadd9b640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010fe90 0x7f1ab0119730 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f1aa002aa00 tx=0x7f1aa0003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:41.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.583+0000 7f1a9f7fe640 1 -- 172.21.15.31:0/4176043048 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa003f020 con 0x7f1ab010fe90 2024-04-25T16:11:41.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.584+0000 7f1a9f7fe640 1 -- 172.21.15.31:0/4176043048 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1aa0031c50 con 0x7f1ab010fe90 2024-04-25T16:11:41.586 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.584+0000 7f1aaf59e640 1 -- 172.21.15.31:0/4176043048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab011b3e0 con 0x7f1ab010fe90 2024-04-25T16:11:41.587 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.585+0000 7f1aaf59e640 1 -- 172.21.15.31:0/4176043048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab011b810 con 0x7f1ab010fe90 2024-04-25T16:11:41.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.588+0000 7f1a9d7fa640 1 -- 172.21.15.31:0/4176043048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a900052c0 con 0x7f1ab010fe90 2024-04-25T16:11:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.596+0000 7f1a9f7fe640 1 -- 172.21.15.31:0/4176043048 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0035070 con 0x7f1ab010fe90 2024-04-25T16:11:41.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.596+0000 7f1a9f7fe640 1 -- 172.21.15.31:0/4176043048 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1aa0042020 con 0x7f1ab010fe90 2024-04-25T16:11:41.600 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.598+0000 7f1a9f7fe640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a980afca0 0x7f1a980b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:41.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.604+0000 7f1a9f7fe640 1 -- 172.21.15.31:0/4176043048 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1aa00c59a0 con 0x7f1ab010fe90 2024-04-25T16:11:41.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.605+0000 7f1aae59c640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a980afca0 0x7f1a980b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:41.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:41 smithi063 ceph-mon[32413]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 746 MiB/s wr, 1.90k op/s 2024-04-25T16:11:41.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.617+0000 7f1aae59c640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a980afca0 0x7f1a980b21b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1ab011a210 tx=0x7f1aa4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:41.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:41.625+0000 7f1a9f7fe640 1 -- 172.21.15.31:0/4176043048 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1aa0049020 con 0x7f1ab010fe90 2024-04-25T16:11:41.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:41 smithi186 ceph-mon[33258]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 746 MiB/s wr, 1.90k op/s 2024-04-25T16:11:41.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:41 smithi031 ceph-mon[29140]: pgmap v662: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 746 MiB/s wr, 1.90k op/s 2024-04-25T16:11:42.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.449+0000 7f1a9d7fa640 1 -- 172.21.15.31:0/4176043048 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a900055b0 con 0x7f1ab010fe90 2024-04-25T16:11:42.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.451+0000 7f1a9f7fe640 1 -- 172.21.15.31:0/4176043048 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7f1aa00c9ab0 con 0x7f1ab010fe90 2024-04-25T16:11:42.458 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:11:42.459 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:11:42.460 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:11:42.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 -- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a980afca0 msgr2=0x7f1a980b21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:42.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a980afca0 0x7f1a980b21b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1ab011a210 tx=0x7f1aa4005f90 comp rx=0 tx=0).stop 2024-04-25T16:11:42.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 -- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010fe90 msgr2=0x7f1ab0119730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:42.468 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010fe90 0x7f1ab0119730 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f1aa002aa00 tx=0x7f1aa0003b90 comp rx=0 tx=0).stop 2024-04-25T16:11:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 -- 172.21.15.31:0/4176043048 shutdown_connections 2024-04-25T16:11:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1a980afca0 0x7f1a980b21b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ab011a610 0x7f1ab011aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1ab010fe90 0x7f1ab0119730 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 --2- 172.21.15.31:0/4176043048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab010a340 0x7f1ab0119170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.465+0000 7f1a9d7fa640 1 -- 172.21.15.31:0/4176043048 >> 172.21.15.31:0/4176043048 conn(0x7f1ab0070320 msgr2=0x7f1ab010de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.470+0000 7f1a9d7fa640 1 -- 172.21.15.31:0/4176043048 shutdown_connections 2024-04-25T16:11:42.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.477+0000 7f1a9d7fa640 1 -- 172.21.15.31:0/4176043048 wait complete. 2024-04-25T16:11:42.528 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:11:42.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.962+0000 7fce43fff640 1 -- 172.21.15.31:0/2575443416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410fe90 msgr2=0x7fce441102a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:42.966 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.962+0000 7fce43fff640 1 --2- 172.21.15.31:0/2575443416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410fe90 0x7fce441102a0 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009ca0 tx=0x7fce3c02a8f0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.963+0000 7fce43fff640 1 -- 172.21.15.31:0/2575443416 shutdown_connections 2024-04-25T16:11:42.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.963+0000 7fce43fff640 1 --2- 172.21.15.31:0/2575443416 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410fe90 0x7fce441102a0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.963+0000 7fce43fff640 1 --2- 172.21.15.31:0/2575443416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce4410b190 0x7fce4410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.963+0000 7fce43fff640 1 --2- 172.21.15.31:0/2575443416 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4410a340 0x7fce4410ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.963+0000 7fce43fff640 1 -- 172.21.15.31:0/2575443416 >> 172.21.15.31:0/2575443416 conn(0x7fce44070320 msgr2=0x7fce440727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:42.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.963+0000 7fce43fff640 1 -- 172.21.15.31:0/2575443416 shutdown_connections 2024-04-25T16:11:42.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.964+0000 7fce43fff640 1 -- 172.21.15.31:0/2575443416 wait complete. 2024-04-25T16:11:42.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.965+0000 7fce43fff640 1 Processor -- start 2024-04-25T16:11:42.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.965+0000 7fce43fff640 1 -- start start 2024-04-25T16:11:42.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.965+0000 7fce43fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410a340 0x7fce441a8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:42.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.965+0000 7fce43fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce4410b190 0x7fce441a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:42.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.966+0000 7fce43fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4410fe90 0x7fce441b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:42.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.966+0000 7fce43fff640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fce4411e340 con 0x7fce4410a340 2024-04-25T16:11:42.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.966+0000 7fce43fff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fce4411e1c0 con 0x7fce4410fe90 2024-04-25T16:11:42.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.966+0000 7fce43fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce4411e4c0 con 0x7fce4410b190 2024-04-25T16:11:42.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.966+0000 7fce42ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410a340 0x7fce441a8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:42.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.966+0000 7fce42ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410a340 0x7fce441a8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41620/0 (socket says 172.21.15.31:41620) 2024-04-25T16:11:42.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.967+0000 7fce42ffd640 1 -- 172.21.15.31:0/1878921557 learned_addr learned my addr 172.21.15.31:0/1878921557 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:42.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.967+0000 7fce42ffd640 1 -- 172.21.15.31:0/1878921557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce4410b190 msgr2=0x7fce441a94f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:11:42.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.967+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce4410b190 0x7fce441a94f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.967+0000 7fce42ffd640 1 -- 172.21.15.31:0/1878921557 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4410fe90 msgr2=0x7fce441b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:42.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.968+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4410fe90 0x7fce441b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.968+0000 7fce42ffd640 1 -- 172.21.15.31:0/1878921557 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce44111c90 con 0x7fce4410a340 2024-04-25T16:11:42.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.968+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410a340 0x7fce441a8fb0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fce3800b450 tx=0x7fce3800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:42.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.970+0000 7fce2bfff640 1 -- 172.21.15.31:0/1878921557 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce38015020 con 0x7fce4410a340 2024-04-25T16:11:42.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.970+0000 7fce43fff640 1 -- 172.21.15.31:0/1878921557 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce44111f80 con 0x7fce4410a340 2024-04-25T16:11:42.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.970+0000 7fce43fff640 1 -- 172.21.15.31:0/1878921557 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce44112550 con 0x7fce4410a340 2024-04-25T16:11:42.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.970+0000 7fce2bfff640 1 -- 172.21.15.31:0/1878921557 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce38011d40 con 0x7fce4410a340 2024-04-25T16:11:42.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.971+0000 7fce2bfff640 1 -- 172.21.15.31:0/1878921557 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce38010e30 con 0x7fce4410a340 2024-04-25T16:11:42.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.980+0000 7fce2bfff640 1 -- 172.21.15.31:0/1878921557 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fce38018020 con 0x7fce4410a340 2024-04-25T16:11:42.984 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.981+0000 7fce2bfff640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce2c0afcf0 0x7fce2c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:42.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.983+0000 7fce2bfff640 1 -- 172.21.15.31:0/1878921557 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fce380df600 con 0x7fce4410a340 2024-04-25T16:11:42.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.983+0000 7fce5062c740 1 Processor -- start 2024-04-25T16:11:42.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.984+0000 7fce5062c740 1 -- start start 2024-04-25T16:11:42.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.984+0000 7fce5062c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad26b390 0x5617ad60d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:42.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.984+0000 7fce5062c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad2424d0 0x5617ad60df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.984+0000 7fce5062c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad24a6b0 0x5617ad60ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.984+0000 7fce5062c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5617ad571d80 con 0x5617ad24a6b0 2024-04-25T16:11:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.984+0000 7fce5062c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5617ad52ba20 con 0x5617ad2424d0 2024-04-25T16:11:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.984+0000 7fce5062c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5617ad56dd40 con 0x5617ad26b390 2024-04-25T16:11:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.986+0000 7fce437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad24a6b0 0x5617ad60ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.986+0000 7fce437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad24a6b0 0x5617ad60ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41630/0 (socket says 172.21.15.31:41630) 2024-04-25T16:11:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.986+0000 7fce437fe640 1 -- 172.21.15.31:0/3806103023 learned_addr learned my addr 172.21.15.31:0/3806103023 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:42.990 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.986+0000 7fce427fc640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce2c0afcf0 0x7fce2c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:42.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.986+0000 7fce42ffd640 1 --2- 172.21.15.31:0/3806103023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad26b390 0x5617ad60d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:42.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.988+0000 7fce42ffd640 1 -- 172.21.15.31:0/3806103023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad2424d0 msgr2=0x5617ad60df60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:42.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.988+0000 7fce42ffd640 1 --2- 172.21.15.31:0/3806103023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad2424d0 0x5617ad60df60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.988+0000 7fce42ffd640 1 -- 172.21.15.31:0/3806103023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad24a6b0 msgr2=0x5617ad60ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:42.993 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.988+0000 7fce42ffd640 1 --2- 172.21.15.31:0/3806103023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad24a6b0 0x5617ad60ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:42.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.988+0000 7fce42ffd640 1 -- 172.21.15.31:0/3806103023 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617ad536c30 con 0x5617ad26b390 2024-04-25T16:11:42.994 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.989+0000 7fce42ffd640 1 --2- 172.21.15.31:0/3806103023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad26b390 0x5617ad60d9e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fce380a4d00 tx=0x7fce380a7990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d9d380c6321ef53 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:42.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.993+0000 7fce297fa640 1 -- 172.21.15.31:0/3806103023 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce380b0240 con 0x5617ad26b390 2024-04-25T16:11:42.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:42.993+0000 7fce297fa640 1 -- 172.21.15.31:0/3806103023 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce380abe40 con 0x5617ad26b390 2024-04-25T16:11:43.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.000+0000 7fce5062c740 1 -- 172.21.15.31:0/3806103023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad26b390 msgr2=0x5617ad60d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.000+0000 7fce5062c740 1 --2- 172.21.15.31:0/3806103023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad26b390 0x5617ad60d9e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fce380a4d00 tx=0x7fce380a7990 comp rx=0 tx=0).stop 2024-04-25T16:11:43.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.000+0000 7fce5062c740 1 -- 172.21.15.31:0/3806103023 shutdown_connections 2024-04-25T16:11:43.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.000+0000 7fce5062c740 1 --2- 172.21.15.31:0/3806103023 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad24a6b0 0x5617ad60ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.000+0000 7fce5062c740 1 --2- 172.21.15.31:0/3806103023 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad2424d0 0x5617ad60df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.000+0000 7fce5062c740 1 --2- 172.21.15.31:0/3806103023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad26b390 0x5617ad60d9e0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.008 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.000+0000 7fce5062c740 1 -- 172.21.15.31:0/3806103023 >> 172.21.15.31:0/3806103023 conn(0x5617ad25f4e0 msgr2=0x5617ad6091f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:43.012 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.009+0000 7fce5062c740 1 -- 172.21.15.31:0/3806103023 shutdown_connections 2024-04-25T16:11:43.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.011+0000 7fce5062c740 1 -- 172.21.15.31:0/3806103023 wait complete. 2024-04-25T16:11:43.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.012+0000 7fce5062c740 1 -- start start 2024-04-25T16:11:43.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.012+0000 7fce5062c740 1 Processor -- start 2024-04-25T16:11:43.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.013+0000 7fce5062c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad2424d0 0x5617ad617780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.013+0000 7fce5062c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad24a6b0 0x5617ad617cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.013+0000 7fce5062c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.013+0000 7fce5062c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5617ad60f750 con 0x5617ad26b390 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.014+0000 7fce5062c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5617ad4f8dc0 con 0x5617ad2424d0 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.014+0000 7fce5062c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5617ad60f8d0 con 0x5617ad24a6b0 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.014+0000 7fce5062c740 10 client.0 _ll_register_callbacks cb 0x5617ad514600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.015+0000 7fce437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.015+0000 7fce437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41644/0 (socket says 172.21.15.31:41644) 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.015+0000 7fce437fe640 1 -- 172.21.15.31:0/1957903381 learned_addr learned my addr 172.21.15.31:0/1957903381 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.017+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad2424d0 0x5617ad617780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.018+0000 7fce437fe640 1 -- 172.21.15.31:0/1957903381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad24a6b0 msgr2=0x5617ad617cc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.018+0000 7fce437fe640 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad24a6b0 0x5617ad617cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.018+0000 7fce437fe640 1 -- 172.21.15.31:0/1957903381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad2424d0 msgr2=0x5617ad617780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.018+0000 7fce437fe640 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad2424d0 0x5617ad617780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.018+0000 7fce437fe640 1 -- 172.21.15.31:0/1957903381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617ad2833f0 con 0x5617ad26b390 2024-04-25T16:11:43.023 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.020+0000 7fce427fc640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce2c0afcf0 0x7fce2c0b2200 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x5617ad60d700 tx=0x7fce34006bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.024 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.020+0000 7fce437fe640 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61ec20 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fce3c031840 tx=0x7fce3c031870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c8b6765f6366a99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.023+0000 7fce26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:43.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.023+0000 7fce26ff5640 1 -- 172.21.15.31:0/1957903381 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c004040 con 0x5617ad26b390 2024-04-25T16:11:43.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.023+0000 7fce26ff5640 1 -- 172.21.15.31:0/1957903381 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fce3c0041e0 con 0x5617ad26b390 2024-04-25T16:11:43.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.023+0000 7fce5062c740 1 -- 172.21.15.31:0/1957903381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce380abe40 con 0x5617ad26b390 2024-04-25T16:11:43.035 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.026+0000 7fce26ff5640 1 -- 172.21.15.31:0/1957903381 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce3c004420 con 0x5617ad26b390 2024-04-25T16:11:43.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.026+0000 7fce5062c740 10 client.17655 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:43.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.027+0000 7fce5062c740 1 -- 172.21.15.31:0/1957903381 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5617ad34d7d0 con 0x5617ad26b390 2024-04-25T16:11:43.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.030+0000 7fce26ff5640 1 -- 172.21.15.31:0/1957903381 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce380abe40 con 0x5617ad26b390 2024-04-25T16:11:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.045+0000 7fce26ff5640 1 -- 172.21.15.31:0/1957903381 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fce3c043070 con 0x5617ad26b390 2024-04-25T16:11:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.045+0000 7fce5062c740 10 client.17655 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.045+0000 7fce5062c740 10 client.17655 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:11:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.045+0000 7fce5062c740 20 client.17655 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.045+0000 7fce5062c740 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5617ad229f60 0x5617ad6254e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.045+0000 7fce5062c740 4 client.17655 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:43.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.045+0000 7fce5062c740 1 -- 172.21.15.31:0/1957903381 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617ad57f990 con 0x5617ad229f60 2024-04-25T16:11:43.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.050+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5617ad229f60 0x5617ad6254e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.063+0000 7fce26ff5640 10 client.17655 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:11:43.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.063+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5617ad229f60 0x5617ad6254e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb090013cba760e3 server_cookie=92b7094e12b1521 in_seq=0 out_seq=0 2024-04-25T16:11:43.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.067+0000 7fce26ff5640 1 -- 172.21.15.31:0/1957903381 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5617ad57f990 con 0x5617ad229f60 2024-04-25T16:11:43.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.067+0000 7fce26ff5640 10 client.17655 handle_command_reply: tid=0 2024-04-25T16:11:43.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.067+0000 7fce5062c740 1 client.17655 shutdown 2024-04-25T16:11:43.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.073+0000 7fce5062c740 1 -- 172.21.15.31:0/1957903381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 msgr2=0x5617ad61ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.073+0000 7fce5062c740 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61ec20 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fce3c031840 tx=0x7fce3c031870 comp rx=0 tx=0).stop 2024-04-25T16:11:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.073+0000 7fce5062c740 1 -- 172.21.15.31:0/1957903381 shutdown_connections 2024-04-25T16:11:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.073+0000 7fce5062c740 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5617ad229f60 0x5617ad6254e0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.073+0000 7fce5062c740 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61ec20 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.073+0000 7fce26ff5640 0 client.17655 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:11:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.073+0000 7fce5062c740 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad24a6b0 0x5617ad617cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.073+0000 7fce5062c740 1 --2- 172.21.15.31:0/1957903381 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad2424d0 0x5617ad617780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.073+0000 7fce5062c740 1 -- 172.21.15.31:0/1957903381 >> 172.21.15.31:0/1957903381 conn(0x5617ad25f4e0 msgr2=0x5617ad6083f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:43.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.074+0000 7fce5062c740 1 -- 172.21.15.31:0/1957903381 shutdown_connections 2024-04-25T16:11:43.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.074+0000 7fce5062c740 1 -- 172.21.15.31:0/1957903381 wait complete. 2024-04-25T16:11:43.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.074+0000 7fce5062c740 20 client.17655 trim_cache size 0 max 16384 2024-04-25T16:11:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.118+0000 7fce5062c740 1 Processor -- start 2024-04-25T16:11:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.118+0000 7fce5062c740 1 -- start start 2024-04-25T16:11:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.118+0000 7fce5062c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 0x5617ad617e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.119+0000 7fce5062c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad24a6b0 0x5617ad616a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.119+0000 7fce5062c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad616f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.119+0000 7fce5062c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fce3000e3b0 con 0x5617ad26b390 2024-04-25T16:11:43.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.119+0000 7fce5062c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fce3000fef0 con 0x5617ad24a6b0 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.119+0000 7fce5062c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce3000ea80 con 0x5617ad2424d0 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.119+0000 7fce437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad616f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.119+0000 7fce437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad616f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41652/0 (socket says 172.21.15.31:41652) 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.119+0000 7fce437fe640 1 -- 172.21.15.31:0/1060036507 learned_addr learned my addr 172.21.15.31:0/1060036507 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.119+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1060036507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 0x5617ad617e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.120+0000 7fce437fe640 1 -- 172.21.15.31:0/1060036507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 msgr2=0x5617ad617e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.120+0000 7fce437fe640 1 --2- 172.21.15.31:0/1060036507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 0x5617ad617e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.120+0000 7fce437fe640 1 -- 172.21.15.31:0/1060036507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad24a6b0 msgr2=0x5617ad616a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.120+0000 7fce437fe640 1 --2- 172.21.15.31:0/1060036507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad24a6b0 0x5617ad616a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.120+0000 7fce437fe640 1 -- 172.21.15.31:0/1060036507 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617ad60f970 con 0x5617ad26b390 2024-04-25T16:11:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.121+0000 7fce437fe640 1 --2- 172.21.15.31:0/1060036507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad616f70 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fce3c002230 tx=0x7fce3c0314f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6029f139c291d201 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.122+0000 7fce28ff9640 1 -- 172.21.15.31:0/1060036507 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c004800 con 0x5617ad26b390 2024-04-25T16:11:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.122+0000 7fce28ff9640 1 -- 172.21.15.31:0/1060036507 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce3c054020 con 0x5617ad26b390 2024-04-25T16:11:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.122+0000 7fce28ff9640 1 -- 172.21.15.31:0/1060036507 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c0305f0 con 0x5617ad26b390 2024-04-25T16:11:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.124+0000 7fce5062c740 1 -- 172.21.15.31:0/1060036507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 msgr2=0x5617ad616f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.124+0000 7fce5062c740 1 --2- 172.21.15.31:0/1060036507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad616f70 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fce3c002230 tx=0x7fce3c0314f0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.124+0000 7fce5062c740 1 -- 172.21.15.31:0/1060036507 shutdown_connections 2024-04-25T16:11:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.125+0000 7fce5062c740 1 --2- 172.21.15.31:0/1060036507 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad616f70 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.125+0000 7fce5062c740 1 --2- 172.21.15.31:0/1060036507 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad24a6b0 0x5617ad616a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.125+0000 7fce5062c740 1 --2- 172.21.15.31:0/1060036507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 0x5617ad617e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.125+0000 7fce5062c740 1 -- 172.21.15.31:0/1060036507 >> 172.21.15.31:0/1060036507 conn(0x5617ad25f4e0 msgr2=0x5617ad617780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.125+0000 7fce5062c740 1 -- 172.21.15.31:0/1060036507 shutdown_connections 2024-04-25T16:11:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.125+0000 7fce5062c740 1 -- 172.21.15.31:0/1060036507 wait complete. 2024-04-25T16:11:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.126+0000 7fce5062c740 1 -- start start 2024-04-25T16:11:43.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.127+0000 7fce5062c740 1 Processor -- start 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.127+0000 7fce5062c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 0x5617ad61baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.127+0000 7fce5062c740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad24a6b0 0x5617ad61bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.127+0000 7fce5062c740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.128+0000 7fce5062c740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fce3000f820 con 0x5617ad26b390 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.128+0000 7fce5062c740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fce3000f150 con 0x5617ad24a6b0 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.128+0000 7fce5062c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce3000c440 con 0x5617ad2424d0 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.128+0000 7fce5062c740 10 client.0 _ll_register_callbacks cb 0x7fce3c02f340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.128+0000 7fce437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.128+0000 7fce437fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61cf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41654/0 (socket says 172.21.15.31:41654) 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.128+0000 7fce437fe640 1 -- 172.21.15.31:0/1930634939 learned_addr learned my addr 172.21.15.31:0/1930634939 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:43.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.129+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 0x5617ad61baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.129+0000 7fce437fe640 1 -- 172.21.15.31:0/1930634939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 msgr2=0x5617ad61baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.129+0000 7fce437fe640 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 0x5617ad61baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.129+0000 7fce437fe640 1 -- 172.21.15.31:0/1930634939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad24a6b0 msgr2=0x5617ad61bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.134 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.129+0000 7fce437fe640 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad24a6b0 0x5617ad61bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.129+0000 7fce437fe640 1 -- 172.21.15.31:0/1930634939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617ad61d4d0 con 0x5617ad26b390 2024-04-25T16:11:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.131+0000 7fce437fe640 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61cf10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009ca0 tx=0x7fce3c005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170cb6272ae33737 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.135 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.131+0000 7fce26ff5640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.135+0000 7fce26ff5640 1 -- 172.21.15.31:0/1930634939 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3c004800 con 0x5617ad26b390 2024-04-25T16:11:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.135+0000 7fce5062c740 1 -- 172.21.15.31:0/1930634939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce3c054020 con 0x5617ad26b390 2024-04-25T16:11:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.135+0000 7fce26ff5640 1 -- 172.21.15.31:0/1930634939 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fce3c004120 con 0x5617ad26b390 2024-04-25T16:11:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.135+0000 7fce26ff5640 1 -- 172.21.15.31:0/1930634939 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce3c02f640 con 0x5617ad26b390 2024-04-25T16:11:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.135+0000 7fce5062c740 10 client.17661 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:43.138 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.136+0000 7fce5062c740 1 -- 172.21.15.31:0/1930634939 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5617ad68d850 con 0x5617ad26b390 2024-04-25T16:11:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.137+0000 7fce26ff5640 1 -- 172.21.15.31:0/1930634939 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce3c054020 con 0x5617ad26b390 2024-04-25T16:11:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.137+0000 7fce26ff5640 1 -- 172.21.15.31:0/1930634939 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fce3c043070 con 0x5617ad26b390 2024-04-25T16:11:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.137+0000 7fce5062c740 10 client.17661 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:43.139 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.137+0000 7fce5062c740 10 client.17661 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:11:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.137+0000 7fce5062c740 20 client.17661 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.137+0000 7fce5062c740 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5617ad1ff3a0 0x5617ad690230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.137+0000 7fce5062c740 4 client.17661 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:43.140 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.137+0000 7fce5062c740 1 -- 172.21.15.31:0/1930634939 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5617ad579800 con 0x5617ad1ff3a0 2024-04-25T16:11:43.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.140+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5617ad1ff3a0 0x5617ad690230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.155+0000 7fce42ffd640 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5617ad1ff3a0 0x5617ad690230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=deaececd525e64cf server_cookie=c7431d9ff192cd22 in_seq=0 out_seq=0 2024-04-25T16:11:43.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.155+0000 7fce26ff5640 10 client.17661 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:11:43.176 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.166+0000 7fce26ff5640 1 -- 172.21.15.31:0/1930634939 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5617ad579800 con 0x5617ad1ff3a0 2024-04-25T16:11:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.166+0000 7fce26ff5640 10 client.17661 handle_command_reply: tid=0 2024-04-25T16:11:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.166+0000 7fce5062c740 1 client.17661 shutdown 2024-04-25T16:11:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce5062c740 1 -- 172.21.15.31:0/1930634939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 msgr2=0x5617ad61cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce5062c740 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61cf10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fce3c009ca0 tx=0x7fce3c005bc0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce5062c740 1 -- 172.21.15.31:0/1930634939 shutdown_connections 2024-04-25T16:11:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce5062c740 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x5617ad1ff3a0 0x5617ad690230 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce5062c740 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5617ad26b390 0x5617ad61cf10 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce5062c740 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5617ad24a6b0 0x5617ad61bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce5062c740 1 --2- 172.21.15.31:0/1930634939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5617ad2424d0 0x5617ad61baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce5062c740 1 -- 172.21.15.31:0/1930634939 >> 172.21.15.31:0/1930634939 conn(0x5617ad25f4e0 msgr2=0x5617ad616a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce26ff5640 0 client.17661 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:11:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.167+0000 7fce5062c740 1 -- 172.21.15.31:0/1930634939 shutdown_connections 2024-04-25T16:11:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.168+0000 7fce5062c740 1 -- 172.21.15.31:0/1930634939 wait complete. 2024-04-25T16:11:43.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.169+0000 7fce5062c740 20 client.17661 trim_cache size 0 max 16384 2024-04-25T16:11:43.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.177+0000 7fce43fff640 1 -- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce2c0afcf0 msgr2=0x7fce2c0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.177+0000 7fce43fff640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce2c0afcf0 0x7fce2c0b2200 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x5617ad60d700 tx=0x7fce34006bd0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.178+0000 7fce43fff640 1 -- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410a340 msgr2=0x7fce441a8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.178+0000 7fce43fff640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410a340 0x7fce441a8fb0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fce3800b450 tx=0x7fce3800b910 comp rx=0 tx=0).stop 2024-04-25T16:11:43.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.179+0000 7fce43fff640 1 -- 172.21.15.31:0/1878921557 shutdown_connections 2024-04-25T16:11:43.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.179+0000 7fce43fff640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce2c0afcf0 0x7fce2c0b2200 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.179+0000 7fce43fff640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce4410fe90 0x7fce441b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.179+0000 7fce43fff640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce4410b190 0x7fce441a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.179+0000 7fce43fff640 1 --2- 172.21.15.31:0/1878921557 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce4410a340 0x7fce441a8fb0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.179+0000 7fce43fff640 1 -- 172.21.15.31:0/1878921557 >> 172.21.15.31:0/1878921557 conn(0x7fce44070320 msgr2=0x7fce4410be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:43.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.179+0000 7fce43fff640 1 -- 172.21.15.31:0/1878921557 shutdown_connections 2024-04-25T16:11:43.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.179+0000 7fce43fff640 1 -- 172.21.15.31:0/1878921557 wait complete. 2024-04-25T16:11:43.278 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:11:43.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:43 smithi063 ceph-mon[32413]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 799 MiB/s wr, 1.92k op/s 2024-04-25T16:11:43.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:43 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4176043048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:11:43.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:43 smithi186 ceph-mon[33258]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 799 MiB/s wr, 1.92k op/s 2024-04-25T16:11:43.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:43 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4176043048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:11:43.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:43 smithi031 ceph-mon[29140]: pgmap v663: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 799 MiB/s wr, 1.92k op/s 2024-04-25T16:11:43.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:43 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4176043048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:11:43.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.738+0000 7fcf69892640 1 -- 172.21.15.31:0/4214778001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 msgr2=0x7fcf64110300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.738+0000 7fcf69892640 1 --2- 172.21.15.31:0/4214778001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 0x7fcf64110300 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcf54009ca0 tx=0x7fcf5402a8f0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.761+0000 7fcf69892640 1 -- 172.21.15.31:0/4214778001 shutdown_connections 2024-04-25T16:11:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.761+0000 7fcf69892640 1 --2- 172.21.15.31:0/4214778001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 0x7fcf64110300 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.761+0000 7fcf69892640 1 --2- 172.21.15.31:0/4214778001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf6410b1d0 0x7fcf6410b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.761+0000 7fcf69892640 1 --2- 172.21.15.31:0/4214778001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf6410a3e0 0x7fcf6410ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.761+0000 7fcf69892640 1 -- 172.21.15.31:0/4214778001 >> 172.21.15.31:0/4214778001 conn(0x7fcf64070400 msgr2=0x7fcf64072840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:43.782 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.776+0000 7fcf69892640 1 -- 172.21.15.31:0/4214778001 shutdown_connections 2024-04-25T16:11:43.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.778+0000 7fcf69892640 1 -- 172.21.15.31:0/4214778001 wait complete. 2024-04-25T16:11:43.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.779+0000 7fcf69892640 1 Processor -- start 2024-04-25T16:11:43.783 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.780+0000 7fcf69892640 1 -- start start 2024-04-25T16:11:43.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.780+0000 7fcf69892640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf6410a3e0 0x7fcf641a8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.780+0000 7fcf69892640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf6410b1d0 0x7fcf641a92e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.781+0000 7fcf69892640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 0x7fcf641b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.781+0000 7fcf69892640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6411e280 con 0x7fcf6410b1d0 2024-04-25T16:11:43.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.781+0000 7fcf69892640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6411e100 con 0x7fcf6410a3e0 2024-04-25T16:11:43.784 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.781+0000 7fcf69892640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6411e400 con 0x7fcf6410fef0 2024-04-25T16:11:43.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.782+0000 7fcf637fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 0x7fcf641b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.782+0000 7fcf637fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 0x7fcf641b0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:42026/0 (socket says 172.21.15.31:42026) 2024-04-25T16:11:43.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.782+0000 7fcf637fe640 1 -- 172.21.15.31:0/2918170894 learned_addr learned my addr 172.21.15.31:0/2918170894 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:43.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.783+0000 7fcf62ffd640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf6410a3e0 0x7fcf641a8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.785+0000 7fcf637fe640 1 -- 172.21.15.31:0/2918170894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf6410a3e0 msgr2=0x7fcf641a8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.785+0000 7fcf637fe640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf6410a3e0 0x7fcf641a8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.785+0000 7fcf637fe640 1 -- 172.21.15.31:0/2918170894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf6410b1d0 msgr2=0x7fcf641a92e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.785+0000 7fcf637fe640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf6410b1d0 0x7fcf641a92e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.785+0000 7fcf637fe640 1 -- 172.21.15.31:0/2918170894 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf641b08e0 con 0x7fcf6410fef0 2024-04-25T16:11:43.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.789+0000 7fcf637fe640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 0x7fcf641b0270 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcf54030820 tx=0x7fcf54030850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.811+0000 7fcf68890640 1 -- 172.21.15.31:0/2918170894 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf54030cb0 con 0x7fcf6410fef0 2024-04-25T16:11:43.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.811+0000 7fcf69892640 1 -- 172.21.15.31:0/2918170894 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf641b0b70 con 0x7fcf6410fef0 2024-04-25T16:11:43.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.811+0000 7fcf69892640 1 -- 172.21.15.31:0/2918170894 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf641b10c0 con 0x7fcf6410fef0 2024-04-25T16:11:43.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.816+0000 7fcf70818740 1 Processor -- start 2024-04-25T16:11:43.819 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.816+0000 7fcf70818740 1 -- start start 2024-04-25T16:11:43.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.826+0000 7fcf70818740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbb39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.826+0000 7fcf70818740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbb3f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.826+0000 7fcf70818740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b7f06b0 0x55581bbb4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.826+0000 7fcf70818740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55581bb17d80 con 0x55581b7e84d0 2024-04-25T16:11:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.827+0000 7fcf70818740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55581bad1a20 con 0x55581b811390 2024-04-25T16:11:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.827+0000 7fcf627fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbb39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.827+0000 7fcf627fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbb39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48628/0 (socket says 172.21.15.31:48628) 2024-04-25T16:11:43.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.827+0000 7fcf627fc640 1 -- 172.21.15.31:0/3949052585 learned_addr learned my addr 172.21.15.31:0/3949052585 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.827+0000 7fcf70818740 1 -- 172.21.15.31:0/3949052585 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55581bb13d40 con 0x55581b7f06b0 2024-04-25T16:11:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.827+0000 7fcf68890640 1 -- 172.21.15.31:0/2918170894 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf5402d070 con 0x7fcf6410fef0 2024-04-25T16:11:43.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.827+0000 7fcf627fc640 1 --2- 172.21.15.31:0/3949052585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b7f06b0 0x55581bbb4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.829+0000 7fcf68890640 1 -- 172.21.15.31:0/2918170894 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5403a680 con 0x7fcf6410fef0 2024-04-25T16:11:43.832 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.830+0000 7fcf627fc640 1 -- 172.21.15.31:0/3949052585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b7f06b0 msgr2=0x55581bbb4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.830+0000 7fcf627fc640 1 --2- 172.21.15.31:0/3949052585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b7f06b0 0x55581bbb4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.830+0000 7fcf627fc640 1 -- 172.21.15.31:0/3949052585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 msgr2=0x55581bbb3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.830+0000 7fcf627fc640 1 --2- 172.21.15.31:0/3949052585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbb3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.830+0000 7fcf627fc640 1 -- 172.21.15.31:0/3949052585 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55581badcc30 con 0x55581b811390 2024-04-25T16:11:43.833 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.831+0000 7fcf68890640 1 -- 172.21.15.31:0/2918170894 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fcf5403a820 con 0x7fcf6410fef0 2024-04-25T16:11:43.840 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.831+0000 7fcf627fc640 1 --2- 172.21.15.31:0/3949052585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbb39e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcf58014810 tx=0x7fcf5800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53697aae96461e19 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.842 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.839+0000 7fcf68890640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcf4c0afc10 0x7fcf4c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.843 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.840+0000 7fcf68890640 1 -- 172.21.15.31:0/2918170894 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcf54100ff0 con 0x7fcf6410fef0 2024-04-25T16:11:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.842+0000 7fcf49ffb640 1 -- 172.21.15.31:0/3949052585 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf580142f0 con 0x55581b811390 2024-04-25T16:11:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.842+0000 7fcf49ffb640 1 -- 172.21.15.31:0/3949052585 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf58014e50 con 0x55581b811390 2024-04-25T16:11:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.843+0000 7fcf62ffd640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcf4c0afc10 0x7fcf4c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.844+0000 7fcf70818740 1 -- 172.21.15.31:0/3949052585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 msgr2=0x55581bbb39e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.849 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.844+0000 7fcf70818740 1 --2- 172.21.15.31:0/3949052585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbb39e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fcf58014810 tx=0x7fcf5800b330 comp rx=0 tx=0).stop 2024-04-25T16:11:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.847+0000 7fcf70818740 1 -- 172.21.15.31:0/3949052585 shutdown_connections 2024-04-25T16:11:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.847+0000 7fcf70818740 1 --2- 172.21.15.31:0/3949052585 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b7f06b0 0x55581bbb4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.847+0000 7fcf70818740 1 --2- 172.21.15.31:0/3949052585 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbb3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.847+0000 7fcf70818740 1 --2- 172.21.15.31:0/3949052585 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbb39e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.847+0000 7fcf70818740 1 -- 172.21.15.31:0/3949052585 >> 172.21.15.31:0/3949052585 conn(0x55581b8054e0 msgr2=0x55581bbaf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.851+0000 7fcf70818740 1 -- 172.21.15.31:0/3949052585 shutdown_connections 2024-04-25T16:11:43.853 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.851+0000 7fcf62ffd640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcf4c0afc10 0x7fcf4c0b2120 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c005fd0 tx=0x7fcf5c009450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.851+0000 7fcf70818740 1 -- 172.21.15.31:0/3949052585 wait complete. 2024-04-25T16:11:43.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.852+0000 7fcf70818740 1 -- start start 2024-04-25T16:11:43.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.853+0000 7fcf70818740 1 Processor -- start 2024-04-25T16:11:43.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.855+0000 7fcf70818740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbbd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.856+0000 7fcf70818740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbbdd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.856+0000 7fcf70818740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 0x55581bbc4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.856+0000 7fcf70818740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55581bbb5750 con 0x55581b7e84d0 2024-04-25T16:11:43.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.857+0000 7fcf70818740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55581ba9edc0 con 0x55581b7f06b0 2024-04-25T16:11:43.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.857+0000 7fcf70818740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55581bbb58d0 con 0x55581b811390 2024-04-25T16:11:43.860 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.857+0000 7fcf70818740 10 client.0 _ll_register_callbacks cb 0x55581baba600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:43.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.860+0000 7fcf62ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbbd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.860+0000 7fcf62ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbbd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39200/0 (socket says 172.21.15.31:39200) 2024-04-25T16:11:43.863 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.860+0000 7fcf62ffd640 1 -- 172.21.15.31:0/3677842190 learned_addr learned my addr 172.21.15.31:0/3677842190 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.862+0000 7fcf637fe640 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 0x55581bbc4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.862+0000 7fcf62ffd640 1 -- 172.21.15.31:0/3677842190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 msgr2=0x55581bbc4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.862+0000 7fcf627fc640 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbbdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.862+0000 7fcf62ffd640 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 0x55581bbc4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.864 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.862+0000 7fcf62ffd640 1 -- 172.21.15.31:0/3677842190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 msgr2=0x55581bbbdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.862+0000 7fcf62ffd640 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbbdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.862+0000 7fcf62ffd640 1 -- 172.21.15.31:0/3677842190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55581b8293f0 con 0x55581b7e84d0 2024-04-25T16:11:43.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.862+0000 7fcf627fc640 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbbdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:11:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.863+0000 7fcf62ffd640 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbbd810 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcf64074760 tx=0x7fcf5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d63754c63c77401 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.863+0000 7fcf42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.864+0000 7fcf42ffd640 1 -- 172.21.15.31:0/3677842190 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5c01a050 con 0x55581b7e84d0 2024-04-25T16:11:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.864+0000 7fcf70818740 1 -- 172.21.15.31:0/3677842190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf58014e50 con 0x55581b7e84d0 2024-04-25T16:11:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.865+0000 7fcf42ffd640 1 -- 172.21.15.31:0/3677842190 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcf5c0108a0 con 0x55581b7e84d0 2024-04-25T16:11:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.866+0000 7fcf42ffd640 1 -- 172.21.15.31:0/3677842190 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf5c010aa0 con 0x55581b7e84d0 2024-04-25T16:11:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.866+0000 7fcf70818740 10 client.17667 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:43.868 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.866+0000 7fcf70818740 1 -- 172.21.15.31:0/3677842190 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55581b8f37d0 con 0x55581b7e84d0 2024-04-25T16:11:43.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.867+0000 7fcf42ffd640 1 -- 172.21.15.31:0/3677842190 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf58014e50 con 0x55581b7e84d0 2024-04-25T16:11:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.867+0000 7fcf42ffd640 1 -- 172.21.15.31:0/3677842190 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcf5c00f040 con 0x55581b7e84d0 2024-04-25T16:11:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.867+0000 7fcf70818740 10 client.17667 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.867+0000 7fcf70818740 10 client.17667 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:11:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.867+0000 7fcf70818740 20 client.17667 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.867+0000 7fcf70818740 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55581b7cff60 0x55581bbcbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.867+0000 7fcf70818740 4 client.17667 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.867+0000 7fcf70818740 1 -- 172.21.15.31:0/3677842190 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55581bb25990 con 0x55581b7cff60 2024-04-25T16:11:43.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.869+0000 7fcf627fc640 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55581b7cff60 0x55581bbcbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.871+0000 7fcf627fc640 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55581b7cff60 0x55581bbcbab0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7609e30e2e0dcf04 server_cookie=e86691e32a511523 in_seq=0 out_seq=0 2024-04-25T16:11:43.875 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.873+0000 7fcf42ffd640 10 client.17667 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:11:43.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.873+0000 7fcf42ffd640 1 -- 172.21.15.31:0/3677842190 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55581bb25990 con 0x55581b7cff60 2024-04-25T16:11:43.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.874+0000 7fcf42ffd640 10 client.17667 handle_command_reply: tid=0 2024-04-25T16:11:43.876 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.874+0000 7fcf70818740 1 client.17667 shutdown 2024-04-25T16:11:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.875+0000 7fcf70818740 1 -- 172.21.15.31:0/3677842190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 msgr2=0x55581bbbd810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.875+0000 7fcf70818740 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbbd810 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fcf64074760 tx=0x7fcf5c004730 comp rx=0 tx=0).stop 2024-04-25T16:11:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.876+0000 7fcf70818740 1 -- 172.21.15.31:0/3677842190 shutdown_connections 2024-04-25T16:11:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.876+0000 7fcf70818740 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55581b7cff60 0x55581bbcbab0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.876+0000 7fcf70818740 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 0x55581bbc4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.876+0000 7fcf70818740 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbbdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.876+0000 7fcf70818740 1 --2- 172.21.15.31:0/3677842190 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbbd810 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.878 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.876+0000 7fcf70818740 1 -- 172.21.15.31:0/3677842190 >> 172.21.15.31:0/3677842190 conn(0x55581b8054e0 msgr2=0x55581bbae3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:43.879 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.876+0000 7fcf42ffd640 0 client.17667 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:11:43.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.883+0000 7fcf70818740 1 -- 172.21.15.31:0/3677842190 shutdown_connections 2024-04-25T16:11:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.884+0000 7fcf70818740 1 -- 172.21.15.31:0/3677842190 wait complete. 2024-04-25T16:11:43.890 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.884+0000 7fcf70818740 20 client.17667 trim_cache size 0 max 16384 2024-04-25T16:11:43.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.933+0000 7fcf70818740 1 Processor -- start 2024-04-25T16:11:43.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.934+0000 7fcf70818740 1 -- start start 2024-04-25T16:11:43.937 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.934+0000 7fcf70818740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b7e84d0 0x55581bbbd3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.947 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.945+0000 7fcf70818740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7f06b0 0x55581bbbd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.945+0000 7fcf70818740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.946+0000 7fcf70818740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5000e470 con 0x55581b7f06b0 2024-04-25T16:11:43.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.946+0000 7fcf70818740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5000ffb0 con 0x55581b811390 2024-04-25T16:11:43.948 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.946+0000 7fcf70818740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5000eb40 con 0x55581b7e84d0 2024-04-25T16:11:43.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.947+0000 7fcf637fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.947+0000 7fcf637fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48654/0 (socket says 172.21.15.31:48654) 2024-04-25T16:11:43.950 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.947+0000 7fcf637fe640 1 -- 172.21.15.31:0/4010965716 learned_addr learned my addr 172.21.15.31:0/4010965716 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:43.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.948+0000 7fcf637fe640 1 -- 172.21.15.31:0/4010965716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b7e84d0 msgr2=0x55581bbbd3e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:43.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.948+0000 7fcf637fe640 1 --2- 172.21.15.31:0/4010965716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b7e84d0 0x55581bbbd3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.948+0000 7fcf637fe640 1 -- 172.21.15.31:0/4010965716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7f06b0 msgr2=0x55581bbbd920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:43.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.948+0000 7fcf637fe640 1 --2- 172.21.15.31:0/4010965716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7f06b0 0x55581bbbd920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.948+0000 7fcf637fe640 1 -- 172.21.15.31:0/4010965716 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55581bba0ba0 con 0x55581b811390 2024-04-25T16:11:43.955 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.949+0000 7fcf637fe640 1 --2- 172.21.15.31:0/4010965716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbad9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf540308c0 tx=0x7fcf5402a8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1fbcf116dc2e561 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.950+0000 7fcf497fa640 1 -- 172.21.15.31:0/4010965716 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf540e3070 con 0x55581b811390 2024-04-25T16:11:43.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.950+0000 7fcf497fa640 1 -- 172.21.15.31:0/4010965716 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf54036030 con 0x55581b811390 2024-04-25T16:11:43.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.950+0000 7fcf497fa640 1 -- 172.21.15.31:0/4010965716 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf540d2360 con 0x55581b811390 2024-04-25T16:11:43.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.960+0000 7fcf70818740 1 -- 172.21.15.31:0/4010965716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 msgr2=0x55581bbad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:43.962 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.960+0000 7fcf70818740 1 --2- 172.21.15.31:0/4010965716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbad9c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcf540308c0 tx=0x7fcf5402a8f0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.961+0000 7fcf70818740 1 -- 172.21.15.31:0/4010965716 shutdown_connections 2024-04-25T16:11:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.961+0000 7fcf70818740 1 --2- 172.21.15.31:0/4010965716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b811390 0x55581bbad9c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.961+0000 7fcf70818740 1 --2- 172.21.15.31:0/4010965716 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7f06b0 0x55581bbbd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.961+0000 7fcf70818740 1 --2- 172.21.15.31:0/4010965716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b7e84d0 0x55581bbbd3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.961+0000 7fcf70818740 1 -- 172.21.15.31:0/4010965716 >> 172.21.15.31:0/4010965716 conn(0x55581b8054e0 msgr2=0x55581bbb8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:43.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.961+0000 7fcf70818740 1 -- 172.21.15.31:0/4010965716 shutdown_connections 2024-04-25T16:11:43.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.964+0000 7fcf70818740 1 -- 172.21.15.31:0/4010965716 wait complete. 2024-04-25T16:11:43.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.966+0000 7fcf70818740 1 -- start start 2024-04-25T16:11:43.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.967+0000 7fcf70818740 1 Processor -- start 2024-04-25T16:11:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.969+0000 7fcf70818740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbc1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.969+0000 7fcf70818740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbc1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.969+0000 7fcf70818740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 0x55581bbc2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.969+0000 7fcf70818740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5000f8e0 con 0x55581b7e84d0 2024-04-25T16:11:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.969+0000 7fcf70818740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5000f210 con 0x55581b7f06b0 2024-04-25T16:11:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.969+0000 7fcf70818740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5000c450 con 0x55581b811390 2024-04-25T16:11:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.969+0000 7fcf70818740 10 client.0 _ll_register_callbacks cb 0x7fcf5c0092e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:43.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.970+0000 7fcf62ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbc1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.970+0000 7fcf62ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbc1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39208/0 (socket says 172.21.15.31:39208) 2024-04-25T16:11:43.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.970+0000 7fcf62ffd640 1 -- 172.21.15.31:0/298226743 learned_addr learned my addr 172.21.15.31:0/298226743 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:43.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.970+0000 7fcf627fc640 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbc1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.971+0000 7fcf637fe640 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 0x55581bbc2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:43.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.971+0000 7fcf627fc640 1 -- 172.21.15.31:0/298226743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 msgr2=0x55581bbc2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.971+0000 7fcf627fc640 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 0x55581bbc2bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.971+0000 7fcf627fc640 1 -- 172.21.15.31:0/298226743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 msgr2=0x55581bbc1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:43.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.971+0000 7fcf627fc640 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbc1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:43.974 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.971+0000 7fcf627fc640 1 -- 172.21.15.31:0/298226743 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55581bbc3180 con 0x55581b7f06b0 2024-04-25T16:11:43.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.973+0000 7fcf627fc640 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbc1c90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x55581bbcc250 tx=0x7fcf5800b2c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db21bf4e1ca9e975 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:43.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.974+0000 7fcf42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:43.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.974+0000 7fcf42ffd640 1 -- 172.21.15.31:0/298226743 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf5802c060 con 0x55581b7f06b0 2024-04-25T16:11:43.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.974+0000 7fcf42ffd640 1 -- 172.21.15.31:0/298226743 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fcf580142f0 con 0x55581b7f06b0 2024-04-25T16:11:43.977 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.975+0000 7fcf70818740 1 -- 172.21.15.31:0/298226743 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf54036030 con 0x55581b7f06b0 2024-04-25T16:11:43.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.980+0000 7fcf42ffd640 1 -- 172.21.15.31:0/298226743 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fcf5800b400 con 0x55581b7f06b0 2024-04-25T16:11:43.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.983+0000 7fcf70818740 10 client.27356 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:43.985 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.983+0000 7fcf70818740 1 -- 172.21.15.31:0/298226743 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55581bbc4400 con 0x55581b7f06b0 2024-04-25T16:11:43.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.994+0000 7fcf42ffd640 1 -- 172.21.15.31:0/298226743 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf58014e50 con 0x55581b7f06b0 2024-04-25T16:11:43.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.994+0000 7fcf42ffd640 1 -- 172.21.15.31:0/298226743 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fcf58026070 con 0x55581b7f06b0 2024-04-25T16:11:43.996 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.994+0000 7fcf70818740 10 client.27356 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:43.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.994+0000 7fcf70818740 10 client.27356 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:11:43.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.994+0000 7fcf70818740 20 client.27356 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:43.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.994+0000 7fcf70818740 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55581b7a53a0 0x55581bc36120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:43.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.994+0000 7fcf70818740 4 client.27356 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:43.997 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.994+0000 7fcf70818740 1 -- 172.21.15.31:0/298226743 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fcf54036030 con 0x55581b7a53a0 2024-04-25T16:11:44.000 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:43.998+0000 7fcf62ffd640 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55581b7a53a0 0x55581bc36120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.001+0000 7fcf62ffd640 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55581b7a53a0 0x55581bc36120 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b56c9469015768a3 server_cookie=95c7cdf80a29d6de in_seq=0 out_seq=0 2024-04-25T16:11:44.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.001+0000 7fcf42ffd640 10 client.27356 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:11:44.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.002+0000 7fcf42ffd640 1 -- 172.21.15.31:0/298226743 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fcf54036030 con 0x55581b7a53a0 2024-04-25T16:11:44.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.002+0000 7fcf42ffd640 10 client.27356 handle_command_reply: tid=0 2024-04-25T16:11:44.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.002+0000 7fcf70818740 1 client.27356 shutdown 2024-04-25T16:11:44.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.011+0000 7fcf70818740 1 -- 172.21.15.31:0/298226743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 msgr2=0x55581bbc1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.011+0000 7fcf70818740 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbc1c90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x55581bbcc250 tx=0x7fcf5800b2c0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.011+0000 7fcf70818740 1 -- 172.21.15.31:0/298226743 shutdown_connections 2024-04-25T16:11:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.011+0000 7fcf70818740 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55581b7a53a0 0x55581bc36120 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.011+0000 7fcf70818740 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55581b811390 0x55581bbc2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.011+0000 7fcf70818740 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55581b7f06b0 0x55581bbc1c90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.011+0000 7fcf70818740 1 --2- 172.21.15.31:0/298226743 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55581b7e84d0 0x55581bbc1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.011+0000 7fcf70818740 1 -- 172.21.15.31:0/298226743 >> 172.21.15.31:0/298226743 conn(0x55581b8054e0 msgr2=0x55581bbbd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:44.014 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.011+0000 7fcf42ffd640 0 client.27356 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:11:44.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.016+0000 7fcf70818740 1 -- 172.21.15.31:0/298226743 shutdown_connections 2024-04-25T16:11:44.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.016+0000 7fcf70818740 1 -- 172.21.15.31:0/298226743 wait complete. 2024-04-25T16:11:44.018 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.016+0000 7fcf70818740 20 client.27356 trim_cache size 0 max 16384 2024-04-25T16:11:44.019 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.032+0000 7fcf69892640 1 -- 172.21.15.31:0/2918170894 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcf4c0afc10 msgr2=0x7fcf4c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.032+0000 7fcf69892640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcf4c0afc10 0x7fcf4c0b2120 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcf5c005fd0 tx=0x7fcf5c009450 comp rx=0 tx=0).stop 2024-04-25T16:11:44.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.032+0000 7fcf69892640 1 -- 172.21.15.31:0/2918170894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 msgr2=0x7fcf641b0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.032+0000 7fcf69892640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 0x7fcf641b0270 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fcf54030820 tx=0x7fcf54030850 comp rx=0 tx=0).stop 2024-04-25T16:11:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.033+0000 7fcf69892640 1 -- 172.21.15.31:0/2918170894 shutdown_connections 2024-04-25T16:11:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.033+0000 7fcf69892640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fcf4c0afc10 0x7fcf4c0b2120 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.033+0000 7fcf69892640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6410fef0 0x7fcf641b0270 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.033+0000 7fcf69892640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fcf6410b1d0 0x7fcf641a92e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.033+0000 7fcf69892640 1 --2- 172.21.15.31:0/2918170894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf6410a3e0 0x7fcf641a8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.033+0000 7fcf69892640 1 -- 172.21.15.31:0/2918170894 >> 172.21.15.31:0/2918170894 conn(0x7fcf64070400 msgr2=0x7fcf6410bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.033+0000 7fcf69892640 1 -- 172.21.15.31:0/2918170894 shutdown_connections 2024-04-25T16:11:44.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.033+0000 7fcf69892640 1 -- 172.21.15.31:0/2918170894 wait complete. 2024-04-25T16:11:44.093 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:11:44.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:44 smithi031 ceph-mon[29140]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 908 MiB/s wr, 2.20k op/s 2024-04-25T16:11:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.605+0000 7fdb59c35640 1 -- 172.21.15.31:0/713823883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb5410a350 msgr2=0x7fdb5410ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.605+0000 7fdb59c35640 1 --2- 172.21.15.31:0/713823883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb5410a350 0x7fdb5410ac90 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdb4801a910 tx=0x7fdb4803b470 comp rx=0 tx=0).stop 2024-04-25T16:11:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.605+0000 7fdb59c35640 1 -- 172.21.15.31:0/713823883 shutdown_connections 2024-04-25T16:11:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.605+0000 7fdb59c35640 1 --2- 172.21.15.31:0/713823883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5410fef0 0x7fdb54110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.605+0000 7fdb59c35640 1 --2- 172.21.15.31:0/713823883 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb5410b1d0 0x7fdb5410b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.605+0000 7fdb59c35640 1 --2- 172.21.15.31:0/713823883 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb5410a350 0x7fdb5410ac90 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.605+0000 7fdb59c35640 1 -- 172.21.15.31:0/713823883 >> 172.21.15.31:0/713823883 conn(0x7fdb54070400 msgr2=0x7fdb54072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:44.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.610+0000 7fdb59c35640 1 -- 172.21.15.31:0/713823883 shutdown_connections 2024-04-25T16:11:44.613 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.610+0000 7fdb59c35640 1 -- 172.21.15.31:0/713823883 wait complete. 2024-04-25T16:11:44.618 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.611+0000 7fdb59c35640 1 Processor -- start 2024-04-25T16:11:44.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:44 smithi063 ceph-mon[32413]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 908 MiB/s wr, 2.20k op/s 2024-04-25T16:11:44.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.619+0000 7fdb59c35640 1 -- start start 2024-04-25T16:11:44.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.619+0000 7fdb59c35640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb5410a350 0x7fdb541a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.619+0000 7fdb59c35640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5410b1d0 0x7fdb541a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.621 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.619+0000 7fdb59c35640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb5410fef0 0x7fdb541a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.619+0000 7fdb59c35640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdb54113b00 con 0x7fdb5410a350 2024-04-25T16:11:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.619+0000 7fdb59c35640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdb54113980 con 0x7fdb5410fef0 2024-04-25T16:11:44.622 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.619+0000 7fdb59c35640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdb54113c80 con 0x7fdb5410b1d0 2024-04-25T16:11:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.620+0000 7fdb59434640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb5410fef0 0x7fdb541a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.620+0000 7fdb59434640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb5410fef0 0x7fdb541a7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48670/0 (socket says 172.21.15.31:48670) 2024-04-25T16:11:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.620+0000 7fdb59434640 1 -- 172.21.15.31:0/701578816 learned_addr learned my addr 172.21.15.31:0/701578816 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.620+0000 7fdb59434640 1 -- 172.21.15.31:0/701578816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5410b1d0 msgr2=0x7fdb541a0bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:11:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.620+0000 7fdb59434640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5410b1d0 0x7fdb541a0bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.620+0000 7fdb59434640 1 -- 172.21.15.31:0/701578816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb5410a350 msgr2=0x7fdb541a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.620+0000 7fdb59434640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb5410a350 0x7fdb541a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.621+0000 7fdb59434640 1 -- 172.21.15.31:0/701578816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb541a81f0 con 0x7fdb5410fef0 2024-04-25T16:11:44.626 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.621+0000 7fdb59434640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb5410fef0 0x7fdb541a7b80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400d720 tx=0x7fdb4400dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:44.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.626+0000 7fdb51ffb640 1 -- 172.21.15.31:0/701578816 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44017070 con 0x7fdb5410fef0 2024-04-25T16:11:44.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.626+0000 7fdb51ffb640 1 -- 172.21.15.31:0/701578816 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb4400de30 con 0x7fdb5410fef0 2024-04-25T16:11:44.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.626+0000 7fdb51ffb640 1 -- 172.21.15.31:0/701578816 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44004c30 con 0x7fdb5410fef0 2024-04-25T16:11:44.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.627+0000 7fdb59c35640 1 -- 172.21.15.31:0/701578816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb541a84e0 con 0x7fdb5410fef0 2024-04-25T16:11:44.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.627+0000 7fdb59c35640 1 -- 172.21.15.31:0/701578816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb541a8a60 con 0x7fdb5410fef0 2024-04-25T16:11:44.636 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.634+0000 7fdb51ffb640 1 -- 172.21.15.31:0/701578816 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdb44009bd0 con 0x7fdb5410fef0 2024-04-25T16:11:44.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.639+0000 7fdb51ffb640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdb3c0afd00 0x7fdb3c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.639+0000 7fdb58c33640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdb3c0afd00 0x7fdb3c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.640+0000 7fdb51ffb640 1 -- 172.21.15.31:0/701578816 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdb440de540 con 0x7fdb5410fef0 2024-04-25T16:11:44.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:44 smithi186 ceph-mon[33258]: pgmap v664: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 908 MiB/s wr, 2.20k op/s 2024-04-25T16:11:44.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.656+0000 7fdb58c33640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdb3c0afd00 0x7fdb3c0b2210 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdb4801a6d0 tx=0x7fdb48000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:44.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.657+0000 7fdb61e21740 1 Processor -- start 2024-04-25T16:11:44.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.657+0000 7fdb61e21740 1 -- start start 2024-04-25T16:11:44.660 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.658+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f022c390 0x5586f05cea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.658+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f02034d0 0x5586f05cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.658+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f020b6b0 0x5586f05cfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.658+0000 7fdb61e21740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5586f0532d80 con 0x5586f020b6b0 2024-04-25T16:11:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.658+0000 7fdb61e21740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5586f04eca20 con 0x5586f022c390 2024-04-25T16:11:44.661 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.658+0000 7fdb61e21740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5586f052ed40 con 0x5586f02034d0 2024-04-25T16:11:44.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.660+0000 7fdb59434640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f020b6b0 0x5586f05cfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.660+0000 7fdb59434640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f020b6b0 0x5586f05cfc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39248/0 (socket says 172.21.15.31:39248) 2024-04-25T16:11:44.662 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.660+0000 7fdb59434640 1 -- 172.21.15.31:0/655728172 learned_addr learned my addr 172.21.15.31:0/655728172 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:44.666 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.664+0000 7fdb58c33640 1 --2- 172.21.15.31:0/655728172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f022c390 0x5586f05cea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.665+0000 7fdb59434640 1 -- 172.21.15.31:0/655728172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f02034d0 msgr2=0x5586f05cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.665+0000 7fdb59434640 1 --2- 172.21.15.31:0/655728172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f02034d0 0x5586f05cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.665+0000 7fdb59434640 1 -- 172.21.15.31:0/655728172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f022c390 msgr2=0x5586f05cea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:44.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.665+0000 7fdb59434640 1 --2- 172.21.15.31:0/655728172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f022c390 0x5586f05cea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.665+0000 7fdb59434640 1 -- 172.21.15.31:0/655728172 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f04f7c30 con 0x5586f020b6b0 2024-04-25T16:11:44.673 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.671+0000 7fdb59434640 1 --2- 172.21.15.31:0/655728172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f020b6b0 0x5586f05cfc10 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdb440a2a50 tx=0x7fdb440a6450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53eb9c6a532712f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:44.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.673+0000 7fdb3affd640 1 -- 172.21.15.31:0/655728172 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44020040 con 0x5586f020b6b0 2024-04-25T16:11:44.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.673+0000 7fdb3affd640 1 -- 172.21.15.31:0/655728172 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb440aad80 con 0x5586f020b6b0 2024-04-25T16:11:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.675+0000 7fdb61e21740 1 -- 172.21.15.31:0/655728172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f020b6b0 msgr2=0x5586f05cfc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.675+0000 7fdb61e21740 1 --2- 172.21.15.31:0/655728172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f020b6b0 0x5586f05cfc10 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fdb440a2a50 tx=0x7fdb440a6450 comp rx=0 tx=0).stop 2024-04-25T16:11:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.675+0000 7fdb61e21740 1 -- 172.21.15.31:0/655728172 shutdown_connections 2024-04-25T16:11:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.675+0000 7fdb61e21740 1 --2- 172.21.15.31:0/655728172 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f020b6b0 0x5586f05cfc10 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.677 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.675+0000 7fdb61e21740 1 --2- 172.21.15.31:0/655728172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f02034d0 0x5586f05cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.675+0000 7fdb61e21740 1 --2- 172.21.15.31:0/655728172 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f022c390 0x5586f05cea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.678 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.675+0000 7fdb61e21740 1 -- 172.21.15.31:0/655728172 >> 172.21.15.31:0/655728172 conn(0x5586f02204e0 msgr2=0x5586f05ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:44.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.677+0000 7fdb61e21740 1 -- 172.21.15.31:0/655728172 shutdown_connections 2024-04-25T16:11:44.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.677+0000 7fdb61e21740 1 -- 172.21.15.31:0/655728172 wait complete. 2024-04-25T16:11:44.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.677+0000 7fdb61e21740 1 -- start start 2024-04-25T16:11:44.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.678+0000 7fdb61e21740 1 Processor -- start 2024-04-25T16:11:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.678+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f02034d0 0x5586f05d85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.678+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01eaf60 0x5586f05d8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.678+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01c03a0 0x5586f05dfa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.679+0000 7fdb61e21740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5586f05d0710 con 0x5586f02034d0 2024-04-25T16:11:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.679+0000 7fdb61e21740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5586f04b9dc0 con 0x5586f01c03a0 2024-04-25T16:11:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.679+0000 7fdb61e21740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5586f05d0890 con 0x5586f01eaf60 2024-04-25T16:11:44.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.679+0000 7fdb61e21740 10 client.0 _ll_register_callbacks cb 0x5586f04d5600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:44.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.680+0000 7fdb59434640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01c03a0 0x5586f05dfa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.683+0000 7fdb59434640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01c03a0 0x5586f05dfa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48692/0 (socket says 172.21.15.31:48692) 2024-04-25T16:11:44.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.683+0000 7fdb59434640 1 -- 172.21.15.31:0/354784746 learned_addr learned my addr 172.21.15.31:0/354784746 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:44.686 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.683+0000 7fdb53fff640 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01eaf60 0x5586f05d8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.688+0000 7fdb59434640 1 -- 172.21.15.31:0/354784746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01eaf60 msgr2=0x5586f05d8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.688+0000 7fdb59434640 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01eaf60 0x5586f05d8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.688+0000 7fdb59434640 1 -- 172.21.15.31:0/354784746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f02034d0 msgr2=0x5586f05d85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:44.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.688+0000 7fdb59434640 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f02034d0 0x5586f05d85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.688+0000 7fdb59434640 1 -- 172.21.15.31:0/354784746 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586f02443f0 con 0x5586f01c03a0 2024-04-25T16:11:44.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.689+0000 7fdb59434640 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01c03a0 0x5586f05dfa70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009610 tx=0x7fdb440a67a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a54bfb4e2992454a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:44.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.689+0000 7fdb33fff640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.691+0000 7fdb33fff640 1 -- 172.21.15.31:0/354784746 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb44020040 con 0x5586f01c03a0 2024-04-25T16:11:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.691+0000 7fdb61e21740 1 -- 172.21.15.31:0/354784746 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5586f0540990 con 0x5586f01c03a0 2024-04-25T16:11:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.692+0000 7fdb33fff640 1 -- 172.21.15.31:0/354784746 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdb440a5c30 con 0x5586f01c03a0 2024-04-25T16:11:44.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.692+0000 7fdb33fff640 1 -- 172.21.15.31:0/354784746 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdb440aad80 con 0x5586f01c03a0 2024-04-25T16:11:44.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.697+0000 7fdb61e21740 10 client.27365 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:44.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.697+0000 7fdb61e21740 1 -- 172.21.15.31:0/354784746 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5586f030e7d0 con 0x5586f01c03a0 2024-04-25T16:11:44.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.698+0000 7fdb33fff640 1 -- 172.21.15.31:0/354784746 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5586f0540990 con 0x5586f01c03a0 2024-04-25T16:11:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.700+0000 7fdb33fff640 1 -- 172.21.15.31:0/354784746 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fdb440a5820 con 0x5586f01c03a0 2024-04-25T16:11:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.701+0000 7fdb61e21740 10 client.27365 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.701+0000 7fdb61e21740 10 client.27365 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:11:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.701+0000 7fdb61e21740 20 client.27365 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.701+0000 7fdb61e21740 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5586f05e4450 0x5586f05e6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.701+0000 7fdb61e21740 4 client.27365 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:44.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.701+0000 7fdb61e21740 1 -- 172.21.15.31:0/354784746 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5586f053a800 con 0x5586f05e4450 2024-04-25T16:11:44.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.703+0000 7fdb58c33640 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5586f05e4450 0x5586f05e6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.706+0000 7fdb58c33640 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5586f05e4450 0x5586f05e6940 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=650d7e609d93994b server_cookie=d12f05bf878dda8 in_seq=0 out_seq=0 2024-04-25T16:11:44.708 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.706+0000 7fdb33fff640 10 client.27365 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:11:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.707+0000 7fdb33fff640 1 -- 172.21.15.31:0/354784746 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5586f053a800 con 0x5586f05e4450 2024-04-25T16:11:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.707+0000 7fdb33fff640 10 client.27365 handle_command_reply: tid=0 2024-04-25T16:11:44.709 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.707+0000 7fdb61e21740 1 client.27365 shutdown 2024-04-25T16:11:44.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.708+0000 7fdb61e21740 1 -- 172.21.15.31:0/354784746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01c03a0 msgr2=0x5586f05dfa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.708+0000 7fdb61e21740 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01c03a0 0x5586f05dfa70 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fdb44009610 tx=0x7fdb440a67a0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.708+0000 7fdb61e21740 1 -- 172.21.15.31:0/354784746 shutdown_connections 2024-04-25T16:11:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.708+0000 7fdb61e21740 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5586f05e4450 0x5586f05e6940 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.708+0000 7fdb61e21740 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01c03a0 0x5586f05dfa70 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.708+0000 7fdb61e21740 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01eaf60 0x5586f05d8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.708+0000 7fdb61e21740 1 --2- 172.21.15.31:0/354784746 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f02034d0 0x5586f05d85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.708+0000 7fdb61e21740 1 -- 172.21.15.31:0/354784746 >> 172.21.15.31:0/354784746 conn(0x5586f02204e0 msgr2=0x5586f05c93f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:44.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.708+0000 7fdb33fff640 0 client.27365 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:11:44.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.712+0000 7fdb61e21740 1 -- 172.21.15.31:0/354784746 shutdown_connections 2024-04-25T16:11:44.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.718+0000 7fdb61e21740 1 -- 172.21.15.31:0/354784746 wait complete. 2024-04-25T16:11:44.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.718+0000 7fdb61e21740 20 client.27365 trim_cache size 0 max 16384 2024-04-25T16:11:44.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.752+0000 7fdb61e21740 1 Processor -- start 2024-04-25T16:11:44.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.753+0000 7fdb61e21740 1 -- start start 2024-04-25T16:11:44.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.753+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01c03a0 0x5586f05e6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.754+0000 7fdb58c33640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01c03a0 0x5586f05e6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.754+0000 7fdb58c33640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01c03a0 0x5586f05e6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39268/0 (socket says 172.21.15.31:39268) 2024-04-25T16:11:44.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.754+0000 7fdb58c33640 1 -- 172.21.15.31:0/1193158461 learned_addr learned my addr 172.21.15.31:0/1193158461 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:44.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.754+0000 7fdb61e21740 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01eaf60 0x5586f05df990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.754+0000 7fdb61e21740 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f02034d0 0x5586f05e69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.755+0000 7fdb61e21740 1 -- 172.21.15.31:0/1193158461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4000e360 con 0x5586f01c03a0 2024-04-25T16:11:44.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.755+0000 7fdb61e21740 1 -- 172.21.15.31:0/1193158461 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4000fea0 con 0x5586f01eaf60 2024-04-25T16:11:44.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.755+0000 7fdb61e21740 1 -- 172.21.15.31:0/1193158461 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4000ea30 con 0x5586f02034d0 2024-04-25T16:11:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.757+0000 7fdb59434640 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f02034d0 0x5586f05e69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.757+0000 7fdb58c33640 1 -- 172.21.15.31:0/1193158461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f02034d0 msgr2=0x5586f05e69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.757+0000 7fdb58c33640 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f02034d0 0x5586f05e69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.757+0000 7fdb58c33640 1 -- 172.21.15.31:0/1193158461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01eaf60 msgr2=0x5586f05df990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.757+0000 7fdb58c33640 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01eaf60 0x5586f05df990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.757+0000 7fdb58c33640 1 -- 172.21.15.31:0/1193158461 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5586f05dfed0 con 0x5586f01c03a0 2024-04-25T16:11:44.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.763+0000 7fdb58c33640 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01c03a0 0x5586f05e6460 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdb4801a910 tx=0x7fdb4804a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09d6851d9e5842e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:44.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.769+0000 7fdb3a7fc640 1 -- 172.21.15.31:0/1193158461 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb480041d0 con 0x5586f01c03a0 2024-04-25T16:11:44.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.769+0000 7fdb3a7fc640 1 -- 172.21.15.31:0/1193158461 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb48065070 con 0x5586f01c03a0 2024-04-25T16:11:44.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.769+0000 7fdb3a7fc640 1 -- 172.21.15.31:0/1193158461 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb480055c0 con 0x5586f01c03a0 2024-04-25T16:11:44.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.770+0000 7fdb61e21740 1 -- 172.21.15.31:0/1193158461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01c03a0 msgr2=0x5586f05e6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.770+0000 7fdb61e21740 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01c03a0 0x5586f05e6460 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fdb4801a910 tx=0x7fdb4804a000 comp rx=0 tx=0).stop 2024-04-25T16:11:44.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.774+0000 7fdb61e21740 1 -- 172.21.15.31:0/1193158461 shutdown_connections 2024-04-25T16:11:44.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.774+0000 7fdb61e21740 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f02034d0 0x5586f05e69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.774+0000 7fdb61e21740 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f01eaf60 0x5586f05df990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.774+0000 7fdb61e21740 1 --2- 172.21.15.31:0/1193158461 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01c03a0 0x5586f05e6460 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.774+0000 7fdb61e21740 1 -- 172.21.15.31:0/1193158461 >> 172.21.15.31:0/1193158461 conn(0x5586f02204e0 msgr2=0x5586f05d3280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:44.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.778+0000 7fdb61e21740 1 -- 172.21.15.31:0/1193158461 shutdown_connections 2024-04-25T16:11:44.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.782+0000 7fdb61e21740 1 -- 172.21.15.31:0/1193158461 wait complete. 2024-04-25T16:11:44.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.783+0000 7fdb61e21740 1 -- start start 2024-04-25T16:11:44.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.783+0000 7fdb61e21740 1 Processor -- start 2024-04-25T16:11:44.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.787+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01c03a0 0x5586f05df490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.787+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01eaf60 0x5586f05dc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.787+0000 7fdb61e21740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f02034d0 0x5586f05dc5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.787+0000 7fdb61e21740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4000f7d0 con 0x5586f01eaf60 2024-04-25T16:11:44.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.787+0000 7fdb61e21740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4000f100 con 0x5586f02034d0 2024-04-25T16:11:44.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.787+0000 7fdb61e21740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdb4000c3f0 con 0x5586f01c03a0 2024-04-25T16:11:44.790 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.787+0000 7fdb61e21740 10 client.0 _ll_register_callbacks cb 0x7fdb440a61d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.788+0000 7fdb58c33640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01c03a0 0x5586f05df490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.788+0000 7fdb58c33640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01c03a0 0x5586f05df490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:42094/0 (socket says 172.21.15.31:42094) 2024-04-25T16:11:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.788+0000 7fdb58c33640 1 -- 172.21.15.31:0/186442436 learned_addr learned my addr 172.21.15.31:0/186442436 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.788+0000 7fdb53fff640 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01eaf60 0x5586f05dc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.789+0000 7fdb53fff640 1 -- 172.21.15.31:0/186442436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01c03a0 msgr2=0x5586f05df490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.789+0000 7fdb53fff640 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01c03a0 0x5586f05df490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.789+0000 7fdb53fff640 1 -- 172.21.15.31:0/186442436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f02034d0 msgr2=0x5586f05dc5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.789+0000 7fdb53fff640 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f02034d0 0x5586f05dc5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.791 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.789+0000 7fdb53fff640 1 -- 172.21.15.31:0/186442436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5586f05dfb30 con 0x5586f01eaf60 2024-04-25T16:11:44.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.789+0000 7fdb53fff640 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01eaf60 0x5586f05dc080 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x5586f05d0260 tx=0x7fdb4c00bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3aa1158d609862d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:44.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.791+0000 7fdb33fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:44.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.800+0000 7fdb33fff640 1 -- 172.21.15.31:0/186442436 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00ab10 con 0x5586f01eaf60 2024-04-25T16:11:44.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.800+0000 7fdb33fff640 1 -- 172.21.15.31:0/186442436 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdb4c00acb0 con 0x5586f01eaf60 2024-04-25T16:11:44.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.800+0000 7fdb61e21740 1 -- 172.21.15.31:0/186442436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdb48065070 con 0x5586f01eaf60 2024-04-25T16:11:44.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.802+0000 7fdb33fff640 1 -- 172.21.15.31:0/186442436 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdb4c014420 con 0x5586f01eaf60 2024-04-25T16:11:44.808 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.806+0000 7fdb61e21740 10 client.17682 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:44.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.806+0000 7fdb61e21740 1 -- 172.21.15.31:0/186442436 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5586f064dbc0 con 0x5586f01eaf60 2024-04-25T16:11:44.811 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.808+0000 7fdb33fff640 1 -- 172.21.15.31:0/186442436 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdb48065070 con 0x5586f01eaf60 2024-04-25T16:11:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.809+0000 7fdb33fff640 1 -- 172.21.15.31:0/186442436 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fdb4c0135e0 con 0x5586f01eaf60 2024-04-25T16:11:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.812+0000 7fdb61e21740 10 client.17682 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.812+0000 7fdb61e21740 10 client.17682 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:11:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.812+0000 7fdb61e21740 20 client.17682 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.812+0000 7fdb61e21740 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5586f064e540 0x5586f0650960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.812+0000 7fdb61e21740 4 client.17682 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:44.814 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.812+0000 7fdb61e21740 1 -- 172.21.15.31:0/186442436 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5586f04b5610 con 0x5586f064e540 2024-04-25T16:11:44.816 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.814+0000 7fdb58c33640 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5586f064e540 0x5586f0650960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:44.820 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.818+0000 7fdb58c33640 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5586f064e540 0x5586f0650960 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e24aa14c1a203e39 server_cookie=5a8c5b667f59429b in_seq=0 out_seq=0 2024-04-25T16:11:44.821 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.819+0000 7fdb33fff640 10 client.17682 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:11:44.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.820+0000 7fdb33fff640 1 -- 172.21.15.31:0/186442436 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5586f04b5610 con 0x5586f064e540 2024-04-25T16:11:44.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.820+0000 7fdb33fff640 10 client.17682 handle_command_reply: tid=0 2024-04-25T16:11:44.822 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.820+0000 7fdb61e21740 1 client.17682 shutdown 2024-04-25T16:11:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.827+0000 7fdb61e21740 1 -- 172.21.15.31:0/186442436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01eaf60 msgr2=0x5586f05dc080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.827+0000 7fdb61e21740 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01eaf60 0x5586f05dc080 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x5586f05d0260 tx=0x7fdb4c00bb60 comp rx=0 tx=0).stop 2024-04-25T16:11:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.827+0000 7fdb61e21740 1 -- 172.21.15.31:0/186442436 shutdown_connections 2024-04-25T16:11:44.829 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.827+0000 7fdb61e21740 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x5586f064e540 0x5586f0650960 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.827+0000 7fdb61e21740 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5586f02034d0 0x5586f05dc5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.827+0000 7fdb61e21740 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5586f01eaf60 0x5586f05dc080 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.827+0000 7fdb61e21740 1 --2- 172.21.15.31:0/186442436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5586f01c03a0 0x5586f05df490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.827+0000 7fdb61e21740 1 -- 172.21.15.31:0/186442436 >> 172.21.15.31:0/186442436 conn(0x5586f02204e0 msgr2=0x5586f05e6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:44.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.827+0000 7fdb33fff640 0 client.17682 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:11:44.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.828+0000 7fdb61e21740 1 -- 172.21.15.31:0/186442436 shutdown_connections 2024-04-25T16:11:44.831 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.828+0000 7fdb61e21740 1 -- 172.21.15.31:0/186442436 wait complete. 2024-04-25T16:11:44.834 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.830+0000 7fdb61e21740 20 client.17682 trim_cache size 0 max 16384 2024-04-25T16:11:44.836 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 -- 172.21.15.31:0/701578816 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdb3c0afd00 msgr2=0x7fdb3c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdb3c0afd00 0x7fdb3c0b2210 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fdb4801a6d0 tx=0x7fdb48000bc0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 -- 172.21.15.31:0/701578816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb5410fef0 msgr2=0x7fdb541a7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb5410fef0 0x7fdb541a7b80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fdb4400d720 tx=0x7fdb4400dbe0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 -- 172.21.15.31:0/701578816 shutdown_connections 2024-04-25T16:11:44.838 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdb3c0afd00 0x7fdb3c0b2210 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdb5410fef0 0x7fdb541a7b80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5410b1d0 0x7fdb541a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 --2- 172.21.15.31:0/701578816 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdb5410a350 0x7fdb541a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 -- 172.21.15.31:0/701578816 >> 172.21.15.31:0/701578816 conn(0x7fdb54070400 msgr2=0x7fdb5410dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 -- 172.21.15.31:0/701578816 shutdown_connections 2024-04-25T16:11:44.839 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:44.834+0000 7fdb59c35640 1 -- 172.21.15.31:0/701578816 wait complete. 2024-04-25T16:11:44.872 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:11:45.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.191+0000 7ffa1cc22640 1 -- 172.21.15.31:0/561846901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa18108740 msgr2=0x7ffa1810ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:45.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.191+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/561846901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa18108740 0x7ffa1810ab60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c009900 tx=0x7ffa0c02a420 comp rx=0 tx=0).stop 2024-04-25T16:11:45.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.195+0000 7ffa1cc22640 1 -- 172.21.15.31:0/561846901 shutdown_connections 2024-04-25T16:11:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.195+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/561846901 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa18108740 0x7ffa1810ab60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.195+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/561846901 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa18103a40 0x7ffa18103e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.195+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/561846901 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa18102800 0x7ffa18103500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.195+0000 7ffa1cc22640 1 -- 172.21.15.31:0/561846901 >> 172.21.15.31:0/561846901 conn(0x7ffa180fd9f0 msgr2=0x7ffa180ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:45.199 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.196+0000 7ffa1cc22640 1 -- 172.21.15.31:0/561846901 shutdown_connections 2024-04-25T16:11:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.198+0000 7ffa1cc22640 1 -- 172.21.15.31:0/561846901 wait complete. 2024-04-25T16:11:45.201 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.199+0000 7ffa1cc22640 1 Processor -- start 2024-04-25T16:11:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.201+0000 7ffa1cc22640 1 -- start start 2024-04-25T16:11:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.201+0000 7ffa1cc22640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa18102800 0x7ffa18072150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.201+0000 7ffa1cc22640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa18103a40 0x7ffa18072690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.201+0000 7ffa1cc22640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa18108740 0x7ffa180735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.201+0000 7ffa1cc22640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1806ce20 con 0x7ffa18102800 2024-04-25T16:11:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.201+0000 7ffa1cc22640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1806cca0 con 0x7ffa18103a40 2024-04-25T16:11:45.206 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.201+0000 7ffa1cc22640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1806cfa0 con 0x7ffa18108740 2024-04-25T16:11:45.207 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.205+0000 7ffa177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa18102800 0x7ffa18072150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.205+0000 7ffa177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa18102800 0x7ffa18072150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39298/0 (socket says 172.21.15.31:39298) 2024-04-25T16:11:45.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.205+0000 7ffa177fe640 1 -- 172.21.15.31:0/163018555 learned_addr learned my addr 172.21.15.31:0/163018555 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:45.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.207+0000 7ffa177fe640 1 -- 172.21.15.31:0/163018555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa18108740 msgr2=0x7ffa180735f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:11:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.207+0000 7ffa177fe640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa18108740 0x7ffa180735f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.207+0000 7ffa177fe640 1 -- 172.21.15.31:0/163018555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa18103a40 msgr2=0x7ffa18072690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:11:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.207+0000 7ffa177fe640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa18103a40 0x7ffa18072690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.207+0000 7ffa177fe640 1 -- 172.21.15.31:0/163018555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa18073c60 con 0x7ffa18102800 2024-04-25T16:11:45.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.209+0000 7ffa177fe640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa18102800 0x7ffa18072150 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800b9b0 tx=0x7ffa0800be70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:45.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.212+0000 7ffa14ff9640 1 -- 172.21.15.31:0/163018555 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08019070 con 0x7ffa18102800 2024-04-25T16:11:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.212+0000 7ffa1cc22640 1 -- 172.21.15.31:0/163018555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa18073f50 con 0x7ffa18102800 2024-04-25T16:11:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.212+0000 7ffa1cc22640 1 -- 172.21.15.31:0/163018555 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa180744a0 con 0x7ffa18102800 2024-04-25T16:11:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.214+0000 7ffa14ff9640 1 -- 172.21.15.31:0/163018555 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa0800cba0 con 0x7ffa18102800 2024-04-25T16:11:45.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.214+0000 7ffa14ff9640 1 -- 172.21.15.31:0/163018555 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa080125a0 con 0x7ffa18102800 2024-04-25T16:11:45.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.219+0000 7ffa24e08740 1 Processor -- start 2024-04-25T16:11:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.220+0000 7ffa24e08740 1 -- start start 2024-04-25T16:11:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.220+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 0x5596e4e7b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.220+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab04d0 0x5596e4e7bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.220+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e7cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.220+0000 7ffa24e08740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5596e4ddfd80 con 0x5596e4ab86b0 2024-04-25T16:11:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.220+0000 7ffa24e08740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5596e4d99a20 con 0x5596e4ad9390 2024-04-25T16:11:45.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.220+0000 7ffa24e08740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5596e4ddbd40 con 0x5596e4ab04d0 2024-04-25T16:11:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.224+0000 7ffa17fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 0x5596e4e7b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.224+0000 7ffa14ff9640 1 -- 172.21.15.31:0/163018555 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ffa0801c070 con 0x7ffa18102800 2024-04-25T16:11:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.225+0000 7ffa16ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e7cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.225+0000 7ffa14ff9640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ffa000afcf0 0x7ffa000b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.225+0000 7ffa16ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e7cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39312/0 (socket says 172.21.15.31:39312) 2024-04-25T16:11:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.225+0000 7ffa16ffd640 1 -- 172.21.15.31:0/3260982301 learned_addr learned my addr 172.21.15.31:0/3260982301 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.226+0000 7ffa14ff9640 1 -- 172.21.15.31:0/163018555 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ffa080de100 con 0x7ffa18102800 2024-04-25T16:11:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.226+0000 7ffa16ffd640 1 -- 172.21.15.31:0/3260982301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab04d0 msgr2=0x5596e4e7bf60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.226+0000 7ffa16ffd640 1 --2- 172.21.15.31:0/3260982301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab04d0 0x5596e4e7bf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.232 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.226+0000 7ffa16ffd640 1 -- 172.21.15.31:0/3260982301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 msgr2=0x5596e4e7b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.226+0000 7ffa16ffd640 1 --2- 172.21.15.31:0/3260982301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 0x5596e4e7b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.226+0000 7ffa16ffd640 1 -- 172.21.15.31:0/3260982301 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596e4da4c30 con 0x5596e4ab86b0 2024-04-25T16:11:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.226+0000 7ffa16ffd640 1 --2- 172.21.15.31:0/3260982301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e7cbf0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ffa10009770 tx=0x7ffa10009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfdc80e30bb52261 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.227+0000 7ff9fdffb640 1 -- 172.21.15.31:0/3260982301 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa10015020 con 0x5596e4ab86b0 2024-04-25T16:11:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.228+0000 7ff9fdffb640 1 -- 172.21.15.31:0/3260982301 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa10011420 con 0x5596e4ab86b0 2024-04-25T16:11:45.233 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.228+0000 7ff9fdffb640 1 -- 172.21.15.31:0/3260982301 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa100105c0 con 0x5596e4ab86b0 2024-04-25T16:11:45.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.229+0000 7ffa24e08740 1 -- 172.21.15.31:0/3260982301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 msgr2=0x5596e4e7cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:45.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.229+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3260982301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e7cbf0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7ffa10009770 tx=0x7ffa10009c30 comp rx=0 tx=0).stop 2024-04-25T16:11:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.233+0000 7ffa24e08740 1 -- 172.21.15.31:0/3260982301 shutdown_connections 2024-04-25T16:11:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.233+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3260982301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e7cbf0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.233+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3260982301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab04d0 0x5596e4e7bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.233+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3260982301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 0x5596e4e7b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.233+0000 7ffa24e08740 1 -- 172.21.15.31:0/3260982301 >> 172.21.15.31:0/3260982301 conn(0x5596e4acd4e0 msgr2=0x5596e4e771f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:45.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.233+0000 7ffa24e08740 1 -- 172.21.15.31:0/3260982301 shutdown_connections 2024-04-25T16:11:45.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.239+0000 7ffa17fff640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ffa000afcf0 0x7ffa000b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.240+0000 7ffa24e08740 1 -- 172.21.15.31:0/3260982301 wait complete. 2024-04-25T16:11:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.241+0000 7ffa24e08740 1 -- start start 2024-04-25T16:11:45.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.241+0000 7ffa24e08740 1 Processor -- start 2024-04-25T16:11:45.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.242+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab04d0 0x5596e4e856f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.242+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab86b0 0x5596e4e85c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.242+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.242+0000 7ffa24e08740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5596e4e7d7a0 con 0x5596e4ab04d0 2024-04-25T16:11:45.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.242+0000 7ffa24e08740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5596e4d66dc0 con 0x5596e4ad9390 2024-04-25T16:11:45.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.242+0000 7ffa24e08740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5596e4e7d920 con 0x5596e4ab86b0 2024-04-25T16:11:45.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.242+0000 7ffa24e08740 10 client.0 _ll_register_callbacks cb 0x5596e4d82600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:45.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.246+0000 7ffa177fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab04d0 0x5596e4e856f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.249+0000 7ffa16ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab86b0 0x5596e4e85c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.250+0000 7ffa16ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab86b0 0x5596e4e85c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:42142/0 (socket says 172.21.15.31:42142) 2024-04-25T16:11:45.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.250+0000 7ffa16ffd640 1 -- 172.21.15.31:0/1997521799 learned_addr learned my addr 172.21.15.31:0/1997521799 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:45.255 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.252+0000 7ffa177fe640 1 -- 172.21.15.31:0/1997521799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab86b0 msgr2=0x5596e4e85c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:45.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.252+0000 7ffa177fe640 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab86b0 0x5596e4e85c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.252+0000 7ffa177fe640 1 -- 172.21.15.31:0/1997521799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 msgr2=0x5596e4e8cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.251+0000 7ffa17fff640 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.254+0000 7ffa177fe640 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.254+0000 7ffa177fe640 1 -- 172.21.15.31:0/1997521799 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596e4af13f0 con 0x5596e4ab04d0 2024-04-25T16:11:45.259 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.255+0000 7ffa17fff640 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:11:45.260 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.257+0000 7ffa177fe640 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab04d0 0x5596e4e856f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x5596e4e7b700 tx=0x7ffa080a60a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2142dcf8d467aebf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:45.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.257+0000 7ff9f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:45.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.257+0000 7ffa17fff640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ffa000afcf0 0x7ffa000b2200 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c0096c0 tx=0x7ffa0c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:45.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.258+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/1997521799 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa080aa940 con 0x5596e4ab04d0 2024-04-25T16:11:45.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.258+0000 7ffa24e08740 1 -- 172.21.15.31:0/1997521799 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa10011420 con 0x5596e4ab04d0 2024-04-25T16:11:45.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.259+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/1997521799 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ffa080af170 con 0x5596e4ab04d0 2024-04-25T16:11:45.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.259+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/1997521799 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffa080af3f0 con 0x5596e4ab04d0 2024-04-25T16:11:45.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.260+0000 7ffa24e08740 10 client.17691 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:45.264 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.260+0000 7ffa24e08740 1 -- 172.21.15.31:0/1997521799 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596e4bbb7d0 con 0x5596e4ab04d0 2024-04-25T16:11:45.265 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.261+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/1997521799 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa10011420 con 0x5596e4ab04d0 2024-04-25T16:11:45.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.265+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/1997521799 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ffa080a2100 con 0x5596e4ab04d0 2024-04-25T16:11:45.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.265+0000 7ffa24e08740 10 client.17691 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:45.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.265+0000 7ffa24e08740 10 client.17691 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:11:45.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.265+0000 7ffa24e08740 20 client.17691 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:45.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.265+0000 7ffa24e08740 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5596e4a97f60 0x5596e4e93910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.265+0000 7ffa24e08740 4 client.17691 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:45.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.265+0000 7ffa24e08740 1 -- 172.21.15.31:0/1997521799 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596e4ded990 con 0x5596e4a97f60 2024-04-25T16:11:45.272 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.269+0000 7ffa16ffd640 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5596e4a97f60 0x5596e4e93910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.297+0000 7ffa16ffd640 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5596e4a97f60 0x5596e4e93910 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=810641ce7f7e0115 server_cookie=5456cd734c3adaff in_seq=0 out_seq=0 2024-04-25T16:11:45.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.297+0000 7ff9f6ffd640 10 client.17691 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:11:45.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.310+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/1997521799 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5596e4ded990 con 0x5596e4a97f60 2024-04-25T16:11:45.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.310+0000 7ff9f6ffd640 10 client.17691 handle_command_reply: tid=0 2024-04-25T16:11:45.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.310+0000 7ffa24e08740 1 client.17691 shutdown 2024-04-25T16:11:45.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.313+0000 7ffa24e08740 1 -- 172.21.15.31:0/1997521799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab04d0 msgr2=0x5596e4e856f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:45.316 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.313+0000 7ffa24e08740 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab04d0 0x5596e4e856f0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x5596e4e7b700 tx=0x7ffa080a60a0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.317+0000 7ffa24e08740 1 -- 172.21.15.31:0/1997521799 shutdown_connections 2024-04-25T16:11:45.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.317+0000 7ffa24e08740 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.317+0000 7ffa24e08740 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab86b0 0x5596e4e85c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.317+0000 7ffa24e08740 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5596e4a97f60 0x5596e4e93910 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.317+0000 7ffa24e08740 1 --2- 172.21.15.31:0/1997521799 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab04d0 0x5596e4e856f0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.319 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.317+0000 7ffa24e08740 1 -- 172.21.15.31:0/1997521799 >> 172.21.15.31:0/1997521799 conn(0x5596e4acd4e0 msgr2=0x5596e4e763f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:45.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.317+0000 7ff9f6ffd640 0 client.17691 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:11:45.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.317+0000 7ffa24e08740 1 -- 172.21.15.31:0/1997521799 shutdown_connections 2024-04-25T16:11:45.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.321+0000 7ffa24e08740 1 -- 172.21.15.31:0/1997521799 wait complete. 2024-04-25T16:11:45.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.322+0000 7ffa24e08740 20 client.17691 trim_cache size 0 max 16384 2024-04-25T16:11:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.385+0000 7ffa24e08740 1 Processor -- start 2024-04-25T16:11:45.388 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.385+0000 7ffa24e08740 1 -- start start 2024-04-25T16:11:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.388+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab04d0 0x5596e4e7bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.391 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.388+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab86b0 0x5596e4e84730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.392 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.389+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.393 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.389+0000 7ffa17fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.394 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.390+0000 7ffa17fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39320/0 (socket says 172.21.15.31:39320) 2024-04-25T16:11:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.390+0000 7ffa17fff640 1 -- 172.21.15.31:0/3703441860 learned_addr learned my addr 172.21.15.31:0/3703441860 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.391+0000 7ffa24e08740 1 -- 172.21.15.31:0/3703441860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400e1d0 con 0x5596e4ad9390 2024-04-25T16:11:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.391+0000 7ffa24e08740 1 -- 172.21.15.31:0/3703441860 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400fd10 con 0x5596e4ab86b0 2024-04-25T16:11:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.391+0000 7ffa16ffd640 1 --2- 172.21.15.31:0/3703441860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab86b0 0x5596e4e84730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.391+0000 7ffa24e08740 1 -- 172.21.15.31:0/3703441860 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400e8a0 con 0x5596e4ab04d0 2024-04-25T16:11:45.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.392+0000 7ffa177fe640 1 --2- 172.21.15.31:0/3703441860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab04d0 0x5596e4e7bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.394+0000 7ffa17fff640 1 -- 172.21.15.31:0/3703441860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab04d0 msgr2=0x5596e4e7bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.394+0000 7ffa17fff640 1 --2- 172.21.15.31:0/3703441860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab04d0 0x5596e4e7bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.394+0000 7ffa17fff640 1 -- 172.21.15.31:0/3703441860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab86b0 msgr2=0x5596e4e84730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.394+0000 7ffa17fff640 1 --2- 172.21.15.31:0/3703441860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab86b0 0x5596e4e84730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.398 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.394+0000 7ffa17fff640 1 -- 172.21.15.31:0/3703441860 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596e4e68ba0 con 0x5596e4ad9390 2024-04-25T16:11:45.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.396+0000 7ffa17fff640 1 --2- 172.21.15.31:0/3703441860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb20 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c005510 tx=0x7ffa0c02a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7effa858090dbd99 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.400+0000 7ff9fd7fa640 1 -- 172.21.15.31:0/3703441860 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c04e070 con 0x5596e4ad9390 2024-04-25T16:11:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.400+0000 7ff9fd7fa640 1 -- 172.21.15.31:0/3703441860 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa0c031a50 con 0x5596e4ad9390 2024-04-25T16:11:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.400+0000 7ff9fd7fa640 1 -- 172.21.15.31:0/3703441860 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0c045040 con 0x5596e4ad9390 2024-04-25T16:11:45.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.405+0000 7ffa24e08740 1 -- 172.21.15.31:0/3703441860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ad9390 msgr2=0x5596e4e8cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:45.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.405+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3703441860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb20 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c005510 tx=0x7ffa0c02a420 comp rx=0 tx=0).stop 2024-04-25T16:11:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.411+0000 7ffa24e08740 1 -- 172.21.15.31:0/3703441860 shutdown_connections 2024-04-25T16:11:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.411+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3703441860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ad9390 0x5596e4e8cb20 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.411+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3703441860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab86b0 0x5596e4e84730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.411+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3703441860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ab04d0 0x5596e4e7bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.411+0000 7ffa24e08740 1 -- 172.21.15.31:0/3703441860 >> 172.21.15.31:0/3703441860 conn(0x5596e4acd4e0 msgr2=0x5596e4e84020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:45.416 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.413+0000 7ffa24e08740 1 -- 172.21.15.31:0/3703441860 shutdown_connections 2024-04-25T16:11:45.419 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.414+0000 7ffa24e08740 1 -- 172.21.15.31:0/3703441860 wait complete. 2024-04-25T16:11:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.415+0000 7ffa24e08740 1 -- start start 2024-04-25T16:11:45.420 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.416+0000 7ffa24e08740 1 Processor -- start 2024-04-25T16:11:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.417+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab04d0 0x5596e4e8c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.417+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e8cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.417+0000 7ffa24e08740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ad9390 0x5596e4e89440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.421 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.417+0000 7ffa24e08740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400f640 con 0x5596e4ab86b0 2024-04-25T16:11:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.417+0000 7ffa24e08740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400ef70 con 0x5596e4ab04d0 2024-04-25T16:11:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.417+0000 7ffa24e08740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ffa0400c260 con 0x5596e4ad9390 2024-04-25T16:11:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.417+0000 7ffa24e08740 10 client.0 _ll_register_callbacks cb 0x7ffa080a5ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:45.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.418+0000 7ffa16ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e8cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.419+0000 7ffa16ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e8cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39328/0 (socket says 172.21.15.31:39328) 2024-04-25T16:11:45.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.420+0000 7ffa16ffd640 1 -- 172.21.15.31:0/3889158524 learned_addr learned my addr 172.21.15.31:0/3889158524 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:45.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.421+0000 7ffa17fff640 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ad9390 0x5596e4e89440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.422+0000 7ffa177fe640 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab04d0 0x5596e4e8c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.423+0000 7ffa16ffd640 1 -- 172.21.15.31:0/3889158524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ad9390 msgr2=0x5596e4e89440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.423+0000 7ffa16ffd640 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ad9390 0x5596e4e89440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.426 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.424+0000 7ffa16ffd640 1 -- 172.21.15.31:0/3889158524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab04d0 msgr2=0x5596e4e8c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.424+0000 7ffa16ffd640 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab04d0 0x5596e4e8c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.427 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.424+0000 7ffa16ffd640 1 -- 172.21.15.31:0/3889158524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596e4e89b80 con 0x5596e4ab86b0 2024-04-25T16:11:45.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.432+0000 7ff9f6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:45.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.432+0000 7ffa16ffd640 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e8cc50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ffa10009740 tx=0x7ffa10006ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801c87452179b7e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.434+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/3889158524 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1002b070 con 0x5596e4ab86b0 2024-04-25T16:11:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.434+0000 7ffa24e08740 1 -- 172.21.15.31:0/3889158524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa0c031a50 con 0x5596e4ab86b0 2024-04-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.434+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/3889158524 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7ffa1000bce0 con 0x5596e4ab86b0 2024-04-25T16:11:45.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.436+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/3889158524 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffa10037050 con 0x5596e4ab86b0 2024-04-25T16:11:45.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.439+0000 7ffa24e08740 10 client.17697 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:45.443 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.439+0000 7ffa24e08740 1 -- 172.21.15.31:0/3889158524 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5596e4efaae0 con 0x5596e4ab86b0 2024-04-25T16:11:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.440+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/3889158524 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa0c031a50 con 0x5596e4ab86b0 2024-04-25T16:11:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.440+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/3889158524 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7ffa10023040 con 0x5596e4ab86b0 2024-04-25T16:11:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.440+0000 7ffa24e08740 10 client.17697 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.440+0000 7ffa24e08740 10 client.17697 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:11:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.440+0000 7ffa24e08740 20 client.17697 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.440+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5596e4a6d3a0 0x5596e4efd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.440+0000 7ffa24e08740 4 client.17697 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.440+0000 7ffa24e08740 1 -- 172.21.15.31:0/3889158524 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5596e4de7800 con 0x5596e4a6d3a0 2024-04-25T16:11:45.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.450+0000 7ffa177fe640 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5596e4a6d3a0 0x5596e4efd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:45.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.455+0000 7ffa177fe640 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5596e4a6d3a0 0x5596e4efd500 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3da151b95d744ea3 server_cookie=d75fb36f3a7f5a24 in_seq=0 out_seq=0 2024-04-25T16:11:45.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.457+0000 7ff9f6ffd640 10 client.17697 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:11:45.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.457+0000 7ff9f6ffd640 1 -- 172.21.15.31:0/3889158524 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5596e4de7800 con 0x5596e4a6d3a0 2024-04-25T16:11:45.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.457+0000 7ff9f6ffd640 10 client.17697 handle_command_reply: tid=0 2024-04-25T16:11:45.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.457+0000 7ffa24e08740 1 client.17697 shutdown 2024-04-25T16:11:45.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.461+0000 7ffa24e08740 1 -- 172.21.15.31:0/3889158524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 msgr2=0x5596e4e8cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:45.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.461+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e8cc50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7ffa10009740 tx=0x7ffa10006ca0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.463 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.461+0000 7ffa24e08740 1 -- 172.21.15.31:0/3889158524 shutdown_connections 2024-04-25T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.461+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5596e4ad9390 0x5596e4e89440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.461+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x5596e4a6d3a0 0x5596e4efd500 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.461+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5596e4ab86b0 0x5596e4e8cc50 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.461+0000 7ffa24e08740 1 --2- 172.21.15.31:0/3889158524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5596e4ab04d0 0x5596e4e8c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.461+0000 7ffa24e08740 1 -- 172.21.15.31:0/3889158524 >> 172.21.15.31:0/3889158524 conn(0x5596e4acd4e0 msgr2=0x5596e4e79ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.461+0000 7ff9f6ffd640 0 client.17697 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:11:45.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.462+0000 7ffa24e08740 1 -- 172.21.15.31:0/3889158524 shutdown_connections 2024-04-25T16:11:45.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.464+0000 7ffa24e08740 1 -- 172.21.15.31:0/3889158524 wait complete. 2024-04-25T16:11:45.467 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.465+0000 7ffa24e08740 20 client.17697 trim_cache size 0 max 16384 2024-04-25T16:11:45.468 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:45.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 -- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ffa000afcf0 msgr2=0x7ffa000b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:45.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ffa000afcf0 0x7ffa000b2200 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffa0c0096c0 tx=0x7ffa0c031400 comp rx=0 tx=0).stop 2024-04-25T16:11:45.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 -- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa18102800 msgr2=0x7ffa18072150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:45.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa18102800 0x7ffa18072150 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800b9b0 tx=0x7ffa0800be70 comp rx=0 tx=0).stop 2024-04-25T16:11:45.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 -- 172.21.15.31:0/163018555 shutdown_connections 2024-04-25T16:11:45.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7ffa000afcf0 0x7ffa000b2200 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffa18108740 0x7ffa180735f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffa18103a40 0x7ffa18072690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 --2- 172.21.15.31:0/163018555 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7ffa18102800 0x7ffa18072150 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:45.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.468+0000 7ffa1cc22640 1 -- 172.21.15.31:0/163018555 >> 172.21.15.31:0/163018555 conn(0x7ffa180fd9f0 msgr2=0x7ffa18108f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:45.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.479+0000 7ffa1cc22640 1 -- 172.21.15.31:0/163018555 shutdown_connections 2024-04-25T16:11:45.486 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:45.481+0000 7ffa1cc22640 1 -- 172.21.15.31:0/163018555 wait complete. 2024-04-25T16:11:45.540 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:11:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.026+0000 7fce2d5e0640 1 -- 172.21.15.31:0/486119752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 msgr2=0x7fce2810b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:46.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.026+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/486119752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 0x7fce2810b5a0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fce18009860 tx=0x7fce1802a380 comp rx=0 tx=0).stop 2024-04-25T16:11:46.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.037+0000 7fce2d5e0640 1 -- 172.21.15.31:0/486119752 shutdown_connections 2024-04-25T16:11:46.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.037+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/486119752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce2810fe90 0x7fce281102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.037+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/486119752 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 0x7fce2810b5a0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.037+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/486119752 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce2810a340 0x7fce2810ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.037+0000 7fce2d5e0640 1 -- 172.21.15.31:0/486119752 >> 172.21.15.31:0/486119752 conn(0x7fce28070320 msgr2=0x7fce280727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:46.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.042+0000 7fce2d5e0640 1 -- 172.21.15.31:0/486119752 shutdown_connections 2024-04-25T16:11:46.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.045+0000 7fce2d5e0640 1 -- 172.21.15.31:0/486119752 wait complete. 2024-04-25T16:11:46.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.047+0000 7fce2d5e0640 1 Processor -- start 2024-04-25T16:11:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.049+0000 7fce2d5e0640 1 -- start start 2024-04-25T16:11:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.050+0000 7fce2d5e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce2810a340 0x7fce2811ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.050+0000 7fce2d5e0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 0x7fce2811a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.050+0000 7fce2d5e0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce2810fe90 0x7fce2811a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.050+0000 7fce2d5e0640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fce28112420 con 0x7fce2810b190 2024-04-25T16:11:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.050+0000 7fce2d5e0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fce281122a0 con 0x7fce2810fe90 2024-04-25T16:11:46.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.050+0000 7fce2d5e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce281125a0 con 0x7fce2810a340 2024-04-25T16:11:46.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.055+0000 7fce277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 0x7fce2811a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.056+0000 7fce27fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce2810a340 0x7fce2811ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.056+0000 7fce277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 0x7fce2811a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39346/0 (socket says 172.21.15.31:39346) 2024-04-25T16:11:46.058 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.056+0000 7fce277fe640 1 -- 172.21.15.31:0/647890918 learned_addr learned my addr 172.21.15.31:0/647890918 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:46.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.060+0000 7fce2cddf640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce2810fe90 0x7fce2811a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.065+0000 7fce277fe640 1 -- 172.21.15.31:0/647890918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce2810a340 msgr2=0x7fce2811ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:46.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.065+0000 7fce277fe640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce2810a340 0x7fce2811ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.065+0000 7fce277fe640 1 -- 172.21.15.31:0/647890918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce2810fe90 msgr2=0x7fce2811a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:46.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.065+0000 7fce277fe640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce2810fe90 0x7fce2811a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.065+0000 7fce277fe640 1 -- 172.21.15.31:0/647890918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce2811ac00 con 0x7fce2810b190 2024-04-25T16:11:46.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.069+0000 7fce277fe640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 0x7fce2811a100 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fce18009980 tx=0x7fce18030d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:46.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.070+0000 7fce257fa640 1 -- 172.21.15.31:0/647890918 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce18037050 con 0x7fce2810b190 2024-04-25T16:11:46.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.070+0000 7fce2d5e0640 1 -- 172.21.15.31:0/647890918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce2811ae90 con 0x7fce2810b190 2024-04-25T16:11:46.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.070+0000 7fce2d5e0640 1 -- 172.21.15.31:0/647890918 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce281b9a00 con 0x7fce2810b190 2024-04-25T16:11:46.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.070+0000 7fce257fa640 1 -- 172.21.15.31:0/647890918 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce18038040 con 0x7fce2810b190 2024-04-25T16:11:46.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.070+0000 7fce257fa640 1 -- 172.21.15.31:0/647890918 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce180046c0 con 0x7fce2810b190 2024-04-25T16:11:46.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.084+0000 7fce357c9740 1 Processor -- start 2024-04-25T16:11:46.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.085+0000 7fce357c9740 1 -- start start 2024-04-25T16:11:46.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.085+0000 7fce357c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe9909e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.086+0000 7fce357c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe990f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.086+0000 7fce357c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 0x5573fe991bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.086+0000 7fce357c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5573fe8f4d80 con 0x5573fe5cd6b0 2024-04-25T16:11:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.086+0000 7fce357c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5573fe8aea20 con 0x5573fe5ee390 2024-04-25T16:11:46.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.086+0000 7fce357c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5573fe8f0d40 con 0x5573fe5c54d0 2024-04-25T16:11:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.087+0000 7fce27fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe990f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.088+0000 7fce27fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe990f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:42200/0 (socket says 172.21.15.31:42200) 2024-04-25T16:11:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.088+0000 7fce27fff640 1 -- 172.21.15.31:0/1492764437 learned_addr learned my addr 172.21.15.31:0/1492764437 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:46.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.088+0000 7fce2cddf640 1 --2- 172.21.15.31:0/1492764437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe9909e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.088+0000 7fce27fff640 1 -- 172.21.15.31:0/1492764437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 msgr2=0x5573fe9909e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.088+0000 7fce27fff640 1 --2- 172.21.15.31:0/1492764437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe9909e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.088+0000 7fce27fff640 1 -- 172.21.15.31:0/1492764437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 msgr2=0x5573fe991bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:11:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.088+0000 7fce27fff640 1 --2- 172.21.15.31:0/1492764437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 0x5573fe991bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.088+0000 7fce27fff640 1 -- 172.21.15.31:0/1492764437 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573fe8b9c30 con 0x5573fe5c54d0 2024-04-25T16:11:46.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.089+0000 7fce27fff640 1 --2- 172.21.15.31:0/1492764437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe990f60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fce20009f90 tx=0x7fce2000f890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe9422ace4946447 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.090+0000 7fce257fa640 1 -- 172.21.15.31:0/647890918 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fce18045070 con 0x7fce2810b190 2024-04-25T16:11:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.090+0000 7fce0e7fc640 1 -- 172.21.15.31:0/1492764437 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce20004b20 con 0x5573fe5c54d0 2024-04-25T16:11:46.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.091+0000 7fce0e7fc640 1 -- 172.21.15.31:0/1492764437 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce20004cc0 con 0x5573fe5c54d0 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.092+0000 7fce257fa640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce100afbb0 0x7fce100b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.092+0000 7fce357c9740 1 -- 172.21.15.31:0/1492764437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 msgr2=0x5573fe990f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.092+0000 7fce357c9740 1 --2- 172.21.15.31:0/1492764437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe990f60 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fce20009f90 tx=0x7fce2000f890 comp rx=0 tx=0).stop 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.093+0000 7fce257fa640 1 -- 172.21.15.31:0/647890918 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fce180fe830 con 0x7fce2810b190 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.093+0000 7fce2cddf640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce100afbb0 0x7fce100b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.097+0000 7fce357c9740 1 -- 172.21.15.31:0/1492764437 shutdown_connections 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.097+0000 7fce357c9740 1 --2- 172.21.15.31:0/1492764437 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 0x5573fe991bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.097+0000 7fce357c9740 1 --2- 172.21.15.31:0/1492764437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe990f60 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.097+0000 7fce357c9740 1 --2- 172.21.15.31:0/1492764437 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe9909e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.097+0000 7fce357c9740 1 -- 172.21.15.31:0/1492764437 >> 172.21.15.31:0/1492764437 conn(0x5573fe5e24e0 msgr2=0x5573fe98c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:46.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.098+0000 7fce357c9740 1 -- 172.21.15.31:0/1492764437 shutdown_connections 2024-04-25T16:11:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.099+0000 7fce357c9740 1 -- 172.21.15.31:0/1492764437 wait complete. 2024-04-25T16:11:46.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.100+0000 7fce357c9740 1 -- start start 2024-04-25T16:11:46.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.103+0000 7fce357c9740 1 Processor -- start 2024-04-25T16:11:46.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.104+0000 7fce2cddf640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce100afbb0 0x7fce100b20c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x5573fe991a30 tx=0x7fce1c00b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.104+0000 7fce357c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe99a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.105+0000 7fce357c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 0x5573fe99ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.105+0000 7fce357c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe9a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.105+0000 7fce357c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5573fe992750 con 0x5573fe5cd6b0 2024-04-25T16:11:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.105+0000 7fce357c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5573fe87bdc0 con 0x5573fe5ee390 2024-04-25T16:11:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.105+0000 7fce357c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5573fe9928d0 con 0x5573fe5c54d0 2024-04-25T16:11:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.105+0000 7fce357c9740 10 client.0 _ll_register_callbacks cb 0x5573fe897600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:46.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.106+0000 7fce2cddf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe9a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.106+0000 7fce27fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe99a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.106+0000 7fce277fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 0x5573fe99ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.106+0000 7fce2cddf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe9a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48774/0 (socket says 172.21.15.31:48774) 2024-04-25T16:11:46.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.106+0000 7fce2cddf640 1 -- 172.21.15.31:0/196355435 learned_addr learned my addr 172.21.15.31:0/196355435 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:46.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.107+0000 7fce277fe640 1 -- 172.21.15.31:0/196355435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 msgr2=0x5573fe99a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:46.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.107+0000 7fce277fe640 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe99a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.107+0000 7fce277fe640 1 -- 172.21.15.31:0/196355435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 msgr2=0x5573fe9a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:46.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.107+0000 7fce277fe640 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe9a1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.107+0000 7fce277fe640 1 -- 172.21.15.31:0/196355435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573fe6063f0 con 0x5573fe5cd6b0 2024-04-25T16:11:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.111+0000 7fce277fe640 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 0x5573fe99ad80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fce180e7fd0 tx=0x7fce180c69a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f1923e7fae1101 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.111+0000 7fce077fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:11:46.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.115+0000 7fce077fe640 1 -- 172.21.15.31:0/196355435 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce180dd070 con 0x5573fe5cd6b0 2024-04-25T16:11:46.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.115+0000 7fce357c9740 1 -- 172.21.15.31:0/196355435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5573fe902990 con 0x5573fe5cd6b0 2024-04-25T16:11:46.117 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.115+0000 7fce077fe640 1 -- 172.21.15.31:0/196355435 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fce180328a0 con 0x5573fe5cd6b0 2024-04-25T16:11:46.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.119+0000 7fce077fe640 1 -- 172.21.15.31:0/196355435 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce180e13f0 con 0x5573fe5cd6b0 2024-04-25T16:11:46.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.120+0000 7fce357c9740 10 client.17706 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:46.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.120+0000 7fce357c9740 1 -- 172.21.15.31:0/196355435 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5573fe6d07d0 con 0x5573fe5cd6b0 2024-04-25T16:11:46.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.123+0000 7fce077fe640 1 -- 172.21.15.31:0/196355435 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5573fe902990 con 0x5573fe5cd6b0 2024-04-25T16:11:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.124+0000 7fce077fe640 1 -- 172.21.15.31:0/196355435 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fce180e0030 con 0x5573fe5cd6b0 2024-04-25T16:11:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.124+0000 7fce357c9740 10 client.17706 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.124+0000 7fce357c9740 10 client.17706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.124+0000 7fce357c9740 20 client.17706 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.124+0000 7fce357c9740 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5573fe5acf60 0x5573fe9a8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.124+0000 7fce357c9740 4 client.17706 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:46.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.124+0000 7fce357c9740 1 -- 172.21.15.31:0/196355435 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5573fe8fc800 con 0x5573fe5acf60 2024-04-25T16:11:46.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.129+0000 7fce27fff640 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5573fe5acf60 0x5573fe9a8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.138+0000 7fce27fff640 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5573fe5acf60 0x5573fe9a8590 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d740e5846e85c4 server_cookie=a604d260f50f2a86 in_seq=0 out_seq=0 2024-04-25T16:11:46.141 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.138+0000 7fce077fe640 10 client.17706 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:46.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.146+0000 7fce077fe640 1 -- 172.21.15.31:0/196355435 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5573fe8fc800 con 0x5573fe5acf60 2024-04-25T16:11:46.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.146+0000 7fce077fe640 10 client.17706 handle_command_reply: tid=0 2024-04-25T16:11:46.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.146+0000 7fce357c9740 1 client.17706 shutdown 2024-04-25T16:11:46.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.148+0000 7fce357c9740 1 -- 172.21.15.31:0/196355435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 msgr2=0x5573fe99ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:46.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.148+0000 7fce357c9740 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 0x5573fe99ad80 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fce180e7fd0 tx=0x7fce180c69a0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.154+0000 7fce357c9740 1 -- 172.21.15.31:0/196355435 shutdown_connections 2024-04-25T16:11:46.156 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.154+0000 7fce357c9740 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5573fe5acf60 0x5573fe9a8590 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.154+0000 7fce357c9740 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe9a1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.154+0000 7fce357c9740 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5cd6b0 0x5573fe99ad80 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.154+0000 7fce357c9740 1 --2- 172.21.15.31:0/196355435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5c54d0 0x5573fe99a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.157 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.154+0000 7fce357c9740 1 -- 172.21.15.31:0/196355435 >> 172.21.15.31:0/196355435 conn(0x5573fe5e24e0 msgr2=0x5573fe98f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:46.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.155+0000 7fce077fe640 0 client.17706 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:46.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.156+0000 7fce357c9740 1 -- 172.21.15.31:0/196355435 shutdown_connections 2024-04-25T16:11:46.165 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.160+0000 7fce357c9740 1 -- 172.21.15.31:0/196355435 wait complete. 2024-04-25T16:11:46.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.164+0000 7fce357c9740 20 client.17706 trim_cache size 0 max 16384 2024-04-25T16:11:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.208+0000 7fce357c9740 1 Processor -- start 2024-04-25T16:11:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.209+0000 7fce357c9740 1 -- start start 2024-04-25T16:11:46.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.209+0000 7fce357c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5c54d0 0x5573fe999af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.210+0000 7fce357c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5cd6b0 0x5573fe9a1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.210+0000 7fce357c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe99a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.210+0000 7fce357c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fce1400e540 con 0x5573fe5c54d0 2024-04-25T16:11:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.210+0000 7fce357c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fce14010080 con 0x5573fe5ee390 2024-04-25T16:11:46.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.210+0000 7fce357c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce1400ec10 con 0x5573fe5cd6b0 2024-04-25T16:11:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.214+0000 7fce277fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5cd6b0 0x5573fe9a1c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.214+0000 7fce2cddf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe99a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.214+0000 7fce2cddf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe99a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48790/0 (socket says 172.21.15.31:48790) 2024-04-25T16:11:46.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.214+0000 7fce2cddf640 1 -- 172.21.15.31:0/4235515475 learned_addr learned my addr 172.21.15.31:0/4235515475 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:46.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.218+0000 7fce277fe640 1 -- 172.21.15.31:0/4235515475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 msgr2=0x5573fe99a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:46.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.218+0000 7fce277fe640 1 --2- 172.21.15.31:0/4235515475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe99a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.218+0000 7fce277fe640 1 -- 172.21.15.31:0/4235515475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5c54d0 msgr2=0x5573fe999af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:46.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.218+0000 7fce277fe640 1 --2- 172.21.15.31:0/4235515475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5c54d0 0x5573fe999af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.218+0000 7fce277fe640 1 -- 172.21.15.31:0/4235515475 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5573fe97dba0 con 0x5573fe5cd6b0 2024-04-25T16:11:46.228 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.226+0000 7fce277fe640 1 --2- 172.21.15.31:0/4235515475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5cd6b0 0x5573fe9a1c00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fce180fc390 tx=0x7fce180057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6f23e7fd8056071 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.234+0000 7fce0dffb640 1 -- 172.21.15.31:0/4235515475 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce180dd030 con 0x5573fe5cd6b0 2024-04-25T16:11:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.234+0000 7fce0dffb640 1 -- 172.21.15.31:0/4235515475 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce180cfb80 con 0x5573fe5cd6b0 2024-04-25T16:11:46.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.234+0000 7fce0dffb640 1 -- 172.21.15.31:0/4235515475 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce180cf860 con 0x5573fe5cd6b0 2024-04-25T16:11:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.236+0000 7fce357c9740 1 -- 172.21.15.31:0/4235515475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5cd6b0 msgr2=0x5573fe9a1c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.236+0000 7fce357c9740 1 --2- 172.21.15.31:0/4235515475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5cd6b0 0x5573fe9a1c00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fce180fc390 tx=0x7fce180057c0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.237+0000 7fce357c9740 1 -- 172.21.15.31:0/4235515475 shutdown_connections 2024-04-25T16:11:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.237+0000 7fce357c9740 1 --2- 172.21.15.31:0/4235515475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5ee390 0x5573fe99a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.237+0000 7fce357c9740 1 --2- 172.21.15.31:0/4235515475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5cd6b0 0x5573fe9a1c00 secure :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fce180fc390 tx=0x7fce180057c0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.237+0000 7fce357c9740 1 --2- 172.21.15.31:0/4235515475 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5c54d0 0x5573fe999af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.237+0000 7fce357c9740 1 -- 172.21.15.31:0/4235515475 >> 172.21.15.31:0/4235515475 conn(0x5573fe5e24e0 msgr2=0x5573fe995470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:46.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.240+0000 7fce357c9740 1 -- 172.21.15.31:0/4235515475 shutdown_connections 2024-04-25T16:11:46.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.240+0000 7fce357c9740 1 -- 172.21.15.31:0/4235515475 wait complete. 2024-04-25T16:11:46.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.241+0000 7fce357c9740 1 -- start start 2024-04-25T16:11:46.244 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.241+0000 7fce357c9740 1 Processor -- start 2024-04-25T16:11:46.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.242+0000 7fce357c9740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5c54d0 0x5573fe99e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.242+0000 7fce357c9740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5ee390 0x5573fe99eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.242+0000 7fce357c9740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5823a0 0x5573fea0e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.242+0000 7fce357c9740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fce1400f9b0 con 0x5573fe5823a0 2024-04-25T16:11:46.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.242+0000 7fce357c9740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fce1400f2e0 con 0x5573fe5c54d0 2024-04-25T16:11:46.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.242+0000 7fce357c9740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fce1400c520 con 0x5573fe5ee390 2024-04-25T16:11:46.245 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.242+0000 7fce357c9740 10 client.0 _ll_register_callbacks cb 0x7fce18005dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:46.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.245+0000 7fce27fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5c54d0 0x5573fe99e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.245+0000 7fce27fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5c54d0 0x5573fe99e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48806/0 (socket says 172.21.15.31:48806) 2024-04-25T16:11:46.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.245+0000 7fce27fff640 1 -- 172.21.15.31:0/128288632 learned_addr learned my addr 172.21.15.31:0/128288632 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:46.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.245+0000 7fce277fe640 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5ee390 0x5573fe99eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.247+0000 7fce2cddf640 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5823a0 0x5573fea0e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.248+0000 7fce27fff640 1 -- 172.21.15.31:0/128288632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5ee390 msgr2=0x5573fe99eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:46.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.248+0000 7fce27fff640 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5ee390 0x5573fe99eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.248+0000 7fce27fff640 1 -- 172.21.15.31:0/128288632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5823a0 msgr2=0x5573fea0e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:46.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.248+0000 7fce27fff640 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5823a0 0x5573fea0e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.251 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.248+0000 7fce27fff640 1 -- 172.21.15.31:0/128288632 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5573fe9a1e60 con 0x5573fe5c54d0 2024-04-25T16:11:46.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.256+0000 7fce27fff640 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5c54d0 0x5573fe99e960 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fce20009f60 tx=0x7fce20009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24073c381de8b871 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:46.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.257+0000 7fce077fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:46.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.259+0000 7fce077fe640 1 -- 172.21.15.31:0/128288632 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce2002d060 con 0x5573fe5c54d0 2024-04-25T16:11:46.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.259+0000 7fce077fe640 1 -- 172.21.15.31:0/128288632 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fce20003970 con 0x5573fe5c54d0 2024-04-25T16:11:46.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.259+0000 7fce357c9740 1 -- 172.21.15.31:0/128288632 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fce180cfb80 con 0x5573fe5c54d0 2024-04-25T16:11:46.266 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.264+0000 7fce077fe640 1 -- 172.21.15.31:0/128288632 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fce20003b70 con 0x5573fe5c54d0 2024-04-25T16:11:46.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.264+0000 7fce357c9740 10 client.27386 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:46.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.264+0000 7fce357c9740 1 -- 172.21.15.31:0/128288632 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5573fea0fd10 con 0x5573fe5c54d0 2024-04-25T16:11:46.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.268+0000 7fce077fe640 1 -- 172.21.15.31:0/128288632 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce180cfb80 con 0x5573fe5c54d0 2024-04-25T16:11:46.270 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.268+0000 7fce077fe640 1 -- 172.21.15.31:0/128288632 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fce2000b050 con 0x5573fe5c54d0 2024-04-25T16:11:46.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.268+0000 7fce357c9740 10 client.27386 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:46.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.268+0000 7fce357c9740 10 client.27386 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:46.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.268+0000 7fce357c9740 20 client.27386 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:46.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.268+0000 7fce357c9740 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5573fea103e0 0x5573fea12800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:46.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.268+0000 7fce357c9740 4 client.27386 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:11:46.273 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.268+0000 7fce357c9740 1 -- 172.21.15.31:0/128288632 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5573fe877610 con 0x5573fea103e0 2024-04-25T16:11:46.275 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.273+0000 7fce277fe640 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5573fea103e0 0x5573fea12800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:46.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.292+0000 7fce077fe640 10 client.27386 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:46.295 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.292+0000 7fce277fe640 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5573fea103e0 0x5573fea12800 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=515f3bd9c1293515 server_cookie=fb170c76e00e5317 in_seq=0 out_seq=0 2024-04-25T16:11:46.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.298+0000 7fce077fe640 1 -- 172.21.15.31:0/128288632 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5573fe877610 con 0x5573fea103e0 2024-04-25T16:11:46.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.298+0000 7fce077fe640 10 client.27386 handle_command_reply: tid=0 2024-04-25T16:11:46.300 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.298+0000 7fce357c9740 1 client.27386 shutdown 2024-04-25T16:11:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.309+0000 7fce357c9740 1 -- 172.21.15.31:0/128288632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5c54d0 msgr2=0x5573fe99e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.309+0000 7fce357c9740 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5c54d0 0x5573fe99e960 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fce20009f60 tx=0x7fce20009240 comp rx=0 tx=0).stop 2024-04-25T16:11:46.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.309+0000 7fce357c9740 1 -- 172.21.15.31:0/128288632 shutdown_connections 2024-04-25T16:11:46.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.309+0000 7fce357c9740 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5573fea103e0 0x5573fea12800 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.309+0000 7fce357c9740 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5573fe5823a0 0x5573fea0e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.309+0000 7fce357c9740 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5573fe5ee390 0x5573fe99eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.309+0000 7fce357c9740 1 --2- 172.21.15.31:0/128288632 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5573fe5c54d0 0x5573fe99e960 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.309+0000 7fce357c9740 1 -- 172.21.15.31:0/128288632 >> 172.21.15.31:0/128288632 conn(0x5573fe5e24e0 msgr2=0x5573fe99a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:46.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.315+0000 7fce077fe640 0 client.27386 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:46.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.315+0000 7fce357c9740 1 -- 172.21.15.31:0/128288632 shutdown_connections 2024-04-25T16:11:46.317 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.315+0000 7fce357c9740 1 -- 172.21.15.31:0/128288632 wait complete. 2024-04-25T16:11:46.321 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.319+0000 7fce357c9740 20 client.27386 trim_cache size 0 max 16384 2024-04-25T16:11:46.323 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:11:46.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.321+0000 7fce2d5e0640 1 -- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce100afbb0 msgr2=0x7fce100b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:46.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.322+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce100afbb0 0x7fce100b20c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x5573fe991a30 tx=0x7fce1c00b2e0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.322+0000 7fce2d5e0640 1 -- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 msgr2=0x7fce2811a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:46.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.322+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 0x7fce2811a100 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fce18009980 tx=0x7fce18030d70 comp rx=0 tx=0).stop 2024-04-25T16:11:46.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.323+0000 7fce2d5e0640 1 -- 172.21.15.31:0/647890918 shutdown_connections 2024-04-25T16:11:46.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.323+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fce100afbb0 0x7fce100b20c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.323+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fce2810fe90 0x7fce2811a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.323+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fce2810b190 0x7fce2811a100 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.323+0000 7fce2d5e0640 1 --2- 172.21.15.31:0/647890918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce2810a340 0x7fce2811ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:46.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.323+0000 7fce2d5e0640 1 -- 172.21.15.31:0/647890918 >> 172.21.15.31:0/647890918 conn(0x7fce28070320 msgr2=0x7fce2810de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:46.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.323+0000 7fce2d5e0640 1 -- 172.21.15.31:0/647890918 shutdown_connections 2024-04-25T16:11:46.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:46.324+0000 7fce2d5e0640 1 -- 172.21.15.31:0/647890918 wait complete. 2024-04-25T16:11:47.376 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:11:47.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:11:47.377 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:11:47.500 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:47 smithi031 ceph-mon[29140]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 749 MiB/s wr, 2.01k op/s 2024-04-25T16:11:47.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:47 smithi063 ceph-mon[32413]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 749 MiB/s wr, 2.01k op/s 2024-04-25T16:11:47.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:47 smithi186 ceph-mon[33258]: pgmap v665: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 749 MiB/s wr, 2.01k op/s 2024-04-25T16:11:47.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.894+0000 7f3cf0292640 1 -- 172.21.15.31:0/712002643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce8108920 msgr2=0x7f3ce810ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:47.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.894+0000 7f3cf0292640 1 --2- 172.21.15.31:0/712002643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce8108920 0x7f3ce810ad40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009ca0 tx=0x7f3ce002a8f0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.895+0000 7f3cf0292640 1 -- 172.21.15.31:0/712002643 shutdown_connections 2024-04-25T16:11:47.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.895+0000 7f3cf0292640 1 --2- 172.21.15.31:0/712002643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce8108920 0x7f3ce810ad40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.895+0000 7f3cf0292640 1 --2- 172.21.15.31:0/712002643 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce8103c20 0x7f3ce8104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.895+0000 7f3cf0292640 1 --2- 172.21.15.31:0/712002643 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ce81029e0 0x7f3ce81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.895+0000 7f3cf0292640 1 -- 172.21.15.31:0/712002643 >> 172.21.15.31:0/712002643 conn(0x7f3ce80fdbd0 msgr2=0x7f3ce8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:47.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.906+0000 7f3cf0292640 1 -- 172.21.15.31:0/712002643 shutdown_connections 2024-04-25T16:11:47.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.908+0000 7f3cf0292640 1 -- 172.21.15.31:0/712002643 wait complete. 2024-04-25T16:11:47.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.910+0000 7f3cf0292640 1 Processor -- start 2024-04-25T16:11:47.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.911+0000 7f3cf0292640 1 -- start start 2024-04-25T16:11:47.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.911+0000 7f3cf0292640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce81029e0 0x7f3ce8071510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.911+0000 7f3cf0292640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ce8103c20 0x7f3ce8071a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.911+0000 7f3cf0292640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce8108920 0x7f3ce81b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.911+0000 7f3cf0292640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce810d450 con 0x7f3ce8108920 2024-04-25T16:11:47.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.911+0000 7f3cf0292640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce810d2d0 con 0x7f3ce8103c20 2024-04-25T16:11:47.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.911+0000 7f3cf0292640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce810d5d0 con 0x7f3ce81029e0 2024-04-25T16:11:47.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.914+0000 7f3ced82d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ce8103c20 0x7f3ce8071a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:47.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.914+0000 7f3ced82d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ce8103c20 0x7f3ce8071a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48828/0 (socket says 172.21.15.31:48828) 2024-04-25T16:11:47.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.914+0000 7f3ced82d640 1 -- 172.21.15.31:0/4201593808 learned_addr learned my addr 172.21.15.31:0/4201593808 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:47.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.915+0000 7f3cee02e640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce81029e0 0x7f3ce8071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:47.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.916+0000 7f3cee02e640 1 -- 172.21.15.31:0/4201593808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ce8103c20 msgr2=0x7f3ce8071a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:47.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.916+0000 7f3cee02e640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ce8103c20 0x7f3ce8071a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.916+0000 7f3cee02e640 1 -- 172.21.15.31:0/4201593808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce8108920 msgr2=0x7f3ce81b0fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:11:47.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.916+0000 7f3cee02e640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce8108920 0x7f3ce81b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.916+0000 7f3cee02e640 1 -- 172.21.15.31:0/4201593808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce81127e0 con 0x7f3ce81029e0 2024-04-25T16:11:47.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.916+0000 7f3cee02e640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce81029e0 0x7f3ce8071510 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3ce81022b0 tx=0x7f3cd800b850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:47.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.917+0000 7f3cdf7fe640 1 -- 172.21.15.31:0/4201593808 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8015020 con 0x7f3ce81029e0 2024-04-25T16:11:47.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.917+0000 7f3cf0292640 1 -- 172.21.15.31:0/4201593808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce8074a20 con 0x7f3ce81029e0 2024-04-25T16:11:47.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.917+0000 7f3cf0292640 1 -- 172.21.15.31:0/4201593808 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce81a8a20 con 0x7f3ce81029e0 2024-04-25T16:11:47.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.917+0000 7f3cdf7fe640 1 -- 172.21.15.31:0/4201593808 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cd8011ce0 con 0x7f3ce81029e0 2024-04-25T16:11:47.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.917+0000 7f3cdf7fe640 1 -- 172.21.15.31:0/4201593808 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8010e10 con 0x7f3ce81029e0 2024-04-25T16:11:47.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.923+0000 7f3cdf7fe640 1 -- 172.21.15.31:0/4201593808 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3cd8018020 con 0x7f3ce81029e0 2024-04-25T16:11:47.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.926+0000 7f3cf7215740 1 Processor -- start 2024-04-25T16:11:47.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.927+0000 7f3cf7215740 1 -- start start 2024-04-25T16:11:47.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.927+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 0x564ad67284a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.927+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad635e550 0x564ad6728a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.927+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad6366730 0x564ad672f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.927+0000 7f3cf7215740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564ad668db00 con 0x564ad635e550 2024-04-25T16:11:47.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.927+0000 7f3cf7215740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564ad6629920 con 0x564ad6366730 2024-04-25T16:11:47.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.927+0000 7f3cf7215740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564ad6689ac0 con 0x564ad6387410 2024-04-25T16:11:47.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.929+0000 7f3cdf7fe640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3cd40afbf0 0x7f3cd40b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.929+0000 7f3cdf7fe640 1 -- 172.21.15.31:0/4201593808 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3cd80de3b0 con 0x7f3ce81029e0 2024-04-25T16:11:47.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.930+0000 7f3cee82f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 0x564ad67284a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:47.932 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.930+0000 7f3cee82f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 0x564ad67284a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:42254/0 (socket says 172.21.15.31:42254) 2024-04-25T16:11:47.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.930+0000 7f3cee82f640 1 -- 172.21.15.31:0/1348484404 learned_addr learned my addr 172.21.15.31:0/1348484404 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:47.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.930+0000 7f3cee02e640 1 --2- 172.21.15.31:0/1348484404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad635e550 0x564ad6728a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:47.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.931+0000 7f3cee02e640 1 -- 172.21.15.31:0/1348484404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 msgr2=0x564ad67284a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:47.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.931+0000 7f3cee02e640 1 --2- 172.21.15.31:0/1348484404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 0x564ad67284a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.931+0000 7f3cee02e640 1 -- 172.21.15.31:0/1348484404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad6366730 msgr2=0x564ad672f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:47.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.931+0000 7f3cee02e640 1 --2- 172.21.15.31:0/1348484404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad6366730 0x564ad672f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.931+0000 7f3cee02e640 1 -- 172.21.15.31:0/1348484404 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad64677e0 con 0x564ad635e550 2024-04-25T16:11:47.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.932+0000 7f3cee02e640 1 --2- 172.21.15.31:0/1348484404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad635e550 0x564ad6728a20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800bd50 tx=0x7f3cd80a6070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1c9867d644ba62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:47.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.933+0000 7f3cee82f640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3cd40afbf0 0x7f3cd40b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.934+0000 7f3cdcbfe640 1 -- 172.21.15.31:0/1348484404 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd80a6840 con 0x564ad635e550 2024-04-25T16:11:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.934+0000 7f3cdcbfe640 1 -- 172.21.15.31:0/1348484404 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cd80aabf0 con 0x564ad635e550 2024-04-25T16:11:47.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.934+0000 7f3cdcbfe640 1 -- 172.21.15.31:0/1348484404 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd808f0d0 con 0x564ad635e550 2024-04-25T16:11:47.941 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.938+0000 7f3cf7215740 1 -- 172.21.15.31:0/1348484404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad635e550 msgr2=0x564ad6728a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:47.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.938+0000 7f3cf7215740 1 --2- 172.21.15.31:0/1348484404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad635e550 0x564ad6728a20 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f3cd800bd50 tx=0x7f3cd80a6070 comp rx=0 tx=0).stop 2024-04-25T16:11:47.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.941+0000 7f3cee82f640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3cd40afbf0 0x7f3cd40b2100 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3ce002ae10 tx=0x7f3ce0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:47.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.942+0000 7f3cf7215740 1 -- 172.21.15.31:0/1348484404 shutdown_connections 2024-04-25T16:11:47.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.942+0000 7f3cf7215740 1 --2- 172.21.15.31:0/1348484404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad6366730 0x564ad672f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.942+0000 7f3cf7215740 1 --2- 172.21.15.31:0/1348484404 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad635e550 0x564ad6728a20 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.942+0000 7f3cf7215740 1 --2- 172.21.15.31:0/1348484404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 0x564ad67284a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.942+0000 7f3cf7215740 1 -- 172.21.15.31:0/1348484404 >> 172.21.15.31:0/1348484404 conn(0x564ad637b560 msgr2=0x564ad6723e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:47.958 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.956+0000 7f3cf7215740 1 -- 172.21.15.31:0/1348484404 shutdown_connections 2024-04-25T16:11:47.963 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.960+0000 7f3cf7215740 1 -- 172.21.15.31:0/1348484404 wait complete. 2024-04-25T16:11:47.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.961+0000 7f3cf7215740 1 -- start start 2024-04-25T16:11:47.964 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.962+0000 7f3cf7215740 1 Processor -- start 2024-04-25T16:11:47.967 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.964+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad635e550 0x564ad6732420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.964+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 0x564ad6732960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.964+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad6387410 0x564ad67398c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.964+0000 7f3cf7215740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x564ad6730370 con 0x564ad6366730 2024-04-25T16:11:47.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.964+0000 7f3cf7215740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x564ad66275d0 con 0x564ad6387410 2024-04-25T16:11:47.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.964+0000 7f3cf7215740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564ad67304f0 con 0x564ad635e550 2024-04-25T16:11:47.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.964+0000 7f3cf7215740 10 client.0 _ll_register_callbacks cb 0x564ad6615080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:47.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.965+0000 7f3cee02e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad635e550 0x564ad6732420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:47.968 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.965+0000 7f3cee82f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad6387410 0x564ad67398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:47.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.967+0000 7f3ced82d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 0x564ad6732960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:47.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.967+0000 7f3ced82d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 0x564ad6732960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39412/0 (socket says 172.21.15.31:39412) 2024-04-25T16:11:47.969 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.967+0000 7f3ced82d640 1 -- 172.21.15.31:0/350707345 learned_addr learned my addr 172.21.15.31:0/350707345 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.968+0000 7f3cee02e640 1 -- 172.21.15.31:0/350707345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad6387410 msgr2=0x564ad67398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.968+0000 7f3cee02e640 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad6387410 0x564ad67398c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.968+0000 7f3cee02e640 1 -- 172.21.15.31:0/350707345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 msgr2=0x564ad6732960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.968+0000 7f3cee02e640 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 0x564ad6732960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:47.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.968+0000 7f3cee02e640 1 -- 172.21.15.31:0/350707345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ad64c7f70 con 0x564ad635e550 2024-04-25T16:11:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.969+0000 7f3cee02e640 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad635e550 0x564ad6732420 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80a2de0 tx=0x7f3cd8006d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f540d787131e64d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.970+0000 7f3cd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.970+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/350707345 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd808f0d0 con 0x564ad635e550 2024-04-25T16:11:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.970+0000 7f3cf7215740 1 -- 172.21.15.31:0/350707345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cd80aabf0 con 0x564ad635e550 2024-04-25T16:11:47.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.970+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/350707345 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3cd808f270 con 0x564ad635e550 2024-04-25T16:11:47.973 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.970+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/350707345 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3cd8086050 con 0x564ad635e550 2024-04-25T16:11:47.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.974+0000 7f3cf7215740 10 client.27388 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:47.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.974+0000 7f3cf7215740 1 -- 172.21.15.31:0/350707345 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ad635f170 con 0x564ad635e550 2024-04-25T16:11:47.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.980+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/350707345 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cd80aabf0 con 0x564ad635e550 2024-04-25T16:11:47.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.981+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/350707345 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3cd808dce0 con 0x564ad635e550 2024-04-25T16:11:47.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.984+0000 7f3cf7215740 10 client.27388 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:47.986 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.984+0000 7f3cf7215740 10 client.27388 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:47.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.984+0000 7f3cf7215740 20 client.27388 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:47.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.984+0000 7f3cf7215740 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564ad63461f0 0x564ad673f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:47.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.984+0000 7f3cf7215740 4 client.27388 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:47.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.984+0000 7f3cf7215740 1 -- 172.21.15.31:0/350707345 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ad669b710 con 0x564ad63461f0 2024-04-25T16:11:47.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.985+0000 7f3ced82d640 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564ad63461f0 0x564ad673f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:47.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.993+0000 7f3ced82d640 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564ad63461f0 0x564ad673f980 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e558d90ec2e7c75 server_cookie=eda652e44e82a975 in_seq=0 out_seq=0 2024-04-25T16:11:47.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.993+0000 7f3cd1ffb640 10 client.27388 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:47.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.996+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/350707345 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564ad669b710 con 0x564ad63461f0 2024-04-25T16:11:47.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.996+0000 7f3cd1ffb640 10 client.27388 handle_command_reply: tid=0 2024-04-25T16:11:47.999 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.997+0000 7f3cf7215740 1 client.27388 shutdown 2024-04-25T16:11:48.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.997+0000 7f3cf7215740 1 -- 172.21.15.31:0/350707345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad635e550 msgr2=0x564ad6732420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.997+0000 7f3cf7215740 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad635e550 0x564ad6732420 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80a2de0 tx=0x7f3cd8006d90 comp rx=0 tx=0).stop 2024-04-25T16:11:48.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.998+0000 7f3cf7215740 1 -- 172.21.15.31:0/350707345 shutdown_connections 2024-04-25T16:11:48.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.998+0000 7f3cf7215740 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564ad63461f0 0x564ad673f980 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.998+0000 7f3cf7215740 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad6387410 0x564ad67398c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.998+0000 7f3cf7215740 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 0x564ad6732960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.998+0000 7f3cf7215740 1 --2- 172.21.15.31:0/350707345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad635e550 0x564ad6732420 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.998+0000 7f3cf7215740 1 -- 172.21.15.31:0/350707345 >> 172.21.15.31:0/350707345 conn(0x564ad637b560 msgr2=0x564ad6722ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.999+0000 7f3cd1ffb640 0 client.27388 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.999+0000 7f3cf7215740 1 -- 172.21.15.31:0/350707345 shutdown_connections 2024-04-25T16:11:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.999+0000 7f3cf7215740 1 -- 172.21.15.31:0/350707345 wait complete. 2024-04-25T16:11:48.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:47.999+0000 7f3cf7215740 20 client.27388 trim_cache size 0 max 16384 2024-04-25T16:11:48.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.119+0000 7f3cf7215740 1 Processor -- start 2024-04-25T16:11:48.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.119+0000 7f3cf7215740 1 -- start start 2024-04-25T16:11:48.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.120+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6731dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.120+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 0x564ad6739850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.121+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 0x564ad67237f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.121+0000 7f3cf7215740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc00e370 con 0x564ad6366730 2024-04-25T16:11:48.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.121+0000 7f3cf7215740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc00feb0 con 0x564ad635e550 2024-04-25T16:11:48.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.121+0000 7f3cf7215740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc00ea40 con 0x564ad6387410 2024-04-25T16:11:48.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.122+0000 7f3cee02e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6731dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.122+0000 7f3cee02e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6731dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48852/0 (socket says 172.21.15.31:48852) 2024-04-25T16:11:48.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.122+0000 7f3cee02e640 1 -- 172.21.15.31:0/1796187894 learned_addr learned my addr 172.21.15.31:0/1796187894 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.122+0000 7f3cee82f640 1 --2- 172.21.15.31:0/1796187894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 0x564ad67237f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.123+0000 7f3cee02e640 1 -- 172.21.15.31:0/1796187894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 msgr2=0x564ad67237f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.123+0000 7f3cee02e640 1 --2- 172.21.15.31:0/1796187894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 0x564ad67237f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.123+0000 7f3cee02e640 1 -- 172.21.15.31:0/1796187894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 msgr2=0x564ad6739850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.123+0000 7f3cee02e640 1 --2- 172.21.15.31:0/1796187894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 0x564ad6739850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.124+0000 7f3cee02e640 1 -- 172.21.15.31:0/1796187894 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ad672d260 con 0x564ad635e550 2024-04-25T16:11:48.142 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.138+0000 7f3cee02e640 1 --2- 172.21.15.31:0/1796187894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6731dc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80a70f0 tx=0x7f3cd80a6070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5627b2813807cee server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.143+0000 7f3cd3fff640 1 -- 172.21.15.31:0/1796187894 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd80af6a0 con 0x564ad635e550 2024-04-25T16:11:48.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.143+0000 7f3cd3fff640 1 -- 172.21.15.31:0/1796187894 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cd80aabf0 con 0x564ad635e550 2024-04-25T16:11:48.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.143+0000 7f3cd3fff640 1 -- 172.21.15.31:0/1796187894 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd808f330 con 0x564ad635e550 2024-04-25T16:11:48.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.145+0000 7f3cf7215740 1 -- 172.21.15.31:0/1796187894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 msgr2=0x564ad6731dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.145+0000 7f3cf7215740 1 --2- 172.21.15.31:0/1796187894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6731dc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3cd80a70f0 tx=0x7f3cd80a6070 comp rx=0 tx=0).stop 2024-04-25T16:11:48.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.148+0000 7f3cf7215740 1 -- 172.21.15.31:0/1796187894 shutdown_connections 2024-04-25T16:11:48.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.148+0000 7f3cf7215740 1 --2- 172.21.15.31:0/1796187894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6387410 0x564ad67237f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.148+0000 7f3cf7215740 1 --2- 172.21.15.31:0/1796187894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6366730 0x564ad6739850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.148+0000 7f3cf7215740 1 --2- 172.21.15.31:0/1796187894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6731dc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.152 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.148+0000 7f3cf7215740 1 -- 172.21.15.31:0/1796187894 >> 172.21.15.31:0/1796187894 conn(0x564ad637b560 msgr2=0x564ad67316b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.166 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.162+0000 7f3cf7215740 1 -- 172.21.15.31:0/1796187894 shutdown_connections 2024-04-25T16:11:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.162+0000 7f3cf7215740 1 -- 172.21.15.31:0/1796187894 wait complete. 2024-04-25T16:11:48.167 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.163+0000 7f3cf7215740 1 -- start start 2024-04-25T16:11:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.164+0000 7f3cf7215740 1 Processor -- start 2024-04-25T16:11:48.168 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.165+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6736b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.169 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.165+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6366730 0x564ad67370a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.166+0000 7f3cf7215740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6387410 0x564ad6737fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.166+0000 7f3cf7215740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc00f7e0 con 0x564ad6387410 2024-04-25T16:11:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.166+0000 7f3cf7215740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc00f110 con 0x564ad635e550 2024-04-25T16:11:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.166+0000 7f3cf7215740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3ccc00c400 con 0x564ad6366730 2024-04-25T16:11:48.170 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.166+0000 7f3cf7215740 10 client.0 _ll_register_callbacks cb 0x7f3cd80a6190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:48.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.167+0000 7f3cee02e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6736b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.168+0000 7f3ced82d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6366730 0x564ad67370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.168+0000 7f3cee02e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6736b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48868/0 (socket says 172.21.15.31:48868) 2024-04-25T16:11:48.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.168+0000 7f3cee02e640 1 -- 172.21.15.31:0/3940753310 learned_addr learned my addr 172.21.15.31:0/3940753310 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.177+0000 7f3cee82f640 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6387410 0x564ad6737fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.182+0000 7f3ced82d640 1 -- 172.21.15.31:0/3940753310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 msgr2=0x564ad6736b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.182+0000 7f3ced82d640 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6736b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.183+0000 7f3ced82d640 1 -- 172.21.15.31:0/3940753310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6387410 msgr2=0x564ad6737fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.183+0000 7f3ced82d640 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6387410 0x564ad6737fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.183+0000 7f3ced82d640 1 -- 172.21.15.31:0/3940753310 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ad6738590 con 0x564ad6366730 2024-04-25T16:11:48.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.190+0000 7f3ced82d640 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6366730 0x564ad67370a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x564ad67337d0 tx=0x7f3ce4009d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c0c6745b46f6d21 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.192+0000 7f3cd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:48.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.193+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/3940753310 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce40140d0 con 0x564ad6366730 2024-04-25T16:11:48.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.193+0000 7f3cf7215740 1 -- 172.21.15.31:0/3940753310 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3cd80aabf0 con 0x564ad6366730 2024-04-25T16:11:48.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.194+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/3940753310 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f3ce4014270 con 0x564ad6366730 2024-04-25T16:11:48.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.196+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/3940753310 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f3ce40035f0 con 0x564ad6366730 2024-04-25T16:11:48.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.200+0000 7f3cf7215740 10 client.27394 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:48.202 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.200+0000 7f3cf7215740 1 -- 172.21.15.31:0/3940753310 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ad6739950 con 0x564ad6366730 2024-04-25T16:11:48.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.212+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/3940753310 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cd80aabf0 con 0x564ad6366730 2024-04-25T16:11:48.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.212+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/3940753310 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3ce4012840 con 0x564ad6366730 2024-04-25T16:11:48.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.219+0000 7f3cf7215740 10 client.27394 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:48.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.219+0000 7f3cf7215740 10 client.27394 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:48.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.219+0000 7f3cf7215740 20 client.27394 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:48.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.219+0000 7f3cf7215740 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564ad631b630 0x564ad67aa050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.219+0000 7f3cf7215740 4 client.27394 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:11:48.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.219+0000 7f3cf7215740 1 -- 172.21.15.31:0/3940753310 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564ad6695580 con 0x564ad631b630 2024-04-25T16:11:48.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.220+0000 7f3cee02e640 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564ad631b630 0x564ad67aa050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.222+0000 7f3cee02e640 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564ad631b630 0x564ad67aa050 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc27c93b5f7cff9a server_cookie=99220b3d2f9a20e6 in_seq=0 out_seq=0 2024-04-25T16:11:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.228+0000 7f3cd1ffb640 10 client.27394 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.228+0000 7f3cd1ffb640 1 -- 172.21.15.31:0/3940753310 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564ad6695580 con 0x564ad631b630 2024-04-25T16:11:48.234 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.228+0000 7f3cd1ffb640 10 client.27394 handle_command_reply: tid=0 2024-04-25T16:11:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cf7215740 1 client.27394 shutdown 2024-04-25T16:11:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cf7215740 1 -- 172.21.15.31:0/3940753310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6366730 msgr2=0x564ad67370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cf7215740 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6366730 0x564ad67370a0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x564ad67337d0 tx=0x7f3ce4009d40 comp rx=0 tx=0).stop 2024-04-25T16:11:48.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cf7215740 1 -- 172.21.15.31:0/3940753310 shutdown_connections 2024-04-25T16:11:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cf7215740 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x564ad631b630 0x564ad67aa050 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cf7215740 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x564ad6387410 0x564ad6737fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cf7215740 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564ad6366730 0x564ad67370a0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cf7215740 1 --2- 172.21.15.31:0/3940753310 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x564ad635e550 0x564ad6736b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cd1ffb640 0 client.27394 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.232+0000 7f3cf7215740 1 -- 172.21.15.31:0/3940753310 >> 172.21.15.31:0/3940753310 conn(0x564ad637b560 msgr2=0x564ad6726c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.233+0000 7f3cf7215740 1 -- 172.21.15.31:0/3940753310 shutdown_connections 2024-04-25T16:11:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.233+0000 7f3cf7215740 1 -- 172.21.15.31:0/3940753310 wait complete. 2024-04-25T16:11:48.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.233+0000 7f3cf7215740 20 client.27394 trim_cache size 0 max 16384 2024-04-25T16:11:48.237 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:11:48.237 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:11:48.237 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "aa8006c0-bbf7-4e98-9077-9884603cb246", 2024-04-25T16:11:48.237 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:11:48.237 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:11:48.261 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.259+0000 7f3cf0292640 1 -- 172.21.15.31:0/4201593808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3cd40afbf0 msgr2=0x7f3cd40b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.259+0000 7f3cf0292640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3cd40afbf0 0x7f3cd40b2100 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3ce002ae10 tx=0x7f3ce0005fd0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.259+0000 7f3cf0292640 1 -- 172.21.15.31:0/4201593808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce81029e0 msgr2=0x7f3ce8071510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.262 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.259+0000 7f3cf0292640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce81029e0 0x7f3ce8071510 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3ce81022b0 tx=0x7f3cd800b850 comp rx=0 tx=0).stop 2024-04-25T16:11:48.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.260+0000 7f3cf0292640 1 -- 172.21.15.31:0/4201593808 shutdown_connections 2024-04-25T16:11:48.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.260+0000 7f3cf0292640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f3cd40afbf0 0x7f3cd40b2100 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.260+0000 7f3cf0292640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f3ce8108920 0x7f3ce81b0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.260+0000 7f3cf0292640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3ce8103c20 0x7f3ce8071a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.260+0000 7f3cf0292640 1 --2- 172.21.15.31:0/4201593808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ce81029e0 0x7f3ce8071510 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.260+0000 7f3cf0292640 1 -- 172.21.15.31:0/4201593808 >> 172.21.15.31:0/4201593808 conn(0x7f3ce80fdbd0 msgr2=0x7f3ce80ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.260+0000 7f3cf0292640 1 -- 172.21.15.31:0/4201593808 shutdown_connections 2024-04-25T16:11:48.263 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.260+0000 7f3cf0292640 1 -- 172.21.15.31:0/4201593808 wait complete. 2024-04-25T16:11:48.332 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:11:48.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:11:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.417+0000 7f73e2622640 1 -- 172.21.15.31:0/2831206732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc10b190 msgr2=0x7f73dc10b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.417+0000 7f73e2622640 1 --2- 172.21.15.31:0/2831206732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc10b190 0x7f73dc10b5a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009900 tx=0x7f73d802a420 comp rx=0 tx=0).stop 2024-04-25T16:11:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.424+0000 7f73e2622640 1 -- 172.21.15.31:0/2831206732 shutdown_connections 2024-04-25T16:11:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.424+0000 7f73e2622640 1 --2- 172.21.15.31:0/2831206732 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73dc10feb0 0x7f73dc1102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.424+0000 7f73e2622640 1 --2- 172.21.15.31:0/2831206732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc10b190 0x7f73dc10b5a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.424+0000 7f73e2622640 1 --2- 172.21.15.31:0/2831206732 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73dc10a340 0x7f73dc10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.427 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.424+0000 7f73e2622640 1 -- 172.21.15.31:0/2831206732 >> 172.21.15.31:0/2831206732 conn(0x7f73dc070330 msgr2=0x7f73dc0727e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.429+0000 7f73e2622640 1 -- 172.21.15.31:0/2831206732 shutdown_connections 2024-04-25T16:11:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.429+0000 7f73e2622640 1 -- 172.21.15.31:0/2831206732 wait complete. 2024-04-25T16:11:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.430+0000 7f73e2622640 1 Processor -- start 2024-04-25T16:11:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.430+0000 7f73e2622640 1 -- start start 2024-04-25T16:11:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.430+0000 7f73e2622640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73dc10a340 0x7f73dc1a8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.430+0000 7f73e2622640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc10b190 0x7f73dc1a9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e2622640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73dc10feb0 0x7f73dc07ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e2622640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc11e390 con 0x7f73dc10a340 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e2622640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc11e210 con 0x7f73dc10feb0 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e2622640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73dc11e510 con 0x7f73dc10b190 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e1e21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73dc10feb0 0x7f73dc07ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e1e21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73dc10feb0 0x7f73dc07ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48886/0 (socket says 172.21.15.31:48886) 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e0e1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc10b190 0x7f73dc1a9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e1e21640 1 -- 172.21.15.31:0/3959792204 learned_addr learned my addr 172.21.15.31:0/3959792204 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e1e21640 1 -- 172.21.15.31:0/3959792204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc10b190 msgr2=0x7f73dc1a9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e1e21640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc10b190 0x7f73dc1a9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e1e21640 1 -- 172.21.15.31:0/3959792204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73dc10a340 msgr2=0x7f73dc1a8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e1e21640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73dc10a340 0x7f73dc1a8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e1e21640 1 -- 172.21.15.31:0/3959792204 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73dc07b470 con 0x7f73dc10feb0 2024-04-25T16:11:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.431+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc10b190 0x7f73dc1a9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:11:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.432+0000 7f73e1e21640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73dc10feb0 0x7f73dc07ae00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f73d400b990 tx=0x7f73d400be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.434+0000 7f73d27fc640 1 -- 172.21.15.31:0/3959792204 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4019070 con 0x7f73dc10feb0 2024-04-25T16:11:48.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.434+0000 7f73e2622640 1 -- 172.21.15.31:0/3959792204 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73dc07b760 con 0x7f73dc10feb0 2024-04-25T16:11:48.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.434+0000 7f73e2622640 1 -- 172.21.15.31:0/3959792204 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73dc07bd00 con 0x7f73dc10feb0 2024-04-25T16:11:48.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.434+0000 7f73d27fc640 1 -- 172.21.15.31:0/3959792204 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73d400cba0 con 0x7f73dc10feb0 2024-04-25T16:11:48.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.436+0000 7f73d27fc640 1 -- 172.21.15.31:0/3959792204 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d40075d0 con 0x7f73dc10feb0 2024-04-25T16:11:48.443 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.441+0000 7f73d27fc640 1 -- 172.21.15.31:0/3959792204 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f73d401c070 con 0x7f73dc10feb0 2024-04-25T16:11:48.451 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.443+0000 7f73ea80b740 1 Processor -- start 2024-04-25T16:11:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.444+0000 7f73ea80b740 1 -- start start 2024-04-25T16:11:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.449+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d311b0 0x5565620e2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.449+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620e30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.449+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d104d0 0x5565620e9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.449+0000 7f73ea80b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x556562037460 con 0x556561d104d0 2024-04-25T16:11:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.449+0000 7f73ea80b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556561ff1590 con 0x556561d082f0 2024-04-25T16:11:48.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.449+0000 7f73ea80b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x556562033420 con 0x556561d311b0 2024-04-25T16:11:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.451+0000 7f73e1620640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620e30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.451+0000 7f73e1620640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620e30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48898/0 (socket says 172.21.15.31:48898) 2024-04-25T16:11:48.453 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.451+0000 7f73e1620640 1 -- 172.21.15.31:0/917587637 learned_addr learned my addr 172.21.15.31:0/917587637 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.457 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.452+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/917587637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d104d0 0x5565620e9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.455+0000 7f73e1620640 1 -- 172.21.15.31:0/917587637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d311b0 msgr2=0x5565620e2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.455+0000 7f73e1620640 1 --2- 172.21.15.31:0/917587637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d311b0 0x5565620e2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.455+0000 7f73e1620640 1 -- 172.21.15.31:0/917587637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d104d0 msgr2=0x5565620e9fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.455+0000 7f73e1620640 1 --2- 172.21.15.31:0/917587637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d104d0 0x5565620e9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.455+0000 7f73e1620640 1 -- 172.21.15.31:0/917587637 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556561ffe0d0 con 0x556561d082f0 2024-04-25T16:11:48.462 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.460+0000 7f73e1620640 1 --2- 172.21.15.31:0/917587637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620e30a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0097f0 tx=0x7f73cc009cb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3905045c0caf1461 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.461+0000 7f73d27fc640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73c80afbd0 0x7f73c80b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.466 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.462+0000 7f73d27fc640 1 -- 172.21.15.31:0/3959792204 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f73d40de0a0 con 0x7f73dc10feb0 2024-04-25T16:11:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.470+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73c80afbd0 0x7f73c80b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.474+0000 7f73c77fe640 1 -- 172.21.15.31:0/917587637 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73cc015020 con 0x556561d082f0 2024-04-25T16:11:48.477 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.474+0000 7f73c77fe640 1 -- 172.21.15.31:0/917587637 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73cc011420 con 0x556561d082f0 2024-04-25T16:11:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.476+0000 7f73ea80b740 1 -- 172.21.15.31:0/917587637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 msgr2=0x5565620e30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.476+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73c80afbd0 0x7f73c80b20e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f73d8031500 tx=0x7f73d8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.478 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.476+0000 7f73ea80b740 1 --2- 172.21.15.31:0/917587637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620e30a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f73cc0097f0 tx=0x7f73cc009cb0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.482+0000 7f73ea80b740 1 -- 172.21.15.31:0/917587637 shutdown_connections 2024-04-25T16:11:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.482+0000 7f73ea80b740 1 --2- 172.21.15.31:0/917587637 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d104d0 0x5565620e9fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.482+0000 7f73ea80b740 1 --2- 172.21.15.31:0/917587637 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620e30a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.482+0000 7f73ea80b740 1 --2- 172.21.15.31:0/917587637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d311b0 0x5565620e2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.482+0000 7f73ea80b740 1 -- 172.21.15.31:0/917587637 >> 172.21.15.31:0/917587637 conn(0x556561d25300 msgr2=0x5565620cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.486+0000 7f73ea80b740 1 -- 172.21.15.31:0/917587637 shutdown_connections 2024-04-25T16:11:48.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.490+0000 7f73ea80b740 1 -- 172.21.15.31:0/917587637 wait complete. 2024-04-25T16:11:48.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.491+0000 7f73ea80b740 1 -- start start 2024-04-25T16:11:48.494 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.491+0000 7f73ea80b740 1 Processor -- start 2024-04-25T16:11:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d082f0 0x5565620ec800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d104d0 0x5565620ecd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d311b0 0x5565620f3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73ea80b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5565620eaa40 con 0x556561d311b0 2024-04-25T16:11:48.495 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73ea80b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x556561fbeb60 con 0x556561d104d0 2024-04-25T16:11:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73ea80b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5565620eabc0 con 0x556561d082f0 2024-04-25T16:11:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73ea80b740 10 client.0 _ll_register_callbacks cb 0x556561fe32b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73e0e1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d104d0 0x5565620ecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73e0e1f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d104d0 0x5565620ecd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48914/0 (socket says 172.21.15.31:48914) 2024-04-25T16:11:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.493+0000 7f73e0e1f640 1 -- 172.21.15.31:0/3358518959 learned_addr learned my addr 172.21.15.31:0/3358518959 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.496 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.494+0000 7f73e0e1f640 1 -- 172.21.15.31:0/3358518959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d082f0 msgr2=0x5565620ec800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.494+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d082f0 0x5565620ec800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.494+0000 7f73e0e1f640 1 -- 172.21.15.31:0/3358518959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d311b0 msgr2=0x5565620f3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.494+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d311b0 0x5565620f3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.497 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.494+0000 7f73e0e1f640 1 -- 172.21.15.31:0/3358518959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556561d49210 con 0x556561d104d0 2024-04-25T16:11:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.494+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d104d0 0x5565620ecd40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f73d80036a0 tx=0x7f73d8009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adb405c1d4feaf72 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.495+0000 7f73c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.495+0000 7f73c4ff9640 1 -- 172.21.15.31:0/3358518959 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d804d070 con 0x556561d104d0 2024-04-25T16:11:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.495+0000 7f73c4ff9640 1 -- 172.21.15.31:0/3358518959 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f73d8049450 con 0x556561d104d0 2024-04-25T16:11:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.495+0000 7f73ea80b740 1 -- 172.21.15.31:0/3358518959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73cc011420 con 0x556561d104d0 2024-04-25T16:11:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.495+0000 7f73c4ff9640 1 -- 172.21.15.31:0/3358518959 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f73d8057050 con 0x556561d104d0 2024-04-25T16:11:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.495+0000 7f73ea80b740 10 client.27407 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.495+0000 7f73ea80b740 1 -- 172.21.15.31:0/3358518959 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556561e15620 con 0x556561d104d0 2024-04-25T16:11:48.498 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.496+0000 7f73c4ff9640 1 -- 172.21.15.31:0/3358518959 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73cc011420 con 0x556561d104d0 2024-04-25T16:11:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.496+0000 7f73c4ff9640 1 -- 172.21.15.31:0/3358518959 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f73d8052600 con 0x556561d104d0 2024-04-25T16:11:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.496+0000 7f73ea80b740 10 client.27407 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.496+0000 7f73ea80b740 10 client.27407 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.496+0000 7f73ea80b740 20 client.27407 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.496+0000 7f73ea80b740 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556561cefd80 0x5565620fa730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.496+0000 7f73ea80b740 4 client.27407 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:48.499 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.496+0000 7f73ea80b740 1 -- 172.21.15.31:0/3358518959 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556562045070 con 0x556561cefd80 2024-04-25T16:11:48.500 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.497+0000 7f73e1620640 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556561cefd80 0x5565620fa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.498+0000 7f73c4ff9640 10 client.27407 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:48.501 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.498+0000 7f73e1620640 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556561cefd80 0x5565620fa730 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5dbfc443badeee6b server_cookie=cc977bf9a83de4f8 in_seq=0 out_seq=0 2024-04-25T16:11:48.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.501+0000 7f73c4ff9640 1 -- 172.21.15.31:0/3358518959 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556562045070 con 0x556561cefd80 2024-04-25T16:11:48.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.501+0000 7f73c4ff9640 10 client.27407 handle_command_reply: tid=0 2024-04-25T16:11:48.503 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.501+0000 7f73ea80b740 1 client.27407 shutdown 2024-04-25T16:11:48.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.502+0000 7f73ea80b740 1 -- 172.21.15.31:0/3358518959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d104d0 msgr2=0x5565620ecd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.504 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.502+0000 7f73ea80b740 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d104d0 0x5565620ecd40 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f73d80036a0 tx=0x7f73d8009260 comp rx=0 tx=0).stop 2024-04-25T16:11:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.505+0000 7f73ea80b740 1 -- 172.21.15.31:0/3358518959 shutdown_connections 2024-04-25T16:11:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.505+0000 7f73ea80b740 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556561cefd80 0x5565620fa730 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.505+0000 7f73ea80b740 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d311b0 0x5565620f3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.505+0000 7f73ea80b740 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d104d0 0x5565620ecd40 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.505+0000 7f73ea80b740 1 --2- 172.21.15.31:0/3358518959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d082f0 0x5565620ec800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.505+0000 7f73c4ff9640 0 client.27407 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:48.507 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.505+0000 7f73ea80b740 1 -- 172.21.15.31:0/3358518959 >> 172.21.15.31:0/3358518959 conn(0x556561d25300 msgr2=0x5565620d03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.508 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.505+0000 7f73ea80b740 1 -- 172.21.15.31:0/3358518959 shutdown_connections 2024-04-25T16:11:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.506+0000 7f73ea80b740 1 -- 172.21.15.31:0/3358518959 wait complete. 2024-04-25T16:11:48.517 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.510+0000 7f73ea80b740 20 client.27407 trim_cache size 0 max 16384 2024-04-25T16:11:48.573 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.571+0000 7f73ea80b740 1 Processor -- start 2024-04-25T16:11:48.585 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.583+0000 7f73ea80b740 1 -- start start 2024-04-25T16:11:48.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.584+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d082f0 0x5565620ecae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.587 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.585+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.585+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d311b0 0x5565620cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.585+0000 7f73ea80b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73c000e540 con 0x556561d082f0 2024-04-25T16:11:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.585+0000 7f73ea80b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73c0010080 con 0x556561d311b0 2024-04-25T16:11:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.585+0000 7f73e0e1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.585+0000 7f73e0e1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:42296/0 (socket says 172.21.15.31:42296) 2024-04-25T16:11:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.585+0000 7f73e0e1f640 1 -- 172.21.15.31:0/2294543362 learned_addr learned my addr 172.21.15.31:0/2294543362 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.588 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.585+0000 7f73ea80b740 1 -- 172.21.15.31:0/2294543362 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73c000ec10 con 0x556561d104d0 2024-04-25T16:11:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.586+0000 7f73e0e1f640 1 -- 172.21.15.31:0/2294543362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d311b0 msgr2=0x5565620cbad0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.586+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/2294543362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d311b0 0x5565620cbad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.586+0000 7f73e0e1f640 1 -- 172.21.15.31:0/2294543362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d082f0 msgr2=0x5565620ecae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.586+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/2294543362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d082f0 0x5565620ecae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.586+0000 7f73e0e1f640 1 -- 172.21.15.31:0/2294543362 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5565620f4100 con 0x556561d104d0 2024-04-25T16:11:48.589 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.587+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/2294543362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f3bc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f73d8052a20 tx=0x7f73d80038e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d041bdf75739f627 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.589+0000 7f73c6ffd640 1 -- 172.21.15.31:0/2294543362 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d804d070 con 0x556561d104d0 2024-04-25T16:11:48.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.589+0000 7f73c6ffd640 1 -- 172.21.15.31:0/2294543362 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73d8049450 con 0x556561d104d0 2024-04-25T16:11:48.591 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.589+0000 7f73c6ffd640 1 -- 172.21.15.31:0/2294543362 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d8035050 con 0x556561d104d0 2024-04-25T16:11:48.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.590+0000 7f73ea80b740 1 -- 172.21.15.31:0/2294543362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 msgr2=0x5565620f3bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.593 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.590+0000 7f73ea80b740 1 --2- 172.21.15.31:0/2294543362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f3bc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f73d8052a20 tx=0x7f73d80038e0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.593+0000 7f73ea80b740 1 -- 172.21.15.31:0/2294543362 shutdown_connections 2024-04-25T16:11:48.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.593+0000 7f73ea80b740 1 --2- 172.21.15.31:0/2294543362 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d311b0 0x5565620cbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.593+0000 7f73ea80b740 1 --2- 172.21.15.31:0/2294543362 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f3bc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.593+0000 7f73ea80b740 1 --2- 172.21.15.31:0/2294543362 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d082f0 0x5565620ecae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.596 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.593+0000 7f73ea80b740 1 -- 172.21.15.31:0/2294543362 >> 172.21.15.31:0/2294543362 conn(0x556561d25300 msgr2=0x5565620ec3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.607+0000 7f73ea80b740 1 -- 172.21.15.31:0/2294543362 shutdown_connections 2024-04-25T16:11:48.609 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.607+0000 7f73ea80b740 1 -- 172.21.15.31:0/2294543362 wait complete. 2024-04-25T16:11:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.608+0000 7f73ea80b740 1 -- start start 2024-04-25T16:11:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.608+0000 7f73ea80b740 1 Processor -- start 2024-04-25T16:11:48.611 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.609+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620f12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.609+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.609+0000 7f73ea80b740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d311b0 0x5565620f26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.609+0000 7f73ea80b740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f73c000f9b0 con 0x556561d311b0 2024-04-25T16:11:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.609+0000 7f73ea80b740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f73c000f2e0 con 0x556561d082f0 2024-04-25T16:11:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.609+0000 7f73ea80b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73c000c520 con 0x556561d104d0 2024-04-25T16:11:48.612 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.609+0000 7f73ea80b740 10 client.0 _ll_register_callbacks cb 0x7f73d80035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.610+0000 7f73e1620640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620f12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.610+0000 7f73e1620640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620f12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48938/0 (socket says 172.21.15.31:48938) 2024-04-25T16:11:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.610+0000 7f73e1620640 1 -- 172.21.15.31:0/1648190640 learned_addr learned my addr 172.21.15.31:0/1648190640 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.610+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.611+0000 7f73e0e1f640 1 -- 172.21.15.31:0/1648190640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 msgr2=0x5565620f12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.613 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.611+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620f12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.611+0000 7f73e0e1f640 1 -- 172.21.15.31:0/1648190640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d311b0 msgr2=0x5565620f26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.611+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d311b0 0x5565620f26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.611+0000 7f73e0e1f640 1 -- 172.21.15.31:0/1648190640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5565620f2c80 con 0x556561d104d0 2024-04-25T16:11:48.614 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.612+0000 7f73e0e1f640 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f1800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f73d80036a0 tx=0x7f73d8005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aea9fad878820c6f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.622+0000 7f73c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:48.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.622+0000 7f73c4ff9640 1 -- 172.21.15.31:0/1648190640 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d8069070 con 0x556561d104d0 2024-04-25T16:11:48.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.622+0000 7f73ea80b740 1 -- 172.21.15.31:0/1648190640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f73d8049450 con 0x556561d104d0 2024-04-25T16:11:48.631 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.622+0000 7f73c4ff9640 1 -- 172.21.15.31:0/1648190640 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f73d8049a10 con 0x556561d104d0 2024-04-25T16:11:48.647 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.645+0000 7f73c4ff9640 1 -- 172.21.15.31:0/1648190640 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f73d8073050 con 0x556561d104d0 2024-04-25T16:11:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.648+0000 7f73ea80b740 10 client.27406 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:48.650 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.648+0000 7f73ea80b740 1 -- 172.21.15.31:0/1648190640 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5565620f3f10 con 0x556561d104d0 2024-04-25T16:11:48.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.650+0000 7f73c4ff9640 1 -- 172.21.15.31:0/1648190640 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f73d8049450 con 0x556561d104d0 2024-04-25T16:11:48.652 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.650+0000 7f73c4ff9640 1 -- 172.21.15.31:0/1648190640 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f73d806e550 con 0x556561d104d0 2024-04-25T16:11:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.650+0000 7f73ea80b740 10 client.27406 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.650+0000 7f73ea80b740 10 client.27406 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.650+0000 7f73ea80b740 20 client.27406 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.650+0000 7f73ea80b740 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556561cc51c0 0x5565621652d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.650+0000 7f73ea80b740 4 client.27406 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:11:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.650+0000 7f73ea80b740 1 -- 172.21.15.31:0/1648190640 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55656203eee0 con 0x556561cc51c0 2024-04-25T16:11:48.653 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.651+0000 7f73e1620640 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556561cc51c0 0x5565621652d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.652+0000 7f73c4ff9640 10 client.27406 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:48.655 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:48.652+0000 7f73e1620640 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556561cc51c0 0x5565621652d0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=211fead8e867515b server_cookie=613bca9133f26ce5 in_seq=0 out_seq=0 2024-04-25T16:11:48.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.851+0000 7f39a9d72640 1 -- 172.21.15.31:0/503553702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4104bc0 msgr2=0x7f39a4106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.854 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.851+0000 7f39a9d72640 1 --2- 172.21.15.31:0/503553702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4104bc0 0x7f39a4106fe0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f399c009ca0 tx=0x7f399c02a8d0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.859+0000 7f39a9d72640 1 -- 172.21.15.31:0/503553702 shutdown_connections 2024-04-25T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.859+0000 7f39a9d72640 1 --2- 172.21.15.31:0/503553702 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4107520 0x7f39a4107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.859+0000 7f39a9d72640 1 --2- 172.21.15.31:0/503553702 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4104bc0 0x7f39a4106fe0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.859+0000 7f39a9d72640 1 --2- 172.21.15.31:0/503553702 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a4102260 0x7f39a4104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.862 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.859+0000 7f39a9d72640 1 -- 172.21.15.31:0/503553702 >> 172.21.15.31:0/503553702 conn(0x7f39a40fda70 msgr2=0x7f39a40fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.865 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.863+0000 7f39a9d72640 1 -- 172.21.15.31:0/503553702 shutdown_connections 2024-04-25T16:11:48.869 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.867+0000 7f39a9d72640 1 -- 172.21.15.31:0/503553702 wait complete. 2024-04-25T16:11:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a9d72640 1 Processor -- start 2024-04-25T16:11:48.871 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a9d72640 1 -- start start 2024-04-25T16:11:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a9d72640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4102260 0x7f39a4110800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a9d72640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4104bc0 0x7f39a4110d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a9d72640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a4107520 0x7f39a4111c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a9d72640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f39a4108890 con 0x7f39a4107520 2024-04-25T16:11:48.872 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a9d72640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f39a4108710 con 0x7f39a4104bc0 2024-04-25T16:11:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a9d72640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f39a4108a10 con 0x7f39a4102260 2024-04-25T16:11:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4102260 0x7f39a4110800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4102260 0x7f39a4110800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:42316/0 (socket says 172.21.15.31:42316) 2024-04-25T16:11:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.870+0000 7f39a37fe640 1 -- 172.21.15.31:0/3096164298 learned_addr learned my addr 172.21.15.31:0/3096164298 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.873 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.871+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4104bc0 0x7f39a4110d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.871+0000 7f39a37fe640 1 -- 172.21.15.31:0/3096164298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4104bc0 msgr2=0x7f39a4110d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.871+0000 7f39a37fe640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4104bc0 0x7f39a4110d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.871+0000 7f39a37fe640 1 -- 172.21.15.31:0/3096164298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a4107520 msgr2=0x7f39a4111c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.871+0000 7f39a37fe640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a4107520 0x7f39a4111c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.874 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.871+0000 7f39a37fe640 1 -- 172.21.15.31:0/3096164298 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39a41122a0 con 0x7f39a4102260 2024-04-25T16:11:48.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.880+0000 7f39a37fe640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4102260 0x7f39a4110800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f399400b450 tx=0x7f399400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.882+0000 7f39a0ff9640 1 -- 172.21.15.31:0/3096164298 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3994015020 con 0x7f39a4102260 2024-04-25T16:11:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.882+0000 7f39a0ff9640 1 -- 172.21.15.31:0/3096164298 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3994011d40 con 0x7f39a4102260 2024-04-25T16:11:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.882+0000 7f39a9d72640 1 -- 172.21.15.31:0/3096164298 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39a4112590 con 0x7f39a4102260 2024-04-25T16:11:48.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.882+0000 7f39a9d72640 1 -- 172.21.15.31:0/3096164298 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39a41b1400 con 0x7f39a4102260 2024-04-25T16:11:48.886 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.884+0000 7f39a0ff9640 1 -- 172.21.15.31:0/3096164298 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399400f340 con 0x7f39a4102260 2024-04-25T16:11:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.897+0000 7f39a0ff9640 1 -- 172.21.15.31:0/3096164298 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3994018020 con 0x7f39a4102260 2024-04-25T16:11:48.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.897+0000 7f39b0c27740 1 Processor -- start 2024-04-25T16:11:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.897+0000 7f39b0c27740 1 -- start start 2024-04-25T16:11:48.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.897+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80b4c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.897+0000 7f39a0ff9640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f398c0afd00 0x7f398c0b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 0x5590e80b51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39b0c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5590e8019d80 con 0x5590e7cf26b0 2024-04-25T16:11:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39b0c27740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5590e7fd38b0 con 0x5590e7d13390 2024-04-25T16:11:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39b0c27740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5590e8015d40 con 0x5590e7cea4d0 2024-04-25T16:11:48.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39a0ff9640 1 -- 172.21.15.31:0/3096164298 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f39940df5a0 con 0x7f39a4102260 2024-04-25T16:11:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39a2ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 0x5590e80b51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39a37fe640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f398c0afd00 0x7f398c0b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39a3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80b4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39a3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80b4c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:48976/0 (socket says 172.21.15.31:48976) 2024-04-25T16:11:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39a3fff640 1 -- 172.21.15.31:0/452835488 learned_addr learned my addr 172.21.15.31:0/452835488 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.898+0000 7f39a3fff640 1 --2- 172.21.15.31:0/452835488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.899+0000 7f39a3fff640 1 -- 172.21.15.31:0/452835488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 msgr2=0x5590e80b51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.899+0000 7f39a3fff640 1 --2- 172.21.15.31:0/452835488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 0x5590e80b51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.899+0000 7f39a3fff640 1 -- 172.21.15.31:0/452835488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 msgr2=0x5590e80b4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.899+0000 7f39a3fff640 1 --2- 172.21.15.31:0/452835488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80b4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.899+0000 7f39a3fff640 1 -- 172.21.15.31:0/452835488 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590e7fa4a40 con 0x5590e7cf26b0 2024-04-25T16:11:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.899+0000 7f39a37fe640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f398c0afd00 0x7f398c0b2210 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3994011040 tx=0x7f39940046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.900+0000 7f39a3fff640 1 --2- 172.21.15.31:0/452835488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80b5e60 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3998009fc0 tx=0x7f3998005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f112cad34d3ca50e server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.900+0000 7f3989ffb640 1 -- 172.21.15.31:0/452835488 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3998004030 con 0x5590e7cf26b0 2024-04-25T16:11:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.900+0000 7f3989ffb640 1 -- 172.21.15.31:0/452835488 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f39980035f0 con 0x5590e7cf26b0 2024-04-25T16:11:48.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.900+0000 7f3989ffb640 1 -- 172.21.15.31:0/452835488 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39980116a0 con 0x5590e7cf26b0 2024-04-25T16:11:48.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.902+0000 7f39b0c27740 1 -- 172.21.15.31:0/452835488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 msgr2=0x5590e80b5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:48.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.902+0000 7f39b0c27740 1 --2- 172.21.15.31:0/452835488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80b5e60 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f3998009fc0 tx=0x7f3998005850 comp rx=0 tx=0).stop 2024-04-25T16:11:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.903+0000 7f39b0c27740 1 -- 172.21.15.31:0/452835488 shutdown_connections 2024-04-25T16:11:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.903+0000 7f39b0c27740 1 --2- 172.21.15.31:0/452835488 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80b5e60 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.903+0000 7f39b0c27740 1 --2- 172.21.15.31:0/452835488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 0x5590e80b51d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.903+0000 7f39b0c27740 1 --2- 172.21.15.31:0/452835488 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80b4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.903+0000 7f39b0c27740 1 -- 172.21.15.31:0/452835488 >> 172.21.15.31:0/452835488 conn(0x5590e7d074e0 msgr2=0x5590e80b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.903+0000 7f39b0c27740 1 -- 172.21.15.31:0/452835488 shutdown_connections 2024-04-25T16:11:48.905 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.903+0000 7f39b0c27740 1 -- 172.21.15.31:0/452835488 wait complete. 2024-04-25T16:11:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.903+0000 7f39b0c27740 1 -- start start 2024-04-25T16:11:48.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.904+0000 7f39b0c27740 1 Processor -- start 2024-04-25T16:11:48.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cea4d0 0x5590e80beaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80befe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39a3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cea4d0 0x5590e80beaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39b0c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5590e80b6970 con 0x5590e7cea4d0 2024-04-25T16:11:48.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39a3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cea4d0 0x5590e80beaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39516/0 (socket says 172.21.15.31:39516) 2024-04-25T16:11:48.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39a3fff640 1 -- 172.21.15.31:0/2744383549 learned_addr learned my addr 172.21.15.31:0/2744383549 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:48.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5590e7fa0d10 con 0x5590e7d13390 2024-04-25T16:11:48.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5590e80b6af0 con 0x5590e7cf26b0 2024-04-25T16:11:48.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39b0c27740 10 client.0 _ll_register_callbacks cb 0x5590e7fcd210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:48.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.906+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80befe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.908+0000 7f39a2ffd640 1 -- 172.21.15.31:0/2744383549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 msgr2=0x5590e80c5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.908+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.908+0000 7f39a2ffd640 1 -- 172.21.15.31:0/2744383549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cea4d0 msgr2=0x5590e80beaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:48.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.908+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cea4d0 0x5590e80beaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:48.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.908+0000 7f39a2ffd640 1 -- 172.21.15.31:0/2744383549 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590e7df57d0 con 0x5590e7cf26b0 2024-04-25T16:11:48.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.908+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80befe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f399c0319e0 tx=0x7f399c031460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaec1fbf3c83ad51 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:48.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.910+0000 7f3982ffd640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:11:48.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.910+0000 7f3982ffd640 1 -- 172.21.15.31:0/2744383549 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c003b70 con 0x5590e7cf26b0 2024-04-25T16:11:48.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.911+0000 7f3982ffd640 1 -- 172.21.15.31:0/2744383549 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f399c003d10 con 0x5590e7cf26b0 2024-04-25T16:11:48.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.911+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f39980035f0 con 0x5590e7cf26b0 2024-04-25T16:11:48.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.911+0000 7f3982ffd640 1 -- 172.21.15.31:0/2744383549 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f399c004040 con 0x5590e7cf26b0 2024-04-25T16:11:48.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.916+0000 7f39b0c27740 10 client.27415 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:48.917 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.916+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5590e7d2b3f0 con 0x5590e7cf26b0 2024-04-25T16:11:48.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.917+0000 7f3982ffd640 1 -- 172.21.15.31:0/2744383549 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f399c036040 con 0x5590e7cf26b0 2024-04-25T16:11:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.917+0000 7f3982ffd640 1 -- 172.21.15.31:0/2744383549 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f399c030a60 con 0x5590e7cf26b0 2024-04-25T16:11:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.918+0000 7f39b0c27740 10 client.27415 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.918+0000 7f39b0c27740 10 client.27415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.918+0000 7f39b0c27740 20 client.27415 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.918+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5590e7cd1f60 0x5590e80cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.918+0000 7f39b0c27740 4 client.27415 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:11:48.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.918+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f39980035f0 con 0x5590e7cd1f60 2024-04-25T16:11:48.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.928+0000 7f39a3fff640 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5590e7cd1f60 0x5590e80cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:48.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.955+0000 7f39a3fff640 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5590e7cd1f60 0x5590e80cc4b0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d78be302e39d9574 server_cookie=52cf657790e0d695 in_seq=0 out_seq=0 2024-04-25T16:11:48.957 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:48.956+0000 7f3982ffd640 10 client.27415 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:49 smithi063 ceph-mon[32413]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 914 MiB/s wr, 2.49k op/s 2024-04-25T16:11:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:49 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:11:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:49 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:49 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:11:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:49 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:11:49.621 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:49 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:49 smithi186 ceph-mon[33258]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 914 MiB/s wr, 2.49k op/s 2024-04-25T16:11:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:49 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:11:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:49 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:49 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:11:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:49 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:11:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:49 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:49 smithi031 ceph-mon[29140]: pgmap v666: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 914 MiB/s wr, 2.49k op/s 2024-04-25T16:11:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:49 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:11:49.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:49 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:49.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:49 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:11:49.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:49 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:11:49.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:49 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:50.481 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.479+0000 7f73c4ff9640 1 -- 172.21.15.31:0/1648190640 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55656203eee0 con 0x556561cc51c0 2024-04-25T16:11:50.697 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.479+0000 7f73c4ff9640 10 client.27406 handle_command_reply: tid=0 2024-04-25T16:11:50.699 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:50 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:50.699 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:50 smithi186 ceph-mon[33258]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 744 MiB/s wr, 1.91k op/s 2024-04-25T16:11:50.699 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:50 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:50.700 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:50 smithi063 ceph-mon[32413]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 744 MiB/s wr, 1.91k op/s 2024-04-25T16:11:50.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.479+0000 7f3982ffd640 1 -- 172.21.15.31:0/2744383549 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f39980035f0 con 0x5590e7cd1f60 2024-04-25T16:11:50.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.479+0000 7f3982ffd640 10 client.27415 handle_command_reply: tid=0 2024-04-25T16:11:50.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.479+0000 7f39b0c27740 1 client.27415 shutdown 2024-04-25T16:11:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.479+0000 7f73ea80b740 1 client.27406 shutdown 2024-04-25T16:11:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.481+0000 7f73ea80b740 1 -- 172.21.15.31:0/1648190640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 msgr2=0x5565620f1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.481+0000 7f73ea80b740 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f1800 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f73d80036a0 tx=0x7f73d8005970 comp rx=0 tx=0).stop 2024-04-25T16:11:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.482+0000 7f73ea80b740 1 -- 172.21.15.31:0/1648190640 shutdown_connections 2024-04-25T16:11:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.482+0000 7f73ea80b740 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x556561cc51c0 0x5565621652d0 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.482+0000 7f73ea80b740 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x556561d311b0 0x5565620f26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.482+0000 7f73ea80b740 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556561d104d0 0x5565620f1800 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.482+0000 7f73ea80b740 1 --2- 172.21.15.31:0/1648190640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x556561d082f0 0x5565620f12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.482+0000 7f73ea80b740 1 -- 172.21.15.31:0/1648190640 >> 172.21.15.31:0/1648190640 conn(0x556561d25300 msgr2=0x5565620ecb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.482+0000 7f73c4ff9640 0 client.27406 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.483+0000 7f73ea80b740 1 -- 172.21.15.31:0/1648190640 shutdown_connections 2024-04-25T16:11:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.483+0000 7f73ea80b740 1 -- 172.21.15.31:0/1648190640 wait complete. 2024-04-25T16:11:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.483+0000 7f73ea80b740 20 client.27406 trim_cache size 0 max 16384 2024-04-25T16:11:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:11:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:11:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:11:50.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:11:50.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.484+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 msgr2=0x5590e80befe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:50.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.484+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80befe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f399c0319e0 tx=0x7f399c031460 comp rx=0 tx=0).stop 2024-04-25T16:11:50.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.488+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 shutdown_connections 2024-04-25T16:11:50.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.488+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5590e7cd1f60 0x5590e80cc4b0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.488+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80c5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.488+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80befe0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.488+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2744383549 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cea4d0 0x5590e80beaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.488+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 >> 172.21.15.31:0/2744383549 conn(0x5590e7d074e0 msgr2=0x5590e80af420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:50.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.488+0000 7f3982ffd640 0 client.27415 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:50.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.488+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 shutdown_connections 2024-04-25T16:11:50.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.488+0000 7f39b0c27740 1 -- 172.21.15.31:0/2744383549 wait complete. 2024-04-25T16:11:50.706 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.489+0000 7f39b0c27740 20 client.27415 trim_cache size 0 max 16384 2024-04-25T16:11:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.489+0000 7f73e2622640 1 -- 172.21.15.31:0/3959792204 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73c80afbd0 msgr2=0x7f73c80b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.489+0000 7f73e2622640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73c80afbd0 0x7f73c80b20e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f73d8031500 tx=0x7f73d8005fd0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.489+0000 7f73e2622640 1 -- 172.21.15.31:0/3959792204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73dc10feb0 msgr2=0x7f73dc07ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.489+0000 7f73e2622640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73dc10feb0 0x7f73dc07ae00 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f73d400b990 tx=0x7f73d400be50 comp rx=0 tx=0).stop 2024-04-25T16:11:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.496+0000 7f73e2622640 1 -- 172.21.15.31:0/3959792204 shutdown_connections 2024-04-25T16:11:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.496+0000 7f73e2622640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f73c80afbd0 0x7f73c80b20e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.496+0000 7f73e2622640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f73dc10feb0 0x7f73dc07ae00 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.496+0000 7f73e2622640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73dc10b190 0x7f73dc1a9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.496+0000 7f73e2622640 1 --2- 172.21.15.31:0/3959792204 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f73dc10a340 0x7f73dc1a8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.496+0000 7f73e2622640 1 -- 172.21.15.31:0/3959792204 >> 172.21.15.31:0/3959792204 conn(0x7f73dc070330 msgr2=0x7f73dc10be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.498+0000 7f73e2622640 1 -- 172.21.15.31:0/3959792204 shutdown_connections 2024-04-25T16:11:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:11:50.498+0000 7f73e2622640 1 -- 172.21.15.31:0/3959792204 wait complete. 2024-04-25T16:11:50.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.604+0000 7f39b0c27740 1 Processor -- start 2024-04-25T16:11:50.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.604+0000 7f39b0c27740 1 -- start start 2024-04-25T16:11:50.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.604+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 0x5590e80bd870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.605+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.605+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.605+0000 7f39b0c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f399000e540 con 0x5590e7cf26b0 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.605+0000 7f39b0c27740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f3990010080 con 0x5590e7d13390 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.605+0000 7f39b0c27740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f399000ec10 con 0x5590e7cea4d0 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.605+0000 7f39a2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.605+0000 7f39a2ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80c5e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:39520/0 (socket says 172.21.15.31:39520) 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.605+0000 7f39a2ffd640 1 -- 172.21.15.31:0/592402153 learned_addr learned my addr 172.21.15.31:0/592402153 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.606+0000 7f39a3fff640 1 --2- 172.21.15.31:0/592402153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 0x5590e80bd870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:50.715 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.606+0000 7f39a2ffd640 1 -- 172.21.15.31:0/592402153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 msgr2=0x5590e80bd870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.606+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/592402153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 0x5590e80bd870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.606+0000 7f39a2ffd640 1 -- 172.21.15.31:0/592402153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 msgr2=0x5590e80c63a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:11:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.606+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/592402153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80c63a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.606+0000 7f39a2ffd640 1 -- 172.21.15.31:0/592402153 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590e7ceb0f0 con 0x5590e7cf26b0 2024-04-25T16:11:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.607+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/592402153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80c5e60 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f399c009bf0 tx=0x7f399c031c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b91620dbd359b1b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.609+0000 7f39897fa640 1 -- 172.21.15.31:0/592402153 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c055030 con 0x5590e7cf26b0 2024-04-25T16:11:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.609+0000 7f39897fa640 1 -- 172.21.15.31:0/592402153 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f399c036070 con 0x5590e7cf26b0 2024-04-25T16:11:50.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.609+0000 7f39897fa640 1 -- 172.21.15.31:0/592402153 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399c003cf0 con 0x5590e7cf26b0 2024-04-25T16:11:50.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.611+0000 7f39b0c27740 1 -- 172.21.15.31:0/592402153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 msgr2=0x5590e80c5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:50.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.611+0000 7f39b0c27740 1 --2- 172.21.15.31:0/592402153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80c5e60 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f399c009bf0 tx=0x7f399c031c80 comp rx=0 tx=0).stop 2024-04-25T16:11:50.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.615+0000 7f39b0c27740 1 -- 172.21.15.31:0/592402153 shutdown_connections 2024-04-25T16:11:50.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.615+0000 7f39b0c27740 1 --2- 172.21.15.31:0/592402153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7d13390 0x5590e80c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.615+0000 7f39b0c27740 1 --2- 172.21.15.31:0/592402153 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7cf26b0 0x5590e80c5e60 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.615+0000 7f39b0c27740 1 --2- 172.21.15.31:0/592402153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cea4d0 0x5590e80bd870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.615+0000 7f39b0c27740 1 -- 172.21.15.31:0/592402153 >> 172.21.15.31:0/592402153 conn(0x5590e7d074e0 msgr2=0x5590e80bd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:50.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.628+0000 7f39b0c27740 1 -- 172.21.15.31:0/592402153 shutdown_connections 2024-04-25T16:11:50.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.628+0000 7f39b0c27740 1 -- 172.21.15.31:0/592402153 wait complete. 2024-04-25T16:11:50.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.630+0000 7f39b0c27740 1 -- start start 2024-04-25T16:11:50.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.631+0000 7f39b0c27740 1 Processor -- start 2024-04-25T16:11:50.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.633+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7cea4d0 0x5590e80c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.633+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80c4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.633+0000 7f39b0c27740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7d13390 0x5590e80c5300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.633+0000 7f39b0c27740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f399000f9b0 con 0x5590e7d13390 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.633+0000 7f39b0c27740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f399000f2e0 con 0x5590e7cea4d0 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.634+0000 7f39b0c27740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f399000c520 con 0x5590e7cf26b0 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.634+0000 7f39b0c27740 10 client.0 _ll_register_callbacks cb 0x7f399c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.634+0000 7f39a2ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.634+0000 7f39a3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7d13390 0x5590e80c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.634+0000 7f39a3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7cea4d0 0x5590e80c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.635+0000 7f39a3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7cea4d0 0x5590e80c3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49006/0 (socket says 172.21.15.31:49006) 2024-04-25T16:11:50.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.635+0000 7f39a3fff640 1 -- 172.21.15.31:0/2379673573 learned_addr learned my addr 172.21.15.31:0/2379673573 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:11:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.635+0000 7f39a3fff640 1 -- 172.21.15.31:0/2379673573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 msgr2=0x5590e80c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.635+0000 7f39a3fff640 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80c4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.636+0000 7f39a3fff640 1 -- 172.21.15.31:0/2379673573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7d13390 msgr2=0x5590e80c5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:11:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.636+0000 7f39a3fff640 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7d13390 0x5590e80c5300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.636+0000 7f39a3fff640 1 -- 172.21.15.31:0/2379673573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590e7fda4b0 con 0x5590e7cea4d0 2024-04-25T16:11:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.636+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:11:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.637+0000 7f39a3fff640 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7cea4d0 0x5590e80c3ee0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3998015010 tx=0x7f3998009a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a042378d3593e755 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:11:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.642+0000 7f3982ffd640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:11:50.720 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.642+0000 7f3982ffd640 1 -- 172.21.15.31:0/2379673573 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f399802ae60 con 0x5590e7cea4d0 2024-04-25T16:11:50.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.642+0000 7f39b0c27740 1 -- 172.21.15.31:0/2379673573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f399c036070 con 0x5590e7cea4d0 2024-04-25T16:11:50.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.642+0000 7f3982ffd640 1 -- 172.21.15.31:0/2379673573 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f39980037f0 con 0x5590e7cea4d0 2024-04-25T16:11:50.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.643+0000 7f3982ffd640 1 -- 172.21.15.31:0/2379673573 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f39980039f0 con 0x5590e7cea4d0 2024-04-25T16:11:50.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.647+0000 7f39b0c27740 10 client.27422 fetch_fsmap learned FSMap version 29 2024-04-25T16:11:50.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.647+0000 7f39b0c27740 1 -- 172.21.15.31:0/2379673573 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5590e80c69e0 con 0x5590e7cea4d0 2024-04-25T16:11:50.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.650+0000 7f3982ffd640 1 -- 172.21.15.31:0/2379673573 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f399c036070 con 0x5590e7cea4d0 2024-04-25T16:11:50.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.650+0000 7f3982ffd640 1 -- 172.21.15.31:0/2379673573 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f3998029580 con 0x5590e7cea4d0 2024-04-25T16:11:50.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.650+0000 7f39b0c27740 10 client.27422 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:11:50.721 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.650+0000 7f39b0c27740 10 client.27422 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:11:50.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.650+0000 7f39b0c27740 20 client.27422 populate_metadata read hostname 'smithi031' 2024-04-25T16:11:50.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.650+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5590e7ca73a0 0x5590e8136fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:11:50.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.650+0000 7f39b0c27740 4 client.27422 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:11:50.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.650+0000 7f39b0c27740 1 -- 172.21.15.31:0/2379673573 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5590e8027990 con 0x5590e7ca73a0 2024-04-25T16:11:50.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.651+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5590e7ca73a0 0x5590e8136fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:11:50.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.658+0000 7f39a2ffd640 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5590e7ca73a0 0x5590e8136fc0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e0c5e1583baa060 server_cookie=c0b2afb92395cca0 in_seq=0 out_seq=0 2024-04-25T16:11:50.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.659+0000 7f3982ffd640 10 client.27422 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:50.722 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.659+0000 7f3982ffd640 1 -- 172.21.15.31:0/2379673573 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5590e8027990 con 0x5590e7ca73a0 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.659+0000 7f3982ffd640 10 client.27422 handle_command_reply: tid=0 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.659+0000 7f39b0c27740 1 client.27422 shutdown 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.670+0000 7f39b0c27740 1 -- 172.21.15.31:0/2379673573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7cea4d0 msgr2=0x5590e80c3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.670+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7cea4d0 0x5590e80c3ee0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3998015010 tx=0x7f3998009a30 comp rx=0 tx=0).stop 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.670+0000 7f39b0c27740 1 -- 172.21.15.31:0/2379673573 shutdown_connections 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.670+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5590e7ca73a0 0x5590e8136fc0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.670+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5590e7d13390 0x5590e80c5300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.670+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5590e7cf26b0 0x5590e80c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.671+0000 7f39b0c27740 1 --2- 172.21.15.31:0/2379673573 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5590e7cea4d0 0x5590e80c3ee0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.723 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.671+0000 7f39b0c27740 1 -- 172.21.15.31:0/2379673573 >> 172.21.15.31:0/2379673573 conn(0x5590e7d074e0 msgr2=0x5590e80b3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:50.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.671+0000 7f3982ffd640 0 client.27422 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:11:50.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.671+0000 7f39b0c27740 1 -- 172.21.15.31:0/2379673573 shutdown_connections 2024-04-25T16:11:50.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.672+0000 7f39b0c27740 1 -- 172.21.15.31:0/2379673573 wait complete. 2024-04-25T16:11:50.724 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.672+0000 7f39b0c27740 20 client.27422 trim_cache size 0 max 16384 2024-04-25T16:11:50.724 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:11:50.724 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-25T16:11:50.725 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:11:50.725 INFO:teuthology.orchestra.run.smithi031.stdout: "aa8006c0-bbf7-4e98-9077-9884603cb246": { 2024-04-25T16:11:50.725 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:11:50.725 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "aa8006c0-bbf7-4e98-9077-9884603cb246", 2024-04-25T16:11:50.725 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:11:50.725 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:11:50.725 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:11:50.725 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:11:50.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.683+0000 7f39a9d72640 1 -- 172.21.15.31:0/3096164298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f398c0afd00 msgr2=0x7f398c0b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:50.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.683+0000 7f39a9d72640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f398c0afd00 0x7f398c0b2210 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f3994011040 tx=0x7f39940046c0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.683+0000 7f39a9d72640 1 -- 172.21.15.31:0/3096164298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4102260 msgr2=0x7f39a4110800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:11:50.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.683+0000 7f39a9d72640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4102260 0x7f39a4110800 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f399400b450 tx=0x7f399400b910 comp rx=0 tx=0).stop 2024-04-25T16:11:50.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.685+0000 7f39a9d72640 1 -- 172.21.15.31:0/3096164298 shutdown_connections 2024-04-25T16:11:50.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.685+0000 7f39a9d72640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f398c0afd00 0x7f398c0b2210 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.685+0000 7f39a9d72640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f39a4107520 0x7f39a4111c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.727 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.685+0000 7f39a9d72640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f39a4104bc0 0x7f39a4110d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.685+0000 7f39a9d72640 1 --2- 172.21.15.31:0/3096164298 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39a4102260 0x7f39a4110800 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:11:50.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.686+0000 7f39a9d72640 1 -- 172.21.15.31:0/3096164298 >> 172.21.15.31:0/3096164298 conn(0x7f39a40fda70 msgr2=0x7f39a40ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:11:50.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.688+0000 7f39a9d72640 1 -- 172.21.15.31:0/3096164298 shutdown_connections 2024-04-25T16:11:50.728 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:11:50.689+0000 7f39a9d72640 1 -- 172.21.15.31:0/3096164298 wait complete. 2024-04-25T16:11:50.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:50 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:11:50.728 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:50 smithi031 ceph-mon[29140]: pgmap v667: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 744 MiB/s wr, 1.91k op/s 2024-04-25T16:11:50.746 INFO:tasks.cephfs.filesystem:scrub status for tag:aa8006c0-bbf7-4e98-9077-9884603cb246 - {'path': '/', 'tag': 'aa8006c0-bbf7-4e98-9077-9884603cb246', 'options': 'recursive,force'} 2024-04-25T16:11:53.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:53 smithi063 ceph-mon[32413]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 815 MiB/s wr, 1.99k op/s 2024-04-25T16:11:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:53 smithi186 ceph-mon[33258]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 815 MiB/s wr, 1.99k op/s 2024-04-25T16:11:53.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:53 smithi031 ceph-mon[29140]: pgmap v668: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 815 MiB/s wr, 1.99k op/s 2024-04-25T16:11:55.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:55 smithi063 ceph-mon[32413]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 890 MiB/s wr, 2.14k op/s 2024-04-25T16:11:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:55 smithi186 ceph-mon[33258]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 890 MiB/s wr, 2.14k op/s 2024-04-25T16:11:55.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:55 smithi031 ceph-mon[29140]: pgmap v669: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 890 MiB/s wr, 2.14k op/s 2024-04-25T16:11:57.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:57 smithi063 ceph-mon[32413]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 744 MiB/s wr, 1.78k op/s 2024-04-25T16:11:57.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:57 smithi186 ceph-mon[33258]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 744 MiB/s wr, 1.78k op/s 2024-04-25T16:11:57.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:57 smithi031 ceph-mon[29140]: pgmap v670: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 744 MiB/s wr, 1.78k op/s 2024-04-25T16:11:58.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:58 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:11:58.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:58 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:11:58.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:58 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:11:59.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:59 smithi063 ceph-mon[32413]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 924 MiB/s wr, 2.16k op/s 2024-04-25T16:11:59.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:59 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:11:59.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:11:59 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:11:59.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:59 smithi186 ceph-mon[33258]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 924 MiB/s wr, 2.16k op/s 2024-04-25T16:11:59.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:59 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:11:59.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:11:59 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:11:59.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:59 smithi031 ceph-mon[29140]: pgmap v671: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 924 MiB/s wr, 2.16k op/s 2024-04-25T16:11:59.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:59 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:11:59.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:11:59 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:12:01.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:01 smithi063 ceph-mon[32413]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 764 MiB/s wr, 1.67k op/s 2024-04-25T16:12:01.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:01.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:01 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:01.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:01 smithi186 ceph-mon[33258]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 764 MiB/s wr, 1.67k op/s 2024-04-25T16:12:01.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:01.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:01 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:01 smithi031 ceph-mon[29140]: pgmap v672: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 764 MiB/s wr, 1.67k op/s 2024-04-25T16:12:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:01.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:01 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:02.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:02 smithi063 ceph-mon[32413]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 839 MiB/s wr, 1.97k op/s 2024-04-25T16:12:02.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:02 smithi186 ceph-mon[33258]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 839 MiB/s wr, 1.97k op/s 2024-04-25T16:12:02.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:02 smithi031 ceph-mon[29140]: pgmap v673: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 839 MiB/s wr, 1.97k op/s 2024-04-25T16:12:03.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.946+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/1177474600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3c810b350 msgr2=0x7fc3c810b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:03.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.946+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/1177474600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3c810b350 0x7fc3c810b760 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fc3c001acb0 tx=0x7fc3c003b8e0 comp rx=0 tx=0).stop 2024-04-25T16:12:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.951+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/1177474600 shutdown_connections 2024-04-25T16:12:03.954 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.951+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/1177474600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3c8110070 0x7fc3c8110480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.951+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/1177474600 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3c810b350 0x7fc3c810b760 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.951+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/1177474600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c810a420 0x7fc3c810ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:03.955 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.951+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/1177474600 >> 172.21.15.31:0/1177474600 conn(0x7fc3c8070540 msgr2=0x7fc3c80729d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:03.960 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.959+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/1177474600 shutdown_connections 2024-04-25T16:12:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.963+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/1177474600 wait complete. 2024-04-25T16:12:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.964+0000 7fc3ce3f3640 1 Processor -- start 2024-04-25T16:12:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.970+0000 7fc3ce3f3640 1 -- start start 2024-04-25T16:12:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.970+0000 7fc3ce3f3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c810a420 0x7fc3c81a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.970+0000 7fc3ce3f3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3c810b350 0x7fc3c81a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.970+0000 7fc3ce3f3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3c8110070 0x7fc3c81a7d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.970+0000 7fc3ce3f3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8113d00 con 0x7fc3c8110070 2024-04-25T16:12:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.970+0000 7fc3ce3f3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8113b80 con 0x7fc3c810a420 2024-04-25T16:12:03.972 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.970+0000 7fc3ce3f3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8113e80 con 0x7fc3c810b350 2024-04-25T16:12:03.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.972+0000 7fc3c7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c810a420 0x7fc3c81a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:03.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.972+0000 7fc3c7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c810a420 0x7fc3c81a08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49804/0 (socket says 172.21.15.31:49804) 2024-04-25T16:12:03.973 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.972+0000 7fc3c7fff640 1 -- 172.21.15.31:0/424792301 learned_addr learned my addr 172.21.15.31:0/424792301 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:03.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.973+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3c810b350 0x7fc3c81a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:03.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.973+0000 7fc3c7fff640 1 -- 172.21.15.31:0/424792301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3c810b350 msgr2=0x7fc3c81a0e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:03.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.973+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3c810b350 0x7fc3c81a0e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:03.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.973+0000 7fc3c7fff640 1 -- 172.21.15.31:0/424792301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3c8110070 msgr2=0x7fc3c81a7d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-25T16:12:03.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.973+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3c8110070 0x7fc3c81a7d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:03.974 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.973+0000 7fc3c7fff640 1 -- 172.21.15.31:0/424792301 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c81a8400 con 0x7fc3c810a420 2024-04-25T16:12:03.975 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.974+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c810a420 0x7fc3c81a08c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800c970 tx=0x7fc3b800ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:03.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.979+0000 7fc3c57fa640 1 -- 172.21.15.31:0/424792301 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b8007bb0 con 0x7fc3c810a420 2024-04-25T16:12:03.982 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.980+0000 7fc3c57fa640 1 -- 172.21.15.31:0/424792301 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b8007d50 con 0x7fc3c810a420 2024-04-25T16:12:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.981+0000 7fc3c57fa640 1 -- 172.21.15.31:0/424792301 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b800f450 con 0x7fc3c810a420 2024-04-25T16:12:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.983+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/424792301 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c81a86f0 con 0x7fc3c810a420 2024-04-25T16:12:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.984+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/424792301 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c81a8bc0 con 0x7fc3c810a420 2024-04-25T16:12:03.989 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.988+0000 7fc3d537e740 1 Processor -- start 2024-04-25T16:12:03.991 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.990+0000 7fc3d537e740 1 -- start start 2024-04-25T16:12:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.991+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868b9d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.991+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.991+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868ba4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.991+0000 7fc3d537e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560868b03460 con 0x5608687dc4d0 2024-04-25T16:12:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.991+0000 7fc3d537e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560868abd590 con 0x5608687fd1b0 2024-04-25T16:12:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.991+0000 7fc3d537e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560868aff420 con 0x5608687d42f0 2024-04-25T16:12:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.992+0000 7fc3c7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.992+0000 7fc3c7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45048/0 (socket says 172.21.15.31:45048) 2024-04-25T16:12:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.992+0000 7fc3c7fff640 1 -- 172.21.15.31:0/1538862986 learned_addr learned my addr 172.21.15.31:0/1538862986 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:03.997 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.994+0000 7fc3cc990640 1 --2- 172.21.15.31:0/1538862986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868b9d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:03.998 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.996+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/1538862986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868ba4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.998+0000 7fc3c7fff640 1 -- 172.21.15.31:0/1538862986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 msgr2=0x560868b9d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.998+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/1538862986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868b9d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.998+0000 7fc3c7fff640 1 -- 172.21.15.31:0/1538862986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 msgr2=0x560868ba4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.998+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/1538862986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868ba4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.998+0000 7fc3c7fff640 1 -- 172.21.15.31:0/1538862986 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560868aca0d0 con 0x5608687d42f0 2024-04-25T16:12:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.999+0000 7fc3c57fa640 1 -- 172.21.15.31:0/424792301 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc3b8004f30 con 0x7fc3c810a420 2024-04-25T16:12:04.000 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:03.999+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/1538862986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9def0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80e1d20 tx=0x7fc3b80e5b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c4323bfe6b5f0eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:04.001 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.000+0000 7fc3c57fa640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3b00afbd0 0x7fc3b00b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.003 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.001+0000 7fc3c57fa640 1 -- 172.21.15.31:0/424792301 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc3b80e42d0 con 0x7fc3c810a420 2024-04-25T16:12:04.004 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.002+0000 7fc3cc990640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3b00afbd0 0x7fc3b00b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.004+0000 7fc3ae7fc640 1 -- 172.21.15.31:0/1538862986 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b80f3070 con 0x5608687d42f0 2024-04-25T16:12:04.006 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.005+0000 7fc3ae7fc640 1 -- 172.21.15.31:0/1538862986 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b80e6400 con 0x5608687d42f0 2024-04-25T16:12:04.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.006+0000 7fc3d537e740 1 -- 172.21.15.31:0/1538862986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 msgr2=0x560868b9def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:04.007 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.006+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1538862986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9def0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fc3b80e1d20 tx=0x7fc3b80e5b10 comp rx=0 tx=0).stop 2024-04-25T16:12:04.011 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.009+0000 7fc3cc990640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3b00afbd0 0x7fc3b00b20e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x560868b9eb40 tx=0x7fc3bc005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.015+0000 7fc3d537e740 1 -- 172.21.15.31:0/1538862986 shutdown_connections 2024-04-25T16:12:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.015+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1538862986 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868ba4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.015+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1538862986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9def0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.015+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1538862986 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868b9d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.015+0000 7fc3d537e740 1 -- 172.21.15.31:0/1538862986 >> 172.21.15.31:0/1538862986 conn(0x5608687f1300 msgr2=0x560868b99300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.015+0000 7fc3d537e740 1 -- 172.21.15.31:0/1538862986 shutdown_connections 2024-04-25T16:12:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.015+0000 7fc3d537e740 1 -- 172.21.15.31:0/1538862986 wait complete. 2024-04-25T16:12:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.016+0000 7fc3d537e740 1 -- start start 2024-04-25T16:12:04.018 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.016+0000 7fc3d537e740 1 Processor -- start 2024-04-25T16:12:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.019+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687d42f0 0x560868ba76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.019+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687dc4d0 0x560868ba7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.019+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868baeb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.019+0000 7fc3d537e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x560868ba5890 con 0x5608687d42f0 2024-04-25T16:12:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.019+0000 7fc3d537e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x560868a8ab60 con 0x5608687fd1b0 2024-04-25T16:12:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.019+0000 7fc3d537e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x560868ba5a10 con 0x5608687dc4d0 2024-04-25T16:12:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.019+0000 7fc3d537e740 10 client.0 _ll_register_callbacks cb 0x560868aaf2b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.020+0000 7fc3c77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687dc4d0 0x560868ba7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.020+0000 7fc3c77fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687dc4d0 0x560868ba7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:45062/0 (socket says 172.21.15.31:45062) 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.020+0000 7fc3c77fe640 1 -- 172.21.15.31:0/1657999972 learned_addr learned my addr 172.21.15.31:0/1657999972 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.020+0000 7fc3cc990640 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868baeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.020+0000 7fc3c77fe640 1 -- 172.21.15.31:0/1657999972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 msgr2=0x560868baeb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.020+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868baeb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.020+0000 7fc3c77fe640 1 -- 172.21.15.31:0/1657999972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687d42f0 msgr2=0x560868ba76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.020+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687d42f0 0x560868ba76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.020+0000 7fc3c77fe640 1 -- 172.21.15.31:0/1657999972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560868815210 con 0x5608687dc4d0 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.021+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687dc4d0 0x560868ba7c20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc3c001acb0 tx=0x7fc3c001aa30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1c31399156da858 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.021+0000 7fc3a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.021+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1657999972 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0003970 con 0x5608687dc4d0 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.021+0000 7fc3d537e740 1 -- 172.21.15.31:0/1657999972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3b80e6400 con 0x5608687dc4d0 2024-04-25T16:12:04.023 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.021+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1657999972 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc3c0003b10 con 0x5608687dc4d0 2024-04-25T16:12:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.022+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1657999972 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc3c0003d10 con 0x5608687dc4d0 2024-04-25T16:12:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.022+0000 7fc3d537e740 10 client.27430 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.022+0000 7fc3d537e740 1 -- 172.21.15.31:0/1657999972 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5608688e1620 con 0x5608687dc4d0 2024-04-25T16:12:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.023+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1657999972 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3b80e6400 con 0x5608687dc4d0 2024-04-25T16:12:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.024+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1657999972 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc3c00419c0 con 0x5608687dc4d0 2024-04-25T16:12:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.024+0000 7fc3d537e740 10 client.27430 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.024+0000 7fc3d537e740 10 client.27430 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.025+0000 7fc3d537e740 20 client.27430 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.025+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5608687bbd80 0x560868bb53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.025+0000 7fc3d537e740 4 client.27430 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:04.026 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.025+0000 7fc3d537e740 1 -- 172.21.15.31:0/1657999972 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560868b11070 con 0x5608687bbd80 2024-04-25T16:12:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.026+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5608687bbd80 0x560868bb53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.031+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5608687bbd80 0x560868bb53b0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbaccae9cbcb8567 server_cookie=17f7278ae6ade96a in_seq=0 out_seq=0 2024-04-25T16:12:04.032 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.031+0000 7fc3a77fe640 10 client.27430 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.042+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1657999972 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560868b11070 con 0x5608687bbd80 2024-04-25T16:12:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.042+0000 7fc3a77fe640 10 client.27430 handle_command_reply: tid=0 2024-04-25T16:12:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.042+0000 7fc3d537e740 1 client.27430 shutdown 2024-04-25T16:12:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.044+0000 7fc3d537e740 1 -- 172.21.15.31:0/1657999972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687dc4d0 msgr2=0x560868ba7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.044+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687dc4d0 0x560868ba7c20 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc3c001acb0 tx=0x7fc3c001aa30 comp rx=0 tx=0).stop 2024-04-25T16:12:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.047+0000 7fc3d537e740 1 -- 172.21.15.31:0/1657999972 shutdown_connections 2024-04-25T16:12:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.047+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5608687bbd80 0x560868bb53b0 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.047+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868baeb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.047+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687dc4d0 0x560868ba7c20 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.047+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1657999972 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687d42f0 0x560868ba76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.047+0000 7fc3d537e740 1 -- 172.21.15.31:0/1657999972 >> 172.21.15.31:0/1657999972 conn(0x5608687f1300 msgr2=0x560868b9c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.050+0000 7fc3a77fe640 0 client.27430 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.051+0000 7fc3d537e740 1 -- 172.21.15.31:0/1657999972 shutdown_connections 2024-04-25T16:12:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.051+0000 7fc3d537e740 1 -- 172.21.15.31:0/1657999972 wait complete. 2024-04-25T16:12:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.051+0000 7fc3d537e740 20 client.27430 trim_cache size 0 max 16384 2024-04-25T16:12:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.103+0000 7fc3d537e740 1 Processor -- start 2024-04-25T16:12:04.105 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.104+0000 7fc3d537e740 1 -- start start 2024-04-25T16:12:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.104+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.104+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868baea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.104+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868b98c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.104+0000 7fc3d537e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b17d0 con 0x5608687dc4d0 2024-04-25T16:12:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.104+0000 7fc3d537e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b3310 con 0x5608687fd1b0 2024-04-25T16:12:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.104+0000 7fc3d537e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b1ea0 con 0x5608687d42f0 2024-04-25T16:12:04.106 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.105+0000 7fc3cc990640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868b98c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.105+0000 7fc3cc990640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868b98c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49830/0 (socket says 172.21.15.31:49830) 2024-04-25T16:12:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.105+0000 7fc3cc990640 1 -- 172.21.15.31:0/471473351 learned_addr learned my addr 172.21.15.31:0/471473351 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.105+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868baea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.105+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.106+0000 7fc3c77fe640 1 -- 172.21.15.31:0/471473351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 msgr2=0x560868b9c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.106+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.106+0000 7fc3c77fe640 1 -- 172.21.15.31:0/471473351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 msgr2=0x560868b98c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.106+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868b98c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.107 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.106+0000 7fc3c77fe640 1 -- 172.21.15.31:0/471473351 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560868ba24b0 con 0x5608687dc4d0 2024-04-25T16:12:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.106+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:12:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.106+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868baea30 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fc3c001ac00 tx=0x7fc3c0056670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84345a90017c30c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.107+0000 7fc3adffb640 1 -- 172.21.15.31:0/471473351 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c0003970 con 0x5608687dc4d0 2024-04-25T16:12:04.108 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.107+0000 7fc3adffb640 1 -- 172.21.15.31:0/471473351 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3c0047040 con 0x5608687dc4d0 2024-04-25T16:12:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.109+0000 7fc3d537e740 1 -- 172.21.15.31:0/471473351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 msgr2=0x560868baea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.109+0000 7fc3d537e740 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868baea30 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fc3c001ac00 tx=0x7fc3c0056670 comp rx=0 tx=0).stop 2024-04-25T16:12:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.111+0000 7fc3d537e740 1 -- 172.21.15.31:0/471473351 shutdown_connections 2024-04-25T16:12:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.111+0000 7fc3d537e740 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687fd1b0 0x560868b98c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.111+0000 7fc3d537e740 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868baea30 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.111+0000 7fc3d537e740 1 --2- 172.21.15.31:0/471473351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687d42f0 0x560868b9c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.111+0000 7fc3d537e740 1 -- 172.21.15.31:0/471473351 >> 172.21.15.31:0/471473351 conn(0x5608687f1300 msgr2=0x560868b9c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:04.122 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.121+0000 7fc3d537e740 1 -- 172.21.15.31:0/471473351 shutdown_connections 2024-04-25T16:12:04.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.121+0000 7fc3d537e740 1 -- 172.21.15.31:0/471473351 wait complete. 2024-04-25T16:12:04.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.121+0000 7fc3d537e740 1 -- start start 2024-04-25T16:12:04.123 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.122+0000 7fc3d537e740 1 Processor -- start 2024-04-25T16:12:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.122+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687d42f0 0x560868babc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.122+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868bac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.122+0000 7fc3d537e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687fd1b0 0x560868bad100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.122+0000 7fc3d537e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b2c40 con 0x5608687dc4d0 2024-04-25T16:12:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.122+0000 7fc3d537e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81b2570 con 0x5608687d42f0 2024-04-25T16:12:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.122+0000 7fc3d537e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81af7b0 con 0x5608687fd1b0 2024-04-25T16:12:04.124 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.122+0000 7fc3d537e740 10 client.0 _ll_register_callbacks cb 0x7fc3c001ab30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.123+0000 7fc3c7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687d42f0 0x560868babc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.123+0000 7fc3c7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687d42f0 0x560868babc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:49840/0 (socket says 172.21.15.31:49840) 2024-04-25T16:12:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.123+0000 7fc3c7fff640 1 -- 172.21.15.31:0/1576724224 learned_addr learned my addr 172.21.15.31:0/1576724224 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:04.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.123+0000 7fc3cc990640 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687fd1b0 0x560868bad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.124+0000 7fc3c7fff640 1 -- 172.21.15.31:0/1576724224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687fd1b0 msgr2=0x560868bad100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.124+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687fd1b0 0x560868bad100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.124+0000 7fc3c7fff640 1 -- 172.21.15.31:0/1576724224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 msgr2=0x560868bac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.124+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868bac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.124+0000 7fc3c7fff640 1 -- 172.21.15.31:0/1576724224 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560868bad6c0 con 0x5608687d42f0 2024-04-25T16:12:04.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.124+0000 7fc3c7fff640 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687d42f0 0x560868babc90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8004a40 tx=0x7fc3b80b5040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47fd57eff33c8bd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:04.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.135+0000 7fc3a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:12:04.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.136+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1576724224 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b80a8620 con 0x5608687d42f0 2024-04-25T16:12:04.137 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.136+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1576724224 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fc3b80a8d50 con 0x5608687d42f0 2024-04-25T16:12:04.138 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.137+0000 7fc3d537e740 1 -- 172.21.15.31:0/1576724224 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3c0047040 con 0x5608687d42f0 2024-04-25T16:12:04.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.144+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1576724224 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fc3b80abed0 con 0x5608687d42f0 2024-04-25T16:12:04.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.146+0000 7fc3d537e740 10 client.27437 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:04.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.146+0000 7fc3d537e740 1 -- 172.21.15.31:0/1576724224 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560868bae9c0 con 0x5608687d42f0 2024-04-25T16:12:04.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.149+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1576724224 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc3c0047040 con 0x5608687d42f0 2024-04-25T16:12:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.149+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1576724224 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fc3b80a89f0 con 0x5608687d42f0 2024-04-25T16:12:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.150+0000 7fc3d537e740 10 client.27437 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.150+0000 7fc3d537e740 10 client.27437 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.150+0000 7fc3d537e740 20 client.27437 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.150+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5608687911c0 0x560868c1f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.150+0000 7fc3d537e740 4 client.27437 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:12:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.150+0000 7fc3d537e740 1 -- 172.21.15.31:0/1576724224 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560868b0aee0 con 0x5608687911c0 2024-04-25T16:12:04.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.151+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5608687911c0 0x560868c1f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.156+0000 7fc3c77fe640 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5608687911c0 0x560868c1f7c0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70915a536b3e9259 server_cookie=45e6809d2872d2ae in_seq=0 out_seq=0 2024-04-25T16:12:04.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:04.156+0000 7fc3a77fe640 10 client.27437 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:05.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:05 smithi063 ceph-mon[32413]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 845 MiB/s wr, 2.25k op/s 2024-04-25T16:12:05.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:05 smithi186 ceph-mon[33258]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 845 MiB/s wr, 2.25k op/s 2024-04-25T16:12:05.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:05 smithi031 ceph-mon[29140]: pgmap v674: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 845 MiB/s wr, 2.25k op/s 2024-04-25T16:12:06.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:06.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:06 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:06.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:06 smithi186 ceph-mon[33258]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 2.15k op/s 2024-04-25T16:12:06.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:06.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:06 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:06.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:06 smithi031 ceph-mon[29140]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 2.15k op/s 2024-04-25T16:12:07.119 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:07.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:06 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:07.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:06 smithi063 ceph-mon[32413]: pgmap v675: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 721 MiB/s wr, 2.15k op/s 2024-04-25T16:12:09.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:09 smithi063 ceph-mon[32413]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 882 MiB/s wr, 2.66k op/s 2024-04-25T16:12:09.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:09 smithi186 ceph-mon[33258]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 882 MiB/s wr, 2.66k op/s 2024-04-25T16:12:09.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:09 smithi031 ceph-mon[29140]: pgmap v676: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 882 MiB/s wr, 2.66k op/s 2024-04-25T16:12:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.700+0000 7fc3a77fe640 1 -- 172.21.15.31:0/1576724224 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560868b0aee0 con 0x5608687911c0 2024-04-25T16:12:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.700+0000 7fc3a77fe640 10 client.27437 handle_command_reply: tid=0 2024-04-25T16:12:09.702 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.700+0000 7fc3d537e740 1 client.27437 shutdown 2024-04-25T16:12:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.701+0000 7fc3d537e740 1 -- 172.21.15.31:0/1576724224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687d42f0 msgr2=0x560868babc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.701+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687d42f0 0x560868babc90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8004a40 tx=0x7fc3b80b5040 comp rx=0 tx=0).stop 2024-04-25T16:12:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.701+0000 7fc3d537e740 1 -- 172.21.15.31:0/1576724224 shutdown_connections 2024-04-25T16:12:09.704 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.701+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5608687911c0 0x560868c1f7c0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.701+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5608687fd1b0 0x560868bad100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.702+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5608687dc4d0 0x560868bac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.702+0000 7fc3d537e740 1 --2- 172.21.15.31:0/1576724224 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5608687d42f0 0x560868babc90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:09.705 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.702+0000 7fc3d537e740 1 -- 172.21.15.31:0/1576724224 >> 172.21.15.31:0/1576724224 conn(0x5608687f1300 msgr2=0x560868ba5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.702+0000 7fc3a77fe640 0 client.27437 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.702+0000 7fc3d537e740 1 -- 172.21.15.31:0/1576724224 shutdown_connections 2024-04-25T16:12:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.702+0000 7fc3d537e740 1 -- 172.21.15.31:0/1576724224 wait complete. 2024-04-25T16:12:09.706 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.702+0000 7fc3d537e740 20 client.27437 trim_cache size 0 max 16384 2024-04-25T16:12:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:12:09.707 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:12:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:12:09.708 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:12:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.704+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/424792301 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3b00afbd0 msgr2=0x7fc3b00b20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.704+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3b00afbd0 0x7fc3b00b20e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x560868b9eb40 tx=0x7fc3bc005d20 comp rx=0 tx=0).stop 2024-04-25T16:12:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.704+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/424792301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c810a420 msgr2=0x7fc3c81a08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:09.709 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.704+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c810a420 0x7fc3c81a08c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc3b800c970 tx=0x7fc3b800ce30 comp rx=0 tx=0).stop 2024-04-25T16:12:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.710+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/424792301 shutdown_connections 2024-04-25T16:12:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.710+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fc3b00afbd0 0x7fc3b00b20e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.710+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fc3c8110070 0x7fc3c81a7d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:09.712 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.710+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3c810b350 0x7fc3c81a0e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.710+0000 7fc3ce3f3640 1 --2- 172.21.15.31:0/424792301 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc3c810a420 0x7fc3c81a08c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.710+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/424792301 >> 172.21.15.31:0/424792301 conn(0x7fc3c8070540 msgr2=0x7fc3c810e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:09.713 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.711+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/424792301 shutdown_connections 2024-04-25T16:12:09.714 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:09.712+0000 7fc3ce3f3640 1 -- 172.21.15.31:0/424792301 wait complete. 2024-04-25T16:12:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:10 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:10.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:10 smithi031 ceph-mon[29140]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 703 MiB/s wr, 2.28k op/s 2024-04-25T16:12:10.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:10.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:10 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:10.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:10 smithi063 ceph-mon[32413]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 703 MiB/s wr, 2.28k op/s 2024-04-25T16:12:10.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:10.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:10 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:10.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:10 smithi186 ceph-mon[33258]: pgmap v677: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 703 MiB/s wr, 2.28k op/s 2024-04-25T16:12:11.770 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:11 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:12:11.770 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:11 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:12:11.770 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:11 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:11.799 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:11 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:12:11.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:11 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:12:11.800 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:11 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:11.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:11 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:12:11.950 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:11 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:12:11.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:11 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:12:12.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:12 smithi063 ceph-mon[32413]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 773 MiB/s wr, 2.47k op/s 2024-04-25T16:12:12.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:12 smithi186 ceph-mon[33258]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 773 MiB/s wr, 2.47k op/s 2024-04-25T16:12:12.951 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:12 smithi031 ceph-mon[29140]: pgmap v678: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 773 MiB/s wr, 2.47k op/s 2024-04-25T16:12:15.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:15 smithi063 ceph-mon[32413]: pgmap v679: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 789 MiB/s wr, 2.25k op/s 2024-04-25T16:12:15.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:15 smithi186 ceph-mon[33258]: pgmap v679: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 789 MiB/s wr, 2.25k op/s 2024-04-25T16:12:15.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:15 smithi031 ceph-mon[29140]: pgmap v679: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 789 MiB/s wr, 2.25k op/s 2024-04-25T16:12:17.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:17 smithi063 ceph-mon[32413]: pgmap v680: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 719 MiB/s wr, 1.94k op/s 2024-04-25T16:12:17.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:17 smithi186 ceph-mon[33258]: pgmap v680: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 719 MiB/s wr, 1.94k op/s 2024-04-25T16:12:17.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:17 smithi031 ceph-mon[29140]: pgmap v680: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 719 MiB/s wr, 1.94k op/s 2024-04-25T16:12:19.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:19 smithi063 ceph-mon[32413]: pgmap v681: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 889 MiB/s wr, 2.32k op/s 2024-04-25T16:12:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:19 smithi186 ceph-mon[33258]: pgmap v681: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 889 MiB/s wr, 2.32k op/s 2024-04-25T16:12:19.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:19 smithi031 ceph-mon[29140]: pgmap v681: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 889 MiB/s wr, 2.32k op/s 2024-04-25T16:12:20.748 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:12:20.748 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:12:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.089+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/1967054178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ab410a340 msgr2=0x7f5ab410ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:21.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.089+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/1967054178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ab410a340 0x7f5ab410ac50 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009860 tx=0x7f5aa402a380 comp rx=0 tx=0).stop 2024-04-25T16:12:21.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.092+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/1967054178 shutdown_connections 2024-04-25T16:12:21.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.092+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/1967054178 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ab410fe90 0x7f5ab41102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.092+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/1967054178 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab410b190 0x7f5ab410b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.092+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/1967054178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ab410a340 0x7f5ab410ac50 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.092+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/1967054178 >> 172.21.15.31:0/1967054178 conn(0x7f5ab4070320 msgr2=0x7f5ab40727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.098+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/1967054178 shutdown_connections 2024-04-25T16:12:21.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.098+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/1967054178 wait complete. 2024-04-25T16:12:21.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.100+0000 7f5ab9e6b640 1 Processor -- start 2024-04-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.100+0000 7f5ab9e6b640 1 -- start start 2024-04-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.100+0000 7f5ab9e6b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab410a340 0x7f5ab41a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.100+0000 7f5ab9e6b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ab410b190 0x7f5ab41a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.100+0000 7f5ab9e6b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ab410fe90 0x7f5ab41a7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.100+0000 7f5ab9e6b640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4113b50 con 0x7f5ab410a340 2024-04-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.100+0000 7f5ab9e6b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab41139d0 con 0x7f5ab410b190 2024-04-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.100+0000 7f5ab9e6b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5ab4113cd0 con 0x7f5ab410fe90 2024-04-25T16:12:21.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.101+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ab410b190 0x7f5ab41a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.101+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ab410b190 0x7f5ab41a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39926/0 (socket says 172.21.15.31:39926) 2024-04-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.101+0000 7f5ab3fff640 1 -- 172.21.15.31:0/878630720 learned_addr learned my addr 172.21.15.31:0/878630720 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.101+0000 7f5ab966a640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ab410fe90 0x7f5ab41a7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:21.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.102+0000 7f5ab966a640 1 -- 172.21.15.31:0/878630720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ab410b190 msgr2=0x7f5ab41a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:21.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.102+0000 7f5ab966a640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ab410b190 0x7f5ab41a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.102+0000 7f5ab966a640 1 -- 172.21.15.31:0/878630720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab410a340 msgr2=0x7f5ab41a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:21.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.102+0000 7f5ab966a640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab410a340 0x7f5ab41a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.102+0000 7f5ab966a640 1 -- 172.21.15.31:0/878630720 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ab41a8190 con 0x7f5ab410fe90 2024-04-25T16:12:21.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.110+0000 7f5ab966a640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ab410fe90 0x7f5ab41a7b20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00e940 tx=0x7f5aac00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:21.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.114+0000 7f5ab1ffb640 1 -- 172.21.15.31:0/878630720 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac004d20 con 0x7f5ab410fe90 2024-04-25T16:12:21.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.114+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/878630720 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ab41a8480 con 0x7f5ab410fe90 2024-04-25T16:12:21.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.115+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/878630720 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ab41a89d0 con 0x7f5ab410fe90 2024-04-25T16:12:21.121 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.120+0000 7f5ac1f78740 1 Processor -- start 2024-04-25T16:12:21.122 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.120+0000 7f5ac1f78740 1 -- start start 2024-04-25T16:12:21.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.122+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3d1390 0x55deee772c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.124 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.123+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee7731d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.123+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3b06b0 0x55deee773e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.123+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55deee6d7d80 con 0x55deee3d1390 2024-04-25T16:12:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.124+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55deee6918b0 con 0x55deee3a84d0 2024-04-25T16:12:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.124+0000 7f5ab8e69640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee7731d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.124+0000 7f5ab8e69640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee7731d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39930/0 (socket says 172.21.15.31:39930) 2024-04-25T16:12:21.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.124+0000 7f5ab8e69640 1 -- 172.21.15.31:0/1127622398 learned_addr learned my addr 172.21.15.31:0/1127622398 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.124+0000 7f5ac1f78740 1 -- 172.21.15.31:0/1127622398 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55deee6d3d40 con 0x55deee3b06b0 2024-04-25T16:12:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.124+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/1127622398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3b06b0 0x55deee773e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:21.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.120+0000 7f5ab1ffb640 1 -- 172.21.15.31:0/878630720 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aac005020 con 0x7f5ab410fe90 2024-04-25T16:12:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.126+0000 7f5ab8e69640 1 -- 172.21.15.31:0/1127622398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3b06b0 msgr2=0x55deee773e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:21.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.126+0000 7f5ab8e69640 1 --2- 172.21.15.31:0/1127622398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3b06b0 0x55deee773e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.126+0000 7f5ab8e69640 1 -- 172.21.15.31:0/1127622398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3d1390 msgr2=0x55deee772c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:21.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.126+0000 7f5ab8e69640 1 --2- 172.21.15.31:0/1127622398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3d1390 0x55deee772c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.126+0000 7f5ab8e69640 1 -- 172.21.15.31:0/1127622398 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deee662a40 con 0x55deee3a84d0 2024-04-25T16:12:21.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.127+0000 7f5ab1ffb640 1 -- 172.21.15.31:0/878630720 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac0116d0 con 0x7f5ab410fe90 2024-04-25T16:12:21.129 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.127+0000 7f5ab8e69640 1 --2- 172.21.15.31:0/1127622398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee7731d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4035040 tx=0x7f5aa4009390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d68ff11aadffef62 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:21.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.140+0000 7f5ab1ffb640 1 -- 172.21.15.31:0/878630720 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5aac00cb20 con 0x7f5ab410fe90 2024-04-25T16:12:21.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.142+0000 7f5ab1ffb640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a9c0afb50 0x7f5a9c0b2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.145 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.143+0000 7f5ab1ffb640 1 -- 172.21.15.31:0/878630720 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5aac0deba0 con 0x7f5ab410fe90 2024-04-25T16:12:21.146 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.144+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a9c0afb50 0x7f5a9c0b2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.144+0000 7f5a9affd640 1 -- 172.21.15.31:0/1127622398 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4037040 con 0x55deee3a84d0 2024-04-25T16:12:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.144+0000 7f5a9affd640 1 -- 172.21.15.31:0/1127622398 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aa4040070 con 0x55deee3a84d0 2024-04-25T16:12:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.144+0000 7f5a9affd640 1 -- 172.21.15.31:0/1127622398 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4003850 con 0x55deee3a84d0 2024-04-25T16:12:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.146+0000 7f5ac1f78740 1 -- 172.21.15.31:0/1127622398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 msgr2=0x55deee7731d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:21.148 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.146+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/1127622398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee7731d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4035040 tx=0x7f5aa4009390 comp rx=0 tx=0).stop 2024-04-25T16:12:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.147+0000 7f5ac1f78740 1 -- 172.21.15.31:0/1127622398 shutdown_connections 2024-04-25T16:12:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.147+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/1127622398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3b06b0 0x55deee773e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.147+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/1127622398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee7731d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.147+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/1127622398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3d1390 0x55deee772c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.147+0000 7f5ac1f78740 1 -- 172.21.15.31:0/1127622398 >> 172.21.15.31:0/1127622398 conn(0x55deee3c54e0 msgr2=0x55deee76e460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.147+0000 7f5ac1f78740 1 -- 172.21.15.31:0/1127622398 shutdown_connections 2024-04-25T16:12:21.149 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.147+0000 7f5ac1f78740 1 -- 172.21.15.31:0/1127622398 wait complete. 2024-04-25T16:12:21.151 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.149+0000 7f5ac1f78740 1 -- start start 2024-04-25T16:12:21.171 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.150+0000 7f5ac1f78740 1 Processor -- start 2024-04-25T16:12:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.151+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3a84d0 0x55deee77caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.151+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3b06b0 0x55deee77cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.151+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee783f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.151+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55deee774910 con 0x55deee3a84d0 2024-04-25T16:12:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.151+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55deee65ed10 con 0x55deee3b06b0 2024-04-25T16:12:21.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.151+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55deee774a90 con 0x55deee3d1390 2024-04-25T16:12:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.151+0000 7f5ac1f78740 10 client.0 _ll_register_callbacks cb 0x55deee68b210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.157+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3b06b0 0x55deee77cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.157+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3b06b0 0x55deee77cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39932/0 (socket says 172.21.15.31:39932) 2024-04-25T16:12:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.157+0000 7f5ab3fff640 1 -- 172.21.15.31:0/2337127996 learned_addr learned my addr 172.21.15.31:0/2337127996 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.158+0000 7f5ab3fff640 1 -- 172.21.15.31:0/2337127996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 msgr2=0x55deee783f40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.158+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee783f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.158+0000 7f5ab3fff640 1 -- 172.21.15.31:0/2337127996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3a84d0 msgr2=0x55deee77caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:21.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.158+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3a84d0 0x55deee77caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:21.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.158+0000 7f5ab3fff640 1 -- 172.21.15.31:0/2337127996 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deee4b37d0 con 0x55deee3b06b0 2024-04-25T16:12:21.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.159+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3b06b0 0x55deee77cfe0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80058c0 tx=0x7f5aa801c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=453065b3c4098df4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:21.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.159+0000 7f5a93fff640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:12:21.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.159+0000 7f5a93fff640 1 -- 172.21.15.31:0/2337127996 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa803c060 con 0x55deee3b06b0 2024-04-25T16:12:21.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.160+0000 7f5ac1f78740 1 -- 172.21.15.31:0/2337127996 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aa4040070 con 0x55deee3b06b0 2024-04-25T16:12:21.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.160+0000 7f5a93fff640 1 -- 172.21.15.31:0/2337127996 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5aa8019040 con 0x55deee3b06b0 2024-04-25T16:12:21.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.160+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a9c0afb50 0x7f5a9c0b2060 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x55deee773ca0 tx=0x7f5aa8025040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:21.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.167+0000 7f5a93fff640 1 -- 172.21.15.31:0/2337127996 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5aa8023610 con 0x55deee3b06b0 2024-04-25T16:12:21.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.168+0000 7f5ac1f78740 10 client.27449 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:21.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.168+0000 7f5ac1f78740 1 -- 172.21.15.31:0/2337127996 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55deee3e93f0 con 0x55deee3b06b0 2024-04-25T16:12:21.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.169+0000 7f5a93fff640 1 -- 172.21.15.31:0/2337127996 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aa4040070 con 0x55deee3b06b0 2024-04-25T16:12:21.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.171+0000 7f5a93fff640 1 -- 172.21.15.31:0/2337127996 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5aa802d050 con 0x55deee3b06b0 2024-04-25T16:12:21.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.171+0000 7f5ac1f78740 10 client.27449 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:21.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.171+0000 7f5ac1f78740 10 client.27449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:21.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.171+0000 7f5ac1f78740 20 client.27449 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:21.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.172+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55deee38ff60 0x55deee78aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:21.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.172+0000 7f5ac1f78740 4 client.27449 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:21.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:21.172+0000 7f5ac1f78740 1 -- 172.21.15.31:0/2337127996 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55deee6e5990 con 0x55deee38ff60 2024-04-25T16:12:21.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:21 smithi063 ceph-mon[32413]: pgmap v682: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 735 MiB/s wr, 1.79k op/s 2024-04-25T16:12:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:21 smithi186 ceph-mon[33258]: pgmap v682: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 735 MiB/s wr, 1.79k op/s 2024-04-25T16:12:21.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:21 smithi031 ceph-mon[29140]: pgmap v682: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 735 MiB/s wr, 1.79k op/s 2024-04-25T16:12:22.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.236+0000 7f5ab8e69640 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55deee38ff60 0x55deee78aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.246+0000 7f5a93fff640 10 client.27449 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:22.247 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.246+0000 7f5ab8e69640 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55deee38ff60 0x55deee78aaf0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1604bc8049d496e2 server_cookie=2548092da3859931 in_seq=0 out_seq=0 2024-04-25T16:12:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.251+0000 7f5a93fff640 1 -- 172.21.15.31:0/2337127996 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55deee6e5990 con 0x55deee38ff60 2024-04-25T16:12:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.251+0000 7f5a93fff640 10 client.27449 handle_command_reply: tid=0 2024-04-25T16:12:22.252 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.251+0000 7f5ac1f78740 1 client.27449 shutdown 2024-04-25T16:12:22.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.252+0000 7f5ac1f78740 1 -- 172.21.15.31:0/2337127996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3b06b0 msgr2=0x55deee77cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:22.253 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.252+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3b06b0 0x55deee77cfe0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5aa80058c0 tx=0x7f5aa801c490 comp rx=0 tx=0).stop 2024-04-25T16:12:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.254+0000 7f5ac1f78740 1 -- 172.21.15.31:0/2337127996 shutdown_connections 2024-04-25T16:12:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.254+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55deee38ff60 0x55deee78aaf0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.254+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee783f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.254+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3b06b0 0x55deee77cfe0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.254+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/2337127996 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3a84d0 0x55deee77caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.254+0000 7f5ac1f78740 1 -- 172.21.15.31:0/2337127996 >> 172.21.15.31:0/2337127996 conn(0x55deee3c54e0 msgr2=0x55deee771370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.254+0000 7f5a93fff640 0 client.27449 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:22.256 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.254+0000 7f5ac1f78740 1 -- 172.21.15.31:0/2337127996 shutdown_connections 2024-04-25T16:12:22.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.256+0000 7f5ac1f78740 1 -- 172.21.15.31:0/2337127996 wait complete. 2024-04-25T16:12:22.257 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.256+0000 7f5ac1f78740 20 client.27449 trim_cache size 0 max 16384 2024-04-25T16:12:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.322+0000 7f5ac1f78740 1 Processor -- start 2024-04-25T16:12:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.322+0000 7f5ac1f78740 1 -- start start 2024-04-25T16:12:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.323+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee78aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.323+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee783e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.323+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee783e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.324+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7843a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.324+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa000e370 con 0x55deee3b06b0 2024-04-25T16:12:22.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.324+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa000feb0 con 0x55deee3a84d0 2024-04-25T16:12:22.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.324+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa000ea40 con 0x55deee3d1390 2024-04-25T16:12:22.326 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.324+0000 7f5ab8e69640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee78aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.325+0000 7f5ab3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee783e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53566/0 (socket says 172.21.15.31:53566) 2024-04-25T16:12:22.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.325+0000 7f5ab3fff640 1 -- 172.21.15.31:0/203006992 learned_addr learned my addr 172.21.15.31:0/203006992 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:22.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.325+0000 7f5ab966a640 1 --2- 172.21.15.31:0/203006992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7843a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.324+0000 7f5ab8e69640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee78aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39936/0 (socket says 172.21.15.31:39936) 2024-04-25T16:12:22.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.326+0000 7f5ab3fff640 1 -- 172.21.15.31:0/203006992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 msgr2=0x55deee7843a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:22.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.326+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/203006992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7843a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.326+0000 7f5ab3fff640 1 -- 172.21.15.31:0/203006992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 msgr2=0x55deee78aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:22.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.326+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/203006992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee78aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.326+0000 7f5ab3fff640 1 -- 172.21.15.31:0/203006992 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deee77b840 con 0x55deee3b06b0 2024-04-25T16:12:22.328 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.327+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/203006992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee783e60 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8003ab0 tx=0x7f5aa8021f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd34f90b4cd49bf server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:22.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.327+0000 7f5a9a7fc640 1 -- 172.21.15.31:0/203006992 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa8047040 con 0x55deee3b06b0 2024-04-25T16:12:22.329 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.328+0000 7f5a9a7fc640 1 -- 172.21.15.31:0/203006992 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aa8023c60 con 0x55deee3b06b0 2024-04-25T16:12:22.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.330+0000 7f5ac1f78740 1 -- 172.21.15.31:0/203006992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 msgr2=0x55deee783e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:22.331 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.330+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/203006992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee783e60 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f5aa8003ab0 tx=0x7f5aa8021f30 comp rx=0 tx=0).stop 2024-04-25T16:12:22.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.331+0000 7f5ac1f78740 1 -- 172.21.15.31:0/203006992 shutdown_connections 2024-04-25T16:12:22.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.331+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/203006992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7843a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.331+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/203006992 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee783e60 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.331+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/203006992 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee78aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.333 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.331+0000 7f5ac1f78740 1 -- 172.21.15.31:0/203006992 >> 172.21.15.31:0/203006992 conn(0x55deee3c54e0 msgr2=0x55deee772bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:22.336 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.335+0000 7f5ac1f78740 1 -- 172.21.15.31:0/203006992 shutdown_connections 2024-04-25T16:12:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.335+0000 7f5ac1f78740 1 -- 172.21.15.31:0/203006992 wait complete. 2024-04-25T16:12:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.335+0000 7f5ac1f78740 1 -- start start 2024-04-25T16:12:22.337 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.336+0000 7f5ac1f78740 1 Processor -- start 2024-04-25T16:12:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.336+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee781710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.336+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee781c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.338 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.336+0000 7f5ac1f78740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7f0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.336+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa000f7e0 con 0x55deee3b06b0 2024-04-25T16:12:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.336+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa000f110 con 0x55deee3a84d0 2024-04-25T16:12:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.336+0000 7f5ac1f78740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa000c400 con 0x55deee3d1390 2024-04-25T16:12:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.337+0000 7f5ac1f78740 10 client.0 _ll_register_callbacks cb 0x7f5aa801cf90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.337+0000 7f5ab966a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7f0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.337+0000 7f5ab966a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7f0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:48694/0 (socket says 172.21.15.31:48694) 2024-04-25T16:12:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.337+0000 7f5ab966a640 1 -- 172.21.15.31:0/138239118 learned_addr learned my addr 172.21.15.31:0/138239118 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.337+0000 7f5ab3fff640 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee781c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.339 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.337+0000 7f5ab966a640 1 -- 172.21.15.31:0/138239118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 msgr2=0x55deee781710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:22.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.338+0000 7f5ab966a640 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee781710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.338+0000 7f5ab966a640 1 -- 172.21.15.31:0/138239118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 msgr2=0x55deee781c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:22.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.338+0000 7f5ab966a640 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee781c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.340 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.338+0000 7f5ab966a640 1 -- 172.21.15.31:0/138239118 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deee78b060 con 0x55deee3d1390 2024-04-25T16:12:22.341 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.339+0000 7f5ab966a640 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7f0e90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5aac005870 tx=0x7f5aac0a7630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a79c2a985cf2d10d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:22.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.348+0000 7f5a93fff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:12:22.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.348+0000 7f5a93fff640 1 -- 172.21.15.31:0/138239118 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aac01f040 con 0x55deee3d1390 2024-04-25T16:12:22.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.348+0000 7f5a93fff640 1 -- 172.21.15.31:0/138239118 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5aac0a3680 con 0x55deee3d1390 2024-04-25T16:12:22.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.351+0000 7f5ac1f78740 1 -- 172.21.15.31:0/138239118 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5aa8023c60 con 0x55deee3d1390 2024-04-25T16:12:22.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.352+0000 7f5a93fff640 1 -- 172.21.15.31:0/138239118 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5aac0c6050 con 0x55deee3d1390 2024-04-25T16:12:22.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.353+0000 7f5ac1f78740 10 client.27451 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:22.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.353+0000 7f5ac1f78740 1 -- 172.21.15.31:0/138239118 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55deee3a90f0 con 0x55deee3d1390 2024-04-25T16:12:22.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.354+0000 7f5a93fff640 1 -- 172.21.15.31:0/138239118 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5aac0a7090 con 0x55deee3d1390 2024-04-25T16:12:22.357 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.355+0000 7f5a93fff640 1 -- 172.21.15.31:0/138239118 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f5aac00f040 con 0x55deee3d1390 2024-04-25T16:12:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.356+0000 7f5ac1f78740 10 client.27451 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.356+0000 7f5ac1f78740 10 client.27451 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.357+0000 7f5ac1f78740 20 client.27451 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:22.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.357+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55deee3653a0 0x55deee7f4c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.357+0000 7f5ac1f78740 4 client.27451 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:12:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.357+0000 7f5ac1f78740 1 -- 172.21.15.31:0/138239118 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f5aa8023c60 con 0x55deee3653a0 2024-04-25T16:12:22.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.358+0000 7f5ab8e69640 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55deee3653a0 0x55deee7f4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.373+0000 7f5a93fff640 10 client.27451 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:22.374 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.373+0000 7f5ab8e69640 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55deee3653a0 0x55deee7f4c80 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b594020239d2a3ab server_cookie=1ca40ca1b9e2ebdc in_seq=0 out_seq=0 2024-04-25T16:12:22.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.375+0000 7f5a93fff640 1 -- 172.21.15.31:0/138239118 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f5aa8023c60 con 0x55deee3653a0 2024-04-25T16:12:22.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.375+0000 7f5a93fff640 10 client.27451 handle_command_reply: tid=0 2024-04-25T16:12:22.377 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.376+0000 7f5ac1f78740 1 client.27451 shutdown 2024-04-25T16:12:22.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.382+0000 7f5ac1f78740 1 -- 172.21.15.31:0/138239118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 msgr2=0x55deee7f0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:22.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.382+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7f0e90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f5aac005870 tx=0x7f5aac0a7630 comp rx=0 tx=0).stop 2024-04-25T16:12:22.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.382+0000 7f5ac1f78740 1 -- 172.21.15.31:0/138239118 shutdown_connections 2024-04-25T16:12:22.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.382+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55deee3653a0 0x55deee7f4c80 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.382+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55deee3d1390 0x55deee7f0e90 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.382+0000 7f5a93fff640 0 client.27451 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:22.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.382+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55deee3b06b0 0x55deee781c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.382+0000 7f5ac1f78740 1 --2- 172.21.15.31:0/138239118 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55deee3a84d0 0x55deee781710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.382+0000 7f5ac1f78740 1 -- 172.21.15.31:0/138239118 >> 172.21.15.31:0/138239118 conn(0x55deee3c54e0 msgr2=0x55deee772bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:22.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.386+0000 7f5ac1f78740 1 -- 172.21.15.31:0/138239118 shutdown_connections 2024-04-25T16:12:22.387 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.386+0000 7f5ac1f78740 1 -- 172.21.15.31:0/138239118 wait complete. 2024-04-25T16:12:22.389 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.388+0000 7f5ac1f78740 20 client.27451 trim_cache size 0 max 16384 2024-04-25T16:12:22.390 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:12:22.390 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:12:22.390 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:12:22.390 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:12:22.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.401+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/878630720 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a9c0afb50 msgr2=0x7f5a9c0b2060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:22.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.401+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a9c0afb50 0x7f5a9c0b2060 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x55deee773ca0 tx=0x7f5aa8025040 comp rx=0 tx=0).stop 2024-04-25T16:12:22.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.401+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/878630720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ab410fe90 msgr2=0x7f5ab41a7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:22.403 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.402+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ab410fe90 0x7f5ab41a7b20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f5aac00e940 tx=0x7f5aac00ee00 comp rx=0 tx=0).stop 2024-04-25T16:12:22.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.404+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/878630720 shutdown_connections 2024-04-25T16:12:22.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.404+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f5a9c0afb50 0x7f5a9c0b2060 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.404+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ab410fe90 0x7f5ab41a7b20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.404+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5ab410b190 0x7f5ab41a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.405+0000 7f5ab9e6b640 1 --2- 172.21.15.31:0/878630720 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5ab410a340 0x7f5ab41a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.405+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/878630720 >> 172.21.15.31:0/878630720 conn(0x7f5ab4070320 msgr2=0x7f5ab406a000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:22.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.407+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/878630720 shutdown_connections 2024-04-25T16:12:22.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.407+0000 7f5ab9e6b640 1 -- 172.21.15.31:0/878630720 wait complete. 2024-04-25T16:12:22.470 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:12:22.471 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:12:22.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.893+0000 7f37e4f57640 1 -- 172.21.15.31:0/1474002113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37e010a420 msgr2=0x7f37e010add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:22.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.893+0000 7f37e4f57640 1 --2- 172.21.15.31:0/1474002113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37e010a420 0x7f37e010add0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f37d0009900 tx=0x7f37d002a410 comp rx=0 tx=0).stop 2024-04-25T16:12:22.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.899+0000 7f37e4f57640 1 -- 172.21.15.31:0/1474002113 shutdown_connections 2024-04-25T16:12:22.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.899+0000 7f37e4f57640 1 --2- 172.21.15.31:0/1474002113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37e0110030 0x7f37e0110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.899+0000 7f37e4f57640 1 --2- 172.21.15.31:0/1474002113 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37e010b310 0x7f37e010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.899+0000 7f37e4f57640 1 --2- 172.21.15.31:0/1474002113 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37e010a420 0x7f37e010add0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.900+0000 7f37e4f57640 1 -- 172.21.15.31:0/1474002113 >> 172.21.15.31:0/1474002113 conn(0x7f37e00704e0 msgr2=0x7f37e0072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:22.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.900+0000 7f37e4f57640 1 -- 172.21.15.31:0/1474002113 shutdown_connections 2024-04-25T16:12:22.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.903+0000 7f37e4f57640 1 -- 172.21.15.31:0/1474002113 wait complete. 2024-04-25T16:12:22.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.905+0000 7f37e4f57640 1 Processor -- start 2024-04-25T16:12:22.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.905+0000 7f37e4f57640 1 -- start start 2024-04-25T16:12:22.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.905+0000 7f37e4f57640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37e010a420 0x7f37e0117010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.905+0000 7f37e4f57640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37e010b310 0x7f37e0117550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.905+0000 7f37e4f57640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37e0110030 0x7f37e01184b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.905+0000 7f37e4f57640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f37e011e4e0 con 0x7f37e0110030 2024-04-25T16:12:22.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.905+0000 7f37e4f57640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f37e011e360 con 0x7f37e010a420 2024-04-25T16:12:22.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.905+0000 7f37e4f57640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f37e011e660 con 0x7f37e010b310 2024-04-25T16:12:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.911+0000 7f37ddd9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37e010b310 0x7f37e0117550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.911+0000 7f37de59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37e010a420 0x7f37e0117010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.911+0000 7f37de59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37e010a420 0x7f37e0117010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39958/0 (socket says 172.21.15.31:39958) 2024-04-25T16:12:22.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.911+0000 7f37de59c640 1 -- 172.21.15.31:0/1619029706 learned_addr learned my addr 172.21.15.31:0/1619029706 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:22.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.913+0000 7f37ded9d640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37e0110030 0x7f37e01184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.914+0000 7f37de59c640 1 -- 172.21.15.31:0/1619029706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37e010b310 msgr2=0x7f37e0117550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:22.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.914+0000 7f37de59c640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37e010b310 0x7f37e0117550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.914+0000 7f37de59c640 1 -- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37e0110030 msgr2=0x7f37e01184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:22.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.914+0000 7f37de59c640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37e0110030 0x7f37e01184b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:22.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.914+0000 7f37de59c640 1 -- 172.21.15.31:0/1619029706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e0118b20 con 0x7f37e010a420 2024-04-25T16:12:22.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.915+0000 7f37de59c640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37e010a420 0x7f37e0117010 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f37d002aa00 tx=0x7f37d00056e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:22.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.920+0000 7f37cf7fe640 1 -- 172.21.15.31:0/1619029706 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d0041070 con 0x7f37e010a420 2024-04-25T16:12:22.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.920+0000 7f37cf7fe640 1 -- 172.21.15.31:0/1619029706 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37d003d450 con 0x7f37e010a420 2024-04-25T16:12:22.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.920+0000 7f37cf7fe640 1 -- 172.21.15.31:0/1619029706 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d00316a0 con 0x7f37e010a420 2024-04-25T16:12:22.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.920+0000 7f37e4f57640 1 -- 172.21.15.31:0/1619029706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e0118db0 con 0x7f37e010a420 2024-04-25T16:12:22.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.920+0000 7f37e4f57640 1 -- 172.21.15.31:0/1619029706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e01192b0 con 0x7f37e010a420 2024-04-25T16:12:22.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.925+0000 7f37e4f57640 1 -- 172.21.15.31:0/1619029706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e0119d90 con 0x7f37e010a420 2024-04-25T16:12:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.941+0000 7f37cf7fe640 1 -- 172.21.15.31:0/1619029706 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f37d0044070 con 0x7f37e010a420 2024-04-25T16:12:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.943+0000 7f37cf7fe640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37c80afc70 0x7f37c80b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.943+0000 7f37cf7fe640 1 -- 172.21.15.31:0/1619029706 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f37d00ffd20 con 0x7f37e010a420 2024-04-25T16:12:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.943+0000 7f37cf7fe640 1 -- 172.21.15.31:0/1619029706 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37d0100240 con 0x7f37e010a420 2024-04-25T16:12:22.951 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.947+0000 7f37ddd9b640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37c80afc70 0x7f37c80b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:22.956 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:22.950+0000 7f37ddd9b640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37c80afc70 0x7f37c80b2180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f37d8005c40 tx=0x7f37d8005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:23.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.096+0000 7fa7c922a640 1 -- 172.21.15.31:0/1324082279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7c4102620 msgr2=0x7fa7c4102af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:23.099 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.096+0000 7fa7c922a640 1 --2- 172.21.15.31:0/1324082279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7c4102620 0x7fa7c4102af0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa7b801a910 tx=0x7fa7b803b470 comp rx=0 tx=0).stop 2024-04-25T16:12:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.115+0000 7fa7c922a640 1 -- 172.21.15.31:0/1324082279 shutdown_connections 2024-04-25T16:12:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.115+0000 7fa7c922a640 1 --2- 172.21.15.31:0/1324082279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4107790 0x7fa7c4107ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.115+0000 7fa7c922a640 1 --2- 172.21.15.31:0/1324082279 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7c4102620 0x7fa7c4102af0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.115+0000 7fa7c922a640 1 --2- 172.21.15.31:0/1324082279 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c4101be0 0x7fa7c4102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.115+0000 7fa7c922a640 1 -- 172.21.15.31:0/1324082279 >> 172.21.15.31:0/1324082279 conn(0x7fa7c40fd1f0 msgr2=0x7fa7c40ff680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.115+0000 7fa7c922a640 1 -- 172.21.15.31:0/1324082279 shutdown_connections 2024-04-25T16:12:23.118 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.115+0000 7fa7c922a640 1 -- 172.21.15.31:0/1324082279 wait complete. 2024-04-25T16:12:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.116+0000 7fa7c922a640 1 Processor -- start 2024-04-25T16:12:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.116+0000 7fa7c922a640 1 -- start start 2024-04-25T16:12:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.117+0000 7fa7c922a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4101be0 0x7fa7c4197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.117+0000 7fa7c922a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7c4102620 0x7fa7c4198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.119 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.117+0000 7fa7c922a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c4107790 0x7fa7c419f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.117+0000 7fa7c922a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c410b3f0 con 0x7fa7c4107790 2024-04-25T16:12:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.117+0000 7fa7c922a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c410b270 con 0x7fa7c4102620 2024-04-25T16:12:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.117+0000 7fa7c922a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c410b570 con 0x7fa7c4101be0 2024-04-25T16:12:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.118+0000 7fa7c8a29640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c4107790 0x7fa7c419f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.118+0000 7fa7c8a29640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c4107790 0x7fa7c419f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53612/0 (socket says 172.21.15.31:53612) 2024-04-25T16:12:23.120 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.118+0000 7fa7c8a29640 1 -- 172.21.15.31:0/3227446623 learned_addr learned my addr 172.21.15.31:0/3227446623 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:23.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.121+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7c4102620 0x7fa7c4198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.121+0000 7fa7c3fff640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4101be0 0x7fa7c4197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.123+0000 7fa7c8a29640 1 -- 172.21.15.31:0/3227446623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4101be0 msgr2=0x7fa7c4197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:23.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.123+0000 7fa7c8a29640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4101be0 0x7fa7c4197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.123+0000 7fa7c8a29640 1 -- 172.21.15.31:0/3227446623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7c4102620 msgr2=0x7fa7c4198350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:23.125 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.123+0000 7fa7c8a29640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7c4102620 0x7fa7c4198350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.123+0000 7fa7c8a29640 1 -- 172.21.15.31:0/3227446623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7c406c6f0 con 0x7fa7c4107790 2024-04-25T16:12:23.126 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.124+0000 7fa7c8a29640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c4107790 0x7fa7c419f2e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400ead0 tx=0x7fa7b400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.131+0000 7fa7c17fa640 1 -- 172.21.15.31:0/3227446623 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b400cea0 con 0x7fa7c4107790 2024-04-25T16:12:23.132 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.131+0000 7fa7c17fa640 1 -- 172.21.15.31:0/3227446623 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7b4005020 con 0x7fa7c4107790 2024-04-25T16:12:23.133 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.131+0000 7fa7c17fa640 1 -- 172.21.15.31:0/3227446623 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4010690 con 0x7fa7c4107790 2024-04-25T16:12:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.133+0000 7fa7c922a640 1 -- 172.21.15.31:0/3227446623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7c406c9e0 con 0x7fa7c4107790 2024-04-25T16:12:23.135 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.133+0000 7fa7c922a640 1 -- 172.21.15.31:0/3227446623 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7c406ceb0 con 0x7fa7c4107790 2024-04-25T16:12:23.146 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.138+0000 7fa7d1410740 1 Processor -- start 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.138+0000 7fa7d1410740 1 -- start start 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.138+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09ff1b0 0x563ac0d9fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.138+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.138+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09de4d0 0x563ac0da6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.138+0000 7fa7d1410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563ac0d05460 con 0x563ac09d62f0 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.138+0000 7fa7d1410740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563ac0cbf590 con 0x563ac09ff1b0 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.138+0000 7fa7d1410740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563ac0d01420 con 0x563ac09de4d0 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.139+0000 7fa7c37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09ff1b0 0x563ac0d9fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.139+0000 7fa7c37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09ff1b0 0x563ac0d9fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39986/0 (socket says 172.21.15.31:39986) 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.139+0000 7fa7c37fe640 1 -- 172.21.15.31:0/3206776084 learned_addr learned my addr 172.21.15.31:0/3206776084 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.139+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/3206776084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09de4d0 0x563ac0da6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.147 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.139+0000 7fa7c37fe640 1 -- 172.21.15.31:0/3206776084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09de4d0 msgr2=0x563ac0da6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.139+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/3206776084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09de4d0 0x563ac0da6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.139+0000 7fa7c37fe640 1 -- 172.21.15.31:0/3206776084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 msgr2=0x563ac0da0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.139+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/3206776084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.140+0000 7fa7c37fe640 1 -- 172.21.15.31:0/3206776084 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac0ccc0d0 con 0x563ac09ff1b0 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.140+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/3206776084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09ff1b0 0x563ac0d9fb10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa7b801a910 tx=0x7fa7b8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4b4d170a374d8c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.143+0000 7fa7c17fa640 1 -- 172.21.15.31:0/3227446623 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa7b40048d0 con 0x7fa7c4107790 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.143+0000 7fa7b27fc640 1 -- 172.21.15.31:0/3206776084 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8018070 con 0x563ac09ff1b0 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.143+0000 7fa7b27fc640 1 -- 172.21.15.31:0/3206776084 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7b8004b60 con 0x563ac09ff1b0 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.144+0000 7fa7c17fa640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7ac0afcf0 0x7fa7ac0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.144+0000 7fa7d1410740 1 -- 172.21.15.31:0/3206776084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09ff1b0 msgr2=0x563ac0d9fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.144+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3206776084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09ff1b0 0x563ac0d9fb10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa7b801a910 tx=0x7fa7b8000bc0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.148 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.145+0000 7fa7c3fff640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7ac0afcf0 0x7fa7ac0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.146+0000 7fa7d1410740 1 -- 172.21.15.31:0/3206776084 shutdown_connections 2024-04-25T16:12:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.146+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3206776084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09de4d0 0x563ac0da6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.146+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3206776084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.146+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3206776084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09ff1b0 0x563ac0d9fb10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.146+0000 7fa7d1410740 1 -- 172.21.15.31:0/3206776084 >> 172.21.15.31:0/3206776084 conn(0x563ac09f3300 msgr2=0x563ac0d9b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.147+0000 7fa7c17fa640 1 -- 172.21.15.31:0/3227446623 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa7b40df2b0 con 0x7fa7c4107790 2024-04-25T16:12:23.149 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.147+0000 7fa7d1410740 1 -- 172.21.15.31:0/3206776084 shutdown_connections 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.147+0000 7fa7d1410740 1 -- 172.21.15.31:0/3206776084 wait complete. 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.148+0000 7fa7d1410740 1 -- start start 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.148+0000 7fa7d1410740 1 Processor -- start 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.149+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.149+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da9e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.149+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 0x563ac0db0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.149+0000 7fa7d1410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563ac0da79d0 con 0x563ac09d62f0 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.149+0000 7fa7d1410740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563ac0c8cb60 con 0x563ac09de4d0 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.149+0000 7fa7d1410740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563ac0da7b50 con 0x563ac09ff1b0 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.149+0000 7fa7d1410740 10 client.0 _ll_register_callbacks cb 0x563ac0cb12b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.150+0000 7fa7c37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.150+0000 7fa7c37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:39996/0 (socket says 172.21.15.31:39996) 2024-04-25T16:12:23.152 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.150+0000 7fa7c37fe640 1 -- 172.21.15.31:0/3536768762 learned_addr learned my addr 172.21.15.31:0/3536768762 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.150+0000 7fa7c3fff640 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.150+0000 7fa7c37fe640 1 -- 172.21.15.31:0/3536768762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 msgr2=0x563ac0db0d90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.150+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 0x563ac0db0d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.150+0000 7fa7c37fe640 1 -- 172.21.15.31:0/3536768762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 msgr2=0x563ac0da98f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.150+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.150+0000 7fa7c37fe640 1 -- 172.21.15.31:0/3536768762 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac0a17210 con 0x563ac09de4d0 2024-04-25T16:12:23.153 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.151+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da9e30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa7b801a2b0 tx=0x7fa7b8040880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a79ed622bb979ca server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:23.154 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.153+0000 7fa7c3fff640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7ac0afcf0 0x7fa7ac0b2200 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x563ac0d9c230 tx=0x7fa7bc00c7f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.154+0000 7fa7a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:12:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.155+0000 7fa7a77fe640 1 -- 172.21.15.31:0/3536768762 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8073070 con 0x563ac09de4d0 2024-04-25T16:12:23.156 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.155+0000 7fa7d1410740 1 -- 172.21.15.31:0/3536768762 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7b8004b60 con 0x563ac09de4d0 2024-04-25T16:12:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.155+0000 7fa7a77fe640 1 -- 172.21.15.31:0/3536768762 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa7b80054d0 con 0x563ac09de4d0 2024-04-25T16:12:23.157 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.156+0000 7fa7a77fe640 1 -- 172.21.15.31:0/3536768762 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7b807f050 con 0x563ac09de4d0 2024-04-25T16:12:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.159+0000 7fa7d1410740 10 client.27470 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:23.161 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.159+0000 7fa7d1410740 1 -- 172.21.15.31:0/3536768762 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563ac0ae3620 con 0x563ac09de4d0 2024-04-25T16:12:23.164 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.162+0000 7fa7a77fe640 1 -- 172.21.15.31:0/3536768762 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7b8004b60 con 0x563ac09de4d0 2024-04-25T16:12:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.164+0000 7fa7a77fe640 1 -- 172.21.15.31:0/3536768762 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa7b800f040 con 0x563ac09de4d0 2024-04-25T16:12:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.164+0000 7fa7d1410740 10 client.27470 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.164+0000 7fa7d1410740 10 client.27470 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.164+0000 7fa7d1410740 20 client.27470 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.164+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563ac09bdd80 0x563ac0dd96f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.166 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.165+0000 7fa7d1410740 4 client.27470 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.165+0000 7fa7d1410740 1 -- 172.21.15.31:0/3536768762 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ac0d13070 con 0x563ac09bdd80 2024-04-25T16:12:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.165+0000 7fa7c3fff640 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563ac09bdd80 0x563ac0dd96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.170+0000 7fa7c3fff640 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563ac09bdd80 0x563ac0dd96f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d76dba45ec0d17b3 server_cookie=8d59866b2babf876 in_seq=0 out_seq=0 2024-04-25T16:12:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.170+0000 7fa7a77fe640 10 client.27470 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.177+0000 7fa7a77fe640 1 -- 172.21.15.31:0/3536768762 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563ac0d13070 con 0x563ac09bdd80 2024-04-25T16:12:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.177+0000 7fa7a77fe640 10 client.27470 handle_command_reply: tid=0 2024-04-25T16:12:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.177+0000 7fa7d1410740 1 client.27470 shutdown 2024-04-25T16:12:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.178+0000 7fa7d1410740 1 -- 172.21.15.31:0/3536768762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 msgr2=0x563ac0da9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:23.179 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.178+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da9e30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fa7b801a2b0 tx=0x7fa7b8040880 comp rx=0 tx=0).stop 2024-04-25T16:12:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.204+0000 7fa7d1410740 1 -- 172.21.15.31:0/3536768762 shutdown_connections 2024-04-25T16:12:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.204+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563ac09bdd80 0x563ac0dd96f0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.204+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 0x563ac0db0d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.204+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da9e30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.204+0000 7fa7d1410740 1 --2- 172.21.15.31:0/3536768762 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da98f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.204+0000 7fa7d1410740 1 -- 172.21.15.31:0/3536768762 >> 172.21.15.31:0/3536768762 conn(0x563ac09f3300 msgr2=0x563ac0d9e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.204+0000 7fa7a77fe640 0 client.27470 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.205+0000 7fa7d1410740 1 -- 172.21.15.31:0/3536768762 shutdown_connections 2024-04-25T16:12:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.206+0000 7fa7d1410740 1 -- 172.21.15.31:0/3536768762 wait complete. 2024-04-25T16:12:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.206+0000 7fa7d1410740 20 client.27470 trim_cache size 0 max 16384 2024-04-25T16:12:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.258+0000 7fa7d1410740 1 Processor -- start 2024-04-25T16:12:23.259 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.258+0000 7fa7d1410740 1 -- start start 2024-04-25T16:12:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.258+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.258+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da4210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.258+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 0x563ac0db0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.258+0000 7fa7d1410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41a0500 con 0x563ac09d62f0 2024-04-25T16:12:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.258+0000 7fa7d1410740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41a2040 con 0x563ac09de4d0 2024-04-25T16:12:23.260 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.258+0000 7fa7d1410740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41a0bd0 con 0x563ac09ff1b0 2024-04-25T16:12:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.259+0000 7fa7c8a29640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 0x563ac0db0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.260+0000 7fa7c3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.260+0000 7fa7c3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:53650/0 (socket says 172.21.15.31:53650) 2024-04-25T16:12:23.261 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.260+0000 7fa7c3fff640 1 -- 172.21.15.31:0/1398229899 learned_addr learned my addr 172.21.15.31:0/1398229899 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:23.262 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.260+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/1398229899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.262+0000 7fa7c8a29640 1 -- 172.21.15.31:0/1398229899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 msgr2=0x563ac0da4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.262+0000 7fa7c8a29640 1 --2- 172.21.15.31:0/1398229899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.262+0000 7fa7c8a29640 1 -- 172.21.15.31:0/1398229899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 msgr2=0x563ac0da3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.262+0000 7fa7c8a29640 1 --2- 172.21.15.31:0/1398229899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.266 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.262+0000 7fa7c8a29640 1 -- 172.21.15.31:0/1398229899 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ac0db11f0 con 0x563ac09ff1b0 2024-04-25T16:12:23.267 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.265+0000 7fa7c8a29640 1 --2- 172.21.15.31:0/1398229899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 0x563ac0db0cb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa7b406cfd0 tx=0x7fa7b40a7150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3ed9f0b7b15912a server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.270+0000 7fa7b1ffb640 1 -- 172.21.15.31:0/1398229899 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b40a7740 con 0x563ac09ff1b0 2024-04-25T16:12:23.271 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.270+0000 7fa7b1ffb640 1 -- 172.21.15.31:0/1398229899 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7b40abaf0 con 0x563ac09ff1b0 2024-04-25T16:12:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.272+0000 7fa7d1410740 1 -- 172.21.15.31:0/1398229899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 msgr2=0x563ac0db0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.272+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1398229899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 0x563ac0db0cb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa7b406cfd0 tx=0x7fa7b40a7150 comp rx=0 tx=0).stop 2024-04-25T16:12:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.272+0000 7fa7d1410740 1 -- 172.21.15.31:0/1398229899 shutdown_connections 2024-04-25T16:12:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.272+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1398229899 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09ff1b0 0x563ac0db0cb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.272+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1398229899 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0da4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.273 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.272+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1398229899 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09d62f0 0x563ac0da3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.274 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.272+0000 7fa7d1410740 1 -- 172.21.15.31:0/1398229899 >> 172.21.15.31:0/1398229899 conn(0x563ac09f3300 msgr2=0x563ac0da7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.279+0000 7fa7d1410740 1 -- 172.21.15.31:0/1398229899 shutdown_connections 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.282+0000 7fa7d1410740 1 -- 172.21.15.31:0/1398229899 wait complete. 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.282+0000 7fa7d1410740 1 -- start start 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.283+0000 7fa7d1410740 1 Processor -- start 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.283+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09d62f0 0x563ac0db0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.283+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0daf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.283+0000 7fa7d1410740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09ff1b0 0x563ac0daf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.283+0000 7fa7d1410740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41a1970 con 0x563ac09ff1b0 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.283+0000 7fa7d1410740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c41a12a0 con 0x563ac09de4d0 2024-04-25T16:12:23.287 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.283+0000 7fa7d1410740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa7c4073b40 con 0x563ac09d62f0 2024-04-25T16:12:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.283+0000 7fa7d1410740 10 client.0 _ll_register_callbacks cb 0x7fa7b8016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.284+0000 7fa7c37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0daf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.284+0000 7fa7c37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0daf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:40008/0 (socket says 172.21.15.31:40008) 2024-04-25T16:12:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.284+0000 7fa7c37fe640 1 -- 172.21.15.31:0/1772561195 learned_addr learned my addr 172.21.15.31:0/1772561195 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.287+0000 7fa7c8a29640 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09ff1b0 0x563ac0daf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.288 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.287+0000 7fa7c3fff640 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09d62f0 0x563ac0db0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.287+0000 7fa7c37fe640 1 -- 172.21.15.31:0/1772561195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09d62f0 msgr2=0x563ac0db0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.287+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09d62f0 0x563ac0db0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.287+0000 7fa7c37fe640 1 -- 172.21.15.31:0/1772561195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09ff1b0 msgr2=0x563ac0daf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.287+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09ff1b0 0x563ac0daf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.289 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.287+0000 7fa7c37fe640 1 -- 172.21.15.31:0/1772561195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ac0dafbc0 con 0x563ac09de4d0 2024-04-25T16:12:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.290+0000 7fa7c37fe640 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0daf070 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8012d10 tx=0x7fa7b8005020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69146b2452bc186f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:23.291 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.290+0000 7fa7a77fe640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:12:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.291+0000 7fa7a77fe640 1 -- 172.21.15.31:0/1772561195 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b8073070 con 0x563ac09de4d0 2024-04-25T16:12:23.292 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.291+0000 7fa7d1410740 1 -- 172.21.15.31:0/1772561195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7b40abaf0 con 0x563ac09de4d0 2024-04-25T16:12:23.293 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.291+0000 7fa7a77fe640 1 -- 172.21.15.31:0/1772561195 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa7b8004ce0 con 0x563ac09de4d0 2024-04-25T16:12:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.293+0000 7fa7a77fe640 1 -- 172.21.15.31:0/1772561195 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa7b8018040 con 0x563ac09de4d0 2024-04-25T16:12:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.293+0000 7fa7d1410740 10 client.27476 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:23.294 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.293+0000 7fa7d1410740 1 -- 172.21.15.31:0/1772561195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563ac0e40830 con 0x563ac09de4d0 2024-04-25T16:12:23.296 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.295+0000 7fa7a77fe640 1 -- 172.21.15.31:0/1772561195 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa7b40abaf0 con 0x563ac09de4d0 2024-04-25T16:12:23.297 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.296+0000 7fa7a77fe640 1 -- 172.21.15.31:0/1772561195 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fa7b8042600 con 0x563ac09de4d0 2024-04-25T16:12:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.296+0000 7fa7d1410740 10 client.27476 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.297+0000 7fa7d1410740 10 client.27476 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.297+0000 7fa7d1410740 20 client.27476 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.297+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563ac09931c0 0x563ac0e43200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.297+0000 7fa7d1410740 4 client.27476 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:12:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.297+0000 7fa7d1410740 1 -- 172.21.15.31:0/1772561195 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563ac0d0cee0 con 0x563ac09931c0 2024-04-25T16:12:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.299+0000 7fa7c3fff640 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563ac09931c0 0x563ac0e43200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:23.304 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.303+0000 7fa7c3fff640 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563ac09931c0 0x563ac0e43200 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf4af2af25dcc6a4 server_cookie=29e98dc5210f783c in_seq=0 out_seq=0 2024-04-25T16:12:23.308 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:23.307+0000 7fa7a77fe640 10 client.27476 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:23.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:23 smithi063 ceph-mon[32413]: pgmap v683: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 791 MiB/s wr, 1.78k op/s 2024-04-25T16:12:23.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:23 smithi186 ceph-mon[33258]: pgmap v683: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 791 MiB/s wr, 1.78k op/s 2024-04-25T16:12:23.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:23 smithi031 ceph-mon[29140]: pgmap v683: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 791 MiB/s wr, 1.78k op/s 2024-04-25T16:12:23.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.756+0000 7f37e4f57640 1 -- 172.21.15.31:0/1619029706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f37e011a100 con 0x7f37e010a420 2024-04-25T16:12:23.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.764+0000 7f37cf7fe640 1 -- 172.21.15.31:0/1619029706 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11917 (secure 0 0 0) 0x7f37d00c8990 con 0x7f37e010a420 2024-04-25T16:12:23.769 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:12:23.769 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:05:03.322392+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[24521,24535,24563,24553,14631]},"id":1}]} 2024-04-25T16:12:23.770 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 29 2024-04-25T16:12:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.778+0000 7f37cd7fa640 1 -- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37c80afc70 msgr2=0x7f37c80b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.778+0000 7f37cd7fa640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37c80afc70 0x7f37c80b2180 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f37d8005c40 tx=0x7f37d8005bd0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.778+0000 7f37cd7fa640 1 -- 172.21.15.31:0/1619029706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37e010a420 msgr2=0x7f37e0117010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:23.780 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.778+0000 7f37cd7fa640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37e010a420 0x7f37e0117010 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f37d002aa00 tx=0x7f37d00056e0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.786+0000 7f37cd7fa640 1 -- 172.21.15.31:0/1619029706 shutdown_connections 2024-04-25T16:12:23.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.786+0000 7f37cd7fa640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f37c80afc70 0x7f37c80b2180 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.786+0000 7f37cd7fa640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f37e0110030 0x7f37e01184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.786+0000 7f37cd7fa640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37e010b310 0x7f37e0117550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.786+0000 7f37cd7fa640 1 --2- 172.21.15.31:0/1619029706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f37e010a420 0x7f37e0117010 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:23.789 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.786+0000 7f37cd7fa640 1 -- 172.21.15.31:0/1619029706 >> 172.21.15.31:0/1619029706 conn(0x7f37e00704e0 msgr2=0x7f37e010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.797+0000 7f37cd7fa640 1 -- 172.21.15.31:0/1619029706 shutdown_connections 2024-04-25T16:12:23.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:23.798+0000 7f37cd7fa640 1 -- 172.21.15.31:0/1619029706 wait complete. 2024-04-25T16:12:23.904 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:12:24.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.344+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2971757859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e0103ac0 msgr2=0x7fd7e0103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:24.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.344+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2971757859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e0103ac0 0x7fd7e0103ed0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd7d8009be0 tx=0x7fd7d802a6c0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.345+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2971757859 shutdown_connections 2024-04-25T16:12:24.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.345+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2971757859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e01087c0 0x7fd7e010abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.345+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2971757859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e0103ac0 0x7fd7e0103ed0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.345+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2971757859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e0102880 0x7fd7e0103580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.347 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.345+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2971757859 >> 172.21.15.31:0/2971757859 conn(0x7fd7e00fda70 msgr2=0x7fd7e00fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:24.359 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:24 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1619029706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:12:24.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.363+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2971757859 shutdown_connections 2024-04-25T16:12:24.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.363+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2971757859 wait complete. 2024-04-25T16:12:24.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.365+0000 7fd7e6b34640 1 Processor -- start 2024-04-25T16:12:24.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.365+0000 7fd7e6b34640 1 -- start start 2024-04-25T16:12:24.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.365+0000 7fd7e6b34640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e0102880 0x7fd7e01a08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.365+0000 7fd7e6b34640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e0103ac0 0x7fd7e01a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.365+0000 7fd7e6b34640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e01087c0 0x7fd7e01a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.366+0000 7fd7e6b34640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e010e360 con 0x7fd7e0102880 2024-04-25T16:12:24.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.366+0000 7fd7e6b34640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e010e1e0 con 0x7fd7e0103ac0 2024-04-25T16:12:24.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.366+0000 7fd7e6b34640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e010e4e0 con 0x7fd7e01087c0 2024-04-25T16:12:24.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.367+0000 7fd7dffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e0103ac0 0x7fd7e01a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.367+0000 7fd7dffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e0103ac0 0x7fd7e01a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:54864/0 (socket says 172.21.15.31:54864) 2024-04-25T16:12:24.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.367+0000 7fd7dffff640 1 -- 172.21.15.31:0/2054803878 learned_addr learned my addr 172.21.15.31:0/2054803878 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:24.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.368+0000 7fd7dffff640 1 -- 172.21.15.31:0/2054803878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e01087c0 msgr2=0x7fd7e01a7b30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:12:24.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.368+0000 7fd7dffff640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e01087c0 0x7fd7e01a7b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.368+0000 7fd7dffff640 1 -- 172.21.15.31:0/2054803878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e0102880 msgr2=0x7fd7e01a08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:24.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.368+0000 7fd7dffff640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e0102880 0x7fd7e01a08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.368+0000 7fd7dffff640 1 -- 172.21.15.31:0/2054803878 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7e01a8170 con 0x7fd7e0103ac0 2024-04-25T16:12:24.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.369+0000 7fd7dffff640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e0103ac0 0x7fd7e01a0ec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd7d802acb0 tx=0x7fd7d8030ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:24.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.393+0000 7fd7ddffb640 1 -- 172.21.15.31:0/2054803878 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8041020 con 0x7fd7e0103ac0 2024-04-25T16:12:24.395 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.393+0000 7fd7ddffb640 1 -- 172.21.15.31:0/2054803878 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7d80318f0 con 0x7fd7e0103ac0 2024-04-25T16:12:24.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.395+0000 7fd7ddffb640 1 -- 172.21.15.31:0/2054803878 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d8036040 con 0x7fd7e0103ac0 2024-04-25T16:12:24.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.395+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2054803878 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7e01a8400 con 0x7fd7e0103ac0 2024-04-25T16:12:24.400 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.395+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2054803878 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7e01a88c0 con 0x7fd7e0103ac0 2024-04-25T16:12:24.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.405+0000 7fd7ed9e6740 1 Processor -- start 2024-04-25T16:12:24.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.405+0000 7fd7ed9e6740 1 -- start start 2024-04-25T16:12:24.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.406+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7449e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.406+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3794d0 0x55a9cc744f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.406+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3816b0 0x55a9cc745bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.407+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a9cc6a8d80 con 0x55a9cc3794d0 2024-04-25T16:12:24.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.407+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a9cc662a20 con 0x55a9cc3816b0 2024-04-25T16:12:24.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.407+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a9cc6a4d40 con 0x55a9cc3a2390 2024-04-25T16:12:24.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.407+0000 7fd7ddffb640 1 -- 172.21.15.31:0/2054803878 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd7d8044020 con 0x7fd7e0103ac0 2024-04-25T16:12:24.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.409+0000 7fd7ddffb640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7c80afa50 0x7fd7c80b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.409+0000 7fd7e48d0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3794d0 0x55a9cc744f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.409+0000 7fd7e48d0640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3794d0 0x55a9cc744f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37108/0 (socket says 172.21.15.31:37108) 2024-04-25T16:12:24.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.409+0000 7fd7e48d0640 1 -- 172.21.15.31:0/1837225273 learned_addr learned my addr 172.21.15.31:0/1837225273 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:24.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.409+0000 7fd7ddffb640 1 -- 172.21.15.31:0/2054803878 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd7d80fef40 con 0x7fd7e0103ac0 2024-04-25T16:12:24.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.410+0000 7fd7e50d1640 1 --2- 172.21.15.31:0/1837225273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7449e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.410+0000 7fd7dffff640 1 --2- 172.21.15.31:0/1837225273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3816b0 0x55a9cc745bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.411+0000 7fd7e50d1640 1 -- 172.21.15.31:0/1837225273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3816b0 msgr2=0x55a9cc745bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.411+0000 7fd7e50d1640 1 --2- 172.21.15.31:0/1837225273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3816b0 0x55a9cc745bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.411+0000 7fd7e50d1640 1 -- 172.21.15.31:0/1837225273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3794d0 msgr2=0x55a9cc744f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.411+0000 7fd7e50d1640 1 --2- 172.21.15.31:0/1837225273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3794d0 0x55a9cc744f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.411+0000 7fd7e50d1640 1 -- 172.21.15.31:0/1837225273 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9cc66dc30 con 0x55a9cc3a2390 2024-04-25T16:12:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.412+0000 7fd7e50d1640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7c80afa50 0x7fd7c80b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.414 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.413+0000 7fd7e50d1640 1 --2- 172.21.15.31:0/1837225273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7449e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009b10 tx=0x7fd7d4009f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2eeaef1bc87e138 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:24.415 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.413+0000 7fd7e50d1640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7c80afa50 0x7fd7c80b1f60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4008000 tx=0x7fd7d4016000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:24.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.434+0000 7fd7c6ffd640 1 -- 172.21.15.31:0/1837225273 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4010a30 con 0x55a9cc3a2390 2024-04-25T16:12:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.434+0000 7fd7c6ffd640 1 -- 172.21.15.31:0/1837225273 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7d4012450 con 0x55a9cc3a2390 2024-04-25T16:12:24.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.434+0000 7fd7c6ffd640 1 -- 172.21.15.31:0/1837225273 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d401b690 con 0x55a9cc3a2390 2024-04-25T16:12:24.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.437+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/1837225273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 msgr2=0x55a9cc7449e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:24.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.437+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/1837225273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7449e0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009b10 tx=0x7fd7d4009f10 comp rx=0 tx=0).stop 2024-04-25T16:12:24.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.438+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/1837225273 shutdown_connections 2024-04-25T16:12:24.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.438+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/1837225273 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3816b0 0x55a9cc745bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.438+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/1837225273 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3794d0 0x55a9cc744f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.438+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/1837225273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7449e0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.438+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/1837225273 >> 172.21.15.31:0/1837225273 conn(0x55a9cc3964e0 msgr2=0x55a9cc7401f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:24.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.438+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/1837225273 shutdown_connections 2024-04-25T16:12:24.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.438+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/1837225273 wait complete. 2024-04-25T16:12:24.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.439+0000 7fd7ed9e6740 1 -- start start 2024-04-25T16:12:24.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.440+0000 7fd7ed9e6740 1 Processor -- start 2024-04-25T16:12:24.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.441+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 0x55a9cc74e600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.442 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.441+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc74eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.441+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc755aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.442+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55a9cc746700 con 0x55a9cc3816b0 2024-04-25T16:12:24.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.442+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55a9cc62fdc0 con 0x55a9cc3794d0 2024-04-25T16:12:24.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.442+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55a9cc746880 con 0x55a9cc3a2390 2024-04-25T16:12:24.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.443+0000 7fd7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc74eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.443+0000 7fd7ed9e6740 10 client.0 _ll_register_callbacks cb 0x55a9cc64b600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:24.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.443+0000 7fd7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc74eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37122/0 (socket says 172.21.15.31:37122) 2024-04-25T16:12:24.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.443+0000 7fd7dffff640 1 -- 172.21.15.31:0/3375836296 learned_addr learned my addr 172.21.15.31:0/3375836296 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:24.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.443+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 0x55a9cc74e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.443+0000 7fd7e50d1640 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc755aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.444+0000 7fd7dffff640 1 -- 172.21.15.31:0/3375836296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 msgr2=0x55a9cc755aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:24.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.444+0000 7fd7dffff640 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc755aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.444+0000 7fd7dffff640 1 -- 172.21.15.31:0/3375836296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 msgr2=0x55a9cc74e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:24.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.444+0000 7fd7dffff640 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 0x55a9cc74e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.444+0000 7fd7dffff640 1 -- 172.21.15.31:0/3375836296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9cc3ba3f0 con 0x55a9cc3816b0 2024-04-25T16:12:24.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.445+0000 7fd7dffff640 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc74eb40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80c3400 tx=0x7fd7d8003da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1833c17f7afa87d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:24.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.445+0000 7fd7bffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:12:24.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.447+0000 7fd7bffff640 1 -- 172.21.15.31:0/3375836296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80c73d0 con 0x55a9cc3816b0 2024-04-25T16:12:24.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.447+0000 7fd7bffff640 1 -- 172.21.15.31:0/3375836296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd7d80cffb0 con 0x55a9cc3816b0 2024-04-25T16:12:24.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.447+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3375836296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7d4012450 con 0x55a9cc3816b0 2024-04-25T16:12:24.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.449+0000 7fd7bffff640 1 -- 172.21.15.31:0/3375836296 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd7d80cb780 con 0x55a9cc3816b0 2024-04-25T16:12:24.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.450+0000 7fd7ed9e6740 10 client.17775 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:24.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.450+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3375836296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a9cc4847d0 con 0x55a9cc3816b0 2024-04-25T16:12:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.451+0000 7fd7bffff640 1 -- 172.21.15.31:0/3375836296 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7d80e1070 con 0x55a9cc3816b0 2024-04-25T16:12:24.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.451+0000 7fd7bffff640 1 -- 172.21.15.31:0/3375836296 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd7d8004b40 con 0x55a9cc3816b0 2024-04-25T16:12:24.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.451+0000 7fd7ed9e6740 10 client.17775 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:24.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.451+0000 7fd7ed9e6740 10 client.17775 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:12:24.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.451+0000 7fd7ed9e6740 20 client.17775 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:24.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.451+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55a9cc360f60 0x55a9cc75c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.451+0000 7fd7ed9e6740 4 client.17775 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:24.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.451+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3375836296 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd7d4012450 con 0x55a9cc360f60 2024-04-25T16:12:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.454+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55a9cc360f60 0x55a9cc75c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.455+0000 7fd7bffff640 10 client.17775 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:12:24.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.455+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55a9cc360f60 0x55a9cc75c1d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e8d0cf0726502df8 server_cookie=15040de7a649d2a0 in_seq=0 out_seq=0 2024-04-25T16:12:24.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.456+0000 7fd7bffff640 1 -- 172.21.15.31:0/3375836296 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fd7d4012450 con 0x55a9cc360f60 2024-04-25T16:12:24.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.456+0000 7fd7bffff640 10 client.17775 handle_command_reply: tid=0 2024-04-25T16:12:24.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.456+0000 7fd7ed9e6740 1 client.17775 shutdown 2024-04-25T16:12:24.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.462+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3375836296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 msgr2=0x55a9cc74eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:24.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.462+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc74eb40 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80c3400 tx=0x7fd7d8003da0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3375836296 shutdown_connections 2024-04-25T16:12:24.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55a9cc360f60 0x55a9cc75c1d0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc755aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc74eb40 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3375836296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 0x55a9cc74e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3375836296 >> 172.21.15.31:0/3375836296 conn(0x55a9cc3964e0 msgr2=0x55a9cc73f3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:24.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7bffff640 0 client.17775 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:12:24.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3375836296 shutdown_connections 2024-04-25T16:12:24.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3375836296 wait complete. 2024-04-25T16:12:24.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.463+0000 7fd7ed9e6740 20 client.17775 trim_cache size 0 max 16384 2024-04-25T16:12:24.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.492+0000 7fd7ed9e6740 1 Processor -- start 2024-04-25T16:12:24.494 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.493+0000 7fd7ed9e6740 1 -- start start 2024-04-25T16:12:24.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.493+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 0x55a9cc749460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.495 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.494+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc7559c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.494+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7499a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.494+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e01b14f0 con 0x55a9cc3816b0 2024-04-25T16:12:24.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.494+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e01b3030 con 0x55a9cc3794d0 2024-04-25T16:12:24.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.494+0000 7fd7e50d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.495+0000 7fd7e50d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7499a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56624/0 (socket says 172.21.15.31:56624) 2024-04-25T16:12:24.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.495+0000 7fd7e50d1640 1 -- 172.21.15.31:0/3828292754 learned_addr learned my addr 172.21.15.31:0/3828292754 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:24.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.495+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 0x55a9cc749460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.495+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3828292754 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e01b1bc0 con 0x55a9cc3a2390 2024-04-25T16:12:24.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.496+0000 7fd7e48d0640 1 -- 172.21.15.31:0/3828292754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 msgr2=0x55a9cc7499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:24.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.496+0000 7fd7dffff640 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc7559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.496+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.496+0000 7fd7e48d0640 1 -- 172.21.15.31:0/3828292754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 msgr2=0x55a9cc7559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:24.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.496+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc7559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.496+0000 7fd7e48d0640 1 -- 172.21.15.31:0/3828292754 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9cc731ba0 con 0x55a9cc3794d0 2024-04-25T16:12:24.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.496+0000 7fd7e50d1640 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:12:24.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.496+0000 7fd7dffff640 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc7559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:12:24.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.498+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 0x55a9cc749460 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55a9cc75c970 tx=0x7fd7d0010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9db041bec69a2b81 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:24.522 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.521+0000 7fd7c67fc640 1 -- 172.21.15.31:0/3828292754 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d0005220 con 0x55a9cc3794d0 2024-04-25T16:12:24.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.521+0000 7fd7c67fc640 1 -- 172.21.15.31:0/3828292754 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7d0010e20 con 0x55a9cc3794d0 2024-04-25T16:12:24.523 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.521+0000 7fd7c67fc640 1 -- 172.21.15.31:0/3828292754 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d001c710 con 0x55a9cc3794d0 2024-04-25T16:12:24.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.523+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3828292754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 msgr2=0x55a9cc749460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:24.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.523+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 0x55a9cc749460 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x55a9cc75c970 tx=0x7fd7d0010800 comp rx=0 tx=0).stop 2024-04-25T16:12:24.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.524+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3828292754 shutdown_connections 2024-04-25T16:12:24.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.524+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3a2390 0x55a9cc7499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.524+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc7559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.524+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/3828292754 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3794d0 0x55a9cc749460 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.524+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3828292754 >> 172.21.15.31:0/3828292754 conn(0x55a9cc3964e0 msgr2=0x55a9cc74e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:24.532 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.531+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3828292754 shutdown_connections 2024-04-25T16:12:24.533 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.532+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/3828292754 wait complete. 2024-04-25T16:12:24.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.533+0000 7fd7ed9e6740 1 -- start start 2024-04-25T16:12:24.534 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.533+0000 7fd7ed9e6740 1 Processor -- start 2024-04-25T16:12:24.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.534+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3794d0 0x55a9cc753130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.534+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc753670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.535+0000 7fd7ed9e6740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3a2390 0x55a9cc7545a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.535+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e01b2960 con 0x55a9cc3816b0 2024-04-25T16:12:24.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.535+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e01b2290 con 0x55a9cc3a2390 2024-04-25T16:12:24.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.535+0000 7fd7ed9e6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd7e01af4d0 con 0x55a9cc3794d0 2024-04-25T16:12:24.537 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.535+0000 7fd7ed9e6740 10 client.0 _ll_register_callbacks cb 0x7fd7d80c71a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:24.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.537+0000 7fd7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc753670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.537+0000 7fd7dffff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc753670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37146/0 (socket says 172.21.15.31:37146) 2024-04-25T16:12:24.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.537+0000 7fd7dffff640 1 -- 172.21.15.31:0/2715721930 learned_addr learned my addr 172.21.15.31:0/2715721930 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:24.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.538+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3794d0 0x55a9cc753130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.539+0000 7fd7e50d1640 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3a2390 0x55a9cc7545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.542+0000 7fd7dffff640 1 -- 172.21.15.31:0/2715721930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3794d0 msgr2=0x55a9cc753130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.542+0000 7fd7dffff640 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3794d0 0x55a9cc753130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.542+0000 7fd7dffff640 1 -- 172.21.15.31:0/2715721930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3a2390 msgr2=0x55a9cc7545a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.542+0000 7fd7dffff640 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3a2390 0x55a9cc7545a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.542+0000 7fd7dffff640 1 -- 172.21.15.31:0/2715721930 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9cc754b60 con 0x55a9cc3816b0 2024-04-25T16:12:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.545+0000 7fd7dffff640 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc753670 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80c70d0 tx=0x7fd7d802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8982ca1aad3c53c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.545+0000 7fd7bffff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:12:24.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.546+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3794d0 0x55a9cc753130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:12:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.548+0000 7fd7bffff640 1 -- 172.21.15.31:0/2715721930 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d80c6d20 con 0x55a9cc3816b0 2024-04-25T16:12:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.548+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/2715721930 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd7d0010e20 con 0x55a9cc3816b0 2024-04-25T16:12:24.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.548+0000 7fd7bffff640 1 -- 172.21.15.31:0/2715721930 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd7d8004b40 con 0x55a9cc3816b0 2024-04-25T16:12:24.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.550+0000 7fd7bffff640 1 -- 172.21.15.31:0/2715721930 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fd7d80c73d0 con 0x55a9cc3816b0 2024-04-25T16:12:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.551+0000 7fd7ed9e6740 10 client.17778 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:24.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.551+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/2715721930 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55a9cc755dc0 con 0x55a9cc3816b0 2024-04-25T16:12:24.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.555+0000 7fd7bffff640 1 -- 172.21.15.31:0/2715721930 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd7d0010e20 con 0x55a9cc3816b0 2024-04-25T16:12:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.557+0000 7fd7bffff640 1 -- 172.21.15.31:0/2715721930 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fd7d80cec10 con 0x55a9cc3816b0 2024-04-25T16:12:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.557+0000 7fd7ed9e6740 10 client.17778 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:24.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.557+0000 7fd7ed9e6740 10 client.17778 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:12:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.557+0000 7fd7ed9e6740 20 client.17778 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.557+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55a9cc3363a0 0x55a9cc7c6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.557+0000 7fd7ed9e6740 4 client.17778 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:12:24.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.557+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/2715721930 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a9cc6b6990 con 0x55a9cc3363a0 2024-04-25T16:12:24.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.561+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55a9cc3363a0 0x55a9cc7c6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.563+0000 7fd7e48d0640 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55a9cc3363a0 0x55a9cc7c6e20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7fc6641bf4950f67 server_cookie=4a80dc4079f5ff2a in_seq=0 out_seq=0 2024-04-25T16:12:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.563+0000 7fd7bffff640 10 client.17778 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:12:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.564+0000 7fd7bffff640 1 -- 172.21.15.31:0/2715721930 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a9cc6b6990 con 0x55a9cc3363a0 2024-04-25T16:12:24.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.564+0000 7fd7bffff640 10 client.17778 handle_command_reply: tid=0 2024-04-25T16:12:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.564+0000 7fd7ed9e6740 1 client.17778 shutdown 2024-04-25T16:12:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.565+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/2715721930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 msgr2=0x55a9cc753670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:24.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.565+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc753670 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fd7d80c70d0 tx=0x7fd7d802a6c0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.565+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/2715721930 shutdown_connections 2024-04-25T16:12:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.565+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x55a9cc3363a0 0x55a9cc7c6e20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.565+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55a9cc3a2390 0x55a9cc7545a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.565+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55a9cc3816b0 0x55a9cc753670 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.565+0000 7fd7ed9e6740 1 --2- 172.21.15.31:0/2715721930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a9cc3794d0 0x55a9cc753130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.565+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/2715721930 >> 172.21.15.31:0/2715721930 conn(0x55a9cc3964e0 msgr2=0x55a9cc74e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.566+0000 7fd7bffff640 0 client.17778 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:12:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.566+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/2715721930 shutdown_connections 2024-04-25T16:12:24.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.566+0000 7fd7ed9e6740 1 -- 172.21.15.31:0/2715721930 wait complete. 2024-04-25T16:12:24.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.566+0000 7fd7ed9e6740 20 client.17778 trim_cache size 0 max 16384 2024-04-25T16:12:24.569 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2054803878 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7c80afa50 msgr2=0x7fd7c80b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7c80afa50 0x7fd7c80b1f60 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4008000 tx=0x7fd7d4016000 comp rx=0 tx=0).stop 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2054803878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e0103ac0 msgr2=0x7fd7e01a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e0103ac0 0x7fd7e01a0ec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd7d802acb0 tx=0x7fd7d8030ca0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2054803878 shutdown_connections 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd7c80afa50 0x7fd7c80b1f60 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7e01087c0 0x7fd7e01a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e0103ac0 0x7fd7e01a0ec0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 --2- 172.21.15.31:0/2054803878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd7e0102880 0x7fd7e01a08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2054803878 >> 172.21.15.31:0/2054803878 conn(0x7fd7e00fda70 msgr2=0x7fd7e00ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:24.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.568+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2054803878 shutdown_connections 2024-04-25T16:12:24.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:24.569+0000 7fd7e6b34640 1 -- 172.21.15.31:0/2054803878 wait complete. 2024-04-25T16:12:24.614 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:12:24.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:24 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1619029706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:12:24.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:24 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1619029706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:12:25.001 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.000+0000 7f15e9522640 1 -- 172.21.15.31:0/3839762102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e410a240 msgr2=0x7f15e410aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.002 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.000+0000 7f15e9522640 1 --2- 172.21.15.31:0/3839762102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e410a240 0x7f15e410aff0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f15d80085a0 tx=0x7f15d802a210 comp rx=0 tx=0).stop 2024-04-25T16:12:25.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.020+0000 7f15e9522640 1 -- 172.21.15.31:0/3839762102 shutdown_connections 2024-04-25T16:12:25.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.020+0000 7f15e9522640 1 --2- 172.21.15.31:0/3839762102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e41102c0 0x7f15e41126e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.020+0000 7f15e9522640 1 --2- 172.21.15.31:0/3839762102 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15e410a760 0x7f15e410b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.021 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.020+0000 7f15e9522640 1 --2- 172.21.15.31:0/3839762102 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e410a240 0x7f15e410aff0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.022 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.020+0000 7f15e9522640 1 -- 172.21.15.31:0/3839762102 >> 172.21.15.31:0/3839762102 conn(0x7f15e4070780 msgr2=0x7f15e4072c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.025+0000 7f15e9522640 1 -- 172.21.15.31:0/3839762102 shutdown_connections 2024-04-25T16:12:25.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.026+0000 7f15e9522640 1 -- 172.21.15.31:0/3839762102 wait complete. 2024-04-25T16:12:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.027+0000 7f15e9522640 1 Processor -- start 2024-04-25T16:12:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.027+0000 7f15e9522640 1 -- start start 2024-04-25T16:12:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.027+0000 7f15e9522640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e410a240 0x7f15e41a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.027+0000 7f15e9522640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15e410a760 0x7f15e41a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.027+0000 7f15e9522640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e41102c0 0x7f15e41a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.027+0000 7f15e9522640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4115a70 con 0x7f15e410a240 2024-04-25T16:12:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.027+0000 7f15e9522640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41158f0 con 0x7f15e410a760 2024-04-25T16:12:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.027+0000 7f15e9522640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15e4115bf0 con 0x7f15e41102c0 2024-04-25T16:12:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.028+0000 7f15e37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15e410a760 0x7f15e41a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.028+0000 7f15e37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15e410a760 0x7f15e41a0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:54922/0 (socket says 172.21.15.31:54922) 2024-04-25T16:12:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.028+0000 7f15e37fe640 1 -- 172.21.15.31:0/1552152640 learned_addr learned my addr 172.21.15.31:0/1552152640 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.030 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.028+0000 7f15e3fff640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e410a240 0x7f15e41a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.028+0000 7f15e8d21640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e41102c0 0x7f15e41a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.029+0000 7f15e37fe640 1 -- 172.21.15.31:0/1552152640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e41102c0 msgr2=0x7f15e41a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.029+0000 7f15e37fe640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e41102c0 0x7f15e41a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.029+0000 7f15e37fe640 1 -- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e410a240 msgr2=0x7f15e41a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.029+0000 7f15e37fe640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e410a240 0x7f15e41a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.029+0000 7f15e37fe640 1 -- 172.21.15.31:0/1552152640 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e41a2130 con 0x7f15e410a760 2024-04-25T16:12:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.029+0000 7f15e37fe640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15e410a760 0x7f15e41a0c60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00acc0 tx=0x7f15dc007590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.031 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.030+0000 7f15e17fa640 1 -- 172.21.15.31:0/1552152640 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc015600 con 0x7f15e410a760 2024-04-25T16:12:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.030+0000 7f15e9522640 1 -- 172.21.15.31:0/1552152640 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15e41a2420 con 0x7f15e410a760 2024-04-25T16:12:25.033 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.031+0000 7f15e9522640 1 -- 172.21.15.31:0/1552152640 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15e41a29c0 con 0x7f15e410a760 2024-04-25T16:12:25.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.035+0000 7f15f1633740 1 Processor -- start 2024-04-25T16:12:25.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.035+0000 7f15f1633740 1 -- start start 2024-04-25T16:12:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.035+0000 7f15f1633740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 0x55e7172bd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.035+0000 7f15f1633740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ef24d0 0x55e7172bdf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.035+0000 7f15f1633740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 0x55e7172bebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.035+0000 7f15f1633740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e717221d80 con 0x55e716f1b390 2024-04-25T16:12:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.035+0000 7f15f1633740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e7171dba20 con 0x55e716ef24d0 2024-04-25T16:12:25.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.035+0000 7f15f1633740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e71721dd40 con 0x55e716efa6b0 2024-04-25T16:12:25.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.037+0000 7f15e3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ef24d0 0x55e7172bdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.037+0000 7f15e8d21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 0x55e7172bd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.037+0000 7f15e8d21640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 0x55e7172bd9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37176/0 (socket says 172.21.15.31:37176) 2024-04-25T16:12:25.038 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.037+0000 7f15e8d21640 1 -- 172.21.15.31:0/255144948 learned_addr learned my addr 172.21.15.31:0/255144948 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.038+0000 7f15e3fff640 1 -- 172.21.15.31:0/255144948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 msgr2=0x55e7172bebf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:25.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.038+0000 7f15e3fff640 1 --2- 172.21.15.31:0/255144948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 0x55e7172bebf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.038+0000 7f15e3fff640 1 -- 172.21.15.31:0/255144948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 msgr2=0x55e7172bd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.038+0000 7f15e3fff640 1 --2- 172.21.15.31:0/255144948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 0x55e7172bd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.039+0000 7f15e3fff640 1 -- 172.21.15.31:0/255144948 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7171e6c30 con 0x55e716ef24d0 2024-04-25T16:12:25.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.040+0000 7f15e3fff640 1 --2- 172.21.15.31:0/255144948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ef24d0 0x55e7172bdf60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f15d80084c0 tx=0x7f15d8004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e296c6aaa75e196 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.040+0000 7f15e17fa640 1 -- 172.21.15.31:0/1552152640 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15dc015c50 con 0x7f15e410a760 2024-04-25T16:12:25.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.040+0000 7f15d27fc640 1 -- 172.21.15.31:0/255144948 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d802ab10 con 0x55e716ef24d0 2024-04-25T16:12:25.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.041+0000 7f15d27fc640 1 -- 172.21.15.31:0/255144948 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15d802acb0 con 0x55e716ef24d0 2024-04-25T16:12:25.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.041+0000 7f15d27fc640 1 -- 172.21.15.31:0/255144948 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d803c6d0 con 0x55e716ef24d0 2024-04-25T16:12:25.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.042+0000 7f15e17fa640 1 -- 172.21.15.31:0/1552152640 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc014ab0 con 0x7f15e410a760 2024-04-25T16:12:25.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.042+0000 7f15e17fa640 1 -- 172.21.15.31:0/1552152640 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f15dc00c030 con 0x7f15e410a760 2024-04-25T16:12:25.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.043+0000 7f15e17fa640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f15cc0afb20 0x7f15cc0b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.044+0000 7f15e17fa640 1 -- 172.21.15.31:0/1552152640 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f15dc0dfcb0 con 0x7f15e410a760 2024-04-25T16:12:25.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.044+0000 7f15e8d21640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f15cc0afb20 0x7f15cc0b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.045+0000 7f15e8d21640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f15cc0afb20 0x7f15cc0b2030 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55e7172bea30 tx=0x7f15d400c4d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.059+0000 7f15f1633740 1 -- 172.21.15.31:0/255144948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ef24d0 msgr2=0x55e7172bdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.059+0000 7f15f1633740 1 --2- 172.21.15.31:0/255144948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ef24d0 0x55e7172bdf60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f15d80084c0 tx=0x7f15d8004600 comp rx=0 tx=0).stop 2024-04-25T16:12:25.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.060+0000 7f15f1633740 1 -- 172.21.15.31:0/255144948 shutdown_connections 2024-04-25T16:12:25.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.060+0000 7f15f1633740 1 --2- 172.21.15.31:0/255144948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 0x55e7172bebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.060+0000 7f15f1633740 1 --2- 172.21.15.31:0/255144948 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ef24d0 0x55e7172bdf60 secure :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f15d80084c0 tx=0x7f15d8004600 comp rx=0 tx=0).stop 2024-04-25T16:12:25.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.060+0000 7f15f1633740 1 --2- 172.21.15.31:0/255144948 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 0x55e7172bd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.060+0000 7f15f1633740 1 -- 172.21.15.31:0/255144948 >> 172.21.15.31:0/255144948 conn(0x55e716f0f4e0 msgr2=0x55e7172b91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.071 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.069+0000 7f15f1633740 1 -- 172.21.15.31:0/255144948 shutdown_connections 2024-04-25T16:12:25.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.069+0000 7f15f1633740 1 -- 172.21.15.31:0/255144948 wait complete. 2024-04-25T16:12:25.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.070+0000 7f15f1633740 1 -- start start 2024-04-25T16:12:25.072 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.071+0000 7f15f1633740 1 Processor -- start 2024-04-25T16:12:25.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.071+0000 7f15f1633740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716efa6b0 0x55e7172c78c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.073 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.071+0000 7f15f1633740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716f1b390 0x55e7172c7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.072+0000 7f15f1633740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ed9f60 0x55e7172ced60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.072+0000 7f15f1633740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55e7172bf6a0 con 0x55e716efa6b0 2024-04-25T16:12:25.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.072+0000 7f15f1633740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55e7171a8dc0 con 0x55e716ed9f60 2024-04-25T16:12:25.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.072+0000 7f15f1633740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55e7172bf820 con 0x55e716f1b390 2024-04-25T16:12:25.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.072+0000 7f15f1633740 10 client.0 _ll_register_callbacks cb 0x55e7171c4600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:25.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.073+0000 7f15e3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716efa6b0 0x55e7172c78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.073+0000 7f15e3fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716efa6b0 0x55e7172c78c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37184/0 (socket says 172.21.15.31:37184) 2024-04-25T16:12:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.073+0000 7f15e3fff640 1 -- 172.21.15.31:0/1754721368 learned_addr learned my addr 172.21.15.31:0/1754721368 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.073+0000 7f15e37fe640 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716f1b390 0x55e7172c7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.073+0000 7f15e8d21640 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ed9f60 0x55e7172ced60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.074+0000 7f15e3fff640 1 -- 172.21.15.31:0/1754721368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716f1b390 msgr2=0x55e7172c7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.074+0000 7f15e3fff640 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716f1b390 0x55e7172c7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.074+0000 7f15e3fff640 1 -- 172.21.15.31:0/1754721368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ed9f60 msgr2=0x55e7172ced60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.074+0000 7f15e3fff640 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ed9f60 0x55e7172ced60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.074+0000 7f15e3fff640 1 -- 172.21.15.31:0/1754721368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e716f333f0 con 0x55e716efa6b0 2024-04-25T16:12:25.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.075+0000 7f15e3fff640 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716efa6b0 0x55e7172c78c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f15d802a800 tx=0x7f15d800a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8282eedabcf6b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.075+0000 7f15c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:12:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.075+0000 7f15c77fe640 1 -- 172.21.15.31:0/1754721368 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15d802ab10 con 0x55e716efa6b0 2024-04-25T16:12:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.076+0000 7f15f1633740 1 -- 172.21.15.31:0/1754721368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15d802acb0 con 0x55e716efa6b0 2024-04-25T16:12:25.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.076+0000 7f15c77fe640 1 -- 172.21.15.31:0/1754721368 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f15d803c880 con 0x55e716efa6b0 2024-04-25T16:12:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.076+0000 7f15c77fe640 1 -- 172.21.15.31:0/1754721368 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f15d802ae60 con 0x55e716efa6b0 2024-04-25T16:12:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.077+0000 7f15f1633740 10 client.17790 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:25.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.077+0000 7f15f1633740 1 -- 172.21.15.31:0/1754721368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e716ffd7d0 con 0x55e716efa6b0 2024-04-25T16:12:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.078+0000 7f15c77fe640 1 -- 172.21.15.31:0/1754721368 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15d802acb0 con 0x55e716efa6b0 2024-04-25T16:12:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.078+0000 7f15c77fe640 1 -- 172.21.15.31:0/1754721368 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f15d8046a20 con 0x55e716efa6b0 2024-04-25T16:12:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.078+0000 7f15f1633740 10 client.17790 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.078+0000 7f15f1633740 10 client.17790 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:12:25.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.078+0000 7f15f1633740 20 client.17790 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.078+0000 7f15f1633740 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55e716eaf3a0 0x55e7172d5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.078+0000 7f15f1633740 4 client.17790 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.078+0000 7f15f1633740 1 -- 172.21.15.31:0/1754721368 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e71722f990 con 0x55e716eaf3a0 2024-04-25T16:12:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.078+0000 7f15e37fe640 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55e716eaf3a0 0x55e7172d5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.079+0000 7f15c77fe640 10 client.17790 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:12:25.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.079+0000 7f15e37fe640 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55e716eaf3a0 0x55e7172d5670 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=457f428c35d9350 server_cookie=34b7dc9653078061 in_seq=0 out_seq=0 2024-04-25T16:12:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.083+0000 7f15c77fe640 1 -- 172.21.15.31:0/1754721368 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55e71722f990 con 0x55e716eaf3a0 2024-04-25T16:12:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.084+0000 7f15c77fe640 10 client.17790 handle_command_reply: tid=0 2024-04-25T16:12:25.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.084+0000 7f15f1633740 1 client.17790 shutdown 2024-04-25T16:12:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.099+0000 7f15f1633740 1 -- 172.21.15.31:0/1754721368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716efa6b0 msgr2=0x55e7172c78c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.099+0000 7f15f1633740 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716efa6b0 0x55e7172c78c0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f15d802a800 tx=0x7f15d800a7b0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15f1633740 1 -- 172.21.15.31:0/1754721368 shutdown_connections 2024-04-25T16:12:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15f1633740 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716ed9f60 0x55e7172ced60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15f1633740 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55e716eaf3a0 0x55e7172d5670 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15f1633740 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716f1b390 0x55e7172c7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15f1633740 1 --2- 172.21.15.31:0/1754721368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716efa6b0 0x55e7172c78c0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15c77fe640 0 client.17790 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:12:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15f1633740 1 -- 172.21.15.31:0/1754721368 >> 172.21.15.31:0/1754721368 conn(0x55e716f0f4e0 msgr2=0x55e7172b79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15f1633740 1 -- 172.21.15.31:0/1754721368 shutdown_connections 2024-04-25T16:12:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15f1633740 1 -- 172.21.15.31:0/1754721368 wait complete. 2024-04-25T16:12:25.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.100+0000 7f15f1633740 20 client.17790 trim_cache size 0 max 16384 2024-04-25T16:12:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.156+0000 7f15f1633740 1 Processor -- start 2024-04-25T16:12:25.158 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.157+0000 7f15f1633740 1 -- start start 2024-04-25T16:12:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.157+0000 7f15f1633740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716ed9f60 0x55e7172b8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.157+0000 7f15f1633740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716efa6b0 0x55e7172b90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.157+0000 7f15f1633740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 0x55e7172c7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.157+0000 7f15f1633740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41c2580 con 0x55e716f1b390 2024-04-25T16:12:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.157+0000 7f15f1633740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41c40c0 con 0x55e716efa6b0 2024-04-25T16:12:25.159 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.157+0000 7f15f1633740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41c2c50 con 0x55e716ed9f60 2024-04-25T16:12:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.158+0000 7f15e3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716ed9f60 0x55e7172b8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.158+0000 7f15e3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716ed9f60 0x55e7172b8bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56680/0 (socket says 172.21.15.31:56680) 2024-04-25T16:12:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.158+0000 7f15e3fff640 1 -- 172.21.15.31:0/780252760 learned_addr learned my addr 172.21.15.31:0/780252760 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.158+0000 7f15e37fe640 1 --2- 172.21.15.31:0/780252760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716efa6b0 0x55e7172b90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.160+0000 7f15e37fe640 1 -- 172.21.15.31:0/780252760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716ed9f60 msgr2=0x55e7172b8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.160+0000 7f15e37fe640 1 --2- 172.21.15.31:0/780252760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716ed9f60 0x55e7172b8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.160+0000 7f15e37fe640 1 -- 172.21.15.31:0/780252760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 msgr2=0x55e7172c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.160+0000 7f15e37fe640 1 --2- 172.21.15.31:0/780252760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 0x55e7172c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.161 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.160+0000 7f15e37fe640 1 -- 172.21.15.31:0/780252760 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7172bee00 con 0x55e716efa6b0 2024-04-25T16:12:25.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.160+0000 7f15e37fe640 1 --2- 172.21.15.31:0/780252760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716efa6b0 0x55e7172b90f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0a5430 tx=0x7f15dc0dfeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8798f6447874f17 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.161+0000 7f15d1ffb640 1 -- 172.21.15.31:0/780252760 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0a7cd0 con 0x55e716efa6b0 2024-04-25T16:12:25.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.161+0000 7f15d1ffb640 1 -- 172.21.15.31:0/780252760 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15dc0a88c0 con 0x55e716efa6b0 2024-04-25T16:12:25.163 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.161+0000 7f15d1ffb640 1 -- 172.21.15.31:0/780252760 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0ac120 con 0x55e716efa6b0 2024-04-25T16:12:25.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.163+0000 7f15f1633740 1 -- 172.21.15.31:0/780252760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716efa6b0 msgr2=0x55e7172b90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.164 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.163+0000 7f15f1633740 1 --2- 172.21.15.31:0/780252760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716efa6b0 0x55e7172b90f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0a5430 tx=0x7f15dc0dfeb0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.175+0000 7f15f1633740 1 -- 172.21.15.31:0/780252760 shutdown_connections 2024-04-25T16:12:25.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.175+0000 7f15f1633740 1 --2- 172.21.15.31:0/780252760 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716f1b390 0x55e7172c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.175+0000 7f15f1633740 1 --2- 172.21.15.31:0/780252760 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716efa6b0 0x55e7172b90f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.175+0000 7f15f1633740 1 --2- 172.21.15.31:0/780252760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716ed9f60 0x55e7172b8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.175+0000 7f15f1633740 1 -- 172.21.15.31:0/780252760 >> 172.21.15.31:0/780252760 conn(0x55e716f0f4e0 msgr2=0x55e7172c5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.179+0000 7f15f1633740 1 -- 172.21.15.31:0/780252760 shutdown_connections 2024-04-25T16:12:25.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.179+0000 7f15f1633740 1 -- 172.21.15.31:0/780252760 wait complete. 2024-04-25T16:12:25.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.179+0000 7f15f1633740 1 -- start start 2024-04-25T16:12:25.181 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.180+0000 7f15f1633740 1 Processor -- start 2024-04-25T16:12:25.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.181+0000 7f15f1633740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716ed9f60 0x55e7172ce130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.181+0000 7f15f1633740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 0x55e7172cad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.182+0000 7f15f1633740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716f1b390 0x55e7172cb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.183 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.182+0000 7f15f1633740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41c39f0 con 0x55e716ed9f60 2024-04-25T16:12:25.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.182+0000 7f15f1633740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41c3320 con 0x55e716f1b390 2024-04-25T16:12:25.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.182+0000 7f15f1633740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15e41c0560 con 0x55e716efa6b0 2024-04-25T16:12:25.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.182+0000 7f15e37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 0x55e7172cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.182+0000 7f15e37fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 0x55e7172cad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56694/0 (socket says 172.21.15.31:56694) 2024-04-25T16:12:25.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.182+0000 7f15e37fe640 1 -- 172.21.15.31:0/2819465725 learned_addr learned my addr 172.21.15.31:0/2819465725 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.182+0000 7f15f1633740 10 client.0 _ll_register_callbacks cb 0x7f15d400c370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:25.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.182+0000 7f15e8d21640 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716f1b390 0x55e7172cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.183+0000 7f15e37fe640 1 -- 172.21.15.31:0/2819465725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716f1b390 msgr2=0x55e7172cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.183+0000 7f15e37fe640 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716f1b390 0x55e7172cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.183+0000 7f15e37fe640 1 -- 172.21.15.31:0/2819465725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716ed9f60 msgr2=0x55e7172ce130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.183+0000 7f15e37fe640 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716ed9f60 0x55e7172ce130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.183+0000 7f15e37fe640 1 -- 172.21.15.31:0/2819465725 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7172cb800 con 0x55e716efa6b0 2024-04-25T16:12:25.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.183+0000 7f15e37fe640 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 0x55e7172cad00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0a8150 tx=0x7f15dc0ac350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bfb86e4ccbe98bd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.185+0000 7f15c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:12:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.185+0000 7f15c77fe640 1 -- 172.21.15.31:0/2819465725 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15dc0a7cd0 con 0x55e716efa6b0 2024-04-25T16:12:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.185+0000 7f15f1633740 1 -- 172.21.15.31:0/2819465725 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15dc0a88c0 con 0x55e716efa6b0 2024-04-25T16:12:25.186 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.185+0000 7f15c77fe640 1 -- 172.21.15.31:0/2819465725 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f15dc0b0da0 con 0x55e716efa6b0 2024-04-25T16:12:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.186+0000 7f15c77fe640 1 -- 172.21.15.31:0/2819465725 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f15dc0ac570 con 0x55e716efa6b0 2024-04-25T16:12:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.186+0000 7f15f1633740 10 client.27496 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:25.187 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.186+0000 7f15f1633740 1 -- 172.21.15.31:0/2819465725 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55e71733ca70 con 0x55e716efa6b0 2024-04-25T16:12:25.193 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.192+0000 7f15c77fe640 1 -- 172.21.15.31:0/2819465725 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15dc0a88c0 con 0x55e716efa6b0 2024-04-25T16:12:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.192+0000 7f15c77fe640 1 -- 172.21.15.31:0/2819465725 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f15dc0a3c40 con 0x55e716efa6b0 2024-04-25T16:12:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.192+0000 7f15f1633740 10 client.27496 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.192+0000 7f15f1633740 10 client.27496 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:12:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.192+0000 7f15f1633740 20 client.27496 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.192+0000 7f15f1633740 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55e71733d440 0x55e71733f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.193+0000 7f15f1633740 4 client.27496 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:12:25.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.193+0000 7f15f1633740 1 -- 172.21.15.31:0/2819465725 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e717229800 con 0x55e71733d440 2024-04-25T16:12:25.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.194+0000 7f15e3fff640 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55e71733d440 0x55e71733f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.195+0000 7f15e3fff640 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55e71733d440 0x55e71733f860 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8e1fc3b356564d1 server_cookie=e031d1e44cd21002 in_seq=0 out_seq=0 2024-04-25T16:12:25.197 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.196+0000 7f15c77fe640 10 client.27496 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:12:25.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.196+0000 7f15c77fe640 1 -- 172.21.15.31:0/2819465725 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e717229800 con 0x55e71733d440 2024-04-25T16:12:25.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.196+0000 7f15c77fe640 10 client.27496 handle_command_reply: tid=0 2024-04-25T16:12:25.198 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.196+0000 7f15f1633740 1 client.27496 shutdown 2024-04-25T16:12:25.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.198+0000 7f15f1633740 1 -- 172.21.15.31:0/2819465725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 msgr2=0x55e7172cad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.198+0000 7f15f1633740 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 0x55e7172cad00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f15dc0a8150 tx=0x7f15dc0ac350 comp rx=0 tx=0).stop 2024-04-25T16:12:25.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.199+0000 7f15f1633740 1 -- 172.21.15.31:0/2819465725 shutdown_connections 2024-04-25T16:12:25.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.199+0000 7f15f1633740 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55e716f1b390 0x55e7172cb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.199+0000 7f15f1633740 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55e71733d440 0x55e71733f860 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.199+0000 7f15f1633740 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55e716efa6b0 0x55e7172cad00 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.199+0000 7f15f1633740 1 --2- 172.21.15.31:0/2819465725 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55e716ed9f60 0x55e7172ce130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.199+0000 7f15c77fe640 0 client.27496 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:12:25.200 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.199+0000 7f15f1633740 1 -- 172.21.15.31:0/2819465725 >> 172.21.15.31:0/2819465725 conn(0x55e716f0f4e0 msgr2=0x55e7172b8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.210 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.204+0000 7f15f1633740 1 -- 172.21.15.31:0/2819465725 shutdown_connections 2024-04-25T16:12:25.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.211+0000 7f15f1633740 1 -- 172.21.15.31:0/2819465725 wait complete. 2024-04-25T16:12:25.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.216+0000 7f15f1633740 20 client.27496 trim_cache size 0 max 16384 2024-04-25T16:12:25.221 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:12:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 -- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f15cc0afb20 msgr2=0x7f15cc0b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.242 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f15cc0afb20 0x7f15cc0b2030 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55e7172bea30 tx=0x7f15d400c4d0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 -- 172.21.15.31:0/1552152640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15e410a760 msgr2=0x7f15e41a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15e410a760 0x7f15e41a0c60 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00acc0 tx=0x7f15dc007590 comp rx=0 tx=0).stop 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 -- 172.21.15.31:0/1552152640 shutdown_connections 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f15cc0afb20 0x7f15cc0b2030 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15e41102c0 0x7f15e41a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15e410a760 0x7f15e41a0c60 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 --2- 172.21.15.31:0/1552152640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f15e410a240 0x7f15e41a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 -- 172.21.15.31:0/1552152640 >> 172.21.15.31:0/1552152640 conn(0x7f15e4070780 msgr2=0x7f15e410c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 -- 172.21.15.31:0/1552152640 shutdown_connections 2024-04-25T16:12:25.243 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.241+0000 7f15e9522640 1 -- 172.21.15.31:0/1552152640 wait complete. 2024-04-25T16:12:25.283 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:12:25.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.605+0000 7f26bebea640 1 -- 172.21.15.31:0/3926875162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26b810a340 msgr2=0x7f26b810ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.605+0000 7f26bebea640 1 --2- 172.21.15.31:0/3926875162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26b810a340 0x7f26b810ac50 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f26a801a910 tx=0x7f26a803b450 comp rx=0 tx=0).stop 2024-04-25T16:12:25.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.611+0000 7f26bebea640 1 -- 172.21.15.31:0/3926875162 shutdown_connections 2024-04-25T16:12:25.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.611+0000 7f26bebea640 1 --2- 172.21.15.31:0/3926875162 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26b810fe90 0x7f26b81102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.611+0000 7f26bebea640 1 --2- 172.21.15.31:0/3926875162 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26b810b190 0x7f26b810b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.611+0000 7f26bebea640 1 --2- 172.21.15.31:0/3926875162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26b810a340 0x7f26b810ac50 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.611+0000 7f26bebea640 1 -- 172.21.15.31:0/3926875162 >> 172.21.15.31:0/3926875162 conn(0x7f26b8070320 msgr2=0x7f26b80727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:25 smithi063 ceph-mon[32413]: pgmap v684: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 837 MiB/s wr, 1.78k op/s 2024-04-25T16:12:25.623 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.621+0000 7f26bebea640 1 -- 172.21.15.31:0/3926875162 shutdown_connections 2024-04-25T16:12:25.625 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.624+0000 7f26bebea640 1 -- 172.21.15.31:0/3926875162 wait complete. 2024-04-25T16:12:25.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.625+0000 7f26bebea640 1 Processor -- start 2024-04-25T16:12:25.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.625+0000 7f26bebea640 1 -- start start 2024-04-25T16:12:25.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.625+0000 7f26bebea640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26b810a340 0x7f26b81a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.625+0000 7f26bebea640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26b810b190 0x7f26b81a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.625+0000 7f26bebea640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26b810fe90 0x7f26b81a7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.625+0000 7f26bebea640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f26b8113bb0 con 0x7f26b810fe90 2024-04-25T16:12:25.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.625+0000 7f26bebea640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f26b8113a30 con 0x7f26b810b190 2024-04-25T16:12:25.627 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.625+0000 7f26bebea640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f26b8113d30 con 0x7f26b810a340 2024-04-25T16:12:25.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.626+0000 7f26be3e9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26b810fe90 0x7f26b81a7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.626+0000 7f26be3e9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26b810fe90 0x7f26b81a7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37230/0 (socket says 172.21.15.31:37230) 2024-04-25T16:12:25.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.626+0000 7f26be3e9640 1 -- 172.21.15.31:0/3008618705 learned_addr learned my addr 172.21.15.31:0/3008618705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.626+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26b810a340 0x7f26b81a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.626+0000 7f26be3e9640 1 -- 172.21.15.31:0/3008618705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26b810a340 msgr2=0x7f26b81a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.628 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.626+0000 7f26be3e9640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26b810a340 0x7f26b81a0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.626+0000 7f26be3e9640 1 -- 172.21.15.31:0/3008618705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26b810b190 msgr2=0x7f26b81a0c70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:12:25.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.626+0000 7f26be3e9640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26b810b190 0x7f26b81a0c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.626+0000 7f26be3e9640 1 -- 172.21.15.31:0/3008618705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26b81a8270 con 0x7f26b810fe90 2024-04-25T16:12:25.629 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.627+0000 7f26be3e9640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26b810fe90 0x7f26b81a7c00 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f26b000eb10 tx=0x7f26b000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.627+0000 7f26aeffd640 1 -- 172.21.15.31:0/3008618705 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b0004b20 con 0x7f26b810fe90 2024-04-25T16:12:25.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.628+0000 7f26aeffd640 1 -- 172.21.15.31:0/3008618705 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26b0004cc0 con 0x7f26b810fe90 2024-04-25T16:12:25.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.628+0000 7f26aeffd640 1 -- 172.21.15.31:0/3008618705 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b0010670 con 0x7f26b810fe90 2024-04-25T16:12:25.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.628+0000 7f26bebea640 1 -- 172.21.15.31:0/3008618705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26b81a8560 con 0x7f26b810fe90 2024-04-25T16:12:25.630 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.628+0000 7f26bebea640 1 -- 172.21.15.31:0/3008618705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26b81a8ae0 con 0x7f26b810fe90 2024-04-25T16:12:25.633 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.632+0000 7f26c6ddc740 1 Processor -- start 2024-04-25T16:12:25.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.633+0000 7f26c6ddc740 1 -- start start 2024-04-25T16:12:25.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.633+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a81390 0x55fac1e239e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.633+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a584d0 0x55fac1e23f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.634 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.633+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a606b0 0x55fac1e24bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.633+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fac1d87d80 con 0x55fac1a606b0 2024-04-25T16:12:25.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.633+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fac1d41a20 con 0x55fac1a584d0 2024-04-25T16:12:25.635 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.633+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fac1d83d40 con 0x55fac1a81390 2024-04-25T16:12:25.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.636+0000 7f26bdbe8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a584d0 0x55fac1e23f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.636+0000 7f26bdbe8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a584d0 0x55fac1e23f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:54978/0 (socket says 172.21.15.31:54978) 2024-04-25T16:12:25.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.636+0000 7f26bdbe8640 1 -- 172.21.15.31:0/1838336567 learned_addr learned my addr 172.21.15.31:0/1838336567 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.637+0000 7f26bdbe8640 1 -- 172.21.15.31:0/1838336567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a81390 msgr2=0x55fac1e239e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.637+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/1838336567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a81390 0x55fac1e239e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.637+0000 7f26bdbe8640 1 -- 172.21.15.31:0/1838336567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a606b0 msgr2=0x55fac1e24bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.637+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/1838336567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a606b0 0x55fac1e24bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.637+0000 7f26bdbe8640 1 -- 172.21.15.31:0/1838336567 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fac1d4cc30 con 0x55fac1a584d0 2024-04-25T16:12:25.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.637+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/1838336567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a584d0 0x55fac1e23f60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f26a801a910 tx=0x7f26a8005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d97d23e3e7a282b server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.637+0000 7f26a3fff640 1 -- 172.21.15.31:0/1838336567 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8046030 con 0x55fac1a584d0 2024-04-25T16:12:25.639 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.638+0000 7f26a3fff640 1 -- 172.21.15.31:0/1838336567 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26a80049c0 con 0x55fac1a584d0 2024-04-25T16:12:25.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.638+0000 7f26aeffd640 1 -- 172.21.15.31:0/3008618705 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f26b0005020 con 0x7f26b810fe90 2024-04-25T16:12:25.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.639+0000 7f26c6ddc740 1 -- 172.21.15.31:0/1838336567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a584d0 msgr2=0x55fac1e23f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.639+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/1838336567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a584d0 0x55fac1e23f60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f26a801a910 tx=0x7f26a8005fd0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.640+0000 7f26c6ddc740 1 -- 172.21.15.31:0/1838336567 shutdown_connections 2024-04-25T16:12:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.640+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/1838336567 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a606b0 0x55fac1e24bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.640+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/1838336567 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a584d0 0x55fac1e23f60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.641+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/1838336567 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a81390 0x55fac1e239e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.641+0000 7f26c6ddc740 1 -- 172.21.15.31:0/1838336567 >> 172.21.15.31:0/1838336567 conn(0x55fac1a754e0 msgr2=0x55fac1e1f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.642 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.641+0000 7f26aeffd640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f26a40afc30 0x7f26a40b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.641+0000 7f26aeffd640 1 -- 172.21.15.31:0/3008618705 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f26b00df1a0 con 0x7f26b810fe90 2024-04-25T16:12:25.643 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.642+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f26a40afc30 0x7f26a40b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.644 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.643+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f26a40afc30 0x7f26a40b2140 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f26a801a2b0 tx=0x7f26a8005260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.645+0000 7f26c6ddc740 1 -- 172.21.15.31:0/1838336567 shutdown_connections 2024-04-25T16:12:25.646 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.645+0000 7f26c6ddc740 1 -- 172.21.15.31:0/1838336567 wait complete. 2024-04-25T16:12:25.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.646+0000 7f26c6ddc740 1 -- start start 2024-04-25T16:12:25.647 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.646+0000 7f26c6ddc740 1 Processor -- start 2024-04-25T16:12:25.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:25 smithi186 ceph-mon[33258]: pgmap v684: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 837 MiB/s wr, 1.78k op/s 2024-04-25T16:12:25.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.648+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e2d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.649 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.648+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 0x55fac1e2d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.649+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 0x55fac1e34950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.649+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fac1e256f0 con 0x55fac1a584d0 2024-04-25T16:12:25.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.649+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fac1d0edc0 con 0x55fac1a81390 2024-04-25T16:12:25.650 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.649+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fac1e25870 con 0x55fac1a606b0 2024-04-25T16:12:25.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.649+0000 7f26bd3e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 0x55fac1e2d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.649+0000 7f26c6ddc740 10 client.0 _ll_register_callbacks cb 0x55fac1d2a600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:25.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.650+0000 7f26be3e9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 0x55fac1e34950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.650+0000 7f26be3e9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 0x55fac1e34950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:54984/0 (socket says 172.21.15.31:54984) 2024-04-25T16:12:25.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.650+0000 7f26be3e9640 1 -- 172.21.15.31:0/4094582929 learned_addr learned my addr 172.21.15.31:0/4094582929 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.650+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e2d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.650+0000 7f26be3e9640 1 -- 172.21.15.31:0/4094582929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 msgr2=0x55fac1e2d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.650+0000 7f26be3e9640 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 0x55fac1e2d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.650+0000 7f26be3e9640 1 -- 172.21.15.31:0/4094582929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 msgr2=0x55fac1e2d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.650+0000 7f26be3e9640 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e2d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.650+0000 7f26be3e9640 1 -- 172.21.15.31:0/4094582929 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fac1a993f0 con 0x55fac1a81390 2024-04-25T16:12:25.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.651+0000 7f26bd3e7640 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 0x55fac1e2d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:12:25.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.651+0000 7f26be3e9640 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 0x55fac1e34950 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f26b0004af0 tx=0x7f26b00a6e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d78dc5615aff6047 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.651+0000 7f26a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:12:25.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.652+0000 7f26a17fa640 1 -- 172.21.15.31:0/4094582929 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26b00ab9e0 con 0x55fac1a81390 2024-04-25T16:12:25.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.652+0000 7f26a17fa640 1 -- 172.21.15.31:0/4094582929 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f26b00a3c20 con 0x55fac1a81390 2024-04-25T16:12:25.653 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.652+0000 7f26c6ddc740 1 -- 172.21.15.31:0/4094582929 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26a80049c0 con 0x55fac1a81390 2024-04-25T16:12:25.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.653+0000 7f26a17fa640 1 -- 172.21.15.31:0/4094582929 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f26b00c6050 con 0x55fac1a81390 2024-04-25T16:12:25.657 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.656+0000 7f26c6ddc740 10 client.27518 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:25.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.656+0000 7f26c6ddc740 1 -- 172.21.15.31:0/4094582929 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fac1b637d0 con 0x55fac1a81390 2024-04-25T16:12:25.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.657+0000 7f26a17fa640 1 -- 172.21.15.31:0/4094582929 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26a80049c0 con 0x55fac1a81390 2024-04-25T16:12:25.658 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.657+0000 7f26a17fa640 1 -- 172.21.15.31:0/4094582929 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f26b00a3070 con 0x55fac1a81390 2024-04-25T16:12:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.657+0000 7f26c6ddc740 10 client.27518 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.657+0000 7f26c6ddc740 10 client.27518 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:12:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.657+0000 7f26c6ddc740 20 client.27518 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.657+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fac1a3ff60 0x55fac1e3b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.657+0000 7f26c6ddc740 4 client.27518 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.657+0000 7f26c6ddc740 1 -- 172.21.15.31:0/4094582929 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fac1d95990 con 0x55fac1a3ff60 2024-04-25T16:12:25.659 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.658+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fac1a3ff60 0x55fac1e3b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.674+0000 7f26a17fa640 10 client.27518 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:12:25.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.674+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fac1a3ff60 0x55fac1e3b820 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=59c839cbe9757ba9 server_cookie=1a826513636a8efb in_seq=0 out_seq=0 2024-04-25T16:12:25.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.687+0000 7f26a17fa640 1 -- 172.21.15.31:0/4094582929 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fac1d95990 con 0x55fac1a3ff60 2024-04-25T16:12:25.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.687+0000 7f26a17fa640 10 client.27518 handle_command_reply: tid=0 2024-04-25T16:12:25.688 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.687+0000 7f26c6ddc740 1 client.27518 shutdown 2024-04-25T16:12:25.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.688+0000 7f26c6ddc740 1 -- 172.21.15.31:0/4094582929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 msgr2=0x55fac1e34950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.688+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 0x55fac1e34950 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f26b0004af0 tx=0x7f26b00a6e60 comp rx=0 tx=0).stop 2024-04-25T16:12:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.688+0000 7f26c6ddc740 1 -- 172.21.15.31:0/4094582929 shutdown_connections 2024-04-25T16:12:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.688+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fac1a3ff60 0x55fac1e3b820 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.688+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 0x55fac1e34950 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.688+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 0x55fac1e2d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.688+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/4094582929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e2d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.688+0000 7f26c6ddc740 1 -- 172.21.15.31:0/4094582929 >> 172.21.15.31:0/4094582929 conn(0x55fac1a754e0 msgr2=0x55fac1e22100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.688+0000 7f26a17fa640 0 client.27518 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:12:25.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.689+0000 7f26c6ddc740 1 -- 172.21.15.31:0/4094582929 shutdown_connections 2024-04-25T16:12:25.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.689+0000 7f26c6ddc740 1 -- 172.21.15.31:0/4094582929 wait complete. 2024-04-25T16:12:25.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.689+0000 7f26c6ddc740 20 client.27518 trim_cache size 0 max 16384 2024-04-25T16:12:25.716 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:25 smithi031 ceph-mon[29140]: pgmap v684: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 837 MiB/s wr, 1.78k op/s 2024-04-25T16:12:25.729 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.728+0000 7f26c6ddc740 1 Processor -- start 2024-04-25T16:12:25.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.728+0000 7f26c6ddc740 1 -- start start 2024-04-25T16:12:25.730 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.729+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e28150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.730+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 0x55fac1e28690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.730+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 0x55fac1e34870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.730+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f269c00e590 con 0x55fac1a584d0 2024-04-25T16:12:25.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.730+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f269c0100d0 con 0x55fac1a81390 2024-04-25T16:12:25.731 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.730+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f269c00ec60 con 0x55fac1a606b0 2024-04-25T16:12:25.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.730+0000 7f26bdbe8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e28150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.731+0000 7f26bdbe8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e28150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37252/0 (socket says 172.21.15.31:37252) 2024-04-25T16:12:25.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.731+0000 7f26bdbe8640 1 -- 172.21.15.31:0/378430274 learned_addr learned my addr 172.21.15.31:0/378430274 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.731+0000 7f26bdbe8640 1 -- 172.21.15.31:0/378430274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 msgr2=0x55fac1e28690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:25.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.731+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/378430274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 0x55fac1e28690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.731+0000 7f26bdbe8640 1 -- 172.21.15.31:0/378430274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 msgr2=0x55fac1e34870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:25.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.731+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/378430274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 0x55fac1e34870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.731+0000 7f26bdbe8640 1 -- 172.21.15.31:0/378430274 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fac1e34db0 con 0x55fac1a584d0 2024-04-25T16:12:25.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.732+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/378430274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e28150 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f26a803ba40 tx=0x7f26a803b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc23306623f3e36d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.733+0000 7f26a37fe640 1 -- 172.21.15.31:0/378430274 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8071070 con 0x55fac1a584d0 2024-04-25T16:12:25.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.733+0000 7f26a37fe640 1 -- 172.21.15.31:0/378430274 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26a806c860 con 0x55fac1a584d0 2024-04-25T16:12:25.734 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.733+0000 7f26a37fe640 1 -- 172.21.15.31:0/378430274 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8046040 con 0x55fac1a584d0 2024-04-25T16:12:25.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.735+0000 7f26c6ddc740 1 -- 172.21.15.31:0/378430274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 msgr2=0x55fac1e28150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.736+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/378430274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e28150 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f26a803ba40 tx=0x7f26a803b450 comp rx=0 tx=0).stop 2024-04-25T16:12:25.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.750+0000 7f26c6ddc740 1 -- 172.21.15.31:0/378430274 shutdown_connections 2024-04-25T16:12:25.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.750+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/378430274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a81390 0x55fac1e34870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.750+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/378430274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a606b0 0x55fac1e28690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.750+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/378430274 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e28150 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.751 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.750+0000 7f26c6ddc740 1 -- 172.21.15.31:0/378430274 >> 172.21.15.31:0/378430274 conn(0x55fac1a754e0 msgr2=0x55fac1e239a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.754+0000 7f26c6ddc740 1 -- 172.21.15.31:0/378430274 shutdown_connections 2024-04-25T16:12:25.755 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.754+0000 7f26c6ddc740 1 -- 172.21.15.31:0/378430274 wait complete. 2024-04-25T16:12:25.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.755+0000 7f26c6ddc740 1 -- start start 2024-04-25T16:12:25.756 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.755+0000 7f26c6ddc740 1 Processor -- start 2024-04-25T16:12:25.757 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.756+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e31660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.756+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a606b0 0x55fac1e31ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.757+0000 7f26c6ddc740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a81390 0x55fac1ea1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.757+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f269c00fa00 con 0x55fac1a584d0 2024-04-25T16:12:25.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.757+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f269c00f330 con 0x55fac1a606b0 2024-04-25T16:12:25.758 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.757+0000 7f26c6ddc740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f269c00c570 con 0x55fac1a81390 2024-04-25T16:12:25.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.757+0000 7f26c6ddc740 10 client.0 _ll_register_callbacks cb 0x7f26b0004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:25.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26bdbe8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e31660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26bd3e7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a606b0 0x55fac1e31ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26bdbe8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e31660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37262/0 (socket says 172.21.15.31:37262) 2024-04-25T16:12:25.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26bdbe8640 1 -- 172.21.15.31:0/213337498 learned_addr learned my addr 172.21.15.31:0/213337498 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:25.759 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26be3e9640 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a81390 0x55fac1ea1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26bdbe8640 1 -- 172.21.15.31:0/213337498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a81390 msgr2=0x55fac1ea1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a81390 0x55fac1ea1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26bdbe8640 1 -- 172.21.15.31:0/213337498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a606b0 msgr2=0x55fac1e31ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:25.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a606b0 0x55fac1e31ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.758+0000 7f26bdbe8640 1 -- 172.21.15.31:0/213337498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fac1e32190 con 0x55fac1a584d0 2024-04-25T16:12:25.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.759+0000 7f26bdbe8640 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e31660 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f26a801a860 tx=0x7f26a8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a38ded54e78172d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:25.760 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.759+0000 7f26a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:12:25.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.760+0000 7f26a17fa640 1 -- 172.21.15.31:0/213337498 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a8046070 con 0x55fac1a584d0 2024-04-25T16:12:25.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.760+0000 7f26c6ddc740 1 -- 172.21.15.31:0/213337498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26a806c860 con 0x55fac1a584d0 2024-04-25T16:12:25.761 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.760+0000 7f26a17fa640 1 -- 172.21.15.31:0/213337498 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f26a8040930 con 0x55fac1a584d0 2024-04-25T16:12:25.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.760+0000 7f26a17fa640 1 -- 172.21.15.31:0/213337498 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f26a807d050 con 0x55fac1a584d0 2024-04-25T16:12:25.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.761+0000 7f26c6ddc740 10 client.17805 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:25.762 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.761+0000 7f26c6ddc740 1 -- 172.21.15.31:0/213337498 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fac1ea3140 con 0x55fac1a584d0 2024-04-25T16:12:25.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.767+0000 7f26a17fa640 1 -- 172.21.15.31:0/213337498 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26a806c860 con 0x55fac1a584d0 2024-04-25T16:12:25.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.770+0000 7f26a17fa640 1 -- 172.21.15.31:0/213337498 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f26a8055bb0 con 0x55fac1a584d0 2024-04-25T16:12:25.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.774+0000 7f26c6ddc740 10 client.17805 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:25.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.774+0000 7f26c6ddc740 10 client.17805 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:12:25.775 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.774+0000 7f26c6ddc740 20 client.17805 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:25.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.774+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fac1a153a0 0x55fac1ea5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:25.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.774+0000 7f26c6ddc740 4 client.17805 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:12:25.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.774+0000 7f26c6ddc740 1 -- 172.21.15.31:0/213337498 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fac1d8f800 con 0x55fac1a153a0 2024-04-25T16:12:25.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.784+0000 7f26bd3e7640 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fac1a153a0 0x55fac1ea5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:25.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.790+0000 7f26a17fa640 10 client.17805 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:12:25.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.790+0000 7f26bd3e7640 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fac1a153a0 0x55fac1ea5500 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=668c6aa87f2cbfa4 server_cookie=29c4f3fa5bc8640e in_seq=0 out_seq=0 2024-04-25T16:12:25.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.790+0000 7f26a17fa640 1 -- 172.21.15.31:0/213337498 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fac1d8f800 con 0x55fac1a153a0 2024-04-25T16:12:25.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.790+0000 7f26a17fa640 10 client.17805 handle_command_reply: tid=0 2024-04-25T16:12:25.793 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.792+0000 7f26c6ddc740 1 client.17805 shutdown 2024-04-25T16:12:25.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.793+0000 7f26c6ddc740 1 -- 172.21.15.31:0/213337498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 msgr2=0x55fac1e31660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.794 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.793+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e31660 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f26a801a860 tx=0x7f26a8005fd0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.795+0000 7f26c6ddc740 1 -- 172.21.15.31:0/213337498 shutdown_connections 2024-04-25T16:12:25.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.795+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fac1a81390 0x55fac1ea1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.795+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fac1a153a0 0x55fac1ea5500 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.796 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.795+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fac1a606b0 0x55fac1e31ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.795+0000 7f26c6ddc740 1 --2- 172.21.15.31:0/213337498 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fac1a584d0 0x55fac1e31660 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.795+0000 7f26c6ddc740 1 -- 172.21.15.31:0/213337498 >> 172.21.15.31:0/213337498 conn(0x55fac1a754e0 msgr2=0x55fac1e239a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.797 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.795+0000 7f26a17fa640 0 client.17805 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:12:25.799 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.798+0000 7f26c6ddc740 1 -- 172.21.15.31:0/213337498 shutdown_connections 2024-04-25T16:12:25.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.801+0000 7f26c6ddc740 1 -- 172.21.15.31:0/213337498 wait complete. 2024-04-25T16:12:25.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.801+0000 7f26c6ddc740 20 client.17805 trim_cache size 0 max 16384 2024-04-25T16:12:25.803 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:12:25.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.803+0000 7f26bebea640 1 -- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f26a40afc30 msgr2=0x7f26a40b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.803+0000 7f26bebea640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f26a40afc30 0x7f26a40b2140 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f26a801a2b0 tx=0x7f26a8005260 comp rx=0 tx=0).stop 2024-04-25T16:12:25.804 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.803+0000 7f26bebea640 1 -- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26b810fe90 msgr2=0x7f26b81a7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.803+0000 7f26bebea640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26b810fe90 0x7f26b81a7c00 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f26b000eb10 tx=0x7f26b000c2d0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.807+0000 7f26bebea640 1 -- 172.21.15.31:0/3008618705 shutdown_connections 2024-04-25T16:12:25.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.807+0000 7f26bebea640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f26a40afc30 0x7f26a40b2140 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.807+0000 7f26bebea640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f26b810fe90 0x7f26b81a7c00 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.807+0000 7f26bebea640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f26b810b190 0x7f26b81a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.807+0000 7f26bebea640 1 --2- 172.21.15.31:0/3008618705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26b810a340 0x7f26b81a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.809 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.808+0000 7f26bebea640 1 -- 172.21.15.31:0/3008618705 >> 172.21.15.31:0/3008618705 conn(0x7f26b8070320 msgr2=0x7f26b810de80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.813+0000 7f26bebea640 1 -- 172.21.15.31:0/3008618705 shutdown_connections 2024-04-25T16:12:25.815 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:25.814+0000 7f26bebea640 1 -- 172.21.15.31:0/3008618705 wait complete. 2024-04-25T16:12:25.866 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:12:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.921+0000 7fa7a77fe640 1 -- 172.21.15.31:0/1772561195 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563ac0d0cee0 con 0x563ac09931c0 2024-04-25T16:12:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.921+0000 7fa7a77fe640 10 client.27476 handle_command_reply: tid=0 2024-04-25T16:12:25.927 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.921+0000 7fa7d1410740 1 client.27476 shutdown 2024-04-25T16:12:25.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.930+0000 7fa7d1410740 1 -- 172.21.15.31:0/1772561195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 msgr2=0x563ac0daf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.932 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.930+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0daf070 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8012d10 tx=0x7fa7b8005020 comp rx=0 tx=0).stop 2024-04-25T16:12:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.933+0000 7fa7d1410740 1 -- 172.21.15.31:0/1772561195 shutdown_connections 2024-04-25T16:12:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.934+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563ac09931c0 0x563ac0e43200 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.934+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563ac09ff1b0 0x563ac0daf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.934+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563ac09de4d0 0x563ac0daf070 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.934+0000 7fa7d1410740 1 --2- 172.21.15.31:0/1772561195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563ac09d62f0 0x563ac0db0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.934+0000 7fa7d1410740 1 -- 172.21.15.31:0/1772561195 >> 172.21.15.31:0/1772561195 conn(0x563ac09f3300 msgr2=0x563ac0da7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.935 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.934+0000 7fa7a77fe640 0 client.27476 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:25.938 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.937+0000 7fa7d1410740 1 -- 172.21.15.31:0/1772561195 shutdown_connections 2024-04-25T16:12:25.939 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.937+0000 7fa7d1410740 1 -- 172.21.15.31:0/1772561195 wait complete. 2024-04-25T16:12:25.943 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.942+0000 7fa7d1410740 20 client.27476 trim_cache size 0 max 16384 2024-04-25T16:12:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:12:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:12:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:12:25.944 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:12:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.947+0000 7fa7c922a640 1 -- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7ac0afcf0 msgr2=0x7fa7ac0b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.947+0000 7fa7c922a640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7ac0afcf0 0x7fa7ac0b2200 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x563ac0d9c230 tx=0x7fa7bc00c7f0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.947+0000 7fa7c922a640 1 -- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c4107790 msgr2=0x7fa7c419f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:25.948 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.947+0000 7fa7c922a640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c4107790 0x7fa7c419f2e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400ead0 tx=0x7fa7b400ef90 comp rx=0 tx=0).stop 2024-04-25T16:12:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.947+0000 7fa7c922a640 1 -- 172.21.15.31:0/3227446623 shutdown_connections 2024-04-25T16:12:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.947+0000 7fa7c922a640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa7ac0afcf0 0x7fa7ac0b2200 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.949 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.947+0000 7fa7c922a640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa7c4107790 0x7fa7c419f2e0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.947+0000 7fa7c922a640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa7c4102620 0x7fa7c4198350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.958+0000 7fa7c922a640 1 --2- 172.21.15.31:0/3227446623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7c4101be0 0x7fa7c4197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:25.959 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.958+0000 7fa7c922a640 1 -- 172.21.15.31:0/3227446623 >> 172.21.15.31:0/3227446623 conn(0x7fa7c40fd1f0 msgr2=0x7fa7c40ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:25.962 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.961+0000 7fa7c922a640 1 -- 172.21.15.31:0/3227446623 shutdown_connections 2024-04-25T16:12:25.963 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:25.961+0000 7fa7c922a640 1 -- 172.21.15.31:0/3227446623 wait complete. 2024-04-25T16:12:26.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.304+0000 7fe872910640 1 -- 172.21.15.31:0/4243371472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe86c10b190 msgr2=0x7fe86c10b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:26.309 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.304+0000 7fe872910640 1 --2- 172.21.15.31:0/4243371472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe86c10b190 0x7fe86c10b5a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe868009900 tx=0x7fe86802a420 comp rx=0 tx=0).stop 2024-04-25T16:12:26.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.309+0000 7fe872910640 1 -- 172.21.15.31:0/4243371472 shutdown_connections 2024-04-25T16:12:26.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.309+0000 7fe872910640 1 --2- 172.21.15.31:0/4243371472 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c10fe90 0x7fe86c1102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.309+0000 7fe872910640 1 --2- 172.21.15.31:0/4243371472 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe86c10b190 0x7fe86c10b5a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.309+0000 7fe872910640 1 --2- 172.21.15.31:0/4243371472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe86c10a340 0x7fe86c10ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.315 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.309+0000 7fe872910640 1 -- 172.21.15.31:0/4243371472 >> 172.21.15.31:0/4243371472 conn(0x7fe86c070320 msgr2=0x7fe86c0727d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:26.320 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.317+0000 7fe872910640 1 -- 172.21.15.31:0/4243371472 shutdown_connections 2024-04-25T16:12:26.323 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.318+0000 7fe872910640 1 -- 172.21.15.31:0/4243371472 wait complete. 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.319+0000 7fe872910640 1 Processor -- start 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.319+0000 7fe872910640 1 -- start start 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.319+0000 7fe872910640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe86c10a340 0x7fe86c1a8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.319+0000 7fe872910640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe86c10b190 0x7fe86c1a9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.319+0000 7fe872910640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c10fe90 0x7fe86c1b04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.319+0000 7fe872910640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c11e360 con 0x7fe86c10fe90 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.319+0000 7fe872910640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c11e1e0 con 0x7fe86c10a340 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.319+0000 7fe872910640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe86c11e4e0 con 0x7fe86c10b190 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.320+0000 7fe87210f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c10fe90 0x7fe86c1b04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.320+0000 7fe87210f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c10fe90 0x7fe86c1b04a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37274/0 (socket says 172.21.15.31:37274) 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.320+0000 7fe87210f640 1 -- 172.21.15.31:0/534572773 learned_addr learned my addr 172.21.15.31:0/534572773 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:26.324 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.320+0000 7fe87190e640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe86c10a340 0x7fe86c1a8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.325 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.320+0000 7fe87110d640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe86c10b190 0x7fe86c1a9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.323+0000 7fe87210f640 1 -- 172.21.15.31:0/534572773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe86c10b190 msgr2=0x7fe86c1a9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:26.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.323+0000 7fe87210f640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe86c10b190 0x7fe86c1a9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.323+0000 7fe87210f640 1 -- 172.21.15.31:0/534572773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe86c10a340 msgr2=0x7fe86c1a8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:26.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.323+0000 7fe87210f640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe86c10a340 0x7fe86c1a8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.323+0000 7fe87210f640 1 -- 172.21.15.31:0/534572773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86c111c90 con 0x7fe86c10fe90 2024-04-25T16:12:26.327 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.325+0000 7fe87210f640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c10fe90 0x7fe86c1b04a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe86400b970 tx=0x7fe86400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:26.342 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.333+0000 7fe862ffd640 1 -- 172.21.15.31:0/534572773 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe864019070 con 0x7fe86c10fe90 2024-04-25T16:12:26.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.333+0000 7fe872910640 1 -- 172.21.15.31:0/534572773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86c111f80 con 0x7fe86c10fe90 2024-04-25T16:12:26.343 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.333+0000 7fe872910640 1 -- 172.21.15.31:0/534572773 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86c112550 con 0x7fe86c10fe90 2024-04-25T16:12:26.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.333+0000 7fe862ffd640 1 -- 172.21.15.31:0/534572773 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe864014440 con 0x7fe86c10fe90 2024-04-25T16:12:26.344 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.333+0000 7fe862ffd640 1 -- 172.21.15.31:0/534572773 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8640075b0 con 0x7fe86c10fe90 2024-04-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.340+0000 7fe87aa25740 1 Processor -- start 2024-04-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.341+0000 7fe87aa25740 1 -- start start 2024-04-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.341+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a37390 0x55f731dd99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.341+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731dd9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.341+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a166b0 0x55f731ddabf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.341+0000 7fe87aa25740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f731d3dd80 con 0x55f731a166b0 2024-04-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.341+0000 7fe87aa25740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f731cf7a20 con 0x55f731a0e4d0 2024-04-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.341+0000 7fe87aa25740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f731d39d40 con 0x55f731a37390 2024-04-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.342+0000 7fe87110d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a166b0 0x55f731ddabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.346 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.342+0000 7fe87110d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a166b0 0x55f731ddabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37278/0 (socket says 172.21.15.31:37278) 2024-04-25T16:12:26.351 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.342+0000 7fe87110d640 1 -- 172.21.15.31:0/744911878 learned_addr learned my addr 172.21.15.31:0/744911878 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.342+0000 7fe87190e640 1 --2- 172.21.15.31:0/744911878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731dd9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.343+0000 7fe87190e640 1 -- 172.21.15.31:0/744911878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a37390 msgr2=0x55f731dd99e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.343+0000 7fe87190e640 1 --2- 172.21.15.31:0/744911878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a37390 0x55f731dd99e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.352 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.343+0000 7fe87190e640 1 -- 172.21.15.31:0/744911878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a166b0 msgr2=0x55f731ddabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:26.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.343+0000 7fe87190e640 1 --2- 172.21.15.31:0/744911878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a166b0 0x55f731ddabf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.343+0000 7fe87190e640 1 -- 172.21.15.31:0/744911878 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f731d02c30 con 0x55f731a0e4d0 2024-04-25T16:12:26.353 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.348+0000 7fe862ffd640 1 -- 172.21.15.31:0/534572773 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe86401c070 con 0x7fe86c10fe90 2024-04-25T16:12:26.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.349+0000 7fe862ffd640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8580afc30 0x7fe8580b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.352+0000 7fe87190e640 1 --2- 172.21.15.31:0/744911878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731dd9f60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe86c074a30 tx=0x7fe85c00bbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93b6c452394d5a69 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:26.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.356+0000 7fe857fff640 1 -- 172.21.15.31:0/744911878 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c00be10 con 0x55f731a0e4d0 2024-04-25T16:12:26.358 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.356+0000 7fe857fff640 1 -- 172.21.15.31:0/744911878 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe85c019070 con 0x55f731a0e4d0 2024-04-25T16:12:26.359 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.356+0000 7fe862ffd640 1 -- 172.21.15.31:0/534572773 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe8640a2bc0 con 0x7fe86c10fe90 2024-04-25T16:12:26.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.357+0000 7fe87aa25740 1 -- 172.21.15.31:0/744911878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 msgr2=0x55f731dd9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:26.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.357+0000 7fe87aa25740 1 --2- 172.21.15.31:0/744911878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731dd9f60 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fe86c074a30 tx=0x7fe85c00bbc0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.359+0000 7fe87110d640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8580afc30 0x7fe8580b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.359+0000 7fe87aa25740 1 -- 172.21.15.31:0/744911878 shutdown_connections 2024-04-25T16:12:26.361 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.359+0000 7fe87aa25740 1 --2- 172.21.15.31:0/744911878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a166b0 0x55f731ddabf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.359+0000 7fe87aa25740 1 --2- 172.21.15.31:0/744911878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731dd9f60 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.359+0000 7fe87aa25740 1 --2- 172.21.15.31:0/744911878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a37390 0x55f731dd99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.359+0000 7fe87aa25740 1 -- 172.21.15.31:0/744911878 >> 172.21.15.31:0/744911878 conn(0x55f731a2b4e0 msgr2=0x55f731dd51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:26.362 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.359+0000 7fe87aa25740 1 -- 172.21.15.31:0/744911878 shutdown_connections 2024-04-25T16:12:26.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.362+0000 7fe87aa25740 1 -- 172.21.15.31:0/744911878 wait complete. 2024-04-25T16:12:26.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.362+0000 7fe87aa25740 1 -- start start 2024-04-25T16:12:26.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.363+0000 7fe87aa25740 1 Processor -- start 2024-04-25T16:12:26.365 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.363+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731de3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.363+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.363+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a37390 0x55f731deae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.363+0000 7fe87aa25740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55f731ddb750 con 0x55f731a37390 2024-04-25T16:12:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.363+0000 7fe87aa25740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55f731cc4dc0 con 0x55f731a0e4d0 2024-04-25T16:12:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.363+0000 7fe87aa25740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55f731ddb8d0 con 0x55f731a166b0 2024-04-25T16:12:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.363+0000 7fe87aa25740 10 client.0 _ll_register_callbacks cb 0x55f731ce0600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.364+0000 7fe87190e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731de3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.364+0000 7fe87190e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731de3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55048/0 (socket says 172.21.15.31:55048) 2024-04-25T16:12:26.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.364+0000 7fe87190e640 1 -- 172.21.15.31:0/2301859040 learned_addr learned my addr 172.21.15.31:0/2301859040 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:26.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.367+0000 7fe87190e640 1 -- 172.21.15.31:0/2301859040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 msgr2=0x55f731de3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:26.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.367+0000 7fe87190e640 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.368 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.367+0000 7fe87190e640 1 -- 172.21.15.31:0/2301859040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a37390 msgr2=0x55f731deae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.367+0000 7fe87190e640 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a37390 0x55f731deae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.367+0000 7fe87190e640 1 -- 172.21.15.31:0/2301859040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f731a4f3f0 con 0x55f731a0e4d0 2024-04-25T16:12:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.368+0000 7fe87190e640 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731de3980 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe85c008010 tx=0x7fe85c014830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac5683291622a2d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.368+0000 7fe8557fa640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:12:26.369 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.368+0000 7fe8557fa640 1 -- 172.21.15.31:0/2301859040 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c023020 con 0x55f731a0e4d0 2024-04-25T16:12:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.369+0000 7fe87aa25740 1 -- 172.21.15.31:0/2301859040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f731d4b990 con 0x55f731a0e4d0 2024-04-25T16:12:26.370 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.369+0000 7fe8557fa640 1 -- 172.21.15.31:0/2301859040 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe85c00db50 con 0x55f731a0e4d0 2024-04-25T16:12:26.373 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.372+0000 7fe87110d640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8580afc30 0x7fe8580b2140 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe8680096c0 tx=0x7fe8680314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:26.378 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.377+0000 7fe8557fa640 1 -- 172.21.15.31:0/2301859040 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe85c00dd50 con 0x55f731a0e4d0 2024-04-25T16:12:26.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.378+0000 7fe87aa25740 10 client.27530 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:26.379 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.378+0000 7fe87aa25740 1 -- 172.21.15.31:0/2301859040 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f731b197d0 con 0x55f731a0e4d0 2024-04-25T16:12:26.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.382+0000 7fe8557fa640 1 -- 172.21.15.31:0/2301859040 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f731d4b990 con 0x55f731a0e4d0 2024-04-25T16:12:26.383 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.382+0000 7fe8557fa640 1 -- 172.21.15.31:0/2301859040 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe85c013070 con 0x55f731a0e4d0 2024-04-25T16:12:26.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.382+0000 7fe87aa25740 10 client.27530 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:26.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.382+0000 7fe87aa25740 10 client.27530 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:12:26.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.382+0000 7fe87aa25740 20 client.27530 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:26.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.382+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55f7319f5f60 0x55f731df1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.382+0000 7fe87aa25740 4 client.27530 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:26.384 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.382+0000 7fe87aa25740 1 -- 172.21.15.31:0/2301859040 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f731d45800 con 0x55f7319f5f60 2024-04-25T16:12:26.386 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.384+0000 7fe87110d640 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55f7319f5f60 0x55f731df1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.395+0000 7fe87110d640 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55f7319f5f60 0x55f731df1af0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce0d8cb87ef07aae server_cookie=77b5b7fe17afec75 in_seq=0 out_seq=0 2024-04-25T16:12:26.399 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.397+0000 7fe8557fa640 10 client.27530 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:12:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.400+0000 7fe8557fa640 1 -- 172.21.15.31:0/2301859040 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f731d45800 con 0x55f7319f5f60 2024-04-25T16:12:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.400+0000 7fe8557fa640 10 client.27530 handle_command_reply: tid=0 2024-04-25T16:12:26.401 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.400+0000 7fe87aa25740 1 client.27530 shutdown 2024-04-25T16:12:26.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.404+0000 7fe87aa25740 1 -- 172.21.15.31:0/2301859040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 msgr2=0x55f731de3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:26.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.404+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731de3980 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe85c008010 tx=0x7fe85c014830 comp rx=0 tx=0).stop 2024-04-25T16:12:26.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.405+0000 7fe87aa25740 1 -- 172.21.15.31:0/2301859040 shutdown_connections 2024-04-25T16:12:26.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.405+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55f7319f5f60 0x55f731df1af0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.405+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a37390 0x55f731deae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.405+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.405+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2301859040 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a0e4d0 0x55f731de3980 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fe85c008010 tx=0x7fe85c014830 comp rx=0 tx=0).stop 2024-04-25T16:12:26.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.405+0000 7fe87aa25740 1 -- 172.21.15.31:0/2301859040 >> 172.21.15.31:0/2301859040 conn(0x55f731a2b4e0 msgr2=0x55f731dd8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:26.407 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.405+0000 7fe8557fa640 0 client.27530 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:12:26.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.410+0000 7fe87aa25740 1 -- 172.21.15.31:0/2301859040 shutdown_connections 2024-04-25T16:12:26.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.410+0000 7fe87aa25740 1 -- 172.21.15.31:0/2301859040 wait complete. 2024-04-25T16:12:26.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.410+0000 7fe87aa25740 20 client.27530 trim_cache size 0 max 16384 2024-04-25T16:12:26.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.447+0000 7fe87aa25740 1 Processor -- start 2024-04-25T16:12:26.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.447+0000 7fe87aa25740 1 -- start start 2024-04-25T16:12:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.447+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.447+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a37390 0x55f731de24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.447+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7319cb3a0 0x55f731de2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.447+0000 7fe87aa25740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe85000e590 con 0x55f7319cb3a0 2024-04-25T16:12:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.447+0000 7fe87aa25740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe8500100d0 con 0x55f731a37390 2024-04-25T16:12:26.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.447+0000 7fe87aa25740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe85000ec60 con 0x55f731a166b0 2024-04-25T16:12:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.448+0000 7fe87110d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a37390 0x55f731de24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.448+0000 7fe87110d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a37390 0x55f731de24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55064/0 (socket says 172.21.15.31:55064) 2024-04-25T16:12:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.448+0000 7fe87110d640 1 -- 172.21.15.31:0/2695288128 learned_addr learned my addr 172.21.15.31:0/2695288128 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.449+0000 7fe87110d640 1 -- 172.21.15.31:0/2695288128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 msgr2=0x55f731de1fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:12:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.449+0000 7fe87110d640 1 --2- 172.21.15.31:0/2695288128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de1fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.449+0000 7fe87110d640 1 -- 172.21.15.31:0/2695288128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7319cb3a0 msgr2=0x55f731de2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:26.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.449+0000 7fe87110d640 1 --2- 172.21.15.31:0/2695288128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7319cb3a0 0x55f731de2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.449+0000 7fe87110d640 1 -- 172.21.15.31:0/2695288128 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f731dc6ba0 con 0x55f731a37390 2024-04-25T16:12:26.451 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:26 smithi031 ceph-mon[29140]: pgmap v685: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 757 MiB/s wr, 1.79k op/s 2024-04-25T16:12:26.451 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.450+0000 7fe87110d640 1 --2- 172.21.15.31:0/2695288128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a37390 0x55f731de24f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe868031440 tx=0x7fe868040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37fe52a7d3d0994c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:26.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.450+0000 7fe8577fe640 1 -- 172.21.15.31:0/2695288128 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868051530 con 0x55f731a37390 2024-04-25T16:12:26.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.450+0000 7fe8577fe640 1 -- 172.21.15.31:0/2695288128 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8680315a0 con 0x55f731a37390 2024-04-25T16:12:26.452 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.451+0000 7fe8577fe640 1 -- 172.21.15.31:0/2695288128 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86803d6b0 con 0x55f731a37390 2024-04-25T16:12:26.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.451+0000 7fe87aa25740 1 -- 172.21.15.31:0/2695288128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a37390 msgr2=0x55f731de24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:26.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.451+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2695288128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a37390 0x55f731de24f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fe868031440 tx=0x7fe868040000 comp rx=0 tx=0).stop 2024-04-25T16:12:26.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.452+0000 7fe87aa25740 1 -- 172.21.15.31:0/2695288128 shutdown_connections 2024-04-25T16:12:26.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.452+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2695288128 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f7319cb3a0 0x55f731de2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.452+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2695288128 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f731a37390 0x55f731de24f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.452+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2695288128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.452+0000 7fe87aa25740 1 -- 172.21.15.31:0/2695288128 >> 172.21.15.31:0/2695288128 conn(0x55f731a2b4e0 msgr2=0x55f731dde600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:26.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.455+0000 7fe87aa25740 1 -- 172.21.15.31:0/2695288128 shutdown_connections 2024-04-25T16:12:26.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.457+0000 7fe87aa25740 1 -- 172.21.15.31:0/2695288128 wait complete. 2024-04-25T16:12:26.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.457+0000 7fe87aa25740 1 -- start start 2024-04-25T16:12:26.459 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.457+0000 7fe87aa25740 1 Processor -- start 2024-04-25T16:12:26.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.458+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7319cb3a0 0x55f731de6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.458+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.458+0000 7fe87aa25740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a37390 0x55f731de9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.458+0000 7fe87aa25740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe85000fa00 con 0x55f731a37390 2024-04-25T16:12:26.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.458+0000 7fe87aa25740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe85000f330 con 0x55f7319cb3a0 2024-04-25T16:12:26.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.458+0000 7fe87aa25740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe85000c570 con 0x55f731a166b0 2024-04-25T16:12:26.460 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.458+0000 7fe87aa25740 10 client.0 _ll_register_callbacks cb 0x7fe85c00b5b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.459+0000 7fe87110d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.459+0000 7fe87110d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56824/0 (socket says 172.21.15.31:56824) 2024-04-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.459+0000 7fe87110d640 1 -- 172.21.15.31:0/2215791078 learned_addr learned my addr 172.21.15.31:0/2215791078 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.459+0000 7fe87210f640 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a37390 0x55f731de9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.460+0000 7fe87110d640 1 -- 172.21.15.31:0/2215791078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7319cb3a0 msgr2=0x55f731de6580 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.460+0000 7fe87110d640 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7319cb3a0 0x55f731de6580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.460+0000 7fe87110d640 1 -- 172.21.15.31:0/2215791078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a37390 msgr2=0x55f731de9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.460+0000 7fe87110d640 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a37390 0x55f731de9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.464 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.460+0000 7fe87110d640 1 -- 172.21.15.31:0/2215791078 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f731de9fc0 con 0x55f731a166b0 2024-04-25T16:12:26.465 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.462+0000 7fe87110d640 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de6ac0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe868031930 tx=0x7fe868031960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=494d1fbb27892ed7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.469+0000 7fe8557fa640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:12:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.469+0000 7fe8557fa640 1 -- 172.21.15.31:0/2215791078 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86805f060 con 0x55f731a166b0 2024-04-25T16:12:26.470 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.469+0000 7fe8557fa640 1 -- 172.21.15.31:0/2215791078 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe868051530 con 0x55f731a166b0 2024-04-25T16:12:26.471 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.470+0000 7fe87aa25740 1 -- 172.21.15.31:0/2215791078 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe8680315a0 con 0x55f731a166b0 2024-04-25T16:12:26.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.472+0000 7fe8557fa640 1 -- 172.21.15.31:0/2215791078 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fe868051730 con 0x55f731a166b0 2024-04-25T16:12:26.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.472+0000 7fe87aa25740 10 client.27511 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:26.473 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.472+0000 7fe87aa25740 1 -- 172.21.15.31:0/2215791078 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f731e586a0 con 0x55f731a166b0 2024-04-25T16:12:26.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.475+0000 7fe8557fa640 1 -- 172.21.15.31:0/2215791078 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8680315a0 con 0x55f731a166b0 2024-04-25T16:12:26.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.475+0000 7fe8557fa640 1 -- 172.21.15.31:0/2215791078 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7fe868044040 con 0x55f731a166b0 2024-04-25T16:12:26.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.475+0000 7fe87aa25740 10 client.27511 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:26.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.475+0000 7fe87aa25740 10 client.27511 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:12:26.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.475+0000 7fe87aa25740 20 client.27511 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:26.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.475+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55f731e58fe0 0x55f731e5b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:26.476 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.475+0000 7fe87aa25740 4 client.27511 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:12:26.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.475+0000 7fe87aa25740 1 -- 172.21.15.31:0/2215791078 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f731cc0610 con 0x55f731e58fe0 2024-04-25T16:12:26.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.476+0000 7fe87190e640 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55f731e58fe0 0x55f731e5b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:26.481 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.480+0000 7fe87190e640 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55f731e58fe0 0x55f731e5b400 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9cda9beab4fcbf5e server_cookie=2378dfe09b814014 in_seq=0 out_seq=0 2024-04-25T16:12:26.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.480+0000 7fe8557fa640 10 client.27511 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:12:26.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.480+0000 7fe8557fa640 1 -- 172.21.15.31:0/2215791078 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f731cc0610 con 0x55f731e58fe0 2024-04-25T16:12:26.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.480+0000 7fe8557fa640 10 client.27511 handle_command_reply: tid=0 2024-04-25T16:12:26.482 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.480+0000 7fe87aa25740 1 client.27511 shutdown 2024-04-25T16:12:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.505+0000 7fe87aa25740 1 -- 172.21.15.31:0/2215791078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 msgr2=0x55f731de6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.505+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de6ac0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe868031930 tx=0x7fe868031960 comp rx=0 tx=0).stop 2024-04-25T16:12:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.506+0000 7fe87aa25740 1 -- 172.21.15.31:0/2215791078 shutdown_connections 2024-04-25T16:12:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.506+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55f731e58fe0 0x55f731e5b400 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.506+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55f731a37390 0x55f731de9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.507 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.506+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55f731a166b0 0x55f731de6ac0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.506+0000 7fe87aa25740 1 --2- 172.21.15.31:0/2215791078 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55f7319cb3a0 0x55f731de6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.506+0000 7fe87aa25740 1 -- 172.21.15.31:0/2215791078 >> 172.21.15.31:0/2215791078 conn(0x55f731a2b4e0 msgr2=0x55f731de1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:26.508 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.507+0000 7fe8557fa640 0 client.27511 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:12:26.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.507+0000 7fe87aa25740 1 -- 172.21.15.31:0/2215791078 shutdown_connections 2024-04-25T16:12:26.509 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.508+0000 7fe87aa25740 1 -- 172.21.15.31:0/2215791078 wait complete. 2024-04-25T16:12:26.510 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.508+0000 7fe87aa25740 20 client.27511 trim_cache size 0 max 16384 2024-04-25T16:12:26.527 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.532+0000 7fe872910640 1 -- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8580afc30 msgr2=0x7fe8580b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.533+0000 7fe872910640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8580afc30 0x7fe8580b2140 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fe8680096c0 tx=0x7fe8680314a0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.533+0000 7fe872910640 1 -- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c10fe90 msgr2=0x7fe86c1b04a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.533+0000 7fe872910640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c10fe90 0x7fe86c1b04a0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fe86400b970 tx=0x7fe86400be30 comp rx=0 tx=0).stop 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.533+0000 7fe872910640 1 -- 172.21.15.31:0/534572773 shutdown_connections 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.533+0000 7fe872910640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe8580afc30 0x7fe8580b2140 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.533+0000 7fe872910640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe86c10fe90 0x7fe86c1b04a0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.533+0000 7fe872910640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe86c10b190 0x7fe86c1a9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.533+0000 7fe872910640 1 --2- 172.21.15.31:0/534572773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe86c10a340 0x7fe86c1a8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:26.535 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.533+0000 7fe872910640 1 -- 172.21.15.31:0/534572773 >> 172.21.15.31:0/534572773 conn(0x7fe86c070320 msgr2=0x7fe86c10be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:26.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.534+0000 7fe872910640 1 -- 172.21.15.31:0/534572773 shutdown_connections 2024-04-25T16:12:26.536 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:26.534+0000 7fe872910640 1 -- 172.21.15.31:0/534572773 wait complete. 2024-04-25T16:12:26.583 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:12:26.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:26 smithi186 ceph-mon[33258]: pgmap v685: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 757 MiB/s wr, 1.79k op/s 2024-04-25T16:12:26.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:26 smithi063 ceph-mon[32413]: pgmap v685: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 757 MiB/s wr, 1.79k op/s 2024-04-25T16:12:27.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.037+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3216503376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b010b310 msgr2=0x7f89b010b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:27.039 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.037+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3216503376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b010b310 0x7f89b010b720 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f89a8009ca0 tx=0x7f89a802a8d0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.040+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3216503376 shutdown_connections 2024-04-25T16:12:27.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.040+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3216503376 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89b0110030 0x7f89b0110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.040+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3216503376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b010b310 0x7f89b010b720 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.040+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3216503376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b010a420 0x7f89b010add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.040+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3216503376 >> 172.21.15.31:0/3216503376 conn(0x7f89b00704e0 msgr2=0x7f89b0072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:27.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.044+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3216503376 shutdown_connections 2024-04-25T16:12:27.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.047+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3216503376 wait complete. 2024-04-25T16:12:27.049 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.048+0000 7f89b4a3c640 1 Processor -- start 2024-04-25T16:12:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.052+0000 7f89b4a3c640 1 -- start start 2024-04-25T16:12:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.052+0000 7f89b4a3c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b010a420 0x7f89b01a9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.052+0000 7f89b4a3c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b010b310 0x7f89b01a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.052+0000 7f89b4a3c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89b0110030 0x7f89b01b0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.052+0000 7f89b4a3c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f89b011e490 con 0x7f89b0110030 2024-04-25T16:12:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.052+0000 7f89b4a3c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f89b011e310 con 0x7f89b010a420 2024-04-25T16:12:27.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.052+0000 7f89b4a3c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89b011e610 con 0x7f89b010b310 2024-04-25T16:12:27.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.054+0000 7f89ae59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b010a420 0x7f89b01a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.056 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.055+0000 7f89aed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89b0110030 0x7f89b01b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.055+0000 7f89aed9d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89b0110030 0x7f89b01b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37326/0 (socket says 172.21.15.31:37326) 2024-04-25T16:12:27.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.055+0000 7f89ae59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b010a420 0x7f89b01a9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55086/0 (socket says 172.21.15.31:55086) 2024-04-25T16:12:27.057 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.055+0000 7f89ae59c640 1 -- 172.21.15.31:0/3794447358 learned_addr learned my addr 172.21.15.31:0/3794447358 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.059+0000 7f89aed9d640 1 -- 172.21.15.31:0/3794447358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b010b310 msgr2=0x7f89b01a95b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:12:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.059+0000 7f89aed9d640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b010b310 0x7f89b01a95b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.059+0000 7f89aed9d640 1 -- 172.21.15.31:0/3794447358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b010a420 msgr2=0x7f89b01a9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:27.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.059+0000 7f89aed9d640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b010a420 0x7f89b01a9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.059+0000 7f89aed9d640 1 -- 172.21.15.31:0/3794447358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89b01b0bb0 con 0x7f89b0110030 2024-04-25T16:12:27.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.070+0000 7f89aed9d640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89b0110030 0x7f89b01b0540 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f89a000e970 tx=0x7f89a000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:27.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.073+0000 7f899f7fe640 1 -- 172.21.15.31:0/3794447358 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a000ccc0 con 0x7f89b0110030 2024-04-25T16:12:27.074 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.073+0000 7f899f7fe640 1 -- 172.21.15.31:0/3794447358 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a0005020 con 0x7f89b0110030 2024-04-25T16:12:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.073+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3794447358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89b01b0ea0 con 0x7f89b0110030 2024-04-25T16:12:27.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.073+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3794447358 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89b01b1310 con 0x7f89b0110030 2024-04-25T16:12:27.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.075+0000 7f899f7fe640 1 -- 172.21.15.31:0/3794447358 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a000ccc0 con 0x7f89b0110030 2024-04-25T16:12:27.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.086+0000 7f89bb9c5740 1 Processor -- start 2024-04-25T16:12:27.089 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.087+0000 7f899f7fe640 1 -- 172.21.15.31:0/3794447358 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f89a00051d0 con 0x7f89b0110030 2024-04-25T16:12:27.090 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.089+0000 7f899f7fe640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89980aff30 0x7f89980b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.090+0000 7f899f7fe640 1 -- 172.21.15.31:0/3794447358 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f89a000cac0 con 0x7f89b0110030 2024-04-25T16:12:27.091 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.090+0000 7f89add9b640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89980aff30 0x7f89980b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.091+0000 7f89bb9c5740 1 -- start start 2024-04-25T16:12:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.092+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010c4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.092+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.092+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010c5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.092+0000 7f89bb9c5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555d01028d80 con 0x555d00cf94d0 2024-04-25T16:12:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.092+0000 7f89bb9c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555d00fe2a20 con 0x555d00d22390 2024-04-25T16:12:27.093 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.092+0000 7f89bb9c5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555d01024d40 con 0x555d00d016b0 2024-04-25T16:12:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.093+0000 7f89add9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.093+0000 7f89add9b640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010c4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37330/0 (socket says 172.21.15.31:37330) 2024-04-25T16:12:27.094 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.093+0000 7f89add9b640 1 -- 172.21.15.31:0/620516424 learned_addr learned my addr 172.21.15.31:0/620516424 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.093+0000 7f89aed9d640 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010c5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.093+0000 7f89add9b640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89980aff30 0x7f89980b2440 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f89a8005e30 tx=0x7f89a8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:27.095 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.093+0000 7f89ae59c640 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010c4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.095+0000 7f89add9b640 1 -- 172.21.15.31:0/620516424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 msgr2=0x555d010c5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.095+0000 7f89add9b640 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010c5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.095+0000 7f89add9b640 1 -- 172.21.15.31:0/620516424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 msgr2=0x555d010c4810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.095+0000 7f89add9b640 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010c4810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.095+0000 7f89add9b640 1 -- 172.21.15.31:0/620516424 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d00fedc30 con 0x555d00cf94d0 2024-04-25T16:12:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.097+0000 7f89ae59c640 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:12:27.098 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.097+0000 7f89aed9d640 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010c5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:12:27.099 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.097+0000 7f89add9b640 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010c4d90 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f89a80099b0 tx=0x7f89a803c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f390a7206d59bcde server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.099+0000 7f899cff9640 1 -- 172.21.15.31:0/620516424 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a804b070 con 0x555d00cf94d0 2024-04-25T16:12:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.099+0000 7f899cff9640 1 -- 172.21.15.31:0/620516424 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a8036040 con 0x555d00cf94d0 2024-04-25T16:12:27.100 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.099+0000 7f899cff9640 1 -- 172.21.15.31:0/620516424 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a8047d10 con 0x555d00cf94d0 2024-04-25T16:12:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.101+0000 7f89bb9c5740 1 -- 172.21.15.31:0/620516424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 msgr2=0x555d010c4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:27.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.101+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010c4d90 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f89a80099b0 tx=0x7f89a803c000 comp rx=0 tx=0).stop 2024-04-25T16:12:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.101+0000 7f89bb9c5740 1 -- 172.21.15.31:0/620516424 shutdown_connections 2024-04-25T16:12:27.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.101+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010c5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.101+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010c4d90 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.101+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/620516424 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010c4810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.101+0000 7f89bb9c5740 1 -- 172.21.15.31:0/620516424 >> 172.21.15.31:0/620516424 conn(0x555d00d164e0 msgr2=0x555d010c0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.102+0000 7f89bb9c5740 1 -- 172.21.15.31:0/620516424 shutdown_connections 2024-04-25T16:12:27.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.102+0000 7f89bb9c5740 1 -- 172.21.15.31:0/620516424 wait complete. 2024-04-25T16:12:27.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.103+0000 7f89bb9c5740 1 -- start start 2024-04-25T16:12:27.105 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.104+0000 7f89bb9c5740 1 Processor -- start 2024-04-25T16:12:27.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.104+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.105+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 0x555d010dfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.105+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 0x555d010e6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.105+0000 7f89bb9c5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555d010c64d0 con 0x555d00cf94d0 2024-04-25T16:12:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.105+0000 7f89bb9c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555d00fafdc0 con 0x555d00d016b0 2024-04-25T16:12:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.105+0000 7f89bb9c5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555d010c6650 con 0x555d00d22390 2024-04-25T16:12:27.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.106+0000 7f89bb9c5740 10 client.0 _ll_register_callbacks cb 0x555d00fcb600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.106+0000 7f89ae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.106+0000 7f89ae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37334/0 (socket says 172.21.15.31:37334) 2024-04-25T16:12:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.106+0000 7f89ae59c640 1 -- 172.21.15.31:0/4091719445 learned_addr learned my addr 172.21.15.31:0/4091719445 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.107+0000 7f89add9b640 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 0x555d010dfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.107+0000 7f89aed9d640 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 0x555d010e6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.108+0000 7f89ae59c640 1 -- 172.21.15.31:0/4091719445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 msgr2=0x555d010e6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:27.109 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.108+0000 7f89ae59c640 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 0x555d010e6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.108+0000 7f89ae59c640 1 -- 172.21.15.31:0/4091719445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 msgr2=0x555d010dfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.108+0000 7f89ae59c640 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 0x555d010dfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.108+0000 7f89ae59c640 1 -- 172.21.15.31:0/4091719445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d00d3a3f0 con 0x555d00cf94d0 2024-04-25T16:12:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.109+0000 7f89ae59c640 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df6b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f89a4011410 tx=0x7f89a40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44125f37940a38f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:27.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.109+0000 7f8995ffb640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:12:27.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.110+0000 7f8995ffb640 1 -- 172.21.15.31:0/4091719445 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a4012980 con 0x555d00cf94d0 2024-04-25T16:12:27.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.110+0000 7f8995ffb640 1 -- 172.21.15.31:0/4091719445 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f89a4012b20 con 0x555d00cf94d0 2024-04-25T16:12:27.111 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.110+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4091719445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89a8036040 con 0x555d00cf94d0 2024-04-25T16:12:27.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.111+0000 7f8995ffb640 1 -- 172.21.15.31:0/4091719445 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f89a4012d20 con 0x555d00cf94d0 2024-04-25T16:12:27.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.111+0000 7f89bb9c5740 10 client.17826 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:27.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.111+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4091719445 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d00e047d0 con 0x555d00cf94d0 2024-04-25T16:12:27.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.112+0000 7f8995ffb640 1 -- 172.21.15.31:0/4091719445 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a401d930 con 0x555d00cf94d0 2024-04-25T16:12:27.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.112+0000 7f8995ffb640 1 -- 172.21.15.31:0/4091719445 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f89a4027740 con 0x555d00cf94d0 2024-04-25T16:12:27.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.112+0000 7f89bb9c5740 10 client.17826 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:27.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.112+0000 7f89bb9c5740 10 client.17826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:27.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.112+0000 7f89bb9c5740 20 client.17826 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:27.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.113+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555d00ce0f60 0x555d010ed2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.113+0000 7f89bb9c5740 4 client.17826 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:27.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.113+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4091719445 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f89a8036040 con 0x555d00ce0f60 2024-04-25T16:12:27.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.116+0000 7f89add9b640 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555d00ce0f60 0x555d010ed2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.119+0000 7f89add9b640 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555d00ce0f60 0x555d010ed2e0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a1f364985cdaa31 server_cookie=2a6f83ab549d3f41 in_seq=0 out_seq=0 2024-04-25T16:12:27.120 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.119+0000 7f8995ffb640 10 client.17826 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.120+0000 7f8995ffb640 1 -- 172.21.15.31:0/4091719445 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f89a8036040 con 0x555d00ce0f60 2024-04-25T16:12:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.120+0000 7f8995ffb640 10 client.17826 handle_command_reply: tid=0 2024-04-25T16:12:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.121+0000 7f89bb9c5740 1 client.17826 shutdown 2024-04-25T16:12:27.123 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.121+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4091719445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 msgr2=0x555d010df6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.121+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df6b0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f89a4011410 tx=0x7f89a40118d0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.122+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4091719445 shutdown_connections 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.122+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555d00ce0f60 0x555d010ed2e0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.122+0000 7f8995ffb640 0 client.17826 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.122+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 0x555d010e6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.122+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 0x555d010dfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.122+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4091719445 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df6b0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.122+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4091719445 >> 172.21.15.31:0/4091719445 conn(0x555d00d164e0 msgr2=0x555d010c2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.122+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4091719445 shutdown_connections 2024-04-25T16:12:27.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.122+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4091719445 wait complete. 2024-04-25T16:12:27.126 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.123+0000 7f89bb9c5740 20 client.17826 trim_cache size 0 max 16384 2024-04-25T16:12:27.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.174+0000 7f89bb9c5740 1 Processor -- start 2024-04-25T16:12:27.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.175+0000 7f89bb9c5740 1 -- start start 2024-04-25T16:12:27.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.175+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.176+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010e6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.176+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.176+0000 7f89bb9c5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f89b01b1930 con 0x555d00cf94d0 2024-04-25T16:12:27.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.176+0000 7f89bb9c5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f89b01b3470 con 0x555d00d22390 2024-04-25T16:12:27.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.176+0000 7f89bb9c5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89b01b2000 con 0x555d00d016b0 2024-04-25T16:12:27.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.177+0000 7f89ae59c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.177+0000 7f89aed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.177+0000 7f89aed9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010f1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55110/0 (socket says 172.21.15.31:55110) 2024-04-25T16:12:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.177+0000 7f89aed9d640 1 -- 172.21.15.31:0/4266611319 learned_addr learned my addr 172.21.15.31:0/4266611319 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.177+0000 7f89add9b640 1 --2- 172.21.15.31:0/4266611319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010e6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.178+0000 7f89add9b640 1 -- 172.21.15.31:0/4266611319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 msgr2=0x555d010f1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.178+0000 7f89add9b640 1 --2- 172.21.15.31:0/4266611319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010f1a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.178+0000 7f89add9b640 1 -- 172.21.15.31:0/4266611319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 msgr2=0x555d010df990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:27.179 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.178+0000 7f89add9b640 1 --2- 172.21.15.31:0/4266611319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.178+0000 7f89add9b640 1 -- 172.21.15.31:0/4266611319 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d010eb040 con 0x555d00d016b0 2024-04-25T16:12:27.180 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.178+0000 7f89add9b640 1 --2- 172.21.15.31:0/4266611319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010e6a70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x555d010e0720 tx=0x7f89a802add0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e011a7c5cdf84790 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:27.184 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.183+0000 7f8997fff640 1 -- 172.21.15.31:0/4266611319 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a803f040 con 0x555d00d016b0 2024-04-25T16:12:27.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.183+0000 7f8997fff640 1 -- 172.21.15.31:0/4266611319 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a8061070 con 0x555d00d016b0 2024-04-25T16:12:27.185 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.183+0000 7f8997fff640 1 -- 172.21.15.31:0/4266611319 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a804b040 con 0x555d00d016b0 2024-04-25T16:12:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.187+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4266611319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 msgr2=0x555d010e6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:27.194 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.187+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4266611319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010e6a70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x555d010e0720 tx=0x7f89a802add0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.187+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4266611319 shutdown_connections 2024-04-25T16:12:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.187+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4266611319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d22390 0x555d010f1a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.187+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4266611319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d016b0 0x555d010e6a70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.187+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/4266611319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010df990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.187+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4266611319 >> 172.21.15.31:0/4266611319 conn(0x555d00d164e0 msgr2=0x555d010df280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:27.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.220+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4266611319 shutdown_connections 2024-04-25T16:12:27.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.220+0000 7f89bb9c5740 1 -- 172.21.15.31:0/4266611319 wait complete. 2024-04-25T16:12:27.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.221+0000 7f89bb9c5740 1 -- start start 2024-04-25T16:12:27.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.221+0000 7f89bb9c5740 1 Processor -- start 2024-04-25T16:12:27.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.223+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010e5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.223+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 0x555d010e6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.225 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.224+0000 7f89bb9c5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 0x555d011538f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.224+0000 7f89add9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 0x555d010e6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.224+0000 7f89add9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 0x555d010e6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55118/0 (socket says 172.21.15.31:55118) 2024-04-25T16:12:27.226 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.224+0000 7f89add9b640 1 -- 172.21.15.31:0/2902430254 learned_addr learned my addr 172.21.15.31:0/2902430254 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:27.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.225+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f89b01b2da0 con 0x555d00cf94d0 2024-04-25T16:12:27.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.226+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f89b01b26d0 con 0x555d00d016b0 2024-04-25T16:12:27.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.226+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f89b0118680 con 0x555d00d22390 2024-04-25T16:12:27.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.226+0000 7f89aed9d640 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 0x555d011538f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.227 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.226+0000 7f89bb9c5740 10 client.0 _ll_register_callbacks cb 0x7f89a400b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:27.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.228+0000 7f89ae59c640 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010e5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.229+0000 7f89aed9d640 1 -- 172.21.15.31:0/2902430254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 msgr2=0x555d010e6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:27.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.229+0000 7f89aed9d640 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 0x555d010e6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.229+0000 7f89aed9d640 1 -- 172.21.15.31:0/2902430254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 msgr2=0x555d010e5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:27.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.229+0000 7f89aed9d640 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010e5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.231 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.229+0000 7f89aed9d640 1 -- 172.21.15.31:0/2902430254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d01153eb0 con 0x555d00d22390 2024-04-25T16:12:27.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.236+0000 7f89aed9d640 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 0x555d011538f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f89a00a3390 tx=0x7f89a00a0f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1b0a993a3d7997cd server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:27.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.247+0000 7f8995ffb640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:12:27.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.247+0000 7f8995ffb640 1 -- 172.21.15.31:0/2902430254 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89a00a3050 con 0x555d00d22390 2024-04-25T16:12:27.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.247+0000 7f8995ffb640 1 -- 172.21.15.31:0/2902430254 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f89a00ad070 con 0x555d00d22390 2024-04-25T16:12:27.250 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.248+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89a8061070 con 0x555d00d22390 2024-04-25T16:12:27.268 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.267+0000 7f8995ffb640 1 -- 172.21.15.31:0/2902430254 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f89a00a31f0 con 0x555d00d22390 2024-04-25T16:12:27.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.269+0000 7f89bb9c5740 10 client.27523 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:27.271 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.269+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555d01155180 con 0x555d00d22390 2024-04-25T16:12:27.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.475+0000 7f8995ffb640 1 -- 172.21.15.31:0/2902430254 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89a8061070 con 0x555d00d22390 2024-04-25T16:12:27.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.475+0000 7f8995ffb640 1 -- 172.21.15.31:0/2902430254 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f89a000ccc0 con 0x555d00d22390 2024-04-25T16:12:27.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.476+0000 7f89bb9c5740 10 client.27523 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:27.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.476+0000 7f89bb9c5740 10 client.27523 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:27.477 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.476+0000 7f89bb9c5740 20 client.27523 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:27.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.476+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555d00cb63a0 0x555d01157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:27.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.476+0000 7f89bb9c5740 4 client.27523 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:12:27.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.476+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d01036990 con 0x555d00cb63a0 2024-04-25T16:12:27.478 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.477+0000 7f89ae59c640 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555d00cb63a0 0x555d01157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.478+0000 7f89ae59c640 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555d00cb63a0 0x555d01157ba0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5a2579f73c6a9b2 server_cookie=6502b44615099e17 in_seq=0 out_seq=0 2024-04-25T16:12:27.479 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.478+0000 7f8995ffb640 10 client.27523 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:27.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.482+0000 7f8995ffb640 1 -- 172.21.15.31:0/2902430254 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d01036990 con 0x555d00cb63a0 2024-04-25T16:12:27.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.482+0000 7f8995ffb640 10 client.27523 handle_command_reply: tid=0 2024-04-25T16:12:27.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.482+0000 7f89bb9c5740 1 client.27523 shutdown 2024-04-25T16:12:27.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.482+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 msgr2=0x555d011538f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:27.483 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.482+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 0x555d011538f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f89a00a3390 tx=0x7f89a00a0f80 comp rx=0 tx=0).stop 2024-04-25T16:12:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.483+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 shutdown_connections 2024-04-25T16:12:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.483+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555d00cb63a0 0x555d01157ba0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.484 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.483+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555d00d22390 0x555d011538f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.483+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555d00d016b0 0x555d010e6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.483+0000 7f89bb9c5740 1 --2- 172.21.15.31:0/2902430254 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555d00cf94d0 0x555d010e5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.483+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 >> 172.21.15.31:0/2902430254 conn(0x555d00d164e0 msgr2=0x555d010df0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.483+0000 7f8995ffb640 0 client.27523 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.483+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 shutdown_connections 2024-04-25T16:12:27.485 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.483+0000 7f89bb9c5740 1 -- 172.21.15.31:0/2902430254 wait complete. 2024-04-25T16:12:27.488 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.487+0000 7f89bb9c5740 20 client.27523 trim_cache size 0 max 16384 2024-04-25T16:12:27.489 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:12:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.490+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89980aff30 msgr2=0x7f89980b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.490+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89980aff30 0x7f89980b2440 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f89a8005e30 tx=0x7f89a8005fd0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.491 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.490+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89b0110030 msgr2=0x7f89b01b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:27.492 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.490+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89b0110030 0x7f89b01b0540 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f89a000e970 tx=0x7f89a000ee30 comp rx=0 tx=0).stop 2024-04-25T16:12:27.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.498+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3794447358 shutdown_connections 2024-04-25T16:12:27.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.498+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f89980aff30 0x7f89980b2440 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.498+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f89b0110030 0x7f89b01b0540 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.498+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89b010b310 0x7f89b01a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.499+0000 7f89b4a3c640 1 --2- 172.21.15.31:0/3794447358 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f89b010a420 0x7f89b01a9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:27.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.499+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3794447358 >> 172.21.15.31:0/3794447358 conn(0x7f89b00704e0 msgr2=0x7f89b010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:27.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.501+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3794447358 shutdown_connections 2024-04-25T16:12:27.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:27.501+0000 7f89b4a3c640 1 -- 172.21.15.31:0/3794447358 wait complete. 2024-04-25T16:12:28.551 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-25T16:12:28.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:12:28.552 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-25T16:12:28.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.969+0000 7f57cb26e640 1 -- 172.21.15.31:0/587339503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c4110030 msgr2=0x7f57c4110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:28.970 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.969+0000 7f57cb26e640 1 --2- 172.21.15.31:0/587339503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c4110030 0x7f57c4110440 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f57b4009900 tx=0x7f57b402a410 comp rx=0 tx=0).stop 2024-04-25T16:12:28.971 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.970+0000 7f57cb26e640 1 -- 172.21.15.31:0/587339503 shutdown_connections 2024-04-25T16:12:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.970+0000 7f57cb26e640 1 --2- 172.21.15.31:0/587339503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c4110030 0x7f57c4110440 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.970+0000 7f57cb26e640 1 --2- 172.21.15.31:0/587339503 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57c410b310 0x7f57c410b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.970+0000 7f57cb26e640 1 --2- 172.21.15.31:0/587339503 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57c410a420 0x7f57c410add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:28.972 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.970+0000 7f57cb26e640 1 -- 172.21.15.31:0/587339503 >> 172.21.15.31:0/587339503 conn(0x7f57c40704e0 msgr2=0x7f57c4072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.975+0000 7f57cb26e640 1 -- 172.21.15.31:0/587339503 shutdown_connections 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.975+0000 7f57cb26e640 1 -- 172.21.15.31:0/587339503 wait complete. 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.976+0000 7f57cb26e640 1 Processor -- start 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.976+0000 7f57cb26e640 1 -- start start 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.976+0000 7f57cb26e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57c410a420 0x7f57c41a9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.976+0000 7f57cb26e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c410b310 0x7f57c41a9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.976+0000 7f57cb26e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57c4110030 0x7f57c41b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.977+0000 7f57cb26e640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f57c411e590 con 0x7f57c4110030 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.977+0000 7f57cb26e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57c411e410 con 0x7f57c410a420 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.977+0000 7f57cb26e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57c411e710 con 0x7f57c410b310 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.977+0000 7f57c900a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57c410a420 0x7f57c41a9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.977+0000 7f57c900a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57c410a420 0x7f57c41a9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55138/0 (socket says 172.21.15.31:55138) 2024-04-25T16:12:28.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.977+0000 7f57c900a640 1 -- 172.21.15.31:0/233375860 learned_addr learned my addr 172.21.15.31:0/233375860 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:28.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.977+0000 7f57c8809640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c410b310 0x7f57c41a9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:28.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.978+0000 7f57c900a640 1 -- 172.21.15.31:0/233375860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c410b310 msgr2=0x7f57c41a9770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:28.980 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.978+0000 7f57c900a640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c410b310 0x7f57c41a9770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:28.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.978+0000 7f57c900a640 1 -- 172.21.15.31:0/233375860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57c4110030 msgr2=0x7f57c41b06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:28.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.978+0000 7f57c900a640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57c4110030 0x7f57c41b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:28.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.978+0000 7f57c900a640 1 -- 172.21.15.31:0/233375860 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c41b0d20 con 0x7f57c410a420 2024-04-25T16:12:28.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.979+0000 7f57c900a640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57c410a420 0x7f57c41a9230 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f57c000b450 tx=0x7f57c000b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:28.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.980+0000 7f57ba7fc640 1 -- 172.21.15.31:0/233375860 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0015020 con 0x7f57c410a420 2024-04-25T16:12:28.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.980+0000 7f57ba7fc640 1 -- 172.21.15.31:0/233375860 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57c0011d40 con 0x7f57c410a420 2024-04-25T16:12:28.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.980+0000 7f57ba7fc640 1 -- 172.21.15.31:0/233375860 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c0010e10 con 0x7f57c410a420 2024-04-25T16:12:28.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.981+0000 7f57cb26e640 1 -- 172.21.15.31:0/233375860 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c41b0fb0 con 0x7f57c410a420 2024-04-25T16:12:28.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.981+0000 7f57cb26e640 1 -- 172.21.15.31:0/233375860 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c41b1440 con 0x7f57c410a420 2024-04-25T16:12:28.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.986+0000 7f57d2202740 1 Processor -- start 2024-04-25T16:12:28.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.987+0000 7f57ba7fc640 1 -- 172.21.15.31:0/233375860 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f57c0018020 con 0x7f57c410a420 2024-04-25T16:12:28.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.993+0000 7f57ba7fc640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57b00afc80 0x7f57b00b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:28.995 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.994+0000 7f57ba7fc640 1 -- 172.21.15.31:0/233375860 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f57c00de5a0 con 0x7f57c410a420 2024-04-25T16:12:29.003 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.998+0000 7f57c980b640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57b00afc80 0x7f57b00b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.999+0000 7f57d2202740 1 -- start start 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.999+0000 7f57d2202740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69bc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.999+0000 7f57d2202740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69bca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.999+0000 7f57d2202740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 0x5616e69c3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.999+0000 7f57d2202740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5616e6921b00 con 0x5616e661b410 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.999+0000 7f57d2202740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5616e68bd920 con 0x5616e65f2550 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:28.999+0000 7f57d2202740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5616e691dac0 con 0x5616e65fa730 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.000+0000 7f57c8809640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69bca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.000+0000 7f57c8809640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69bca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55142/0 (socket says 172.21.15.31:55142) 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.000+0000 7f57c8809640 1 -- 172.21.15.31:0/2490089355 learned_addr learned my addr 172.21.15.31:0/2490089355 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.000+0000 7f57c900a640 1 --2- 172.21.15.31:0/2490089355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69bc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.004 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.000+0000 7f57c8809640 1 -- 172.21.15.31:0/2490089355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 msgr2=0x5616e69c3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:29.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.000+0000 7f57c8809640 1 --2- 172.21.15.31:0/2490089355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 0x5616e69c3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.000+0000 7f57c8809640 1 -- 172.21.15.31:0/2490089355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 msgr2=0x5616e69bc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:29.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.000+0000 7f57c8809640 1 --2- 172.21.15.31:0/2490089355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69bc4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.000+0000 7f57c8809640 1 -- 172.21.15.31:0/2490089355 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616e66fb7e0 con 0x5616e65f2550 2024-04-25T16:12:29.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.001+0000 7f57c900a640 1 --2- 172.21.15.31:0/2490089355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69bc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:12:29.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.001+0000 7f57c8809640 1 --2- 172.21.15.31:0/2490089355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69bca20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00b500 tx=0x7f57bc0096b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73155df03340b5aa server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:29.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.002+0000 7f57af7fe640 1 -- 172.21.15.31:0/2490089355 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc00a970 con 0x5616e65f2550 2024-04-25T16:12:29.005 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.002+0000 7f57af7fe640 1 -- 172.21.15.31:0/2490089355 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57bc019070 con 0x5616e65f2550 2024-04-25T16:12:29.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.004+0000 7f57d2202740 1 -- 172.21.15.31:0/2490089355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 msgr2=0x5616e69bca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:29.006 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.004+0000 7f57d2202740 1 --2- 172.21.15.31:0/2490089355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69bca20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f57bc00b500 tx=0x7f57bc0096b0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.013 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.012+0000 7f57c980b640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57b00afc80 0x7f57b00b2190 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005dc0 tx=0x7f57b4031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.015+0000 7f57d2202740 1 -- 172.21.15.31:0/2490089355 shutdown_connections 2024-04-25T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.015+0000 7f57d2202740 1 --2- 172.21.15.31:0/2490089355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 0x5616e69c3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.015+0000 7f57d2202740 1 --2- 172.21.15.31:0/2490089355 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69bca20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.015+0000 7f57d2202740 1 --2- 172.21.15.31:0/2490089355 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69bc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.016 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.015+0000 7f57d2202740 1 -- 172.21.15.31:0/2490089355 >> 172.21.15.31:0/2490089355 conn(0x5616e660f560 msgr2=0x5616e69b7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:29.028 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.026+0000 7f57d2202740 1 -- 172.21.15.31:0/2490089355 shutdown_connections 2024-04-25T16:12:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.027+0000 7f57d2202740 1 -- 172.21.15.31:0/2490089355 wait complete. 2024-04-25T16:12:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.028+0000 7f57d2202740 1 -- start start 2024-04-25T16:12:29.029 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.028+0000 7f57d2202740 1 Processor -- start 2024-04-25T16:12:29.036 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.035+0000 7f57d2202740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69c62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.035+0000 7f57d2202740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 0x5616e69c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.035+0000 7f57d2202740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.035+0000 7f57d2202740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5616e69c43c0 con 0x5616e661b410 2024-04-25T16:12:29.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.035+0000 7f57d2202740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5616e68bb5d0 con 0x5616e65f2550 2024-04-25T16:12:29.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.035+0000 7f57d2202740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5616e69c4540 con 0x5616e65fa730 2024-04-25T16:12:29.037 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.035+0000 7f57d2202740 10 client.0 _ll_register_callbacks cb 0x5616e68a9080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:29.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.035+0000 7f57c8809640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 0x5616e69c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.036+0000 7f57c900a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69c62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.036+0000 7f57c900a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69c62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55144/0 (socket says 172.21.15.31:55144) 2024-04-25T16:12:29.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.036+0000 7f57c900a640 1 -- 172.21.15.31:0/2769206629 learned_addr learned my addr 172.21.15.31:0/2769206629 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:29.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.037+0000 7f57c900a640 1 -- 172.21.15.31:0/2769206629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 msgr2=0x5616e69c67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:29.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.035+0000 7f57c8809640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 0x5616e69c67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56926/0 (socket says 172.21.15.31:56926) 2024-04-25T16:12:29.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.039+0000 7f57c900a640 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 0x5616e69c67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.039+0000 7f57c900a640 1 -- 172.21.15.31:0/2769206629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 msgr2=0x5616e69cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:29.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.039+0000 7f57c900a640 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.042 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.039+0000 7f57c900a640 1 -- 172.21.15.31:0/2769206629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616e675bf70 con 0x5616e65f2550 2024-04-25T16:12:29.043 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.042+0000 7f57c8809640 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 0x5616e69c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:12:29.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.042+0000 7f57c900a640 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69c62a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f57c00a3d00 tx=0x7f57c0010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f26c958423e98ef5 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:29.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.044+0000 7f57acff9640 10 client.0 ms_handle_connect on v2:172.21.15.63:3300/0 2024-04-25T16:12:29.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.045+0000 7f57acff9640 1 -- 172.21.15.31:0/2769206629 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c00a6a90 con 0x5616e65f2550 2024-04-25T16:12:29.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.045+0000 7f57d2202740 1 -- 172.21.15.31:0/2769206629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57bc019070 con 0x5616e65f2550 2024-04-25T16:12:29.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.045+0000 7f57acff9640 1 -- 172.21.15.31:0/2769206629 <== mon.2 v2:172.21.15.63:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f57c0011aa0 con 0x5616e65f2550 2024-04-25T16:12:29.060 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.056+0000 7f57acff9640 1 -- 172.21.15.31:0/2769206629 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f57c00c7050 con 0x5616e65f2550 2024-04-25T16:12:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.059+0000 7f57d2202740 10 client.27557 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:29.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.059+0000 7f57d2202740 1 -- 172.21.15.31:0/2769206629 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5616e65f3170 con 0x5616e65f2550 2024-04-25T16:12:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.062+0000 7f57acff9640 1 -- 172.21.15.31:0/2769206629 <== mon.2 v2:172.21.15.63:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57bc019070 con 0x5616e65f2550 2024-04-25T16:12:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.063+0000 7f57acff9640 1 -- 172.21.15.31:0/2769206629 <== mon.2 v2:172.21.15.63:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f57c00a5cf0 con 0x5616e65f2550 2024-04-25T16:12:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.063+0000 7f57d2202740 10 client.27557 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.063+0000 7f57d2202740 10 client.27557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.063+0000 7f57d2202740 20 client.27557 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.064+0000 7f57d2202740 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5616e65da1f0 0x5616e69d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.064+0000 7f57d2202740 4 client.27557 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:29.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.064+0000 7f57d2202740 1 -- 172.21.15.31:0/2769206629 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5616e692f710 con 0x5616e65da1f0 2024-04-25T16:12:29.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.066+0000 7f57c8809640 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5616e65da1f0 0x5616e69d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.070 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.068+0000 7f57c8809640 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5616e65da1f0 0x5616e69d3c60 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d78a3af28a228f2d server_cookie=33f08bf01b24fbf3 in_seq=0 out_seq=0 2024-04-25T16:12:29.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.077+0000 7f57acff9640 10 client.27557 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:29.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.077+0000 7f57acff9640 1 -- 172.21.15.31:0/2769206629 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5616e692f710 con 0x5616e65da1f0 2024-04-25T16:12:29.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.077+0000 7f57acff9640 10 client.27557 handle_command_reply: tid=0 2024-04-25T16:12:29.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.077+0000 7f57d2202740 1 client.27557 shutdown 2024-04-25T16:12:29.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.077+0000 7f57d2202740 1 -- 172.21.15.31:0/2769206629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 msgr2=0x5616e69c62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:29.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.077+0000 7f57d2202740 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69c62a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f57c00a3d00 tx=0x7f57c0010800 comp rx=0 tx=0).stop 2024-04-25T16:12:29.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.085+0000 7f57d2202740 1 -- 172.21.15.31:0/2769206629 shutdown_connections 2024-04-25T16:12:29.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.085+0000 7f57d2202740 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5616e65da1f0 0x5616e69d3c60 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.085+0000 7f57d2202740 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.085+0000 7f57d2202740 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65fa730 0x5616e69c67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.085+0000 7f57d2202740 1 --2- 172.21.15.31:0/2769206629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65f2550 0x5616e69c62a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.085+0000 7f57acff9640 0 client.27557 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:29.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.085+0000 7f57d2202740 1 -- 172.21.15.31:0/2769206629 >> 172.21.15.31:0/2769206629 conn(0x5616e660f560 msgr2=0x5616e69b6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:29.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.099+0000 7f57d2202740 1 -- 172.21.15.31:0/2769206629 shutdown_connections 2024-04-25T16:12:29.101 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.099+0000 7f57d2202740 1 -- 172.21.15.31:0/2769206629 wait complete. 2024-04-25T16:12:29.102 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.100+0000 7f57d2202740 20 client.27557 trim_cache size 0 max 16384 2024-04-25T16:12:29.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.167+0000 7f57d2202740 1 Processor -- start 2024-04-25T16:12:29.174 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.172+0000 7f57d2202740 1 -- start start 2024-04-25T16:12:29.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.173+0000 7f57d2202740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65f2550 0x5616e69c4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.175 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.174+0000 7f57d2202740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 0x5616e69c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.174+0000 7f57d2202740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69d8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.176 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.174+0000 7f57c900a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65f2550 0x5616e69c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.174+0000 7f57c900a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65f2550 0x5616e69c4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56934/0 (socket says 172.21.15.31:56934) 2024-04-25T16:12:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.174+0000 7f57c900a640 1 -- 172.21.15.31:0/1184750272 learned_addr learned my addr 172.21.15.31:0/1184750272 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.174+0000 7f57d2202740 1 -- 172.21.15.31:0/1184750272 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41ba1a0 con 0x5616e661b410 2024-04-25T16:12:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.174+0000 7f57d2202740 1 -- 172.21.15.31:0/1184750272 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41bbce0 con 0x5616e65fa730 2024-04-25T16:12:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.174+0000 7f57d2202740 1 -- 172.21.15.31:0/1184750272 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41ba870 con 0x5616e65f2550 2024-04-25T16:12:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.175+0000 7f57c980b640 1 --2- 172.21.15.31:0/1184750272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69d8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.177 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.175+0000 7f57c8809640 1 --2- 172.21.15.31:0/1184750272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 0x5616e69c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.176+0000 7f57c8809640 1 -- 172.21.15.31:0/1184750272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65f2550 msgr2=0x5616e69c4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.176+0000 7f57c8809640 1 --2- 172.21.15.31:0/1184750272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65f2550 0x5616e69c4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.176+0000 7f57c8809640 1 -- 172.21.15.31:0/1184750272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 msgr2=0x5616e69d8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.176+0000 7f57c8809640 1 --2- 172.21.15.31:0/1184750272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69d8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.178 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.176+0000 7f57c8809640 1 -- 172.21.15.31:0/1184750272 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616e69c6920 con 0x5616e65fa730 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.181+0000 7f57c8809640 1 --2- 172.21.15.31:0/1184750272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 0x5616e69c4ea0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x5616e69c7310 tx=0x7f57bc0149d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2df142d03cee728c server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.182+0000 7f57aeffd640 1 -- 172.21.15.31:0/1184750272 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc0144c0 con 0x5616e65fa730 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.182+0000 7f57aeffd640 1 -- 172.21.15.31:0/1184750272 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57bc019040 con 0x5616e65fa730 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.182+0000 7f57aeffd640 1 -- 172.21.15.31:0/1184750272 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57bc028330 con 0x5616e65fa730 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.184+0000 7f57d2202740 1 -- 172.21.15.31:0/1184750272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 msgr2=0x5616e69c4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.184+0000 7f57d2202740 1 --2- 172.21.15.31:0/1184750272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 0x5616e69c4ea0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x5616e69c7310 tx=0x7f57bc0149d0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.194+0000 7f57d2202740 1 -- 172.21.15.31:0/1184750272 shutdown_connections 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.194+0000 7f57d2202740 1 --2- 172.21.15.31:0/1184750272 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e661b410 0x5616e69d8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.194+0000 7f57d2202740 1 --2- 172.21.15.31:0/1184750272 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 0x5616e69c4ea0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.195 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.194+0000 7f57d2202740 1 --2- 172.21.15.31:0/1184750272 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e65f2550 0x5616e69c4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.196 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.194+0000 7f57d2202740 1 -- 172.21.15.31:0/1184750272 >> 172.21.15.31:0/1184750272 conn(0x5616e660f560 msgr2=0x5616e69c0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.200+0000 7f57d2202740 1 -- 172.21.15.31:0/1184750272 shutdown_connections 2024-04-25T16:12:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.200+0000 7f57d2202740 1 -- 172.21.15.31:0/1184750272 wait complete. 2024-04-25T16:12:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.201+0000 7f57d2202740 1 -- start start 2024-04-25T16:12:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.201+0000 7f57d2202740 1 Processor -- start 2024-04-25T16:12:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.202+0000 7f57d2202740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e65f2550 0x5616e69ca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.203+0000 7f57d2202740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 0x5616e69ca780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.203+0000 7f57d2202740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e661b410 0x5616e69cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.203+0000 7f57d2202740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41bb610 con 0x5616e65f2550 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.203+0000 7f57d2202740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41baf40 con 0x5616e65fa730 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.203+0000 7f57d2202740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57c41b8180 con 0x5616e661b410 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.203+0000 7f57d2202740 10 client.0 _ll_register_callbacks cb 0x7f57c00a6270 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.204+0000 7f57c900a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e65f2550 0x5616e69ca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.206+0000 7f57c900a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e65f2550 0x5616e69ca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37394/0 (socket says 172.21.15.31:37394) 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.206+0000 7f57c900a640 1 -- 172.21.15.31:0/3077014438 learned_addr learned my addr 172.21.15.31:0/3077014438 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.207+0000 7f57c900a640 1 -- 172.21.15.31:0/3077014438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e661b410 msgr2=0x5616e69cb6b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.207+0000 7f57c900a640 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e661b410 0x5616e69cb6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.207+0000 7f57c900a640 1 -- 172.21.15.31:0/3077014438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 msgr2=0x5616e69ca780 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.207+0000 7f57c900a640 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 0x5616e69ca780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.207+0000 7f57c900a640 1 -- 172.21.15.31:0/3077014438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616e69c6050 con 0x5616e65f2550 2024-04-25T16:12:29.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.207+0000 7f57c900a640 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e65f2550 0x5616e69ca240 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f57c0006e70 tx=0x7f57c00a72d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9116995c813c6eb server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:29.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.208+0000 7f57acff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:12:29.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.208+0000 7f57acff9640 1 -- 172.21.15.31:0/3077014438 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57c00b4080 con 0x5616e65f2550 2024-04-25T16:12:29.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.209+0000 7f57d2202740 1 -- 172.21.15.31:0/3077014438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57bc019040 con 0x5616e65f2550 2024-04-25T16:12:29.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.209+0000 7f57acff9640 1 -- 172.21.15.31:0/3077014438 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f57c00c90b0 con 0x5616e65f2550 2024-04-25T16:12:29.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.209+0000 7f57acff9640 1 -- 172.21.15.31:0/3077014438 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f57c00de3d0 con 0x5616e65f2550 2024-04-25T16:12:29.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.209+0000 7f57d2202740 10 client.17841 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:29.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.209+0000 7f57d2202740 1 -- 172.21.15.31:0/3077014438 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5616e69ccdd0 con 0x5616e65f2550 2024-04-25T16:12:29.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.210+0000 7f57acff9640 1 -- 172.21.15.31:0/3077014438 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57bc019040 con 0x5616e65f2550 2024-04-25T16:12:29.215 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.214+0000 7f57acff9640 1 -- 172.21.15.31:0/3077014438 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f57c00a5760 con 0x5616e65f2550 2024-04-25T16:12:29.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.214+0000 7f57d2202740 10 client.17841 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:29.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.214+0000 7f57d2202740 10 client.17841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:29.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.214+0000 7f57d2202740 20 client.17841 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:29.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.214+0000 7f57d2202740 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5616e65af630 0x5616e6a60430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:29.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.214+0000 7f57d2202740 4 client.17841 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-25T16:12:29.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.214+0000 7f57d2202740 1 -- 172.21.15.31:0/3077014438 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5616e6929580 con 0x5616e65af630 2024-04-25T16:12:29.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.216+0000 7f57c8809640 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5616e65af630 0x5616e6a60430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:29.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.217+0000 7f57c8809640 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5616e65af630 0x5616e6a60430 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9083ee38c90c6742 server_cookie=fcfd4e3ec046b0da in_seq=0 out_seq=0 2024-04-25T16:12:29.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.218+0000 7f57acff9640 10 client.17841 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:29.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:29 smithi063 ceph-mon[32413]: pgmap v686: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 877 MiB/s wr, 2.19k op/s 2024-04-25T16:12:29.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:29 smithi186 ceph-mon[33258]: pgmap v686: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 877 MiB/s wr, 2.19k op/s 2024-04-25T16:12:29.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:29 smithi031 ceph-mon[29140]: pgmap v686: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 877 MiB/s wr, 2.19k op/s 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.852+0000 7f57acff9640 1 -- 172.21.15.31:0/3077014438 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5616e6929580 con 0x5616e65af630 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.852+0000 7f57acff9640 10 client.17841 handle_command_reply: tid=0 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.852+0000 7f57d2202740 1 client.17841 shutdown 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 -- 172.21.15.31:0/3077014438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e65f2550 msgr2=0x5616e69ca240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e65f2550 0x5616e69ca240 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f57c0006e70 tx=0x7f57c00a72d0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 -- 172.21.15.31:0/3077014438 shutdown_connections 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5616e65af630 0x5616e6a60430 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5616e661b410 0x5616e69cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5616e65fa730 0x5616e69ca780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 --2- 172.21.15.31:0/3077014438 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x5616e65f2550 0x5616e69ca240 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 -- 172.21.15.31:0/3077014438 >> 172.21.15.31:0/3077014438 conn(0x5616e660f560 msgr2=0x5616e69c4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:29.855 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57acff9640 0 client.17841 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:29.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 -- 172.21.15.31:0/3077014438 shutdown_connections 2024-04-25T16:12:29.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.853+0000 7f57d2202740 1 -- 172.21.15.31:0/3077014438 wait complete. 2024-04-25T16:12:29.856 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.854+0000 7f57d2202740 20 client.17841 trim_cache size 0 max 16384 2024-04-25T16:12:29.856 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:12:29.856 INFO:teuthology.orchestra.run.smithi031.stdout: "return_code": 0, 2024-04-25T16:12:29.857 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_tag": "710ae31f-7e5e-4124-8ee7-49badec9c8f8", 2024-04-25T16:12:29.857 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": "asynchronous" 2024-04-25T16:12:29.857 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:12:29.857 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 -- 172.21.15.31:0/233375860 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57b00afc80 msgr2=0x7f57b00b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57b00afc80 0x7f57b00b2190 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f57b4005dc0 tx=0x7f57b4031400 comp rx=0 tx=0).stop 2024-04-25T16:12:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 -- 172.21.15.31:0/233375860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57c410a420 msgr2=0x7f57c41a9230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57c410a420 0x7f57c41a9230 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f57c000b450 tx=0x7f57c000b910 comp rx=0 tx=0).stop 2024-04-25T16:12:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 -- 172.21.15.31:0/233375860 shutdown_connections 2024-04-25T16:12:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57b00afc80 0x7f57b00b2190 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f57c4110030 0x7f57c41b06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57c410b310 0x7f57c41a9770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 --2- 172.21.15.31:0/233375860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57c410a420 0x7f57c41a9230 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:29.858 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 -- 172.21.15.31:0/233375860 >> 172.21.15.31:0/233375860 conn(0x7f57c40704e0 msgr2=0x7f57c410c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:29.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.855+0000 7f57cb26e640 1 -- 172.21.15.31:0/233375860 shutdown_connections 2024-04-25T16:12:29.859 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:29.856+0000 7f57cb26e640 1 -- 172.21.15.31:0/233375860 wait complete. 2024-04-25T16:12:29.941 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:12:29.941 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:12:30.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.517+0000 7faa32ae2640 1 -- 172.21.15.31:0/2181066101 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa2c110030 msgr2=0x7faa2c110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:30.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.517+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2181066101 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa2c110030 0x7faa2c110440 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7faa24009ca0 tx=0x7faa2402a8f0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.517+0000 7faa32ae2640 1 -- 172.21.15.31:0/2181066101 shutdown_connections 2024-04-25T16:12:30.519 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.517+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2181066101 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa2c110030 0x7faa2c110440 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.517+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2181066101 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa2c10b310 0x7faa2c10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.517+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2181066101 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c10a420 0x7faa2c10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.517+0000 7faa32ae2640 1 -- 172.21.15.31:0/2181066101 >> 172.21.15.31:0/2181066101 conn(0x7faa2c0704e0 msgr2=0x7faa2c072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:30.520 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.519+0000 7faa32ae2640 1 -- 172.21.15.31:0/2181066101 shutdown_connections 2024-04-25T16:12:30.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.522+0000 7faa32ae2640 1 -- 172.21.15.31:0/2181066101 wait complete. 2024-04-25T16:12:30.524 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.523+0000 7faa32ae2640 1 Processor -- start 2024-04-25T16:12:30.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.523+0000 7faa32ae2640 1 -- start start 2024-04-25T16:12:30.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.524+0000 7faa32ae2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa2c10a420 0x7faa2c116bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.524+0000 7faa32ae2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa2c10b310 0x7faa2c117130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.524+0000 7faa32ae2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c1b6c40 0x7faa2c1b9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.524+0000 7faa32ae2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c11e540 con 0x7faa2c10a420 2024-04-25T16:12:30.525 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.524+0000 7faa32ae2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c11e3c0 con 0x7faa2c10b310 2024-04-25T16:12:30.526 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.524+0000 7faa32ae2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faa2c11e6c0 con 0x7faa2c1b6c40 2024-04-25T16:12:30.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.524+0000 7faa3107f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c1b6c40 0x7faa2c1b9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.525+0000 7faa3107f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c1b6c40 0x7faa2c1b9060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:56962/0 (socket says 172.21.15.31:56962) 2024-04-25T16:12:30.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.525+0000 7faa3107f640 1 -- 172.21.15.31:0/2453456823 learned_addr learned my addr 172.21.15.31:0/2453456823 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:30.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.525+0000 7faa2bfff640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa2c10b310 0x7faa2c117130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.525+0000 7faa3107f640 1 -- 172.21.15.31:0/2453456823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa2c10b310 msgr2=0x7faa2c117130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:30.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.525+0000 7faa3107f640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa2c10b310 0x7faa2c117130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.525+0000 7faa3107f640 1 -- 172.21.15.31:0/2453456823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa2c10a420 msgr2=0x7faa2c116bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:30.527 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.525+0000 7faa3107f640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa2c10a420 0x7faa2c116bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.525+0000 7faa3107f640 1 -- 172.21.15.31:0/2453456823 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa2c1b95a0 con 0x7faa2c1b6c40 2024-04-25T16:12:30.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.526+0000 7faa2bfff640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa2c10b310 0x7faa2c117130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:12:30.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.526+0000 7faa3107f640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c1b6c40 0x7faa2c1b9060 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7faa24009dc0 tx=0x7faa24030800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:30.528 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.526+0000 7faa29ffb640 1 -- 172.21.15.31:0/2453456823 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa24030d00 con 0x7faa2c1b6c40 2024-04-25T16:12:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.526+0000 7faa29ffb640 1 -- 172.21.15.31:0/2453456823 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa24030ea0 con 0x7faa2c1b6c40 2024-04-25T16:12:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.526+0000 7faa29ffb640 1 -- 172.21.15.31:0/2453456823 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2403a660 con 0x7faa2c1b6c40 2024-04-25T16:12:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.526+0000 7faa32ae2640 1 -- 172.21.15.31:0/2453456823 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa2c1b9770 con 0x7faa2c1b6c40 2024-04-25T16:12:30.529 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.527+0000 7faa32ae2640 1 -- 172.21.15.31:0/2453456823 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa2c1b9cc0 con 0x7faa2c1b6c40 2024-04-25T16:12:30.538 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.537+0000 7faa3998e740 1 Processor -- start 2024-04-25T16:12:30.539 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.537+0000 7faa3998e740 1 -- start start 2024-04-25T16:12:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.538+0000 7faa3998e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 0x55eed30bca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.538+0000 7faa3998e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cf24d0 0x55eed30bcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.538+0000 7faa3998e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cfa6b0 0x55eed30bdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.539+0000 7faa3998e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55eed3021d80 con 0x55eed2d1b390 2024-04-25T16:12:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.539+0000 7faa3998e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55eed2fdb8b0 con 0x55eed2cfa6b0 2024-04-25T16:12:30.540 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.539+0000 7faa3998e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55eed301dd40 con 0x55eed2cf24d0 2024-04-25T16:12:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.539+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cfa6b0 0x55eed30bdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.539+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cfa6b0 0x55eed30bdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55190/0 (socket says 172.21.15.31:55190) 2024-04-25T16:12:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.539+0000 7faa2bfff640 1 -- 172.21.15.31:0/1572000577 learned_addr learned my addr 172.21.15.31:0/1572000577 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:30.541 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.539+0000 7faa3087e640 1 --2- 172.21.15.31:0/1572000577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cf24d0 0x55eed30bcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.540+0000 7faa2bfff640 1 -- 172.21.15.31:0/1572000577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cf24d0 msgr2=0x55eed30bcfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:30.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.540+0000 7faa2bfff640 1 --2- 172.21.15.31:0/1572000577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cf24d0 0x55eed30bcfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.540+0000 7faa2bfff640 1 -- 172.21.15.31:0/1572000577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 msgr2=0x55eed30bca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:30.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.540+0000 7faa2bfff640 1 --2- 172.21.15.31:0/1572000577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 0x55eed30bca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.542 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.541+0000 7faa2bfff640 1 -- 172.21.15.31:0/1572000577 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eed2faca40 con 0x55eed2cfa6b0 2024-04-25T16:12:30.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.541+0000 7faa3087e640 1 --2- 172.21.15.31:0/1572000577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cf24d0 0x55eed30bcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:12:30.543 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.541+0000 7faa2bfff640 1 --2- 172.21.15.31:0/1572000577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cfa6b0 0x55eed30bdc70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faa20009d90 tx=0x7faa20012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae9bc6862fa4cf7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:30.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.547+0000 7faa29ffb640 1 -- 172.21.15.31:0/2453456823 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7faa2403a800 con 0x7faa2c1b6c40 2024-04-25T16:12:30.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.547+0000 7faa12ffd640 1 -- 172.21.15.31:0/1572000577 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2001b640 con 0x55eed2cfa6b0 2024-04-25T16:12:30.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.547+0000 7faa12ffd640 1 -- 172.21.15.31:0/1572000577 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa2001bc90 con 0x55eed2cfa6b0 2024-04-25T16:12:30.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.547+0000 7faa12ffd640 1 -- 172.21.15.31:0/1572000577 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa2000fcf0 con 0x55eed2cfa6b0 2024-04-25T16:12:30.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.549+0000 7faa3998e740 1 -- 172.21.15.31:0/1572000577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cfa6b0 msgr2=0x55eed30bdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:30.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.549+0000 7faa3998e740 1 --2- 172.21.15.31:0/1572000577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cfa6b0 0x55eed30bdc70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7faa20009d90 tx=0x7faa20012820 comp rx=0 tx=0).stop 2024-04-25T16:12:30.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.550+0000 7faa3998e740 1 -- 172.21.15.31:0/1572000577 shutdown_connections 2024-04-25T16:12:30.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.550+0000 7faa3998e740 1 --2- 172.21.15.31:0/1572000577 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cfa6b0 0x55eed30bdc70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.550+0000 7faa3998e740 1 --2- 172.21.15.31:0/1572000577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cf24d0 0x55eed30bcfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.550+0000 7faa3998e740 1 --2- 172.21.15.31:0/1572000577 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 0x55eed30bca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.550+0000 7faa3998e740 1 -- 172.21.15.31:0/1572000577 >> 172.21.15.31:0/1572000577 conn(0x55eed2d0f4e0 msgr2=0x55eed30b8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:30.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.552+0000 7faa3998e740 1 -- 172.21.15.31:0/1572000577 shutdown_connections 2024-04-25T16:12:30.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.552+0000 7faa3998e740 1 -- 172.21.15.31:0/1572000577 wait complete. 2024-04-25T16:12:30.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.553+0000 7faa3998e740 1 -- start start 2024-04-25T16:12:30.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.554+0000 7faa3998e740 1 Processor -- start 2024-04-25T16:12:30.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.555+0000 7faa3998e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 0x55eed30c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.556+0000 7faa3998e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.556+0000 7faa3998e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 0x55eed30cde70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.556+0000 7faa3998e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55eed30be780 con 0x55eed2d1b390 2024-04-25T16:12:30.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.556+0000 7faa3998e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55eed2fa8d10 con 0x55eed2cf24d0 2024-04-25T16:12:30.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.556+0000 7faa3998e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55eed30be900 con 0x55eed2cfa6b0 2024-04-25T16:12:30.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.557+0000 7faa3998e740 10 client.0 _ll_register_callbacks cb 0x55eed2fd5210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:30.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.558+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 0x55eed30c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.558+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 0x55eed30cde70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.558+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 0x55eed30cde70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37434/0 (socket says 172.21.15.31:37434) 2024-04-25T16:12:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.558+0000 7faa2bfff640 1 -- 172.21.15.31:0/2546112200 learned_addr learned my addr 172.21.15.31:0/2546112200 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.558+0000 7faa3087e640 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.558+0000 7faa29ffb640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa140afc00 0x7faa140b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.559+0000 7faa29ffb640 1 -- 172.21.15.31:0/2453456823 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faa24100370 con 0x7faa2c1b6c40 2024-04-25T16:12:30.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.561+0000 7faa3107f640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa140afc00 0x7faa140b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.562+0000 7faa3087e640 1 -- 172.21.15.31:0/2546112200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 msgr2=0x55eed30c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.562+0000 7faa3087e640 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 0x55eed30c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.562+0000 7faa3087e640 1 -- 172.21.15.31:0/2546112200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 msgr2=0x55eed30cde70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.562+0000 7faa3087e640 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 0x55eed30cde70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.564 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.562+0000 7faa3087e640 1 -- 172.21.15.31:0/2546112200 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eed2dfd7d0 con 0x55eed2cfa6b0 2024-04-25T16:12:30.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.564+0000 7faa3087e640 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30c6f10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55eed30bc780 tx=0x7faa1c00d5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f89e1c0cc61b8da server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:30.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.564+0000 7faa0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:12:30.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.564+0000 7faa3107f640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa140afc00 0x7faa140b2110 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7faa240c8500 tx=0x7faa240c4040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:30.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.566+0000 7faa0bfff640 1 -- 172.21.15.31:0/2546112200 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c014430 con 0x55eed2cfa6b0 2024-04-25T16:12:30.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.566+0000 7faa3998e740 1 -- 172.21.15.31:0/2546112200 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa2001bc90 con 0x55eed2cfa6b0 2024-04-25T16:12:30.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.566+0000 7faa0bfff640 1 -- 172.21.15.31:0/2546112200 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faa1c0145d0 con 0x55eed2cfa6b0 2024-04-25T16:12:30.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.570+0000 7faa0bfff640 1 -- 172.21.15.31:0/2546112200 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faa1c0147d0 con 0x55eed2cfa6b0 2024-04-25T16:12:30.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.570+0000 7faa3998e740 10 client.27541 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:30.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.570+0000 7faa3998e740 1 -- 172.21.15.31:0/2546112200 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eed2d333f0 con 0x55eed2cfa6b0 2024-04-25T16:12:30.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.574+0000 7faa0bfff640 1 -- 172.21.15.31:0/2546112200 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa2001bc90 con 0x55eed2cfa6b0 2024-04-25T16:12:30.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.574+0000 7faa0bfff640 1 -- 172.21.15.31:0/2546112200 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7faa1c013070 con 0x55eed2cfa6b0 2024-04-25T16:12:30.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.574+0000 7faa3998e740 10 client.27541 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:30.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.574+0000 7faa3998e740 10 client.27541 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:30.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.574+0000 7faa3998e740 20 client.27541 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:30.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.574+0000 7faa3998e740 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55eed2cd9f60 0x55eed30d4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.574+0000 7faa3998e740 4 client.27541 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:30.576 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.574+0000 7faa3998e740 1 -- 172.21.15.31:0/2546112200 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eed302f990 con 0x55eed2cd9f60 2024-04-25T16:12:30.578 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.577+0000 7faa2bfff640 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55eed2cd9f60 0x55eed30d4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.586+0000 7faa2bfff640 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55eed2cd9f60 0x55eed30d4890 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64b6c2ba227b2751 server_cookie=856de0884baad3b8 in_seq=0 out_seq=0 2024-04-25T16:12:30.588 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.586+0000 7faa0bfff640 10 client.27541 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:30.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.595+0000 7faa0bfff640 1 -- 172.21.15.31:0/2546112200 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55eed302f990 con 0x55eed2cd9f60 2024-04-25T16:12:30.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.595+0000 7faa0bfff640 10 client.27541 handle_command_reply: tid=0 2024-04-25T16:12:30.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.595+0000 7faa3998e740 1 client.27541 shutdown 2024-04-25T16:12:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.596+0000 7faa3998e740 1 -- 172.21.15.31:0/2546112200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 msgr2=0x55eed30c6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.596+0000 7faa3998e740 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30c6f10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55eed30bc780 tx=0x7faa1c00d5f0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.596+0000 7faa3998e740 1 -- 172.21.15.31:0/2546112200 shutdown_connections 2024-04-25T16:12:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.596+0000 7faa3998e740 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55eed2cd9f60 0x55eed30d4890 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.596+0000 7faa3998e740 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2d1b390 0x55eed30cde70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.596+0000 7faa3998e740 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30c6f10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.596+0000 7faa3998e740 1 --2- 172.21.15.31:0/2546112200 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 0x55eed30c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.596+0000 7faa3998e740 1 -- 172.21.15.31:0/2546112200 >> 172.21.15.31:0/2546112200 conn(0x55eed2d0f4e0 msgr2=0x55eed30b73f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:30.599 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.596+0000 7faa0bfff640 0 client.27541 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:30.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.603+0000 7faa3998e740 1 -- 172.21.15.31:0/2546112200 shutdown_connections 2024-04-25T16:12:30.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.604+0000 7faa3998e740 1 -- 172.21.15.31:0/2546112200 wait complete. 2024-04-25T16:12:30.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.604+0000 7faa3998e740 20 client.27541 trim_cache size 0 max 16384 2024-04-25T16:12:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.666+0000 7faa3998e740 1 Processor -- start 2024-04-25T16:12:30.667 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.666+0000 7faa3998e740 1 -- start start 2024-04-25T16:12:30.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.667+0000 7faa3998e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 0x55eed30c4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.667+0000 7faa3998e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cfa6b0 0x55eed30c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.668 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.667+0000 7faa3998e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2d1b390 0x55eed30cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.667+0000 7faa3998e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faa1800e550 con 0x55eed2cfa6b0 2024-04-25T16:12:30.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.667+0000 7faa3998e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa18010090 con 0x55eed2cf24d0 2024-04-25T16:12:30.669 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.667+0000 7faa3998e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faa1800ec20 con 0x55eed2d1b390 2024-04-25T16:12:30.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.673+0000 7faa3087e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cfa6b0 0x55eed30c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.674 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.673+0000 7faa3087e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cfa6b0 0x55eed30c54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37450/0 (socket says 172.21.15.31:37450) 2024-04-25T16:12:30.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.673+0000 7faa3087e640 1 -- 172.21.15.31:0/3681848339 learned_addr learned my addr 172.21.15.31:0/3681848339 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:30.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.673+0000 7faa2bfff640 1 --2- 172.21.15.31:0/3681848339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 0x55eed30c4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.675 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.673+0000 7faa2bfff640 1 --2- 172.21.15.31:0/3681848339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2d1b390 0x55eed30cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.678+0000 7faa3087e640 1 -- 172.21.15.31:0/3681848339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2d1b390 msgr2=0x55eed30cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:30.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.678+0000 7faa3087e640 1 --2- 172.21.15.31:0/3681848339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2d1b390 0x55eed30cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.678+0000 7faa3087e640 1 -- 172.21.15.31:0/3681848339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 msgr2=0x55eed30c4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:30.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.678+0000 7faa3087e640 1 --2- 172.21.15.31:0/3681848339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 0x55eed30c4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.679 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.678+0000 7faa3087e640 1 -- 172.21.15.31:0/3681848339 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eed2cf30f0 con 0x55eed2cfa6b0 2024-04-25T16:12:30.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.678+0000 7faa3087e640 1 --2- 172.21.15.31:0/3681848339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cfa6b0 0x55eed30c54b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7faa1c020be0 tx=0x7faa1c00bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aec48a9317c1ac4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:30.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.679+0000 7faa127fc640 1 -- 172.21.15.31:0/3681848339 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c020dd0 con 0x55eed2cfa6b0 2024-04-25T16:12:30.680 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.679+0000 7faa127fc640 1 -- 172.21.15.31:0/3681848339 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa1c019070 con 0x55eed2cfa6b0 2024-04-25T16:12:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.679+0000 7faa127fc640 1 -- 172.21.15.31:0/3681848339 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c029440 con 0x55eed2cfa6b0 2024-04-25T16:12:30.681 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.680+0000 7faa3998e740 1 -- 172.21.15.31:0/3681848339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cfa6b0 msgr2=0x55eed30c54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:30.682 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.680+0000 7faa3998e740 1 --2- 172.21.15.31:0/3681848339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cfa6b0 0x55eed30c54b0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7faa1c020be0 tx=0x7faa1c00bf30 comp rx=0 tx=0).stop 2024-04-25T16:12:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.683+0000 7faa3998e740 1 -- 172.21.15.31:0/3681848339 shutdown_connections 2024-04-25T16:12:30.684 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.683+0000 7faa3998e740 1 --2- 172.21.15.31:0/3681848339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2d1b390 0x55eed30cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.683+0000 7faa3998e740 1 --2- 172.21.15.31:0/3681848339 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cfa6b0 0x55eed30c54b0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.683+0000 7faa3998e740 1 --2- 172.21.15.31:0/3681848339 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2cf24d0 0x55eed30c4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.685 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.683+0000 7faa3998e740 1 -- 172.21.15.31:0/3681848339 >> 172.21.15.31:0/3681848339 conn(0x55eed2d0f4e0 msgr2=0x55eed30c16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:30.689 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.687+0000 7faa3998e740 1 -- 172.21.15.31:0/3681848339 shutdown_connections 2024-04-25T16:12:30.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.689+0000 7faa3998e740 1 -- 172.21.15.31:0/3681848339 wait complete. 2024-04-25T16:12:30.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.689+0000 7faa3998e740 1 -- start start 2024-04-25T16:12:30.691 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.689+0000 7faa3998e740 1 Processor -- start 2024-04-25T16:12:30.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa3998e740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cf24d0 0x55eed30ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa3998e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30cab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa3998e740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2d1b390 0x55eed30cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa3998e740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7faa1800f9c0 con 0x55eed2cf24d0 2024-04-25T16:12:30.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa3998e740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7faa1800f2f0 con 0x55eed2d1b390 2024-04-25T16:12:30.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa3998e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7faa1800c530 con 0x55eed2cfa6b0 2024-04-25T16:12:30.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa3998e740 10 client.0 _ll_register_callbacks cb 0x7faa1c00b0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:30.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cf24d0 0x55eed30ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa3087e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa2bfff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cf24d0 0x55eed30ca5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:37462/0 (socket says 172.21.15.31:37462) 2024-04-25T16:12:30.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa3087e640 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30cab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:57002/0 (socket says 172.21.15.31:57002) 2024-04-25T16:12:30.693 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.691+0000 7faa2bfff640 1 -- 172.21.15.31:0/2521372724 learned_addr learned my addr 172.21.15.31:0/2521372724 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.692+0000 7faa2bfff640 1 -- 172.21.15.31:0/2521372724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 msgr2=0x55eed30cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.692+0000 7faa2bfff640 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.692+0000 7faa2bfff640 1 -- 172.21.15.31:0/2521372724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2d1b390 msgr2=0x55eed30cda50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.692+0000 7faa2bfff640 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2d1b390 0x55eed30cda50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.692+0000 7faa2bfff640 1 -- 172.21.15.31:0/2521372724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eed2fe24b0 con 0x55eed2cf24d0 2024-04-25T16:12:30.696 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.693+0000 7faa2bfff640 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cf24d0 0x55eed30ca5d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7faa2000f7d0 tx=0x7faa20009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=551507905af47f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:30.700 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.699+0000 7faa0bfff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:12:30.701 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.699+0000 7faa0bfff640 1 -- 172.21.15.31:0/2521372724 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa200050b0 con 0x55eed2cf24d0 2024-04-25T16:12:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.699+0000 7faa0bfff640 1 -- 172.21.15.31:0/2521372724 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7faa20005250 con 0x55eed2cf24d0 2024-04-25T16:12:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.699+0000 7faa3998e740 1 -- 172.21.15.31:0/2521372724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa1c019070 con 0x55eed2cf24d0 2024-04-25T16:12:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.700+0000 7faa0bfff640 1 -- 172.21.15.31:0/2521372724 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7faa20005490 con 0x55eed2cf24d0 2024-04-25T16:12:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.700+0000 7faa3998e740 10 client.17853 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:30.702 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.700+0000 7faa3998e740 1 -- 172.21.15.31:0/2521372724 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55eed313b1a0 con 0x55eed2cf24d0 2024-04-25T16:12:30.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.701+0000 7faa0bfff640 1 -- 172.21.15.31:0/2521372724 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa1c019070 con 0x55eed2cf24d0 2024-04-25T16:12:30.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.701+0000 7faa0bfff640 1 -- 172.21.15.31:0/2521372724 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7faa20019020 con 0x55eed2cf24d0 2024-04-25T16:12:30.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.701+0000 7faa3998e740 10 client.17853 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:30.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.701+0000 7faa3998e740 10 client.17853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:30.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.701+0000 7faa3998e740 20 client.17853 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:30.703 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.702+0000 7faa3998e740 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55eed2caf3a0 0x55eed313db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.702+0000 7faa3998e740 4 client.17853 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:12:30.704 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.702+0000 7faa3998e740 1 -- 172.21.15.31:0/2521372724 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eed3029800 con 0x55eed2caf3a0 2024-04-25T16:12:30.705 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.704+0000 7faa3087e640 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55eed2caf3a0 0x55eed313db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:30.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.706+0000 7faa3087e640 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55eed2caf3a0 0x55eed313db70 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d05aa9c69f190d16 server_cookie=9d1971f76eaf04bb in_seq=0 out_seq=0 2024-04-25T16:12:30.710 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.706+0000 7faa0bfff640 10 client.17853 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:30.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.709+0000 7faa0bfff640 1 -- 172.21.15.31:0/2521372724 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x55eed3029800 con 0x55eed2caf3a0 2024-04-25T16:12:30.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.709+0000 7faa0bfff640 10 client.17853 handle_command_reply: tid=0 2024-04-25T16:12:30.711 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.709+0000 7faa3998e740 1 client.17853 shutdown 2024-04-25T16:12:30.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.710+0000 7faa3998e740 1 -- 172.21.15.31:0/2521372724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cf24d0 msgr2=0x55eed30ca5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:30.712 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.710+0000 7faa3998e740 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cf24d0 0x55eed30ca5d0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7faa2000f7d0 tx=0x7faa20009840 comp rx=0 tx=0).stop 2024-04-25T16:12:30.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.711+0000 7faa3998e740 1 -- 172.21.15.31:0/2521372724 shutdown_connections 2024-04-25T16:12:30.713 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.711+0000 7faa3998e740 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x55eed2caf3a0 0x55eed313db70 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.711+0000 7faa0bfff640 0 client.17853 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:30.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.711+0000 7faa3998e740 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55eed2d1b390 0x55eed30cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.711+0000 7faa3998e740 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eed2cfa6b0 0x55eed30cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.711+0000 7faa3998e740 1 --2- 172.21.15.31:0/2521372724 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55eed2cf24d0 0x55eed30ca5d0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.711+0000 7faa3998e740 1 -- 172.21.15.31:0/2521372724 >> 172.21.15.31:0/2521372724 conn(0x55eed2d0f4e0 msgr2=0x55eed30c5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:30.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.712+0000 7faa3998e740 1 -- 172.21.15.31:0/2521372724 shutdown_connections 2024-04-25T16:12:30.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.712+0000 7faa3998e740 1 -- 172.21.15.31:0/2521372724 wait complete. 2024-04-25T16:12:30.714 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.713+0000 7faa3998e740 20 client.17853 trim_cache size 0 max 16384 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "scrub active (3 inodes in the stack)", 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": { 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout: "710ae31f-7e5e-4124-8ee7-49badec9c8f8": { 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout: "path": "/", 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout: "tag": "710ae31f-7e5e-4124-8ee7-49badec9c8f8", 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout: "options": "recursive,force" 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:12:30.715 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:12:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 -- 172.21.15.31:0/2453456823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa140afc00 msgr2=0x7faa140b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa140afc00 0x7faa140b2110 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7faa240c8500 tx=0x7faa240c4040 comp rx=0 tx=0).stop 2024-04-25T16:12:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 -- 172.21.15.31:0/2453456823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c1b6c40 msgr2=0x7faa2c1b9060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c1b6c40 0x7faa2c1b9060 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7faa24009dc0 tx=0x7faa24030800 comp rx=0 tx=0).stop 2024-04-25T16:12:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 -- 172.21.15.31:0/2453456823 shutdown_connections 2024-04-25T16:12:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7faa140afc00 0x7faa140b2110 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.716 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c1b6c40 0x7faa2c1b9060 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7faa2c10b310 0x7faa2c117130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 --2- 172.21.15.31:0/2453456823 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7faa2c10a420 0x7faa2c116bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 -- 172.21.15.31:0/2453456823 >> 172.21.15.31:0/2453456823 conn(0x7faa2c0704e0 msgr2=0x7faa2c071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 -- 172.21.15.31:0/2453456823 shutdown_connections 2024-04-25T16:12:30.717 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:12:30.714+0000 7faa32ae2640 1 -- 172.21.15.31:0/2453456823 wait complete. 2024-04-25T16:12:30.747 INFO:tasks.cephfs.filesystem:scrub status for tag:710ae31f-7e5e-4124-8ee7-49badec9c8f8 - {'path': '/', 'tag': '710ae31f-7e5e-4124-8ee7-49badec9c8f8', 'options': 'recursive,force'} 2024-04-25T16:12:31.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:31 smithi063 ceph-mon[32413]: scrub queued for path: / 2024-04-25T16:12:31.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:31 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:12:31.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:31 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:12:31.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:31 smithi063 ceph-mon[32413]: pgmap v687: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 683 MiB/s wr, 1.65k op/s 2024-04-25T16:12:31.624 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:31 smithi063 ceph-mon[32413]: scrub summary: active paths [/] 2024-04-25T16:12:31.624 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:31 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:12:31.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:31 smithi186 ceph-mon[33258]: scrub queued for path: / 2024-04-25T16:12:31.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:31 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:12:31.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:31 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:12:31.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:31 smithi186 ceph-mon[33258]: pgmap v687: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 683 MiB/s wr, 1.65k op/s 2024-04-25T16:12:31.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:31 smithi186 ceph-mon[33258]: scrub summary: active paths [/] 2024-04-25T16:12:31.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:31 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:12:31.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:31 smithi031 ceph-mon[29140]: scrub queued for path: / 2024-04-25T16:12:31.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:31 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:12:31.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:31 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:12:31.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:31 smithi031 ceph-mon[29140]: pgmap v687: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 683 MiB/s wr, 1.65k op/s 2024-04-25T16:12:31.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:31 smithi031 ceph-mon[29140]: scrub summary: active paths [/] 2024-04-25T16:12:31.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:31 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:12:33.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:33 smithi063 ceph-mon[32413]: pgmap v688: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 778 MiB/s wr, 1.75k op/s 2024-04-25T16:12:33.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:33 smithi186 ceph-mon[33258]: pgmap v688: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 778 MiB/s wr, 1.75k op/s 2024-04-25T16:12:33.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:33 smithi031 ceph-mon[29140]: pgmap v688: 97 pgs: 97 active+clean; 10 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 778 MiB/s wr, 1.75k op/s 2024-04-25T16:12:34.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:34 smithi063 ceph-mon[32413]: scrub summary: idle+waiting paths [/] 2024-04-25T16:12:34.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:34 smithi186 ceph-mon[33258]: scrub summary: idle+waiting paths [/] 2024-04-25T16:12:34.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:34 smithi031 ceph-mon[29140]: scrub summary: idle+waiting paths [/] 2024-04-25T16:12:35.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:35 smithi063 ceph-mon[32413]: pgmap v689: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 810 MiB/s wr, 1.73k op/s 2024-04-25T16:12:35.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:35 smithi186 ceph-mon[33258]: pgmap v689: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 810 MiB/s wr, 1.73k op/s 2024-04-25T16:12:35.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:35 smithi031 ceph-mon[29140]: pgmap v689: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 810 MiB/s wr, 1.73k op/s 2024-04-25T16:12:36.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:36 smithi186 ceph-mon[33258]: pgmap v690: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 718 MiB/s wr, 1.60k op/s 2024-04-25T16:12:36.720 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:36 smithi031 ceph-mon[29140]: pgmap v690: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 718 MiB/s wr, 1.60k op/s 2024-04-25T16:12:36.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:36 smithi063 ceph-mon[32413]: pgmap v690: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 718 MiB/s wr, 1.60k op/s 2024-04-25T16:12:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.360+0000 7f86df030640 1 -- 172.21.15.31:0/1017712918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86d810a350 msgr2=0x7f86d810ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.360+0000 7f86df030640 1 --2- 172.21.15.31:0/1017712918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86d810a350 0x7f86d810ac90 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009900 tx=0x7f86c802a410 comp rx=0 tx=0).stop 2024-04-25T16:12:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.360+0000 7f86df030640 1 -- 172.21.15.31:0/1017712918 shutdown_connections 2024-04-25T16:12:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.360+0000 7f86df030640 1 --2- 172.21.15.31:0/1017712918 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d810fef0 0x7f86d8110300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.360+0000 7f86df030640 1 --2- 172.21.15.31:0/1017712918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d810b1d0 0x7f86d810b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.360+0000 7f86df030640 1 --2- 172.21.15.31:0/1017712918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86d810a350 0x7f86d810ac90 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.360+0000 7f86df030640 1 -- 172.21.15.31:0/1017712918 >> 172.21.15.31:0/1017712918 conn(0x7f86d8070390 msgr2=0x7f86d8072820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.360+0000 7f86df030640 1 -- 172.21.15.31:0/1017712918 shutdown_connections 2024-04-25T16:12:39.363 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.360+0000 7f86df030640 1 -- 172.21.15.31:0/1017712918 wait complete. 2024-04-25T16:12:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.361+0000 7f86df030640 1 Processor -- start 2024-04-25T16:12:39.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.362+0000 7f86df030640 1 -- start start 2024-04-25T16:12:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.362+0000 7f86df030640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86d810a350 0x7f86d81a8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.362+0000 7f86df030640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d810b1d0 0x7f86d81a93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.362+0000 7f86df030640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d810fef0 0x7f86d81b0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.362+0000 7f86df030640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86d811e300 con 0x7f86d810b1d0 2024-04-25T16:12:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.362+0000 7f86df030640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f86d811e180 con 0x7f86d810fef0 2024-04-25T16:12:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.362+0000 7f86df030640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86d811e480 con 0x7f86d810a350 2024-04-25T16:12:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.363+0000 7f86dcdcc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86d810a350 0x7f86d81a8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.363+0000 7f86dcdcc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86d810a350 0x7f86d81a8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54860/0 (socket says 172.21.15.31:54860) 2024-04-25T16:12:39.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.363+0000 7f86dcdcc640 1 -- 172.21.15.31:0/4285902537 learned_addr learned my addr 172.21.15.31:0/4285902537 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.363+0000 7f86d7fff640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d810b1d0 0x7f86d81a93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.363+0000 7f86d7fff640 1 -- 172.21.15.31:0/4285902537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86d810a350 msgr2=0x7f86d81a8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.363+0000 7f86d7fff640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86d810a350 0x7f86d81a8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.363+0000 7f86d7fff640 1 -- 172.21.15.31:0/4285902537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d810fef0 msgr2=0x7f86d81b0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.364+0000 7f86d7fff640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d810fef0 0x7f86d81b0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.364+0000 7f86d7fff640 1 -- 172.21.15.31:0/4285902537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d8111c90 con 0x7f86d810b1d0 2024-04-25T16:12:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.364+0000 7f86d7fff640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d810b1d0 0x7f86d81a93b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f86d000e9f0 tx=0x7f86d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:39.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.365+0000 7f86d5ffb640 1 -- 172.21.15.31:0/4285902537 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d000cd80 con 0x7f86d810b1d0 2024-04-25T16:12:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.365+0000 7f86df030640 1 -- 172.21.15.31:0/4285902537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d8111f80 con 0x7f86d810b1d0 2024-04-25T16:12:39.373 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.366+0000 7f86df030640 1 -- 172.21.15.31:0/4285902537 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d81124d0 con 0x7f86d810b1d0 2024-04-25T16:12:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.371+0000 7f86e5fb4740 1 Processor -- start 2024-04-25T16:12:39.374 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.371+0000 7f86d5ffb640 1 -- 172.21.15.31:0/4285902537 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d0011920 con 0x7f86d810b1d0 2024-04-25T16:12:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.371+0000 7f86d5ffb640 1 -- 172.21.15.31:0/4285902537 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0010970 con 0x7f86d810b1d0 2024-04-25T16:12:39.375 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.372+0000 7f86e5fb4740 1 -- start start 2024-04-25T16:12:39.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.374+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.376 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.374+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e0a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.374+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 0x563e0e0a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.375+0000 7f86dd5cd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.377 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.375+0000 7f86dd5cd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:54870/0 (socket says 172.21.15.31:54870) 2024-04-25T16:12:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.375+0000 7f86dd5cd640 1 -- 172.21.15.31:0/1953001705 learned_addr learned my addr 172.21.15.31:0/1953001705 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.375+0000 7f86e5fb4740 1 -- 172.21.15.31:0/1953001705 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563e0e006460 con 0x563e0dcd72f0 2024-04-25T16:12:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.375+0000 7f86e5fb4740 1 -- 172.21.15.31:0/1953001705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563e0dfc0590 con 0x563e0dcdf4d0 2024-04-25T16:12:39.378 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.375+0000 7f86e5fb4740 1 -- 172.21.15.31:0/1953001705 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563e0e002420 con 0x563e0dd001b0 2024-04-25T16:12:39.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.378+0000 7f86d5ffb640 1 -- 172.21.15.31:0/4285902537 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f86d0010bf0 con 0x7f86d810b1d0 2024-04-25T16:12:39.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.379+0000 7f86d7fff640 1 --2- 172.21.15.31:0/1953001705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 0x563e0e0a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.380+0000 7f86d5ffb640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86c00afcf0 0x7f86c00b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.382 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.380+0000 7f86d7fff640 1 -- 172.21.15.31:0/1953001705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 msgr2=0x563e0e0a0b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:39.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.380+0000 7f86d7fff640 1 --2- 172.21.15.31:0/1953001705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0a0b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.380+0000 7f86d7fff640 1 -- 172.21.15.31:0/1953001705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 msgr2=0x563e0e0a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:39.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.380+0000 7f86d7fff640 1 --2- 172.21.15.31:0/1953001705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e0a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.380+0000 7f86d7fff640 1 -- 172.21.15.31:0/1953001705 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e0dfcd0d0 con 0x563e0dcdf4d0 2024-04-25T16:12:39.383 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.381+0000 7f86dd5cd640 1 --2- 172.21.15.31:0/1953001705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:12:39.384 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.382+0000 7f86dd5cd640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86c00afcf0 0x7f86c00b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.385 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.383+0000 7f86d7fff640 1 --2- 172.21.15.31:0/1953001705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 0x563e0e0a7fa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f86d00dca80 tx=0x7f86d0021000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=898dab05c17998ba server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.384+0000 7f86d5ffb640 1 -- 172.21.15.31:0/4285902537 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f86d00df030 con 0x7f86d810b1d0 2024-04-25T16:12:39.386 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.384+0000 7f86dd5cd640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86c00afcf0 0x7f86c00b2200 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x563e0e0a1cf0 tx=0x7f86cc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.384+0000 7f86beffd640 1 -- 172.21.15.31:0/1953001705 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d00a7290 con 0x563e0dcdf4d0 2024-04-25T16:12:39.387 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.385+0000 7f86beffd640 1 -- 172.21.15.31:0/1953001705 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d00a6fd0 con 0x563e0dcdf4d0 2024-04-25T16:12:39.388 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.386+0000 7f86beffd640 1 -- 172.21.15.31:0/1953001705 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d00151b0 con 0x563e0dcdf4d0 2024-04-25T16:12:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.386+0000 7f86e5fb4740 1 -- 172.21.15.31:0/1953001705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 msgr2=0x563e0e0a7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:39.389 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.386+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/1953001705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 0x563e0e0a7fa0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f86d00dca80 tx=0x7f86d0021000 comp rx=0 tx=0).stop 2024-04-25T16:12:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.388+0000 7f86e5fb4740 1 -- 172.21.15.31:0/1953001705 shutdown_connections 2024-04-25T16:12:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.388+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/1953001705 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 0x563e0e0a7fa0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.388+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/1953001705 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e0a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.390 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.388+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/1953001705 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0a0b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.391 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.388+0000 7f86e5fb4740 1 -- 172.21.15.31:0/1953001705 >> 172.21.15.31:0/1953001705 conn(0x563e0dcf4300 msgr2=0x563e0e09c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:39.393 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.390+0000 7f86e5fb4740 1 -- 172.21.15.31:0/1953001705 shutdown_connections 2024-04-25T16:12:39.395 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.393+0000 7f86e5fb4740 1 -- 172.21.15.31:0/1953001705 wait complete. 2024-04-25T16:12:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.394+0000 7f86e5fb4740 1 -- start start 2024-04-25T16:12:39.396 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.394+0000 7f86e5fb4740 1 Processor -- start 2024-04-25T16:12:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.395+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcd72f0 0x563e0e0aa7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.397 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.395+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcdf4d0 0x563e0e0aad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.395+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.396+0000 7f86e5fb4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x563e0e0a8990 con 0x563e0dcdf4d0 2024-04-25T16:12:39.398 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.396+0000 7f86e5fb4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x563e0df8db60 con 0x563e0dcd72f0 2024-04-25T16:12:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.396+0000 7f86e5fb4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563e0e0a8b10 con 0x563e0dd001b0 2024-04-25T16:12:39.399 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.396+0000 7f86e5fb4740 10 client.0 _ll_register_callbacks cb 0x563e0dfb22b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.400+0000 7f86d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcdf4d0 0x563e0e0aad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.400+0000 7f86d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcdf4d0 0x563e0e0aad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44396/0 (socket says 172.21.15.31:44396) 2024-04-25T16:12:39.402 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.400+0000 7f86d7fff640 1 -- 172.21.15.31:0/4242353368 learned_addr learned my addr 172.21.15.31:0/4242353368 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.401+0000 7f86dd5cd640 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.402+0000 7f86dd5cd640 1 -- 172.21.15.31:0/4242353368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcd72f0 msgr2=0x563e0e0aa7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-25T16:12:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.402+0000 7f86dd5cd640 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcd72f0 0x563e0e0aa7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.402+0000 7f86dd5cd640 1 -- 172.21.15.31:0/4242353368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcdf4d0 msgr2=0x563e0e0aad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.402+0000 7f86dd5cd640 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcdf4d0 0x563e0e0aad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.402+0000 7f86dd5cd640 1 -- 172.21.15.31:0/4242353368 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e0dd18210 con 0x563e0dd001b0 2024-04-25T16:12:39.406 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.402+0000 7f86dd5cd640 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0b1c60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00f9e0 tx=0x7f86cc009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8507543c343b6157 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.410+0000 7f86b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.186:3300/0 2024-04-25T16:12:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.411+0000 7f86b7fff640 1 -- 172.21.15.31:0/4242353368 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86cc01f7c0 con 0x563e0dd001b0 2024-04-25T16:12:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.411+0000 7f86b7fff640 1 -- 172.21.15.31:0/4242353368 <== mon.1 v2:172.21.15.186:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f86cc01f960 con 0x563e0dd001b0 2024-04-25T16:12:39.418 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.416+0000 7f86e5fb4740 1 -- 172.21.15.31:0/4242353368 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86d00a6fd0 con 0x563e0dd001b0 2024-04-25T16:12:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.417+0000 7f86b7fff640 1 -- 172.21.15.31:0/4242353368 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86cc01fb60 con 0x563e0dd001b0 2024-04-25T16:12:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.422+0000 7f86e5fb4740 10 client.27559 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:39.424 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.422+0000 7f86e5fb4740 1 -- 172.21.15.31:0/4242353368 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e0dde4620 con 0x563e0dd001b0 2024-04-25T16:12:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.434+0000 7f86b7fff640 1 -- 172.21.15.31:0/4242353368 <== mon.1 v2:172.21.15.186:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d00a6fd0 con 0x563e0dd001b0 2024-04-25T16:12:39.438 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.435+0000 7f86b7fff640 1 -- 172.21.15.31:0/4242353368 <== mon.1 v2:172.21.15.186:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f86cc013380 con 0x563e0dd001b0 2024-04-25T16:12:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.436+0000 7f86e5fb4740 10 client.27559 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.436+0000 7f86e5fb4740 10 client.27559 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.436+0000 7f86e5fb4740 20 client.27559 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.436+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563e0dcbed80 0x563e0e0b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.436+0000 7f86e5fb4740 4 client.27559 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:12:39.439 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.436+0000 7f86e5fb4740 1 -- 172.21.15.31:0/4242353368 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e0e014070 con 0x563e0dcbed80 2024-04-25T16:12:39.440 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.438+0000 7f86dcdcc640 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563e0dcbed80 0x563e0e0b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.448+0000 7f86b7fff640 10 client.27559 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:39.452 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.448+0000 7f86dcdcc640 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563e0dcbed80 0x563e0e0b85c0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b5988a218cebf4f server_cookie=cdfc696934ea9721 in_seq=0 out_seq=0 2024-04-25T16:12:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.453+0000 7f86b7fff640 1 -- 172.21.15.31:0/4242353368 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563e0e014070 con 0x563e0dcbed80 2024-04-25T16:12:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.453+0000 7f86b7fff640 10 client.27559 handle_command_reply: tid=0 2024-04-25T16:12:39.456 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.454+0000 7f86e5fb4740 1 client.27559 shutdown 2024-04-25T16:12:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.456+0000 7f86e5fb4740 1 -- 172.21.15.31:0/4242353368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 msgr2=0x563e0e0b1c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:39.458 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.456+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0b1c60 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f86cc00f9e0 tx=0x7f86cc009190 comp rx=0 tx=0).stop 2024-04-25T16:12:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.461+0000 7f86e5fb4740 1 -- 172.21.15.31:0/4242353368 shutdown_connections 2024-04-25T16:12:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.461+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563e0dcbed80 0x563e0e0b85c0 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.461+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dd001b0 0x563e0e0b1c60 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.461+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcdf4d0 0x563e0e0aad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.461+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/4242353368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcd72f0 0x563e0e0aa7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.461+0000 7f86b7fff640 0 client.27559 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:39.464 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.461+0000 7f86e5fb4740 1 -- 172.21.15.31:0/4242353368 >> 172.21.15.31:0/4242353368 conn(0x563e0dcf4300 msgr2=0x563e0e09f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:39.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.462+0000 7f86e5fb4740 1 -- 172.21.15.31:0/4242353368 shutdown_connections 2024-04-25T16:12:39.465 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.462+0000 7f86e5fb4740 1 -- 172.21.15.31:0/4242353368 wait complete. 2024-04-25T16:12:39.475 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.473+0000 7f86e5fb4740 20 client.27559 trim_cache size 0 max 16384 2024-04-25T16:12:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.524+0000 7f86e5fb4740 1 Processor -- start 2024-04-25T16:12:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.524+0000 7f86e5fb4740 1 -- start start 2024-04-25T16:12:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.524+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dcbed80 0x563e0e09f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.524+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e09ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.524+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 0x563e0e0b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.530 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.524+0000 7f86e5fb4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86c400e4c0 con 0x563e0dcd72f0 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.524+0000 7f86e5fb4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f86c4010000 con 0x563e0dcdf4d0 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.524+0000 7f86e5fb4740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86c400eb90 con 0x563e0dcbed80 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.525+0000 7f86d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e09ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.525+0000 7f86d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e09ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:44408/0 (socket says 172.21.15.31:44408) 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.525+0000 7f86d7fff640 1 -- 172.21.15.31:0/2488384938 learned_addr learned my addr 172.21.15.31:0/2488384938 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.525+0000 7f86dd5cd640 1 --2- 172.21.15.31:0/2488384938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 0x563e0e0b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.525+0000 7f86d7fff640 1 -- 172.21.15.31:0/2488384938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dcbed80 msgr2=0x563e0e09f9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.525+0000 7f86d7fff640 1 --2- 172.21.15.31:0/2488384938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dcbed80 0x563e0e09f9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.525+0000 7f86d7fff640 1 -- 172.21.15.31:0/2488384938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 msgr2=0x563e0e0b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.525+0000 7f86d7fff640 1 --2- 172.21.15.31:0/2488384938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 0x563e0e0b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.525+0000 7f86d7fff640 1 -- 172.21.15.31:0/2488384938 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e0e0a5630 con 0x563e0dcd72f0 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.526+0000 7f86d7fff640 1 --2- 172.21.15.31:0/2488384938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e09ff00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f86d00a4730 tx=0x7f86d0009280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a7a431157b4486 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.526+0000 7f86be7fc640 1 -- 172.21.15.31:0/2488384938 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d00a6c30 con 0x563e0dcd72f0 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.526+0000 7f86be7fc640 1 -- 172.21.15.31:0/2488384938 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d00a6fd0 con 0x563e0dcd72f0 2024-04-25T16:12:39.531 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.527+0000 7f86be7fc640 1 -- 172.21.15.31:0/2488384938 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d00a6e10 con 0x563e0dcd72f0 2024-04-25T16:12:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.528+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2488384938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 msgr2=0x563e0e09ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.528+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2488384938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e09ff00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f86d00a4730 tx=0x7f86d0009280 comp rx=0 tx=0).stop 2024-04-25T16:12:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.528+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2488384938 shutdown_connections 2024-04-25T16:12:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.528+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2488384938 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcdf4d0 0x563e0e0b1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.532 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.528+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2488384938 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e09ff00 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.528+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2488384938 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dcbed80 0x563e0e09f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.528+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2488384938 >> 172.21.15.31:0/2488384938 conn(0x563e0dcf4300 msgr2=0x563e0e09f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.529+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2488384938 shutdown_connections 2024-04-25T16:12:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.529+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2488384938 wait complete. 2024-04-25T16:12:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.530+0000 7f86e5fb4740 1 -- start start 2024-04-25T16:12:39.533 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.531+0000 7f86e5fb4740 1 Processor -- start 2024-04-25T16:12:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.532+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcbed80 0x563e0e0aee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.535 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.533+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e0af380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.533+0000 7f86e5fb4740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dcdf4d0 0x563e0e0b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.533+0000 7f86e5fb4740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f86c400f930 con 0x563e0dcd72f0 2024-04-25T16:12:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.533+0000 7f86e5fb4740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f86c400f260 con 0x563e0dcbed80 2024-04-25T16:12:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.533+0000 7f86dcdcc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcbed80 0x563e0e0aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.533+0000 7f86dcdcc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcbed80 0x563e0e0aee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:57590/0 (socket says 172.21.15.31:57590) 2024-04-25T16:12:39.536 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.533+0000 7f86dcdcc640 1 -- 172.21.15.31:0/2768376689 learned_addr learned my addr 172.21.15.31:0/2768376689 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:12:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.534+0000 7f86dd5cd640 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dcdf4d0 0x563e0e0b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.534+0000 7f86d7fff640 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e0af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.534+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2768376689 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86c400c4a0 con 0x563e0dcdf4d0 2024-04-25T16:12:39.537 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.535+0000 7f86e5fb4740 10 client.0 _ll_register_callbacks cb 0x7f86cc006bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:12:39.538 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.536+0000 7f86d7fff640 1 -- 172.21.15.31:0/2768376689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dcdf4d0 msgr2=0x563e0e0b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.536+0000 7f86d7fff640 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dcdf4d0 0x563e0e0b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.536+0000 7f86d7fff640 1 -- 172.21.15.31:0/2768376689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcbed80 msgr2=0x563e0e0aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:12:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.536+0000 7f86d7fff640 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcbed80 0x563e0e0aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:39.539 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.536+0000 7f86d7fff640 1 -- 172.21.15.31:0/2768376689 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e0e0b20c0 con 0x563e0dcd72f0 2024-04-25T16:12:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.538+0000 7f86d7fff640 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e0af380 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f86d00104e0 tx=0x7f86d00225d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ee743b346b31e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:12:39.541 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.538+0000 7f86b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:12:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.539+0000 7f86b7fff640 1 -- 172.21.15.31:0/2768376689 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d00a6e10 con 0x563e0dcd72f0 2024-04-25T16:12:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.539+0000 7f86b7fff640 1 -- 172.21.15.31:0/2768376689 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f86d00afe90 con 0x563e0dcd72f0 2024-04-25T16:12:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.540+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2768376689 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86d00a6fd0 con 0x563e0dcd72f0 2024-04-25T16:12:39.542 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.540+0000 7f86b7fff640 1 -- 172.21.15.31:0/2768376689 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86d00a6c30 con 0x563e0dcd72f0 2024-04-25T16:12:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.543+0000 7f86e5fb4740 10 client.17865 fetch_fsmap learned FSMap version 29 2024-04-25T16:12:39.546 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.543+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2768376689 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563e0e0b18f0 con 0x563e0dcd72f0 2024-04-25T16:12:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.546+0000 7f86b7fff640 1 -- 172.21.15.31:0/2768376689 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86d00151b0 con 0x563e0dcd72f0 2024-04-25T16:12:39.549 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.546+0000 7f86b7fff640 1 -- 172.21.15.31:0/2768376689 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 29) ==== 8354+0+0 (secure 0 0 0) 0x7f86d000f040 con 0x563e0dcd72f0 2024-04-25T16:12:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.548+0000 7f86e5fb4740 10 client.17865 fetch_fsmap finished waiting for FSMap version 29 2024-04-25T16:12:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.548+0000 7f86e5fb4740 10 client.17865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-25T16:12:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.548+0000 7f86e5fb4740 20 client.17865 populate_metadata read hostname 'smithi031' 2024-04-25T16:12:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.548+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563e0dd001b0 0x563e0e122980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:12:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.548+0000 7f86e5fb4740 4 client.17865 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-25T16:12:39.550 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.548+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2768376689 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f86d00a6fd0 con 0x563e0dd001b0 2024-04-25T16:12:39.555 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.552+0000 7f86dcdcc640 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563e0dd001b0 0x563e0e122980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:12:39.557 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.555+0000 7f86dcdcc640 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563e0dd001b0 0x563e0e122980 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4da2ef1cd42611 server_cookie=3c4abf7b2d984d58 in_seq=0 out_seq=0 2024-04-25T16:12:39.558 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:39.556+0000 7f86b7fff640 10 client.17865 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:39.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:39 smithi063 ceph-mon[32413]: pgmap v691: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 864 MiB/s wr, 1.96k op/s 2024-04-25T16:12:39.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:39 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:12:39.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:39 smithi063 ceph-mon[32413]: scrub summary: idle 2024-04-25T16:12:39.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:39 smithi186 ceph-mon[33258]: pgmap v691: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 864 MiB/s wr, 1.96k op/s 2024-04-25T16:12:39.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:39 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:12:39.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:39 smithi186 ceph-mon[33258]: scrub summary: idle 2024-04-25T16:12:39.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:39 smithi031 ceph-mon[29140]: pgmap v691: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 864 MiB/s wr, 1.96k op/s 2024-04-25T16:12:39.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:39 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:12:39.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:39 smithi031 ceph-mon[29140]: scrub summary: idle 2024-04-25T16:12:41.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:41 smithi063 ceph-mon[32413]: pgmap v692: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 722 MiB/s wr, 1.48k op/s 2024-04-25T16:12:41.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:41 smithi186 ceph-mon[33258]: pgmap v692: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 722 MiB/s wr, 1.48k op/s 2024-04-25T16:12:41.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:41 smithi031 ceph-mon[29140]: pgmap v692: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 722 MiB/s wr, 1.48k op/s 2024-04-25T16:12:42.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:42 smithi031 ceph-mon[29140]: pgmap v693: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 873 MiB/s wr, 1.70k op/s 2024-04-25T16:12:42.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:42 smithi063 ceph-mon[32413]: pgmap v693: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 873 MiB/s wr, 1.70k op/s 2024-04-25T16:12:42.898 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:42 smithi186 ceph-mon[33258]: pgmap v693: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 873 MiB/s wr, 1.70k op/s 2024-04-25T16:12:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.361+0000 7f86b7fff640 1 -- 172.21.15.31:0/2768376689 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f86d00a6fd0 con 0x563e0dd001b0 2024-04-25T16:12:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.361+0000 7f86b7fff640 10 client.17865 handle_command_reply: tid=0 2024-04-25T16:12:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.361+0000 7f86e5fb4740 1 client.17865 shutdown 2024-04-25T16:12:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.362+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2768376689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 msgr2=0x563e0e0af380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:44.364 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.362+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e0af380 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f86d00104e0 tx=0x7f86d00225d0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.362+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2768376689 shutdown_connections 2024-04-25T16:12:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.362+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x563e0dd001b0 0x563e0e122980 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.363+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563e0dcdf4d0 0x563e0e0b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.363+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x563e0dcd72f0 0x563e0e0af380 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.363+0000 7f86e5fb4740 1 --2- 172.21.15.31:0/2768376689 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x563e0dcbed80 0x563e0e0aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.363+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2768376689 >> 172.21.15.31:0/2768376689 conn(0x563e0dcf4300 msgr2=0x563e0e09fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:44.365 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.363+0000 7f86b7fff640 0 client.17865 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:12:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.363+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2768376689 shutdown_connections 2024-04-25T16:12:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.363+0000 7f86e5fb4740 1 -- 172.21.15.31:0/2768376689 wait complete. 2024-04-25T16:12:44.366 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.364+0000 7f86e5fb4740 20 client.17865 trim_cache size 0 max 16384 2024-04-25T16:12:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:{ 2024-04-25T16:12:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "message": "", 2024-04-25T16:12:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout: "return_code": 0 2024-04-25T16:12:44.367 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stdout:} 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 -- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86c00afcf0 msgr2=0x7f86c00b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86c00afcf0 0x7f86c00b2200 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x563e0e0a1cf0 tx=0x7f86cc008040 comp rx=0 tx=0).stop 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 -- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d810b1d0 msgr2=0x7f86d81a93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d810b1d0 0x7f86d81a93b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f86d000e9f0 tx=0x7f86d000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 -- 172.21.15.31:0/4285902537 shutdown_connections 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f86c00afcf0 0x7f86c00b2200 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f86d810fef0 0x7f86d81b0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f86d810b1d0 0x7f86d81a93b0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 --2- 172.21.15.31:0/4285902537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86d810a350 0x7f86d81a8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:12:44.368 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.365+0000 7f86df030640 1 -- 172.21.15.31:0/4285902537 >> 172.21.15.31:0/4285902537 conn(0x7f86d8070390 msgr2=0x7f86d810bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:12:44.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.366+0000 7f86df030640 1 -- 172.21.15.31:0/4285902537 shutdown_connections 2024-04-25T16:12:44.369 INFO:teuthology.task.background_exec.ubuntu@smithi031.front.sepia.ceph.com.smithi031.stderr:2024-04-25T16:12:44.367+0000 7f86df030640 1 -- 172.21.15.31:0/4285902537 wait complete. 2024-04-25T16:12:44.801 INFO:tasks.workunit.client.0.smithi031.stdout:stopping iogen 2024-04-25T16:12:44.801 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo 'stopping iogen' 2024-04-25T16:12:44.801 INFO:tasks.workunit.client.0.smithi031.stderr:+ ./iogen -k 2024-04-25T16:12:44.953 INFO:tasks.workunit.client.0.smithi031.stdout:OK 2024-04-25T16:12:44.954 INFO:tasks.workunit.client.0.smithi031.stderr:+ echo OK 2024-04-25T16:12:45.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:45 smithi063 ceph-mon[32413]: pgmap v694: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 861 MiB/s wr, 1.74k op/s 2024-04-25T16:12:45.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:45 smithi186 ceph-mon[33258]: pgmap v694: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 861 MiB/s wr, 1.74k op/s 2024-04-25T16:12:45.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:45 smithi031 ceph-mon[29140]: pgmap v694: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 861 MiB/s wr, 1.74k op/s 2024-04-25T16:12:47.281 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T16:12:47.281 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T16:12:47.372 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:47 smithi031 ceph-mon[29140]: pgmap v695: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 759 MiB/s wr, 1.64k op/s 2024-04-25T16:12:47.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:47 smithi063 ceph-mon[32413]: pgmap v695: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 759 MiB/s wr, 1.64k op/s 2024-04-25T16:12:47.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:47 smithi186 ceph-mon[33258]: pgmap v695: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 759 MiB/s wr, 1.64k op/s 2024-04-25T16:12:47.831 INFO:tasks.workunit:Stopping ['suites/iogen.sh'] on client.0... 2024-04-25T16:12:47.831 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T16:12:48.142 DEBUG:teuthology.parallel:result is None 2024-04-25T16:12:48.142 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T16:12:48.177 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T16:12:48.177 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-25T16:12:48.189 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T16:12:48.225 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-25T16:12:48.235 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-25T16:12:49.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:49 smithi063 ceph-mon[32413]: pgmap v696: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 919 MiB/s wr, 1.93k op/s 2024-04-25T16:12:49.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:49 smithi186 ceph-mon[33258]: pgmap v696: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 919 MiB/s wr, 1.93k op/s 2024-04-25T16:12:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:49 smithi031 ceph-mon[29140]: pgmap v696: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 919 MiB/s wr, 1.93k op/s 2024-04-25T16:12:50.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:50 smithi063 ceph-mon[32413]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:12:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:50 smithi186 ceph-mon[33258]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:12:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:50 smithi031 ceph-mon[29140]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-25T16:12:51.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:51 smithi063 ceph-mon[32413]: pgmap v697: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 736 MiB/s wr, 1.42k op/s 2024-04-25T16:12:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:51 smithi186 ceph-mon[33258]: pgmap v697: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 736 MiB/s wr, 1.42k op/s 2024-04-25T16:12:51.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:51 smithi031 ceph-mon[29140]: pgmap v697: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 736 MiB/s wr, 1.42k op/s 2024-04-25T16:12:53.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:53 smithi063 ceph-mon[32413]: pgmap v698: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 727 MiB/s wr, 1.38k op/s 2024-04-25T16:12:53.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:53 smithi186 ceph-mon[33258]: pgmap v698: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 727 MiB/s wr, 1.38k op/s 2024-04-25T16:12:53.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:53 smithi031 ceph-mon[29140]: pgmap v698: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 727 MiB/s wr, 1.38k op/s 2024-04-25T16:12:55.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:55 smithi063 ceph-mon[32413]: pgmap v699: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 548 MiB/s wr, 1.06k op/s 2024-04-25T16:12:55.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:55 smithi186 ceph-mon[33258]: pgmap v699: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 548 MiB/s wr, 1.06k op/s 2024-04-25T16:12:55.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:55 smithi031 ceph-mon[29140]: pgmap v699: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 548 MiB/s wr, 1.06k op/s 2024-04-25T16:12:57.621 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:57 smithi063 ceph-mon[32413]: pgmap v700: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 398 MiB/s wr, 790 op/s 2024-04-25T16:12:57.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:57 smithi186 ceph-mon[33258]: pgmap v700: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 398 MiB/s wr, 790 op/s 2024-04-25T16:12:57.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:57 smithi031 ceph-mon[29140]: pgmap v700: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 398 MiB/s wr, 790 op/s 2024-04-25T16:12:59.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:12:59 smithi063 ceph-mon[32413]: pgmap v701: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 367 MiB/s wr, 734 op/s 2024-04-25T16:12:59.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:12:59 smithi186 ceph-mon[33258]: pgmap v701: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 367 MiB/s wr, 734 op/s 2024-04-25T16:12:59.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:12:59 smithi031 ceph-mon[29140]: pgmap v701: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 367 MiB/s wr, 734 op/s 2024-04-25T16:13:00.749 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-25T16:13:00.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.878+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/3725059860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf24106f40 msgr2=0x7fdf24109360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.878+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/3725059860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf24106f40 0x7fdf24109360 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009900 tx=0x7fdf0c02a410 comp rx=0 tx=0).stop 2024-04-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.878+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/3725059860 shutdown_connections 2024-04-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.878+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/3725059860 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf241098a0 0x7fdf2410bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.878+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/3725059860 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf24106f40 0x7fdf24109360 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.878+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/3725059860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf241045e0 0x7fdf24106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.878+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/3725059860 >> 172.21.15.31:0/3725059860 conn(0x7fdf240fdbc0 msgr2=0x7fdf24100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.878+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/3725059860 shutdown_connections 2024-04-25T16:13:00.880 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.878+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/3725059860 wait complete. 2024-04-25T16:13:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf2a0d8640 1 Processor -- start 2024-04-25T16:13:00.881 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf2a0d8640 1 -- start start 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf2a0d8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf241045e0 0x7fdf2419a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf2a0d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf24106f40 0x7fdf2419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf2a0d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf241098a0 0x7fdf241a1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf2a0d8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2410f620 con 0x7fdf241098a0 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf2a0d8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2410f4a0 con 0x7fdf24106f40 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf2a0d8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdf2410f7a0 con 0x7fdf241045e0 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf23fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf241098a0 0x7fdf241a1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf23fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf241098a0 0x7fdf241a1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57292/0 (socket says 172.21.15.31:57292) 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.879+0000 7fdf23fff640 1 -- 172.21.15.31:0/2596007368 learned_addr learned my addr 172.21.15.31:0/2596007368 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf1affd640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf24106f40 0x7fdf2419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.883 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf237fe640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf241045e0 0x7fdf2419a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf23fff640 1 -- 172.21.15.31:0/2596007368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf241045e0 msgr2=0x7fdf2419a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf23fff640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf241045e0 0x7fdf2419a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf23fff640 1 -- 172.21.15.31:0/2596007368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf24106f40 msgr2=0x7fdf2419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf23fff640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf24106f40 0x7fdf2419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf23fff640 1 -- 172.21.15.31:0/2596007368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf241a1dc0 con 0x7fdf241098a0 2024-04-25T16:13:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf23fff640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf241098a0 0x7fdf241a1750 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00eb40 tx=0x7fdf1c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf217fa640 1 -- 172.21.15.31:0/2596007368 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c004b20 con 0x7fdf241098a0 2024-04-25T16:13:00.884 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf217fa640 1 -- 172.21.15.31:0/2596007368 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf1c004cc0 con 0x7fdf241098a0 2024-04-25T16:13:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/2596007368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf241a20b0 con 0x7fdf241098a0 2024-04-25T16:13:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf217fa640 1 -- 172.21.15.31:0/2596007368 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf1c010720 con 0x7fdf241098a0 2024-04-25T16:13:00.885 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.880+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/2596007368 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf241a25b0 con 0x7fdf241098a0 2024-04-25T16:13:00.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf217fa640 1 -- 172.21.15.31:0/2596007368 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fdf1c005560 con 0x7fdf241098a0 2024-04-25T16:13:00.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf30f88740 1 Processor -- start 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf30f88740 1 -- start start 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558001457390 0x5580017f8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580017f8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 0x5580017f9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf30f88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55800175dd80 con 0x55800142e4d0 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf30f88740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5580017178b0 con 0x558001457390 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf30f88740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x558001759d40 con 0x5580014366b0 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf237fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580017f8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf237fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580017f8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57306/0 (socket says 172.21.15.31:57306) 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf237fe640 1 -- 172.21.15.31:0/3760840700 learned_addr learned my addr 172.21.15.31:0/3760840700 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:00.897 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.883+0000 7fdf217fa640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdf100afaf0 0x7fdf100b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf1affd640 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 0x5580017f9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf1affd640 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558001457390 0x5580017f8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf23fff640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdf100afaf0 0x7fdf100b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf237fe640 1 -- 172.21.15.31:0/3760840700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 msgr2=0x5580017f9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf237fe640 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 0x5580017f9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf237fe640 1 -- 172.21.15.31:0/3760840700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558001457390 msgr2=0x5580017f8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf237fe640 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558001457390 0x5580017f8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf237fe640 1 -- 172.21.15.31:0/3760840700 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580016e8a40 con 0x55800142e4d0 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf217fa640 1 -- 172.21.15.31:0/2596007368 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdf1c0df450 con 0x7fdf241098a0 2024-04-25T16:13:00.898 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf1affd640 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 0x5580017f9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf237fe640 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580017f8fe0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400bc90 tx=0x7fdf1400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73db497ab40035f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf23fff640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdf100afaf0 0x7fdf100b2000 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c0a7640 tx=0x7fdf1c072160 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf19ffb640 1 -- 172.21.15.31:0/3760840700 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf14015070 con 0x55800142e4d0 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf19ffb640 1 -- 172.21.15.31:0/3760840700 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf1401a910 con 0x55800142e4d0 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.884+0000 7fdf19ffb640 1 -- 172.21.15.31:0/3760840700 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf14011830 con 0x55800142e4d0 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.885+0000 7fdf30f88740 1 -- 172.21.15.31:0/3760840700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 msgr2=0x5580017f8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.885+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580017f8fe0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fdf1400bc90 tx=0x7fdf1400f730 comp rx=0 tx=0).stop 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.885+0000 7fdf30f88740 1 -- 172.21.15.31:0/3760840700 shutdown_connections 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.885+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 0x5580017f9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.885+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580017f8fe0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.885+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3760840700 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558001457390 0x5580017f8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.885+0000 7fdf30f88740 1 -- 172.21.15.31:0/3760840700 >> 172.21.15.31:0/3760840700 conn(0x55800144b4e0 msgr2=0x5580017f4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.885+0000 7fdf30f88740 1 -- 172.21.15.31:0/3760840700 shutdown_connections 2024-04-25T16:13:00.899 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.885+0000 7fdf30f88740 1 -- 172.21.15.31:0/3760840700 wait complete. 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf30f88740 1 -- start start 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf30f88740 1 Processor -- start 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580018028d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 0x558001802e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558001457390 0x558001809d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf30f88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x5580017fa780 con 0x55800142e4d0 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf30f88740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x5580016e4d10 con 0x558001457390 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf30f88740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5580017fa900 con 0x5580014366b0 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf30f88740 10 client.0 _ll_register_callbacks cb 0x558001711210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf1affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580018028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf1affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580018028d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57314/0 (socket says 172.21.15.31:57314) 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.886+0000 7fdf1affd640 1 -- 172.21.15.31:0/3775295890 learned_addr learned my addr 172.21.15.31:0/3775295890 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:00.900 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf237fe640 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 0x558001802e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf1affd640 1 -- 172.21.15.31:0/3775295890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 msgr2=0x558001802e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf1affd640 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 0x558001802e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf1affd640 1 -- 172.21.15.31:0/3775295890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558001457390 msgr2=0x558001809d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf1affd640 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558001457390 0x558001809d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf1affd640 1 -- 172.21.15.31:0/3775295890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580015397d0 con 0x55800142e4d0 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf1affd640 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580018028d0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0096c0 tx=0x7fdf0c032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1e0503bb3e86758 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf02ffd640 1 -- 172.21.15.31:0/3775295890 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c043070 con 0x55800142e4d0 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf02ffd640 1 -- 172.21.15.31:0/3775295890 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdf0c004040 con 0x55800142e4d0 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf30f88740 1 -- 172.21.15.31:0/3775295890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf1401a910 con 0x55800142e4d0 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf02ffd640 1 -- 172.21.15.31:0/3775295890 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdf0c035040 con 0x55800142e4d0 2024-04-25T16:13:00.901 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf30f88740 10 client.17877 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.887+0000 7fdf30f88740 1 -- 172.21.15.31:0/3775295890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55800146f3f0 con 0x55800142e4d0 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf02ffd640 1 -- 172.21.15.31:0/3775295890 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf1401a910 con 0x55800142e4d0 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf02ffd640 1 -- 172.21.15.31:0/3775295890 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fdf0c048460 con 0x55800142e4d0 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf30f88740 10 client.17877 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf30f88740 10 client.17877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf30f88740 20 client.17877 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558001415f60 0x558001810260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf30f88740 4 client.17877 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf30f88740 1 -- 172.21.15.31:0/3775295890 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55800176b990 con 0x558001415f60 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf237fe640 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558001415f60 0x558001810260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf02ffd640 10 client.17877 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:13:00.902 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.888+0000 7fdf237fe640 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558001415f60 0x558001810260 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36c4e04813ae684a server_cookie=53920a6be38da38a in_seq=0 out_seq=0 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.889+0000 7fdf02ffd640 1 -- 172.21.15.31:0/3775295890 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55800176b990 con 0x558001415f60 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.889+0000 7fdf02ffd640 10 client.17877 handle_command_reply: tid=0 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.889+0000 7fdf30f88740 1 client.17877 shutdown 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.889+0000 7fdf30f88740 1 -- 172.21.15.31:0/3775295890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 msgr2=0x5580018028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.889+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580018028d0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0096c0 tx=0x7fdf0c032dc0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf30f88740 1 -- 172.21.15.31:0/3775295890 shutdown_connections 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x558001415f60 0x558001810260 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x558001457390 0x558001809d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5580014366b0 0x558001802e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf02ffd640 0 client.17877 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf30f88740 1 --2- 172.21.15.31:0/3775295890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55800142e4d0 0x5580018028d0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.903 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf30f88740 1 -- 172.21.15.31:0/3775295890 >> 172.21.15.31:0/3775295890 conn(0x55800144b4e0 msgr2=0x5580017f7180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:00.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf30f88740 1 -- 172.21.15.31:0/3775295890 shutdown_connections 2024-04-25T16:13:00.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf30f88740 1 -- 172.21.15.31:0/3775295890 wait complete. 2024-04-25T16:13:00.904 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.890+0000 7fdf30f88740 20 client.17877 trim_cache size 0 max 16384 2024-04-25T16:13:00.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf30f88740 1 Processor -- start 2024-04-25T16:13:00.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf30f88740 1 -- start start 2024-04-25T16:13:00.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 0x558001802bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 0x558001809d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001800f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf30f88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf241ab210 con 0x558001457390 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf30f88740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdf241acd80 con 0x5580014366b0 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf30f88740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdf241ab8e0 con 0x55800142e4d0 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf1affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001800f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf1affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001800f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57326/0 (socket says 172.21.15.31:57326) 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.908+0000 7fdf1affd640 1 -- 172.21.15.31:0/2315457149 learned_addr learned my addr 172.21.15.31:0/2315457149 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf1affd640 1 --2- 172.21.15.31:0/2315457149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 0x558001802bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf237fe640 1 --2- 172.21.15.31:0/2315457149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 0x558001809d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf1affd640 1 -- 172.21.15.31:0/2315457149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 msgr2=0x558001802bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf1affd640 1 --2- 172.21.15.31:0/2315457149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 0x558001802bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf1affd640 1 -- 172.21.15.31:0/2315457149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 msgr2=0x558001809d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:00.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf1affd640 1 --2- 172.21.15.31:0/2315457149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 0x558001809d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf1affd640 1 -- 172.21.15.31:0/2315457149 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55800142f0f0 con 0x558001457390 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf1affd640 1 --2- 172.21.15.31:0/2315457149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001800f00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c031500 tx=0x7fdf0c040040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36045cb458b3e462 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf197fa640 1 -- 172.21.15.31:0/2315457149 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c043070 con 0x558001457390 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf197fa640 1 -- 172.21.15.31:0/2315457149 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf0c048560 con 0x558001457390 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.909+0000 7fdf197fa640 1 -- 172.21.15.31:0/2315457149 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c000e60 con 0x558001457390 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 -- 172.21.15.31:0/2315457149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 msgr2=0x558001800f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 --2- 172.21.15.31:0/2315457149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001800f00 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c031500 tx=0x7fdf0c040040 comp rx=0 tx=0).stop 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 -- 172.21.15.31:0/2315457149 shutdown_connections 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 --2- 172.21.15.31:0/2315457149 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001800f00 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 --2- 172.21.15.31:0/2315457149 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 0x558001809d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 --2- 172.21.15.31:0/2315457149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 0x558001802bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 -- 172.21.15.31:0/2315457149 >> 172.21.15.31:0/2315457149 conn(0x55800144b4e0 msgr2=0x5580018024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 -- 172.21.15.31:0/2315457149 shutdown_connections 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 -- 172.21.15.31:0/2315457149 wait complete. 2024-04-25T16:13:00.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 -- start start 2024-04-25T16:13:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.910+0000 7fdf30f88740 1 Processor -- start 2024-04-25T16:13:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 0x558001809960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 0x558001806530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf30f88740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001806a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.918 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf30f88740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fdf241ac6b0 con 0x558001457390 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf30f88740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fdf241abfb0 con 0x5580014366b0 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf30f88740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdf241a91f0 con 0x55800142e4d0 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf30f88740 10 client.0 _ll_register_callbacks cb 0x7fdf0c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001806a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001806a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57332/0 (socket says 172.21.15.31:57332) 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 -- 172.21.15.31:0/907062857 learned_addr learned my addr 172.21.15.31:0/907062857 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 0x558001809960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf237fe640 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 0x558001806530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 -- 172.21.15.31:0/907062857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 msgr2=0x558001809960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 0x558001809960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 -- 172.21.15.31:0/907062857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 msgr2=0x558001806530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 0x558001806530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 -- 172.21.15.31:0/907062857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55800171e4b0 con 0x558001457390 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf1affd640 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001806a70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009300 tx=0x7fdf0c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337304653a7fedea server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:00.919 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf02ffd640 1 -- 172.21.15.31:0/907062857 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c043070 con 0x558001457390 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf02ffd640 1 -- 172.21.15.31:0/907062857 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fdf0c048ae0 con 0x558001457390 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.911+0000 7fdf30f88740 1 -- 172.21.15.31:0/907062857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdf0c048560 con 0x558001457390 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf02ffd640 1 -- 172.21.15.31:0/907062857 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fdf0c048ce0 con 0x558001457390 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf30f88740 10 client.17883 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf30f88740 1 -- 172.21.15.31:0/907062857 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x558001877310 con 0x558001457390 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf02ffd640 1 -- 172.21.15.31:0/907062857 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf0c048560 con 0x558001457390 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf02ffd640 1 -- 172.21.15.31:0/907062857 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fdf0c03d820 con 0x558001457390 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf30f88740 10 client.17883 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf30f88740 10 client.17883 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf30f88740 20 client.17883 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf30f88740 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5580013eb3a0 0x558001879c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf30f88740 4 client.17883 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf30f88740 1 -- 172.21.15.31:0/907062857 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558001765800 con 0x5580013eb3a0 2024-04-25T16:13:00.920 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.912+0000 7fdf237fe640 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5580013eb3a0 0x558001879c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf02ffd640 10 client.17883 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf237fe640 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5580013eb3a0 0x558001879c90 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb74f114872f017e server_cookie=a99432b05ac9b3a8 in_seq=0 out_seq=0 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf02ffd640 1 -- 172.21.15.31:0/907062857 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x558001765800 con 0x5580013eb3a0 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf02ffd640 10 client.17883 handle_command_reply: tid=0 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 client.17883 shutdown 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 -- 172.21.15.31:0/907062857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 msgr2=0x558001806a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001806a70 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009300 tx=0x7fdf0c02a410 comp rx=0 tx=0).stop 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 -- 172.21.15.31:0/907062857 shutdown_connections 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x5580013eb3a0 0x558001879c90 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x558001457390 0x558001806a70 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x5580014366b0 0x558001806530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf02ffd640 0 client.17883 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 --2- 172.21.15.31:0/907062857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55800142e4d0 0x558001809960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.921 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 -- 172.21.15.31:0/907062857 >> 172.21.15.31:0/907062857 conn(0x55800144b4e0 msgr2=0x558001800f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:00.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 -- 172.21.15.31:0/907062857 shutdown_connections 2024-04-25T16:13:00.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 1 -- 172.21.15.31:0/907062857 wait complete. 2024-04-25T16:13:00.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.913+0000 7fdf30f88740 20 client.17883 trim_cache size 0 max 16384 2024-04-25T16:13:00.922 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:13:00.922 INFO:teuthology.orchestra.run.smithi031.stdout: "status": "no active scrubs running", 2024-04-25T16:13:00.922 INFO:teuthology.orchestra.run.smithi031.stdout: "scrubs": {} 2024-04-25T16:13:00.922 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:13:00.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdf100afaf0 msgr2=0x7fdf100b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:00.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdf100afaf0 0x7fdf100b2000 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c0a7640 tx=0x7fdf1c072160 comp rx=0 tx=0).stop 2024-04-25T16:13:00.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf241098a0 msgr2=0x7fdf241a1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:00.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf241098a0 0x7fdf241a1750 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c00eb40 tx=0x7fdf1c00c2d0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/2596007368 shutdown_connections 2024-04-25T16:13:00.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fdf100afaf0 0x7fdf100b2000 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fdf241098a0 0x7fdf241a1750 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf24106f40 0x7fdf2419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 --2- 172.21.15.31:0/2596007368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf241045e0 0x7fdf2419a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:00.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/2596007368 >> 172.21.15.31:0/2596007368 conn(0x7fdf240fdbc0 msgr2=0x7fdf241065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:00.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/2596007368 shutdown_connections 2024-04-25T16:13:00.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:00.914+0000 7fdf2a0d8640 1 -- 172.21.15.31:0/2596007368 wait complete. 2024-04-25T16:13:00.927 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-25T16:13:00.928 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:13:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.038+0000 7f44ffaf9640 1 -- 172.21.15.31:0/422020929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8103b00 msgr2=0x7f44f8104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.038+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/422020929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8103b00 0x7f44f8104800 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009900 tx=0x7f44f402a410 comp rx=0 tx=0).stop 2024-04-25T16:13:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.038+0000 7f44ffaf9640 1 -- 172.21.15.31:0/422020929 shutdown_connections 2024-04-25T16:13:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.038+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/422020929 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44f8109a40 0x7f44f810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.038+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/422020929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44f8104d40 0x7f44f8105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.038+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/422020929 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8103b00 0x7f44f8104800 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.040 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.038+0000 7f44ffaf9640 1 -- 172.21.15.31:0/422020929 >> 172.21.15.31:0/422020929 conn(0x7f44f80fed10 msgr2=0x7f44f81011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.038+0000 7f44ffaf9640 1 -- 172.21.15.31:0/422020929 shutdown_connections 2024-04-25T16:13:01.041 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.038+0000 7f44ffaf9640 1 -- 172.21.15.31:0/422020929 wait complete. 2024-04-25T16:13:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.039+0000 7f44ffaf9640 1 Processor -- start 2024-04-25T16:13:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.039+0000 7f44ffaf9640 1 -- start start 2024-04-25T16:13:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.039+0000 7f44ffaf9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44f8103b00 0x7f44f819c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.039+0000 7f44ffaf9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44f8104d40 0x7f44f819c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.039+0000 7f44ffaf9640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8109a40 0x7f44f81a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.039+0000 7f44ffaf9640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f44f810f600 con 0x7f44f8109a40 2024-04-25T16:13:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.039+0000 7f44ffaf9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f44f810f480 con 0x7f44f8103b00 2024-04-25T16:13:01.044 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.039+0000 7f44ffaf9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f44f810f780 con 0x7f44f8104d40 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fe096640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8109a40 0x7f44f81a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fe096640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8109a40 0x7f44f81a3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57340/0 (socket says 172.21.15.31:57340) 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fe096640 1 -- 172.21.15.31:0/589456185 learned_addr learned my addr 172.21.15.31:0/589456185 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fd094640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44f8104d40 0x7f44f819c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fd895640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44f8103b00 0x7f44f819c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fe096640 1 -- 172.21.15.31:0/589456185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44f8104d40 msgr2=0x7f44f819c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fe096640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44f8104d40 0x7f44f819c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fe096640 1 -- 172.21.15.31:0/589456185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44f8103b00 msgr2=0x7f44f819c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fe096640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44f8103b00 0x7f44f819c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fe096640 1 -- 172.21.15.31:0/589456185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44f81a3f20 con 0x7f44f8109a40 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44fe096640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8109a40 0x7f44f81a3960 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f44f000cce0 tx=0x7f44f000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44eeffd640 1 -- 172.21.15.31:0/589456185 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f000f450 con 0x7f44f8109a40 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44eeffd640 1 -- 172.21.15.31:0/589456185 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f44f000faa0 con 0x7f44f8109a40 2024-04-25T16:13:01.045 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44f81a4210 con 0x7f44f8109a40 2024-04-25T16:13:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44eeffd640 1 -- 172.21.15.31:0/589456185 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0018b70 con 0x7f44f8109a40 2024-04-25T16:13:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.040+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44f81a4760 con 0x7f44f8109a40 2024-04-25T16:13:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.041+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44f806a190 con 0x7f44f8109a40 2024-04-25T16:13:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.043+0000 7f44eeffd640 1 -- 172.21.15.31:0/589456185 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f44f0018d10 con 0x7f44f8109a40 2024-04-25T16:13:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.043+0000 7f44eeffd640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f44e40afb70 0x7f44e40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.044+0000 7f44eeffd640 1 -- 172.21.15.31:0/589456185 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f44f00df0b0 con 0x7f44f8109a40 2024-04-25T16:13:01.046 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.044+0000 7f44fd895640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f44e40afb70 0x7f44e40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.047 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.044+0000 7f44fd895640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f44e40afb70 0x7f44e40b2080 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009230 tx=0x7f44f4031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.048 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.047+0000 7f44eeffd640 1 -- 172.21.15.31:0/589456185 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f44f0014030 con 0x7f44f8109a40 2024-04-25T16:13:01.279 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.278+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44f81a4c00 con 0x7f44f8109a40 2024-04-25T16:13:01.280 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.278+0000 7f44eeffd640 1 -- 172.21.15.31:0/589456185 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11918 (secure 0 0 0) 0x7f44f00a7db0 con 0x7f44f8109a40 2024-04-25T16:13:01.280 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:13:01.280 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:12:49.118320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[14631,24553,24563,24521,24535]},"id":1}]} 2024-04-25T16:13:01.280 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f44e40afb70 msgr2=0x7f44e40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f44e40afb70 0x7f44e40b2080 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f44f4009230 tx=0x7f44f4031400 comp rx=0 tx=0).stop 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8109a40 msgr2=0x7f44f81a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8109a40 0x7f44f81a3960 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f44f000cce0 tx=0x7f44f000a620 comp rx=0 tx=0).stop 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 shutdown_connections 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f44e40afb70 0x7f44e40b2080 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f44f8109a40 0x7f44f81a3960 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44f8104d40 0x7f44f819c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 --2- 172.21.15.31:0/589456185 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f44f8103b00 0x7f44f819c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 >> 172.21.15.31:0/589456185 conn(0x7f44f80fed10 msgr2=0x7f44f80ff120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.284 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 shutdown_connections 2024-04-25T16:13:01.285 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.282+0000 7f44ffaf9640 1 -- 172.21.15.31:0/589456185 wait complete. 2024-04-25T16:13:01.302 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-25T16:13:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.418+0000 7fa0a368c640 1 -- 172.21.15.31:0/1925182582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c1029f0 msgr2=0x7fa09c1036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.418+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1925182582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c1029f0 0x7fa09c1036f0 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fa098009900 tx=0x7fa09802a410 comp rx=0 tx=0).stop 2024-04-25T16:13:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.418+0000 7fa0a368c640 1 -- 172.21.15.31:0/1925182582 shutdown_connections 2024-04-25T16:13:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.418+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1925182582 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa09c108930 0x7fa09c10ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.422 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.418+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1925182582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa09c103c30 0x7fa09c104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.418+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1925182582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c1029f0 0x7fa09c1036f0 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.418+0000 7fa0a368c640 1 -- 172.21.15.31:0/1925182582 >> 172.21.15.31:0/1925182582 conn(0x7fa09c0fdbc0 msgr2=0x7fa09c100090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 -- 172.21.15.31:0/1925182582 shutdown_connections 2024-04-25T16:13:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 -- 172.21.15.31:0/1925182582 wait complete. 2024-04-25T16:13:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 Processor -- start 2024-04-25T16:13:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 -- start start 2024-04-25T16:13:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa09c1029f0 0x7fa09c075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c103c30 0x7fa09c19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.423 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa09c108930 0x7fa09c1a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c10e4f0 con 0x7fa09c103c30 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c10e370 con 0x7fa09c1029f0 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.419+0000 7fa0a368c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c10e670 con 0x7fa09c108930 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a0c27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c103c30 0x7fa09c19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a0c27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c103c30 0x7fa09c19efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57362/0 (socket says 172.21.15.31:57362) 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a0c27640 1 -- 172.21.15.31:0/1385482001 learned_addr learned my addr 172.21.15.31:0/1385482001 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a1428640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa09c1029f0 0x7fa09c075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a0c27640 1 -- 172.21.15.31:0/1385482001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa09c108930 msgr2=0x7fa09c1a1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa09c108930 0x7fa09c1a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa09c108930 0x7fa09c1a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a0c27640 1 -- 172.21.15.31:0/1385482001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa09c1029f0 msgr2=0x7fa09c075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa09c1029f0 0x7fa09c075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a0c27640 1 -- 172.21.15.31:0/1385482001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa09c1a1d40 con 0x7fa09c103c30 2024-04-25T16:13:01.424 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a1428640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa09c1029f0 0x7fa09c075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:01.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c103c30 0x7fa09c19efd0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00e9f0 tx=0x7fa08c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0927fc640 1 -- 172.21.15.31:0/1385482001 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00cd80 con 0x7fa09c103c30 2024-04-25T16:13:01.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0927fc640 1 -- 172.21.15.31:0/1385482001 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa08c0054d0 con 0x7fa09c103c30 2024-04-25T16:13:01.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a368c640 1 -- 172.21.15.31:0/1385482001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa09c1a1fd0 con 0x7fa09c103c30 2024-04-25T16:13:01.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0927fc640 1 -- 172.21.15.31:0/1385482001 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08c010690 con 0x7fa09c103c30 2024-04-25T16:13:01.425 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.420+0000 7fa0a368c640 1 -- 172.21.15.31:0/1385482001 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa09c1a2470 con 0x7fa09c103c30 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0aa60f740 1 Processor -- start 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0aa60f740 1 -- start start 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0927fc640 1 -- 172.21.15.31:0/1385482001 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa08c004e90 con 0x7fa09c103c30 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 0x557ceaadb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaadbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7186b0 0x557ceaadca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557ceaa3fd80 con 0x557cea7186b0 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557cea9f9a20 con 0x557cea7104d0 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ceaa3bd40 con 0x557cea739390 2024-04-25T16:13:01.432 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0a0c27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7186b0 0x557ceaadca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0a0c27640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7186b0 0x557ceaadca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57368/0 (socket says 172.21.15.31:57368) 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.423+0000 7fa0a0c27640 1 -- 172.21.15.31:0/1468393896 learned_addr learned my addr 172.21.15.31:0/1468393896 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/1468393896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 0x557ceaadb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0a1428640 1 --2- 172.21.15.31:0/1468393896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaadbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0927fc640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0880afaf0 0x7fa0880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0a0c27640 1 -- 172.21.15.31:0/1468393896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 msgr2=0x557ceaadb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/1468393896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 0x557ceaadb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0a0c27640 1 -- 172.21.15.31:0/1468393896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 msgr2=0x557ceaadbd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/1468393896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaadbd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0a0c27640 1 -- 172.21.15.31:0/1468393896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ceaa04c30 con 0x557cea7186b0 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0880afaf0 0x7fa0880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/1468393896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7186b0 0x557ceaadca20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa08c0c4190 tx=0x7fa08c0a34e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76487b0347e81b16 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0927fc640 1 -- 172.21.15.31:0/1385482001 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa08c0df130 con 0x7fa09c103c30 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0877fe640 1 -- 172.21.15.31:0/1468393896 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08c0a73f0 con 0x557cea7186b0 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0877fe640 1 -- 172.21.15.31:0/1468393896 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa08c0a7130 con 0x557cea7186b0 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.424+0000 7fa0877fe640 1 -- 172.21.15.31:0/1468393896 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa08c005180 con 0x557cea7186b0 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1468393896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7186b0 msgr2=0x557ceaadca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0880afaf0 0x7fa0880b2000 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x557ceaadc860 tx=0x7fa094004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1468393896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7186b0 0x557ceaadca20 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa08c0c4190 tx=0x7fa08c0a34e0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1468393896 shutdown_connections 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1468393896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7186b0 0x557ceaadca20 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1468393896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaadbd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1468393896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 0x557ceaadb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1468393896 >> 172.21.15.31:0/1468393896 conn(0x557cea72d4e0 msgr2=0x557ceaad7020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1468393896 shutdown_connections 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.425+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1468393896 wait complete. 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0aa60f740 1 -- start start 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0aa60f740 1 Processor -- start 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7104d0 0x557ceaae55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7186b0 0x557ceaae5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 0x557ceaaeca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x557ceaadd530 con 0x557cea7104d0 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x557cea9c6dc0 con 0x557cea7186b0 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x557ceaadd6b0 con 0x557cea739390 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0aa60f740 10 client.0 _ll_register_callbacks cb 0x557cea9e2600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0a1428640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7104d0 0x557ceaae55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0a1428640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7104d0 0x557ceaae55d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57384/0 (socket says 172.21.15.31:57384) 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.426+0000 7fa0a1428640 1 -- 172.21.15.31:0/2876076848 learned_addr learned my addr 172.21.15.31:0/2876076848 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7186b0 0x557ceaae5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 0x557ceaaeca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a1428640 1 -- 172.21.15.31:0/2876076848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 msgr2=0x557ceaaeca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a1428640 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 0x557ceaaeca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a1428640 1 -- 172.21.15.31:0/2876076848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7186b0 msgr2=0x557ceaae5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a1428640 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7186b0 0x557ceaae5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a1428640 1 -- 172.21.15.31:0/2876076848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557cea7513f0 con 0x557cea7104d0 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7186b0 0x557ceaae5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 0x557ceaaeca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0a1428640 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7104d0 0x557ceaae55d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fa09c1022c0 tx=0x7fa098032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=571bc4f0e3cfa09d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa084ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa084ff9640 1 -- 172.21.15.31:0/2876076848 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa098040070 con 0x557cea7104d0 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa084ff9640 1 -- 172.21.15.31:0/2876076848 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa098004040 con 0x557cea7104d0 2024-04-25T16:13:01.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2876076848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa08c0a7130 con 0x557cea7104d0 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa084ff9640 1 -- 172.21.15.31:0/2876076848 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa098035040 con 0x557cea7104d0 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0aa60f740 10 client.17901 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2876076848 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557cea81b7d0 con 0x557cea7104d0 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.427+0000 7fa084ff9640 1 -- 172.21.15.31:0/2876076848 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa08c0a7130 con 0x557cea7104d0 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa084ff9640 1 -- 172.21.15.31:0/2876076848 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fa098004880 con 0x557cea7104d0 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa0aa60f740 10 client.17901 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa0aa60f740 10 client.17901 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa0aa60f740 20 client.17901 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557cea6f7f60 0x557ceaaf3320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa0aa60f740 4 client.17901 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2876076848 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ceaa4d990 con 0x557cea6f7f60 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557cea6f7f60 0x557ceaaf3320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa084ff9640 10 client.17901 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:13:01.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.428+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557cea6f7f60 0x557ceaaf3320 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c267b4b3439a1b server_cookie=d70c7060871d4a78 in_seq=0 out_seq=0 2024-04-25T16:13:01.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa084ff9640 1 -- 172.21.15.31:0/2876076848 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557ceaa4d990 con 0x557cea6f7f60 2024-04-25T16:13:01.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa084ff9640 10 client.17901 handle_command_reply: tid=0 2024-04-25T16:13:01.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 client.17901 shutdown 2024-04-25T16:13:01.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2876076848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7104d0 msgr2=0x557ceaae55d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7104d0 0x557ceaae55d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fa09c1022c0 tx=0x7fa098032dc0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2876076848 shutdown_connections 2024-04-25T16:13:01.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557cea6f7f60 0x557ceaaf3320 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea739390 0x557ceaaeca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7186b0 0x557ceaae5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa084ff9640 0 client.17901 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:13:01.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2876076848 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea7104d0 0x557ceaae55d0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2876076848 >> 172.21.15.31:0/2876076848 conn(0x557cea72d4e0 msgr2=0x557ceaad9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2876076848 shutdown_connections 2024-04-25T16:13:01.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2876076848 wait complete. 2024-04-25T16:13:01.439 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.429+0000 7fa0aa60f740 20 client.17901 trim_cache size 0 max 16384 2024-04-25T16:13:01.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0aa60f740 1 Processor -- start 2024-04-25T16:13:01.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0aa60f740 1 -- start start 2024-04-25T16:13:01.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaad6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 0x557ceaaec920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaad6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c1ab2c0 con 0x557cea739390 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c1ace30 con 0x557cea7104d0 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c1ab990 con 0x557cea7186b0 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0a1c29640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaad6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0a1c29640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaad6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57396/0 (socket says 172.21.15.31:57396) 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0a1c29640 1 -- 172.21.15.31:0/2365406751 learned_addr learned my addr 172.21.15.31:0/2365406751 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.443+0000 7fa0a1428640 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaad6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 0x557ceaaec920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa0a1c29640 1 -- 172.21.15.31:0/2365406751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 msgr2=0x557ceaaec920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 0x557ceaaec920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa0a1c29640 1 -- 172.21.15.31:0/2365406751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 msgr2=0x557ceaad6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaad6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa0a1c29640 1 -- 172.21.15.31:0/2365406751 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ceaaece60 con 0x557cea739390 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa0a1428640 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaad6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaad6f90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa09400f820 tx=0x7fa094015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1359d59310424afe server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa086ffd640 1 -- 172.21.15.31:0/2365406751 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09400fe60 con 0x557cea739390 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa086ffd640 1 -- 172.21.15.31:0/2365406751 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa09401ddf0 con 0x557cea739390 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.444+0000 7fa086ffd640 1 -- 172.21.15.31:0/2365406751 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09401bb70 con 0x557cea739390 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2365406751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 msgr2=0x557ceaad6f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaad6f90 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fa09400f820 tx=0x7fa094015000 comp rx=0 tx=0).stop 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2365406751 shutdown_connections 2024-04-25T16:13:01.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaad6f90 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 0x557ceaaec920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/2365406751 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaad6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2365406751 >> 172.21.15.31:0/2365406751 conn(0x557cea72d4e0 msgr2=0x557ceaae0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2365406751 shutdown_connections 2024-04-25T16:13:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 -- 172.21.15.31:0/2365406751 wait complete. 2024-04-25T16:13:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 -- start start 2024-04-25T16:13:01.449 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.445+0000 7fa0aa60f740 1 Processor -- start 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaaec720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 0x557ceaae92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0aa60f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaae97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c1ac760 con 0x557cea739390 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c1ac060 con 0x557cea7104d0 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0aa60f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa09c1a92a0 con 0x557cea7186b0 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0aa60f740 10 client.0 _ll_register_callbacks cb 0x7fa08c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1c29640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaae97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1c29640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaae97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57408/0 (socket says 172.21.15.31:57408) 2024-04-25T16:13:01.453 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1c29640 1 -- 172.21.15.31:0/1980891494 learned_addr learned my addr 172.21.15.31:0/1980891494 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1428640 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaaec720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a0c27640 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 0x557ceaae92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1c29640 1 -- 172.21.15.31:0/1980891494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 msgr2=0x557ceaae92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 0x557ceaae92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1c29640 1 -- 172.21.15.31:0/1980891494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 msgr2=0x557ceaaec720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaaec720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1c29640 1 -- 172.21.15.31:0/1980891494 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ceab59680 con 0x557cea739390 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa0a1c29640 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaae97e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa09401b630 tx=0x7fa09401d0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c02638fd1c8c37 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.446+0000 7fa084ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa084ff9640 1 -- 172.21.15.31:0/1980891494 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09401db50 con 0x557cea739390 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa084ff9640 1 -- 172.21.15.31:0/1980891494 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fa09402c070 con 0x557cea739390 2024-04-25T16:13:01.454 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1980891494 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa09401ddf0 con 0x557cea739390 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa084ff9640 1 -- 172.21.15.31:0/1980891494 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fa094004ab0 con 0x557cea739390 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0aa60f740 10 client.17907 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1980891494 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x557ceab5a7f0 con 0x557cea739390 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa084ff9640 1 -- 172.21.15.31:0/1980891494 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa09401ddf0 con 0x557cea739390 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa084ff9640 1 -- 172.21.15.31:0/1980891494 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7fa094005050 con 0x557cea739390 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0aa60f740 10 client.17907 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0aa60f740 10 client.17907 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0aa60f740 20 client.17907 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557cea6cd3a0 0x557ceab5d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0aa60f740 4 client.17907 mds_command: new command op to 14631 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1980891494 --> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ceaa47800 con 0x557cea6cd3a0 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.447+0000 7fa0a1428640 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557cea6cd3a0 0x557ceab5d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa084ff9640 10 client.17907 ms_handle_connect on v2:172.21.15.31:6840/2149984264 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0a1428640 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557cea6cd3a0 0x557ceab5d1b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=382d891c8a53f4a9 server_cookie=f01be27dacf97934 in_seq=0 out_seq=0 2024-04-25T16:13:01.455 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa084ff9640 1 -- 172.21.15.31:0/1980891494 <== mds.4 v2:172.21.15.31:6840/2149984264 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ceaa47800 con 0x557cea6cd3a0 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa084ff9640 10 client.17907 handle_command_reply: tid=0 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 client.17907 shutdown 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1980891494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 msgr2=0x557ceaae97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaae97e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fa09401b630 tx=0x7fa09401d0d0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1980891494 shutdown_connections 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.31:6840/2149984264,v1:172.21.15.31:6841/2149984264] conn(0x557cea6cd3a0 0x557ceab5d1b0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x557cea739390 0x557ceaae97e0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x557cea7186b0 0x557ceaae92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa084ff9640 0 client.17907 ms_handle_reset on v2:172.21.15.31:6840/2149984264 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 --2- 172.21.15.31:0/1980891494 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x557cea7104d0 0x557ceaaec720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1980891494 >> 172.21.15.31:0/1980891494 conn(0x557cea72d4e0 msgr2=0x557ceaae5140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1980891494 shutdown_connections 2024-04-25T16:13:01.456 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 1 -- 172.21.15.31:0/1980891494 wait complete. 2024-04-25T16:13:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.448+0000 7fa0aa60f740 20 client.17907 trim_cache size 0 max 16384 2024-04-25T16:13:01.457 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:13:01.457 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 -- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0880afaf0 msgr2=0x7fa0880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0880afaf0 0x7fa0880b2000 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x557ceaadc860 tx=0x7fa094004a40 comp rx=0 tx=0).stop 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 -- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c103c30 msgr2=0x7fa09c19efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c103c30 0x7fa09c19efd0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fa08c00e9f0 tx=0x7fa08c00edf0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 -- 172.21.15.31:0/1385482001 shutdown_connections 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fa0880afaf0 0x7fa0880b2000 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa09c108930 0x7fa09c1a1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fa09c103c30 0x7fa09c19efd0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 --2- 172.21.15.31:0/1385482001 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa09c1029f0 0x7fa09c075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 -- 172.21.15.31:0/1385482001 >> 172.21.15.31:0/1385482001 conn(0x7fa09c0fdbc0 msgr2=0x7fa09c104910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 -- 172.21.15.31:0/1385482001 shutdown_connections 2024-04-25T16:13:01.458 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.449+0000 7fa0a368c640 1 -- 172.21.15.31:0/1385482001 wait complete. 2024-04-25T16:13:01.462 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.574+0000 7f575f13f640 1 -- 172.21.15.31:0/794781449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758102a00 msgr2=0x7f5758103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.574+0000 7f575f13f640 1 --2- 172.21.15.31:0/794781449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758102a00 0x7f5758103700 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f574c009900 tx=0x7f574c02a410 comp rx=0 tx=0).stop 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.574+0000 7f575f13f640 1 -- 172.21.15.31:0/794781449 shutdown_connections 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.574+0000 7f575f13f640 1 --2- 172.21.15.31:0/794781449 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5758108940 0x7f575810ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.574+0000 7f575f13f640 1 --2- 172.21.15.31:0/794781449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5758103c40 0x7f5758104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.574+0000 7f575f13f640 1 --2- 172.21.15.31:0/794781449 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758102a00 0x7f5758103700 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.574+0000 7f575f13f640 1 -- 172.21.15.31:0/794781449 >> 172.21.15.31:0/794781449 conn(0x7f57580fdbf0 msgr2=0x7f57581000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.574+0000 7f575f13f640 1 -- 172.21.15.31:0/794781449 shutdown_connections 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.574+0000 7f575f13f640 1 -- 172.21.15.31:0/794781449 wait complete. 2024-04-25T16:13:01.589 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f575f13f640 1 Processor -- start 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f575f13f640 1 -- start start 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f575f13f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5758102a00 0x7f575819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f575f13f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758103c40 0x7f575819a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f575f13f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5758108940 0x7f57581a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f575f13f640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f575810e530 con 0x7f5758103c40 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f575f13f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f575810e3b0 con 0x7f5758108940 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f575f13f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f575810e6b0 con 0x7f5758102a00 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f5757fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758103c40 0x7f575819a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f5757fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758103c40 0x7f575819a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57430/0 (socket says 172.21.15.31:57430) 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f5757fff640 1 -- 172.21.15.31:0/2818891065 learned_addr learned my addr 172.21.15.31:0/2818891065 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.575+0000 7f575cedb640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5758102a00 0x7f575819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.590 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f575d6dc640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5758108940 0x7f57581a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f5757fff640 1 -- 172.21.15.31:0/2818891065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5758102a00 msgr2=0x7f575819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f5757fff640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5758102a00 0x7f575819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f5757fff640 1 -- 172.21.15.31:0/2818891065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5758108940 msgr2=0x7f57581a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f5757fff640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5758108940 0x7f57581a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f5757fff640 1 -- 172.21.15.31:0/2818891065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57581a1d70 con 0x7f5758103c40 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f5757fff640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758103c40 0x7f575819a770 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f574400d770 tx=0x7f574400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f5755ffb640 1 -- 172.21.15.31:0/2818891065 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5744017070 con 0x7f5758103c40 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f5755ffb640 1 -- 172.21.15.31:0/2818891065 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57440054d0 con 0x7f5758103c40 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f5755ffb640 1 -- 172.21.15.31:0/2818891065 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5744004c30 con 0x7f5758103c40 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f575f13f640 1 -- 172.21.15.31:0/2818891065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57581a2060 con 0x7f5758103c40 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.576+0000 7f575f13f640 1 -- 172.21.15.31:0/2818891065 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57581a2560 con 0x7f5758103c40 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5755ffb640 1 -- 172.21.15.31:0/2818891065 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5744004e10 con 0x7f5758103c40 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5765ff5740 1 Processor -- start 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5765ff5740 1 -- start start 2024-04-25T16:13:01.591 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a980390 0x55854ad22810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad22d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad23a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5765ff5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55854ac86d80 con 0x55854a95f6b0 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5765ff5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55854ac40a20 con 0x55854a9574d0 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5765ff5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55854ac82d40 con 0x55854a980390 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5757fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad23a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5757fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad23a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57434/0 (socket says 172.21.15.31:57434) 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5757fff640 1 -- 172.21.15.31:0/3502730631 learned_addr learned my addr 172.21.15.31:0/3502730631 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f575d6dc640 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a980390 0x55854ad22810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f5755ffb640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57480afaf0 0x7f57480b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.579+0000 7f575cedb640 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad22d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5757fff640 1 -- 172.21.15.31:0/3502730631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a980390 msgr2=0x55854ad22810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5757fff640 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a980390 0x55854ad22810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5757fff640 1 -- 172.21.15.31:0/3502730631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 msgr2=0x55854ad22d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.592 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5757fff640 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad22d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5757fff640 1 -- 172.21.15.31:0/3502730631 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55854ac4bc30 con 0x55854a95f6b0 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f575d6dc640 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a980390 0x55854ad22810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f575cedb640 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad22d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f575d6dc640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57480afaf0 0x7f57480b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5755ffb640 1 -- 172.21.15.31:0/2818891065 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f57440de780 con 0x7f5758103c40 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5757fff640 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad23a20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f57440513f0 tx=0x7f5744022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdfe762e220bcb9d server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5742ffd640 1 -- 172.21.15.31:0/3502730631 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440a6a40 con 0x55854a95f6b0 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5742ffd640 1 -- 172.21.15.31:0/3502730631 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57440a6780 con 0x55854a95f6b0 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.580+0000 7f5742ffd640 1 -- 172.21.15.31:0/3502730631 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440a63e0 con 0x55854a95f6b0 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 -- 172.21.15.31:0/3502730631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 msgr2=0x55854ad23a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f575d6dc640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57480afaf0 0x7f57480b2000 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x55854ad23860 tx=0x7f5750004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad23a20 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f57440513f0 tx=0x7f5744022000 comp rx=0 tx=0).stop 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 -- 172.21.15.31:0/3502730631 shutdown_connections 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad23a20 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.593 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad22d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3502730631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a980390 0x55854ad22810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 -- 172.21.15.31:0/3502730631 >> 172.21.15.31:0/3502730631 conn(0x55854a9744e0 msgr2=0x55854ad1e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 -- 172.21.15.31:0/3502730631 shutdown_connections 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 -- 172.21.15.31:0/3502730631 wait complete. 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 -- start start 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.581+0000 7f5765ff5740 1 Processor -- start 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 0x55854ad2c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a95f6b0 0x55854ad2c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad338f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f5765ff5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55854ad24470 con 0x55854a980390 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f5765ff5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55854ac0ddc0 con 0x55854a95f6b0 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f5765ff5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55854ad245f0 con 0x55854a9574d0 2024-04-25T16:13:01.594 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f5765ff5740 10 client.0 _ll_register_callbacks cb 0x55854ac29600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575d6dc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575d6dc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad338f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57446/0 (socket says 172.21.15.31:57446) 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575d6dc640 1 -- 172.21.15.31:0/3147212690 learned_addr learned my addr 172.21.15.31:0/3147212690 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f5757fff640 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a95f6b0 0x55854ad2c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575cedb640 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 0x55854ad2c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575d6dc640 1 -- 172.21.15.31:0/3147212690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 msgr2=0x55854ad2c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575d6dc640 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 0x55854ad2c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575d6dc640 1 -- 172.21.15.31:0/3147212690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a95f6b0 msgr2=0x55854ad2c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575d6dc640 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a95f6b0 0x55854ad2c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575d6dc640 1 -- 172.21.15.31:0/3147212690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55854a9983f0 con 0x55854a980390 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f575d6dc640 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad338f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f57500138e0 tx=0x7f5750004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65767bc6f5984cd8 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.582+0000 7f5737fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:01.595 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5737fff640 1 -- 172.21.15.31:0/3147212690 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5750013e60 con 0x55854a980390 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5737fff640 1 -- 172.21.15.31:0/3147212690 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f575001ed70 con 0x55854a980390 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5765ff5740 1 -- 172.21.15.31:0/3147212690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57440a6780 con 0x55854a980390 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5737fff640 1 -- 172.21.15.31:0/3147212690 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5750028470 con 0x55854a980390 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5765ff5740 10 client.17919 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5765ff5740 1 -- 172.21.15.31:0/3147212690 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55854aa627d0 con 0x55854a980390 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5737fff640 1 -- 172.21.15.31:0/3147212690 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57440a6780 con 0x55854a980390 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5737fff640 1 -- 172.21.15.31:0/3147212690 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f575000f040 con 0x55854a980390 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5765ff5740 10 client.17919 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5765ff5740 10 client.17919 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5765ff5740 20 client.17919 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55854a93ef60 0x55854ad3a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5765ff5740 4 client.17919 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.583+0000 7f5765ff5740 1 -- 172.21.15.31:0/3147212690 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55854ac94990 con 0x55854a93ef60 2024-04-25T16:13:01.596 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.584+0000 7f575cedb640 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55854a93ef60 0x55854ad3a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.584+0000 7f5737fff640 10 client.17919 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.584+0000 7f575cedb640 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55854a93ef60 0x55854ad3a1e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=139f0a160e228223 server_cookie=a95645681879103 in_seq=0 out_seq=0 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5737fff640 1 -- 172.21.15.31:0/3147212690 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55854ac94990 con 0x55854a93ef60 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5737fff640 10 client.17919 handle_command_reply: tid=0 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 client.17919 shutdown 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 -- 172.21.15.31:0/3147212690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 msgr2=0x55854ad338f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad338f0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f57500138e0 tx=0x7f5750004ab0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 -- 172.21.15.31:0/3147212690 shutdown_connections 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad338f0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a95f6b0 0x55854ad2c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55854a93ef60 0x55854ad3a1e0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 --2- 172.21.15.31:0/3147212690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 0x55854ad2c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5737fff640 0 client.17919 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 -- 172.21.15.31:0/3147212690 >> 172.21.15.31:0/3147212690 conn(0x55854a9744e0 msgr2=0x55854ad1d040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.597 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 -- 172.21.15.31:0/3147212690 shutdown_connections 2024-04-25T16:13:01.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.585+0000 7f5765ff5740 1 -- 172.21.15.31:0/3147212690 wait complete. 2024-04-25T16:13:01.598 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.586+0000 7f5765ff5740 20 client.17919 trim_cache size 0 max 16384 2024-04-25T16:13:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.599+0000 7f5765ff5740 1 Processor -- start 2024-04-25T16:13:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.599+0000 7f5765ff5740 1 -- start start 2024-04-25T16:13:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.599+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 0x55854ad3e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.606 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.599+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad33810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.599+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a980390 0x55854ad3eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.599+0000 7f5765ff5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f57581ab270 con 0x55854a95f6b0 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.599+0000 7f5765ff5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57581acde0 con 0x55854a980390 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.599+0000 7f5765ff5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57581ab940 con 0x55854a9574d0 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f5757fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad33810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f5757fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad33810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57452/0 (socket says 172.21.15.31:57452) 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f5757fff640 1 -- 172.21.15.31:0/1237988593 learned_addr learned my addr 172.21.15.31:0/1237988593 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f575cedb640 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 0x55854ad3e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f575d6dc640 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a980390 0x55854ad3eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f5757fff640 1 -- 172.21.15.31:0/1237988593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 msgr2=0x55854ad3e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f5757fff640 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 0x55854ad3e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f5757fff640 1 -- 172.21.15.31:0/1237988593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a980390 msgr2=0x55854ad3eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f5757fff640 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a980390 0x55854ad3eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.607 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f5757fff640 1 -- 172.21.15.31:0/1237988593 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55854ad27c80 con 0x55854a95f6b0 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f575cedb640 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 0x55854ad3e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f575d6dc640 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a980390 0x55854ad3eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f5757fff640 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad33810 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f57440a6c00 tx=0x7f57440a7370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3254ea1c0091a50 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f57427fc640 1 -- 172.21.15.31:0/1237988593 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440a65c0 con 0x55854a95f6b0 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f57427fc640 1 -- 172.21.15.31:0/1237988593 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57440a6780 con 0x55854a95f6b0 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.600+0000 7f57427fc640 1 -- 172.21.15.31:0/1237988593 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57440afad0 con 0x55854a95f6b0 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 -- 172.21.15.31:0/1237988593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 msgr2=0x55854ad33810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad33810 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f57440a6c00 tx=0x7f57440a7370 comp rx=0 tx=0).stop 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 -- 172.21.15.31:0/1237988593 shutdown_connections 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a980390 0x55854ad3eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a95f6b0 0x55854ad33810 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 --2- 172.21.15.31:0/1237988593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a9574d0 0x55854ad3e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 -- 172.21.15.31:0/1237988593 >> 172.21.15.31:0/1237988593 conn(0x55854a9744e0 msgr2=0x55854ad22790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.608 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 -- 172.21.15.31:0/1237988593 shutdown_connections 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 -- 172.21.15.31:0/1237988593 wait complete. 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 -- start start 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.601+0000 7f5765ff5740 1 Processor -- start 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad31040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a95f6b0 0x55854ad31580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f5765ff5740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad324b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f5765ff5740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f57581ac710 con 0x55854a980390 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f5765ff5740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f57581ac010 con 0x55854a9574d0 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f5765ff5740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f57581a9250 con 0x55854a95f6b0 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f5765ff5740 10 client.0 _ll_register_callbacks cb 0x7f5750013560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575d6dc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad324b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575d6dc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad324b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57468/0 (socket says 172.21.15.31:57468) 2024-04-25T16:13:01.609 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575d6dc640 1 -- 172.21.15.31:0/2526149896 learned_addr learned my addr 172.21.15.31:0/2526149896 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575cedb640 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad31040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f5757fff640 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a95f6b0 0x55854ad31580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575d6dc640 1 -- 172.21.15.31:0/2526149896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a95f6b0 msgr2=0x55854ad31580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575d6dc640 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a95f6b0 0x55854ad31580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575d6dc640 1 -- 172.21.15.31:0/2526149896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 msgr2=0x55854ad31040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575d6dc640 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad31040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575d6dc640 1 -- 172.21.15.31:0/2526149896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55854ad33d50 con 0x55854a980390 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575cedb640 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad31040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f575d6dc640 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad324b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5750009470 tx=0x7f5750012e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34fe8be7932ca113 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.602+0000 7f5737fff640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5737fff640 1 -- 172.21.15.31:0/2526149896 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5750013e60 con 0x55854a980390 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5737fff640 1 -- 172.21.15.31:0/2526149896 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5750028610 con 0x55854a980390 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5765ff5740 1 -- 172.21.15.31:0/2526149896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57440a6780 con 0x55854a980390 2024-04-25T16:13:01.610 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5737fff640 1 -- 172.21.15.31:0/2526149896 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f575001f690 con 0x55854a980390 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5765ff5740 10 client.17925 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5765ff5740 1 -- 172.21.15.31:0/2526149896 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55854ad33b60 con 0x55854a980390 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5737fff640 1 -- 172.21.15.31:0/2526149896 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f57440a6780 con 0x55854a980390 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5737fff640 1 -- 172.21.15.31:0/2526149896 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f575003a020 con 0x55854a980390 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5765ff5740 10 client.17925 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5765ff5740 10 client.17925 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5765ff5740 20 client.17925 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5765ff5740 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55854a9143a0 0x55854ada4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5765ff5740 4 client.17925 mds_command: new command op to 24521 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f5765ff5740 1 -- 172.21.15.31:0/2526149896 --> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55854ac8e800 con 0x55854a9143a0 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.603+0000 7f575cedb640 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55854a9143a0 0x55854ada4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5737fff640 10 client.17925 ms_handle_connect on v2:172.21.15.63:6836/546178402 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f575cedb640 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55854a9143a0 0x55854ada4dc0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b17f9a7d9af3a409 server_cookie=ba3cbcdaea357487 in_seq=0 out_seq=0 2024-04-25T16:13:01.611 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5737fff640 1 -- 172.21.15.31:0/2526149896 <== mds.3 v2:172.21.15.63:6836/546178402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55854ac8e800 con 0x55854a9143a0 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5737fff640 10 client.17925 handle_command_reply: tid=0 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 client.17925 shutdown 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 -- 172.21.15.31:0/2526149896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 msgr2=0x55854ad324b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad324b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f5750009470 tx=0x7f5750012e10 comp rx=0 tx=0).stop 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 -- 172.21.15.31:0/2526149896 shutdown_connections 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55854a980390 0x55854ad324b0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.63:6836/546178402,v1:172.21.15.63:6837/546178402] conn(0x55854a9143a0 0x55854ada4dc0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55854a95f6b0 0x55854ad31580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 --2- 172.21.15.31:0/2526149896 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55854a9574d0 0x55854ad31040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 -- 172.21.15.31:0/2526149896 >> 172.21.15.31:0/2526149896 conn(0x55854a9744e0 msgr2=0x55854ad22790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5737fff640 0 client.17925 ms_handle_reset on v2:172.21.15.63:6836/546178402 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.604+0000 7f5765ff5740 1 -- 172.21.15.31:0/2526149896 shutdown_connections 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f5765ff5740 1 -- 172.21.15.31:0/2526149896 wait complete. 2024-04-25T16:13:01.612 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f5765ff5740 20 client.17925 trim_cache size 0 max 16384 2024-04-25T16:13:01.613 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 -- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57480afaf0 msgr2=0x7f57480b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57480afaf0 0x7f57480b2000 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x55854ad23860 tx=0x7f5750004a40 comp rx=0 tx=0).stop 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 -- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758103c40 msgr2=0x7f575819a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758103c40 0x7f575819a770 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f574400d770 tx=0x7f574400dc30 comp rx=0 tx=0).stop 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 -- 172.21.15.31:0/2818891065 shutdown_connections 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f57480afaf0 0x7f57480b2000 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5758108940 0x7f57581a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f5758103c40 0x7f575819a770 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 --2- 172.21.15.31:0/2818891065 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5758102a00 0x7f575819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 -- 172.21.15.31:0/2818891065 >> 172.21.15.31:0/2818891065 conn(0x7f57580fdbf0 msgr2=0x7f5758109e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 -- 172.21.15.31:0/2818891065 shutdown_connections 2024-04-25T16:13:01.614 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.605+0000 7f575f13f640 1 -- 172.21.15.31:0/2818891065 wait complete. 2024-04-25T16:13:01.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:01 smithi031 ceph-mon[29140]: pgmap v702: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 118 MiB/s wr, 245 op/s 2024-04-25T16:13:01.617 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:01 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/589456185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:13:01.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-25T16:13:01.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:01 smithi063 ceph-mon[32413]: pgmap v702: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 118 MiB/s wr, 245 op/s 2024-04-25T16:13:01.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:01 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/589456185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:13:01.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:01 smithi186 ceph-mon[33258]: pgmap v702: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 118 MiB/s wr, 245 op/s 2024-04-25T16:13:01.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:01 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/589456185' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:13:01.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 -- 172.21.15.31:0/3019496258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 msgr2=0x7f1da8103ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/3019496258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 0x7f1da8103ff0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f1d90009900 tx=0x7f1d9002a410 comp rx=0 tx=0).stop 2024-04-25T16:13:01.732 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 -- 172.21.15.31:0/3019496258 shutdown_connections 2024-04-25T16:13:01.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/3019496258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1da8104680 0x7f1da810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/3019496258 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 0x7f1da8103ff0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/3019496258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1da8102fc0 0x7f1da81033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 -- 172.21.15.31:0/3019496258 >> 172.21.15.31:0/3019496258 conn(0x7f1da80fd340 msgr2=0x7f1da80ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 -- 172.21.15.31:0/3019496258 shutdown_connections 2024-04-25T16:13:01.733 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 -- 172.21.15.31:0/3019496258 wait complete. 2024-04-25T16:13:01.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.731+0000 7f1dad8d8640 1 Processor -- start 2024-04-25T16:13:01.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1dad8d8640 1 -- start start 2024-04-25T16:13:01.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1dad8d8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1da8102fc0 0x7f1da8102280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.735 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1dad8d8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 0x7f1da819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1dad8d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1da819f330 0x7f1da81a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1dad8d8640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1da810e750 con 0x7f1da8103b20 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1dad8d8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1da810e5d0 con 0x7f1da819f330 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1dad8d8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1da810e8d0 con 0x7f1da8102fc0 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 0x7f1da819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 0x7f1da819edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57484/0 (socket says 172.21.15.31:57484) 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da67fc640 1 -- 172.21.15.31:0/46540067 learned_addr learned my addr 172.21.15.31:0/46540067 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da77fe640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1da819f330 0x7f1da81a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1da8102fc0 0x7f1da8102280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da67fc640 1 -- 172.21.15.31:0/46540067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1da8102fc0 msgr2=0x7f1da8102280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da67fc640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1da8102fc0 0x7f1da8102280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da67fc640 1 -- 172.21.15.31:0/46540067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1da819f330 msgr2=0x7f1da81a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da67fc640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1da819f330 0x7f1da81a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da67fc640 1 -- 172.21.15.31:0/46540067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da81a1f90 con 0x7f1da8103b20 2024-04-25T16:13:01.736 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.732+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1da8102fc0 0x7f1da8102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:01.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.733+0000 7f1da67fc640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 0x7f1da819edf0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1d90006010 tx=0x7f1d90031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.733+0000 7f1dac8d6640 1 -- 172.21.15.31:0/46540067 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9003f030 con 0x7f1da8103b20 2024-04-25T16:13:01.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.733+0000 7f1dac8d6640 1 -- 172.21.15.31:0/46540067 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d90003aa0 con 0x7f1da8103b20 2024-04-25T16:13:01.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.733+0000 7f1dad8d8640 1 -- 172.21.15.31:0/46540067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da81a2220 con 0x7f1da8103b20 2024-04-25T16:13:01.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.733+0000 7f1dac8d6640 1 -- 172.21.15.31:0/46540067 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d90035070 con 0x7f1da8103b20 2024-04-25T16:13:01.737 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.733+0000 7f1dad8d8640 1 -- 172.21.15.31:0/46540067 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da81a2710 con 0x7f1da8103b20 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.735+0000 7f1db4785740 1 Processor -- start 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1db4785740 1 -- start start 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1dac8d6640 1 -- 172.21.15.31:0/46540067 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1d90042030 con 0x7f1da8103b20 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1db4785740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37c0390 0x55fde3b62810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1db4785740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37974d0 0x55fde3b62d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1db4785740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b63a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1db4785740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fde3ac6d80 con 0x55fde379f6b0 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1db4785740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fde3a80a20 con 0x55fde37c0390 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1db4785740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fde3ac2d40 con 0x55fde37974d0 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b63a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37c0390 0x55fde3b62810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.744 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b63a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57486/0 (socket says 172.21.15.31:57486) 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37c0390 0x55fde3b62810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:36666/0 (socket says 172.21.15.31:36666) 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da67fc640 1 -- 172.21.15.31:0/1910318235 learned_addr learned my addr 172.21.15.31:0/1910318235 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/1910318235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37974d0 0x55fde3b62d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da67fc640 1 -- 172.21.15.31:0/1910318235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37974d0 msgr2=0x55fde3b62d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da67fc640 1 --2- 172.21.15.31:0/1910318235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37974d0 0x55fde3b62d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da67fc640 1 -- 172.21.15.31:0/1910318235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37c0390 msgr2=0x55fde3b62810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da67fc640 1 --2- 172.21.15.31:0/1910318235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37c0390 0x55fde3b62810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da67fc640 1 -- 172.21.15.31:0/1910318235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde3a8bc30 con 0x55fde379f6b0 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.736+0000 7f1da77fe640 1 --2- 172.21.15.31:0/1910318235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37c0390 0x55fde3b62810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.737+0000 7f1da67fc640 1 --2- 172.21.15.31:0/1910318235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b63a20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1d901074b0 tx=0x7f1d901075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f10358d46184b83 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.745 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.737+0000 7f1dac8d6640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d940afaf0 0x7f1d940b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.737+0000 7f1da77fe640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d940afaf0 0x7f1d940b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.737+0000 7f1d9dffb640 1 -- 172.21.15.31:0/1910318235 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d900c35e0 con 0x55fde379f6b0 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.737+0000 7f1dac8d6640 1 -- 172.21.15.31:0/46540067 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1d900ff290 con 0x7f1da8103b20 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.737+0000 7f1d9dffb640 1 -- 172.21.15.31:0/1910318235 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d900cee50 con 0x55fde379f6b0 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.737+0000 7f1d9dffb640 1 -- 172.21.15.31:0/1910318235 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d900c6230 con 0x55fde379f6b0 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 -- 172.21.15.31:0/1910318235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 msgr2=0x55fde3b63a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1da77fe640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d940afaf0 0x7f1d940b2000 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x55fde3b63860 tx=0x7f1d98009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 --2- 172.21.15.31:0/1910318235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b63a20 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f1d901074b0 tx=0x7f1d901075a0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 -- 172.21.15.31:0/1910318235 shutdown_connections 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 --2- 172.21.15.31:0/1910318235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b63a20 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 --2- 172.21.15.31:0/1910318235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37974d0 0x55fde3b62d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 --2- 172.21.15.31:0/1910318235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37c0390 0x55fde3b62810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 -- 172.21.15.31:0/1910318235 >> 172.21.15.31:0/1910318235 conn(0x55fde37b44e0 msgr2=0x55fde3b5e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.746 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 -- 172.21.15.31:0/1910318235 shutdown_connections 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 -- 172.21.15.31:0/1910318235 wait complete. 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 -- start start 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.738+0000 7f1db4785740 1 Processor -- start 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1db4785740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 0x55fde3b6c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1db4785740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b6c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1db4785740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b736f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1db4785740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55fde3b642f0 con 0x55fde379f6b0 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1db4785740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55fde3a4ddc0 con 0x55fde37974d0 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1db4785740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55fde3b64470 con 0x55fde37c0390 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1db4785740 10 client.0 _ll_register_callbacks cb 0x55fde3a69600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b6c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b6c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57500/0 (socket says 172.21.15.31:57500) 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da67fc640 1 -- 172.21.15.31:0/1876358890 learned_addr learned my addr 172.21.15.31:0/1876358890 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.747 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 0x55fde3b6c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da77fe640 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b736f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da67fc640 1 -- 172.21.15.31:0/1876358890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 msgr2=0x55fde3b736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da67fc640 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b736f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da67fc640 1 -- 172.21.15.31:0/1876358890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 msgr2=0x55fde3b6c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da67fc640 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 0x55fde3b6c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da67fc640 1 -- 172.21.15.31:0/1876358890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde37d83f0 con 0x55fde379f6b0 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da77fe640 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1da67fc640 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b6c790 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1d90107480 tx=0x7f1d900ff510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ddae45444400b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1d86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1d86ffd640 1 -- 172.21.15.31:0/1876358890 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d90049070 con 0x55fde379f6b0 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1d86ffd640 1 -- 172.21.15.31:0/1876358890 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1d9002f900 con 0x55fde379f6b0 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.739+0000 7f1db4785740 1 -- 172.21.15.31:0/1876358890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d900cee50 con 0x55fde379f6b0 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1d86ffd640 1 -- 172.21.15.31:0/1876358890 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1d900c6e90 con 0x55fde379f6b0 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1db4785740 10 client.17937 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:01.748 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1db4785740 1 -- 172.21.15.31:0/1876358890 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fde38a27d0 con 0x55fde379f6b0 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1d86ffd640 1 -- 172.21.15.31:0/1876358890 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1d900cee50 con 0x55fde379f6b0 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1d86ffd640 1 -- 172.21.15.31:0/1876358890 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f1d9002fd40 con 0x55fde379f6b0 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1db4785740 10 client.17937 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1db4785740 10 client.17937 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1db4785740 20 client.17937 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1db4785740 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fde377ef60 0x55fde3b79ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1db4785740 4 client.17937 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1db4785740 1 -- 172.21.15.31:0/1876358890 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fde3ad4990 con 0x55fde377ef60 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.740+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fde377ef60 0x55fde3b79ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.741+0000 7f1d86ffd640 10 client.17937 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.741+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fde377ef60 0x55fde3b79ca0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4c757f273a090b4c server_cookie=9437f123f89383fc in_seq=0 out_seq=0 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1d86ffd640 1 -- 172.21.15.31:0/1876358890 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fde3ad4990 con 0x55fde377ef60 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1d86ffd640 10 client.17937 handle_command_reply: tid=0 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 client.17937 shutdown 2024-04-25T16:13:01.749 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 -- 172.21.15.31:0/1876358890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 msgr2=0x55fde3b6c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b6c790 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f1d90107480 tx=0x7f1d900ff510 comp rx=0 tx=0).stop 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 -- 172.21.15.31:0/1876358890 shutdown_connections 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b736f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b6c790 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fde377ef60 0x55fde3b79ca0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 --2- 172.21.15.31:0/1876358890 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 0x55fde3b6c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 -- 172.21.15.31:0/1876358890 >> 172.21.15.31:0/1876358890 conn(0x55fde37b44e0 msgr2=0x55fde3b60f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1d86ffd640 0 client.17937 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 -- 172.21.15.31:0/1876358890 shutdown_connections 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 1 -- 172.21.15.31:0/1876358890 wait complete. 2024-04-25T16:13:01.750 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.742+0000 7f1db4785740 20 client.17937 trim_cache size 0 max 16384 2024-04-25T16:13:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1db4785740 1 Processor -- start 2024-04-25T16:13:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1db4785740 1 -- start start 2024-04-25T16:13:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1db4785740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde37974d0 0x55fde3b5da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1db4785740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde379f6b0 0x55fde3b73610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1db4785740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b5df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1db4785740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1da81ab1b0 con 0x55fde37974d0 2024-04-25T16:13:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1db4785740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1da81acd20 con 0x55fde379f6b0 2024-04-25T16:13:01.764 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1db4785740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1da81ab880 con 0x55fde37c0390 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde37974d0 0x55fde3b5da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde37974d0 0x55fde3b5da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57516/0 (socket says 172.21.15.31:57516) 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da6ffd640 1 -- 172.21.15.31:0/639703345 learned_addr learned my addr 172.21.15.31:0/639703345 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da77fe640 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b5df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da6ffd640 1 -- 172.21.15.31:0/639703345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 msgr2=0x55fde3b5df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b5df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da67fc640 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde379f6b0 0x55fde3b73610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da6ffd640 1 -- 172.21.15.31:0/639703345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde379f6b0 msgr2=0x55fde3b73610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde379f6b0 0x55fde3b73610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da6ffd640 1 -- 172.21.15.31:0/639703345 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fde3b67b60 con 0x55fde37974d0 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.756+0000 7f1da67fc640 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde379f6b0 0x55fde3b73610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde37974d0 0x55fde3b5da50 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x55fde3b62530 tx=0x7f1da0013b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7a15d45a76291c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1d9d7fa640 1 -- 172.21.15.31:0/639703345 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0005020 con 0x55fde37974d0 2024-04-25T16:13:01.765 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1d9d7fa640 1 -- 172.21.15.31:0/639703345 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da001d430 con 0x55fde37974d0 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1d9d7fa640 1 -- 172.21.15.31:0/639703345 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0004a90 con 0x55fde37974d0 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1db4785740 1 -- 172.21.15.31:0/639703345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde37974d0 msgr2=0x55fde3b5da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1db4785740 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde37974d0 0x55fde3b5da50 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x55fde3b62530 tx=0x7f1da0013b00 comp rx=0 tx=0).stop 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1db4785740 1 -- 172.21.15.31:0/639703345 shutdown_connections 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1db4785740 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b5df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1db4785740 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde379f6b0 0x55fde3b73610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1db4785740 1 --2- 172.21.15.31:0/639703345 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde37974d0 0x55fde3b5da50 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.757+0000 7f1db4785740 1 -- 172.21.15.31:0/639703345 >> 172.21.15.31:0/639703345 conn(0x55fde37b44e0 msgr2=0x55fde3b6a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 -- 172.21.15.31:0/639703345 shutdown_connections 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 -- 172.21.15.31:0/639703345 wait complete. 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 -- start start 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 Processor -- start 2024-04-25T16:13:01.766 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 0x55fde3b70bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b71110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b71fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1da81ac650 con 0x55fde379f6b0 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1da81abf50 con 0x55fde37974d0 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1da81a9190 con 0x55fde37c0390 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1db4785740 10 client.0 _ll_register_callbacks cb 0x7f1d98009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.758+0000 7f1da67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b71110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da67fc640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b71110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57526/0 (socket says 172.21.15.31:57526) 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da67fc640 1 -- 172.21.15.31:0/3284556894 learned_addr learned my addr 172.21.15.31:0/3284556894 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 0x55fde3b70bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da77fe640 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b71fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da67fc640 1 -- 172.21.15.31:0/3284556894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 msgr2=0x55fde3b71fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da67fc640 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b71fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da67fc640 1 -- 172.21.15.31:0/3284556894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 msgr2=0x55fde3b70bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da67fc640 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 0x55fde3b70bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da67fc640 1 -- 172.21.15.31:0/3284556894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fde3b73b50 con 0x55fde379f6b0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1da67fc640 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b71110 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1d901074b0 tx=0x7f1d90005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67aa5640a161a2b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1d86ffd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1d86ffd640 1 -- 172.21.15.31:0/3284556894 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d90049070 con 0x55fde379f6b0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1d86ffd640 1 -- 172.21.15.31:0/3284556894 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1d900b0070 con 0x55fde379f6b0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1db4785740 1 -- 172.21.15.31:0/3284556894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1da001d430 con 0x55fde379f6b0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1d86ffd640 1 -- 172.21.15.31:0/3284556894 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1d900c7070 con 0x55fde379f6b0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1db4785740 10 client.17943 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.759+0000 7f1db4785740 1 -- 172.21.15.31:0/3284556894 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55fde3b73620 con 0x55fde379f6b0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1d86ffd640 1 -- 172.21.15.31:0/3284556894 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1da001d430 con 0x55fde379f6b0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1d86ffd640 1 -- 172.21.15.31:0/3284556894 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f1d9002f300 con 0x55fde379f6b0 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1db4785740 10 client.17943 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1db4785740 10 client.17943 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-04-25T16:13:01.768 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1db4785740 20 client.17943 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1db4785740 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fde37543a0 0x55fde3be4130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1db4785740 4 client.17943 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1db4785740 1 -- 172.21.15.31:0/3284556894 --> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fde3ace800 con 0x55fde37543a0 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fde37543a0 0x55fde3be4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1d86ffd640 10 client.17943 ms_handle_connect on v2:172.21.15.186:6836/509868288 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.760+0000 7f1da6ffd640 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fde37543a0 0x55fde3be4130 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2a3d292001b9f1e0 server_cookie=5cad19785e136bfe in_seq=0 out_seq=0 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1d86ffd640 1 -- 172.21.15.31:0/3284556894 <== mds.2 v2:172.21.15.186:6836/509868288 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fde3ace800 con 0x55fde37543a0 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1d86ffd640 10 client.17943 handle_command_reply: tid=0 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 client.17943 shutdown 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 -- 172.21.15.31:0/3284556894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 msgr2=0x55fde3b71110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b71110 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f1d901074b0 tx=0x7f1d90005c90 comp rx=0 tx=0).stop 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 -- 172.21.15.31:0/3284556894 shutdown_connections 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55fde37c0390 0x55fde3b71fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55fde379f6b0 0x55fde3b71110 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.186:6836/509868288,v1:172.21.15.186:6837/509868288] conn(0x55fde37543a0 0x55fde3be4130 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 --2- 172.21.15.31:0/3284556894 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55fde37974d0 0x55fde3b70bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1d86ffd640 0 client.17943 ms_handle_reset on v2:172.21.15.186:6836/509868288 2024-04-25T16:13:01.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 -- 172.21.15.31:0/3284556894 >> 172.21.15.31:0/3284556894 conn(0x55fde37b44e0 msgr2=0x55fde3b60e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 -- 172.21.15.31:0/3284556894 shutdown_connections 2024-04-25T16:13:01.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 1 -- 172.21.15.31:0/3284556894 wait complete. 2024-04-25T16:13:01.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1db4785740 20 client.17943 trim_cache size 0 max 16384 2024-04-25T16:13:01.770 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:13:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1dad8d8640 1 -- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d940afaf0 msgr2=0x7f1d940b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.761+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d940afaf0 0x7f1d940b2000 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x55fde3b63860 tx=0x7f1d98009500 comp rx=0 tx=0).stop 2024-04-25T16:13:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 -- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 msgr2=0x7f1da819edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 0x7f1da819edf0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f1d90006010 tx=0x7f1d90031a50 comp rx=0 tx=0).stop 2024-04-25T16:13:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 -- 172.21.15.31:0/46540067 shutdown_connections 2024-04-25T16:13:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1d940afaf0 0x7f1d940b2000 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1da819f330 0x7f1da81a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1da8103b20 0x7f1da819edf0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 --2- 172.21.15.31:0/46540067 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1da8102fc0 0x7f1da8102280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 -- 172.21.15.31:0/46540067 >> 172.21.15.31:0/46540067 conn(0x7f1da80fd340 msgr2=0x7f1da80ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 -- 172.21.15.31:0/46540067 shutdown_connections 2024-04-25T16:13:01.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.762+0000 7f1dad8d8640 1 -- 172.21.15.31:0/46540067 wait complete. 2024-04-25T16:13:01.775 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-25T16:13:01.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 -- 172.21.15.31:0/299722524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 msgr2=0x7f52b0101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 --2- 172.21.15.31:0/299722524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 0x7f52b0101bc0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f5298009900 tx=0x7f529802a410 comp rx=0 tx=0).stop 2024-04-25T16:13:01.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 -- 172.21.15.31:0/299722524 shutdown_connections 2024-04-25T16:13:01.892 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 --2- 172.21.15.31:0/299722524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b0102250 0x7f52b010bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 --2- 172.21.15.31:0/299722524 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 0x7f52b0101bc0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 --2- 172.21.15.31:0/299722524 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52b01050a0 0x7f52b01054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 -- 172.21.15.31:0/299722524 >> 172.21.15.31:0/299722524 conn(0x7f52b00fd350 msgr2=0x7f52b00ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 -- 172.21.15.31:0/299722524 shutdown_connections 2024-04-25T16:13:01.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 -- 172.21.15.31:0/299722524 wait complete. 2024-04-25T16:13:01.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 Processor -- start 2024-04-25T16:13:01.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.888+0000 7f52b5be3640 1 -- start start 2024-04-25T16:13:01.893 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52b5be3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 0x7f52b019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52b5be3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52b0102250 0x7f52b019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52b5be3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01050a0 0x7f52b01a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52b5be3640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f52b010f610 con 0x7f52b0101750 2024-04-25T16:13:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52b5be3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f52b010f490 con 0x7f52b0102250 2024-04-25T16:13:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52b5be3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52b010f790 con 0x7f52b01050a0 2024-04-25T16:13:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 0x7f52b019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.894 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 0x7f52b019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57538/0 (socket says 172.21.15.31:57538) 2024-04-25T16:13:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52af7fe640 1 -- 172.21.15.31:0/2225111142 learned_addr learned my addr 172.21.15.31:0/2225111142 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52aeffd640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52b0102250 0x7f52b019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52affff640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01050a0 0x7f52b01a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52af7fe640 1 -- 172.21.15.31:0/2225111142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01050a0 msgr2=0x7f52b01a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52af7fe640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01050a0 0x7f52b01a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52af7fe640 1 -- 172.21.15.31:0/2225111142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52b0102250 msgr2=0x7f52b019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52af7fe640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52b0102250 0x7f52b019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.895 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52af7fe640 1 -- 172.21.15.31:0/2225111142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b01a1d80 con 0x7f52b0101750 2024-04-25T16:13:01.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52af7fe640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 0x7f52b019a240 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f52a000e9f0 tx=0x7f52a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.889+0000 7f52acff9640 1 -- 172.21.15.31:0/2225111142 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a000ccc0 con 0x7f52b0101750 2024-04-25T16:13:01.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.890+0000 7f52acff9640 1 -- 172.21.15.31:0/2225111142 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52a00035f0 con 0x7f52b0101750 2024-04-25T16:13:01.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.890+0000 7f52b5be3640 1 -- 172.21.15.31:0/2225111142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b01a2070 con 0x7f52b0101750 2024-04-25T16:13:01.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.890+0000 7f52acff9640 1 -- 172.21.15.31:0/2225111142 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a000ce60 con 0x7f52b0101750 2024-04-25T16:13:01.896 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.890+0000 7f52b5be3640 1 -- 172.21.15.31:0/2225111142 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b01a25c0 con 0x7f52b0101750 2024-04-25T16:13:01.906 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52bcb6f740 1 Processor -- start 2024-04-25T16:13:01.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52bcb6f740 1 -- start start 2024-04-25T16:13:01.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e60810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a954d0 0x55ee23e60d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52acff9640 1 -- 172.21.15.31:0/2225111142 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f52a0003aa0 con 0x7f52b0101750 2024-04-25T16:13:01.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a9d6b0 0x55ee23e61a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ee23dc4d80 con 0x55ee23a954d0 2024-04-25T16:13:01.907 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ee23d7ea20 con 0x55ee23abe390 2024-04-25T16:13:01.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ee23dc0d40 con 0x55ee23a9d6b0 2024-04-25T16:13:01.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a954d0 0x55ee23e60d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52af7fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a954d0 0x55ee23e60d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57540/0 (socket says 172.21.15.31:57540) 2024-04-25T16:13:01.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52af7fe640 1 -- 172.21.15.31:0/1514635627 learned_addr learned my addr 172.21.15.31:0/1514635627 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52affff640 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e60810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52aeffd640 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a9d6b0 0x55ee23e61a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52af7fe640 1 -- 172.21.15.31:0/1514635627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a9d6b0 msgr2=0x55ee23e61a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.908 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52af7fe640 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a9d6b0 0x55ee23e61a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.893+0000 7f52af7fe640 1 -- 172.21.15.31:0/1514635627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 msgr2=0x55ee23e60810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52af7fe640 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e60810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52af7fe640 1 -- 172.21.15.31:0/1514635627 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee23d89c30 con 0x55ee23a954d0 2024-04-25T16:13:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52affff640 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e60810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52acff9640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f529c0afaf0 0x7f529c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52af7fe640 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a954d0 0x55ee23e60d90 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f52a00e2f30 tx=0x7f52a00e71a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50d617336bc368e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52affff640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f529c0afaf0 0x7f529c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52acff9640 1 -- 172.21.15.31:0/2225111142 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f52a00defb0 con 0x7f52b0101750 2024-04-25T16:13:01.909 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52a5ffb640 1 -- 172.21.15.31:0/1514635627 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a00a3360 con 0x55ee23a954d0 2024-04-25T16:13:01.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52a5ffb640 1 -- 172.21.15.31:0/1514635627 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52a00a6fb0 con 0x55ee23a954d0 2024-04-25T16:13:01.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.894+0000 7f52a5ffb640 1 -- 172.21.15.31:0/1514635627 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a00a6c10 con 0x55ee23a954d0 2024-04-25T16:13:01.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1514635627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a954d0 msgr2=0x55ee23e60d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52affff640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f529c0afaf0 0x7f529c0b2000 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x55ee23e61860 tx=0x7f52a8008860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a954d0 0x55ee23e60d90 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f52a00e2f30 tx=0x7f52a00e71a0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1514635627 shutdown_connections 2024-04-25T16:13:01.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a9d6b0 0x55ee23e61a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.910 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a954d0 0x55ee23e60d90 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1514635627 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e60810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1514635627 >> 172.21.15.31:0/1514635627 conn(0x55ee23ab24e0 msgr2=0x55ee23e5c020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1514635627 shutdown_connections 2024-04-25T16:13:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1514635627 wait complete. 2024-04-25T16:13:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.895+0000 7f52bcb6f740 1 -- start start 2024-04-25T16:13:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52bcb6f740 1 Processor -- start 2024-04-25T16:13:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e6a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.911 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e71960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x55ee23e624d0 con 0x55ee23a9d6b0 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x55ee23d4bdc0 con 0x55ee23abe390 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ee23e62650 con 0x55ee23a954d0 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52bcb6f740 10 client.0 _ll_register_callbacks cb 0x55ee23d67600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57550/0 (socket says 172.21.15.31:57550) 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52aeffd640 1 -- 172.21.15.31:0/2463084873 learned_addr learned my addr 172.21.15.31:0/2463084873 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52af7fe640 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e6a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52affff640 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e71960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52aeffd640 1 -- 172.21.15.31:0/2463084873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 msgr2=0x55ee23e6a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52aeffd640 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e6a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.912 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52aeffd640 1 -- 172.21.15.31:0/2463084873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 msgr2=0x55ee23e71960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52aeffd640 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e71960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.896+0000 7f52aeffd640 1 -- 172.21.15.31:0/2463084873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee23ad63f0 con 0x55ee23a9d6b0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f52aeffd640 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6aa70 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f5298009900 tx=0x7f5298031e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a9b21899305c74 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f528effd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f528effd640 1 -- 172.21.15.31:0/2463084873 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298035070 con 0x55ee23a9d6b0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f528effd640 1 -- 172.21.15.31:0/2463084873 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5298004040 con 0x55ee23a9d6b0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f52bcb6f740 1 -- 172.21.15.31:0/2463084873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52a00a6fb0 con 0x55ee23a9d6b0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f528effd640 1 -- 172.21.15.31:0/2463084873 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5298004240 con 0x55ee23a9d6b0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f52bcb6f740 10 client.17955 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f52bcb6f740 1 -- 172.21.15.31:0/2463084873 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ee23ba07d0 con 0x55ee23a9d6b0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f528effd640 1 -- 172.21.15.31:0/2463084873 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f52a00a6fb0 con 0x55ee23a9d6b0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f528effd640 1 -- 172.21.15.31:0/2463084873 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f5298043020 con 0x55ee23a9d6b0 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f52bcb6f740 10 client.17955 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:01.913 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f52bcb6f740 10 client.17955 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.897+0000 7f52bcb6f740 20 client.17955 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.898+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ee23a7cf60 0x55ee23e78250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.898+0000 7f52bcb6f740 4 client.17955 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.898+0000 7f52bcb6f740 1 -- 172.21.15.31:0/2463084873 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee23dd2990 con 0x55ee23a7cf60 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.898+0000 7f52af7fe640 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ee23a7cf60 0x55ee23e78250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.898+0000 7f528effd640 10 client.17955 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.898+0000 7f52af7fe640 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ee23a7cf60 0x55ee23e78250 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8164aefbeca79777 server_cookie=3ce156c3e88a2267 in_seq=0 out_seq=0 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.899+0000 7f528effd640 1 -- 172.21.15.31:0/2463084873 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ee23dd2990 con 0x55ee23a7cf60 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.899+0000 7f528effd640 10 client.17955 handle_command_reply: tid=0 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.899+0000 7f52bcb6f740 1 client.17955 shutdown 2024-04-25T16:13:01.914 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.899+0000 7f52bcb6f740 1 -- 172.21.15.31:0/2463084873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 msgr2=0x55ee23e6aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.899+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6aa70 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f5298009900 tx=0x7f5298031e60 comp rx=0 tx=0).stop 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.899+0000 7f52bcb6f740 1 -- 172.21.15.31:0/2463084873 shutdown_connections 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.899+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e71960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.899+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ee23a7cf60 0x55ee23e78250 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.900+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6aa70 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.900+0000 7f528effd640 0 client.17955 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.900+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/2463084873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e6a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.900+0000 7f52bcb6f740 1 -- 172.21.15.31:0/2463084873 >> 172.21.15.31:0/2463084873 conn(0x55ee23ab24e0 msgr2=0x55ee23e5b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.900+0000 7f52bcb6f740 1 -- 172.21.15.31:0/2463084873 shutdown_connections 2024-04-25T16:13:01.915 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.900+0000 7f52bcb6f740 1 -- 172.21.15.31:0/2463084873 wait complete. 2024-04-25T16:13:01.916 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.900+0000 7f52bcb6f740 20 client.17955 trim_cache size 0 max 16384 2024-04-25T16:13:01.922 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.913+0000 7f52bcb6f740 1 Processor -- start 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.913+0000 7f52bcb6f740 1 -- start start 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.913+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e68cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e691f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e69730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f52b01ab2a0 con 0x55ee23a9d6b0 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f52b01ace10 con 0x55ee23abe390 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52b01ab970 con 0x55ee23a954d0 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e691f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52aeffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e691f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57566/0 (socket says 172.21.15.31:57566) 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52aeffd640 1 -- 172.21.15.31:0/4235355941 learned_addr learned my addr 172.21.15.31:0/4235355941 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52affff640 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e69730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52af7fe640 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e68cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52aeffd640 1 -- 172.21.15.31:0/4235355941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 msgr2=0x55ee23e68cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.923 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52aeffd640 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e68cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52aeffd640 1 -- 172.21.15.31:0/4235355941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 msgr2=0x55ee23e69730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52aeffd640 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e69730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52aeffd640 1 -- 172.21.15.31:0/4235355941 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee23e69c70 con 0x55ee23a9d6b0 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52affff640 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e69730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52aeffd640 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e691f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f5298031a00 tx=0x7f529802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b90b57604b13aa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52a57fa640 1 -- 172.21.15.31:0/4235355941 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5298035070 con 0x55ee23a9d6b0 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52a57fa640 1 -- 172.21.15.31:0/4235355941 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5298055070 con 0x55ee23a9d6b0 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.914+0000 7f52a57fa640 1 -- 172.21.15.31:0/4235355941 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529803b430 con 0x55ee23a9d6b0 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 -- 172.21.15.31:0/4235355941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 msgr2=0x55ee23e691f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e691f0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f5298031a00 tx=0x7f529802a410 comp rx=0 tx=0).stop 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 -- 172.21.15.31:0/4235355941 shutdown_connections 2024-04-25T16:13:01.924 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23abe390 0x55ee23e69730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23a9d6b0 0x55ee23e691f0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/4235355941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e68cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 -- 172.21.15.31:0/4235355941 >> 172.21.15.31:0/4235355941 conn(0x55ee23ab24e0 msgr2=0x55ee23e652d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 -- 172.21.15.31:0/4235355941 shutdown_connections 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 -- 172.21.15.31:0/4235355941 wait complete. 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.915+0000 7f52bcb6f740 1 -- start start 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52bcb6f740 1 Processor -- start 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e6e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.925 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52bcb6f740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23abe390 0x55ee23e6f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f52b01ac740 con 0x55ee23abe390 2024-04-25T16:13:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f52b01ac040 con 0x55ee23a9d6b0 2024-04-25T16:13:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52bcb6f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f52b01a9280 con 0x55ee23a954d0 2024-04-25T16:13:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52bcb6f740 10 client.0 _ll_register_callbacks cb 0x7f5298009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23abe390 0x55ee23e6f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52affff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23abe390 0x55ee23e6f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57578/0 (socket says 172.21.15.31:57578) 2024-04-25T16:13:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52affff640 1 -- 172.21.15.31:0/1171423699 learned_addr learned my addr 172.21.15.31:0/1171423699 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:01.926 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52aeffd640 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52af7fe640 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e6e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52affff640 1 -- 172.21.15.31:0/1171423699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 msgr2=0x55ee23e6e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52affff640 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e6e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52affff640 1 -- 172.21.15.31:0/1171423699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23a9d6b0 msgr2=0x55ee23e6e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52affff640 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.916+0000 7f52affff640 1 -- 172.21.15.31:0/1171423699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee23e6fdf0 con 0x55ee23abe390 2024-04-25T16:13:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52affff640 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23abe390 0x55ee23e6f830 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f52a8011db0 tx=0x7f52a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7407abe97765370 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f528effd640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:01.927 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f528effd640 1 -- 172.21.15.31:0/1171423699 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a8020640 con 0x55ee23abe390 2024-04-25T16:13:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f528effd640 1 -- 172.21.15.31:0/1171423699 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f52a80207e0 con 0x55ee23abe390 2024-04-25T16:13:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1171423699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5298055070 con 0x55ee23abe390 2024-04-25T16:13:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f528effd640 1 -- 172.21.15.31:0/1171423699 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f52a80209e0 con 0x55ee23abe390 2024-04-25T16:13:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52bcb6f740 10 client.17961 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1171423699 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55ee23e710b0 con 0x55ee23abe390 2024-04-25T16:13:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f528effd640 1 -- 172.21.15.31:0/1171423699 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5298055070 con 0x55ee23abe390 2024-04-25T16:13:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f528effd640 1 -- 172.21.15.31:0/1171423699 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f52a8028c40 con 0x55ee23abe390 2024-04-25T16:13:01.928 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52bcb6f740 10 client.17961 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:01.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52bcb6f740 10 client.17961 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-04-25T16:13:01.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52bcb6f740 20 client.17961 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:01.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ee23a523a0 0x55ee23ee2c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:01.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52bcb6f740 4 client.17961 mds_command: new command op to 24553 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:13:01.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.917+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1171423699 --> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee23dcc800 con 0x55ee23a523a0 2024-04-25T16:13:01.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.918+0000 7f52af7fe640 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ee23a523a0 0x55ee23ee2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:01.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.918+0000 7f52af7fe640 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ee23a523a0 0x55ee23ee2c60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=edf9da0ca4d1335a server_cookie=8f993ea96d5f3856 in_seq=0 out_seq=0 2024-04-25T16:13:01.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.918+0000 7f528effd640 10 client.17961 ms_handle_connect on v2:172.21.15.186:6838/2846674462 2024-04-25T16:13:01.929 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.918+0000 7f528effd640 1 -- 172.21.15.31:0/1171423699 <== mds.1 v2:172.21.15.186:6838/2846674462 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee23dcc800 con 0x55ee23a523a0 2024-04-25T16:13:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.918+0000 7f528effd640 10 client.17961 handle_command_reply: tid=0 2024-04-25T16:13:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.918+0000 7f52bcb6f740 1 client.17961 shutdown 2024-04-25T16:13:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1171423699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23abe390 msgr2=0x55ee23e6f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23abe390 0x55ee23e6f830 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f52a8011db0 tx=0x7f52a8004a40 comp rx=0 tx=0).stop 2024-04-25T16:13:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1171423699 shutdown_connections 2024-04-25T16:13:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.186:6838/2846674462,v1:172.21.15.186:6839/2846674462] conn(0x55ee23a523a0 0x55ee23ee2c60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x55ee23abe390 0x55ee23e6f830 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f528effd640 0 client.17961 ms_handle_reset on v2:172.21.15.186:6838/2846674462 2024-04-25T16:13:01.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x55ee23a9d6b0 0x55ee23e6e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 --2- 172.21.15.31:0/1171423699 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ee23a954d0 0x55ee23e6e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1171423699 >> 172.21.15.31:0/1171423699 conn(0x55ee23ab24e0 msgr2=0x55ee23e68cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1171423699 shutdown_connections 2024-04-25T16:13:01.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 1 -- 172.21.15.31:0/1171423699 wait complete. 2024-04-25T16:13:01.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52bcb6f740 20 client.17961 trim_cache size 0 max 16384 2024-04-25T16:13:01.932 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:13:01.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 -- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f529c0afaf0 msgr2=0x7f529c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f529c0afaf0 0x7f529c0b2000 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x55ee23e61860 tx=0x7f52a8008860 comp rx=0 tx=0).stop 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 -- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 msgr2=0x7f52b019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 0x7f52b019a240 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f52a000e9f0 tx=0x7f52a000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 -- 172.21.15.31:0/2225111142 shutdown_connections 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f529c0afaf0 0x7f529c0b2000 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01050a0 0x7f52b01a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f52b0102250 0x7f52b019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 --2- 172.21.15.31:0/2225111142 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f52b0101750 0x7f52b019a240 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.919+0000 7f52b5be3640 1 -- 172.21.15.31:0/2225111142 >> 172.21.15.31:0/2225111142 conn(0x7f52b00fd350 msgr2=0x7f52b0105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.920+0000 7f52b5be3640 1 -- 172.21.15.31:0/2225111142 shutdown_connections 2024-04-25T16:13:01.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:01.920+0000 7f52b5be3640 1 -- 172.21.15.31:0/2225111142 wait complete. 2024-04-25T16:13:01.935 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-25T16:13:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.048+0000 7f53732f2640 1 -- 172.21.15.31:0/1215778441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 msgr2=0x7f536c10ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.048+0000 7f53732f2640 1 --2- 172.21.15.31:0/1215778441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 0x7f536c10ad60 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f5364009ca0 tx=0x7f536402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.048+0000 7f53732f2640 1 -- 172.21.15.31:0/1215778441 shutdown_connections 2024-04-25T16:13:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.048+0000 7f53732f2640 1 --2- 172.21.15.31:0/1215778441 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 0x7f536c10ad60 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.048+0000 7f53732f2640 1 --2- 172.21.15.31:0/1215778441 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f536c103c40 0x7f536c104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.048+0000 7f53732f2640 1 --2- 172.21.15.31:0/1215778441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c102a00 0x7f536c103700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.052 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.048+0000 7f53732f2640 1 -- 172.21.15.31:0/1215778441 >> 172.21.15.31:0/1215778441 conn(0x7f536c0fdbf0 msgr2=0x7f536c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.048+0000 7f53732f2640 1 -- 172.21.15.31:0/1215778441 shutdown_connections 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.048+0000 7f53732f2640 1 -- 172.21.15.31:0/1215778441 wait complete. 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f53732f2640 1 Processor -- start 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f53732f2640 1 -- start start 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f53732f2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f536c102a00 0x7f536c10d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f53732f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c103c40 0x7f536c10b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f53732f2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 0x7f536c10beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f53732f2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f536c10f630 con 0x7f536c108940 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f53732f2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f536c10f4b0 con 0x7f536c102a00 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f53732f2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f536c10f7b0 con 0x7f536c103c40 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f537188f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 0x7f536c10beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f537188f640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 0x7f536c10beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57600/0 (socket says 172.21.15.31:57600) 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.049+0000 7f537188f640 1 -- 172.21.15.31:0/2830215199 learned_addr learned my addr 172.21.15.31:0/2830215199 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:02.053 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537088d640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c103c40 0x7f536c10b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537108e640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f536c102a00 0x7f536c10d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537188f640 1 -- 172.21.15.31:0/2830215199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c103c40 msgr2=0x7f536c10b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537188f640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c103c40 0x7f536c10b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537188f640 1 -- 172.21.15.31:0/2830215199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f536c102a00 msgr2=0x7f536c10d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537188f640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f536c102a00 0x7f536c10d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537188f640 1 -- 172.21.15.31:0/2830215199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536c10c5e0 con 0x7f536c108940 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537088d640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c103c40 0x7f536c10b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537108e640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f536c102a00 0x7f536c10d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f537188f640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 0x7f536c10beb0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5364006010 tx=0x7f536402f3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f53627fc640 1 -- 172.21.15.31:0/2830215199 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5364003c70 con 0x7f536c108940 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f53732f2640 1 -- 172.21.15.31:0/2830215199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536c1a4300 con 0x7f536c108940 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f53627fc640 1 -- 172.21.15.31:0/2830215199 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5364003e10 con 0x7f536c108940 2024-04-25T16:13:02.054 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f53627fc640 1 -- 172.21.15.31:0/2830215199 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f536402fcc0 con 0x7f536c108940 2024-04-25T16:13:02.055 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.050+0000 7f53732f2640 1 -- 172.21.15.31:0/2830215199 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f536c1a4760 con 0x7f536c108940 2024-04-25T16:13:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f53627fc640 1 -- 172.21.15.31:0/2830215199 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5364004040 con 0x7f536c108940 2024-04-25T16:13:02.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537a1a3740 1 Processor -- start 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537a1a3740 1 -- start start 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 0x555b70348810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 0x555b70348d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70349a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537a1a3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555b702acd80 con 0x555b6ff856b0 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537a1a3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555b70266a20 con 0x555b6ff7d4d0 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537a1a3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555b702a8d40 con 0x555b6ffa6390 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537088d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70349a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537088d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70349a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57604/0 (socket says 172.21.15.31:57604) 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.053+0000 7f537088d640 1 -- 172.21.15.31:0/2024594296 learned_addr learned my addr 172.21.15.31:0/2024594296 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f537088d640 1 --2- 172.21.15.31:0/2024594296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 0x555b70348810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f537108e640 1 --2- 172.21.15.31:0/2024594296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 0x555b70348d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.062 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f53627fc640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f53580afaf0 0x7f53580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f537088d640 1 -- 172.21.15.31:0/2024594296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 msgr2=0x555b70348810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f537088d640 1 --2- 172.21.15.31:0/2024594296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 0x555b70348810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f537088d640 1 -- 172.21.15.31:0/2024594296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 msgr2=0x555b70348d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f537088d640 1 --2- 172.21.15.31:0/2024594296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 0x555b70348d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f537088d640 1 -- 172.21.15.31:0/2024594296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b70271c30 con 0x555b6ff856b0 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f537188f640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f53580afaf0 0x7f53580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f537088d640 1 --2- 172.21.15.31:0/2024594296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70349a20 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f535c013610 tx=0x7f535c00c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ef83a9159d3587 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f53627fc640 1 -- 172.21.15.31:0/2830215199 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f53641004a0 con 0x7f536c108940 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f53577fe640 1 -- 172.21.15.31:0/2024594296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c019030 con 0x555b6ff856b0 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f53577fe640 1 -- 172.21.15.31:0/2024594296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f535c0035f0 con 0x555b6ff856b0 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.054+0000 7f53577fe640 1 -- 172.21.15.31:0/2024594296 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c011860 con 0x555b6ff856b0 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537a1a3740 1 -- 172.21.15.31:0/2024594296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 msgr2=0x555b70349a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2024594296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70349a20 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f535c013610 tx=0x7f535c00c780 comp rx=0 tx=0).stop 2024-04-25T16:13:02.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537188f640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f53580afaf0 0x7f53580b2000 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f53640c8690 tx=0x7f536402a8c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537a1a3740 1 -- 172.21.15.31:0/2024594296 shutdown_connections 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2024594296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70349a20 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2024594296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 0x555b70348d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2024594296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 0x555b70348810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537a1a3740 1 -- 172.21.15.31:0/2024594296 >> 172.21.15.31:0/2024594296 conn(0x555b6ff9a4e0 msgr2=0x555b70344020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537a1a3740 1 -- 172.21.15.31:0/2024594296 shutdown_connections 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.055+0000 7f537a1a3740 1 -- 172.21.15.31:0/2024594296 wait complete. 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537a1a3740 1 -- start start 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537a1a3740 1 Processor -- start 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 0x555b70352540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70352a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 0x555b703599e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537a1a3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x555b7034a530 con 0x555b6ff856b0 2024-04-25T16:13:02.064 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537a1a3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x555b70233dc0 con 0x555b6ff7d4d0 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537a1a3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x555b7034a6b0 con 0x555b6ffa6390 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537a1a3740 10 client.0 _ll_register_callbacks cb 0x555b7024f600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537108e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70352a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537108e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70352a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57610/0 (socket says 172.21.15.31:57610) 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537108e640 1 -- 172.21.15.31:0/2167297713 learned_addr learned my addr 172.21.15.31:0/2167297713 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537088d640 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 0x555b703599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537088d640 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 0x555b70352540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537108e640 1 -- 172.21.15.31:0/2167297713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 msgr2=0x555b703599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537108e640 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 0x555b703599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537108e640 1 -- 172.21.15.31:0/2167297713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 msgr2=0x555b70352540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537108e640 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 0x555b70352540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.056+0000 7f537108e640 1 -- 172.21.15.31:0/2167297713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b6ffbe3f0 con 0x555b6ff856b0 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f537088d640 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 0x555b70352540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f537108e640 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70352a80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f536c1022d0 tx=0x7f53680117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd4b20a0a9a4c9f server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:02.065 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f5354ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f5354ff9640 1 -- 172.21.15.31:0/2167297713 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368004440 con 0x555b6ff856b0 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f5354ff9640 1 -- 172.21.15.31:0/2167297713 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f53680035f0 con 0x555b6ff856b0 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f537a1a3740 1 -- 172.21.15.31:0/2167297713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f535c0035f0 con 0x555b6ff856b0 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f5354ff9640 1 -- 172.21.15.31:0/2167297713 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f53680037f0 con 0x555b6ff856b0 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f537a1a3740 10 client.17973 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f537a1a3740 1 -- 172.21.15.31:0/2167297713 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555b700887d0 con 0x555b6ff856b0 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f5354ff9640 1 -- 172.21.15.31:0/2167297713 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f535c0035f0 con 0x555b6ff856b0 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f5354ff9640 1 -- 172.21.15.31:0/2167297713 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f536801bb70 con 0x555b6ff856b0 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f537a1a3740 10 client.17973 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f537a1a3740 10 client.17973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f537a1a3740 20 client.17973 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.057+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555b6ff64f60 0x555b70360310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.058+0000 7f537a1a3740 4 client.17973 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-25T16:13:02.066 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.058+0000 7f537a1a3740 1 -- 172.21.15.31:0/2167297713 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b702ba990 con 0x555b6ff64f60 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.058+0000 7f537088d640 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555b6ff64f60 0x555b70360310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.058+0000 7f537088d640 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555b6ff64f60 0x555b70360310 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=905512704109459d server_cookie=4e65dff04b0c2409 in_seq=0 out_seq=0 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.058+0000 7f5354ff9640 10 client.17973 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f5354ff9640 1 -- 172.21.15.31:0/2167297713 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555b702ba990 con 0x555b6ff64f60 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f5354ff9640 10 client.17973 handle_command_reply: tid=0 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 client.17973 shutdown 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 -- 172.21.15.31:0/2167297713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 msgr2=0x555b70352a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70352a80 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f536c1022d0 tx=0x7f53680117b0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 -- 172.21.15.31:0/2167297713 shutdown_connections 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555b6ff64f60 0x555b70360310 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ffa6390 0x555b703599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f5354ff9640 0 client.17973 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b70352a80 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2167297713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ff7d4d0 0x555b70352540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.067 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 -- 172.21.15.31:0/2167297713 >> 172.21.15.31:0/2167297713 conn(0x555b6ff9a4e0 msgr2=0x555b70346f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 -- 172.21.15.31:0/2167297713 shutdown_connections 2024-04-25T16:13:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 1 -- 172.21.15.31:0/2167297713 wait complete. 2024-04-25T16:13:02.068 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.059+0000 7f537a1a3740 20 client.17973 trim_cache size 0 max 16384 2024-04-25T16:13:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537a1a3740 1 Processor -- start 2024-04-25T16:13:02.076 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537a1a3740 1 -- start start 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff7d4d0 0x555b70344160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff856b0 0x555b70359900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ffa6390 0x555b70350d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537a1a3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1ad3f0 con 0x555b6ff7d4d0 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537a1a3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1aef60 con 0x555b6ffa6390 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537a1a3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1adac0 con 0x555b6ff856b0 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537088d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff7d4d0 0x555b70344160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537088d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff7d4d0 0x555b70344160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57612/0 (socket says 172.21.15.31:57612) 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.073+0000 7f537088d640 1 -- 172.21.15.31:0/885332582 learned_addr learned my addr 172.21.15.31:0/885332582 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f537088d640 1 -- 172.21.15.31:0/885332582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff856b0 msgr2=0x555b70359900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f537108e640 1 --2- 172.21.15.31:0/885332582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff856b0 0x555b70359900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f537088d640 1 --2- 172.21.15.31:0/885332582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff856b0 0x555b70359900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f537088d640 1 -- 172.21.15.31:0/885332582 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ffa6390 msgr2=0x555b70350d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:02.077 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f537088d640 1 --2- 172.21.15.31:0/885332582 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ffa6390 0x555b70350d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f537088d640 1 -- 172.21.15.31:0/885332582 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b7034dca0 con 0x555b6ff7d4d0 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f537108e640 1 --2- 172.21.15.31:0/885332582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff856b0 0x555b70359900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f537088d640 1 --2- 172.21.15.31:0/885332582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff7d4d0 0x555b70344160 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f535c0058c0 tx=0x7f535c013460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327f89575c2630e4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f5356ffd640 1 -- 172.21.15.31:0/885332582 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c019040 con 0x555b6ff7d4d0 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f5356ffd640 1 -- 172.21.15.31:0/885332582 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f535c0054f0 con 0x555b6ff7d4d0 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.074+0000 7f5356ffd640 1 -- 172.21.15.31:0/885332582 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535c028a20 con 0x555b6ff7d4d0 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 -- 172.21.15.31:0/885332582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff7d4d0 msgr2=0x555b70344160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 --2- 172.21.15.31:0/885332582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff7d4d0 0x555b70344160 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f535c0058c0 tx=0x7f535c013460 comp rx=0 tx=0).stop 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 -- 172.21.15.31:0/885332582 shutdown_connections 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 --2- 172.21.15.31:0/885332582 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ffa6390 0x555b70350d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 --2- 172.21.15.31:0/885332582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff856b0 0x555b70359900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 --2- 172.21.15.31:0/885332582 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff7d4d0 0x555b70344160 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 -- 172.21.15.31:0/885332582 >> 172.21.15.31:0/885332582 conn(0x555b6ff9a4e0 msgr2=0x555b70343a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:02.078 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 -- 172.21.15.31:0/885332582 shutdown_connections 2024-04-25T16:13:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 -- 172.21.15.31:0/885332582 wait complete. 2024-04-25T16:13:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 -- start start 2024-04-25T16:13:02.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.075+0000 7f537a1a3740 1 Processor -- start 2024-04-25T16:13:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff7d4d0 0x555b70356ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b703570e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537a1a3740 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ffa6390 0x555b70358010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.082 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537a1a3740 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1ae890 con 0x555b6ff856b0 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537a1a3740 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1ae190 con 0x555b6ffa6390 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537a1a3740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f536c1ab3d0 con 0x555b6ff7d4d0 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537a1a3740 10 client.0 _ll_register_callbacks cb 0x7f536800b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537108e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b703570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537108e640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b703570e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57628/0 (socket says 172.21.15.31:57628) 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537108e640 1 -- 172.21.15.31:0/2921995167 learned_addr learned my addr 172.21.15.31:0/2921995167 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537088d640 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff7d4d0 0x555b70356ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537088d640 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ffa6390 0x555b70358010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537108e640 1 -- 172.21.15.31:0/2921995167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff7d4d0 msgr2=0x555b70356ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537108e640 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff7d4d0 0x555b70356ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537108e640 1 -- 172.21.15.31:0/2921995167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ffa6390 msgr2=0x555b70358010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537108e640 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ffa6390 0x555b70358010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537108e640 1 -- 172.21.15.31:0/2921995167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555b70359e40 con 0x555b6ff856b0 2024-04-25T16:13:02.083 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537108e640 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b703570e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5368003710 tx=0x7f5368003740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d2bd297bb70c2a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f5354ff9640 10 client.0 ms_handle_connect on v2:172.21.15.31:3300/0 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f5354ff9640 1 -- 172.21.15.31:0/2921995167 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5368004450 con 0x555b6ff856b0 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f5354ff9640 1 -- 172.21.15.31:0/2921995167 <== mon.0 v2:172.21.15.31:3300/0 2 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f5368012a20 con 0x555b6ff856b0 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.076+0000 7f537a1a3740 1 -- 172.21.15.31:0/2921995167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f535c0054f0 con 0x555b6ff856b0 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f5354ff9640 1 -- 172.21.15.31:0/2921995167 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f5368037050 con 0x555b6ff856b0 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f537a1a3740 10 client.17979 fetch_fsmap learned FSMap version 30 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f537a1a3740 1 -- 172.21.15.31:0/2921995167 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555b703596f0 con 0x555b6ff856b0 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f5354ff9640 1 -- 172.21.15.31:0/2921995167 <== mon.0 v2:172.21.15.31:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f535c0054f0 con 0x555b6ff856b0 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f5354ff9640 1 -- 172.21.15.31:0/2921995167 <== mon.0 v2:172.21.15.31:3300/0 5 ==== fsmap(e 30) ==== 8358+0+0 (secure 0 0 0) 0x7f53680051e0 con 0x555b6ff856b0 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f537a1a3740 10 client.17979 fetch_fsmap finished waiting for FSMap version 30 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f537a1a3740 10 client.17979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f537a1a3740 20 client.17979 populate_metadata read hostname 'smithi031' 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555b6ff3a3a0 0x555b703ca980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:02.084 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f537a1a3740 4 client.17979 mds_command: new command op to 24563 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f537a1a3740 1 -- 172.21.15.31:0/2921995167 --> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555b702b4800 con 0x555b6ff3a3a0 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.077+0000 7f537088d640 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555b6ff3a3a0 0x555b703ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f5354ff9640 10 client.17979 ms_handle_connect on v2:172.21.15.63:6838/495769167 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537088d640 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555b6ff3a3a0 0x555b703ca980 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42ce7f8bfaf6cd98 server_cookie=28682a077b9ea96a in_seq=0 out_seq=0 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f5354ff9640 1 -- 172.21.15.31:0/2921995167 <== mds.0 v2:172.21.15.63:6838/495769167 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555b702b4800 con 0x555b6ff3a3a0 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f5354ff9640 10 client.17979 handle_command_reply: tid=0 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 client.17979 shutdown 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 -- 172.21.15.31:0/2921995167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 msgr2=0x555b703570e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b703570e0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f5368003710 tx=0x7f5368003740 comp rx=0 tx=0).stop 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 -- 172.21.15.31:0/2921995167 shutdown_connections 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.63:6838/495769167,v1:172.21.15.63:6839/495769167] conn(0x555b6ff3a3a0 0x555b703ca980 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x555b6ffa6390 0x555b70358010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x555b6ff856b0 0x555b703570e0 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.085 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f5354ff9640 0 client.17979 ms_handle_reset on v2:172.21.15.63:6838/495769167 2024-04-25T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 --2- 172.21.15.31:0/2921995167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555b6ff7d4d0 0x555b70356ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 -- 172.21.15.31:0/2921995167 >> 172.21.15.31:0/2921995167 conn(0x555b6ff9a4e0 msgr2=0x555b70346e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 -- 172.21.15.31:0/2921995167 shutdown_connections 2024-04-25T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 1 -- 172.21.15.31:0/2921995167 wait complete. 2024-04-25T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.078+0000 7f537a1a3740 20 client.17979 trim_cache size 0 max 16384 2024-04-25T16:13:02.086 INFO:teuthology.orchestra.run.smithi031.stdout:[] 2024-04-25T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 -- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f53580afaf0 msgr2=0x7f53580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f53580afaf0 0x7f53580b2000 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f53640c8690 tx=0x7f536402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 -- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 msgr2=0x7f536c10beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 0x7f536c10beb0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f5364006010 tx=0x7f536402f3e0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 -- 172.21.15.31:0/2830215199 shutdown_connections 2024-04-25T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f53580afaf0 0x7f53580b2000 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f536c108940 0x7f536c10beb0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c103c40 0x7f536c10b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.087 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 --2- 172.21.15.31:0/2830215199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f536c102a00 0x7f536c10d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 -- 172.21.15.31:0/2830215199 >> 172.21.15.31:0/2830215199 conn(0x7f536c0fdbf0 msgr2=0x7f536c104920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 -- 172.21.15.31:0/2830215199 shutdown_connections 2024-04-25T16:13:02.088 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:02.079+0000 7f53732f2640 1 -- 172.21.15.31:0/2830215199 wait complete. 2024-04-25T16:13:03.092 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-25T16:13:03.092 INFO:tasks.fwd_scrub:done joining 2024-04-25T16:13:03.093 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-25T16:13:03.103 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:13:03.235 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.230+0000 7fe0ed28c640 1 -- 172.21.15.31:0/1408898157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e8103c20 msgr2=0x7fe0e8104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/1408898157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e8103c20 0x7fe0e8104030 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0009900 tx=0x7fe0d002a410 comp rx=0 tx=0).stop 2024-04-25T16:13:03.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 -- 172.21.15.31:0/1408898157 shutdown_connections 2024-04-25T16:13:03.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/1408898157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e8108920 0x7fe0e810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/1408898157 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e8103c20 0x7fe0e8104030 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/1408898157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e81029e0 0x7fe0e81036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 -- 172.21.15.31:0/1408898157 >> 172.21.15.31:0/1408898157 conn(0x7fe0e80fdbf0 msgr2=0x7fe0e8100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:03.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 -- 172.21.15.31:0/1408898157 shutdown_connections 2024-04-25T16:13:03.236 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 -- 172.21.15.31:0/1408898157 wait complete. 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 Processor -- start 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.231+0000 7fe0ed28c640 1 -- start start 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0ed28c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e81029e0 0x7fe0e8075860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0ed28c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e8103c20 0x7fe0e819f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0ed28c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e8108920 0x7fe0e81a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0ed28c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e810e560 con 0x7fe0e81029e0 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0ed28c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e810e3e0 con 0x7fe0e8108920 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0ed28c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e810e6e0 con 0x7fe0e8103c20 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e81029e0 0x7fe0e8075860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:03.237 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e81029e0 0x7fe0e8075860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57654/0 (socket says 172.21.15.31:57654) 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e6ffd640 1 -- 172.21.15.31:0/652663859 learned_addr learned my addr 172.21.15.31:0/652663859 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e77fe640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e8108920 0x7fe0e81a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0dffff640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e8103c20 0x7fe0e819f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e6ffd640 1 -- 172.21.15.31:0/652663859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e8103c20 msgr2=0x7fe0e819f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e6ffd640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e8103c20 0x7fe0e819f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e6ffd640 1 -- 172.21.15.31:0/652663859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e8108920 msgr2=0x7fe0e81a15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e6ffd640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e8108920 0x7fe0e81a15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e6ffd640 1 -- 172.21.15.31:0/652663859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e81a1e10 con 0x7fe0e81029e0 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.232+0000 7fe0e6ffd640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e81029e0 0x7fe0e8075860 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800d740 tx=0x7fe0d800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.233+0000 7fe0e4ff9640 1 -- 172.21.15.31:0/652663859 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8017070 con 0x7fe0e81029e0 2024-04-25T16:13:03.238 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.233+0000 7fe0e4ff9640 1 -- 172.21.15.31:0/652663859 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe0d800de50 con 0x7fe0e81029e0 2024-04-25T16:13:03.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.233+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0e81a2100 con 0x7fe0e81029e0 2024-04-25T16:13:03.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.233+0000 7fe0e4ff9640 1 -- 172.21.15.31:0/652663859 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0d8003c20 con 0x7fe0e81029e0 2024-04-25T16:13:03.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.233+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e81a2650 con 0x7fe0e81029e0 2024-04-25T16:13:03.239 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.234+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e8107730 con 0x7fe0e81029e0 2024-04-25T16:13:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.235+0000 7fe0e4ff9640 1 -- 172.21.15.31:0/652663859 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe0d8003dc0 con 0x7fe0e81029e0 2024-04-25T16:13:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.236+0000 7fe0e4ff9640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0d40afaf0 0x7fe0d40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.236+0000 7fe0e4ff9640 1 -- 172.21.15.31:0/652663859 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fe0d80dea90 con 0x7fe0e81029e0 2024-04-25T16:13:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.236+0000 7fe0dffff640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0d40afaf0 0x7fe0d40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:03.240 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.237+0000 7fe0dffff640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0d40afaf0 0x7fe0d40b2000 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0006010 tx=0x7fe0d0031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:03.241 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.240+0000 7fe0e4ff9640 1 -- 172.21.15.31:0/652663859 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0d80a7680 con 0x7fe0e81029e0 2024-04-25T16:13:03.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.432+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0e802dda0 con 0x7fe0e81029e0 2024-04-25T16:13:03.435 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.433+0000 7fe0e4ff9640 1 -- 172.21.15.31:0/652663859 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7fe0d80a6f80 con 0x7fe0e81029e0 2024-04-25T16:13:03.435 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:13:03.435 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:13:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.435+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0d40afaf0 msgr2=0x7fe0d40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.435+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0d40afaf0 0x7fe0d40b2000 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe0d0006010 tx=0x7fe0d0031400 comp rx=0 tx=0).stop 2024-04-25T16:13:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.435+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e81029e0 msgr2=0x7fe0e8075860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.435+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e81029e0 0x7fe0e8075860 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe0d800d740 tx=0x7fe0d800dc00 comp rx=0 tx=0).stop 2024-04-25T16:13:03.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.436+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 shutdown_connections 2024-04-25T16:13:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.436+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fe0d40afaf0 0x7fe0d40b2000 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.436+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe0e8108920 0x7fe0e81a15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.436+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e8103c20 0x7fe0e819f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.436+0000 7fe0ed28c640 1 --2- 172.21.15.31:0/652663859 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fe0e81029e0 0x7fe0e8075860 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.436+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 >> 172.21.15.31:0/652663859 conn(0x7fe0e80fdbf0 msgr2=0x7fe0e8104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.436+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 shutdown_connections 2024-04-25T16:13:03.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.436+0000 7fe0ed28c640 1 -- 172.21.15.31:0/652663859 wait complete. 2024-04-25T16:13:03.448 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:03 smithi031 ceph-mon[29140]: pgmap v703: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 106 MiB/s wr, 250 op/s 2024-04-25T16:13:03.449 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:59:00.978038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:59:01.829535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:13:03.450 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:13:03.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.564+0000 7f8d98106640 1 -- 172.21.15.31:0/3523265115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 msgr2=0x7f8d9010be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.564+0000 7f8d98106640 1 --2- 172.21.15.31:0/3523265115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 0x7f8d9010be50 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009ca0 tx=0x7f8d8802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.564+0000 7f8d98106640 1 -- 172.21.15.31:0/3523265115 shutdown_connections 2024-04-25T16:13:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.564+0000 7f8d98106640 1 --2- 172.21.15.31:0/3523265115 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 0x7f8d9010be50 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.564+0000 7f8d98106640 1 --2- 172.21.15.31:0/3523265115 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90104d30 0x7f8d90105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.564+0000 7f8d98106640 1 --2- 172.21.15.31:0/3523265115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d90103af0 0x7f8d901047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.564+0000 7f8d98106640 1 -- 172.21.15.31:0/3523265115 >> 172.21.15.31:0/3523265115 conn(0x7f8d900fece0 msgr2=0x7f8d90101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.564+0000 7f8d98106640 1 -- 172.21.15.31:0/3523265115 shutdown_connections 2024-04-25T16:13:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.564+0000 7f8d98106640 1 -- 172.21.15.31:0/3523265115 wait complete. 2024-04-25T16:13:03.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d98106640 1 Processor -- start 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d98106640 1 -- start start 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d98106640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d90103af0 0x7f8d9019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d98106640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90104d30 0x7f8d9019a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d98106640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 0x7f8d901a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d98106640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9010f600 con 0x7f8d90109a30 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d98106640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9010f480 con 0x7f8d90104d30 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d98106640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9010f780 con 0x7f8d90103af0 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d966a3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 0x7f8d901a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d966a3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 0x7f8d901a1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:57666/0 (socket says 172.21.15.31:57666) 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d966a3640 1 -- 172.21.15.31:0/2333106235 learned_addr learned my addr 172.21.15.31:0/2333106235 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:03.570 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.565+0000 7f8d95ea2640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d90103af0 0x7f8d9019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d956a1640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90104d30 0x7f8d9019a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d966a3640 1 -- 172.21.15.31:0/2333106235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d90103af0 msgr2=0x7f8d9019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d966a3640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d90103af0 0x7f8d9019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d966a3640 1 -- 172.21.15.31:0/2333106235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90104d30 msgr2=0x7f8d9019a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d966a3640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90104d30 0x7f8d9019a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d966a3640 1 -- 172.21.15.31:0/2333106235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d901a1d00 con 0x7f8d90109a30 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d956a1640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90104d30 0x7f8d9019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d966a3640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 0x7f8d901a1690 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f8d880097b0 tx=0x7f8d88003c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d86ffd640 1 -- 172.21.15.31:0/2333106235 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d88004240 con 0x7f8d90109a30 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d86ffd640 1 -- 172.21.15.31:0/2333106235 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d8802f380 con 0x7f8d90109a30 2024-04-25T16:13:03.571 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d901a1f90 con 0x7f8d90109a30 2024-04-25T16:13:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d86ffd640 1 -- 172.21.15.31:0/2333106235 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8802fc30 con 0x7f8d90109a30 2024-04-25T16:13:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.566+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d901a2450 con 0x7f8d90109a30 2024-04-25T16:13:03.572 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.567+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d90108840 con 0x7f8d90109a30 2024-04-25T16:13:03.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.570+0000 7f8d86ffd640 1 -- 172.21.15.31:0/2333106235 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f8d880043e0 con 0x7f8d90109a30 2024-04-25T16:13:03.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.570+0000 7f8d86ffd640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d7c0afb70 0x7f8d7c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:03.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.570+0000 7f8d95ea2640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d7c0afb70 0x7f8d7c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:03.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.570+0000 7f8d86ffd640 1 -- 172.21.15.31:0/2333106235 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f8d88100440 con 0x7f8d90109a30 2024-04-25T16:13:03.573 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.571+0000 7f8d95ea2640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d7c0afb70 0x7f8d7c0b2080 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c005fd0 tx=0x7f8d8c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:03.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.574+0000 7f8d86ffd640 1 -- 172.21.15.31:0/2333106235 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d880c9030 con 0x7f8d90109a30 2024-04-25T16:13:03.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:03 smithi063 ceph-mon[32413]: pgmap v703: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 106 MiB/s wr, 250 op/s 2024-04-25T16:13:03.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:03 smithi186 ceph-mon[33258]: pgmap v703: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 106 MiB/s wr, 250 op/s 2024-04-25T16:13:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.939+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d9002dce0 con 0x7f8d90109a30 2024-04-25T16:13:03.942 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.940+0000 7f8d86ffd640 1 -- 172.21.15.31:0/2333106235 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7f8d880c8930 con 0x7f8d90109a30 2024-04-25T16:13:03.943 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T16:13:03.943 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d7c0afb70 msgr2=0x7f8d7c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d7c0afb70 0x7f8d7c0b2080 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c005fd0 tx=0x7f8d8c00b410 comp rx=0 tx=0).stop 2024-04-25T16:13:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 msgr2=0x7f8d901a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 0x7f8d901a1690 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f8d880097b0 tx=0x7f8d88003c20 comp rx=0 tx=0).stop 2024-04-25T16:13:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 shutdown_connections 2024-04-25T16:13:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f8d7c0afb70 0x7f8d7c0b2080 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f8d90109a30 0x7f8d901a1690 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8d90104d30 0x7f8d9019a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 --2- 172.21.15.31:0/2333106235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d90103af0 0x7f8d9019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:03.944 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 >> 172.21.15.31:0/2333106235 conn(0x7f8d900fece0 msgr2=0x7f8d90105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:03.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 shutdown_connections 2024-04-25T16:13:03.945 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:03.942+0000 7f8d98106640 1 -- 172.21.15.31:0/2333106235 wait complete. 2024-04-25T16:13:03.965 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:13:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.101+0000 7f9a9ee4a640 1 -- 172.21.15.31:0/2790042878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 msgr2=0x7f9a98103dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.101+0000 7f9a9ee4a640 1 --2- 172.21.15.31:0/2790042878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 0x7f9a98103dd0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009900 tx=0x7f9a8402a440 comp rx=0 tx=0).stop 2024-04-25T16:13:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.101+0000 7f9a9ee4a640 1 -- 172.21.15.31:0/2790042878 shutdown_connections 2024-04-25T16:13:04.103 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.101+0000 7f9a9ee4a640 1 --2- 172.21.15.31:0/2790042878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a98104460 0x7f9a9810aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.101+0000 7f9a9ee4a640 1 --2- 172.21.15.31:0/2790042878 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 0x7f9a98103dd0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.101+0000 7f9a9ee4a640 1 --2- 172.21.15.31:0/2790042878 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a98102f40 0x7f9a98103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.101+0000 7f9a9ee4a640 1 -- 172.21.15.31:0/2790042878 >> 172.21.15.31:0/2790042878 conn(0x7f9a980fd340 msgr2=0x7f9a980ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:04.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.101+0000 7f9a9ee4a640 1 -- 172.21.15.31:0/2790042878 shutdown_connections 2024-04-25T16:13:04.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.101+0000 7f9a9ee4a640 1 -- 172.21.15.31:0/2790042878 wait complete. 2024-04-25T16:13:04.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.103+0000 7f9a9ee4a640 1 Processor -- start 2024-04-25T16:13:04.104 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.103+0000 7f9a9ee4a640 1 -- start start 2024-04-25T16:13:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.103+0000 7f9a9ee4a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a98102f40 0x7f9a98197520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.103+0000 7f9a9ee4a640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 0x7f9a98197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a9ee4a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a98104460 0x7f9a98194070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a9ee4a640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9810e610 con 0x7f9a98103920 2024-04-25T16:13:04.106 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a9ee4a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9810e490 con 0x7f9a98104460 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a9ee4a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9810e790 con 0x7f9a98102f40 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 0x7f9a98197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 0x7f9a98197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34266/0 (socket says 172.21.15.31:34266) 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a97fff640 1 -- 172.21.15.31:0/3129934316 learned_addr learned my addr 172.21.15.31:0/3129934316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a97fff640 1 -- 172.21.15.31:0/3129934316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a98102f40 msgr2=0x7f9a98197520 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a97fff640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a98102f40 0x7f9a98197520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a97fff640 1 -- 172.21.15.31:0/3129934316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a98104460 msgr2=0x7f9a98194070 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a97fff640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a98104460 0x7f9a98194070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.104+0000 7f9a97fff640 1 -- 172.21.15.31:0/3129934316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a98194890 con 0x7f9a98103920 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.105+0000 7f9a97fff640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 0x7f9a98197a60 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000c00 tx=0x7f9a840043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:04.107 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.105+0000 7f9a95ffb640 1 -- 172.21.15.31:0/3129934316 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8403f030 con 0x7f9a98103920 2024-04-25T16:13:04.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.105+0000 7f9a95ffb640 1 -- 172.21.15.31:0/3129934316 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9a84003800 con 0x7f9a98103920 2024-04-25T16:13:04.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.105+0000 7f9a95ffb640 1 -- 172.21.15.31:0/3129934316 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84035070 con 0x7f9a98103920 2024-04-25T16:13:04.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.105+0000 7f9a9ee4a640 1 -- 172.21.15.31:0/3129934316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a98194b20 con 0x7f9a98103920 2024-04-25T16:13:04.108 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.106+0000 7f9a9ee4a640 1 -- 172.21.15.31:0/3129934316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a981a8c30 con 0x7f9a98103920 2024-04-25T16:13:04.110 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.109+0000 7f9a837fe640 1 -- 172.21.15.31:0/3129934316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a980637a0 con 0x7f9a98103920 2024-04-25T16:13:04.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.110+0000 7f9a95ffb640 1 -- 172.21.15.31:0/3129934316 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9a84003cb0 con 0x7f9a98103920 2024-04-25T16:13:04.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.112+0000 7f9a95ffb640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9a880afb70 0x7f9a880b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.113+0000 7f9a95ffb640 1 -- 172.21.15.31:0/3129934316 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f9a840ff7e0 con 0x7f9a98103920 2024-04-25T16:13:04.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.114+0000 7f9a9cbe6640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9a880afb70 0x7f9a880b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.116 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.115+0000 7f9a9cbe6640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9a880afb70 0x7f9a880b2080 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9a98069e00 tx=0x7f9a8c009250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:04.125 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.123+0000 7f9a95ffb640 1 -- 172.21.15.31:0/3129934316 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9a840c83d0 con 0x7f9a98103920 2024-04-25T16:13:04.345 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.344+0000 7f9a837fe640 1 -- 172.21.15.31:0/3129934316 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9a98063970 con 0x7f9a98103920 2024-04-25T16:13:04.350 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.349+0000 7f9a95ffb640 1 -- 172.21.15.31:0/3129934316 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7f9a840c7cd0 con 0x7f9a98103920 2024-04-25T16:13:04.350 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 -- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9a880afb70 msgr2=0x7f9a880b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9a880afb70 0x7f9a880b2080 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9a98069e00 tx=0x7f9a8c009250 comp rx=0 tx=0).stop 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 -- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 msgr2=0x7f9a98197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 0x7f9a98197a60 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000c00 tx=0x7f9a840043c0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 -- 172.21.15.31:0/3129934316 shutdown_connections 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f9a880afb70 0x7f9a880b2080 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9a98104460 0x7f9a98194070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f9a98103920 0x7f9a98197a60 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 --2- 172.21.15.31:0/3129934316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a98102f40 0x7f9a98197520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 -- 172.21.15.31:0/3129934316 >> 172.21.15.31:0/3129934316 conn(0x7f9a980fd340 msgr2=0x7f9a980ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:04.354 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 -- 172.21.15.31:0/3129934316 shutdown_connections 2024-04-25T16:13:04.355 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.352+0000 7f9a837fe640 1 -- 172.21.15.31:0/3129934316 wait complete. 2024-04-25T16:13:04.372 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:13:04.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.554+0000 7f16d2451640 1 -- 172.21.15.31:0/793224468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc110030 msgr2=0x7f16cc110440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.554+0000 7f16d2451640 1 --2- 172.21.15.31:0/793224468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc110030 0x7f16cc110440 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f16c401acb0 tx=0x7f16c403b920 comp rx=0 tx=0).stop 2024-04-25T16:13:04.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.555+0000 7f16d2451640 1 -- 172.21.15.31:0/793224468 shutdown_connections 2024-04-25T16:13:04.557 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.555+0000 7f16d2451640 1 --2- 172.21.15.31:0/793224468 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc110030 0x7f16cc110440 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.555+0000 7f16d2451640 1 --2- 172.21.15.31:0/793224468 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16cc10b310 0x7f16cc10b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.555+0000 7f16d2451640 1 --2- 172.21.15.31:0/793224468 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16cc10a420 0x7f16cc10add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.555+0000 7f16d2451640 1 -- 172.21.15.31:0/793224468 >> 172.21.15.31:0/793224468 conn(0x7f16cc0704e0 msgr2=0x7f16cc072990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:04.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.556+0000 7f16d2451640 1 -- 172.21.15.31:0/793224468 shutdown_connections 2024-04-25T16:13:04.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.556+0000 7f16d2451640 1 -- 172.21.15.31:0/793224468 wait complete. 2024-04-25T16:13:04.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.557+0000 7f16d2451640 1 Processor -- start 2024-04-25T16:13:04.558 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.557+0000 7f16d2451640 1 -- start start 2024-04-25T16:13:04.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16d2451640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16cc10a420 0x7f16cc1a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16d2451640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc10b310 0x7f16cc1a0f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16d2451640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16cc110030 0x7f16cc1a7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16d2451640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc113d50 con 0x7f16cc10b310 2024-04-25T16:13:04.559 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16d2451640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc113bd0 con 0x7f16cc10a420 2024-04-25T16:13:04.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16d2451640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f16cc113ed0 con 0x7f16cc110030 2024-04-25T16:13:04.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16d09ee640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16cc110030 0x7f16cc1a7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16d09ee640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16cc110030 0x7f16cc1a7e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.31:40200/0 (socket says 172.21.15.31:40200) 2024-04-25T16:13:04.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16d09ee640 1 -- 172.21.15.31:0/3112364927 learned_addr learned my addr 172.21.15.31:0/3112364927 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:04.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.558+0000 7f16c3fff640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc10b310 0x7f16cc1a0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.559+0000 7f16cbfff640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16cc10a420 0x7f16cc1a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.560 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.559+0000 7f16cbfff640 1 -- 172.21.15.31:0/3112364927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16cc110030 msgr2=0x7f16cc1a7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.559+0000 7f16cbfff640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16cc110030 0x7f16cc1a7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.559+0000 7f16cbfff640 1 -- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc10b310 msgr2=0x7f16cc1a0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.559+0000 7f16cbfff640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc10b310 0x7f16cc1a0f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.559+0000 7f16cbfff640 1 -- 172.21.15.31:0/3112364927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16cc1a8500 con 0x7f16cc10a420 2024-04-25T16:13:04.561 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.560+0000 7f16cbfff640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16cc10a420 0x7f16cc1a09c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009fd0 tx=0x7f16bc00b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:04.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.561+0000 7f16c9ffb640 1 -- 172.21.15.31:0/3112364927 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc015020 con 0x7f16cc10a420 2024-04-25T16:13:04.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.561+0000 7f16c9ffb640 1 -- 172.21.15.31:0/3112364927 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16bc011ce0 con 0x7f16cc10a420 2024-04-25T16:13:04.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.561+0000 7f16c9ffb640 1 -- 172.21.15.31:0/3112364927 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16bc00f2e0 con 0x7f16cc10a420 2024-04-25T16:13:04.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.561+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16cc1a87f0 con 0x7f16cc10a420 2024-04-25T16:13:04.563 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.561+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16cc1a8d70 con 0x7f16cc10a420 2024-04-25T16:13:04.565 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.564+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16cc069d10 con 0x7f16cc10a420 2024-04-25T16:13:04.566 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.565+0000 7f16c9ffb640 1 -- 172.21.15.31:0/3112364927 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f16bc018020 con 0x7f16cc10a420 2024-04-25T16:13:04.567 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.566+0000 7f16c9ffb640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f16b80afd00 0x7f16b80b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.568 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.566+0000 7f16c9ffb640 1 -- 172.21.15.31:0/3112364927 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f16bc0de2f0 con 0x7f16cc10a420 2024-04-25T16:13:04.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.567+0000 7f16c3fff640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f16b80afd00 0x7f16b80b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.569 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.568+0000 7f16c3fff640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f16b80afd00 0x7f16b80b2210 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f16cc1a1a60 tx=0x7f16b4009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:04.575 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.574+0000 7f16c9ffb640 1 -- 172.21.15.31:0/3112364927 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16bc01f020 con 0x7f16cc10a420 2024-04-25T16:13:04.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:04 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/652663859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:13:04.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:04 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2333106235' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:13:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:04 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/652663859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:13:04.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:04 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2333106235' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:13:04.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:04 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/652663859' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:13:04.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:04 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2333106235' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:13:04.802 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.801+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f16cc1a8fd0 con 0x7f16cc10a420 2024-04-25T16:13:04.803 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.801+0000 7f16c9ffb640 1 -- 172.21.15.31:0/3112364927 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7f16bc010e90 con 0x7f16cc10a420 2024-04-25T16:13:04.803 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T16:13:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f16b80afd00 msgr2=0x7f16b80b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f16b80afd00 0x7f16b80b2210 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f16cc1a1a60 tx=0x7f16b4009110 comp rx=0 tx=0).stop 2024-04-25T16:13:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16cc10a420 msgr2=0x7f16cc1a09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16cc10a420 0x7f16cc1a09c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f16bc009fd0 tx=0x7f16bc00b7b0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 shutdown_connections 2024-04-25T16:13:04.805 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f16b80afd00 0x7f16b80b2210 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16cc110030 0x7f16cc1a7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f16cc10b310 0x7f16cc1a0f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 --2- 172.21.15.31:0/3112364927 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16cc10a420 0x7f16cc1a09c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 >> 172.21.15.31:0/3112364927 conn(0x7f16cc0704e0 msgr2=0x7f16cc10e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.803+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 shutdown_connections 2024-04-25T16:13:04.806 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.804+0000 7f16d2451640 1 -- 172.21.15.31:0/3112364927 wait complete. 2024-04-25T16:13:04.817 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-25T16:13:04.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.928+0000 7f569a80c640 1 -- 172.21.15.31:0/3359677491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 msgr2=0x7f56941037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.928+0000 7f569a80c640 1 --2- 172.21.15.31:0/3359677491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 0x7f56941037b0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f5688009900 tx=0x7f568802a410 comp rx=0 tx=0).stop 2024-04-25T16:13:04.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.928+0000 7f569a80c640 1 -- 172.21.15.31:0/3359677491 shutdown_connections 2024-04-25T16:13:04.930 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.928+0000 7f569a80c640 1 --2- 172.21.15.31:0/3359677491 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56941080a0 0x7f569410a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.928+0000 7f569a80c640 1 --2- 172.21.15.31:0/3359677491 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 0x7f56941037b0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.928+0000 7f569a80c640 1 --2- 172.21.15.31:0/3359677491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5694102160 0x7f5694102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.928+0000 7f569a80c640 1 -- 172.21.15.31:0/3359677491 >> 172.21.15.31:0/3359677491 conn(0x7f56940fd330 msgr2=0x7f56940ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:04.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.928+0000 7f569a80c640 1 -- 172.21.15.31:0/3359677491 shutdown_connections 2024-04-25T16:13:04.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.928+0000 7f569a80c640 1 -- 172.21.15.31:0/3359677491 wait complete. 2024-04-25T16:13:04.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f569a80c640 1 Processor -- start 2024-04-25T16:13:04.931 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f569a80c640 1 -- start start 2024-04-25T16:13:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f569a80c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5694102160 0x7f5694198020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f569a80c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 0x7f5694198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f569a80c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56941080a0 0x7f569419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f569a80c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f569410d3f0 con 0x7f56941033a0 2024-04-25T16:13:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f569a80c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f569410d270 con 0x7f56941080a0 2024-04-25T16:13:04.933 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f569a80c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f569410d570 con 0x7f5694102160 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f56937fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 0x7f5694198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f56937fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 0x7f5694198560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:34300/0 (socket says 172.21.15.31:34300) 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f56937fe640 1 -- 172.21.15.31:0/3178235717 learned_addr learned my addr 172.21.15.31:0/3178235717 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.929+0000 7f5693fff640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5694102160 0x7f5694198020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f5698da9640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56941080a0 0x7f569419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f56937fe640 1 -- 172.21.15.31:0/3178235717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5694102160 msgr2=0x7f5694198020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f56937fe640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5694102160 0x7f5694198020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f56937fe640 1 -- 172.21.15.31:0/3178235717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56941080a0 msgr2=0x7f569419f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f56937fe640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56941080a0 0x7f569419f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f56937fe640 1 -- 172.21.15.31:0/3178235717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f569419fb60 con 0x7f56941033a0 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f56937fe640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 0x7f5694198560 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f5688006010 tx=0x7f56880042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:04.934 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f56917fa640 1 -- 172.21.15.31:0/3178235717 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f568803f030 con 0x7f56941033a0 2024-04-25T16:13:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f56917fa640 1 -- 172.21.15.31:0/3178235717 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5688031910 con 0x7f56941033a0 2024-04-25T16:13:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f569419fdf0 con 0x7f56941033a0 2024-04-25T16:13:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f56917fa640 1 -- 172.21.15.31:0/3178235717 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5688035070 con 0x7f56941033a0 2024-04-25T16:13:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.930+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56941a0220 con 0x7f56941033a0 2024-04-25T16:13:04.935 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.931+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5694106eb0 con 0x7f56941033a0 2024-04-25T16:13:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.932+0000 7f56917fa640 1 -- 172.21.15.31:0/3178235717 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5688042030 con 0x7f56941033a0 2024-04-25T16:13:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.933+0000 7f56917fa640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f567c0afaf0 0x7f567c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.933+0000 7f56917fa640 1 -- 172.21.15.31:0/3178235717 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f56880ff810 con 0x7f56941033a0 2024-04-25T16:13:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.933+0000 7f5693fff640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f567c0afaf0 0x7f567c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:04.936 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.934+0000 7f5693fff640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f567c0afaf0 0x7f567c0b2000 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5694102d60 tx=0x7f5684005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:04.938 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:04.937+0000 7f56917fa640 1 -- 172.21.15.31:0/3178235717 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56880c8400 con 0x7f56941033a0 2024-04-25T16:13:05.173 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.172+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f56940637a0 con 0x7f56941033a0 2024-04-25T16:13:05.182 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.181+0000 7f56917fa640 1 -- 172.21.15.31:0/3178235717 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+803125 (secure 0 0 0) 0x7f56880c7d00 con 0x7f56941033a0 2024-04-25T16:13:05.193 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:13:05.193 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:13:05.193 INFO:teuthology.orchestra.run.smithi031.stdout: "epoch": 18, 2024-04-25T16:13:05.193 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.193 INFO:teuthology.orchestra.run.smithi031.stdout: "active_gid": 14152, 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: "active_name": "x", 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addrs": { 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6800", 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3984865329 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v1", 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:6801", 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 3984865329 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.194 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "active_addr": "172.21.15.31:6801/3984865329", 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "active_change": "2024-04-25T15:51:49.220007+0000", 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "active_mgr_features": 4540701547738038271, 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "available": true, 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "standbys": [ 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 24109, 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "z", 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540701547738038271, 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.196 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.197 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T16:13:05.197 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.197 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.197 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.197 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.198 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.198 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.198 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.199 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.200 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.201 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.203 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.204 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.205 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.207 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.208 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.208 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.208 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-04-25T16:13:05.208 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-04-25T16:13:05.208 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.208 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.208 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-04-25T16:13:05.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.210 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.213 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.214 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.215 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.218 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.219 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.220 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.221 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.223 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.224 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.225 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.226 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.227 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-04-25T16:13:05.229 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.230 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:05.231 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.232 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.233 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-04-25T16:13:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-04-25T16:13:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.234 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T16:13:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T16:13:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-04-25T16:13:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-04-25T16:13:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.236 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T16:13:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.237 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-04-25T16:13:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-04-25T16:13:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-04-25T16:13:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T16:13:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T16:13:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.239 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-04-25T16:13:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-04-25T16:13:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-04-25T16:13:05.240 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T16:13:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-04-25T16:13:05.241 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-04-25T16:13:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-04-25T16:13:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.242 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T16:13:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-04-25T16:13:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-04-25T16:13:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.243 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T16:13:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T16:13:05.244 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-04-25T16:13:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-04-25T16:13:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.245 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T16:13:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.246 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-04-25T16:13:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T16:13:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-04-25T16:13:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.247 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.248 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.249 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-04-25T16:13:05.250 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-04-25T16:13:05.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-04-25T16:13:05.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.253 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.254 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-04-25T16:13:05.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-04-25T16:13:05.256 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-04-25T16:13:05.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-04-25T16:13:05.258 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-04-25T16:13:05.259 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-04-25T16:13:05.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.262 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.263 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.264 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.265 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.266 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.267 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.268 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.269 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T16:13:05.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-04-25T16:13:05.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.272 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.273 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.274 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-04-25T16:13:05.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-04-25T16:13:05.276 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.277 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-04-25T16:13:05.278 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.279 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.280 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.281 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.282 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.283 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.284 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T16:13:05.285 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T16:13:05.286 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-04-25T16:13:05.287 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.289 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T16:13:05.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.291 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-04-25T16:13:05.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-04-25T16:13:05.293 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-04-25T16:13:05.294 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-04-25T16:13:05.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.296 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.297 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-04-25T16:13:05.298 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.300 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.301 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.302 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.303 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.304 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.305 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.306 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.307 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.308 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.310 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.311 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-04-25T16:13:05.312 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.314 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.315 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.316 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.317 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.318 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.320 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.321 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.322 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.323 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.324 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.325 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.326 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.327 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.328 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.329 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.330 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.331 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.332 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.333 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.334 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.335 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.336 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.337 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T16:13:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T16:13:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.338 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.339 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T16:13:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T16:13:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.340 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "[]", 2024-04-25T16:13:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.341 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T16:13:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T16:13:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.342 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-04-25T16:13:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.343 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.344 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T16:13:05.344 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T16:13:05.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.344 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.345 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T16:13:05.346 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.347 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.348 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T16:13:05.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.351 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.352 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.353 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.356 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T16:13:05.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.358 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T16:13:05.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T16:13:05.360 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T16:13:05.361 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.362 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.364 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T16:13:05.365 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.366 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-04-25T16:13:05.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-04-25T16:13:05.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-04-25T16:13:05.369 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.370 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.372 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.373 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.376 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.377 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.379 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.381 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.382 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.383 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.385 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.386 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.387 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.388 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.389 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.390 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.391 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.392 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.393 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.394 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.395 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T16:13:05.396 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T16:13:05.397 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.398 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.399 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.400 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.401 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.402 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.403 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.404 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.405 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.406 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.407 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.409 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.409 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.410 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.411 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.412 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.414 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.415 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.416 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.417 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.418 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-04-25T16:13:05.419 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.420 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.421 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.423 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.424 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.425 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.426 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.427 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.428 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.429 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.430 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.431 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-04-25T16:13:05.432 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.433 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.434 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.435 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.436 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.437 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.438 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.439 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.440 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.441 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.442 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.443 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.445 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.448 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.449 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.449 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.449 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.449 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.449 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.449 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.449 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.450 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.451 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.452 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.453 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.454 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.455 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.456 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.457 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.458 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.459 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.460 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.462 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.463 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.465 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.466 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.467 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.468 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.469 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.470 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.471 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.472 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.473 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.474 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.475 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.476 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.477 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.478 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.480 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.482 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.483 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.484 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.485 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.486 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.487 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.488 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.490 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.491 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.492 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.493 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.494 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.495 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.496 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.497 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.498 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.500 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.502 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-04-25T16:13:05.503 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.504 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.505 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.507 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.508 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.509 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.510 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.511 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.512 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.513 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.514 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.515 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.517 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-04-25T16:13:05.518 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.519 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.520 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.521 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.522 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.523 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.524 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.525 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.526 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.527 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.528 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.530 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T16:13:05.531 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.532 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.533 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.535 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.536 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.537 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.538 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.539 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.540 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.541 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.542 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-04-25T16:13:05.543 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.544 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.545 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T16:13:05.546 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.547 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.548 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.548 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.548 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.548 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.548 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.548 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.548 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.548 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.548 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.549 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.549 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.549 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.549 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.549 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.549 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.549 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.550 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.550 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.550 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.550 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.550 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.551 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.551 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.551 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.551 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.551 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.551 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.552 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.552 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.552 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.552 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.552 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.552 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.553 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.553 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.553 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.553 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.553 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.553 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.554 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.554 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.554 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.554 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.554 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.554 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.555 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.555 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.555 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.555 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.555 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.555 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.556 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.556 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.556 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.556 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.556 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.557 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.557 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.557 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.557 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.557 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.558 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.558 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.558 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.558 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-04-25T16:13:05.558 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-04-25T16:13:05.559 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.559 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.559 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.559 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.559 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.559 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.559 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.559 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.560 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.560 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.560 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.560 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-04-25T16:13:05.560 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T16:13:05.561 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.561 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.561 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.561 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.561 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.561 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.562 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.562 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.562 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.562 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.563 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.563 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.563 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.563 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.563 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.563 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.564 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.564 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.564 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.564 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.564 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.564 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.565 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.565 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-04-25T16:13:05.565 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-04-25T16:13:05.565 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.565 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.566 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.566 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.566 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.566 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.566 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.567 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.567 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.567 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.567 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.567 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.567 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.568 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.568 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-04-25T16:13:05.568 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.568 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.568 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.569 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-04-25T16:13:05.569 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-04-25T16:13:05.569 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.569 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.569 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.569 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.570 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.570 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.570 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.570 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.570 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.570 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.571 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.571 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T16:13:05.571 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T16:13:05.571 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.572 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.572 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.572 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.572 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.572 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.572 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.573 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.573 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.573 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.573 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.573 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.574 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.574 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.574 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.574 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.574 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.575 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.575 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.575 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.575 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.575 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.575 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.576 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.576 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.576 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.576 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.576 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.576 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.577 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.577 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.577 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.577 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.577 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.578 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.578 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.578 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.578 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.578 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.578 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.579 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.580 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.581 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.582 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.583 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T16:13:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T16:13:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.584 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.585 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.586 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.588 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.589 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.590 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.591 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.592 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.593 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.594 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.595 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.596 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.597 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.598 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.599 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.600 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.601 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.602 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.603 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.604 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.605 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.606 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.607 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-04-25T16:13:05.608 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.609 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.610 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-04-25T16:13:05.614 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.615 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-04-25T16:13:05.616 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-04-25T16:13:05.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-04-25T16:13:05.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.620 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.621 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.622 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-25T16:13:05.623 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.624 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.625 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.626 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.627 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.629 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.630 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.631 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.633 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.634 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.635 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.637 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.638 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.639 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.640 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.641 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.642 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.643 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.644 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T16:13:05.646 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-04-25T16:13:05.647 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.648 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.649 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.650 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.651 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.652 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.653 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.654 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.655 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.656 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.657 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T16:13:05.658 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.659 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.660 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.661 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.662 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.663 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.664 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.665 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.667 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.668 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.669 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.670 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.671 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.672 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.673 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.674 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.675 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.676 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.676 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.676 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.676 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.676 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.676 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-04-25T16:13:05.676 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-04-25T16:13:05.676 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.677 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.677 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.677 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T16:13:05.677 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.677 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.677 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.677 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.677 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.678 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.678 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.678 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.678 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.678 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.678 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-04-25T16:13:05.678 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.679 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.679 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.679 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.679 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.679 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.679 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.679 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.680 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.680 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.680 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.680 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.680 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.680 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.681 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.681 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.681 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.681 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.681 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.681 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.682 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.683 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.684 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.685 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.686 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.686 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.686 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.686 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.686 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.686 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.687 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.687 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.687 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.687 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.687 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.688 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.689 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.691 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.692 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.693 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.694 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-04-25T16:13:05.695 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.696 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.697 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.698 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.700 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.701 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.702 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.704 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.705 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.706 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.707 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.708 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.709 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.710 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.711 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.712 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-04-25T16:13:05.712 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-04-25T16:13:05.712 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.712 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.712 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.712 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T16:13:05.712 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.712 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.713 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.713 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.713 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.713 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.713 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.713 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.713 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.713 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.714 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T16:13:05.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.714 INFO:teuthology.orchestra.run.smithi031.stdout: "gid": 24119, 2024-04-25T16:13:05.714 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "y", 2024-04-25T16:13:05.714 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr_features": 4540701547738038271, 2024-04-25T16:13:05.714 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-04-25T16:13:05.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-04-25T16:13:05.715 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.715 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.715 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.715 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:05.715 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:05.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.715 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.715 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.716 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:05.716 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.716 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.716 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.716 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T16:13:05.716 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.716 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.716 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.717 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.718 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.718 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.718 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.718 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.718 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.718 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.718 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.718 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.719 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.719 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.719 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.719 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.719 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.719 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.719 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.719 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.720 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.720 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.720 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.720 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.720 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.720 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.720 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.721 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.721 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.721 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.721 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.721 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.721 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.721 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.721 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.722 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.722 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.722 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.722 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.722 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.722 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.722 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.722 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.723 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.723 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.723 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.723 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.723 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.723 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.723 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.723 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.723 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.724 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.724 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.724 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.724 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.724 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.724 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.724 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.724 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-04-25T16:13:05.725 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.726 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.726 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.726 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.726 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.726 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.726 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.726 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-04-25T16:13:05.726 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.727 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.728 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.729 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.730 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.731 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.732 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-04-25T16:13:05.733 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.734 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.735 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.736 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.737 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:05.738 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.740 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.741 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-04-25T16:13:05.742 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.743 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.744 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.745 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.746 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.747 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.748 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.749 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-04-25T16:13:05.750 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T16:13:05.751 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T16:13:05.752 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.753 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.754 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.755 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-04-25T16:13:05.756 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-04-25T16:13:05.757 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-04-25T16:13:05.758 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-04-25T16:13:05.759 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.760 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.761 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.762 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.763 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.764 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.765 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.766 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.767 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.768 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.769 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.770 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.771 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-04-25T16:13:05.772 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-04-25T16:13:05.773 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-04-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-04-25T16:13:05.775 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.776 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.777 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.778 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.779 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.780 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.781 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-04-25T16:13:05.782 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.783 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.784 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.785 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-04-25T16:13:05.786 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-04-25T16:13:05.787 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-04-25T16:13:05.788 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-04-25T16:13:05.789 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.790 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-04-25T16:13:05.791 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-04-25T16:13:05.792 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.793 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.794 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.795 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T16:13:05.796 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.797 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.798 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T16:13:05.799 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T16:13:05.800 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.801 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-04-25T16:13:05.802 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-04-25T16:13:05.803 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.804 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.805 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T16:13:05.806 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.807 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.809 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.810 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.811 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.811 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.811 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.811 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.811 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T16:13:05.811 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.811 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.811 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.812 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.813 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.814 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.815 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.816 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T16:13:05.818 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.819 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.820 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.821 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.822 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.823 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.824 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.825 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.826 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.827 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.828 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.829 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-04-25T16:13:05.830 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.831 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.832 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.833 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.834 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.835 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.835 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.835 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.835 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T16:13:05.835 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T16:13:05.835 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.835 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.835 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.835 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T16:13:05.836 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T16:13:05.837 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.837 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.837 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.837 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.837 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.837 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.837 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.837 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.838 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.838 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.838 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.838 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.838 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T16:13:05.838 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T16:13:05.838 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.839 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.839 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.839 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.839 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.839 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.839 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.839 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.839 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.840 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.840 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.840 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.840 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T16:13:05.840 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T16:13:05.840 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.840 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.841 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.841 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.841 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.841 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.841 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.841 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.841 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.842 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.842 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.842 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.842 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T16:13:05.842 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T16:13:05.842 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.842 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.842 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.843 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.843 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.843 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.843 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.843 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.843 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.843 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.844 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.844 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.844 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T16:13:05.844 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T16:13:05.844 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.844 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.844 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.844 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.845 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T16:13:05.846 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T16:13:05.846 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.846 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.846 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.846 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.846 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.846 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.847 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.847 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.847 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.847 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.847 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.847 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.847 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T16:13:05.847 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T16:13:05.848 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.848 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.848 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.848 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.848 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.848 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.848 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.848 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.849 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.849 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.849 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.849 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.849 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T16:13:05.849 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T16:13:05.849 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.849 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.850 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.851 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T16:13:05.852 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-04-25T16:13:05.853 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T16:13:05.854 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T16:13:05.855 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.856 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T16:13:05.857 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T16:13:05.858 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.859 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "[]", 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.860 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.861 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.862 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.863 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.864 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.865 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.866 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T16:13:05.867 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T16:13:05.868 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T16:13:05.869 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T16:13:05.870 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.871 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.872 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T16:13:05.873 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.874 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T16:13:05.875 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T16:13:05.876 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T16:13:05.877 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.878 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.879 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.880 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.880 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.880 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.880 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.880 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T16:13:05.880 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T16:13:05.881 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.881 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.881 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.881 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.881 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.881 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.881 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.882 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.882 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.882 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.882 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.882 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.882 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T16:13:05.883 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T16:13:05.883 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.883 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.883 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.883 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.883 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.884 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.884 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.884 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.884 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.884 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.884 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.884 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.885 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T16:13:05.885 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T16:13:05.885 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.885 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.885 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.885 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.886 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.886 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.886 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.886 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.886 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.886 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.887 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.887 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.887 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T16:13:05.887 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T16:13:05.887 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.887 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.888 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.888 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.888 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.888 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.888 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.889 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.889 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.889 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.889 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.889 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.889 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T16:13:05.889 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T16:13:05.889 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.889 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.890 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.890 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:05.890 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.890 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.890 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.890 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.891 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.891 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.891 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.891 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.891 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-04-25T16:13:05.892 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-04-25T16:13:05.892 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.892 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.892 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.892 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.892 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.893 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.893 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.893 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.893 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.893 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.893 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.894 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.894 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-04-25T16:13:05.894 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-04-25T16:13:05.894 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.894 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.895 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.895 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.895 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.895 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.895 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.895 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.896 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.896 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.896 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.896 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.896 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-04-25T16:13:05.896 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-04-25T16:13:05.897 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.897 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.897 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.897 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.897 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.897 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.898 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.898 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-04-25T16:13:05.898 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.898 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.898 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.899 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.899 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-04-25T16:13:05.899 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-04-25T16:13:05.899 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.899 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.899 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.900 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-04-25T16:13:05.900 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.900 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.900 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.900 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.900 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.901 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.901 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.901 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.901 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T16:13:05.901 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T16:13:05.901 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.902 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.902 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.902 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.902 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.902 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.902 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.903 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.903 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.903 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.903 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.903 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.903 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.904 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.904 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.904 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.904 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.904 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.904 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.905 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.905 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.905 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.905 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.905 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.905 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.906 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.906 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.906 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.906 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.906 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.907 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.907 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.907 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.907 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.907 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.907 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.908 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.908 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.908 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.908 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.908 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.908 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.909 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.909 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.909 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.909 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.909 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.910 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.910 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.910 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.910 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.910 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.910 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.911 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.911 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.911 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.911 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.911 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.911 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.911 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.911 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.912 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.912 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.912 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.912 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.912 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.912 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.912 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.912 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.913 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.913 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.913 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.913 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.913 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.913 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.913 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.913 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.913 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.914 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.914 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.914 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.914 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.914 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-04-25T16:13:05.914 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-04-25T16:13:05.914 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.915 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.915 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.915 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.915 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.915 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.915 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.915 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-04-25T16:13:05.915 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.916 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.917 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.918 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.918 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.918 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.918 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T16:13:05.918 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T16:13:05.918 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.918 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.918 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.919 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.920 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.921 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.922 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T16:13:05.923 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-04-25T16:13:05.924 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.925 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.926 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.927 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.928 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.929 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.930 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.931 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.932 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.933 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.934 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.935 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.936 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.937 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.938 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.939 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.941 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.942 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.943 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.944 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-04-25T16:13:05.945 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-04-25T16:13:05.946 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.946 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.946 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.946 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T16:13:05.946 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.946 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.946 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.946 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.947 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:05.948 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.949 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.950 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.951 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.952 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.953 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.954 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.955 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.956 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.957 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.958 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.959 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-04-25T16:13:05.960 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.961 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.962 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.963 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.964 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.965 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.966 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.967 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.968 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.969 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.970 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.971 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.972 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.973 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.974 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.975 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.976 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.977 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.978 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.979 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.980 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.981 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.982 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.983 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.984 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.985 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.986 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.987 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.988 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:05.989 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T16:13:05.990 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.991 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:05.992 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:05.993 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:05.994 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:05.995 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-04-25T16:13:05.996 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.997 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.998 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:05.999 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.000 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.001 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.002 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.003 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.004 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.005 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.006 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.007 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.008 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.009 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.010 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.011 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.013 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.014 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.015 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.016 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.017 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.018 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.019 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.020 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.021 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.022 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.023 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.024 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.025 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.026 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.027 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.028 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.029 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.029 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.029 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.029 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.029 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.029 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-25T16:13:06.029 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-04-25T16:13:06.030 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-04-25T16:13:06.030 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.030 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-04-25T16:13:06.030 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.030 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.030 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.030 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.030 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.030 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.031 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.032 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.033 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.034 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.035 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.036 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.037 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.038 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.039 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.040 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.041 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.042 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.043 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.044 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.045 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.046 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.047 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.047 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.047 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.047 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.047 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.047 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.047 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.047 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.047 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.048 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.049 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.050 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:06.051 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.052 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.053 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.054 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T16:13:06.055 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.056 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.057 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.058 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.059 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.060 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-04-25T16:13:06.061 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.062 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.063 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.064 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.065 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.066 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.067 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.068 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.069 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.070 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.071 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.072 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T16:13:06.073 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.074 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.075 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T16:13:06.076 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.077 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-04-25T16:13:06.078 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T16:13:06.079 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.080 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.081 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.082 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.083 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.084 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.085 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.086 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.087 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.088 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.089 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.090 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.091 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.092 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.093 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.094 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.095 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.096 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.097 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.098 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.099 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.100 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.101 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.102 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.103 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.104 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.105 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.106 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.107 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.108 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.109 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.110 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.111 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.112 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.113 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.114 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.114 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.114 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.114 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.114 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.114 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-04-25T16:13:06.115 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.115 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.115 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.115 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.115 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.115 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.115 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.115 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.116 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.117 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.118 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.119 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.120 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.121 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.122 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.123 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.124 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.125 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.126 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.127 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.128 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.129 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-04-25T16:13:06.130 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-04-25T16:13:06.131 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-04-25T16:13:06.132 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.133 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.134 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-04-25T16:13:06.135 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-04-25T16:13:06.136 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.137 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-04-25T16:13:06.138 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-04-25T16:13:06.139 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-04-25T16:13:06.140 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.141 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.142 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.143 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.144 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.145 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.146 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.147 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.148 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.150 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.151 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.152 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.153 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.154 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.155 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.156 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.157 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.158 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.159 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.160 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.161 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.162 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.163 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.164 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.165 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.166 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.167 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.168 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.169 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.170 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.171 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.172 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.173 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.174 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.175 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.176 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.177 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.178 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.179 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.180 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.181 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.182 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.183 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.184 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.185 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.186 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.187 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.188 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.189 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.190 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.191 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.192 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.193 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.194 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.195 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.196 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.197 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.198 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.199 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.200 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.201 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.202 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.203 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.204 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.205 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.206 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.207 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.207 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.207 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.207 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.207 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.207 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.207 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.207 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-04-25T16:13:06.207 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-04-25T16:13:06.208 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.208 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.208 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.208 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-04-25T16:13:06.208 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.208 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.208 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.208 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T16:13:06.208 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-04-25T16:13:06.209 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.210 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.211 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.212 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-04-25T16:13:06.213 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-04-25T16:13:06.214 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.215 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.216 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.217 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.218 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.219 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.220 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.221 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.222 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.223 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.224 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.225 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.225 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.225 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.225 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.225 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.225 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.225 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-04-25T16:13:06.225 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-04-25T16:13:06.226 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.226 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.226 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.226 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T16:13:06.226 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.226 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.226 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.226 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: "modules": [ 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard", 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: "iostat", 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: "nfs", 2024-04-25T16:13:06.227 INFO:teuthology.orchestra.run.smithi031.stdout: "restful" 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "available_modules": [ 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "alerts", 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.228 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How frequently to reexamine health status", 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.229 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.230 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.231 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.232 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.233 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.234 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_destination": { 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_destination", 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email address to send alerts to", 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.235 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_from_name": { 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_from_name", 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "Ceph", 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Email From: name", 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.236 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_host": { 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_host", 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP server", 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.237 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_password": { 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_password", 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Password to authenticate with", 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.238 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_port": { 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_port", 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "465", 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP port", 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.239 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_sender": { 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_sender", 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SMTP envelope sender", 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.240 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_ssl": { 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_ssl", 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.241 INFO:teuthology.orchestra.run.smithi031.stdout: "smtp_user": { 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "smtp_user", 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "User to authenticate as", 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.242 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.243 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "balancer", 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "active": { 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "active", 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.244 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically balance PGs across cluster", 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_time": { 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_time", 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.245 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0000", 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "beginning time of day to automatically balance", 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "begin_weekday": { 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "begin_weekday", 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:06.246 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_max_iterations": { 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_max_iterations", 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.247 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "25", 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "250", 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_metrics": { 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_metrics", 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.248 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "pgs,objects,bytes", 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "crush_compat_step": { 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crush_compat_step", 2024-04-25T16:13:06.249 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.5", 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0.001", 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "0.999", 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "aggressiveness of optimization", 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "end_time": { 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_time", 2024-04-25T16:13:06.250 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2359", 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ending time of day to automatically balance", 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "end_weekday": { 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "end_weekday", 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:06.251 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "0", 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "6", 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.252 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.253 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.254 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.255 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.256 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "min_score": { 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_score", 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.257 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.258 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "upmap", 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "crush-compat", 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "none", 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "read", 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap", 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap-read" 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Balancer mode", 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.259 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_ids": { 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_ids", 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-25T16:13:06.260 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.261 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.262 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_deviation": { 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_deviation", 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:06.263 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "upmap_max_optimizations": { 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "upmap_max_optimizations", 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.264 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm", 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.265 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_down_multiplier": { 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_down_multiplier", 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.266 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_refresh_rate": { 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_refresh_rate", 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "20", 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.267 INFO:teuthology.orchestra.run.smithi031.stdout: "agent_starting_port": { 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "agent_starting_port", 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4721", 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_ptrace": { 2024-04-25T16:13:06.268 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_ptrace", 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_interval": { 2024-04-25T16:13:06.269 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_interval", 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.270 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "autotune_memory_target_ratio": { 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "autotune_memory_target_ratio", 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0.7", 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.271 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm_log_destination": { 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cephadm_log_destination", 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "file", 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "file,syslog", 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "syslog" 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-25T16:13:06.272 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "cgroups_split": { 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cgroups_split", 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-25T16:13:06.273 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "config_checks_enabled": { 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_checks_enabled", 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.274 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "config_dashboard": { 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_dashboard", 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.275 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_alertmanager": { 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_alertmanager", 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.276 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_base": { 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_base", 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.277 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Container image name, without the tag", 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_elasticsearch": { 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_elasticsearch", 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.278 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "elasticsearch container image", 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_grafana": { 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_grafana", 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.279 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_haproxy": { 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_haproxy", 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.280 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "HAproxy container image", 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_agent": { 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_agent", 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.281 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger agent container image", 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_collector": { 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_collector", 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.282 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger collector container image", 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_jaeger_query": { 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_jaeger_query", 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.283 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Jaeger query container image", 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_keepalived": { 2024-04-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_keepalived", 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Keepalived container image", 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.285 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_loki": { 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_loki", 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Loki container image", 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.286 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_node_exporter": { 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_node_exporter", 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.287 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_nvmeof": { 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_nvmeof", 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Nvme-of container image", 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.288 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_prometheus": { 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_prometheus", 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Prometheus container image", 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.289 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_promtail": { 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_promtail", 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Promtail container image", 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.290 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_samba": { 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_samba", 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Samba/SMB container image", 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.291 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "container_image_snmp_gateway": { 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_image_snmp_gateway", 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "SNMP Gateway container image", 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.292 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "container_init": { 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "container_init", 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Run podman/docker with `--init`", 2024-04-25T16:13:06.293 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "daemon_cache_timeout": { 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "daemon_cache_timeout", 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.294 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "default_cephadm_command_timeout": { 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_cephadm_command_timeout", 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "900", 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.295 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "default_registry": { 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "default_registry", 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "docker.io", 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.296 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "device_cache_timeout": { 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_cache_timeout", 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1800", 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.297 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache device inventory", 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "device_enhanced_scan": { 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_enhanced_scan", 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.298 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "facts_cache_timeout": { 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "facts_cache_timeout", 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.299 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "seconds to cache host facts data", 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "grafana_dashboards_path": { 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "grafana_dashboards_path", 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.300 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "host_check_interval": { 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "host_check_interval", 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.301 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to perform a host check", 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "hw_monitoring": { 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hw_monitoring", 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.302 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "inventory_list_all": { 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "inventory_list_all", 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.303 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.304 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "log_refresh_metadata": { 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_refresh_metadata", 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.305 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.306 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.307 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.308 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.309 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.310 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.311 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "*", 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "max_count_per_host": { 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_count_per_host", 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.312 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of daemons per service per host", 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "max_osd_draining_count": { 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_osd_draining_count", 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.313 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "migration_current": { 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "migration_current", 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.314 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "internal - do not modify", 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "mode": { 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mode", 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.315 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "root", 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm-package", 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "root" 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "mode for remote execution of cephadm", 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.316 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "oob_default_addr": { 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "oob_default_addr", 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "169.254.1.1", 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.317 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "prometheus_alerts_path": { 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus_alerts_path", 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.318 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_insecure": { 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_insecure", 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.319 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_password": { 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_password", 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.320 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_url": { 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_url", 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.321 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "registry_username": { 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "registry_username", 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.322 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "secure_monitoring_stack": { 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "secure_monitoring_stack", 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.323 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "service_discovery_port": { 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "service_discovery_port", 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8765", 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.324 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "cephadm service discovery port", 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.325 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "ssh_config_file": { 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssh_config_file", 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.326 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "use_agent": { 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_agent", 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.327 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "use_repo_digest": { 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "use_repo_digest", 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.328 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_failed_host_check": { 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_failed_host_check", 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.329 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if the host check fails", 2024-04-25T16:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_daemons": { 2024-04-25T16:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_daemons", 2024-04-25T16:13:06.330 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-25T16:13:06.331 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_on_stray_hosts": { 2024-04-25T16:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_on_stray_hosts", 2024-04-25T16:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.332 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.333 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crash", 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.334 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.335 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.336 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.337 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.338 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "retain_interval": { 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retain_interval", 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "31536000", 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.339 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.340 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_recent_interval": { 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_recent_interval", 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1209600", 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.341 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dashboard", 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.342 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_HOST": { 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.343 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.344 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_ENABLED": { 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_ENABLED", 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.345 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.346 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "ENABLE_BROWSABLE_API": { 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.347 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.348 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-25T16:13:06.349 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-25T16:13:06.350 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-25T16:13:06.351 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-25T16:13:06.352 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-25T16:13:06.353 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-25T16:13:06.354 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-25T16:13:06.355 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_PASSWORD": { 2024-04-25T16:13:06.356 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-25T16:13:06.357 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_URL": { 2024-04-25T16:13:06.358 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_URL", 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_API_USERNAME": { 2024-04-25T16:13:06.359 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-25T16:13:06.360 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.361 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.362 INFO:teuthology.orchestra.run.smithi031.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-25T16:13:06.363 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.364 INFO:teuthology.orchestra.run.smithi031.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "[]", 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.365 INFO:teuthology.orchestra.run.smithi031.stdout: "MULTICLUSTER_CONFIG": { 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "{}", 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.366 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_HOST": { 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.370 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.372 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.373 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.374 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.375 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_ENABLED": { 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.376 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.377 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.378 INFO:teuthology.orchestra.run.smithi031.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8", 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-25T16:13:06.379 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "45", 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.380 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ACCESS_KEY": { 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.381 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "admin", 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.382 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SECRET_KEY": { 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.383 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "RGW_API_SSL_VERIFY": { 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.384 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "UNSAFE_TLS_v1_2": { 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.385 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.386 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.387 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.388 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "cross_origin_url": { 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cross_origin_url", 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.389 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "crt_file": { 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "crt_file", 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.390 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "debug": { 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "debug", 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Enable/disable debug options", 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.391 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "jwt_token_ttl": { 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "jwt_token_ttl", 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "28800", 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.392 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.393 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.394 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.395 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.396 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.397 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.398 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "motd": { 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "motd", 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "The message of the day", 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect_resolve_ip_addr": { 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "redirect_resolve_ip_addr", 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.400 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.401 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8080", 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.402 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T16:13:06.403 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl_server_port": { 2024-04-25T16:13:06.404 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl_server_port", 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8443", 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T16:13:06.405 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "redirect", 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "redirect" 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.406 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.407 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "url_prefix": { 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url_prefix", 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.408 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_monitoring": { 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_monitoring", 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.409 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "monitor device health metrics", 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.410 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.411 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.412 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.413 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.414 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "mark_out_threshold": { 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mark_out_threshold", 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.415 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "2419200", 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "pool_name": { 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pool_name", 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.416 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "device_health_metrics", 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "retention_period": { 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "retention_period", 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.417 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15552000", 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long to retain device health metrics", 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_frequency": { 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_frequency", 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "self_heal": { 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "self_heal", 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.419 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.420 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how frequently to wake up and check device health", 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.421 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "warn_threshold": { 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "warn_threshold", 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.422 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7257600", 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "diskprediction_local", 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.423 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.424 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.425 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.426 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.427 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.428 INFO:teuthology.orchestra.run.smithi031.stdout: "predict_interval": { 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predict_interval", 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "86400", 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.429 INFO:teuthology.orchestra.run.smithi031.stdout: "predictor_model": { 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "predictor_model", 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "prophetstor", 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.430 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:06.431 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:06.431 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.431 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.431 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.431 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "600", 2024-04-25T16:13:06.431 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.431 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.431 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.431 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.432 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.432 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.432 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.432 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.432 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.432 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.432 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.433 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.433 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.433 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.433 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.433 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.433 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.434 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.434 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.434 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.434 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.434 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.435 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.435 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.435 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "influx", 2024-04-25T16:13:06.435 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": false, 2024-04-25T16:13:06.435 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "influxdb python module not found", 2024-04-25T16:13:06.435 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.435 INFO:teuthology.orchestra.run.smithi031.stdout: "batch_size": { 2024-04-25T16:13:06.436 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "batch_size", 2024-04-25T16:13:06.436 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.436 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.436 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.436 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5000", 2024-04-25T16:13:06.436 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.436 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.437 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.437 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-25T16:13:06.437 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.437 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.437 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.437 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "database": { 2024-04-25T16:13:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "database", 2024-04-25T16:13:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "ceph", 2024-04-25T16:13:06.438 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-25T16:13:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.439 INFO:teuthology.orchestra.run.smithi031.stdout: "hostname": { 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "hostname", 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server hostname", 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.440 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "30", 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "5", 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.441 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.442 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.443 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.444 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.445 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.446 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "password": { 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "password", 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.447 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "password of InfluxDB server user", 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "port": { 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "port", 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "8086", 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.448 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "InfluxDB server port", 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.449 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.450 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.450 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.450 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.450 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.450 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.450 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.451 INFO:teuthology.orchestra.run.smithi031.stdout: "ssl": { 2024-04-25T16:13:06.451 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ssl", 2024-04-25T16:13:06.451 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.451 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.451 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.451 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "false", 2024-04-25T16:13:06.451 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.451 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "threads": { 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threads", 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:06.452 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "32", 2024-04-25T16:13:06.453 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.453 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-25T16:13:06.453 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.453 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.453 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.453 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.453 INFO:teuthology.orchestra.run.smithi031.stdout: "username": { 2024-04-25T16:13:06.454 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "username", 2024-04-25T16:13:06.454 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.454 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.454 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.454 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.454 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.454 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.454 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.454 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "username of InfluxDB server user", 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "verify_ssl": { 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "verify_ssl", 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "true", 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-25T16:13:06.455 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "insights", 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.456 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.457 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.459 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.460 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.461 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.462 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.463 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.463 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "iostat", 2024-04-25T16:13:06.463 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.463 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.463 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.463 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.464 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.464 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.464 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.464 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.464 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.464 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.464 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.465 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.466 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.467 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.468 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.469 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "k8sevents", 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_event_retention_days": { 2024-04-25T16:13:06.470 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "ceph_event_retention_days", 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "7", 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "config_check_secs": { 2024-04-25T16:13:06.471 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "config_check_secs", 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "10", 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.472 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.473 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.474 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.475 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.476 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.477 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.478 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "localpool", 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "failure_domain": { 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "failure_domain", 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "host", 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.479 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "failure domain for any created local pool", 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.480 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.481 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.482 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.483 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.484 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "min_size": { 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "min_size", 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.485 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default min_size for any created local pool", 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "num_rep": { 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "num_rep", 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3", 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default replica count for any created local pool", 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_num": { 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_num", 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.487 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "128", 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "default pg_num for any created local pool", 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "prefix": { 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prefix", 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.488 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "name prefix for any created local pool", 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.489 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "subtree": { 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "subtree", 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "rack", 2024-04-25T16:13:06.490 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mds_autoscaler", 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.491 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.492 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.493 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.494 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.495 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.496 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.497 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirroring", 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.498 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.499 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.500 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.501 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.502 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.503 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "nfs", 2024-04-25T16:13:06.504 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.505 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.506 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.507 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.508 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.509 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.510 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: "fail_fs": { 2024-04-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "fail_fs", 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.512 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.513 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.514 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.515 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.516 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.517 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator": { 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "orchestrator", 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "cephadm", 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "rook", 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "test_orchestrator" 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Orchestrator backend", 2024-04-25T16:13:06.518 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.519 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_perf_query", 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.520 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.521 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.522 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.523 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.524 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.525 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "osd_support", 2024-04-25T16:13:06.526 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.527 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.528 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.529 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.530 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.531 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.532 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "pg_autoscaler", 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.533 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.534 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.535 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.536 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.537 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.538 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.539 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "threshold": { 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "threshold", 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "3.0", 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1.0", 2024-04-25T16:13:06.540 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "scaling threshold", 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "progress", 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.541 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_pg_recovery_event": { 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_pg_recovery_event", 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.544 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.545 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.546 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.547 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "max_completed_events": { 2024-04-25T16:13:06.548 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_completed_events", 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "50", 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "number of past completed events to remember", 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "sleep_interval": { 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sleep_interval", 2024-04-25T16:13:06.549 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "5", 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "how long the module is going to sleep", 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.550 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.551 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "prometheus", 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "cache": { 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "cache", 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.552 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "exclude_perf_counters": { 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "exclude_perf_counters", 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.553 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.554 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.555 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.556 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.557 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.558 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools": { 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools", 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.559 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-25T16:13:06.560 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "300", 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "scrape_interval": { 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "scrape_interval", 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.561 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15.0", 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T16:13:06.562 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "::", 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T16:13:06.563 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "9283", 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.564 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "stale_cache_strategy": { 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stale_cache_strategy", 2024-04-25T16:13:06.565 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "log", 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_behaviour": { 2024-04-25T16:13:06.566 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_behaviour", 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "default", 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "default", 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "error" 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.567 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "standby_error_status_code": { 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "standby_error_status_code", 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "500", 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "400", 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "599", 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.568 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.569 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.570 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.571 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.572 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.573 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_snap_create": { 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_snap_create", 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10", 2024-04-25T16:13:06.574 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "mirror_snapshot_schedule": { 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "mirror_snapshot_schedule", 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.575 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.576 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "trash_purge_schedule": { 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "trash_purge_schedule", 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.577 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "restful", 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.578 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "enable_auth": { 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enable_auth", 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.579 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "key_file": { 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "key_file", 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.580 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.581 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.582 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.583 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.584 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.585 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "server_addr": { 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_addr", 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.586 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "server_port": { 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "server_port", 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.587 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.588 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rgw", 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.589 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.590 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.591 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.592 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.593 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.594 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.595 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rook", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.596 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.597 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.598 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.599 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.600 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.601 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "storage_class": { 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "storage_class", 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "local", 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "selftest", 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.603 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.604 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.605 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.606 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.607 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "roption1": { 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption1", 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.608 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "roption2": { 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "roption2", 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.609 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "xyz", 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption1": { 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption1", 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.610 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption2": { 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption2", 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.611 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption3": { 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption3", 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "float", 2024-04-25T16:13:06.612 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption4": { 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption4", 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.613 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption5": { 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption5", 2024-04-25T16:13:06.614 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption6": { 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption6", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.615 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "rwoption7": { 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rwoption7", 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.616 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "1", 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "42", 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.617 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "testkey": { 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testkey", 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.618 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "testlkey": { 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testlkey", 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.619 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "testnewline": { 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "testnewline", 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.620 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snap_schedule", 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.621 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "allow_m_granularity": { 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "allow_m_granularity", 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "allow minute scheduled snapshots", 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.622 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "dump_on_update": { 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "dump_on_update", 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "dump database to debug log on update", 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.623 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.624 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.625 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.626 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.627 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.628 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.629 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.630 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.630 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "stats", 2024-04-25T16:13:06.630 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.630 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.630 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.630 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.634 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.635 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.636 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.637 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.638 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.639 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "status", 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.640 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.641 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.642 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.643 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.644 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.645 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telegraf", 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.646 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "address": { 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "address", 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.647 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "15", 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.648 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.649 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.650 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.651 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.652 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.653 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.654 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "telemetry", 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_basic": { 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_basic", 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.655 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_crash": { 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_crash", 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.656 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_device": { 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_device", 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.657 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_ident": { 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_ident", 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.658 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "channel_perf": { 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "channel_perf", 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.659 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "contact": { 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "contact", 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.660 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "description": { 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "description", 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.661 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "device_url": { 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "device_url", 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.662 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "enabled": { 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "enabled", 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.663 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.664 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "24", 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "8", 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "last_opt_revision": { 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "last_opt_revision", 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.665 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "1", 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard": { 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard", 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.666 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "leaderboard_description": { 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "leaderboard_description", 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.667 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.668 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.669 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.670 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.671 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.672 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "organization": { 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "organization", 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.673 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "proxy": { 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "proxy", 2024-04-25T16:13:06.674 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.675 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "url": { 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "url", 2024-04-25T16:13:06.676 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.677 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "test_orchestrator", 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.678 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.679 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.680 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.681 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.682 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.683 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.684 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.685 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.686 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.687 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.688 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "max_concurrent_clones": { 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "max_concurrent_clones", 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.689 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "4", 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.690 INFO:teuthology.orchestra.run.smithi031.stdout: "periodic_async_work": { 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "periodic_async_work", 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Periodically check for async work", 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.691 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_delay": { 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_delay", 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-25T16:13:06.692 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "snapshot_clone_no_wait": { 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "snapshot_clone_no_wait", 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "True", 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.693 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.694 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix", 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "can_run": true, 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "error_string": "", 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "module_options": { 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "discovery_interval": { 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "discovery_interval", 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "uint", 2024-04-25T16:13:06.695 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "100", 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "identifier": { 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "identifier", 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.696 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "interval": { 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "interval", 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "secs", 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.697 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "60", 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "log_level": { 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_level", 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.698 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.699 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster": { 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster", 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.700 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_cluster_level": { 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_cluster_level", 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "info", 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [ 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "", 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "critical", 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "debug", 2024-04-25T16:13:06.701 INFO:teuthology.orchestra.run.smithi031.stdout: "error", 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "info", 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "warning" 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "log_to_file": { 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "log_to_file", 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "bool", 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.702 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "False", 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "sqlite3_killpoint": { 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "sqlite3_killpoint", 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "dev", 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 1, 2024-04-25T16:13:06.703 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "0", 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_host": { 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_host", 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "", 2024-04-25T16:13:06.704 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_port": { 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_port", 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "int", 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "10051", 2024-04-25T16:13:06.705 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "zabbix_sender": { 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "zabbix_sender", 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "str", 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "level": "advanced", 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "flags": 0, 2024-04-25T16:13:06.706 INFO:teuthology.orchestra.run.smithi031.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: "min": "", 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: "max": "", 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: "enum_allowed": [], 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: "desc": "", 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: "long_desc": "", 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: "tags": [], 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: "see_also": [] 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: "services": { 2024-04-25T16:13:06.707 INFO:teuthology.orchestra.run.smithi031.stdout: "dashboard": "https://172.21.15.31:8443/" 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "always_on_modules": { 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "octopus": [ 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T16:13:06.708 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "pacific": [ 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "quincy": [ 2024-04-25T16:13:06.709 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "reef": [ 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T16:13:06.710 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: ], 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "squid": [ 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "balancer", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "crash", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "devicehealth", 2024-04-25T16:13:06.711 INFO:teuthology.orchestra.run.smithi031.stdout: "orchestrator", 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "pg_autoscaler", 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "progress", 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "rbd_support", 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "status", 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "telemetry", 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "volumes" 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "last_failure_osd_epoch": 3, 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "active_clients": [ 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "devicehealth", 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T16:13:06.712 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 989398132 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "libcephsqlite", 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2463152784 2024-04-25T16:13:06.713 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "rbd_support", 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 2663702827 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.714 INFO:teuthology.orchestra.run.smithi031.stdout: "name": "volumes", 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: "addrvec": [ 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: { 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: "type": "v2", 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: "addr": "172.21.15.31:0", 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: "nonce": 186916348 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2024-04-25T16:13:06.715 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:13:06.716 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:05 smithi186 ceph-mon[33258]: pgmap v704: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 84 MiB/s wr, 226 op/s 2024-04-25T16:13:06.716 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:05 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3129934316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:13:06.716 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:05 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3112364927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:13:06.716 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:05 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3178235717' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T16:13:06.716 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:05 smithi063 ceph-mon[32413]: pgmap v704: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 84 MiB/s wr, 226 op/s 2024-04-25T16:13:06.717 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:05 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3129934316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:13:06.717 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:05 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3112364927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:13:06.717 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:05 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3178235717' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T16:13:06.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f567c0afaf0 msgr2=0x7f567c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:06.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f567c0afaf0 0x7f567c0b2000 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5694102d60 tx=0x7f5684005f90 comp rx=0 tx=0).stop 2024-04-25T16:13:06.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 msgr2=0x7f5694198560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:06.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 0x7f5694198560 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f5688006010 tx=0x7f56880042f0 comp rx=0 tx=0).stop 2024-04-25T16:13:06.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 shutdown_connections 2024-04-25T16:13:06.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f567c0afaf0 0x7f567c0b2000 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:06.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56941080a0 0x7f569419f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:06.718 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56941033a0 0x7f5694198560 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:06.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 --2- 172.21.15.31:0/3178235717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5694102160 0x7f5694198020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:06.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 >> 172.21.15.31:0/3178235717 conn(0x7f56940fd330 msgr2=0x7f5694104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:06.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 shutdown_connections 2024-04-25T16:13:06.719 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:05.187+0000 7f569a80c640 1 -- 172.21.15.31:0/3178235717 wait complete. 2024-04-25T16:13:06.725 DEBUG:tasks.check_counter:Getting stats from a 2024-04-25T16:13:06.725 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-25T16:13:06.726 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:05 smithi031 ceph-mon[29140]: pgmap v704: 97 pgs: 97 active+clean; 10 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 84 MiB/s wr, 226 op/s 2024-04-25T16:13:06.726 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:05 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3129934316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:13:06.726 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:05 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3112364927' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:13:06.726 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:05 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3178235717' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-25T16:13:06.993 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:13:07.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:07 smithi031 ceph-mon[29140]: pgmap v705: 97 pgs: 97 active+clean; 9.7 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 76 MiB/s wr, 222 op/s 2024-04-25T16:13:07.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:07 smithi063 ceph-mon[32413]: pgmap v705: 97 pgs: 97 active+clean; 9.7 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 76 MiB/s wr, 222 op/s 2024-04-25T16:13:07.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:07 smithi186 ceph-mon[33258]: pgmap v705: 97 pgs: 97 active+clean; 9.7 GiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 76 MiB/s wr, 222 op/s 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 535, 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 695, 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 166923, 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 177015, 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 9, 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 4, 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.167381158, 2024-04-25T16:13:09.331 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.035594231, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.135334106, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.007793088, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 693, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.248567901, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000358683 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1172, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000838658, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000715 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:13:09.332 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 120, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 291, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 56423, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 922227, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 49, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.066576652, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.021973563, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.042752079, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000203028, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 290, 2024-04-25T16:13:09.333 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.115977563, 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000399922 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 164, 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000075245, 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000458 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 390240 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 355, 2024-04-25T16:13:09.334 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 346, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 619621, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 198665, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 7, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 5, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.065335232, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.025021071, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.031420588, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.010257902, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 345, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.135491379, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000392728 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:09.335 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 126, 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000082753, 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000656 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 570928, 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 146688 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-04-25T16:13:09.336 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 46, 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.001301878, 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000028301 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 2, 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000007450, 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000003725 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 0, 2024-04-25T16:13:09.337 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 0, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 0, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 11, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 0, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 0, 2024-04-25T16:13:09.338 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 10, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 10, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 10, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 1, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:09.339 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 4, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 234, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:13:09.340 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:13:09.341 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 0 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 1, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 0, 2024-04-25T16:13:09.342 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:09.343 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "evlrg": 0, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 2, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 0, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 0, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 2, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 0, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 2, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 0, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 0, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 2, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "segmjr": 2, 2024-04-25T16:13:09.344 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 0, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 4194304, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 4194892, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 0, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.001426181, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.001426181 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 0 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 13, 2024-04-25T16:13:09.345 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 13, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 0, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 11, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 12, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 1, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 10, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 10, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 0, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 0, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 0, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 0, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 123216, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 223632 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.346 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 0, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 0, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 0, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 0, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 0, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.347 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.348 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.349 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2024-04-25T16:13:09.350 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.351 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.351 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.351 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.351 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2024-04-25T16:13:09.351 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.351 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2024-04-25T16:13:09.352 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.353 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.354 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.355 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2024-04-25T16:13:09.356 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.357 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.358 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:09.359 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:09.360 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:09.361 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 253959, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 81, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:09.362 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 159, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 50872, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 198, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-25T16:13:09.363 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 35, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 10710, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 33, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 35, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 35, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.229996056, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.006571315 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2024-04-25T16:13:09.364 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 35, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 57 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 35, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 0, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 35, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 0, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 10, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 0, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 1, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 5, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2024-04-25T16:13:09.365 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 6, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 20, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2024-04-25T16:13:09.366 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2024-04-25T16:13:09.367 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 77, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 2, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 9, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 9, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 14, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 0, 2024-04-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 1 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 1, 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 0 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2024-04-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 0, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed_ops": 0, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 0, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 1006, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 735603, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 1006, 2024-04-25T16:13:09.370 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1006, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 735603, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T16:13:09.371 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 35, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 10710, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 35, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 10710, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:09.372 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 35, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 35, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 35, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 35, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:09.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 0, 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:09.374 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x5575193b88f0": { 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 2, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 588, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 2, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T16:13:09.375 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1, 2024-04-25T16:13:09.376 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 588, 2024-04-25T16:13:09.376 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:09.376 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:09.376 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:09.376 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:09.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:09.376 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:09.376 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:13:09.558 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:09 smithi031 ceph-mon[29140]: pgmap v706: 97 pgs: 97 active+clean; 8.3 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 97 MiB/s wr, 338 op/s 2024-04-25T16:13:09.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:09 smithi063 ceph-mon[32413]: pgmap v706: 97 pgs: 97 active+clean; 8.3 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 97 MiB/s wr, 338 op/s 2024-04-25T16:13:09.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:09 smithi186 ceph-mon[33258]: pgmap v706: 97 pgs: 97 active+clean; 8.3 GiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 97 MiB/s wr, 338 op/s 2024-04-25T16:13:10.124 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-04-25T16:13:10.124 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-04-25T16:13:10.124 DEBUG:tasks.check_counter:Getting stats from d 2024-04-25T16:13:10.124 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-25T16:13:10.387 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:13:11.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:11 smithi063 ceph-mon[32413]: pgmap v707: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 86 MiB/s wr, 328 op/s 2024-04-25T16:13:11.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:11 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:13:11.643 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:11 smithi031 ceph-mon[29140]: pgmap v707: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 86 MiB/s wr, 328 op/s 2024-04-25T16:13:11.643 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:11 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:13:11.648 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:11 smithi186 ceph-mon[33258]: pgmap v707: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 86 MiB/s wr, 328 op/s 2024-04-25T16:13:11.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:11 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-25T16:13:12.418 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:13:12.418 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:12.418 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2024-04-25T16:13:12.418 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.001347564, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000298416, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:12.419 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 242, 2024-04-25T16:13:12.420 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 237, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 515360, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 145152, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.061063816, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.025482906, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.023618575, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.011610170, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 237, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.024897229, 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000105051 2024-04-25T16:13:12.421 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 515360, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 145152 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 1, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 179, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 160, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 109904, 2024-04-25T16:13:12.422 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 3, 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 2, 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.020124254, 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.013641828, 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.001320575, 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 177, 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.014859908, 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000083954 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.423 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 109904 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-25T16:13:12.424 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:12.425 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:12.426 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:12.427 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 149001, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 31, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-04-25T16:13:12.428 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:12.429 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 239, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 478780, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 239, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 239, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 478780, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:12.430 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:12.431 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:12.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:12.431 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:12.431 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: pgmap v708: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 95 MiB/s wr, 366 op/s 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:13:13.120 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:12 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.136 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.136 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.136 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.136 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.136 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: pgmap v708: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 95 MiB/s wr, 366 op/s 2024-04-25T16:13:13.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:13:13.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:13:13.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:12 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.149 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: pgmap v708: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 95 MiB/s wr, 366 op/s 2024-04-25T16:13:13.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-25T16:13:13.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-25T16:13:13.150 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:12 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' 2024-04-25T16:13:13.165 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-25T16:13:13.166 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-25T16:13:13.166 DEBUG:tasks.check_counter:Getting stats from g 2024-04-25T16:13:13.166 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-25T16:13:13.420 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:13:15.365 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:15 smithi031 ceph-mon[29140]: pgmap v709: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 84 MiB/s wr, 336 op/s 2024-04-25T16:13:15.365 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:13:15.365 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 828, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 830, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 278153, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 263305, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 6, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 2, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.175819061, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.048468947, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.050230048, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.059826295, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 828, 2024-04-25T16:13:15.366 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.335374165, 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000405041 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 834, 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 1007, 2024-04-25T16:13:15.367 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 220922, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 655887, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 5, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 4, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.146312990, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.043935428, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.041107884, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.050516254, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1006, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.148014883, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000147132 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.368 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 392208 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 248, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 241, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 556068, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 144293, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 3, 2024-04-25T16:13:15.369 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.053627798, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.019708372, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.024943256, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.009252398, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 240, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.057977673, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000241573 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.370 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 554720, 2024-04-25T16:13:15.371 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 143328 2024-04-25T16:13:15.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.371 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-25T16:13:15.371 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 1, 2024-04-25T16:13:15.371 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-25T16:13:15.371 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.371 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-MDSRank": { 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 3312, 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.158962727, 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000047995 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "queue_len": 0, 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "complete_latency": { 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.372 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "mds": { 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "request": 0, 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "reply": 0, 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "reply_latency": { 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_reply": 0, 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "forward": 0, 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:15.373 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_commit": 0, 2024-04-25T16:13:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_split": 0, 2024-04-25T16:13:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_merge": 0, 2024-04-25T16:13:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes": 0, 2024-04-25T16:13:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_top": 0, 2024-04-25T16:13:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_bottom": 0, 2024-04-25T16:13:15.374 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_pinned": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_expired": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "inodes_with_caps": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "caps": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "subtrees": 1, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_hit": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_forward": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_discover": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "traverse_lock": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "load_cent": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "q": 0, 2024-04-25T16:13:15.375 INFO:teuthology.orchestra.run.smithi031.stdout: "exported": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "exported_inodes": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "imported": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "imported_inodes": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "openino_peer_discover": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rfiles": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rbytes": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "root_rsnaps": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:13:15.376 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:13:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:13:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "scrub_file_inodes": 0, 2024-04-25T16:13:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:13:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:13:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:13:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:15.377 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_cap_release": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "process_request_cap_release": 0 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_cache": { 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_try_discover": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_send_discover": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "dir_handle_discover": 0, 2024-04-25T16:13:15.378 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_created": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_enqueued": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "strays_migrated": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_started": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "recovery_completed": 0, 2024-04-25T16:13:15.379 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_exportdir": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_flush": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "ireq_inodestats": 0 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_log": { 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "evlrg": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "evadd": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "evex": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "evtrm": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "ev": 2, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "evexg": 0, 2024-04-25T16:13:15.380 INFO:teuthology.orchestra.run.smithi031.stdout: "evexd": 0, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "segadd": 0, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "segex": 0, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "segtrm": 0, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "seg": 2, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "segmjr": 2, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "segexg": 0, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "segexd": 0, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "expos": 4194304, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "wrpos": 4194892, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "rdpos": 4194352, 2024-04-25T16:13:15.381 INFO:teuthology.orchestra.run.smithi031.stdout: "jlat": { 2024-04-25T16:13:15.382 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.382 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.382 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.382 INFO:teuthology.orchestra.run.smithi031.stdout: "replayed": 2 2024-04-25T16:13:15.382 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.382 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_mem": { 2024-04-25T16:13:15.383 INFO:teuthology.orchestra.run.smithi031.stdout: "ino": 3, 2024-04-25T16:13:15.383 INFO:teuthology.orchestra.run.smithi031.stdout: "ino+": 3, 2024-04-25T16:13:15.383 INFO:teuthology.orchestra.run.smithi031.stdout: "ino-": 0, 2024-04-25T16:13:15.383 INFO:teuthology.orchestra.run.smithi031.stdout: "dir": 1, 2024-04-25T16:13:15.383 INFO:teuthology.orchestra.run.smithi031.stdout: "dir+": 1, 2024-04-25T16:13:15.383 INFO:teuthology.orchestra.run.smithi031.stdout: "dir-": 0, 2024-04-25T16:13:15.383 INFO:teuthology.orchestra.run.smithi031.stdout: "dn": 0, 2024-04-25T16:13:15.383 INFO:teuthology.orchestra.run.smithi031.stdout: "dn+": 0, 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: "dn-": 0, 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: "cap": 0, 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: "cap+": 0, 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: "cap-": 0, 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: "rss": 118992, 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: "heap": 215440 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_server": { 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_client_request": 0, 2024-04-25T16:13:15.384 INFO:teuthology.orchestra.run.smithi031.stdout: "dispatch_server_request": 0, 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_request": 0, 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_client_session": 0, 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "handle_peer_request": 0, 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "req_create_latency": { 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getattr_latency": { 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.385 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.386 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.386 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:15.386 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.386 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.386 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.386 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.387 INFO:teuthology.orchestra.run.smithi031.stdout: "req_link_latency": { 2024-04-25T16:13:15.387 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.387 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.387 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.387 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.387 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookup_latency": { 2024-04-25T16:13:15.388 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.388 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.388 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.388 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.388 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:15.388 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupino_latency": { 2024-04-25T16:13:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.389 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupname_latency": { 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.390 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: "req_lssnap_latency": { 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mkdir_latency": { 2024-04-25T16:13:15.391 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.392 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.392 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.392 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.392 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mknod_latency": { 2024-04-25T16:13:15.392 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.392 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.393 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.393 INFO:teuthology.orchestra.run.smithi031.stdout: "req_mksnap_latency": { 2024-04-25T16:13:15.393 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.393 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.393 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.393 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.393 INFO:teuthology.orchestra.run.smithi031.stdout: "req_open_latency": { 2024-04-25T16:13:15.393 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "req_readdir_latency": { 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rename_latency": { 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.394 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.395 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmdir_latency": { 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.396 INFO:teuthology.orchestra.run.smithi031.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setattr_latency": { 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.397 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setlayout_latency": { 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "req_setxattr_latency": { 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.398 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "req_symlink_latency": { 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "req_unlink_latency": { 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:15.399 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_sessions": { 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "session_count": 0, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "session_add": 0, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "session_remove": 0, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_open": 0, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "sessions_stale": 0, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "total_load": 0, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "average_load": 0, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "avg_session_uptime": 0, 2024-04-25T16:13:15.400 INFO:teuthology.orchestra.run.smithi031.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:15.401 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:15.402 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 197463, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 54, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:15.403 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:15.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-25T16:13:15.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-25T16:13:15.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:15.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:15.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:15.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:15.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:15.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 159, 2024-04-25T16:13:15.404 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 8512, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 13, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "objecter": { 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "op_active": 0, 2024-04-25T16:13:15.405 INFO:teuthology.orchestra.run.smithi031.stdout: "op_laggy": 0, 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send": 1660, 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: "op_send_bytes": 21, 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: "op_resend": 3, 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: "op_reply": 1660, 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: "op_latency": { 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1660, 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 16.783713071, 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.010110670 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.406 INFO:teuthology.orchestra.run.smithi031.stdout: "op_inflight": 0, 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "oplen_avg": { 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 1660, 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 1661 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "op": 1660, 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "op_r": 1660, 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "op_w": 0, 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "op_rmw": 0, 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "op_pg": 0, 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_stat": 827, 2024-04-25T16:13:15.407 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_create": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_read": 831, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_write": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writefull": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_writesame": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_append": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_zero": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_truncate": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_delete": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_mapext": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_getxattr": 1, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_setxattr": 0, 2024-04-25T16:13:15.408 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_call": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_watch": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_notify": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "osdop_other": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_active": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_send": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_resend": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "linger_ping": 0, 2024-04-25T16:13:15.409 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_active": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_send": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "poolop_resend": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_active": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_send": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "poolstat_resend": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_active": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_send": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "statfs_resend": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "command_active": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "command_send": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "command_resend": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "map_epoch": 77, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "map_full": 0, 2024-04-25T16:13:15.410 INFO:teuthology.orchestra.run.smithi031.stdout: "map_inc": 2, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_sessions": 6, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_open": 6, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_session_close": 0, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_laggy": 0, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_wr": 0, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_rd": 2, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_del": 0 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "oft": { 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_objs": 0, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_updates": 0, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "omap_total_removes": 0 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.411 INFO:teuthology.orchestra.run.smithi031.stdout: "osd-slow-ops": { 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "slow_ops_count": 0 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "purge_queue": { 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing": 0, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executing_high_water": 0, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed_ops": 0, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_executed": 0, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:15.412 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 1906, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 868087, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 1906, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1906, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 868087, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.413 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 1660, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 593, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1660, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 593, 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:15.414 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.415 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.415 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.415 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:15.415 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.415 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:15.415 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:15.415 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 1024, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 1660, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 1660, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 1660, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 1660, 2024-04-25T16:13:15.416 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:15.417 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 0, 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 0, 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-write_buf_throttle-0x56212ca908f0": { 2024-04-25T16:13:15.418 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 3758096384, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:15.419 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:15.420 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:13:15.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:15 smithi063 ceph-mon[32413]: pgmap v709: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 84 MiB/s wr, 336 op/s 2024-04-25T16:13:15.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:15 smithi186 ceph-mon[33258]: pgmap v709: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 84 MiB/s wr, 336 op/s 2024-04-25T16:13:16.320 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-25T16:13:16.321 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-25T16:13:16.321 DEBUG:tasks.check_counter:Getting stats from j 2024-04-25T16:13:16.321 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-25T16:13:16.577 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.a/config 2024-04-25T16:13:17.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:17 smithi063 ceph-mon[32413]: pgmap v710: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 71 MiB/s wr, 299 op/s 2024-04-25T16:13:17.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:17 smithi186 ceph-mon[33258]: pgmap v710: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 71 MiB/s wr, 299 op/s 2024-04-25T16:13:17.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:17 smithi031 ceph-mon[29140]: pgmap v710: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 71 MiB/s wr, 299 op/s 2024-04-25T16:13:18.540 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2024-04-25T16:13:18.540 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 0, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 2, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 0, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 0, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 4, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 0, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.001433745, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.000000000, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.000283137, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:18.541 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 1, 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 181, 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 160, 2024-04-25T16:13:18.542 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 110256, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 3, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 2, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.016489037, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.013292515, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.001285441, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 179, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.022029526, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000123069 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:18.543 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 110256 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_messages": 244, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages": 239, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_bytes": 515904, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_bytes": 146432, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_created_connections": 2, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_active_connections": 1, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_total_time": 0.065070060, 2024-04-25T16:13:18.544 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_send_time": 0.020420868, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_recv_time": 0.022697296, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_running_fast_dispatch_time": 0.020847666, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 239, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.051768269, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000216603 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_recv_encrypted_bytes": 515904, 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: "msgr_send_encrypted_bytes": 146432 2024-04-25T16:13:18.545 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "cct": { 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "total_workers": 0, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "unhealthy_workers": 0 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "mempool": { 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:18.546 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:18.547 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_items": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_bytes": 148984, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_anon_items": 30, 2024-04-25T16:13:18.548 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_bytes": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_items": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_bytes": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_items": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "pgmap_items": 0, 2024-04-25T16:13:18.549 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_bytes": 928, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "mds_co_items": 2, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_1_items": 0, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "unittest_2_items": 0 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "val": 0, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "max": 104857600, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "get_started": 0, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "get": 241, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "get_sum": 479040, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:18.550 INFO:teuthology.orchestra.run.smithi031.stdout: "get_or_fail_success": 241, 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: "take": 0, 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: "take_sum": 0, 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: "put": 241, 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: "put_sum": 479040, 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: "wait": { 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: "avgcount": 0, 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: "sum": 0.000000000, 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: "avgtime": 0.000000000 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout: } 2024-04-25T16:13:18.551 INFO:teuthology.orchestra.run.smithi031.stdout:} 2024-04-25T16:13:19.328 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-25T16:13:19.328 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-25T16:13:19.329 DEBUG:tasks.check_counter:Getting stats from b 2024-04-25T16:13:19.329 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-25T16:13:19.450 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:19 smithi031 ceph-mon[29140]: pgmap v711: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 93 MiB/s wr, 374 op/s 2024-04-25T16:13:19.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:19 smithi063 ceph-mon[32413]: pgmap v711: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 93 MiB/s wr, 374 op/s 2024-04-25T16:13:19.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:19 smithi186 ceph-mon[33258]: pgmap v711: 97 pgs: 97 active+clean; 8.0 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 93 MiB/s wr, 374 op/s 2024-04-25T16:13:19.768 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:13:21.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:21 smithi063 ceph-mon[32413]: pgmap v712: 97 pgs: 97 active+clean; 7.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 69 MiB/s wr, 269 op/s 2024-04-25T16:13:21.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:21 smithi186 ceph-mon[33258]: pgmap v712: 97 pgs: 97 active+clean; 7.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 69 MiB/s wr, 269 op/s 2024-04-25T16:13:21.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:21 smithi031 ceph-mon[29140]: pgmap v712: 97 pgs: 97 active+clean; 7.9 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 69 MiB/s wr, 269 op/s 2024-04-25T16:13:22.040 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T16:13:22.040 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:22.040 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 5693, 2024-04-25T16:13:22.040 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 6211, 2024-04-25T16:13:22.040 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1515316, 2024-04-25T16:13:22.040 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 313666468, 2024-04-25T16:13:22.040 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 19, 2024-04-25T16:13:22.040 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 2.479199440, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 1.469752289, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.210915614, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.748933244, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 6212, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 2.736333280, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000440491 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3703, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.001312303, 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000354 2024-04-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 3995, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 4001, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 1057331, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 340078530, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 9, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 6, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 2.245262875, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 1.460947021, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.117212328, 2024-04-25T16:13:22.042 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.638656882, 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 4004, 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 3.755636884, 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000937971 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1066, 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000305475, 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000286 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 3104, 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 399632 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.043 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 527039, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 529459, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 194254721, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 465952654, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 230, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 12, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 33.919261639, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 15.559356804, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 50.337030498, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.707779285, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 529451, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 75.942346062, 2024-04-25T16:13:22.044 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000143436 2024-04-25T16:13:22.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.045 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:22.045 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 776467, 2024-04-25T16:13:22.045 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.246670934, 2024-04-25T16:13:22.045 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000317 2024-04-25T16:13:22.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.045 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 193431552, 2024-04-25T16:13:22.045 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 210588472 2024-04-25T16:13:22.045 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 3414, 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 409.741063791, 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.120017886 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.046 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 980, 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.032724233, 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000033392 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 101, 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 100, 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 100, 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.513803139, 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.005138031 2024-04-25T16:13:22.047 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 1, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 121, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 0, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 38, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 7, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 31, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 28, 2024-04-25T16:13:22.048 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 2, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 6, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 6, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 3, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 156, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 142, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 0, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 0, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 0, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 3, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 1, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 2, 2024-04-25T16:13:22.049 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 501, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 4, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 234, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 0, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 501, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 189, 2024-04-25T16:13:22.050 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 18, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 171, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 312, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 3, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 17, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 2935, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 522867, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 522834, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 8, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 47 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.051 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 6, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 11, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 20, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 9, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:22.052 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 18, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 1, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-04-25T16:13:22.053 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 524050, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 518871, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 517847, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 6203, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 14856, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 646, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 639, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 638, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 8, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 1, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 19, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 1028193643, 2024-04-25T16:13:22.054 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 1040445004, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 0, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 5538, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 1571.663489238, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.283796224 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 0 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 35, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 63, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 28, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 21, 2024-04-25T16:13:22.055 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 22, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 1, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 38, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 65, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 27, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 6, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 36, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 30, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 143216, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 231824 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 142, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 36, 2024-04-25T16:13:22.056 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 101, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 17, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 74, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 22, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.026204280, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001191103 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 15, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.004102835, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000273522 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.057 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 16, 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.006062425, 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000378901 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.058 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2, 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000270749, 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000135374 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.059 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-04-25T16:13:22.060 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.061 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.061 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.061 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.061 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-04-25T16:13:22.061 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 10, 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.018664481, 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001866448 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.062 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1, 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000202583, 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000202583 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 4, 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.013211725, 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.003302931 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2, 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.002101572, 2024-04-25T16:13:22.063 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001050786 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 4, 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.398328403, 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.099582100 2024-04-25T16:13:22.064 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1, 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.001276090, 2024-04-25T16:13:22.065 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001276090 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.066 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 7, 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.019626181, 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.002803740 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 16, 2024-04-25T16:13:22.067 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.023751815, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001484488 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 1, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 2, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 1, 2024-04-25T16:13:22.068 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 1, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 38, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 38, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 729, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:22.069 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:22.070 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:22.071 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 316907, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 130, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-04-25T16:13:22.072 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 159, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 146000, 2024-04-25T16:13:22.073 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 2369, 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 9032, 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 1036607135, 2024-04-25T16:13:22.074 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 37, 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 9032, 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 9032, 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 450.211549324, 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.049846274 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 9032, 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 9354 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 9032, 2024-04-25T16:13:22.075 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 540, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 8492, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 99, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 44, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 21, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 5541, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 210, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 0, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 2036, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-04-25T16:13:22.076 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 519, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 88, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-04-25T16:13:22.077 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-04-25T16:13:22.078 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 77, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 3, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 12, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 12, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 782, 2024-04-25T16:13:22.079 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 0, 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 14 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 1, 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-04-25T16:13:22.080 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 513, 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 6, 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 1033, 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 9, 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.081 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 536723, 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 126499123, 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 536723, 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:22.082 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 536723, 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 126499123, 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:22.083 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 9032, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 1036608044, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 9032, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1036608044, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.084 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 9032, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 9032, 2024-04-25T16:13:22.085 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 9032, 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 9032, 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 9, 2024-04-25T16:13:22.086 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 909, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 9, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 3, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 909, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:22.087 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x563ac3e848f0": { 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 11493, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 524050, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1036250700, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 524050, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 5538, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1036239207, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:22.088 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:22.089 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:22.089 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:22.089 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:22.089 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T16:13:23.044 INFO:tasks.check_counter:Daemon mds.b mds.exported=1 2024-04-25T16:13:23.044 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-25T16:13:23.044 DEBUG:tasks.check_counter:Getting stats from e 2024-04-25T16:13:23.044 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-25T16:13:23.318 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:13:23.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:23 smithi063 ceph-mon[32413]: pgmap v713: 97 pgs: 97 active+clean; 6.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 76 MiB/s wr, 448 op/s 2024-04-25T16:13:23.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:23 smithi186 ceph-mon[33258]: pgmap v713: 97 pgs: 97 active+clean; 6.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 76 MiB/s wr, 448 op/s 2024-04-25T16:13:23.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:23 smithi031 ceph-mon[29140]: pgmap v713: 97 pgs: 97 active+clean; 6.9 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 76 MiB/s wr, 448 op/s 2024-04-25T16:13:25.253 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T16:13:25.253 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 1108, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 1098, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 861120, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 418740, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 1, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.148266461, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.050577616, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.046845358, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.042309802, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1097, 2024-04-25T16:13:25.254 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.066852156, 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000060940 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 574176, 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 147168 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 3, 2024-04-25T16:13:25.255 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 183, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 716, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 396570, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 4, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.010788161, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.007799325, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.001367305, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000065090, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 181, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.011939911, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000065966 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.256 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 395872 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 1596, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 1596, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 433560, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 506711, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 4, 2024-04-25T16:13:25.257 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.125456149, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.027618612, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.035052783, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.055097432, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1595, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.067475888, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000042304 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.258 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 4074, 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.137636419, 2024-04-25T16:13:25.259 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000033784 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 0, 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 0, 2024-04-25T16:13:25.260 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 0, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 0, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 0, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 6, 2024-04-25T16:13:25.261 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 6, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 6, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 3, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:25.262 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 0, 2024-04-25T16:13:25.263 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 0, 2024-04-25T16:13:25.264 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 0 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 0, 2024-04-25T16:13:25.265 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:25.266 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 16, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 0, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 2, 2024-04-25T16:13:25.267 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 2, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 0, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 4194304, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 4218979, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 4217328, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 16 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 7, 2024-04-25T16:13:25.268 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 16, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 9, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 5, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 13, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 8, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 7, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 15, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 8, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 0, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 0, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 0, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 118680, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 215440 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 0, 2024-04-25T16:13:25.269 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 0, 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 0, 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 0, 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 0, 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.270 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.271 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.272 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.273 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.274 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.274 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.274 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-04-25T16:13:25.274 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.274 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.275 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.276 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-04-25T16:13:25.277 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.278 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.279 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.280 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.281 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 1, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 1, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 0, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 1, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 0, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 0, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 1824, 2024-04-25T16:13:25.282 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:25.283 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:25.284 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 181340, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 47, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:25.285 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 159, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:25.286 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 27633, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 100, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 2448, 2024-04-25T16:13:25.287 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 21, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 3, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 2448, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2448, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 3.440965939, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001405623 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2448, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 2449 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 2448, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 2448, 2024-04-25T16:13:25.288 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 1581, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 865, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-04-25T16:13:25.289 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 1, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-04-25T16:13:25.290 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-04-25T16:13:25.291 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 77, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 2, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 4, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 4, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 0, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 2, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 0 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 0, 2024-04-25T16:13:25.292 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 0, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 0, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 0, 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:25.293 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 2703, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1036171, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 2703, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2703, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1036171, 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:25.294 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2448, 2024-04-25T16:13:25.295 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 24680, 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2448, 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 24680, 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:25.296 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2448, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 2448, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2448, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2448, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.297 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.298 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x563f57890c30": { 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-04-25T16:13:25.299 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:25.300 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:25.300 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:25.300 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:25.300 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:25.300 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:25.300 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T16:13:25.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:25 smithi063 ceph-mon[32413]: pgmap v714: 97 pgs: 97 active+clean; 6.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 68 MiB/s wr, 546 op/s 2024-04-25T16:13:25.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:25 smithi186 ceph-mon[33258]: pgmap v714: 97 pgs: 97 active+clean; 6.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 68 MiB/s wr, 546 op/s 2024-04-25T16:13:25.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:25 smithi031 ceph-mon[29140]: pgmap v714: 97 pgs: 97 active+clean; 6.0 GiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 68 MiB/s wr, 546 op/s 2024-04-25T16:13:26.456 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-25T16:13:26.457 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-25T16:13:26.457 DEBUG:tasks.check_counter:Getting stats from h 2024-04-25T16:13:26.457 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-25T16:13:26.713 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:13:27.369 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:27 smithi063 ceph-mon[32413]: pgmap v715: 97 pgs: 97 active+clean; 5.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 57 MiB/s wr, 509 op/s 2024-04-25T16:13:27.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:27 smithi186 ceph-mon[33258]: pgmap v715: 97 pgs: 97 active+clean; 5.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 57 MiB/s wr, 509 op/s 2024-04-25T16:13:27.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:27 smithi031 ceph-mon[29140]: pgmap v715: 97 pgs: 97 active+clean; 5.8 GiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 57 MiB/s wr, 509 op/s 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 1208, 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 939, 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 345580, 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 367164, 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 10, 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 3, 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.116697417, 2024-04-25T16:13:28.586 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.046553029, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.078198046, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.005156708, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 942, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.072906573, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000077395 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1974, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000550270, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000278 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T16:13:28.587 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 81, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 260, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 48597, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 448509, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 9, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 6, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.019601320, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.009605228, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.008268496, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.000282167, 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:28.588 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 258, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.028401228, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000110082 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 127, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000036998, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000291 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 397312 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 411, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 407, 2024-04-25T16:13:28.589 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 637802, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 686793, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 48, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 7, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.083503201, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.019724881, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.055555149, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.009274618, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 402, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.031100384, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000077364 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:28.590 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 194, 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000053409, 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000275 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 585088, 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 155760 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-04-25T16:13:28.591 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 44, 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.001106323, 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000025143 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2, 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000005977, 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000002988 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 2, 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 1, 2024-04-25T16:13:28.592 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000903328, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000903328 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 1, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 11, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 0, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 14, 2024-04-25T16:13:28.593 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 0, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 14, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 14, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 0, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 1, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 1, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 2, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 3, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 1, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 0, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 1, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:28.594 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 4, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 234, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:13:28.595 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 2, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 3, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:28.596 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 2, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 0, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 1 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 2, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 0, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:28.597 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 0, 2024-04-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 3, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 0, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 0, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 3, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 0, 2024-04-25T16:13:28.599 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 2, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 0, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 0, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 2, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 2, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 0, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 4194304, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 4195200, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 0, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.003965365, 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001982682 2024-04-25T16:13:28.600 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 0 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 17, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 17, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 0, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 15, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 16, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 1, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 14, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 14, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 0, 2024-04-25T16:13:28.601 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 1, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 1, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 0, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 120784, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 223632 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 3, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 0, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 2, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 10, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 0, 2024-04-25T16:13:28.602 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1, 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000903328, 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000903328 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-04-25T16:13:28.603 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.604 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.605 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-04-25T16:13:28.605 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.605 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.605 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.605 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.605 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-04-25T16:13:28.605 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.605 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-04-25T16:13:28.606 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.607 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.607 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.607 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.607 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-04-25T16:13:28.607 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.607 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.607 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.608 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.609 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.610 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.611 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.612 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.613 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.614 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.615 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 1, 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 1, 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 0, 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 1, 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 1, 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 1, 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 737, 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.616 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-04-25T16:13:28.617 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:28.617 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:28.617 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:28.617 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:28.617 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:28.617 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:28.617 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:28.617 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:28.618 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:28.618 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:28.618 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:28.618 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:28.618 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:28.618 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:28.619 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:28.619 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:28.619 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:28.619 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:28.619 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:28.619 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:28.619 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:28.620 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:28.620 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:28.620 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:28.620 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:28.620 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:28.620 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:28.621 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 246609, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 83, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 159, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:28.622 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 68037, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 288, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 39, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 11513, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 33, 2024-04-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 39, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 39, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.212997921, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.005461485 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 39, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 63 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 39, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 0, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 39, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-04-25T16:13:28.624 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 10, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 2, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 6, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 6, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 0, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 20, 2024-04-25T16:13:28.625 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-04-25T16:13:28.626 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 77, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-04-25T16:13:28.627 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 2, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 9, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 9, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 18, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 0, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 1 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 1, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.628 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 0, 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 0, 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 0, 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:28.629 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 1696, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 861414, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 1696, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 1696, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 861414, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.630 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 39, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 11513, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 39, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 11513, 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:28.631 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 39, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 39, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 39, 2024-04-25T16:13:28.632 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 39, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:28.633 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x55974de428f0": { 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 3, 2024-04-25T16:13:28.634 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 896, 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 3, 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2, 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 896, 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:28.635 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T16:13:29.483 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-25T16:13:29.483 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-25T16:13:29.483 DEBUG:tasks.check_counter:Getting stats from k 2024-04-25T16:13:29.483 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-25T16:13:29.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:29 smithi063 ceph-mon[32413]: pgmap v716: 97 pgs: 97 active+clean; 4.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 53 MiB/s wr, 532 op/s 2024-04-25T16:13:29.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:29 smithi186 ceph-mon[33258]: pgmap v716: 97 pgs: 97 active+clean; 4.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 53 MiB/s wr, 532 op/s 2024-04-25T16:13:29.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:29 smithi031 ceph-mon[29140]: pgmap v716: 97 pgs: 97 active+clean; 4.2 GiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 53 MiB/s wr, 532 op/s 2024-04-25T16:13:29.802 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.b/config 2024-04-25T16:13:31.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:31 smithi063 ceph-mon[32413]: pgmap v717: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 28 MiB/s wr, 453 op/s 2024-04-25T16:13:31.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:31 smithi186 ceph-mon[33258]: pgmap v717: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 28 MiB/s wr, 453 op/s 2024-04-25T16:13:31.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:31 smithi031 ceph-mon[29140]: pgmap v717: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 28 MiB/s wr, 453 op/s 2024-04-25T16:13:31.967 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-04-25T16:13:31.967 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 1122, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 1110, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 867728, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 423674, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 1, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.149333834, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.050140567, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.048003090, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.042905369, 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:31.968 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 1110, 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.069572584, 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000062678 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 579104, 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 150512 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:31.969 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 744, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 927, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 183590, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 635974, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 4, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.085753719, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.024314587, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.018527667, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.032010253, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 925, 2024-04-25T16:13:31.970 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.040612577, 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000043905 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 399632 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_messages": 864, 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages": 865, 2024-04-25T16:13:31.971 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_bytes": 227421, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_bytes": 274435, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_created_connections": 5, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_active_connections": 4, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_total_time": 0.086092509, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_send_time": 0.027449626, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_recv_time": 0.022897385, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_running_fast_dispatch_time": 0.027400109, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 863, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.037676226, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000043657 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.972 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "cct": { 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "total_workers": 1, 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "unhealthy_workers": 0 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-MDSRank": { 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 4140, 2024-04-25T16:13:31.973 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.153357710, 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000037042 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: "queue_len": 0, 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: "complete_latency": { 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: "mds": { 2024-04-25T16:13:31.974 INFO:teuthology.orchestra.run.smithi063.stdout: "request": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "reply": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "reply_latency": { 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_reply": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "forward": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_commit": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_split": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_merge": 0, 2024-04-25T16:13:31.975 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_top": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_bottom": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_pinned": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_expired": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "inodes_with_caps": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "caps": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "subtrees": 1, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_hit": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_forward": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_discover": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:31.976 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "traverse_lock": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "load_cent": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "q": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "exported": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "exported_inodes": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "imported": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "imported_inodes": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "openino_peer_discover": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rfiles": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rbytes": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "root_rsnaps": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:13:31.977 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "scrub_file_inodes": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:31.978 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_cap_release": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "process_request_cap_release": 0 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_cache": { 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_try_discover": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_send_discover": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "dir_handle_discover": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:31.979 INFO:teuthology.orchestra.run.smithi063.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_created": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_enqueued": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "strays_migrated": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_started": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "recovery_completed": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_exportdir": 0, 2024-04-25T16:13:31.980 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_flush": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "ireq_inodestats": 0 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_log": { 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "evlrg": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "evadd": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "evex": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "evtrm": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "ev": 3, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "evexg": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "evexd": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "segadd": 0, 2024-04-25T16:13:31.981 INFO:teuthology.orchestra.run.smithi063.stdout: "segex": 0, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "segtrm": 0, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "seg": 2, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "segmjr": 2, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "segexg": 0, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "segexd": 0, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "expos": 4194304, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "wrpos": 4195200, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "rdpos": 4194892, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "jlat": { 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.982 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "replayed": 3 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_mem": { 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "ino": 3, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "ino+": 3, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "ino-": 0, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "dir": 1, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "dir+": 1, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "dir-": 0, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "dn": 0, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "dn+": 0, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "dn-": 0, 2024-04-25T16:13:31.983 INFO:teuthology.orchestra.run.smithi063.stdout: "cap": 0, 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "cap+": 0, 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "cap-": 0, 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "rss": 117360, 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "heap": 215440 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_server": { 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_client_request": 0, 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "dispatch_server_request": 0, 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_request": 0, 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_client_session": 0, 2024-04-25T16:13:31.984 INFO:teuthology.orchestra.run.smithi063.stdout: "handle_peer_request": 0, 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "req_create_latency": { 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getattr_latency": { 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.985 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "req_link_latency": { 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookup_latency": { 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:31.986 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupino_latency": { 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupname_latency": { 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.987 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "req_lssnap_latency": { 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.988 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mkdir_latency": { 2024-04-25T16:13:31.989 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.989 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.989 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.989 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.989 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mknod_latency": { 2024-04-25T16:13:31.989 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "req_mksnap_latency": { 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "req_open_latency": { 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.990 INFO:teuthology.orchestra.run.smithi063.stdout: "req_readdir_latency": { 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rename_latency": { 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:31.991 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmdir_latency": { 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.992 INFO:teuthology.orchestra.run.smithi063.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setattr_latency": { 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.993 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setlayout_latency": { 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "req_setxattr_latency": { 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.994 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: "req_symlink_latency": { 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: "req_unlink_latency": { 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.995 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_sessions": { 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "session_count": 1, 2024-04-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi063.stdout: "session_add": 1, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "session_remove": 0, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_open": 1, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "sessions_stale": 0, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "total_load": 0, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "average_load": 0, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "avg_session_uptime": 1830, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "mempool": { 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:31.998 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:31.999 INFO:teuthology.orchestra.run.smithi063.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_items": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_bytes": 197463, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_anon_items": 54, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_bytes": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_items": 0, 2024-04-25T16:13:32.000 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_items": 159, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "pgmap_items": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_bytes": 8512, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "mds_co_items": 13, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_1_items": 0, 2024-04-25T16:13:32.001 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "unittest_2_items": 0 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "objecter": { 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "op_active": 0, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "op_laggy": 0, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send": 2465, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "op_send_bytes": 21, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "op_resend": 3, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "op_reply": 2465, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "op_latency": { 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2465, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 4.081959175, 2024-04-25T16:13:32.002 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.001655967 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "op_inflight": 0, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "oplen_avg": { 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 2465, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 2466 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "op": 2465, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "op_r": 2465, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "op_w": 0, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "op_rmw": 0, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "op_pg": 0, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_stat": 1599, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_create": 0, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_read": 864, 2024-04-25T16:13:32.003 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_write": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writefull": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_writesame": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_append": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_zero": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_truncate": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_delete": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_mapext": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_getxattr": 1, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_setxattr": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:32.004 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_call": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_watch": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_notify": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "osdop_other": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_active": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_send": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_resend": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "linger_ping": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_active": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_send": 0, 2024-04-25T16:13:32.005 INFO:teuthology.orchestra.run.smithi063.stdout: "poolop_resend": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_active": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_send": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "poolstat_resend": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_active": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_send": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "statfs_resend": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "command_active": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "command_send": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "command_resend": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "map_epoch": 77, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "map_full": 0, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "map_inc": 2, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_sessions": 6, 2024-04-25T16:13:32.006 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_open": 6, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_session_close": 0, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "osd_laggy": 0, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_wr": 0, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_rd": 2, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_del": 0 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "oft": { 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_objs": 0, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_updates": 0, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "omap_total_removes": 0 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.007 INFO:teuthology.orchestra.run.smithi063.stdout: "osd-slow-ops": { 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "slow_ops_count": 0 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "purge_queue": { 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing": 0, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executing_high_water": 0, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed_ops": 0, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_executed": 0, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T16:13:32.008 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 2726, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 1017197, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 2726, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2726, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 1017197, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.009 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 104857600, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2465, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 901, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2465, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 901, 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:32.010 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:32.011 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:32.011 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:32.011 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:32.011 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.011 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:32.011 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:32.011 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 1024, 2024-04-25T16:13:32.011 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:32.011 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 2465, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 2465, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 2465, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 2465, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:32.012 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-04-25T16:13:32.013 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: }, 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "throttle-write_buf_throttle-0x55f8c32ac8f0": { 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "val": 0, 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "max": 3758096384, 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "get_started": 0, 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "get": 0, 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "get_sum": 0, 2024-04-25T16:13:32.014 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: "take": 0, 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: "take_sum": 0, 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: "put": 0, 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: "put_sum": 0, 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: "wait": { 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: "avgcount": 0, 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: "sum": 0.000000000, 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: "avgtime": 0.000000000 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout: } 2024-04-25T16:13:32.015 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-04-25T16:13:32.819 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-25T16:13:32.820 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-25T16:13:32.820 DEBUG:tasks.check_counter:Getting stats from c 2024-04-25T16:13:32.820 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-25T16:13:33.137 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:13:33.554 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:33 smithi186 ceph-mon[33258]: pgmap v718: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 42 MiB/s wr, 545 op/s 2024-04-25T16:13:33.594 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:33 smithi063 ceph-mon[32413]: pgmap v718: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 42 MiB/s wr, 545 op/s 2024-04-25T16:13:33.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:33 smithi031 ceph-mon[29140]: pgmap v718: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 42 MiB/s wr, 545 op/s 2024-04-25T16:13:35.210 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 785, 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 937, 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 731115, 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 305092, 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 8, 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 2, 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.117880625, 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.038818596, 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.063370501, 2024-04-25T16:13:35.211 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.013465403, 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 937, 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.050482302, 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000053876 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1098, 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000420113, 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000382 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 590864, 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 153696 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.212 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 151, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 332, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 94326, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 503376, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 8, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 6, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.023855691, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.011574564, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.012448327, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000349923, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 330, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.036844036, 2024-04-25T16:13:35.213 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000111648 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 252, 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000093321, 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000370 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 400896 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 125, 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 127, 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 56028, 2024-04-25T16:13:35.214 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 529709, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 50, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.034365881, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.004827280, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.028505246, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000329079, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 125, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.019087499, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000152699 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 175, 2024-04-25T16:13:35.215 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000041373, 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000236 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 1584, 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 1440 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 32, 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.001882464, 2024-04-25T16:13:35.216 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000058827 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2, 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000008986, 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000004493 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 2, 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 1, 2024-04-25T16:13:35.217 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000184498, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000184498 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 1, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 11, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 10, 2024-04-25T16:13:35.218 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 10, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 10, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 1, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 3, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 2, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-04-25T16:13:35.219 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 4, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 234, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:35.220 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 0, 2024-04-25T16:13:35.221 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 1 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 1, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:35.222 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.223 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 4, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 4, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 2, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 2, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 2, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-04-25T16:13:35.224 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 4194304, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 4195424, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 0, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.006589037, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.002196345 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 0 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 13, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 13, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 0, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 11, 2024-04-25T16:13:35.225 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 12, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 1, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 10, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 10, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 0, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 1, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 1, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 124064, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 223632 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 2, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 0, 2024-04-25T16:13:35.226 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 2, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 4, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 0, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000184498, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000184498 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.227 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.228 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.229 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.230 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-04-25T16:13:35.231 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.231 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.231 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.231 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.232 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.233 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:35.234 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:35.235 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-04-25T16:13:35.236 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:35.237 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 0, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 1, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 1, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 0, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 1, 2024-04-25T16:13:35.238 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 1, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 68, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:35.239 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:35.240 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 262823, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 89, 2024-04-25T16:13:35.241 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 159, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 50888, 2024-04-25T16:13:35.242 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 200, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 38, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 11332, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 33, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 38, 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-04-25T16:13:35.243 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 38, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.234995893, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.006184102 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 38, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 60 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 38, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 0, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 38, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 0, 2024-04-25T16:13:35.244 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 10, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 3, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 6, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 6, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 20, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:35.245 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-04-25T16:13:35.246 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 77, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 2, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 8, 2024-04-25T16:13:35.247 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 8, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 0, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 14, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 0, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 1 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 1, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-04-25T16:13:35.248 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 0, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 0, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 0, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:35.249 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 1057, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 768504, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 1057, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1057, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 768504, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.250 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 38, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 11332, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 38, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 11332, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.251 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 38, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 38, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 38, 2024-04-25T16:13:35.252 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 38, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:35.253 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x55c41a766680": { 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:35.254 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 4, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 1120, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 4, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 3, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1120, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:35.255 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-25T16:13:35.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:35 smithi063 ceph-mon[32413]: pgmap v719: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 41 MiB/s wr, 421 op/s 2024-04-25T16:13:35.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:35 smithi186 ceph-mon[33258]: pgmap v719: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 41 MiB/s wr, 421 op/s 2024-04-25T16:13:35.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:35 smithi031 ceph-mon[29140]: pgmap v719: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 41 MiB/s wr, 421 op/s 2024-04-25T16:13:36.152 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-25T16:13:36.152 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-25T16:13:36.152 DEBUG:tasks.check_counter:Getting stats from f 2024-04-25T16:13:36.153 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-25T16:13:36.400 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:13:37.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:37 smithi186 ceph-mon[33258]: pgmap v720: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 35 MiB/s wr, 307 op/s 2024-04-25T16:13:37.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:37 smithi063 ceph-mon[32413]: pgmap v720: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 35 MiB/s wr, 307 op/s 2024-04-25T16:13:37.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:37 smithi031 ceph-mon[29140]: pgmap v720: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 35 MiB/s wr, 307 op/s 2024-04-25T16:13:38.527 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-25T16:13:38.527 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:38.527 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 2119, 2024-04-25T16:13:38.527 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 2119, 2024-04-25T16:13:38.527 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 361391830, 2024-04-25T16:13:38.527 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 673028, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 8, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.464022435, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.058754622, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.304062210, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.080297827, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2118, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.097003326, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000045799 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.528 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 2944, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 448 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 1233, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 1415, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 403050057, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 792914, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 7, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 6, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.393265672, 2024-04-25T16:13:38.529 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.035654841, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.298642756, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.044940942, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1413, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.059995498, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000042459 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:13:38.530 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 401072 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 1282, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 1272, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 273786842, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 474881, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 5, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.306731135, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.047629308, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.207041672, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.043578365, 2024-04-25T16:13:38.531 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1271, 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 1.185593371, 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000932803 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 575152, 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 149248 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-04-25T16:13:38.532 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 6406, 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.205955048, 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000032150 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.533 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 0, 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 0, 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 0, 2024-04-25T16:13:38.534 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 0, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 51, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 4, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 47, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 44, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 0, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 3, 2024-04-25T16:13:38.535 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:13:38.536 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 4, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 234, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 0 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-04-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:38.539 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 524056, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-04-25T16:13:38.540 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 0, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 8, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 75, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 1028193643, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 1040449137, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 1040447999, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.541 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 524056 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 20, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 36, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 16, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 12, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 13, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 1, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 51, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 55, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 4, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-04-25T16:13:38.542 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 0, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 0, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 123520, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 215440 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 0, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 0, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 0, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 0, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 0, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.543 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.544 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.545 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.546 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.547 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.547 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-04-25T16:13:38.547 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.547 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.547 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.547 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.547 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.548 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.549 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-04-25T16:13:38.550 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-04-25T16:13:38.551 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.552 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.553 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-04-25T16:13:38.554 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 1, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 2, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 1, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 1, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 0, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 0, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 1838, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:38.555 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:38.556 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:38.557 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 238926, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 84, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 159, 2024-04-25T16:13:38.558 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 97444, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 1064, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 4371, 2024-04-25T16:13:38.559 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 26, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 4, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 4371, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 4371, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 30.488468503, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.006975170 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 4371, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 4372 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 4371, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 4371, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 0, 2024-04-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 2828, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 1540, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:38.561 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 2, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-04-25T16:13:38.562 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 77, 2024-04-25T16:13:38.563 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 2, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 11, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 11, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 0, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 0, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 2, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 0 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 0, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-04-25T16:13:38.564 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 0, 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 0, 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 0, 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:38.565 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 4630, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 1037795872, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 4630, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 4630, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1037795872, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.566 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 4371, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 1036465105, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 4371, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1036465105, 2024-04-25T16:13:38.567 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 4371, 2024-04-25T16:13:38.568 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 4371, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 4371, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 4371, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:38.569 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x55d0cad328f0": { 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-25T16:13:38.570 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:38.571 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-25T16:13:39.343 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-25T16:13:39.343 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-25T16:13:39.343 DEBUG:tasks.check_counter:Getting stats from i 2024-04-25T16:13:39.343 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-25T16:13:39.588 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:13:39.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:39 smithi063 ceph-mon[32413]: pgmap v721: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 58 MiB/s wr, 548 op/s 2024-04-25T16:13:39.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:39 smithi186 ceph-mon[33258]: pgmap v721: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 58 MiB/s wr, 548 op/s 2024-04-25T16:13:39.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:39 smithi031 ceph-mon[29140]: pgmap v721: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 58 MiB/s wr, 548 op/s 2024-04-25T16:13:41.383 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-25T16:13:41.383 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 2338, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 1666, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 1157506, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 655524, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 8, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.190152590, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.065382754, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.143478341, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.014371272, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1666, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.096722109, 2024-04-25T16:13:41.384 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000058056 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3119, 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000882085, 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000282 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 599792, 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 166512 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 104, 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 292, 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 53598, 2024-04-25T16:13:41.385 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 511793, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 15, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 6, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.032737778, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.013492889, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.017063748, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.000440564, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 288, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.028101148, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000097573 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.386 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 163, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000062052, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000380 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 4304, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 411856 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 162, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 162, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 76165, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 504118, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 44, 2024-04-25T16:13:41.387 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.035673802, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.005242265, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.027970384, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.002219790, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 162, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.028356130, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000175037 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 153, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000053871, 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000352 2024-04-25T16:13:41.388 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 100, 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.006615651, 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000066156 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:41.389 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 16, 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000422755, 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000026422 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 5, 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 4, 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 4, 2024-04-25T16:13:41.390 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.007146697, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.001786674 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 1, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 11, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 15, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 0, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 15, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:41.391 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 14, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 7, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 4, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 11, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 6, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 1, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-04-25T16:13:41.392 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 1, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 2, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 30, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 1, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 4, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 234, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 0, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 30, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:41.393 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 30, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 1, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 3, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 1, 2024-04-25T16:13:41.394 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 2, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 4 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 2, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:41.395 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 2, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 2, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:13:41.396 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 16, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 16, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-04-25T16:13:41.397 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 2, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 2, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 2, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 4194304, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 4218979, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 0, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 11, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.036568200, 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.003324381 2024-04-25T16:13:41.398 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 0 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 20, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 25, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 5, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 19, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 23, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 4, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 16, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 27, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 11, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 5, 2024-04-25T16:13:41.399 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 5, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 122872, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 223632 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 9, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 1, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 5, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 10, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 57, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.400 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.401 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1, 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000458814, 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000458814 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.402 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.403 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.404 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.404 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.404 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-04-25T16:13:41.404 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.404 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.404 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.404 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.404 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-04-25T16:13:41.404 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.405 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1, 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000517646, 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000517646 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.406 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.407 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.408 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.409 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.410 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.006170237, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.003085118 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.411 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 1, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 1, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 0, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 1, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 3, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 3, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 743, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.412 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:41.413 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:41.414 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 275172, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 95, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:41.415 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 159, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 81366, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 366, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:41.416 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 96, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 37760, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 33, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 96, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 96, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 2.085963625, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.021728787 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-04-25T16:13:41.417 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 96, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 124 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 96, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 31, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 65, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 0, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 10, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 1, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 12, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 11, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-04-25T16:13:41.418 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 14, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 30, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 20, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-04-25T16:13:41.419 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-04-25T16:13:41.420 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 77, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 2, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 9, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 9, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 0, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 22, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 0, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 4 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 1, 2024-04-25T16:13:41.421 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 2, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 2, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 2, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 2, 2024-04-25T16:13:41.422 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 2600, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 1033540, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 2600, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2600, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1033540, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.423 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 96, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 37962, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 96, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 37962, 2024-04-25T16:13:41.424 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:41.425 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.425 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:41 smithi186 ceph-mon[33258]: pgmap v722: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 61 MiB/s wr, 550 op/s 2024-04-25T16:13:41.425 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.425 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.425 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:41.425 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.425 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:41.425 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:41.425 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-04-25T16:13:41.425 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 96, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 96, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 96, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 96, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:41.426 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 2, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 202, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 2, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 1, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 202, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.427 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x564f197c2680": { 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 16, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 24675, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 16, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 11, 2024-04-25T16:13:41.428 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 24675, 2024-04-25T16:13:41.429 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:41.429 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:41.429 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:41.429 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:41.429 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:41.429 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:41.429 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-25T16:13:41.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:41 smithi063 ceph-mon[32413]: pgmap v722: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 61 MiB/s wr, 550 op/s 2024-04-25T16:13:41.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:41 smithi031 ceph-mon[29140]: pgmap v722: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 61 MiB/s wr, 550 op/s 2024-04-25T16:13:42.216 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-25T16:13:42.216 INFO:tasks.check_counter:Daemon mds.i mds.imported=1 2024-04-25T16:13:42.217 DEBUG:tasks.check_counter:Getting stats from l 2024-04-25T16:13:42.217 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8 shell --fsid 74fd3742-031b-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-25T16:13:42.463 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/mon.c/config 2024-04-25T16:13:43.604 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:43 smithi186 ceph-mon[33258]: pgmap v723: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 59 MiB/s wr, 535 op/s 2024-04-25T16:13:43.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:43 smithi063 ceph-mon[32413]: pgmap v723: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 59 MiB/s wr, 535 op/s 2024-04-25T16:13:43.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:43 smithi031 ceph-mon[29140]: pgmap v723: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 59 MiB/s wr, 535 op/s 2024-04-25T16:13:44.219 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-04-25T16:13:44.219 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-0": { 2024-04-25T16:13:44.219 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 268, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 256, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 581026, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 153396, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 5, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 1, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.044973934, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.014937021, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.021763073, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.007921498, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 256, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.012290574, 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000048010 2024-04-25T16:13:44.220 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 580192, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 153072 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-1": { 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 743, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 929, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 182714, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 639353, 2024-04-25T16:13:44.221 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 5, 2024-04-25T16:13:44.222 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 4, 2024-04-25T16:13:44.222 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.097293355, 2024-04-25T16:13:44.222 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.025586199, 2024-04-25T16:13:44.222 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.024781794, 2024-04-25T16:13:44.222 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.039297432, 2024-04-25T16:13:44.222 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:44.222 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 927, 2024-04-25T16:13:44.222 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.039708240, 2024-04-25T16:13:44.222 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000042835 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 403392 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "AsyncMessenger::Worker-2": { 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_messages": 1624, 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages": 1626, 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_bytes": 486583, 2024-04-25T16:13:44.223 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_bytes": 516490, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_created_connections": 4, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_active_connections": 3, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_total_time": 0.175322462, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_send_time": 0.048533555, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_recv_time": 0.050113953, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_running_fast_dispatch_time": 0.062379698, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_messages_queue_lat": { 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 1624, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.067329676, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000041459 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_handle_ack_lat": { 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.224 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.225 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.225 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.225 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-25T16:13:44.225 INFO:teuthology.orchestra.run.smithi186.stdout: "msgr_send_encrypted_bytes": 0 2024-04-25T16:13:44.225 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.225 INFO:teuthology.orchestra.run.smithi186.stdout: "cct": { 2024-04-25T16:13:44.225 INFO:teuthology.orchestra.run.smithi186.stdout: "total_workers": 1, 2024-04-25T16:13:44.225 INFO:teuthology.orchestra.run.smithi186.stdout: "unhealthy_workers": 0 2024-04-25T16:13:44.225 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-MDSRank": { 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 3966, 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.133494435, 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000033659 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "finisher-PurgeQueue": { 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "queue_len": 0, 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "complete_latency": { 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.226 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "mds": { 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "request": 0, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "reply": 0, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "reply_latency": { 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_reply": 0, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "forward": 0, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_complete": 0, 2024-04-25T16:13:44.227 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_fetch_keys": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_commit": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_split": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_merge": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_top": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_bottom": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pin_tail": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_pinned": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_expired": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "inodes_with_caps": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "caps": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "subtrees": 1, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_hit": 0, 2024-04-25T16:13:44.228 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_forward": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_discover": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_dir_fetch": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_remote_ino": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "traverse_lock": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "load_cent": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "q": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "exported": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "exported_inodes": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "imported": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "imported_inodes": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_dir_fetch": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_backtrace_fetch": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "openino_peer_discover": 0, 2024-04-25T16:13:44.229 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rfiles": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rbytes": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "root_rsnaps": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_fetch": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_set_tag": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_backtrace_repaired": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_inotable_repaired": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_inodes": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dir_base_inodes": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_dirfrag_rstats": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "scrub_file_inodes": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_inode_file_caps": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_revoke": 0, 2024-04-25T16:13:44.230 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_grant": 0, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_trunc": 0, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps": 0, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_caps_dirty": 0, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_cap_release": 0, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "process_request_cap_release": 0 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_cache": { 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update": 0, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_update_receipt": 0, 2024-04-25T16:13:44.231 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_try_discover": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_send_discover": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "dir_handle_discover": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_delayed": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "num_strays_enqueuing": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_created": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_enqueued": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_reintegrated": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "strays_migrated": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_processing": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_enqueued": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "num_recovering_prioritized": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_started": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "recovery_completed": 0, 2024-04-25T16:13:44.232 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_path": 0, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_quiesce_inode": 0, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_enqueue_scrub": 0, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_exportdir": 0, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_flush": 0, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragmentdir": 0, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_fragstats": 0, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "ireq_inodestats": 0 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_log": { 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "evlrg": 0, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "evadd": 0, 2024-04-25T16:13:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: "evex": 0, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "evtrm": 0, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "ev": 4, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "evexg": 0, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "evexd": 0, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "segadd": 0, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "segex": 0, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "segtrm": 0, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "seg": 2, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "segmjr": 2, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "segexg": 0, 2024-04-25T16:13:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: "segexd": 0, 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: "expos": 4194304, 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: "wrpos": 4195424, 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: "rdpos": 4195283, 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: "jlat": { 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: "replayed": 4 2024-04-25T16:13:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_mem": { 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "ino": 3, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "ino+": 3, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "ino-": 0, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "dir": 1, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "dir+": 1, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "dir-": 0, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "dn": 0, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "dn+": 0, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "dn-": 0, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "cap": 0, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "cap+": 0, 2024-04-25T16:13:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: "cap-": 0, 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "rss": 118616, 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "heap": 215440 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_server": { 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_client_request": 0, 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "dispatch_server_request": 0, 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_request": 0, 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_client_session": 0, 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "handle_peer_request": 0, 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "req_create_latency": { 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getattr_latency": { 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getfilelock_latency": { 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "req_link_latency": { 2024-04-25T16:13:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookup_latency": { 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookuphash_latency": { 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupino_latency": { 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupname_latency": { 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupparent_latency": { 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lookupsnap_latency": { 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: "req_lssnap_latency": { 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mkdir_latency": { 2024-04-25T16:13:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mknod_latency": { 2024-04-25T16:13:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: "req_mksnap_latency": { 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: "req_open_latency": { 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: "req_readdir_latency": { 2024-04-25T16:13:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rename_latency": { 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "req_renamesnap_latency": { 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "req_snapdiff_latency": { 2024-04-25T16:13:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmdir_latency": { 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmsnap_latency": { 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: "req_rmxattr_latency": { 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setattr_latency": { 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setdirlayout_latency": { 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setfilelock_latency": { 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setlayout_latency": { 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "req_setxattr_latency": { 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.247 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "req_symlink_latency": { 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "req_unlink_latency": { 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_revoke_eviction": 0, 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "cap_acquisition_throttle": 0, 2024-04-25T16:13:44.248 INFO:teuthology.orchestra.run.smithi186.stdout: "req_getvxattr_latency": { 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_sessions": { 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: "session_count": 0, 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: "session_add": 1, 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: "session_remove": 1, 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_open": 1, 2024-04-25T16:13:44.249 INFO:teuthology.orchestra.run.smithi186.stdout: "sessions_stale": 0, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "total_load": 0, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "average_load": 0, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "avg_session_uptime": 1038, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "mdthresh_evicted": 0 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "mempool": { 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_bytes": 0, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "bloom_filter_items": 0, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_bytes": 0, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_alloc_items": 0, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_bytes": 0, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_data_items": 0, 2024-04-25T16:13:44.250 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_onode_items": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_meta_items": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_bytes": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_other_items": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_cache_buffer_items": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_bytes": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_extent_items": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_bytes": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_blob_items": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_shared_blob_items": 0, 2024-04-25T16:13:44.251 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_inline_bl_items": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_bytes": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_fsck_items": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_bytes": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_txc_items": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_deferred_items": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_bytes": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluestore_writing_items": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_bytes": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_items": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_bytes": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_reader_items": 0, 2024-04-25T16:13:44.252 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_bytes": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "bluefs_file_writer_items": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_bytes": 189383, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_anon_items": 50, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_bytes": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "buffer_meta_items": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_bytes": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_items": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_bytes": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_mapbl_items": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_bytes": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_pglog_items": 0, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_bytes": 7536, 2024-04-25T16:13:44.253 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_items": 159, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_bytes": 0, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "osdmap_mapping_items": 0, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_bytes": 0, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "pgmap_items": 0, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_bytes": 8512, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "mds_co_items": 13, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_bytes": 0, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_1_items": 0, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_bytes": 0, 2024-04-25T16:13:44.254 INFO:teuthology.orchestra.run.smithi186.stdout: "unittest_2_items": 0 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "objecter": { 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "op_active": 0, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "op_laggy": 0, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send": 2367, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "op_send_bytes": 21, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "op_resend": 3, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "op_reply": 2367, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "op_latency": { 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2367, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 65.383859836, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.027623092 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "op_inflight": 0, 2024-04-25T16:13:44.255 INFO:teuthology.orchestra.run.smithi186.stdout: "oplen_avg": { 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 2367, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 2368 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "op": 2367, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "op_r": 2367, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "op_w": 0, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "op_rmw": 0, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "op_pg": 0, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_stat": 1550, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_create": 0, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_read": 815, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_write": 0, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writefull": 0, 2024-04-25T16:13:44.256 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_writesame": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_append": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_zero": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_truncate": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_delete": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_mapext": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_sparse_read": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_clonerange": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_getxattr": 1, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_setxattr": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_cmpxattr": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_rmxattr": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_resetxattrs": 0, 2024-04-25T16:13:44.257 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_call": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_watch": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_notify": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_src_cmpxattr": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_pgls_filter": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "osdop_other": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_active": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_send": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_resend": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "linger_ping": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_active": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_send": 0, 2024-04-25T16:13:44.258 INFO:teuthology.orchestra.run.smithi186.stdout: "poolop_resend": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_active": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_send": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "poolstat_resend": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_active": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_send": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "statfs_resend": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "command_active": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "command_send": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "command_resend": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "map_epoch": 77, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "map_full": 0, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "map_inc": 2, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_sessions": 5, 2024-04-25T16:13:44.259 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_open": 5, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_session_close": 0, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "osd_laggy": 0, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_wr": 0, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_rd": 2, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_del": 0 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "oft": { 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_objs": 0, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_kv_pairs": 0, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_updates": 0, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "omap_total_removes": 0 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "osd-slow-ops": { 2024-04-25T16:13:44.260 INFO:teuthology.orchestra.run.smithi186.stdout: "slow_ops_count": 0 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "purge_queue": { 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops": 0, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_ops_high_water": 0, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing": 0, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executing_high_water": 0, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed_ops": 0, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_executed": 0, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "pq_item_in_journal": 0 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-25T16:13:44.261 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 2631, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 997123, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 2631, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2631, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 997123, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.262 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_bytes": { 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 104857600, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 2367, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 1125, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2367, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 1125, 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:44.263 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-objecter_ops": { 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 1024, 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:44.264 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 2367, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 2367, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 2367, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 2367, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle": { 2024-04-25T16:13:44.265 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.266 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: }, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "throttle-write_buf_throttle-0x556f3432c8f0": { 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "val": 0, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "max": 3758096384, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "get_started": 0, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "get": 0, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "get_sum": 0, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_fail": 0, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "get_or_fail_success": 0, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "take": 0, 2024-04-25T16:13:44.267 INFO:teuthology.orchestra.run.smithi186.stdout: "take_sum": 0, 2024-04-25T16:13:44.268 INFO:teuthology.orchestra.run.smithi186.stdout: "put": 0, 2024-04-25T16:13:44.268 INFO:teuthology.orchestra.run.smithi186.stdout: "put_sum": 0, 2024-04-25T16:13:44.268 INFO:teuthology.orchestra.run.smithi186.stdout: "wait": { 2024-04-25T16:13:44.268 INFO:teuthology.orchestra.run.smithi186.stdout: "avgcount": 0, 2024-04-25T16:13:44.268 INFO:teuthology.orchestra.run.smithi186.stdout: "sum": 0.000000000, 2024-04-25T16:13:44.268 INFO:teuthology.orchestra.run.smithi186.stdout: "avgtime": 0.000000000 2024-04-25T16:13:44.268 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:44.268 INFO:teuthology.orchestra.run.smithi186.stdout: } 2024-04-25T16:13:44.268 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-04-25T16:13:45.003 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-25T16:13:45.003 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-25T16:13:45.004 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-25T16:13:45.017 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-25T16:13:45.017 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T16:13:45.017 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T16:13:45.053 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2024-04-25T16:13:45.053 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-25T16:13:45.121 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:13:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.430+0000 7fd21c738640 1 -- 172.21.15.31:0/2446637610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214103970 msgr2=0x7fd214103dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:45.433 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.430+0000 7fd21c738640 1 --2- 172.21.15.31:0/2446637610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214103970 0x7fd214103dc0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fd204009900 tx=0x7fd20402a410 comp rx=0 tx=0).stop 2024-04-25T16:13:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.430+0000 7fd21c738640 1 -- 172.21.15.31:0/2446637610 shutdown_connections 2024-04-25T16:13:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.431+0000 7fd21c738640 1 --2- 172.21.15.31:0/2446637610 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd214104450 0x7fd21410ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.431+0000 7fd21c738640 1 --2- 172.21.15.31:0/2446637610 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214103970 0x7fd214103dc0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.431+0000 7fd21c738640 1 --2- 172.21.15.31:0/2446637610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd214102f90 0x7fd2141033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.431+0000 7fd21c738640 1 -- 172.21.15.31:0/2446637610 >> 172.21.15.31:0/2446637610 conn(0x7fd2140fd330 msgr2=0x7fd2140ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.431+0000 7fd21c738640 1 -- 172.21.15.31:0/2446637610 shutdown_connections 2024-04-25T16:13:45.434 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.431+0000 7fd21c738640 1 -- 172.21.15.31:0/2446637610 wait complete. 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21c738640 1 Processor -- start 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21c738640 1 -- start start 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21c738640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd214102f90 0x7fd214199180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21c738640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd214103970 0x7fd2141996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21c738640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214104450 0x7fd2141a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21acd5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214104450 0x7fd2141a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21c738640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd21410e5e0 con 0x7fd214104450 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21c738640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd21410e460 con 0x7fd214103970 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21c738640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd21410e760 con 0x7fd214102f90 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21acd5640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214104450 0x7fd2141a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41538/0 (socket says 172.21.15.31:41538) 2024-04-25T16:13:45.436 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.432+0000 7fd21acd5640 1 -- 172.21.15.31:0/2335013535 learned_addr learned my addr 172.21.15.31:0/2335013535 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:45.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.433+0000 7fd21acd5640 1 -- 172.21.15.31:0/2335013535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd214102f90 msgr2=0x7fd214199180 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-25T16:13:45.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.433+0000 7fd21acd5640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd214102f90 0x7fd214199180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.433+0000 7fd21acd5640 1 -- 172.21.15.31:0/2335013535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd214103970 msgr2=0x7fd2141996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:45.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.433+0000 7fd21acd5640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd214103970 0x7fd2141996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.433+0000 7fd21acd5640 1 -- 172.21.15.31:0/2335013535 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2141a0cc0 con 0x7fd214104450 2024-04-25T16:13:45.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.433+0000 7fd21acd5640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214104450 0x7fd2141a0650 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00eaa0 tx=0x7fd20c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:45.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.433+0000 7fd20b7fe640 1 -- 172.21.15.31:0/2335013535 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c00cdb0 con 0x7fd214104450 2024-04-25T16:13:45.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.434+0000 7fd20b7fe640 1 -- 172.21.15.31:0/2335013535 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd20c005020 con 0x7fd214104450 2024-04-25T16:13:45.437 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.434+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2141a0fb0 con 0x7fd214104450 2024-04-25T16:13:45.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.435+0000 7fd20b7fe640 1 -- 172.21.15.31:0/2335013535 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd20c0116d0 con 0x7fd214104450 2024-04-25T16:13:45.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.435+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2141a14e0 con 0x7fd214104450 2024-04-25T16:13:45.438 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.436+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd214107930 con 0x7fd214104450 2024-04-25T16:13:45.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.437+0000 7fd20b7fe640 1 -- 172.21.15.31:0/2335013535 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd20c0051d0 con 0x7fd214104450 2024-04-25T16:13:45.440 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.438+0000 7fd20b7fe640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2000afb70 0x7fd2000b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:45.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.438+0000 7fd20b7fe640 1 -- 172.21.15.31:0/2335013535 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd20c0dee00 con 0x7fd214104450 2024-04-25T16:13:45.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.439+0000 7fd21a4d4640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2000afb70 0x7fd2000b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:45.441 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.439+0000 7fd21a4d4640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2000afb70 0x7fd2000b2080 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd214069b90 tx=0x7fd210006c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:45.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.444+0000 7fd20b7fe640 1 -- 172.21.15.31:0/2335013535 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd20c016030 con 0x7fd214104450 2024-04-25T16:13:45.619 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:45 smithi063 ceph-mon[32413]: pgmap v724: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 42 MiB/s wr, 421 op/s 2024-04-25T16:13:45.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:45 smithi186 ceph-mon[33258]: pgmap v724: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 42 MiB/s wr, 421 op/s 2024-04-25T16:13:45.651 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.649+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2140637d0 con 0x7fd214104450 2024-04-25T16:13:45.652 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.650+0000 7fd20b7fe640 1 -- 172.21.15.31:0/2335013535 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7fd20c0a79f0 con 0x7fd214104450 2024-04-25T16:13:45.653 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:13:45.653 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2000afb70 msgr2=0x7fd2000b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2000afb70 0x7fd2000b2080 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fd214069b90 tx=0x7fd210006c20 comp rx=0 tx=0).stop 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214104450 msgr2=0x7fd2141a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214104450 0x7fd2141a0650 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fd20c00eaa0 tx=0x7fd20c00ef60 comp rx=0 tx=0).stop 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 shutdown_connections 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2000afb70 0x7fd2000b2080 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd214104450 0x7fd2141a0650 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd214103970 0x7fd2141996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 --2- 172.21.15.31:0/2335013535 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd214102f90 0x7fd214199180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 >> 172.21.15.31:0/2335013535 conn(0x7fd2140fd330 msgr2=0x7fd214104b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:45.655 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 shutdown_connections 2024-04-25T16:13:45.656 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.652+0000 7fd21c738640 1 -- 172.21.15.31:0/2335013535 wait complete. 2024-04-25T16:13:45.665 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:45 smithi031 ceph-mon[29140]: pgmap v724: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 42 MiB/s wr, 421 op/s 2024-04-25T16:13:45.667 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-25T15:54:17.311776+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-25T15:59:00.978038+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-25T15:59:01.829535+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-25T16:13:45.667 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.780+0000 7fd224027640 1 -- 172.21.15.31:0/2119161626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1098b0 msgr2=0x7fd21c10bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.780+0000 7fd224027640 1 --2- 172.21.15.31:0/2119161626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1098b0 0x7fd21c10bcd0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd214009ca0 tx=0x7fd21402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.780+0000 7fd224027640 1 -- 172.21.15.31:0/2119161626 shutdown_connections 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.780+0000 7fd224027640 1 --2- 172.21.15.31:0/2119161626 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1098b0 0x7fd21c10bcd0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.780+0000 7fd224027640 1 --2- 172.21.15.31:0/2119161626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c106f50 0x7fd21c109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.780+0000 7fd224027640 1 --2- 172.21.15.31:0/2119161626 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd21c1045f0 0x7fd21c106a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.780+0000 7fd224027640 1 -- 172.21.15.31:0/2119161626 >> 172.21.15.31:0/2119161626 conn(0x7fd21c0fdbf0 msgr2=0x7fd21c1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.780+0000 7fd224027640 1 -- 172.21.15.31:0/2119161626 shutdown_connections 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.780+0000 7fd224027640 1 -- 172.21.15.31:0/2119161626 wait complete. 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd224027640 1 Processor -- start 2024-04-25T16:13:45.785 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd224027640 1 -- start start 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd224027640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1045f0 0x7fd21c19a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd224027640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c106f50 0x7fd21c19a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd224027640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd21c1098b0 0x7fd21c1a1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd224027640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fd21c10f630 con 0x7fd21c1045f0 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd224027640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fd21c10f4b0 con 0x7fd21c106f50 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd224027640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd21c10f7b0 con 0x7fd21c1098b0 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd221dc3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1045f0 0x7fd21c19a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd221dc3640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1045f0 0x7fd21c19a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41548/0 (socket says 172.21.15.31:41548) 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd221dc3640 1 -- 172.21.15.31:0/2973255312 learned_addr learned my addr 172.21.15.31:0/2973255312 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd2225c4640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd21c1098b0 0x7fd21c1a1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.781+0000 7fd2215c2640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c106f50 0x7fd21c19a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd221dc3640 1 -- 172.21.15.31:0/2973255312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd21c1098b0 msgr2=0x7fd21c1a1700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd221dc3640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd21c1098b0 0x7fd21c1a1700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.786 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd221dc3640 1 -- 172.21.15.31:0/2973255312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c106f50 msgr2=0x7fd21c19a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd221dc3640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c106f50 0x7fd21c19a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd221dc3640 1 -- 172.21.15.31:0/2973255312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd21c1a1d70 con 0x7fd21c1045f0 2024-04-25T16:13:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd2215c2640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c106f50 0x7fd21c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd221dc3640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1045f0 0x7fd21c19a230 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c970 tx=0x7fd21800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd212ffd640 1 -- 172.21.15.31:0/2973255312 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd218007bb0 con 0x7fd21c1045f0 2024-04-25T16:13:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd21c1a2060 con 0x7fd21c1045f0 2024-04-25T16:13:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd212ffd640 1 -- 172.21.15.31:0/2973255312 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd218007d50 con 0x7fd21c1045f0 2024-04-25T16:13:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd212ffd640 1 -- 172.21.15.31:0/2973255312 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd218018730 con 0x7fd21c1045f0 2024-04-25T16:13:45.787 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.782+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd21c1a2560 con 0x7fd21c1045f0 2024-04-25T16:13:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.783+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd21c0ff710 con 0x7fd21c1045f0 2024-04-25T16:13:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.784+0000 7fd212ffd640 1 -- 172.21.15.31:0/2973255312 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd21800f990 con 0x7fd21c1045f0 2024-04-25T16:13:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.785+0000 7fd212ffd640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2080afaf0 0x7fd2080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.785+0000 7fd212ffd640 1 -- 172.21.15.31:0/2973255312 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fd2180dec20 con 0x7fd21c1045f0 2024-04-25T16:13:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.785+0000 7fd2215c2640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2080afaf0 0x7fd2080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:45.788 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.786+0000 7fd2215c2640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2080afaf0 0x7fd2080b2000 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd20c005d00 tx=0x7fd20c005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:45.792 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.790+0000 7fd212ffd640 1 -- 172.21.15.31:0/2973255312 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd218014030 con 0x7fd21c1045f0 2024-04-25T16:13:45.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.976+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd21c02db50 con 0x7fd21c1045f0 2024-04-25T16:13:45.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.976+0000 7fd212ffd640 1 -- 172.21.15.31:0/2973255312 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) ==== 93+0+10 (secure 0 0 0) 0x7fd2180a7810 con 0x7fd21c1045f0 2024-04-25T16:13:45.979 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2024-04-25T16:13:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2080afaf0 msgr2=0x7fd2080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2080afaf0 0x7fd2080b2000 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fd20c005d00 tx=0x7fd20c005ed0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1045f0 msgr2=0x7fd21c19a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1045f0 0x7fd21c19a230 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd21800c970 tx=0x7fd21800ce30 comp rx=0 tx=0).stop 2024-04-25T16:13:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 shutdown_connections 2024-04-25T16:13:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fd2080afaf0 0x7fd2080b2000 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd21c1098b0 0x7fd21c1a1700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.981 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd21c106f50 0x7fd21c19a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 --2- 172.21.15.31:0/2973255312 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fd21c1045f0 0x7fd21c19a230 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 >> 172.21.15.31:0/2973255312 conn(0x7fd21c0fdbf0 msgr2=0x7fd21c105260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.978+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 shutdown_connections 2024-04-25T16:13:45.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:45.979+0000 7fd224027640 1 -- 172.21.15.31:0/2973255312 wait complete. 2024-04-25T16:13:45.993 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-25T16:13:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.107+0000 7f74b74b6640 1 -- 172.21.15.31:0/1296816059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0102fe0 msgr2=0x7f74b01033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.107+0000 7f74b74b6640 1 --2- 172.21.15.31:0/1296816059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0102fe0 0x7f74b01033f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009900 tx=0x7f74ac02a410 comp rx=0 tx=0).stop 2024-04-25T16:13:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.107+0000 7f74b74b6640 1 -- 172.21.15.31:0/1296816059 shutdown_connections 2024-04-25T16:13:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.107+0000 7f74b74b6640 1 --2- 172.21.15.31:0/1296816059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b01046a0 0x7f74b010adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.107+0000 7f74b74b6640 1 --2- 172.21.15.31:0/1296816059 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b0103b40 0x7f74b0104010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.107+0000 7f74b74b6640 1 --2- 172.21.15.31:0/1296816059 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0102fe0 0x7f74b01033f0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.107+0000 7f74b74b6640 1 -- 172.21.15.31:0/1296816059 >> 172.21.15.31:0/1296816059 conn(0x7f74b00fd380 msgr2=0x7f74b00ff830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:46.112 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.107+0000 7f74b74b6640 1 -- 172.21.15.31:0/1296816059 shutdown_connections 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b74b6640 1 -- 172.21.15.31:0/1296816059 wait complete. 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b74b6640 1 Processor -- start 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b74b6640 1 -- start start 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b74b6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b0102fe0 0x7f74b019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b74b6640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0103b40 0x7f74b019a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b74b6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b019f540 0x7f74b01a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b74b6640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f74b010f870 con 0x7f74b0103b40 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b74b6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f74b010f6f0 con 0x7f74b0102fe0 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b74b6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74b010f9f0 con 0x7f74b019f540 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b4a51640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0103b40 0x7f74b019a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.108+0000 7f74b4a51640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0103b40 0x7f74b019a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41566/0 (socket says 172.21.15.31:41566) 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b4a51640 1 -- 172.21.15.31:0/3707470158 learned_addr learned my addr 172.21.15.31:0/3707470158 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b5252640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b0102fe0 0x7f74b019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b5a53640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b019f540 0x7f74b01a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.113 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b4a51640 1 -- 172.21.15.31:0/3707470158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b019f540 msgr2=0x7f74b01a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b4a51640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b019f540 0x7f74b01a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b4a51640 1 -- 172.21.15.31:0/3707470158 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b0102fe0 msgr2=0x7f74b019a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b4a51640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b0102fe0 0x7f74b019a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b4a51640 1 -- 172.21.15.31:0/3707470158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b01a1fd0 con 0x7f74b0103b40 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b4a51640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0103b40 0x7f74b019a610 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f74a000e9f0 tx=0x7f74a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74a67fc640 1 -- 172.21.15.31:0/3707470158 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a000cd80 con 0x7f74b0103b40 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74a67fc640 1 -- 172.21.15.31:0/3707470158 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74a00108e0 con 0x7f74b0103b40 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b01a22c0 con 0x7f74b0103b40 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74a67fc640 1 -- 172.21.15.31:0/3707470158 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74a0018980 con 0x7f74b0103b40 2024-04-25T16:13:46.114 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.109+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b01a2810 con 0x7f74b0103b40 2024-04-25T16:13:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.110+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b006b350 con 0x7f74b0103b40 2024-04-25T16:13:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.112+0000 7f74a67fc640 1 -- 172.21.15.31:0/3707470158 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f74a0010430 con 0x7f74b0103b40 2024-04-25T16:13:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.112+0000 7f74a67fc640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f749c0afaf0 0x7f749c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.112+0000 7f74a67fc640 1 -- 172.21.15.31:0/3707470158 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f74a00df3c0 con 0x7f74b0103b40 2024-04-25T16:13:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.112+0000 7f74b5252640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f749c0afaf0 0x7f749c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.115 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.113+0000 7f74b5252640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f749c0afaf0 0x7f749c0b2000 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009230 tx=0x7f74ac031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:46.118 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.116+0000 7f74a67fc640 1 -- 172.21.15.31:0/3707470158 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74a00a7960 con 0x7f74b0103b40 2024-04-25T16:13:46.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.305+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f74b002dda0 con 0x7f74b0103b40 2024-04-25T16:13:46.308 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.306+0000 7f74a67fc640 1 -- 172.21.15.31:0/3707470158 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v77) ==== 104+0+11 (secure 0 0 0) 0x7f74a00abc60 con 0x7f74b0103b40 2024-04-25T16:13:46.308 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 32 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f749c0afaf0 msgr2=0x7f749c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f749c0afaf0 0x7f749c0b2000 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009230 tx=0x7f74ac031400 comp rx=0 tx=0).stop 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0103b40 msgr2=0x7f74b019a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0103b40 0x7f74b019a610 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f74a000e9f0 tx=0x7f74a000eeb0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 shutdown_connections 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f749c0afaf0 0x7f749c0b2000 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b019f540 0x7f74b01a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b0103b40 0x7f74b019a610 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 --2- 172.21.15.31:0/3707470158 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b0102fe0 0x7f74b019a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.311 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 >> 172.21.15.31:0/3707470158 conn(0x7f74b00fd380 msgr2=0x7f74b00ff7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:46.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 shutdown_connections 2024-04-25T16:13:46.312 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.308+0000 7f74b74b6640 1 -- 172.21.15.31:0/3707470158 wait complete. 2024-04-25T16:13:46.322 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-25T16:13:46.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.439+0000 7fafe3215640 1 -- 172.21.15.31:0/2747233531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc102a00 msgr2=0x7fafdc103700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.444 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.439+0000 7fafe3215640 1 --2- 172.21.15.31:0/2747233531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc102a00 0x7fafdc103700 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fafd0009900 tx=0x7fafd002a410 comp rx=0 tx=0).stop 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.439+0000 7fafe3215640 1 -- 172.21.15.31:0/2747233531 shutdown_connections 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.439+0000 7fafe3215640 1 --2- 172.21.15.31:0/2747233531 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafdc108940 0x7fafdc10ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.439+0000 7fafe3215640 1 --2- 172.21.15.31:0/2747233531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafdc103c40 0x7fafdc104050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.439+0000 7fafe3215640 1 --2- 172.21.15.31:0/2747233531 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc102a00 0x7fafdc103700 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.439+0000 7fafe3215640 1 -- 172.21.15.31:0/2747233531 >> 172.21.15.31:0/2747233531 conn(0x7fafdc0fdbf0 msgr2=0x7fafdc1000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 -- 172.21.15.31:0/2747233531 shutdown_connections 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 -- 172.21.15.31:0/2747233531 wait complete. 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 Processor -- start 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 -- start start 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafdc102a00 0x7fafdc19a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafdc103c40 0x7fafdc19a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc108940 0x7fafdc1a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.445 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10e4e0 con 0x7fafdc108940 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10e360 con 0x7fafdc102a00 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.440+0000 7fafe3215640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10e660 con 0x7fafdc103c40 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe17b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc108940 0x7fafdc1a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe17b2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc108940 0x7fafdc1a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41576/0 (socket says 172.21.15.31:41576) 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe17b2640 1 -- 172.21.15.31:0/149116629 learned_addr learned my addr 172.21.15.31:0/149116629 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe0fb1640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafdc102a00 0x7fafdc19a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafdbfff640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafdc103c40 0x7fafdc19a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe17b2640 1 -- 172.21.15.31:0/149116629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafdc103c40 msgr2=0x7fafdc19a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe17b2640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafdc103c40 0x7fafdc19a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe17b2640 1 -- 172.21.15.31:0/149116629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafdc102a00 msgr2=0x7fafdc19a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe17b2640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafdc102a00 0x7fafdc19a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe17b2640 1 -- 172.21.15.31:0/149116629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafdc1a1da0 con 0x7fafdc108940 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafdbfff640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafdc103c40 0x7fafdc19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:46.446 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe17b2640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc108940 0x7fafdc1a1730 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fafd400eaa0 tx=0x7fafd400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:46.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafd9ffb640 1 -- 172.21.15.31:0/149116629 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd400cdb0 con 0x7fafdc108940 2024-04-25T16:13:46.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafdc1a2090 con 0x7fafdc108940 2024-04-25T16:13:46.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafd9ffb640 1 -- 172.21.15.31:0/149116629 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafd4005020 con 0x7fafdc108940 2024-04-25T16:13:46.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.441+0000 7fafd9ffb640 1 -- 172.21.15.31:0/149116629 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd40116d0 con 0x7fafdc108940 2024-04-25T16:13:46.447 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.442+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafdc1a2590 con 0x7fafdc108940 2024-04-25T16:13:46.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.443+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafdc1a2850 con 0x7fafdc108940 2024-04-25T16:13:46.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.444+0000 7fafd9ffb640 1 -- 172.21.15.31:0/149116629 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fafd40051d0 con 0x7fafdc108940 2024-04-25T16:13:46.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.445+0000 7fafd9ffb640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fafcc0afaf0 0x7fafcc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.445+0000 7fafd9ffb640 1 -- 172.21.15.31:0/149116629 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7fafd40df180 con 0x7fafdc108940 2024-04-25T16:13:46.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.445+0000 7fafe0fb1640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fafcc0afaf0 0x7fafcc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.448 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.445+0000 7fafe0fb1640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fafcc0afaf0 0x7fafcc0b2000 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fafd0005ab0 tx=0x7fafd0031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:46.450 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.448+0000 7fafd9ffb640 1 -- 172.21.15.31:0/149116629 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafd4016030 con 0x7fafdc108940 2024-04-25T16:13:46.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:46 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2335013535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:13:46.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:46 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2973255312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:13:46.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:46 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3707470158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:13:46.637 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.635+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fafdc1a2a20 con 0x7fafdc108940 2024-04-25T16:13:46.638 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.636+0000 7fafd9ffb640 1 -- 172.21.15.31:0/149116629 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v77) ==== 100+0+11 (secure 0 0 0) 0x7fafd40a7d10 con 0x7fafdc108940 2024-04-25T16:13:46.638 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 64 2024-04-25T16:13:46.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fafcc0afaf0 msgr2=0x7fafcc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fafcc0afaf0 0x7fafcc0b2000 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fafd0005ab0 tx=0x7fafd0031400 comp rx=0 tx=0).stop 2024-04-25T16:13:46.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc108940 msgr2=0x7fafdc1a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc108940 0x7fafdc1a1730 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fafd400eaa0 tx=0x7fafd400ef60 comp rx=0 tx=0).stop 2024-04-25T16:13:46.640 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 shutdown_connections 2024-04-25T16:13:46.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fafcc0afaf0 0x7fafcc0b2000 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fafdc108940 0x7fafdc1a1730 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafdc103c40 0x7fafdc19a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 --2- 172.21.15.31:0/149116629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fafdc102a00 0x7fafdc19a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 >> 172.21.15.31:0/149116629 conn(0x7fafdc0fdbf0 msgr2=0x7fafdc074dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:46.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 shutdown_connections 2024-04-25T16:13:46.641 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.638+0000 7fafe3215640 1 -- 172.21.15.31:0/149116629 wait complete. 2024-04-25T16:13:46.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:46 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2335013535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:13:46.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:46 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2973255312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:13:46.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:46 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3707470158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:13:46.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:46 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2335013535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:13:46.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:46 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2973255312' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-25T16:13:46.651 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:46 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3707470158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-25T16:13:46.652 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-25T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.764+0000 7f7938d2d640 1 -- 172.21.15.31:0/1599880398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7934103c20 msgr2=0x7f7934104030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.764+0000 7f7938d2d640 1 --2- 172.21.15.31:0/1599880398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7934103c20 0x7f7934104030 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f7928009900 tx=0x7f792802a410 comp rx=0 tx=0).stop 2024-04-25T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.764+0000 7f7938d2d640 1 -- 172.21.15.31:0/1599880398 shutdown_connections 2024-04-25T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.764+0000 7f7938d2d640 1 --2- 172.21.15.31:0/1599880398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7934108920 0x7f793410ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.764+0000 7f7938d2d640 1 --2- 172.21.15.31:0/1599880398 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f7934103c20 0x7f7934104030 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.769 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.764+0000 7f7938d2d640 1 --2- 172.21.15.31:0/1599880398 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79341029e0 0x7f79341036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.764+0000 7f7938d2d640 1 -- 172.21.15.31:0/1599880398 >> 172.21.15.31:0/1599880398 conn(0x7f79340fdbd0 msgr2=0x7f7934100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.764+0000 7f7938d2d640 1 -- 172.21.15.31:0/1599880398 shutdown_connections 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.764+0000 7f7938d2d640 1 -- 172.21.15.31:0/1599880398 wait complete. 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f7938d2d640 1 Processor -- start 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f7938d2d640 1 -- start start 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f7938d2d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79341029e0 0x7f793419a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f7938d2d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7934103c20 0x7f793419aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f7938d2d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7934108920 0x7f79341a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f7938d2d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f793410e510 con 0x7f79341029e0 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f7938d2d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f793410e390 con 0x7f7934108920 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f7938d2d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f793410e690 con 0x7f7934103c20 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f793259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79341029e0 0x7f793419a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f793259c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79341029e0 0x7f793419a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41592/0 (socket says 172.21.15.31:41592) 2024-04-25T16:13:46.770 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f793259c640 1 -- 172.21.15.31:0/2872242271 learned_addr learned my addr 172.21.15.31:0/2872242271 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.765+0000 7f7932d9d640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7934108920 0x7f79341a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f7931d9b640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7934103c20 0x7f793419aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f793259c640 1 -- 172.21.15.31:0/2872242271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7934103c20 msgr2=0x7f793419aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f793259c640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7934103c20 0x7f793419aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f793259c640 1 -- 172.21.15.31:0/2872242271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7934108920 msgr2=0x7f79341a1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f793259c640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7934108920 0x7f79341a1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f793259c640 1 -- 172.21.15.31:0/2872242271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79341a2070 con 0x7f79341029e0 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f7932d9d640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7934108920 0x7f79341a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f793259c640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79341029e0 0x7f793419a530 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f792400e9f0 tx=0x7f792400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f79237fe640 1 -- 172.21.15.31:0/2872242271 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792400ccc0 con 0x7f79341029e0 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f79237fe640 1 -- 172.21.15.31:0/2872242271 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f79240035f0 con 0x7f79341029e0 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f79237fe640 1 -- 172.21.15.31:0/2872242271 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792400ce60 con 0x7f79341029e0 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79341a2360 con 0x7f79341029e0 2024-04-25T16:13:46.771 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.766+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79341a2860 con 0x7f79341029e0 2024-04-25T16:13:46.772 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.767+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f793406c2a0 con 0x7f79341029e0 2024-04-25T16:13:46.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.768+0000 7f79237fe640 1 -- 172.21.15.31:0/2872242271 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7924003aa0 con 0x7f79341029e0 2024-04-25T16:13:46.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.769+0000 7f79237fe640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f791c0afaf0 0x7f791c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:46.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.770+0000 7f7931d9b640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f791c0afaf0 0x7f791c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:46.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.770+0000 7f79237fe640 1 -- 172.21.15.31:0/2872242271 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f79240df0a0 con 0x7f79341029e0 2024-04-25T16:13:46.773 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.770+0000 7f7931d9b640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f791c0afaf0 0x7f791c0b2000 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f792802a930 tx=0x7f7928000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:46.776 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.774+0000 7f79237fe640 1 -- 172.21.15.31:0/2872242271 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79240a7c30 con 0x7f79341029e0 2024-04-25T16:13:46.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.973+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f79341a2b20 con 0x7f79341029e0 2024-04-25T16:13:46.976 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.974+0000 7f79237fe640 1 -- 172.21.15.31:0/2872242271 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v77) ==== 81+0+590 (secure 0 0 0) 0x7f79240a7530 con 0x7f79341029e0 2024-04-25T16:13:46.976 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/2182379244 2024-04-26T15:51:49.219867+0000 2024-04-25T16:13:46.976 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3143860956 2024-04-26T15:51:49.219867+0000 2024-04-25T16:13:46.976 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/3323797811 2024-04-26T15:51:49.219867+0000 2024-04-25T16:13:46.977 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/3023595010 2024-04-26T15:51:49.219867+0000 2024-04-25T16:13:46.977 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/3549555101 2024-04-26T15:51:21.857593+0000 2024-04-25T16:13:46.977 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6801/3549555101 2024-04-26T15:51:21.857593+0000 2024-04-25T16:13:46.977 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/650992382 2024-04-26T15:51:21.857593+0000 2024-04-25T16:13:46.977 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:6800/3023595010 2024-04-26T15:51:49.219867+0000 2024-04-25T16:13:46.977 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/106399623 2024-04-26T15:51:21.857593+0000 2024-04-25T16:13:46.977 INFO:teuthology.orchestra.run.smithi031.stdout:172.21.15.31:0/2862878115 2024-04-26T15:51:21.857593+0000 2024-04-25T16:13:46.977 INFO:teuthology.orchestra.run.smithi031.stderr:listed 10 entries 2024-04-25T16:13:46.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f791c0afaf0 msgr2=0x7f791c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.978 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f791c0afaf0 0x7f791c0b2000 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f792802a930 tx=0x7f7928000bc0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79341029e0 msgr2=0x7f793419a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79341029e0 0x7f793419a530 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f792400e9f0 tx=0x7f792400eeb0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 shutdown_connections 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f791c0afaf0 0x7f791c0b2000 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7934108920 0x7f79341a1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7934103c20 0x7f793419aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 --2- 172.21.15.31:0/2872242271 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f79341029e0 0x7f793419a530 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 >> 172.21.15.31:0/2872242271 conn(0x7f79340fdbd0 msgr2=0x7f7934109e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 shutdown_connections 2024-04-25T16:13:46.979 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:46.976+0000 7f7938d2d640 1 -- 172.21.15.31:0/2872242271 wait complete. 2024-04-25T16:13:46.990 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-25T16:13:46.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:13:46.990 DEBUG:teuthology.orchestra.run.smithi031:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-25T16:13:47.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:47 smithi186 ceph-mon[33258]: pgmap v725: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 33 MiB/s wr, 356 op/s 2024-04-25T16:13:47.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:47 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/149116629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:13:47.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:47 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2872242271' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:13:47.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:47 smithi031 ceph-mon[29140]: pgmap v725: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 33 MiB/s wr, 356 op/s 2024-04-25T16:13:47.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:47 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/149116629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:13:47.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:47 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2872242271' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:13:47.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:47 smithi063 ceph-mon[32413]: pgmap v725: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 33 MiB/s wr, 356 op/s 2024-04-25T16:13:47.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:47 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/149116629' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-25T16:13:47.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:47 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2872242271' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-25T16:13:48.102 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T16:13:48.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:13:48.102 DEBUG:teuthology.orchestra.run.smithi031:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-25T16:13:48.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:13:48.157 DEBUG:teuthology.orchestra.run.smithi031:> ip netns list 2024-04-25T16:13:48.205 INFO:teuthology.orchestra.run.smithi031.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-25T16:13:48.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:13:48.206 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-25T16:13:48.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-25T16:13:48.247 DEBUG:teuthology.orchestra.run.smithi031:> sudo ip link delete ceph-brx 2024-04-25T16:13:48.349 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-25T16:13:48.360 INFO:tasks.cephadm:Teardown begin 2024-04-25T16:13:48.360 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-25T16:13:48.360 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:13:48.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.543+0000 7f80dddb2640 1 -- 172.21.15.31:0/2778775167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 msgr2=0x7f80d8105490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:48.546 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.543+0000 7f80dddb2640 1 --2- 172.21.15.31:0/2778775167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 0x7f80d8105490 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f80c8009900 tx=0x7f80c802a410 comp rx=0 tx=0).stop 2024-04-25T16:13:48.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.545+0000 7f80dddb2640 1 -- 172.21.15.31:0/2778775167 shutdown_connections 2024-04-25T16:13:48.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.545+0000 7f80dddb2640 1 --2- 172.21.15.31:0/2778775167 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80d8102230 0x7f80d810bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:48.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.545+0000 7f80dddb2640 1 --2- 172.21.15.31:0/2778775167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d8101730 0x7f80d8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:48.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.545+0000 7f80dddb2640 1 --2- 172.21.15.31:0/2778775167 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 0x7f80d8105490 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:48.547 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.545+0000 7f80dddb2640 1 -- 172.21.15.31:0/2778775167 >> 172.21.15.31:0/2778775167 conn(0x7f80d80fd330 msgr2=0x7f80d80ff800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:48.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.545+0000 7f80dddb2640 1 -- 172.21.15.31:0/2778775167 shutdown_connections 2024-04-25T16:13:48.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.546+0000 7f80dddb2640 1 -- 172.21.15.31:0/2778775167 wait complete. 2024-04-25T16:13:48.548 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.546+0000 7f80dddb2640 1 Processor -- start 2024-04-25T16:13:48.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80dddb2640 1 -- start start 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80dddb2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d8101730 0x7f80d81a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80dddb2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80d8102230 0x7f80d81a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80dddb2640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 0x7f80d81a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80dddb2640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f80d810d460 con 0x7f80d8105080 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80dddb2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f80d810d2e0 con 0x7f80d8102230 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80dddb2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f80d810d5e0 con 0x7f80d8101730 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 0x7f80d81a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80d7fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 0x7f80d81a7b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41608/0 (socket says 172.21.15.31:41608) 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80d7fff640 1 -- 172.21.15.31:0/3940547316 learned_addr learned my addr 172.21.15.31:0/3940547316 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80d77fe640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d8101730 0x7f80d81a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.547+0000 7f80d6ffd640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80d8102230 0x7f80d81a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:48.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.548+0000 7f80d77fe640 1 -- 172.21.15.31:0/3940547316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80d8102230 msgr2=0x7f80d81a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.548+0000 7f80d77fe640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80d8102230 0x7f80d81a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.548+0000 7f80d77fe640 1 -- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 msgr2=0x7f80d81a7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.548+0000 7f80d77fe640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 0x7f80d81a7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.548+0000 7f80d77fe640 1 -- 172.21.15.31:0/3940547316 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80d81a81c0 con 0x7f80d8101730 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.548+0000 7f80d7fff640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 0x7f80d81a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.548+0000 7f80d77fe640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d8101730 0x7f80d81a0680 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f80c802aa00 tx=0x7f80c8003f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.549+0000 7f80d4ff9640 1 -- 172.21.15.31:0/3940547316 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c803f070 con 0x7f80d8101730 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.549+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80d81a8450 con 0x7f80d8101730 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.549+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80d81a8960 con 0x7f80d8101730 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.549+0000 7f80d4ff9640 1 -- 172.21.15.31:0/3940547316 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80c8031d60 con 0x7f80d8101730 2024-04-25T16:13:48.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.549+0000 7f80d4ff9640 1 -- 172.21.15.31:0/3940547316 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80c8035030 con 0x7f80d8101730 2024-04-25T16:13:48.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.550+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80b8005290 con 0x7f80d8101730 2024-04-25T16:13:48.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.552+0000 7f80d4ff9640 1 -- 172.21.15.31:0/3940547316 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f80c8042070 con 0x7f80d8101730 2024-04-25T16:13:48.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.553+0000 7f80d4ff9640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80c40afc20 0x7f80c40b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:48.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.553+0000 7f80d6ffd640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80c40afc20 0x7f80c40b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:48.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.553+0000 7f80d4ff9640 1 -- 172.21.15.31:0/3940547316 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f80c80ffa60 con 0x7f80d8101730 2024-04-25T16:13:48.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.553+0000 7f80d6ffd640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80c40afc20 0x7f80c40b2130 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f80d81a1720 tx=0x7f80c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:48.562 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.560+0000 7f80d4ff9640 1 -- 172.21.15.31:0/3940547316 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80c80c8030 con 0x7f80d8101730 2024-04-25T16:13:48.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:48 smithi186 ceph-mon[33258]: pgmap v726: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 65 MiB/s wr, 589 op/s 2024-04-25T16:13:48.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:48 smithi031 ceph-mon[29140]: pgmap v726: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 65 MiB/s wr, 589 op/s 2024-04-25T16:13:48.869 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:48 smithi063 ceph-mon[32413]: pgmap v726: 97 pgs: 97 active+clean; 4.0 GiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 65 MiB/s wr, 589 op/s 2024-04-25T16:13:48.982 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.980+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f80b8005580 con 0x7f80d8101730 2024-04-25T16:13:48.983 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.981+0000 7f80d4ff9640 1 -- 172.21.15.31:0/3940547316 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11918 (secure 0 0 0) 0x7f80c80cc330 con 0x7f80d8101730 2024-04-25T16:13:48.984 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:13:48.984 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:12:49.118320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[14631,24553,24563,24521,24535]},"id":1}]} 2024-04-25T16:13:48.984 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2024-04-25T16:13:48.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.985+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80c40afc20 msgr2=0x7f80c40b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:48.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.985+0000 7f80dddb2640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80c40afc20 0x7f80c40b2130 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f80d81a1720 tx=0x7f80c0005f90 comp rx=0 tx=0).stop 2024-04-25T16:13:48.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.985+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d8101730 msgr2=0x7f80d81a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:48.987 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.985+0000 7f80dddb2640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d8101730 0x7f80d81a0680 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f80c802aa00 tx=0x7f80c8003f00 comp rx=0 tx=0).stop 2024-04-25T16:13:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.986+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 shutdown_connections 2024-04-25T16:13:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.986+0000 7f80dddb2640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f80c40afc20 0x7f80c40b2130 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.986+0000 7f80dddb2640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f80d8105080 0x7f80d81a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.986+0000 7f80dddb2640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80d8102230 0x7f80d81a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.986+0000 7f80dddb2640 1 --2- 172.21.15.31:0/3940547316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80d8101730 0x7f80d81a0680 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:48.988 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.986+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 >> 172.21.15.31:0/3940547316 conn(0x7f80d80fd330 msgr2=0x7f80d80ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:48.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.986+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 shutdown_connections 2024-04-25T16:13:48.989 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:48.986+0000 7f80dddb2640 1 -- 172.21.15.31:0/3940547316 wait complete. 2024-04-25T16:13:49.007 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-25T16:13:49.127 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.124+0000 7f1e3e049640 1 -- 172.21.15.31:0/3151280084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38103b00 msgr2=0x7f1e38104800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.124+0000 7f1e3e049640 1 --2- 172.21.15.31:0/3151280084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38103b00 0x7f1e38104800 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f1e28009ca0 tx=0x7f1e2802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.125+0000 7f1e3e049640 1 -- 172.21.15.31:0/3151280084 shutdown_connections 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.125+0000 7f1e3e049640 1 --2- 172.21.15.31:0/3151280084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38109a40 0x7f1e3810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.125+0000 7f1e3e049640 1 --2- 172.21.15.31:0/3151280084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e38104d40 0x7f1e38105150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.125+0000 7f1e3e049640 1 --2- 172.21.15.31:0/3151280084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38103b00 0x7f1e38104800 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.125+0000 7f1e3e049640 1 -- 172.21.15.31:0/3151280084 >> 172.21.15.31:0/3151280084 conn(0x7f1e380fed10 msgr2=0x7f1e381011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.125+0000 7f1e3e049640 1 -- 172.21.15.31:0/3151280084 shutdown_connections 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.125+0000 7f1e3e049640 1 -- 172.21.15.31:0/3151280084 wait complete. 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.125+0000 7f1e3e049640 1 Processor -- start 2024-04-25T16:13:49.128 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.125+0000 7f1e3e049640 1 -- start start 2024-04-25T16:13:49.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e3e049640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38103b00 0x7f1e3819c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e3e049640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e38104d40 0x7f1e3819c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e3e049640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38109a40 0x7f1e381a3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.130 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e3e049640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3810f600 con 0x7f1e38109a40 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e3e049640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3810f480 con 0x7f1e38103b00 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e3e049640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1e3810f780 con 0x7f1e38104d40 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e37fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38109a40 0x7f1e381a3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e377fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38103b00 0x7f1e3819c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e377fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38103b00 0x7f1e3819c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:54944/0 (socket says 172.21.15.31:54944) 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e37fff640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38109a40 0x7f1e381a3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41620/0 (socket says 172.21.15.31:41620) 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e377fe640 1 -- 172.21.15.31:0/4098202220 learned_addr learned my addr 172.21.15.31:0/4098202220 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e36ffd640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e38104d40 0x7f1e3819c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e377fe640 1 -- 172.21.15.31:0/4098202220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e38104d40 msgr2=0x7f1e3819c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e377fe640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e38104d40 0x7f1e3819c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e377fe640 1 -- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38109a40 msgr2=0x7f1e381a3910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e377fe640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38109a40 0x7f1e381a3910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.131 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e377fe640 1 -- 172.21.15.31:0/4098202220 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e381a3f80 con 0x7f1e38103b00 2024-04-25T16:13:49.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e37fff640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38109a40 0x7f1e381a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:13:49.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.126+0000 7f1e377fe640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38103b00 0x7f1e3819c440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1e2802ade0 tx=0x7f1e280316e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:49.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.127+0000 7f1e34ff9640 1 -- 172.21.15.31:0/4098202220 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e28003b70 con 0x7f1e38103b00 2024-04-25T16:13:49.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.127+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e381a4210 con 0x7f1e38103b00 2024-04-25T16:13:49.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.127+0000 7f1e34ff9640 1 -- 172.21.15.31:0/4098202220 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e28003d10 con 0x7f1e38103b00 2024-04-25T16:13:49.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.127+0000 7f1e34ff9640 1 -- 172.21.15.31:0/4098202220 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2802fb90 con 0x7f1e38103b00 2024-04-25T16:13:49.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.127+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e381a4760 con 0x7f1e38103b00 2024-04-25T16:13:49.132 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.128+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e381a49c0 con 0x7f1e38103b00 2024-04-25T16:13:49.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.129+0000 7f1e34ff9640 1 -- 172.21.15.31:0/4098202220 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1e2802fd30 con 0x7f1e38103b00 2024-04-25T16:13:49.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.130+0000 7f1e34ff9640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e240afaf0 0x7f1e240b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.130+0000 7f1e36ffd640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e240afaf0 0x7f1e240b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.130+0000 7f1e34ff9640 1 -- 172.21.15.31:0/4098202220 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f1e281001e0 con 0x7f1e38103b00 2024-04-25T16:13:49.133 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.130+0000 7f1e36ffd640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e240afaf0 0x7f1e240b2000 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1e3819d4e0 tx=0x7f1e20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:49.136 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.133+0000 7f1e34ff9640 1 -- 172.21.15.31:0/4098202220 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e280c8d70 con 0x7f1e38103b00 2024-04-25T16:13:49.363 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.361+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e381a4b90 con 0x7f1e38103b00 2024-04-25T16:13:49.364 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.362+0000 7f1e34ff9640 1 -- 172.21.15.31:0/4098202220 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11918 (secure 0 0 0) 0x7f1e280c8670 con 0x7f1e38103b00 2024-04-25T16:13:49.364 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:13:49.364 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14538,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6835/3224554516","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6834","nonce":3224554516},{"type":"v1","addr":"172.21.15.31:6835","nonce":3224554516}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.31:6837/533105912","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6836","nonce":533105912},{"type":"v1","addr":"172.21.15.31:6837","nonce":533105912}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-25T15:59:02.905418+0000","modified":"2024-04-25T16:12:49.118320+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24563,"mds_1":24553,"mds_2":24535,"mds_3":24521,"mds_4":14631},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14613":{"gid":14613,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.31:6839/3019802236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6838","nonce":3019802236},{"type":"v1","addr":"172.21.15.31:6839","nonce":3019802236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14631":{"gid":14631,"name":"a","rank":4,"incarnation":24,"state":"up:active","state_seq":5,"addr":"172.21.15.31:6841/2149984264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6840","nonce":2149984264},{"type":"v1","addr":"172.21.15.31:6841","nonce":2149984264}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24455":{"gid":24455,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6833/927313408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6832","nonce":927313408},{"type":"v1","addr":"172.21.15.63:6833","nonce":927313408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24460":{"gid":24460,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6833/1701597397","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1701597397},{"type":"v1","addr":"172.21.15.186:6833","nonce":1701597397}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24488":{"gid":24488,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.63:6835/1584006856","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6834","nonce":1584006856},{"type":"v1","addr":"172.21.15.63:6835","nonce":1584006856}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.186:6835/1885074975","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1885074975},{"type":"v1","addr":"172.21.15.186:6835","nonce":1885074975}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24521":{"gid":24521,"name":"h","rank":3,"incarnation":22,"state":"up:active","state_seq":7,"addr":"172.21.15.63:6837/546178402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6836","nonce":546178402},{"type":"v1","addr":"172.21.15.63:6837","nonce":546178402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":20,"state":"up:active","state_seq":8,"addr":"172.21.15.186:6837/509868288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":509868288},{"type":"v1","addr":"172.21.15.186:6837","nonce":509868288}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24553":{"gid":24553,"name":"i","rank":1,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2846674462","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2846674462},{"type":"v1","addr":"172.21.15.186:6839","nonce":2846674462}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24563":{"gid":24563,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.63:6839/495769167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6838","nonce":495769167},{"type":"v1","addr":"172.21.15.63:6839","nonce":495769167}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24563,"qdb_cluster":[14631,24553,24563,24521,24535]},"id":1}]} 2024-04-25T16:13:49.365 INFO:teuthology.orchestra.run.smithi031.stderr:dumped fsmap epoch 30 2024-04-25T16:13:49.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e240afaf0 msgr2=0x7f1e240b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.366 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e240afaf0 0x7f1e240b2000 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f1e3819d4e0 tx=0x7f1e20005f90 comp rx=0 tx=0).stop 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38103b00 msgr2=0x7f1e3819c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38103b00 0x7f1e3819c440 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f1e2802ade0 tx=0x7f1e280316e0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 shutdown_connections 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1e240afaf0 0x7f1e240b2000 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1e38109a40 0x7f1e381a3910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e38104d40 0x7f1e3819c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 --2- 172.21.15.31:0/4098202220 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1e38103b00 0x7f1e3819c440 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 >> 172.21.15.31:0/4098202220 conn(0x7f1e380fed10 msgr2=0x7f1e38101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 shutdown_connections 2024-04-25T16:13:49.367 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.364+0000 7f1e3e049640 1 -- 172.21.15.31:0/4098202220 wait complete. 2024-04-25T16:13:49.379 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-25T16:13:49.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 -- 172.21.15.31:0/214333598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 msgr2=0x7f74b81069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/214333598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 0x7f74b81069d0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f74a8009900 tx=0x7f74a802a410 comp rx=0 tx=0).stop 2024-04-25T16:13:49.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 -- 172.21.15.31:0/214333598 shutdown_connections 2024-04-25T16:13:49.496 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/214333598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b8109870 0x7f74b810bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/214333598 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b8106f10 0x7f74b8109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/214333598 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 0x7f74b81069d0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 -- 172.21.15.31:0/214333598 >> 172.21.15.31:0/214333598 conn(0x7f74b80fdbd0 msgr2=0x7f74b8100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 -- 172.21.15.31:0/214333598 shutdown_connections 2024-04-25T16:13:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 -- 172.21.15.31:0/214333598 wait complete. 2024-04-25T16:13:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.490+0000 7f74bd5d7640 1 Processor -- start 2024-04-25T16:13:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74bd5d7640 1 -- start start 2024-04-25T16:13:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74bd5d7640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 0x7f74b819a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.497 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74bd5d7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b8106f10 0x7f74b819a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74bd5d7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b8109870 0x7f74b81a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74bd5d7640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f74b810f5c0 con 0x7f74b81045b0 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74bd5d7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f74b810f440 con 0x7f74b8109870 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74bd5d7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f74b810f740 con 0x7f74b8106f10 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 0x7f74b819a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b6ffd640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 0x7f74b819a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41638/0 (socket says 172.21.15.31:41638) 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b6ffd640 1 -- 172.21.15.31:0/3952183373 learned_addr learned my addr 172.21.15.31:0/3952183373 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b67fc640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b8106f10 0x7f74b819a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b77fe640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b8109870 0x7f74b81a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b67fc640 1 -- 172.21.15.31:0/3952183373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b8109870 msgr2=0x7f74b81a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.498 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b67fc640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b8109870 0x7f74b81a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b67fc640 1 -- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 msgr2=0x7f74b819a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b67fc640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 0x7f74b819a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b67fc640 1 -- 172.21.15.31:0/3952183373 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74b81a1d40 con 0x7f74b8106f10 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.491+0000 7f74b6ffd640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 0x7f74b819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.492+0000 7f74b67fc640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b8106f10 0x7f74b819a740 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f74ac005fd0 tx=0x7f74ac00dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.492+0000 7f749ffff640 1 -- 172.21.15.31:0/3952183373 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac009c30 con 0x7f74b8106f10 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.492+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74b81a1fd0 con 0x7f74b8106f10 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.492+0000 7f749ffff640 1 -- 172.21.15.31:0/3952183373 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74ac009dd0 con 0x7f74b8106f10 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.492+0000 7f749ffff640 1 -- 172.21.15.31:0/3952183373 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74ac010460 con 0x7f74b8106f10 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.492+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74b81a24c0 con 0x7f74b8106f10 2024-04-25T16:13:49.499 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.493+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74b80ff6f0 con 0x7f74b8106f10 2024-04-25T16:13:49.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.494+0000 7f749ffff640 1 -- 172.21.15.31:0/3952183373 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f74ac003e80 con 0x7f74b8106f10 2024-04-25T16:13:49.500 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.495+0000 7f749ffff640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74a00afaf0 0x7f74a00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.495+0000 7f74b6ffd640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74a00afaf0 0x7f74a00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.495+0000 7f749ffff640 1 -- 172.21.15.31:0/3952183373 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f74ac0dec20 con 0x7f74b8106f10 2024-04-25T16:13:49.501 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.495+0000 7f74b6ffd640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74a00afaf0 0x7f74a00b2000 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f74a8004730 tx=0x7f74a80046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:49.502 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.499+0000 7f749ffff640 1 -- 172.21.15.31:0/3952183373 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f74ac017020 con 0x7f74b8106f10 2024-04-25T16:13:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:49 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3940547316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:13:49.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:49 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/4098202220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:13:49.690 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.687+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f74b802dd70 con 0x7f74b8106f10 2024-04-25T16:13:49.692 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.689+0000 7f749ffff640 1 -- 172.21.15.31:0/3952183373 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) ==== 74+0+22634 (secure 0 0 0) 0x7f74ac0a7810 con 0x7f74b8106f10 2024-04-25T16:13:49.692 INFO:teuthology.orchestra.run.smithi031.stdout: 2024-04-25T16:13:49.692 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":77,"fsid":"74fd3742-031b-11ef-bc93-c7b262605968","created":"2024-04-25T15:50:52.065833+0000","modified":"2024-04-25T15:59:50.662616+0000","last_up_change":"2024-04-25T15:58:12.158407+0000","last_in_change":"2024-04-25T15:57:51.641662+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-25T15:54:17.311776+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-25T15:59:00.978038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-25T15:59:01.829535+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cd266854-c19f-4bb8-9461-5b329b2aa0b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6803","nonce":2387250399}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6805","nonce":2387250399}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6809","nonce":2387250399}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":2387250399},{"type":"v1","addr":"172.21.15.31:6807","nonce":2387250399}]},"public_addr":"172.21.15.31:6803/2387250399","cluster_addr":"172.21.15.31:6805/2387250399","heartbeat_back_addr":"172.21.15.31:6809/2387250399","heartbeat_front_addr":"172.21.15.31:6807/2387250399","state":["exists","up"]},{"osd":1,"uuid":"4b7ee32d-a410-468a-ad35-a2113b0bc4d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6801","nonce":1253577374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6803","nonce":1253577374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6807","nonce":1253577374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":1253577374},{"type":"v1","addr":"172.21.15.63:6805","nonce":1253577374}]},"public_addr":"172.21.15.63:6801/1253577374","cluster_addr":"172.21.15.63:6803/1253577374","heartbeat_back_addr":"172.21.15.63:6807/1253577374","heartbeat_front_addr":"172.21.15.63:6805/1253577374","state":["exists","up"]},{"osd":2,"uuid":"e4927da5-24d1-4c79-b1a6-b8cbc4d6ace4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6801","nonce":1381996284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6803","nonce":1381996284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6807","nonce":1381996284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1381996284},{"type":"v1","addr":"172.21.15.186:6805","nonce":1381996284}]},"public_addr":"172.21.15.186:6801/1381996284","cluster_addr":"172.21.15.186:6803/1381996284","heartbeat_back_addr":"172.21.15.186:6807/1381996284","heartbeat_front_addr":"172.21.15.186:6805/1381996284","state":["exists","up"]},{"osd":3,"uuid":"5c56990d-8026-4d79-bb6f-f1ee793e6a66","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6811","nonce":1212428989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6813","nonce":1212428989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6817","nonce":1212428989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":1212428989},{"type":"v1","addr":"172.21.15.31:6815","nonce":1212428989}]},"public_addr":"172.21.15.31:6811/1212428989","cluster_addr":"172.21.15.31:6813/1212428989","heartbeat_back_addr":"172.21.15.31:6817/1212428989","heartbeat_front_addr":"172.21.15.31:6815/1212428989","state":["exists","up"]},{"osd":4,"uuid":"e948c581-682c-4d78-9e92-d70304b5076e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6809","nonce":2163634076}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6811","nonce":2163634076}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6815","nonce":2163634076}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":2163634076},{"type":"v1","addr":"172.21.15.63:6813","nonce":2163634076}]},"public_addr":"172.21.15.63:6809/2163634076","cluster_addr":"172.21.15.63:6811/2163634076","heartbeat_back_addr":"172.21.15.63:6815/2163634076","heartbeat_front_addr":"172.21.15.63:6813/2163634076","state":["exists","up"]},{"osd":5,"uuid":"79c9c971-2a3e-4117-843f-efd5270a8de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6809","nonce":1998951869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6811","nonce":1998951869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6815","nonce":1998951869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1998951869},{"type":"v1","addr":"172.21.15.186:6813","nonce":1998951869}]},"public_addr":"172.21.15.186:6809/1998951869","cluster_addr":"172.21.15.186:6811/1998951869","heartbeat_back_addr":"172.21.15.186:6815/1998951869","heartbeat_front_addr":"172.21.15.186:6813/1998951869","state":["exists","up"]},{"osd":6,"uuid":"b8955799-23fe-4161-890c-c77eb55760b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6818","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6819","nonce":3847071731}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6820","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6821","nonce":3847071731}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6824","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6825","nonce":3847071731}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6822","nonce":3847071731},{"type":"v1","addr":"172.21.15.31:6823","nonce":3847071731}]},"public_addr":"172.21.15.31:6819/3847071731","cluster_addr":"172.21.15.31:6821/3847071731","heartbeat_back_addr":"172.21.15.31:6825/3847071731","heartbeat_front_addr":"172.21.15.31:6823/3847071731","state":["exists","up"]},{"osd":7,"uuid":"33003c67-ad68-41a1-98eb-aea9138f1432","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6817","nonce":4096527906}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6819","nonce":4096527906}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6823","nonce":4096527906}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":4096527906},{"type":"v1","addr":"172.21.15.63:6821","nonce":4096527906}]},"public_addr":"172.21.15.63:6817/4096527906","cluster_addr":"172.21.15.63:6819/4096527906","heartbeat_back_addr":"172.21.15.63:6823/4096527906","heartbeat_front_addr":"172.21.15.63:6821/4096527906","state":["exists","up"]},{"osd":8,"uuid":"75e3156a-b34c-4a5b-b1bb-3cc3f24a3e0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6817","nonce":82309382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6819","nonce":82309382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6823","nonce":82309382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":82309382},{"type":"v1","addr":"172.21.15.186:6821","nonce":82309382}]},"public_addr":"172.21.15.186:6817/82309382","cluster_addr":"172.21.15.186:6819/82309382","heartbeat_back_addr":"172.21.15.186:6823/82309382","heartbeat_front_addr":"172.21.15.186:6821/82309382","state":["exists","up"]},{"osd":9,"uuid":"37886022-d7c3-400c-aa91-02eedf2a12ee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6826","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6827","nonce":3856015669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6828","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6829","nonce":3856015669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6832","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6833","nonce":3856015669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6830","nonce":3856015669},{"type":"v1","addr":"172.21.15.31:6831","nonce":3856015669}]},"public_addr":"172.21.15.31:6827/3856015669","cluster_addr":"172.21.15.31:6829/3856015669","heartbeat_back_addr":"172.21.15.31:6833/3856015669","heartbeat_front_addr":"172.21.15.31:6831/3856015669","state":["exists","up"]},{"osd":10,"uuid":"97d9d39e-2bd0-47a2-b5a4-cca5bde83613","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6825","nonce":1440937910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6826","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6827","nonce":1440937910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6830","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6831","nonce":1440937910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6828","nonce":1440937910},{"type":"v1","addr":"172.21.15.63:6829","nonce":1440937910}]},"public_addr":"172.21.15.63:6825/1440937910","cluster_addr":"172.21.15.63:6827/1440937910","heartbeat_back_addr":"172.21.15.63:6831/1440937910","heartbeat_front_addr":"172.21.15.63:6829/1440937910","state":["exists","up"]},{"osd":11,"uuid":"ba832dc5-0d0e-420c-9430-12fdc737a86a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6825","nonce":2008780285}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6827","nonce":2008780285}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6831","nonce":2008780285}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":2008780285},{"type":"v1","addr":"172.21.15.186:6829","nonce":2008780285}]},"public_addr":"172.21.15.186:6825/2008780285","cluster_addr":"172.21.15.186:6827/2008780285","heartbeat_back_addr":"172.21.15.186:6831/2008780285","heartbeat_front_addr":"172.21.15.186:6829/2008780285","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:26.018756+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:53:50.471837+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:14.205543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:54:39.679915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:05.663925+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:31.215744+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:55:57.438434+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:24.142097+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:56:49.387823+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:16.584546+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:57:44.073882+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-25T15:58:10.178381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.c","mappings":[{"from":2,"to":1}]},{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":8,"to":11}]},{"pgid":"3.b","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":6}]},{"pgid":"3.1d","mappings":[{"from":4,"to":8}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:0/2182379244":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3143860956":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/3323797811":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6801/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:6800/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6801/3549555101":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/650992382":"2024-04-26T15:51:21.857593+0000","172.21.15.31:6800/3023595010":"2024-04-26T15:51:49.219867+0000","172.21.15.31:0/106399623":"2024-04-26T15:51:21.857593+0000","172.21.15.31:0/2862878115":"2024-04-26T15:51:21.857593+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-25T16:13:49.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.691+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74a00afaf0 msgr2=0x7f74a00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.694 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74a00afaf0 0x7f74a00b2000 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f74a8004730 tx=0x7f74a80046c0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b8106f10 msgr2=0x7f74b819a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b8106f10 0x7f74b819a740 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f74ac005fd0 tx=0x7f74ac00dde0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 shutdown_connections 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f74a00afaf0 0x7f74a00b2000 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f74b8109870 0x7f74b81a16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74b8106f10 0x7f74b819a740 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 --2- 172.21.15.31:0/3952183373 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f74b81045b0 0x7f74b819a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 >> 172.21.15.31:0/3952183373 conn(0x7f74b80fdbd0 msgr2=0x7f74b81051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 shutdown_connections 2024-04-25T16:13:49.695 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.692+0000 7f74bd5d7640 1 -- 172.21.15.31:0/3952183373 wait complete. 2024-04-25T16:13:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:49 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3940547316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:13:49.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:49 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/4098202220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:13:49.707 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-25T16:13:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 -- 172.21.15.31:0/810322657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 msgr2=0x7f56581036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 --2- 172.21.15.31:0/810322657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 0x7f56581036e0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f5654009900 tx=0x7f565402a410 comp rx=0 tx=0).stop 2024-04-25T16:13:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 -- 172.21.15.31:0/810322657 shutdown_connections 2024-04-25T16:13:49.824 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 --2- 172.21.15.31:0/810322657 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5658108920 0x7f565810ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 --2- 172.21.15.31:0/810322657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5658103c20 0x7f5658104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 --2- 172.21.15.31:0/810322657 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 0x7f56581036e0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 -- 172.21.15.31:0/810322657 >> 172.21.15.31:0/810322657 conn(0x7f56580fdbd0 msgr2=0x7f5658100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 -- 172.21.15.31:0/810322657 shutdown_connections 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 -- 172.21.15.31:0/810322657 wait complete. 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 Processor -- start 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.819+0000 7f565f60c640 1 -- start start 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565f60c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 0x7f565819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565f60c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5658103c20 0x7f565819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565f60c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5658108920 0x7f56581a1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565f60c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f565810f610 con 0x7f56581029e0 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565f60c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f565810f490 con 0x7f5658103c20 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565f60c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f565810f790 con 0x7f5658108920 2024-04-25T16:13:49.825 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565d3a8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 0x7f565819a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565d3a8640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 0x7f565819a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41654/0 (socket says 172.21.15.31:41654) 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565d3a8640 1 -- 172.21.15.31:0/1698205296 learned_addr learned my addr 172.21.15.31:0/1698205296 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565cba7640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5658103c20 0x7f565819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565dba9640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5658108920 0x7f56581a1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565d3a8640 1 -- 172.21.15.31:0/1698205296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5658108920 msgr2=0x7f56581a1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565d3a8640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5658108920 0x7f56581a1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565d3a8640 1 -- 172.21.15.31:0/1698205296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5658103c20 msgr2=0x7f565819aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565d3a8640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5658103c20 0x7f565819aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565d3a8640 1 -- 172.21.15.31:0/1698205296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56581a2090 con 0x7f56581029e0 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565cba7640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5658103c20 0x7f565819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565dba9640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5658108920 0x7f56581a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f565d3a8640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 0x7f565819a550 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5654006010 tx=0x7f5654004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:49.826 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f564e7fc640 1 -- 172.21.15.31:0/1698205296 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f565403f030 con 0x7f56581029e0 2024-04-25T16:13:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f564e7fc640 1 -- 172.21.15.31:0/1698205296 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5654003aa0 con 0x7f56581029e0 2024-04-25T16:13:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.820+0000 7f564e7fc640 1 -- 172.21.15.31:0/1698205296 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5654035070 con 0x7f56581029e0 2024-04-25T16:13:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.821+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56581a2320 con 0x7f56581029e0 2024-04-25T16:13:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.821+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56581a2810 con 0x7f56581029e0 2024-04-25T16:13:49.827 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.821+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5658107730 con 0x7f56581029e0 2024-04-25T16:13:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.823+0000 7f564e7fc640 1 -- 172.21.15.31:0/1698205296 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5654042030 con 0x7f56581029e0 2024-04-25T16:13:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.824+0000 7f564e7fc640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f56440afaf0 0x7f56440b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.824+0000 7f565cba7640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f56440afaf0 0x7f56440b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.824+0000 7f564e7fc640 1 -- 172.21.15.31:0/1698205296 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8219+0+0 (secure 0 0 0) 0x7f56540ff7c0 con 0x7f56581029e0 2024-04-25T16:13:49.828 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.824+0000 7f565cba7640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f56440afaf0 0x7f56440b2000 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5648005fd0 tx=0x7f5648005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:49.830 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:49.828+0000 7f564e7fc640 1 -- 172.21.15.31:0/1698205296 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56540c83b0 con 0x7f56581029e0 2024-04-25T16:13:49.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:49 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3940547316' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:13:49.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:49 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/4098202220' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-25T16:13:50.061 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.059+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f565802dd10 con 0x7f56581029e0 2024-04-25T16:13:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.061+0000 7f564e7fc640 1 -- 172.21.15.31:0/1698205296 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v31) ==== 161+0+0 (secure 0 0 0) 0x7f56540c7cb0 con 0x7f56581029e0 2024-04-25T16:13:50.063 INFO:teuthology.orchestra.run.smithi031.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f56440afaf0 msgr2=0x7f56440b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f56440afaf0 0x7f56440b2000 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5648005fd0 tx=0x7f5648005850 comp rx=0 tx=0).stop 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 msgr2=0x7f565819a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 0x7f565819a550 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f5654006010 tx=0x7f5654004270 comp rx=0 tx=0).stop 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 shutdown_connections 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f56440afaf0 0x7f56440b2000 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5658108920 0x7f56581a1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5658103c20 0x7f565819aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 --2- 172.21.15.31:0/1698205296 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f56581029e0 0x7f565819a550 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 >> 172.21.15.31:0/1698205296 conn(0x7f56580fdbd0 msgr2=0x7f5658104900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 shutdown_connections 2024-04-25T16:13:50.069 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.066+0000 7f565f60c640 1 -- 172.21.15.31:0/1698205296 wait complete. 2024-04-25T16:13:50.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-25T16:13:50.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.205+0000 7f1cce48c640 1 -- 172.21.15.31:0/2321840319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cc8101d70 msgr2=0x7f1cc81021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:50.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.205+0000 7f1cce48c640 1 --2- 172.21.15.31:0/2321840319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cc8101d70 0x7f1cc81021c0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009900 tx=0x7f1cb802a410 comp rx=0 tx=0).stop 2024-04-25T16:13:50.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.205+0000 7f1cce48c640 1 -- 172.21.15.31:0/2321840319 shutdown_connections 2024-04-25T16:13:50.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.205+0000 7f1cce48c640 1 --2- 172.21.15.31:0/2321840319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc8107920 0x7f1cc8107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:50.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.205+0000 7f1cce48c640 1 --2- 172.21.15.31:0/2321840319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cc81027b0 0x7f1cc8102c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:50.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.205+0000 7f1cce48c640 1 --2- 172.21.15.31:0/2321840319 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cc8101d70 0x7f1cc81021c0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:50.208 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.205+0000 7f1cce48c640 1 -- 172.21.15.31:0/2321840319 >> 172.21.15.31:0/2321840319 conn(0x7f1cc80fd340 msgr2=0x7f1cc80ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:50.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.205+0000 7f1cce48c640 1 -- 172.21.15.31:0/2321840319 shutdown_connections 2024-04-25T16:13:50.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.205+0000 7f1cce48c640 1 -- 172.21.15.31:0/2321840319 wait complete. 2024-04-25T16:13:50.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cce48c640 1 Processor -- start 2024-04-25T16:13:50.209 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cce48c640 1 -- start start 2024-04-25T16:13:50.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cce48c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cc8101d70 0x7f1cc81a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:50.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cce48c640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cc81027b0 0x7f1cc81a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:50.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cce48c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc8107920 0x7f1cc81a7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:50.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cce48c640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc8115bf0 con 0x7f1cc81027b0 2024-04-25T16:13:50.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cce48c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc8115a70 con 0x7f1cc8101d70 2024-04-25T16:13:50.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cce48c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc8115d70 con 0x7f1cc8107920 2024-04-25T16:13:50.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cc7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cc8101d70 0x7f1cc81a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:50.211 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cc77fe640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cc81027b0 0x7f1cc81a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cc7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cc8101d70 0x7f1cc81a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.31:55024/0 (socket says 172.21.15.31:55024) 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1cc7fff640 1 -- 172.21.15.31:0/1300043888 learned_addr learned my addr 172.21.15.31:0/1300043888 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.206+0000 7f1ccca29640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc8107920 0x7f1cc81a7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cc7fff640 1 -- 172.21.15.31:0/1300043888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc8107920 msgr2=0x7f1cc81a7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cc7fff640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc8107920 0x7f1cc81a7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cc7fff640 1 -- 172.21.15.31:0/1300043888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cc81027b0 msgr2=0x7f1cc81a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cc7fff640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cc81027b0 0x7f1cc81a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cc7fff640 1 -- 172.21.15.31:0/1300043888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc81a8320 con 0x7f1cc8101d70 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cc7fff640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cc8101d70 0x7f1cc81a07e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8006010 tx=0x7f1cb80042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cc57fa640 1 -- 172.21.15.31:0/1300043888 <== mon.2 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb803f030 con 0x7f1cc8101d70 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cc57fa640 1 -- 172.21.15.31:0/1300043888 <== mon.2 v2:172.21.15.63:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cb8031910 con 0x7f1cc8101d70 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc81a85b0 con 0x7f1cc8101d70 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cc57fa640 1 -- 172.21.15.31:0/1300043888 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb8035070 con 0x7f1cc8101d70 2024-04-25T16:13:50.212 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.207+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc81a8af0 con 0x7f1cc8101d70 2024-04-25T16:13:50.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.208+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc8106730 con 0x7f1cc8101d70 2024-04-25T16:13:50.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.210+0000 7f1cc57fa640 1 -- 172.21.15.31:0/1300043888 <== mon.2 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1cb8042030 con 0x7f1cc8101d70 2024-04-25T16:13:50.213 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.211+0000 7f1cc57fa640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1cb00afb70 0x7f1cb00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:50.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.211+0000 7f1cc77fe640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1cb00afb70 0x7f1cb00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:50.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.211+0000 7f1cc57fa640 1 -- 172.21.15.31:0/1300043888 <== mon.2 v2:172.21.15.63:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8649+0+0 (secure 0 0 0) 0x7f1cb80ff940 con 0x7f1cc8101d70 2024-04-25T16:13:50.214 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.211+0000 7f1cc77fe640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1cb00afb70 0x7f1cb00b2080 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1cc81a1880 tx=0x7f1cbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:50.216 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.214+0000 7f1cc57fa640 1 -- 172.21.15.31:0/1300043888 <== mon.2 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cb80c7cd0 con 0x7f1cc8101d70 2024-04-25T16:13:50.372 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a[71143]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.372 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a[71143]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.372 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mds[71149]: -- 172.21.15.31:0/4000059702 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ed853fa40 con 0x564ed8587600 2024-04-25T16:13:50.373 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a[71143]: starting mds.a at 2024-04-25T16:13:50.373 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g[70756]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.373 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g[70756]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.373 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mds[70760]: -- 172.21.15.31:0/2424062986 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5617f1a2ba40 con 0x5617f1a73600 2024-04-25T16:13:50.373 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g[70756]: starting mds.g at 2024-04-25T16:13:50.373 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a[29136]: 2024-04-25T16:13:50.060+0000 7fb6a099e700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T16:13:50.390 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c[64212]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.390 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c[64212]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.390 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mds[64216]: -- 172.21.15.186:0/960568626 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d42ebdba40 con 0x55d42ec23600 2024-04-25T16:13:50.390 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c[64212]: starting mds.c at 2024-04-25T16:13:50.390 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f[63901]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.390 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f[63901]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.390 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mds[63905]: -- 172.21.15.186:0/3435802204 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dff02fda40 con 0x55dff0345a80 2024-04-25T16:13:50.390 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f[63901]: starting mds.f at 2024-04-25T16:13:50.391 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-l[63593]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.391 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-l[63593]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.391 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mds[63597]: -- 172.21.15.186:0/2283849206 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a67bcb1a40 con 0x55a67bcf9600 2024-04-25T16:13:50.391 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-l[63593]: starting mds.l at 2024-04-25T16:13:50.391 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i[64519]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.391 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i[64519]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.391 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mds[64523]: -- 172.21.15.186:0/3599817261 <== mon.0 v2:172.21.15.31:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55596cefda40 con 0x55596cf45a80 2024-04-25T16:13:50.391 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i[64519]: starting mds.i at 2024-04-25T16:13:50.462 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:50.459+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f1cc80637a0 con 0x7f1cc8101d70 2024-04-25T16:13:50.620 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.620 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.621 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: starting mds.b at 2024-04-25T16:13:50.621 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h[62651]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.621 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h[62651]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.621 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mds[62655]: -- 172.21.15.63:0/4251591180 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fd2d4dfa40 con 0x55fd2d527600 2024-04-25T16:13:50.622 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h[62651]: starting mds.h at 2024-04-25T16:13:50.622 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/3952183373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:13:50.622 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1698205296' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T16:13:50.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mon[32413]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:13:50.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mon[32413]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T16:13:50.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1698205296' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T16:13:50.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mon[32413]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T16:13:50.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mon[32413]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T16:13:50.623 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mon[32413]: pgmap v728: 97 pgs: 97 active+clean; 3.5 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 64 MiB/s wr, 507 op/s 2024-04-25T16:13:50.624 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e[62343]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.624 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e[62343]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.624 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mds[62347]: -- 172.21.15.63:0/3245718109 <== mon.1 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f07bd2ba40 con 0x55f07bd73180 2024-04-25T16:13:50.624 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e[62343]: starting mds.e at 2024-04-25T16:13:50.625 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k[62024]: ignoring --setuser ceph since I am not root 2024-04-25T16:13:50.625 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k[62024]: ignoring --setgroup ceph since I am not root 2024-04-25T16:13:50.625 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mds[62028]: -- 172.21.15.63:0/1762023677 <== mon.2 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x559f556b84e0 con 0x559f55703180 2024-04-25T16:13:50.625 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-mds[62028]: -- 172.21.15.63:0/1762023677 <== mon.2 v2:172.21.15.63:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559f556bba40 con 0x559f55703180 2024-04-25T16:13:50.625 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:13:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k[62024]: starting mds.k at 2024-04-25T16:13:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/3952183373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:13:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1698205296' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T16:13:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mon[33258]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:13:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mon[33258]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T16:13:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1698205296' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T16:13:50.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mon[33258]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T16:13:50.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mon[33258]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T16:13:50.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:50 smithi186 ceph-mon[33258]: pgmap v728: 97 pgs: 97 active+clean; 3.5 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 64 MiB/s wr, 507 op/s 2024-04-25T16:13:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/3952183373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-25T16:13:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1698205296' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-25T16:13:50.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mon[29140]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-25T16:13:50.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mon[29140]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-25T16:13:50.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1698205296' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-25T16:13:50.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mon[29140]: osdmap e78: 12 total, 12 up, 12 in 2024-04-25T16:13:50.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mon[29140]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-25T16:13:50.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:50 smithi031 ceph-mon[29140]: pgmap v728: 97 pgs: 97 active+clean; 3.5 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 64 MiB/s wr, 507 op/s 2024-04-25T16:13:51.075 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.073+0000 7f1cc57fa640 1 -- 172.21.15.31:0/1300043888 <== mon.2 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v32) ==== 104+0+0 (secure 0 0 0) 0x7f1cb80cbfd0 con 0x7f1cc8101d70 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.076+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1cb00afb70 msgr2=0x7f1cb00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.076+0000 7f1cce48c640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1cb00afb70 0x7f1cb00b2080 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f1cc81a1880 tx=0x7f1cbc005f90 comp rx=0 tx=0).stop 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.076+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cc8101d70 msgr2=0x7f1cc81a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.076+0000 7f1cce48c640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cc8101d70 0x7f1cc81a07e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8006010 tx=0x7f1cb80042f0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.076+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 shutdown_connections 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.076+0000 7f1cce48c640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f1cb00afb70 0x7f1cb00b2080 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.076+0000 7f1cce48c640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc8107920 0x7f1cc81a7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.076+0000 7f1cce48c640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f1cc81027b0 0x7f1cc81a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.076+0000 7f1cce48c640 1 --2- 172.21.15.31:0/1300043888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cc8101d70 0x7f1cc81a07e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.077+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 >> 172.21.15.31:0/1300043888 conn(0x7f1cc80fd340 msgr2=0x7f1cc8103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:51.079 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.077+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 shutdown_connections 2024-04-25T16:13:51.080 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.077+0000 7f1cce48c640 1 -- 172.21.15.31:0/1300043888 wait complete. 2024-04-25T16:13:51.096 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-25T16:13:51.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 -- 172.21.15.31:0/3749690084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca80108920 msgr2=0x7fca8010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:51.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 --2- 172.21.15.31:0/3749690084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca80108920 0x7fca8010ad40 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fca78009ca0 tx=0x7fca7802a8c0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 -- 172.21.15.31:0/3749690084 shutdown_connections 2024-04-25T16:13:51.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 --2- 172.21.15.31:0/3749690084 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca80108920 0x7fca8010ad40 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.217 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 --2- 172.21.15.31:0/3749690084 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80103c20 0x7fca80104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 --2- 172.21.15.31:0/3749690084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca801029e0 0x7fca801036e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 -- 172.21.15.31:0/3749690084 >> 172.21.15.31:0/3749690084 conn(0x7fca800fdbd0 msgr2=0x7fca80100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:51.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 -- 172.21.15.31:0/3749690084 shutdown_connections 2024-04-25T16:13:51.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 -- 172.21.15.31:0/3749690084 wait complete. 2024-04-25T16:13:51.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 Processor -- start 2024-04-25T16:13:51.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.212+0000 7fca87f7d640 1 -- start start 2024-04-25T16:13:51.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca87f7d640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca801029e0 0x7fca8019a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:51.218 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca87f7d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca80103c20 0x7fca8019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca87f7d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80108920 0x7fca801a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca87f7d640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7fca8010e510 con 0x7fca801029e0 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca87f7d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7fca8010e390 con 0x7fca80108920 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca87f7d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fca8010e690 con 0x7fca80103c20 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85d19640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca801029e0 0x7fca8019a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85d19640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca801029e0 0x7fca8019a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41726/0 (socket says 172.21.15.31:41726) 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85d19640 1 -- 172.21.15.31:0/2067288133 learned_addr learned my addr 172.21.15.31:0/2067288133 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca8651a640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80108920 0x7fca801a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85518640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca80103c20 0x7fca8019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:51.219 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85d19640 1 -- 172.21.15.31:0/2067288133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca80103c20 msgr2=0x7fca8019aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85d19640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca80103c20 0x7fca8019aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85d19640 1 -- 172.21.15.31:0/2067288133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80108920 msgr2=0x7fca801a1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85d19640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80108920 0x7fca801a1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85d19640 1 -- 172.21.15.31:0/2067288133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca801a20f0 con 0x7fca801029e0 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca8651a640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80108920 0x7fca801a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca85d19640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca801029e0 0x7fca8019a5b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00e9f0 tx=0x7fca7c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca76ffd640 1 -- 172.21.15.31:0/2067288133 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c00ccc0 con 0x7fca801029e0 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca801a23e0 con 0x7fca801029e0 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca76ffd640 1 -- 172.21.15.31:0/2067288133 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca7c0035f0 con 0x7fca801029e0 2024-04-25T16:13:51.220 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.213+0000 7fca76ffd640 1 -- 172.21.15.31:0/2067288133 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c00ce60 con 0x7fca801029e0 2024-04-25T16:13:51.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.214+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca801a28e0 con 0x7fca801029e0 2024-04-25T16:13:51.221 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.215+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca8006c2a0 con 0x7fca801029e0 2024-04-25T16:13:51.222 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.216+0000 7fca76ffd640 1 -- 172.21.15.31:0/2067288133 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fca7c003aa0 con 0x7fca801029e0 2024-04-25T16:13:51.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.217+0000 7fca76ffd640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca6c0afaf0 0x7fca6c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:51.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.217+0000 7fca85518640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca6c0afaf0 0x7fca6c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:51.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.217+0000 7fca76ffd640 1 -- 172.21.15.31:0/2067288133 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 8649+0+0 (secure 0 0 0) 0x7fca7c0df2b0 con 0x7fca801029e0 2024-04-25T16:13:51.223 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.218+0000 7fca85518640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca6c0afaf0 0x7fca6c0b2000 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fca8019b650 tx=0x7fca70005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:51.224 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.222+0000 7fca76ffd640 1 -- 172.21.15.31:0/2067288133 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca7c0a7c90 con 0x7fca801029e0 2024-04-25T16:13:51.413 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:51.410+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fca801a2fa0 con 0x7fca801029e0 2024-04-25T16:13:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/1300043888' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:13:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:13:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:13:51.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: Cluster is now healthy 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6832/2162660317,v1:172.21.15.186:6834/2162660317] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6833/1348533253,v1:172.21.15.186:6835/1348533253] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6836/2806723983,v1:172.21.15.186:6838/2806723983] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.186:6837/4032297517,v1:172.21.15.186:6839/4032297517] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6834/4001199304,v1:172.21.15.63:6838/4001199304] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6833/2086424345,v1:172.21.15.63:6837/2086424345] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6832/726273908,v1:172.21.15.63:6836/726273908] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.63:6835/4153059994,v1:172.21.15.63:6839/4153059994] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.31:6838/986862247,v1:172.21.15.31:6840/986862247] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: mds.? [v2:172.21.15.31:6839/4250105961,v1:172.21.15.31:6841/4250105961] up:boot 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: fsmap 12 up:standby 2024-04-25T16:13:51.650 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:13:51.651 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:13:51.651 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:13:51.651 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:13:51.651 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:13:51.651 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:13:51.651 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:13:51.651 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:13:51.651 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:13:51.651 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:51 smithi186 ceph-mon[33258]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:13:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/1300043888' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:13:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:13:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:13:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T16:13:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: Cluster is now healthy 2024-04-25T16:13:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T16:13:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6832/2162660317,v1:172.21.15.186:6834/2162660317] up:boot 2024-04-25T16:13:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6833/1348533253,v1:172.21.15.186:6835/1348533253] up:boot 2024-04-25T16:13:51.701 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6836/2806723983,v1:172.21.15.186:6838/2806723983] up:boot 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.186:6837/4032297517,v1:172.21.15.186:6839/4032297517] up:boot 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6834/4001199304,v1:172.21.15.63:6838/4001199304] up:boot 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6833/2086424345,v1:172.21.15.63:6837/2086424345] up:boot 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6832/726273908,v1:172.21.15.63:6836/726273908] up:boot 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.63:6835/4153059994,v1:172.21.15.63:6839/4153059994] up:boot 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.31:6838/986862247,v1:172.21.15.31:6840/986862247] up:boot 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: mds.? [v2:172.21.15.31:6839/4250105961,v1:172.21.15.31:6841/4250105961] up:boot 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: fsmap 12 up:standby 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:13:51.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:13:51.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:13:51.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:13:51.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:13:51.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:13:51.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:51 smithi031 ceph-mon[29140]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:13:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/1300043888' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:13:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-25T16:13:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-25T16:13:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-25T16:13:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: Cluster is now healthy 2024-04-25T16:13:51.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6832/2162660317,v1:172.21.15.186:6834/2162660317] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6833/1348533253,v1:172.21.15.186:6835/1348533253] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6836/2806723983,v1:172.21.15.186:6838/2806723983] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.186:6837/4032297517,v1:172.21.15.186:6839/4032297517] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6834/4001199304,v1:172.21.15.63:6838/4001199304] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6833/2086424345,v1:172.21.15.63:6837/2086424345] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6832/726273908,v1:172.21.15.63:6836/726273908] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.63:6835/4153059994,v1:172.21.15.63:6839/4153059994] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.31:6838/986862247,v1:172.21.15.31:6840/986862247] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: mds.? [v2:172.21.15.31:6839/4250105961,v1:172.21.15.31:6841/4250105961] up:boot 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: fsmap 12 up:standby 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-25T16:13:51.871 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-25T16:13:51.872 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-25T16:13:51.872 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-25T16:13:51.872 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-25T16:13:51.872 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-25T16:13:51.872 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-25T16:13:51.872 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-25T16:13:51.872 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:51 smithi063 ceph-mon[32413]: from='mgr.14152 172.21.15.31:0/1357020457' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-25T16:13:52.404 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.403+0000 7fca76ffd640 1 -- 172.21.15.31:0/2067288133 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v79) ==== 181+0+0 (secure 0 0 0) 0x7fca7c0a7590 con 0x7fca801029e0 2024-04-25T16:13:52.404 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_metadata' removed 2024-04-25T16:13:52.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.406+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca6c0afaf0 msgr2=0x7fca6c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:52.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.406+0000 7fca87f7d640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca6c0afaf0 0x7fca6c0b2000 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fca8019b650 tx=0x7fca70005f90 comp rx=0 tx=0).stop 2024-04-25T16:13:52.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.406+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca801029e0 msgr2=0x7fca8019a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:52.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.406+0000 7fca87f7d640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca801029e0 0x7fca8019a5b0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fca7c00e9f0 tx=0x7fca7c00eeb0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.407+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 shutdown_connections 2024-04-25T16:13:52.408 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.407+0000 7fca87f7d640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7fca6c0afaf0 0x7fca6c0b2000 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.407+0000 7fca87f7d640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fca80108920 0x7fca801a1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.407+0000 7fca87f7d640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca80103c20 0x7fca8019aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.407+0000 7fca87f7d640 1 --2- 172.21.15.31:0/2067288133 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7fca801029e0 0x7fca8019a5b0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.407+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 >> 172.21.15.31:0/2067288133 conn(0x7fca800fdbd0 msgr2=0x7fca80109e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:52.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.407+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 shutdown_connections 2024-04-25T16:13:52.409 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.407+0000 7fca87f7d640 1 -- 172.21.15.31:0/2067288133 wait complete. 2024-04-25T16:13:52.423 DEBUG:teuthology.orchestra.run.smithi031:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-25T16:13:52.549 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.546+0000 7f235aa68640 1 -- 172.21.15.31:0/3752395640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354103c00 msgr2=0x7f2354104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.546+0000 7f235aa68640 1 --2- 172.21.15.31:0/3752395640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354103c00 0x7f2354104010 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f2344009ca0 tx=0x7f234402a8c0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.550 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.546+0000 7f235aa68640 1 -- 172.21.15.31:0/3752395640 shutdown_connections 2024-04-25T16:13:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.546+0000 7f235aa68640 1 --2- 172.21.15.31:0/3752395640 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2354108900 0x7f235410ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.546+0000 7f235aa68640 1 --2- 172.21.15.31:0/3752395640 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354103c00 0x7f2354104010 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.546+0000 7f235aa68640 1 --2- 172.21.15.31:0/3752395640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23541029c0 0x7f23541036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.546+0000 7f235aa68640 1 -- 172.21.15.31:0/3752395640 >> 172.21.15.31:0/3752395640 conn(0x7f23540fdbd0 msgr2=0x7f2354100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.546+0000 7f235aa68640 1 -- 172.21.15.31:0/3752395640 shutdown_connections 2024-04-25T16:13:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.546+0000 7f235aa68640 1 -- 172.21.15.31:0/3752395640 wait complete. 2024-04-25T16:13:52.551 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f235aa68640 1 Processor -- start 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f235aa68640 1 -- start start 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f235aa68640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23541029c0 0x7f2354199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f235aa68640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2354103c00 0x7f2354199640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f235aa68640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354108900 0x7f23541a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f235aa68640 1 -- --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_getmap magic: 0 -- 0x7f235410e4e0 con 0x7f2354108900 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f235aa68640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 -- 0x7f235410e360 con 0x7f23541029c0 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f235aa68640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f235410e660 con 0x7f2354103c00 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2359005640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354108900 0x7f23541a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2359005640 1 --2- >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354108900 0x7f23541a05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.31:3300/0 says I am v2:172.21.15.31:41748/0 (socket says 172.21.15.31:41748) 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2359005640 1 -- 172.21.15.31:0/2692470703 learned_addr learned my addr 172.21.15.31:0/2692470703 (peer_addr_for_me v2:172.21.15.31:0/0) 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2353fff640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2354103c00 0x7f2354199640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2358804640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23541029c0 0x7f2354199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:52.552 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2359005640 1 -- 172.21.15.31:0/2692470703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2354103c00 msgr2=0x7f2354199640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2359005640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2354103c00 0x7f2354199640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2359005640 1 -- 172.21.15.31:0/2692470703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23541029c0 msgr2=0x7f2354199100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2359005640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23541029c0 0x7f2354199100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.547+0000 7f2359005640 1 -- 172.21.15.31:0/2692470703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23541a0b90 con 0x7f2354108900 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.548+0000 7f2359005640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354108900 0x7f23541a05d0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f233c007900 tx=0x7f233c007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.548+0000 7f2351ffb640 1 -- 172.21.15.31:0/2692470703 <== mon.0 v2:172.21.15.31:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c00e6f0 con 0x7f2354108900 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.548+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2354069ef0 con 0x7f2354108900 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.548+0000 7f2351ffb640 1 -- 172.21.15.31:0/2692470703 <== mon.0 v2:172.21.15.31:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f233c00ed40 con 0x7f2354108900 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.548+0000 7f2351ffb640 1 -- 172.21.15.31:0/2692470703 <== mon.0 v2:172.21.15.31:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233c00ac40 con 0x7f2354108900 2024-04-25T16:13:52.553 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.548+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f235406a3f0 con 0x7f2354108900 2024-04-25T16:13:52.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.549+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f235406af30 con 0x7f2354108900 2024-04-25T16:13:52.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.551+0000 7f2351ffb640 1 -- 172.21.15.31:0/2692470703 <== mon.0 v2:172.21.15.31:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f233c00e890 con 0x7f2354108900 2024-04-25T16:13:52.554 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.551+0000 7f2351ffb640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23400afaf0 0x7f23400b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-25T16:13:52.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.551+0000 7f2351ffb640 1 -- 172.21.15.31:0/2692470703 <== mon.0 v2:172.21.15.31:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8116+0+0 (secure 0 0 0) 0x7f233c0e0530 con 0x7f2354108900 2024-04-25T16:13:52.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.551+0000 7f2358804640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23400afaf0 0x7f23400b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-25T16:13:52.555 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.552+0000 7f2358804640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23400afaf0 0x7f23400b2000 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f234c0058c0 tx=0x7f234c00a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-25T16:13:52.556 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.555+0000 7f2351ffb640 1 -- 172.21.15.31:0/2692470703 <== mon.0 v2:172.21.15.31:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f233c023020 con 0x7f2354108900 2024-04-25T16:13:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:52 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2067288133' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T16:13:52.620 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:52 smithi063 ceph-mon[32413]: pgmap v729: 97 pgs: 97 active+clean; 1.4 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 104 MiB/s wr, 895 op/s 2024-04-25T16:13:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:52 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2067288133' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T16:13:52.649 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:52 smithi186 ceph-mon[33258]: pgmap v729: 97 pgs: 97 active+clean; 1.4 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 104 MiB/s wr, 895 op/s 2024-04-25T16:13:52.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:52 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2067288133' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T16:13:52.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:52 smithi031 ceph-mon[29140]: pgmap v729: 97 pgs: 97 active+clean; 1.4 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 104 MiB/s wr, 895 op/s 2024-04-25T16:13:52.743 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:52.742+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 --> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f2354107180 con 0x7f2354108900 2024-04-25T16:13:53.406 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.404+0000 7f2351ffb640 1 -- 172.21.15.31:0/2692470703 <== mon.0 v2:172.21.15.31:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v80) ==== 169+0+0 (secure 0 0 0) 0x7f233c0a91d0 con 0x7f2354108900 2024-04-25T16:13:53.406 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'cephfs_data' removed 2024-04-25T16:13:53.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23400afaf0 msgr2=0x7f23400b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:53.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23400afaf0 0x7f23400b2000 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f234c0058c0 tx=0x7f234c00a4c0 comp rx=0 tx=0).stop 2024-04-25T16:13:53.410 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354108900 msgr2=0x7f23541a05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-25T16:13:53.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354108900 0x7f23541a05d0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f233c007900 tx=0x7f233c007dc0 comp rx=0 tx=0).stop 2024-04-25T16:13:53.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 shutdown_connections 2024-04-25T16:13:53.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:6800/3984865329,v1:172.21.15.31:6801/3984865329] conn(0x7f23400afaf0 0x7f23400b2000 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:53.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] conn(0x7f2354108900 0x7f23541a05d0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:53.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2354103c00 0x7f2354199640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:53.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 --2- 172.21.15.31:0/2692470703 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f23541029c0 0x7f2354199100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-25T16:13:53.411 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.408+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 >> 172.21.15.31:0/2692470703 conn(0x7f23540fdbd0 msgr2=0x7f23541048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-25T16:13:53.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.409+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 shutdown_connections 2024-04-25T16:13:53.412 INFO:teuthology.orchestra.run.smithi031.stderr:2024-04-25T16:13:53.409+0000 7f235aa68640 1 -- 172.21.15.31:0/2692470703 wait complete. 2024-04-25T16:13:53.430 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:13:53.474 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:13:53.504 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:13:53.533 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-25T16:13:53.533 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T16:13:53.552 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T16:13:53.568 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-25T16:13:53.588 INFO:tasks.cephadm:Stopping all daemons... 2024-04-25T16:13:53.588 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-25T16:13:53.588 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.a 2024-04-25T16:13:53.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:53 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2067288133' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T16:13:53.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:53 smithi031 ceph-mon[29140]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T16:13:53.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:53 smithi031 ceph-mon[29140]: from='client.? 172.21.15.31:0/2692470703' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T16:13:53.700 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:53 smithi031 systemd[1]: Stopping Ceph mon.a for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:13:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:53 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2067288133' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T16:13:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:53 smithi063 ceph-mon[32413]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T16:13:53.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:53 smithi063 ceph-mon[32413]: from='client.? 172.21.15.31:0/2692470703' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T16:13:53.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:53 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2067288133' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-25T16:13:53.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:53 smithi186 ceph-mon[33258]: osdmap e79: 12 total, 12 up, 12 in 2024-04-25T16:13:53.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:53 smithi186 ceph-mon[33258]: from='client.? 172.21.15.31:0/2692470703' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-25T16:13:54.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:53 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a[29136]: 2024-04-25T16:13:53.918+0000 7fb6a61a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:13:54.200 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:53 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a[29136]: 2024-04-25T16:13:53.918+0000 7fb6a61a9700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-25T16:13:54.201 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:54 smithi031 podman[98693]: 2024-04-25 16:13:54.117231516 +0000 UTC m=+0.318087834 container died 3c77fdc22f43d6e43451cecbbab44433d99a3d1440a2a6dea41d3bf7aef738c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-25T16:13:54.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:54 smithi031 podman[98693]: 2024-04-25 16:13:54.56786827 +0000 UTC m=+0.768724576 container cleanup 3c77fdc22f43d6e43451cecbbab44433d99a3d1440a2a6dea41d3bf7aef738c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T16:13:54.804 INFO:journalctl@ceph.mon.a.smithi031.stdout:Apr 25 16:13:54 smithi031 bash[98693]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-a 2024-04-25T16:13:55.024 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.a.service' 2024-04-25T16:13:55.067 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:13:55.067 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-25T16:13:55.067 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-25T16:13:55.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.b 2024-04-25T16:13:55.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:55 smithi063 systemd[1]: Stopping Ceph mon.b for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:13:55.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:55 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-b[32409]: 2024-04-25T16:13:55.540+0000 7fe53151b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:13:55.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:55 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-b[32409]: 2024-04-25T16:13:55.540+0000 7fe53151b700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-25T16:13:55.870 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:55 smithi063 podman[67458]: 2024-04-25 16:13:55.592458927 +0000 UTC m=+0.288837372 container died c5ed416e9877ab5005402e48655d818e709866be333382ec51d4331ceeaefe59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:13:56.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:56 smithi063 podman[67458]: 2024-04-25 16:13:56.041440134 +0000 UTC m=+0.737818573 container cleanup c5ed416e9877ab5005402e48655d818e709866be333382ec51d4331ceeaefe59 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T16:13:56.370 INFO:journalctl@ceph.mon.b.smithi063.stdout:Apr 25 16:13:56 smithi063 bash[67458]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-b 2024-04-25T16:13:56.490 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.b.service' 2024-04-25T16:13:56.538 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:13:56.539 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-25T16:13:56.539 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-25T16:13:56.539 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.c 2024-04-25T16:13:56.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:56 smithi186 systemd[1]: Stopping Ceph mon.c for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:13:57.399 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:57 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-c[33253]: 2024-04-25T16:13:57.117+0000 7f799cf5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:13:57.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:57 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-c[33253]: 2024-04-25T16:13:57.117+0000 7f799cf5b700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-25T16:13:57.400 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:57 smithi186 podman[69510]: 2024-04-25 16:13:57.280853033 +0000 UTC m=+0.515625392 container died ef9abd0df57580b2e405538e3fd04a29dcb29e9eafdb967eed65f8e5fccf3067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424) 2024-04-25T16:13:57.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:57 smithi186 podman[69510]: 2024-04-25 16:13:57.651159787 +0000 UTC m=+0.885932135 container cleanup ef9abd0df57580b2e405538e3fd04a29dcb29e9eafdb967eed65f8e5fccf3067 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-c, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:13:57.899 INFO:journalctl@ceph.mon.c.smithi186.stdout:Apr 25 16:13:57 smithi186 bash[69510]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mon-c 2024-04-25T16:13:58.080 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mon.c.service' 2024-04-25T16:13:58.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:13:58.127 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-25T16:13:58.127 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-25T16:13:58.127 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.x 2024-04-25T16:13:58.450 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 16:13:58 smithi031 systemd[1]: Stopping Ceph mgr.x for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:13:58.849 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 16:13:58 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x[29375]: 2024-04-25T16:13:58.488+0000 7fdb6d85c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:13:58.849 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 16:13:58 smithi031 podman[98815]: 2024-04-25 16:13:58.519514694 +0000 UTC m=+0.198083715 container died 8ac56c1f2c768fd7526ba22c4f7f3042b84356f5d215de729c304af1b4c2a315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T16:13:59.199 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 16:13:58 smithi031 podman[98815]: 2024-04-25 16:13:58.946603756 +0000 UTC m=+0.625172767 container cleanup 8ac56c1f2c768fd7526ba22c4f7f3042b84356f5d215de729c304af1b4c2a315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:13:59.199 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 16:13:58 smithi031 bash[98815]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x 2024-04-25T16:13:59.433 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-25T16:13:59.450 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 16:13:59 smithi031 podman[98829]: 2024-04-25 16:13:59.198275014 +0000 UTC m=+0.674177692 container remove 8ac56c1f2c768fd7526ba22c4f7f3042b84356f5d215de729c304af1b4c2a315 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:994e2f20102a20fa9a27966c14b625cc01884df8, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:13:59.450 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 16:13:59 smithi031 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.x.service: Deactivated successfully. 2024-04-25T16:13:59.450 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 16:13:59 smithi031 systemd[1]: Stopped Ceph mgr.x for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T16:13:59.450 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Apr 25 16:13:59 smithi031 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.x.service: Consumed 1min 30.449s CPU time. 2024-04-25T16:13:59.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:13:59.468 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-25T16:13:59.468 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-25T16:13:59.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.y 2024-04-25T16:13:59.870 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 16:13:59 smithi063 systemd[1]: Stopping Ceph mgr.y for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:14:00.233 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 16:13:59 smithi063 podman[67579]: 2024-04-25 16:13:59.926512688 +0000 UTC m=+0.260823810 container died 1513da908ef814fa568b9849a2e30a87f4af28b59c2380bd402df6e13b855008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-04-25T16:14:00.617 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 16:14:00 smithi063 podman[67579]: 2024-04-25 16:14:00.339738697 +0000 UTC m=+0.674049861 container cleanup 1513da908ef814fa568b9849a2e30a87f4af28b59c2380bd402df6e13b855008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-04-25T16:14:00.617 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 16:14:00 smithi063 bash[67579]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y 2024-04-25T16:14:00.870 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 16:14:00 smithi063 podman[67593]: 2024-04-25 16:14:00.616195317 +0000 UTC m=+0.684821386 container remove 1513da908ef814fa568b9849a2e30a87f4af28b59c2380bd402df6e13b855008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T16:14:00.870 INFO:journalctl@ceph.mgr.y.smithi063.stdout:Apr 25 16:14:00 smithi063 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-25T16:14:00.930 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-25T16:14:00.964 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:14:00.964 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-25T16:14:00.964 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-25T16:14:00.965 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.z 2024-04-25T16:14:01.311 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 16:14:01 smithi186 systemd[1]: Stopping Ceph mgr.z for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:14:01.626 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 16:14:01 smithi186 podman[69633]: 2024-04-25 16:14:01.310664592 +0000 UTC m=+0.152022872 container died 5ec702eb857bc1f8a4b494aaa1cc173195d3bcf3da5c3505a6509e5a97212faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424) 2024-04-25T16:14:01.899 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 16:14:01 smithi186 podman[69633]: 2024-04-25 16:14:01.73309626 +0000 UTC m=+0.574454539 container cleanup 5ec702eb857bc1f8a4b494aaa1cc173195d3bcf3da5c3505a6509e5a97212faa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:14:01.899 INFO:journalctl@ceph.mgr.z.smithi186.stdout:Apr 25 16:14:01 smithi186 bash[69633]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mgr-z 2024-04-25T16:14:02.162 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-25T16:14:02.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:14:02.196 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-25T16:14:02.196 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-25T16:14:02.196 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.0 2024-04-25T16:14:02.519 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:02 smithi031 systemd[1]: Stopping Ceph osd.0 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:14:02.950 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:02 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0[44315]: 2024-04-25T16:14:02.517+0000 7f2b4ffc3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:14:02.950 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:02 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0[44315]: 2024-04-25T16:14:02.517+0000 7f2b4ffc3700 -1 osd.0 80 *** Got signal Terminated *** 2024-04-25T16:14:02.950 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:02 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0[44315]: 2024-04-25T16:14:02.517+0000 7f2b4ffc3700 -1 osd.0 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:14:08.200 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:07 smithi031 podman[98937]: 2024-04-25 16:14:07.893505986 +0000 UTC m=+5.504683625 container died 2d80d52e1ea0ffb33e0f1368df834d78379bed283dd25ff045c39d2825938d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-25T16:14:08.578 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:08 smithi031 podman[98937]: 2024-04-25 16:14:08.317500436 +0000 UTC m=+5.928678119 container cleanup 2d80d52e1ea0ffb33e0f1368df834d78379bed283dd25ff045c39d2825938d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-25T16:14:08.578 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:08 smithi031 bash[98937]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0 2024-04-25T16:14:08.950 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:08 smithi031 podman[98954]: 2024-04-25 16:14:08.576972691 +0000 UTC m=+0.684293325 container remove 2d80d52e1ea0ffb33e0f1368df834d78379bed283dd25ff045c39d2825938d29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:14:09.282 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:09 smithi031 podman[99042]: 2024-04-25 16:14:08.907597603 +0000 UTC m=+0.018682659 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:14:09.282 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:09 smithi031 podman[99042]: 2024-04-25 16:14:09.028030433 +0000 UTC m=+0.139115479 container create 15fb5bd97204371aa47ac8091b518ea4d45014ded6f2ee44f265ecd9375f3178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True) 2024-04-25T16:14:09.282 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:09 smithi031 podman[99042]: 2024-04-25 16:14:09.28151364 +0000 UTC m=+0.392598689 container init 15fb5bd97204371aa47ac8091b518ea4d45014ded6f2ee44f265ecd9375f3178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-25T16:14:09.700 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:09 smithi031 podman[99042]: 2024-04-25 16:14:09.285225589 +0000 UTC m=+0.396310636 container start 15fb5bd97204371aa47ac8091b518ea4d45014ded6f2ee44f265ecd9375f3178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-25T16:14:09.700 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:09 smithi031 podman[99042]: 2024-04-25 16:14:09.414129151 +0000 UTC m=+0.525214197 container attach 15fb5bd97204371aa47ac8091b518ea4d45014ded6f2ee44f265ecd9375f3178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:14:10.200 INFO:journalctl@ceph.osd.0.smithi031.stdout:Apr 25 16:14:09 smithi031 podman[99042]: 2024-04-25 16:14:09.876999503 +0000 UTC m=+0.988084544 container died 15fb5bd97204371aa47ac8091b518ea4d45014ded6f2ee44f265ecd9375f3178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:14:10.583 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.0.service' 2024-04-25T16:14:10.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:14:10.627 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-25T16:14:10.627 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-25T16:14:10.627 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.1 2024-04-25T16:14:10.958 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:10 smithi063 systemd[1]: Stopping Ceph osd.1 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:14:11.369 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1[37016]: 2024-04-25T16:14:10.956+0000 7f1ea8ccf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:14:11.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1[37016]: 2024-04-25T16:14:10.956+0000 7f1ea8ccf700 -1 osd.1 80 *** Got signal Terminated *** 2024-04-25T16:14:11.370 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1[37016]: 2024-04-25T16:14:10.956+0000 7f1ea8ccf700 -1 osd.1 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:14:16.815 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:16 smithi063 podman[67701]: 2024-04-25 16:14:16.50752066 +0000 UTC m=+5.679497698 container died ce26214f0cda43adc565691e7b798bfb1a7c12fae3c7f701fd4806657333791a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-04-25T16:14:17.120 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:16 smithi063 podman[67701]: 2024-04-25 16:14:16.913239866 +0000 UTC m=+6.085216899 container cleanup ce26214f0cda43adc565691e7b798bfb1a7c12fae3c7f701fd4806657333791a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-25T16:14:17.120 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:16 smithi063 bash[67701]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1 2024-04-25T16:14:17.614 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:17 smithi063 podman[67730]: 2024-04-25 16:14:17.189855181 +0000 UTC m=+0.683824659 container remove ce26214f0cda43adc565691e7b798bfb1a7c12fae3c7f701fd4806657333791a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T16:14:17.870 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:17 smithi063 podman[67822]: 2024-04-25 16:14:17.516544334 +0000 UTC m=+0.018432128 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:14:17.870 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:17 smithi063 podman[67822]: 2024-04-25 16:14:17.666252491 +0000 UTC m=+0.168140302 container create aaf787e70641463254b852ee1b57141d392d5a29b04665467971ef0cc10c3984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-25T16:14:18.237 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:17 smithi063 podman[67822]: 2024-04-25 16:14:17.893139973 +0000 UTC m=+0.395027742 container init aaf787e70641463254b852ee1b57141d392d5a29b04665467971ef0cc10c3984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:14:18.237 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:17 smithi063 podman[67822]: 2024-04-25 16:14:17.896548471 +0000 UTC m=+0.398436241 container start aaf787e70641463254b852ee1b57141d392d5a29b04665467971ef0cc10c3984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2) 2024-04-25T16:14:18.620 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:18 smithi063 podman[67822]: 2024-04-25 16:14:18.236521987 +0000 UTC m=+0.738409825 container attach aaf787e70641463254b852ee1b57141d392d5a29b04665467971ef0cc10c3984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:14:19.040 INFO:journalctl@ceph.osd.1.smithi063.stdout:Apr 25 16:14:18 smithi063 podman[67822]: 2024-04-25 16:14:18.683279176 +0000 UTC m=+1.185166956 container died aaf787e70641463254b852ee1b57141d392d5a29b04665467971ef0cc10c3984 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424) 2024-04-25T16:14:19.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.1.service' 2024-04-25T16:14:19.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:14:19.397 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-25T16:14:19.397 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-25T16:14:19.397 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.2 2024-04-25T16:14:19.697 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:19 smithi186 systemd[1]: Stopping Ceph osd.2 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:14:19.957 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2[38392]: 2024-04-25T16:14:19.695+0000 7f0fe2566700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:14:19.957 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2[38392]: 2024-04-25T16:14:19.695+0000 7f0fe2566700 -1 osd.2 80 *** Got signal Terminated *** 2024-04-25T16:14:19.957 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2[38392]: 2024-04-25T16:14:19.695+0000 7f0fe2566700 -1 osd.2 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:14:25.399 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:25 smithi186 podman[69759]: 2024-04-25 16:14:25.141268628 +0000 UTC m=+5.552737569 container died 4b192dd9dbd4e68aef7b86c6b7e38172d0828008213f76bcfcbbdb52e46a0330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-25T16:14:25.808 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:25 smithi186 podman[69759]: 2024-04-25 16:14:25.538447809 +0000 UTC m=+5.949916753 container cleanup 4b192dd9dbd4e68aef7b86c6b7e38172d0828008213f76bcfcbbdb52e46a0330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-25T16:14:25.808 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:25 smithi186 bash[69759]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2 2024-04-25T16:14:26.142 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:25 smithi186 podman[69787]: 2024-04-25 16:14:25.8067842 +0000 UTC m=+0.660021453 container remove 4b192dd9dbd4e68aef7b86c6b7e38172d0828008213f76bcfcbbdb52e46a0330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-25T16:14:26.399 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:26 smithi186 podman[69876]: 2024-04-25 16:14:26.044336833 +0000 UTC m=+0.018526381 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:14:26.399 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:26 smithi186 podman[69876]: 2024-04-25 16:14:26.16637766 +0000 UTC m=+0.140567201 container create f17a46d598a4dbf4d42f4fd29769198fc1e3d370b4d5c447c90863bcfe1bd045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-04-25T16:14:26.738 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:26 smithi186 podman[69876]: 2024-04-25 16:14:26.421853182 +0000 UTC m=+0.396042723 container init f17a46d598a4dbf4d42f4fd29769198fc1e3d370b4d5c447c90863bcfe1bd045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:14:26.738 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:26 smithi186 podman[69876]: 2024-04-25 16:14:26.424821475 +0000 UTC m=+0.399011016 container start f17a46d598a4dbf4d42f4fd29769198fc1e3d370b4d5c447c90863bcfe1bd045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T16:14:26.738 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:26 smithi186 podman[69876]: 2024-04-25 16:14:26.536905819 +0000 UTC m=+0.511095358 container attach f17a46d598a4dbf4d42f4fd29769198fc1e3d370b4d5c447c90863bcfe1bd045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:14:27.086 INFO:journalctl@ceph.osd.2.smithi186.stdout:Apr 25 16:14:26 smithi186 podman[69876]: 2024-04-25 16:14:26.737803162 +0000 UTC m=+0.711992700 container died f17a46d598a4dbf4d42f4fd29769198fc1e3d370b4d5c447c90863bcfe1bd045 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-25T16:14:27.392 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.2.service' 2024-04-25T16:14:27.431 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:14:27.431 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-25T16:14:27.431 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-25T16:14:27.432 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.3 2024-04-25T16:14:27.754 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:27 smithi031 systemd[1]: Stopping Ceph osd.3 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:14:28.024 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:27 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3[50849]: 2024-04-25T16:14:27.752+0000 7f02bcd09700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:14:28.025 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:27 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3[50849]: 2024-04-25T16:14:27.752+0000 7f02bcd09700 -1 osd.3 80 *** Got signal Terminated *** 2024-04-25T16:14:28.025 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:27 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3[50849]: 2024-04-25T16:14:27.752+0000 7f02bcd09700 -1 osd.3 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:14:28.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:28 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:28.810+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:28.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:28 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:28.635+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:29.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:28 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:28.874+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:29.200 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:28 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3[50849]: 2024-04-25T16:14:28.917+0000 7f02b92bf700 -1 osd.3 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:07.256548+0000 front 2024-04-25T16:14:07.256449+0000 (oldest deadline 2024-04-25T16:14:28.257245+0000) 2024-04-25T16:14:29.895 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:29 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:29.465+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:29.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:29.772+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:29.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:29.595+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:29.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:29.547+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:30.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:29 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:29.858+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:30.152 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:29 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3[50849]: 2024-04-25T16:14:29.894+0000 7f02b92bf700 -1 osd.3 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:07.256548+0000 front 2024-04-25T16:14:07.256449+0000 (oldest deadline 2024-04-25T16:14:28.257245+0000) 2024-04-25T16:14:30.419 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:30 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:30.151+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:30.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:30 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:30.418+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:30.803 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:30.549+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:30.803 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:30.542+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:31.119 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:30 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:30.826+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:31.130 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:30 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3[50849]: 2024-04-25T16:14:30.863+0000 7f02b92bf700 -1 osd.3 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:07.256548+0000 front 2024-04-25T16:14:07.256449+0000 (oldest deadline 2024-04-25T16:14:28.257245+0000) 2024-04-25T16:14:31.148 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:30.801+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:31.444 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:31 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:31.129+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:31.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:31 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:31.442+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:31.801 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:31.514+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:31.802 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:31.522+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:32.119 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:31 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:31.842+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:32.149 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:31.800+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:32.158 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:31 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3[50849]: 2024-04-25T16:14:31.862+0000 7f02b92bf700 -1 osd.3 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:07.256548+0000 front 2024-04-25T16:14:07.256449+0000 (oldest deadline 2024-04-25T16:14:28.257245+0000) 2024-04-25T16:14:32.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:32 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:32.157+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:32.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:32.755+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:32.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:32.547+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:32.900 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:32.524+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:32.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:32 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:32.455+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:33.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:32 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:32.796+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:33.439 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:33 smithi031 podman[99150]: 2024-04-25 16:14:33.256124353 +0000 UTC m=+5.625054164 container died c678ce22756e9d087f906a254fadb42a2974bd633af34360e6b6e20b45b8050f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS) 2024-04-25T16:14:33.439 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:33 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:33.140+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:33.701 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:33 smithi031 podman[99150]: 2024-04-25 16:14:33.569357522 +0000 UTC m=+5.938287332 container cleanup c678ce22756e9d087f906a254fadb42a2974bd633af34360e6b6e20b45b8050f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T16:14:33.701 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:33 smithi031 bash[99150]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3 2024-04-25T16:14:33.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:33 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:33.437+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:33.753 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:33.500+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:33.753 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:33.510+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:34.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:33 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:33.776+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:34.122 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:33 smithi031 podman[99179]: 2024-04-25 16:14:33.837169535 +0000 UTC m=+0.581497914 container remove c678ce22756e9d087f906a254fadb42a2974bd633af34360e6b6e20b45b8050f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:14:34.149 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:33.751+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:34.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:34 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:34.121+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:34.450 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:34 smithi031 podman[99267]: 2024-04-25 16:14:34.167804146 +0000 UTC m=+0.018989076 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:14:34.451 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:34 smithi031 podman[99267]: 2024-04-25 16:14:34.321328393 +0000 UTC m=+0.172513311 container create 91f471dda1d997a8bdf7f63124f11e7c594090a3c7d8a8f57823cded2ca4272c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T16:14:34.736 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:34 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:34.459+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:34.737 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:34 smithi031 podman[99267]: 2024-04-25 16:14:34.623703624 +0000 UTC m=+0.474888545 container init 91f471dda1d997a8bdf7f63124f11e7c594090a3c7d8a8f57823cded2ca4272c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-25T16:14:34.737 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:34 smithi031 podman[99267]: 2024-04-25 16:14:34.627018057 +0000 UTC m=+0.478202984 container start 91f471dda1d997a8bdf7f63124f11e7c594090a3c7d8a8f57823cded2ca4272c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424) 2024-04-25T16:14:34.737 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:34 smithi031 podman[99267]: 2024-04-25 16:14:34.735178185 +0000 UTC m=+0.586363121 container attach 91f471dda1d997a8bdf7f63124f11e7c594090a3c7d8a8f57823cded2ca4272c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:14:34.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:34.483+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:34.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:34.715+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:34.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:34.483+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:35.089 INFO:journalctl@ceph.osd.3.smithi031.stdout:Apr 25 16:14:34 smithi031 podman[99267]: 2024-04-25 16:14:34.931673215 +0000 UTC m=+0.782858134 container died 91f471dda1d997a8bdf7f63124f11e7c594090a3c7d8a8f57823cded2ca4272c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-25T16:14:35.119 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:34 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:34.818+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:35.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:35 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:35.088+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:35.523 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.3.service' 2024-04-25T16:14:35.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:14:35.567 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-25T16:14:35.568 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-25T16:14:35.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.4 2024-04-25T16:14:35.870 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:35 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:35.793+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:35.871 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:35 smithi063 systemd[1]: Stopping Ceph osd.4 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:14:35.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:35.685+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:35.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:35.515+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:35.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:35.471+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:35.953 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:35 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:35.478+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:36.370 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:35 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4[43959]: 2024-04-25T16:14:35.877+0000 7f9adc873700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:14:36.370 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:35 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4[43959]: 2024-04-25T16:14:35.877+0000 7f9adc873700 -1 osd.4 80 *** Got signal Terminated *** 2024-04-25T16:14:36.370 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:35 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4[43959]: 2024-04-25T16:14:35.877+0000 7f9adc873700 -1 osd.4 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:14:36.439 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:36 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:36.045+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:36.703 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:36 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:36.438+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:36.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:36.495+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:36.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:36.651+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:36.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:36.489+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:37.119 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:36 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:36.804+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:37.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:37 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:37.076+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:37.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:37 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:37.076+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:37.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:37.639+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:37.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:37.520+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:37.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:37.520+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:37.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:37.463+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:37.900 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:37.463+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:37.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:37 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:37.452+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:37.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:37 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:37.452+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:38.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:37 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:37.834+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:38.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:37 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:37.834+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:38.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:38 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:38.111+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:38.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:38 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:38.111+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:38.620 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:38 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4[43959]: 2024-04-25T16:14:38.295+0000 7f9ad8e29700 -1 osd.4 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.265736+0000 front 2024-04-25T16:14:15.265852+0000 (oldest deadline 2024-04-25T16:14:37.565938+0000) 2024-04-25T16:14:38.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:38 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:38.684+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:38.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:38 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:38.516+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:38.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:38 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:38.516+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:38.900 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:38 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:38.466+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:38.900 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:38 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:38.466+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:38.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:38 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:38.459+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:38.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:38 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:38.459+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:39.119 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:38 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:38.823+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:39.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:38 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:38.823+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:39.426 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:39 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:39.093+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:39.426 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:39 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:39.093+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:39.619 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:39 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4[43959]: 2024-04-25T16:14:39.333+0000 7f9ad8e29700 -1 osd.4 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.265736+0000 front 2024-04-25T16:14:15.265852+0000 (oldest deadline 2024-04-25T16:14:37.565938+0000) 2024-04-25T16:14:39.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:39 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:39.425+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:39.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:39 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:39.425+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:39.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:39 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:39.653+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:39.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:39 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:39.504+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:39.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:39 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:39.504+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:39.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:39 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:39.515+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:39.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:39 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:39.515+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:40.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:39 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:39.961+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:40.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:39 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:39.815+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:40.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:39 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:39.815+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:40.380 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:40 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:40.061+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:40.380 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:40 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:40.061+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:40.619 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:40 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4[43959]: 2024-04-25T16:14:40.341+0000 7f9ad8e29700 -1 osd.4 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.265736+0000 front 2024-04-25T16:14:15.265852+0000 (oldest deadline 2024-04-25T16:14:37.565938+0000) 2024-04-25T16:14:40.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:40 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:40.379+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:40.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:40 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:40.379+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:40.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:40 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:40.543+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:40.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:40 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:40.543+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:40.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:40 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:40.649+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:40.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:40 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:40.649+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:40.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:40 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:40.563+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:40.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:40 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:40.563+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:41.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:40 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:40.951+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:41.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:40 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:40.783+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:41.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:40 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:40.783+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:41.424 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:41 smithi063 podman[67926]: 2024-04-25 16:14:41.275265679 +0000 UTC m=+5.512851544 container died a504b7666b84dfc8728637ccccc8ecca8baf170745c91ea1f6cc3a256ecf119d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T16:14:41.424 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:41 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:41.049+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:41.424 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:41 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:41.049+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:41.694 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:41 smithi063 podman[67926]: 2024-04-25 16:14:41.670387353 +0000 UTC m=+5.907973213 container cleanup a504b7666b84dfc8728637ccccc8ecca8baf170745c91ea1f6cc3a256ecf119d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-25T16:14:41.694 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:41 smithi063 bash[67926]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4 2024-04-25T16:14:41.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:41 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:41.424+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:41.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:41 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:41.424+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:41.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:41 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:41.534+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:41.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:41 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:41.534+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:41.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:41 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:41.616+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:41.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:41 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:41.616+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:41.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:41 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:41.523+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:41.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:41 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:41.523+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:42.120 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:41 smithi063 podman[67945]: 2024-04-25 16:14:41.947183662 +0000 UTC m=+0.675499636 container remove a504b7666b84dfc8728637ccccc8ecca8baf170745c91ea1f6cc3a256ecf119d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:14:42.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:41 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:41.996+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:42.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:41 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:41.787+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:42.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:41 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:41.787+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:42.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:42.018+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:42.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:42.018+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:42.552 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:42 smithi063 podman[68044]: 2024-04-25 16:14:42.167858702 +0000 UTC m=+0.019590120 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:14:42.552 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:42 smithi063 podman[68044]: 2024-04-25 16:14:42.298334228 +0000 UTC m=+0.150065632 container create f2983da84fa336778fe23220f8dcba846e6be4bb2f97726edeccf256589795ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-25T16:14:42.812 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:42 smithi063 podman[68044]: 2024-04-25 16:14:42.551285589 +0000 UTC m=+0.403016994 container init f2983da84fa336778fe23220f8dcba846e6be4bb2f97726edeccf256589795ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:14:42.812 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:42 smithi063 podman[68044]: 2024-04-25 16:14:42.55462033 +0000 UTC m=+0.406351742 container start f2983da84fa336778fe23220f8dcba846e6be4bb2f97726edeccf256589795ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-25T16:14:42.812 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:42 smithi063 podman[68044]: 2024-04-25 16:14:42.659681638 +0000 UTC m=+0.511413049 container attach f2983da84fa336778fe23220f8dcba846e6be4bb2f97726edeccf256589795ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-25T16:14:42.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:42 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:42.489+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:42.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:42 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:42.489+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:42.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:42 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:42.553+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:42.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:42 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:42.553+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:42.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:42 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:42.601+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:42.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:42 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:42.601+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:42.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:42.467+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:42.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:42 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:42.467+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:43.120 INFO:journalctl@ceph.osd.4.smithi063.stdout:Apr 25 16:14:42 smithi063 podman[68044]: 2024-04-25 16:14:42.867133242 +0000 UTC m=+0.718864641 container died f2983da84fa336778fe23220f8dcba846e6be4bb2f97726edeccf256589795ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-04-25T16:14:43.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:42 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:42.968+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:43.121 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:42 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:42.810+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:43.121 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:42 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:42.810+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:43.447 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:43 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:43.046+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:43.447 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:43 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:43.046+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:43.495 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.4.service' 2024-04-25T16:14:43.537 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:14:43.537 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-25T16:14:43.537 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-25T16:14:43.537 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.5 2024-04-25T16:14:43.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:43 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:43.445+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:43.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:43 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:43.445+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:43.873 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:43.612+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:43.873 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:43.612+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:43.874 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:43.465+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:43.874 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:43.465+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:43.874 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:43.537+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:43.874 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:43.537+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:43.874 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:43 smithi186 systemd[1]: Stopping Ceph osd.5 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:14:44.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:43 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:43.968+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:44.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:43 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:43.826+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:44.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:43 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:43.826+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:44.148 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:43.871+0000 7f85f56db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:14:44.149 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:43.871+0000 7f85f56db700 -1 osd.5 80 *** Got signal Terminated *** 2024-04-25T16:14:44.149 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:43 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:43.871+0000 7f85f56db700 -1 osd.5 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:14:44.436 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:44 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:44.068+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:44.436 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:44 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:44.068+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:44.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:44 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:44.434+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:44.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:44 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:44.434+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:44.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:44 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:44.544+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:44.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:44 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:44.544+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:44.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:44 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:44.418+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:44.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:44 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:44.418+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:44.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:44 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:44.643+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:44.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:44 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:44.643+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:45.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:44 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:44.997+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:45.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:44 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:44.861+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:45.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:44 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:44.861+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:45.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:45 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:45.093+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:45.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:45 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:45.093+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:45.649 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:45 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:45.565+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:45.649 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:45 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:45.565+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:45.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:45 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:45.387+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:45.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:45 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:45.387+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:45.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:45 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:45.468+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:45.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:45 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:45.468+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:46.149 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:45 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:45.649+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:46.149 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:45 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:45.649+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:46.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:45 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:45.968+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:46.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:45 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:45.894+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:46.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:45 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:45.894+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:46.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:46.065+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:46.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:46.065+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:46.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:46.065+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:46.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:46 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:46.568+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:46.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:46 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:46.568+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:46.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:46 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:46.608+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:46.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:46 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:46.608+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:46.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:46 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:46.409+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:46.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:46 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:46.409+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:46.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:46.513+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:46.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:46.513+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:46.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:46 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:46.513+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:47.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:47 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:47.010+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:47.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:47 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:47.010+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:47.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:46 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:46.875+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:47.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:46 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:46.875+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:47.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:47.055+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:47.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:47.055+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:47.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:47.055+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:47.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:47 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:47.408+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:47.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:47 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:47.408+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:47.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:47 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:47.590+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:47.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:47 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:47.590+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:47.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:47 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:47.622+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:47.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:47 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:47.622+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:47.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:47.475+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:47.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:47.475+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:47.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:47.475+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:48.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:47 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:47.856+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:48.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:47 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:47.856+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:48.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:47 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:47.989+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:48.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:47 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:47.989+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:48.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:48 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:48.047+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:48.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:48 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:48.047+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:48.451 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:48 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:48.047+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:48.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:48 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:48.560+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.337753+0000 front 2024-04-25T16:14:06.337944+0000 (oldest deadline 2024-04-25T16:14:29.237934+0000) 2024-04-25T16:14:48.899 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:48 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5[45202]: 2024-04-25T16:14:48.560+0000 7f85edc89700 -1 osd.5 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.839832+0000 front 2024-04-25T16:14:13.839719+0000 (oldest deadline 2024-04-25T16:14:37.339836+0000) 2024-04-25T16:14:48.899 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:48 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:48.434+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:48.900 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:48 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:48.434+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:48.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:48 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:48.614+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:48.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:48 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:48.614+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:48.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:48 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:48.614+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:48.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:48 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:48.520+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:48.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:48 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:48.520+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:48.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:48 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:48.520+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:49.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:48 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:48.943+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:49.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:48 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:48.943+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:49.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:48 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:48.906+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:49.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:48 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:48.906+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:49.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:49 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:49.027+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:49.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:49 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:49.027+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:49.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:49 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:49.027+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:49.515 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:49 smithi186 podman[69980]: 2024-04-25 16:14:49.220305441 +0000 UTC m=+5.489338924 container died f35558d125f245b84da3fb81de7f123bc8331b64c0e64da9081030cf42b4a08d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:14:49.516 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:49 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:49.421+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:49.516 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:49 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:49.421+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:49.516 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:49 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:49.421+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:49.842 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:49 smithi186 podman[69980]: 2024-04-25 16:14:49.631197084 +0000 UTC m=+5.900230571 container cleanup f35558d125f245b84da3fb81de7f123bc8331b64c0e64da9081030cf42b4a08d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T16:14:49.842 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:49 smithi186 bash[69980]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5 2024-04-25T16:14:49.842 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:49 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:49.608+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:49.842 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:49 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:49.608+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:49.842 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:49 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:49.608+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:49.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:49 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:49.521+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:49.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:49 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:49.521+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:49.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:49 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:49.521+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:50.149 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:49 smithi186 podman[70000]: 2024-04-25 16:14:49.841099527 +0000 UTC m=+0.620487592 container remove f35558d125f245b84da3fb81de7f123bc8331b64c0e64da9081030cf42b4a08d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:14:50.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:49 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:49.946+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:50.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:49 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:49.946+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:50.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:49 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:49.908+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:50.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:49 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:49.908+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:50.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:49 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:49.908+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:50.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:50.002+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:50.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:50.002+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:50.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:50.002+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:50.581 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:50 smithi186 podman[70090]: 2024-04-25 16:14:50.161230291 +0000 UTC m=+0.018231751 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:14:50.581 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:50 smithi186 podman[70090]: 2024-04-25 16:14:50.267975663 +0000 UTC m=+0.124977116 container create 27eccd5b300dacfb48d21ca4b12fee1a953747b2479e11e5f9ebbe0e9be372b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-25T16:14:50.581 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:50 smithi186 podman[70090]: 2024-04-25 16:14:50.479680701 +0000 UTC m=+0.336682153 container init 27eccd5b300dacfb48d21ca4b12fee1a953747b2479e11e5f9ebbe0e9be372b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5) 2024-04-25T16:14:50.581 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:50 smithi186 podman[70090]: 2024-04-25 16:14:50.482790037 +0000 UTC m=+0.339791492 container start 27eccd5b300dacfb48d21ca4b12fee1a953747b2479e11e5f9ebbe0e9be372b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T16:14:50.581 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:50.374+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:50.581 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:50.374+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:50.581 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:50.374+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:50.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:50.635+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:50.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:50.635+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:50.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:50 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:50.635+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:50.900 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:50 smithi186 podman[70090]: 2024-04-25 16:14:50.579820569 +0000 UTC m=+0.436822046 container attach 27eccd5b300dacfb48d21ca4b12fee1a953747b2479e11e5f9ebbe0e9be372b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:14:50.900 INFO:journalctl@ceph.osd.5.smithi186.stdout:Apr 25 16:14:50 smithi186 podman[70090]: 2024-04-25 16:14:50.784534956 +0000 UTC m=+0.641536410 container died 27eccd5b300dacfb48d21ca4b12fee1a953747b2479e11e5f9ebbe0e9be372b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:14:50.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:50.520+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:50.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:50.520+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:50.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:50 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:50.520+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:51.369 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:50.934+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:51.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:50.934+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:51.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:50.930+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:51.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:50.930+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:51.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:50 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:50.930+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:51.406 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.5.service' 2024-04-25T16:14:51.446 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:14:51.446 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-25T16:14:51.447 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-25T16:14:51.447 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.6 2024-04-25T16:14:51.451 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:51.030+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:51.451 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:51.030+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:51.451 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:51.030+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:51.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:51 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:51.590+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:51.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:51 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:51.590+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:51.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:51 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:51.590+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:51.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:51 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:51.379+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:51.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:51 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:51.379+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:51.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:51 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:51.379+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:51.780 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:51 smithi031 systemd[1]: Stopping Ceph osd.6 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:14:51.781 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:51.778+0000 7f3ad5df3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:14:51.781 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:51.550+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:51.781 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:51.550+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:51.781 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:51.550+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:52.046 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:51.778+0000 7f3ad5df3700 -1 osd.6 80 *** Got signal Terminated *** 2024-04-25T16:14:52.046 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:51.778+0000 7f3ad5df3700 -1 osd.6 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:14:52.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:51 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:51.889+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:52.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:51 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:51.889+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:52.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:51 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:51.947+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:52.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:51 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:51.947+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:52.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:51 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:51.947+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:52.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:52.042+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:52.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:52.042+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:52.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:52.042+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:52.594 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:52 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:52.575+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:52.594 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:52 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:52.575+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:52.594 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:52 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:52.575+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:52.594 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:52 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:52.336+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:52.594 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:52 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:52.336+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:52.594 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:52 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:52.336+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:52.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:52.515+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:52.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:52.515+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:52.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:52.515+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:53.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:52 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:52.888+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:53.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:52 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:52.888+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:53.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:52 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:52.931+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:53.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:52 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:52.931+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:53.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:52 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:52.931+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:53.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:52.998+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:53.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:52.998+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:53.451 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:52 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:52.998+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:53.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:53 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:53.536+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:53.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:53 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:53.536+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:53.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:53 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:53.536+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:53.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:53 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:53.342+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:53.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:53 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:53.342+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:53.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:53 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:53.342+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:53.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:53 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:53.544+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:53.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:53 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:53.544+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:53.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:53 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:53.544+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:54.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:53 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:53.888+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:54.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:53 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:53.888+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:54.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:53 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:53.893+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:54.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:53 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:53.893+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:54.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:53 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:53.893+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:54.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:54 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:54.014+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:54.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:54 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:54.014+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:54.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:54 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:54.014+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:54.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:54 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:54.562+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:54.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:54 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:54.562+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:54.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:54 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:54.562+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:54.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:54 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:54.366+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:54.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:54 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:54.366+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:54.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:54 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:54.366+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:54.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:54 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:54.567+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:54.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:54 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:54.567+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:54.951 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:54 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:54.567+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:55.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:54 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:54.878+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:55.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:54 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:54.878+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:55.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:54 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:54.878+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:14:55.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:54 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:54.943+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:55.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:54 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:54.943+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:55.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:54 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:54.943+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:55.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:54 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:54.943+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:14:55.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:55 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:55.002+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:55.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:55 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:55.002+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:55.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:55 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:55.002+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:55.451 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:55 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:55.002+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:28.829744+0000 front 2024-04-25T16:14:28.829770+0000 (oldest deadline 2024-04-25T16:14:54.129785+0000) 2024-04-25T16:14:55.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:55 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:55.542+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:55.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:55 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:55.542+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:55.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:55 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:55.542+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:55.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:55 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:55.347+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:55.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:55 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:55.347+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:55.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:55 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:55.347+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:55.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:55 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:55.547+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:55.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:55 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:55.547+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:55.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:55 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:55.547+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:55.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:55 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:55.547+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:14:56.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:55 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:55.890+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:56.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:55 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:55.890+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:56.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:55 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:55.890+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:14:56.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:55 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:55.941+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:56.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:55 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:55.941+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:56.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:55 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:55.941+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:56.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:55 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:55.942+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:14:56.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:56.039+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.324781+0000 front 2024-04-25T16:14:03.324806+0000 (oldest deadline 2024-04-25T16:14:29.224951+0000) 2024-04-25T16:14:56.450 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:56.039+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:15.626902+0000 front 2024-04-25T16:14:15.626835+0000 (oldest deadline 2024-04-25T16:14:36.726608+0000) 2024-04-25T16:14:56.451 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:56.039+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:20.629109+0000 front 2024-04-25T16:14:20.629058+0000 (oldest deadline 2024-04-25T16:14:45.928966+0000) 2024-04-25T16:14:56.451 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6[57513]: 2024-04-25T16:14:56.039+0000 7f3ad23a9700 -1 osd.6 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:28.829744+0000 front 2024-04-25T16:14:28.829770+0000 (oldest deadline 2024-04-25T16:14:54.129785+0000) 2024-04-25T16:14:56.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:56 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:56.539+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:56.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:56 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:56.539+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:56.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:56 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:56.539+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:56.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:56 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:56.539+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:14:56.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:56 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:56.315+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:56.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:56 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:56.315+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:56.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:56 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:56.315+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:56.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:56.535+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:56.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:56.535+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:56.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:56.535+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:56.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:56 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:56.535+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:14:57.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:56 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:56.843+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:57.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:56 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:56.843+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:57.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:56 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:56.843+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:14:57.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:56 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:56.958+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:57.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:56 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:56.958+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:57.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:56 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:56.958+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:57.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:56 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:56.958+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:14:57.504 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:57 smithi031 podman[99373]: 2024-04-25 16:14:57.232973235 +0000 UTC m=+5.580712231 container stop 0e1d38009e03e6b1d684bc6f06a42fb98bca75d0cb2c329272e9a60a59457b82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T16:14:57.504 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:57 smithi031 podman[99373]: 2024-04-25 16:14:57.258674415 +0000 UTC m=+5.606413414 container died 0e1d38009e03e6b1d684bc6f06a42fb98bca75d0cb2c329272e9a60a59457b82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-25T16:14:57.546 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:57 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:57.278+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:57.546 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:57 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:57.278+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:57.546 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:57 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:57.278+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:57.755 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:57 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:57.502+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:57.755 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:57 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:57.502+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:57.755 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:57 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:57.502+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:57.755 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:57 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:57.502+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:14:57.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:57 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:57.545+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:57.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:57 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:57.545+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:57.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:57 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:57.545+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:57.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:57 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:57.545+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:14:58.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:57 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:57.866+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:58.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:57 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:57.867+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:58.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:57 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:57.867+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:14:58.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:57 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:57.992+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:58.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:57 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:57.992+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:58.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:57 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:57.992+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:58.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:57 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:57.992+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:14:58.200 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:57 smithi031 podman[99373]: 2024-04-25 16:14:57.754025957 +0000 UTC m=+6.101764943 container cleanup 0e1d38009e03e6b1d684bc6f06a42fb98bca75d0cb2c329272e9a60a59457b82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:14:58.200 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:57 smithi031 bash[99373]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6 2024-04-25T16:14:58.200 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:57 smithi031 podman[99390]: 2024-04-25 16:14:57.988882512 +0000 UTC m=+0.748897461 container remove 0e1d38009e03e6b1d684bc6f06a42fb98bca75d0cb2c329272e9a60a59457b82 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:14:58.533 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:58 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:58.259+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:58.533 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:58 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:58.259+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:58.533 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:58 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:58.259+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:58.776 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:58 smithi031 podman[99491]: 2024-04-25 16:14:58.381420716 +0000 UTC m=+0.019766733 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:14:58.776 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:58 smithi031 podman[99491]: 2024-04-25 16:14:58.514674629 +0000 UTC m=+0.153020629 container create 22b917d05da7abed36e40ef64b325e372afa64c5fc54c2bd48022a7dd667b09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:14:58.777 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:58 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:58.463+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:58.777 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:58 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:58.463+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:58.777 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:58 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:58.463+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:58.777 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:58 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:58.463+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:14:58.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:58 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:58.532+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:58.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:58 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:58.532+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:58.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:58 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:58.532+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:58.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:58 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:58.532+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:14:59.085 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:58 smithi031 podman[99491]: 2024-04-25 16:14:58.775624212 +0000 UTC m=+0.413970225 container init 22b917d05da7abed36e40ef64b325e372afa64c5fc54c2bd48022a7dd667b09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T16:14:59.086 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:58 smithi031 podman[99491]: 2024-04-25 16:14:58.779158414 +0000 UTC m=+0.417504417 container start 22b917d05da7abed36e40ef64b325e372afa64c5fc54c2bd48022a7dd667b09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-25T16:14:59.086 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:58 smithi031 podman[99491]: 2024-04-25 16:14:58.867269269 +0000 UTC m=+0.505615270 container attach 22b917d05da7abed36e40ef64b325e372afa64c5fc54c2bd48022a7dd667b09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:14:59.119 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:58 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:58.861+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:14:59.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:58 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:58.861+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:14:59.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:58 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:58.861+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:14:59.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:59.020+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:14:59.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:59.020+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:14:59.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:59.020+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:14:59.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:59.020+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:14:59.404 INFO:journalctl@ceph.osd.6.smithi031.stdout:Apr 25 16:14:59 smithi031 podman[99491]: 2024-04-25 16:14:59.085123332 +0000 UTC m=+0.723469339 container died 22b917d05da7abed36e40ef64b325e372afa64c5fc54c2bd48022a7dd667b09d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True) 2024-04-25T16:14:59.497 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:59 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:59.236+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:14:59.497 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:59 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:59.236+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:14:59.497 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:14:59 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:14:59.236+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:14:59.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:59 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:59.416+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:14:59.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:59 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:59.416+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:14:59.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:59 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:59.416+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:14:59.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:14:59 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:14:59.416+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:14:59.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.6.service' 2024-04-25T16:14:59.836 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:14:59.836 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-25T16:14:59.836 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-25T16:14:59.836 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.7 2024-04-25T16:14:59.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:59 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:59.496+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:14:59.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:59 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:59.496+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:14:59.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:59 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:59.496+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:14:59.900 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:14:59 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:14:59.496+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:00.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:59 smithi063 systemd[1]: Stopping Ceph osd.7 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:00.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:59.894+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:15:00.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:59.894+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:15:00.121 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:14:59.894+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:15:00.121 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:59.985+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:00.121 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:59.985+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:00.121 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:59.985+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:00.121 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:14:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:14:59.985+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:00.484 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:00 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:00.206+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:00.484 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:00 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:00.206+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:00.484 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:00 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:00.206+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:00.620 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:00 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:00.157+0000 7f02f8120700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:00.620 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:00 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:00.157+0000 7f02f8120700 -1 osd.7 80 *** Got signal Terminated *** 2024-04-25T16:15:00.620 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:00 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:00.157+0000 7f02f8120700 -1 osd.7 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:15:00.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:00 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:00.418+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:00.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:00 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:00.418+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:00.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:00 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:00.418+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:00.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:00 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:00.418+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:00.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:00 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:00.483+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:00.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:00 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:00.483+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:00.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:00 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:00.483+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:00.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:00 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:00.483+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:01.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:00 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:00.858+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:15:01.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:00 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:00.858+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:15:01.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:00 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:00.858+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:15:01.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:01.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:01.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:01.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:01.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:01.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:01.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:01.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:01.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:01.247+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:01.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:01.247+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:01.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:01.247+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:01.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:01.469+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:01.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:01.469+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:01.650 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:01.469+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:01.650 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:01 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:01.469+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:01.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:01 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:01.413+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:01.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:01 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:01.413+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:01.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:01 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:01.413+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:01.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:01 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:01.413+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:02.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:01.854+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:15:02.121 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:01.854+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:15:02.121 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:01.854+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:15:02.121 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:01.997+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:02.122 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:01.997+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:02.122 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:01.997+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:02.122 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:01 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:01.997+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:02.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:02 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:02.258+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:02.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:02 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:02.258+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:02.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:02 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:02.258+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:02.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:02 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:02.382+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:02.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:02 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:02.382+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:02.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:02 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:02.382+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:02.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:02 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:02.382+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:02.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:02 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:02.497+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:02.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:02 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:02.497+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:02.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:02 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:02.497+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:02.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:02 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:02.497+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:03.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:02 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:02.864+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:15:03.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:02 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:02.864+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:15:03.120 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:02 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:02.864+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:15:03.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.002+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:03.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.002+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:03.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.002+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:03.120 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.002+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:03.121 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.002+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:03.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:03.488+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:03.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:03.488+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:03.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:03.488+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:03.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:03.488+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:03.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:03.260+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:03.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:03.260+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:03.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:03.260+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:03.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:03 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:03.260+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:03.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:03 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:03.349+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:03.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:03 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:03.349+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:03.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:03 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:03.349+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:03.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:03 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:03.349+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:04.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:03.888+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:15:04.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:03.888+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:15:04.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:03.888+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:15:04.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:04.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:04.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:04.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:04.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:03.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:04.534 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:04.233+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:04.534 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:04.233+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:04.534 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:04.233+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:04.534 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:04.233+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:04.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:04 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:04.328+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:04.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:04 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:04.328+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:04.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:04 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:04.328+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:04.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:04 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:04.328+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:04.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:04.533+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:04.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:04.533+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:04.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:04.533+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:04.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:04 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:04.533+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:05.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:04.881+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.890786+0000 front 2024-04-25T16:14:14.890676+0000 (oldest deadline 2024-04-25T16:14:39.590945+0000) 2024-04-25T16:15:05.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:04.881+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.292187+0000 front 2024-04-25T16:14:24.292159+0000 (oldest deadline 2024-04-25T16:14:46.591957+0000) 2024-04-25T16:15:05.370 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7[50675]: 2024-04-25T16:15:04.881+0000 7f02f06ce700 -1 osd.7 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.993761+0000 front 2024-04-25T16:14:29.993563+0000 (oldest deadline 2024-04-25T16:14:54.693585+0000) 2024-04-25T16:15:05.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:04.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:05.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:04.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:05.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:04.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:05.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:04.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:05.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:04 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:04.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:05.565 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:05.210+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:05.565 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:05.210+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:05.565 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:05.210+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:05.565 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:05.210+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:05.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:05 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:05.310+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:05.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:05 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:05.310+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:05.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:05 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:05.310+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:05.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:05 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:05.310+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:05.869 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:05 smithi063 podman[68151]: 2024-04-25 16:15:05.617687334 +0000 UTC m=+5.583974363 container died 0a9359540769e2abaabf61b6a0d7fb463d368eeacf7cdb1b7d07a86d3ba5cc9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-25T16:15:05.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:05.564+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:05.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:05.564+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:05.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:05.564+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:05.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:05 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:05.564+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:06.345 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:06 smithi063 podman[68151]: 2024-04-25 16:15:06.0344884 +0000 UTC m=+6.000775426 container cleanup 0a9359540769e2abaabf61b6a0d7fb463d368eeacf7cdb1b7d07a86d3ba5cc9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:15:06.345 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:06 smithi063 bash[68151]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7 2024-04-25T16:15:06.345 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:05 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:05.955+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:06.345 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:05 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:05.955+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:06.345 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:05 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:05.955+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:06.346 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:05 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:05.955+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:06.346 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:05 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:05.955+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:06.555 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:06.257+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:06.556 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:06.257+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:06.556 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:06.257+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:06.556 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:06.257+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:06.619 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:06 smithi063 podman[68169]: 2024-04-25 16:15:06.344420082 +0000 UTC m=+0.725710387 container remove 0a9359540769e2abaabf61b6a0d7fb463d368eeacf7cdb1b7d07a86d3ba5cc9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-04-25T16:15:06.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:06 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:06.361+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:06.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:06 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:06.361+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:06.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:06 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:06.361+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:06.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:06 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:06.361+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:06.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:06.554+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:06.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:06.554+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:06.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:06.554+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:06.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:06 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:06.554+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:07.040 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:06 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:06.950+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:07.040 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:06 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:06.950+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:07.040 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:06 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:06.950+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:07.040 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:06 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:06.950+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:07.040 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:06 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:06.950+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:07.041 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:06 smithi063 podman[68263]: 2024-04-25 16:15:06.671122562 +0000 UTC m=+0.018544307 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:15:07.041 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:06 smithi063 podman[68263]: 2024-04-25 16:15:06.795352453 +0000 UTC m=+0.142774190 container create 9c1576f3ae1334662ea7b7b48dc7ca0c03f85cdb583a8129c655ddba6efbe8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5) 2024-04-25T16:15:07.352 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:07 smithi063 podman[68263]: 2024-04-25 16:15:07.039515733 +0000 UTC m=+0.386937466 container init 9c1576f3ae1334662ea7b7b48dc7ca0c03f85cdb583a8129c655ddba6efbe8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-25T16:15:07.353 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:07 smithi063 podman[68263]: 2024-04-25 16:15:07.043274159 +0000 UTC m=+0.390695895 container start 9c1576f3ae1334662ea7b7b48dc7ca0c03f85cdb583a8129c655ddba6efbe8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T16:15:07.353 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:07 smithi063 podman[68263]: 2024-04-25 16:15:07.148034203 +0000 UTC m=+0.495455938 container attach 9c1576f3ae1334662ea7b7b48dc7ca0c03f85cdb583a8129c655ddba6efbe8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-25T16:15:07.557 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:07.232+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:07.557 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:07.232+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:07.557 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:07.232+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:07.557 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:07.232+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:07.620 INFO:journalctl@ceph.osd.7.smithi063.stdout:Apr 25 16:15:07 smithi063 podman[68263]: 2024-04-25 16:15:07.352160995 +0000 UTC m=+0.699582730 container died 9c1576f3ae1334662ea7b7b48dc7ca0c03f85cdb583a8129c655ddba6efbe8fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424) 2024-04-25T16:15:07.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:07 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:07.376+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:07.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:07 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:07.376+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:07.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:07 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:07.376+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:07.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:07 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:07.376+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:07.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:07 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:07.376+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:07.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:07.555+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:07.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:07.555+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:07.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:07.555+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:07.899 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:07 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:07.555+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:07.997 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.7.service' 2024-04-25T16:15:08.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:08.036 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-25T16:15:08.036 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-25T16:15:08.036 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.8 2024-04-25T16:15:08.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:07.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:08.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:07.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:08.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:07.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:08.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:07.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:08.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:07.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:08.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:08 smithi186 systemd[1]: Stopping Ceph osd.8 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:08.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:08.371+0000 7f7698d1a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:08.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:08.371+0000 7f7698d1a700 -1 osd.8 80 *** Got signal Terminated *** 2024-04-25T16:15:08.373 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:08.371+0000 7f7698d1a700 -1 osd.8 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:15:08.373 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:08.279+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:08.373 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:08.279+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:08.373 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:08.279+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:08.373 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:08.279+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:08.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:08.524+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:08.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:08.524+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:08.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:08.524+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:08.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:08 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:08.524+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:08.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:08.374+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:08.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:08.374+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:08.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:08.374+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:08.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:08.374+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:08.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:08 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:08.374+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:09.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:08.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:09.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:08.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:09.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:08.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:09.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:08.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:09.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:08 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:08.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:09.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:09 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:09.510+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:09.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:09 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:09.510+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:09.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:09 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:09.510+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:09.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:09 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:09.510+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:09.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:09 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:09.319+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:09.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:09 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:09.319+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:09.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:09 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:09.319+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:09.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:09 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:09.319+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:09.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:09 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:09.331+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:09.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:09 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:09.331+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:09.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:09 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:09.331+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:09.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:09 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:09.331+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:09.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:09 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:09.331+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:10.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:10.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:10.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:10.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:10.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:10.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.015+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:10.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:10 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:10.474+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:10.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:10 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:10.474+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:10.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:10 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:10.474+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:10.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:10 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:10.474+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:10.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:10 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:10.308+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:10.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:10 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:10.308+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:10.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:10 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:10.308+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:10.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:10 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:10.308+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:10.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:10 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:10.339+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:10.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:10 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:10.339+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:10.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:10 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:10.339+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:10.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:10 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:10.339+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:10.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:10 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:10.339+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:11.271 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:11.272 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:11.272 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:11.272 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:11.272 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:11.272 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:10 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:10.968+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:11.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:11.331+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:11.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:11.331+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:11.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:11.331+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:11.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:11.331+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:11.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:11.479+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:11.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:11.479+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:11.649 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:11.479+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:11.650 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:11.479+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:11.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:11 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:11.373+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:11.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:11 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:11.373+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:11.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:11 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:11.373+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:11.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:11 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:11.373+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:11.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:11 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:11.373+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:12.372 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:11 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:11.942+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:12.372 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:11 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:11.942+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:12.372 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:11 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:11.942+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:12.372 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:11 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:11.942+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:12.372 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:11 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:11.942+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:12.372 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:11 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:11.942+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:12.650 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:12 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:12.502+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:02.738774+0000 front 2024-04-25T16:14:02.738577+0000 (oldest deadline 2024-04-25T16:14:28.638633+0000) 2024-04-25T16:15:12.650 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:12 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:12.502+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:14.539956+0000 front 2024-04-25T16:14:14.539942+0000 (oldest deadline 2024-04-25T16:14:40.439707+0000) 2024-04-25T16:15:12.651 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:12 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:12.502+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.441256+0000 front 2024-04-25T16:14:24.441163+0000 (oldest deadline 2024-04-25T16:14:47.941210+0000) 2024-04-25T16:15:12.651 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:12 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8[51761]: 2024-04-25T16:15:12.502+0000 7f76912c8700 -1 osd.8 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:32.543038+0000 front 2024-04-25T16:14:32.543308+0000 (oldest deadline 2024-04-25T16:14:56.042977+0000) 2024-04-25T16:15:12.651 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:12 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:12.310+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:12.651 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:12 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:12.310+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:12.651 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:12 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:12.310+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:12.651 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:12 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:12.310+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:12.651 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:12 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:12.310+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:12.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:12 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:12.397+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:12.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:12 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:12.397+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:12.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:12 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:12.397+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:12.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:12 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:12.397+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:12.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:12 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:12.397+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:13.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:12 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:12.961+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:13.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:12 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:12.961+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:13.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:12 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:12.961+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:13.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:12 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:12.961+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:13.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:12 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:12.961+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:13.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:12 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:12.961+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:13.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:13 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:13.270+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:13.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:13 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:13.270+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:13.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:13 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:13.270+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:13.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:13 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:13.270+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:13.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:13 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:13.270+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:13.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:13 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:13.350+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:13.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:13 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:13.350+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:13.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:13 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:13.350+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:13.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:13 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:13.350+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:13.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:13 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:13.350+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:13.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:13 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:13.350+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.634563+0000 front 2024-04-25T16:14:46.634528+0000 (oldest deadline 2024-04-25T16:15:12.534552+0000) 2024-04-25T16:15:14.149 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:13 smithi186 podman[70208]: 2024-04-25 16:15:13.866522532 +0000 UTC m=+5.628850126 container died 27c75e9fe86989c7f7b7915636836da208738daaafa8bda84a6cd8f454217be4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-04-25T16:15:14.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:13 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:13.978+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:14.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:13 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:13.978+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:14.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:13 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:13.978+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:14.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:13 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:13.978+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:14.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:13 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:13.978+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:14.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:13 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:13.978+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:14.499 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:14 smithi186 podman[70208]: 2024-04-25 16:15:14.263858123 +0000 UTC m=+6.026185760 container cleanup 27c75e9fe86989c7f7b7915636836da208738daaafa8bda84a6cd8f454217be4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-25T16:15:14.499 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:14 smithi186 bash[70208]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8 2024-04-25T16:15:14.499 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:14 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:14.299+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:14.500 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:14 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:14.299+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:14.500 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:14 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:14.299+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:14.500 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:14 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:14.299+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:14.500 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:14 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:14.299+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:14.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:14 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:14.366+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:14.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:14 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:14.366+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:14.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:14 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:14.366+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:14.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:14 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:14.366+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:14.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:14 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:14.366+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:14.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:14 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:14.366+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.634563+0000 front 2024-04-25T16:14:46.634528+0000 (oldest deadline 2024-04-25T16:15:12.534552+0000) 2024-04-25T16:15:14.814 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:14 smithi186 podman[70225]: 2024-04-25 16:15:14.498608702 +0000 UTC m=+0.629382261 container remove 27c75e9fe86989c7f7b7915636836da208738daaafa8bda84a6cd8f454217be4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T16:15:15.136 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:14 smithi186 podman[70314]: 2024-04-25 16:15:14.716952841 +0000 UTC m=+0.018331775 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:15:15.136 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:14 smithi186 podman[70314]: 2024-04-25 16:15:14.81621599 +0000 UTC m=+0.117594908 container create 919aef76e9e1b875aaa121db0d4b374df7e3de106c230d4ea95242966df62efe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-25T16:15:15.136 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:15 smithi186 podman[70314]: 2024-04-25 16:15:15.060268886 +0000 UTC m=+0.361647805 container init 919aef76e9e1b875aaa121db0d4b374df7e3de106c230d4ea95242966df62efe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:15:15.136 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:15 smithi186 podman[70314]: 2024-04-25 16:15:15.063281463 +0000 UTC m=+0.364660379 container start 919aef76e9e1b875aaa121db0d4b374df7e3de106c230d4ea95242966df62efe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:15:15.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:14 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:14.957+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:15.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:14 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:14.957+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:15.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:14 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:14.957+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:15.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:14 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:14.957+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:15.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:14 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:14.957+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:15.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:14 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:14.957+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:15.399 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:15 smithi186 podman[70314]: 2024-04-25 16:15:15.135266956 +0000 UTC m=+0.436645888 container attach 919aef76e9e1b875aaa121db0d4b374df7e3de106c230d4ea95242966df62efe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8-deactivate, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-25T16:15:15.399 INFO:journalctl@ceph.osd.8.smithi186.stdout:Apr 25 16:15:15 smithi186 podman[70314]: 2024-04-25 16:15:15.355479549 +0000 UTC m=+0.656858466 container died 919aef76e9e1b875aaa121db0d4b374df7e3de106c230d4ea95242966df62efe (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-25T16:15:15.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:15 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:15.261+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:15.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:15 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:15.261+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:15.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:15 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:15.261+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:15.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:15 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:15.261+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:15.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:15 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:15.261+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:15.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:15.393+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:15.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:15.393+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:15.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:15.393+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:15.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:15.393+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:15.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:15.393+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:15.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:15 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:15.393+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.634563+0000 front 2024-04-25T16:14:46.634528+0000 (oldest deadline 2024-04-25T16:15:12.534552+0000) 2024-04-25T16:15:16.005 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.8.service' 2024-04-25T16:15:16.043 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:16.044 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-25T16:15:16.044 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-25T16:15:16.044 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.9 2024-04-25T16:15:16.365 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 systemd[1]: Stopping Ceph osd.9 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:16.369 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:15 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:15.980+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:16.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:15 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:15.980+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:16.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:15 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:15.980+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:16.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:15 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:15.980+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:16.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:15 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:15.980+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:16.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:15 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:15.980+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:16.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:16 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:16.237+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:16.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:16 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:16.237+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:16.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:16 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:16.237+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:16.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:16 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:16.237+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:16.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:16 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:16.237+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:16.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:16.364+0000 7f45303cb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:16.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:16.364+0000 7f45303cb700 -1 osd.9 80 *** Got signal Terminated *** 2024-04-25T16:15:16.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:16.364+0000 7f45303cb700 -1 osd.9 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:15:16.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:16.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:16.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:16.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:16.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:16.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:16.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:16.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:16.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:16.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:16.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:16 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:16.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.634563+0000 front 2024-04-25T16:14:46.634528+0000 (oldest deadline 2024-04-25T16:15:12.534552+0000) 2024-04-25T16:15:17.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:16 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:16.952+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:17.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:16 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:16.952+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:17.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:16 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:16.952+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:17.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:16 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:16.952+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:17.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:16 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:16.952+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:17.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:16 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:16.952+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:17.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:17 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:17.216+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:17.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:17 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:17.216+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:17.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:17 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:17.216+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:17.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:17 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:17.217+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:17.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:17 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:17.217+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:17.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:17.414+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:17.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:17.414+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:17.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:17.414+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:17.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:17.414+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:17.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:17.414+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:17.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:17 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:17.414+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.634563+0000 front 2024-04-25T16:14:46.634528+0000 (oldest deadline 2024-04-25T16:15:12.534552+0000) 2024-04-25T16:15:18.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:17 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:17.983+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:18.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:17 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:17.983+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:18.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:17 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:17.983+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:18.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:17 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:17.983+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:18.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:17 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:17.983+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:18.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:17 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:17.983+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:18.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:17 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:17.983+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:18.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:18 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:18.217+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:18.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:18 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:18.217+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:18.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:18 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:18.217+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:18.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:18 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:18.217+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:18.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:18 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:18.217+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:18.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:18 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:18.392+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:18.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:18 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:18.392+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:18.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:18 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:18.392+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:18.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:18 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:18.392+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:18.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:18 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:18.392+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:18.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:18 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:18.392+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.634563+0000 front 2024-04-25T16:14:46.634528+0000 (oldest deadline 2024-04-25T16:15:12.534552+0000) 2024-04-25T16:15:19.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:18 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:18.975+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:19.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:18 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:18.975+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:19.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:18 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:18.975+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:19.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:18 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:18.975+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:19.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:18 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:18.975+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:19.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:18 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:18.975+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:19.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:18 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:18.975+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:19.550 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:19.194+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:19.550 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:19.194+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:19.550 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:19.194+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:19.550 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:19.194+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:19.551 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:19.194+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:19.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:19.394+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:19.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:19.394+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:19.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:19.394+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:19.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:19.394+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:19.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:19.394+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:19.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:19 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:19.394+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.634563+0000 front 2024-04-25T16:14:46.634528+0000 (oldest deadline 2024-04-25T16:15:12.534552+0000) 2024-04-25T16:15:20.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:19 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:19.974+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:20.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:19 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:19.974+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:20.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:19 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:19.974+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:20.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:19 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:19.974+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:20.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:19 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:19.974+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:20.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:19 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:19.974+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:20.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:19 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:19.974+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:20.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:20 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:20.222+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:20.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:20 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:20.222+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:20.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:20 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:20.222+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:20.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:20 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:20.222+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:20.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:20 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:20.222+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:20.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:20.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:05.227273+0000 front 2024-04-25T16:14:05.227310+0000 (oldest deadline 2024-04-25T16:14:29.327402+0000) 2024-04-25T16:15:20.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:20.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.929296+0000 front 2024-04-25T16:14:13.929478+0000 (oldest deadline 2024-04-25T16:14:36.829321+0000) 2024-04-25T16:15:20.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:20.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:21.431283+0000 front 2024-04-25T16:14:21.431233+0000 (oldest deadline 2024-04-25T16:14:45.531116+0000) 2024-04-25T16:15:20.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:20.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:29.632114+0000 front 2024-04-25T16:14:29.632157+0000 (oldest deadline 2024-04-25T16:14:55.532213+0000) 2024-04-25T16:15:20.700 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:20.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:40.733870+0000 front 2024-04-25T16:14:40.733925+0000 (oldest deadline 2024-04-25T16:15:06.633953+0000) 2024-04-25T16:15:20.701 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:20 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9[64569]: 2024-04-25T16:15:20.417+0000 7f452c981700 -1 osd.9 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.634563+0000 front 2024-04-25T16:14:46.634528+0000 (oldest deadline 2024-04-25T16:15:12.534552+0000) 2024-04-25T16:15:21.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:20 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:20.959+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:21.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:20 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:20.959+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:21.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:20 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:20.959+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:21.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:20 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:20.959+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:21.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:20 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:20.959+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:21.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:20 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:20.959+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:21.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:20 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:20.959+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:21.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:21 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:21.173+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:21.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:21 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:21.173+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:21.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:21 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:21.173+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:21.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:21 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:21.173+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:21.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:21 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:21.173+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:21.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:21 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:21.173+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:21.973 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:21 smithi031 podman[99597]: 2024-04-25 16:15:21.70990892 +0000 UTC m=+5.463489538 container died 1bf01118a0ce409eee922d905994ac5b4528e9230008268e5f64e16cc1bb3a20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-25T16:15:22.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:21 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:21.964+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:22.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:21 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:21.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:22.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:21 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:21.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:22.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:21 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:21.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:22.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:21 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:21.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:22.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:21 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:21.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:22.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:21 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:21.965+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:22.374 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:22 smithi031 podman[99597]: 2024-04-25 16:15:22.06363814 +0000 UTC m=+5.817218753 container cleanup 1bf01118a0ce409eee922d905994ac5b4528e9230008268e5f64e16cc1bb3a20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=) 2024-04-25T16:15:22.374 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:22 smithi031 bash[99597]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9 2024-04-25T16:15:22.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:22 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:22.139+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:22.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:22 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:22.139+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:22.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:22 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:22.139+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:22.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:22 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:22.139+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:22.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:22 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:22.139+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:22.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:22 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:22.139+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:22.677 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:22 smithi031 podman[99616]: 2024-04-25 16:15:22.373259262 +0000 UTC m=+0.662133651 container remove 1bf01118a0ce409eee922d905994ac5b4528e9230008268e5f64e16cc1bb3a20 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-25T16:15:22.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:22 smithi031 podman[99708]: 2024-04-25 16:15:22.579834731 +0000 UTC m=+0.018407784 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:15:22.950 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:22 smithi031 podman[99708]: 2024-04-25 16:15:22.715626387 +0000 UTC m=+0.154199425 container create e5b82495860a3df5f034aea10293b4ef5a6f71aef94adca0837c202f386d5cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-04-25T16:15:23.304 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:22 smithi031 podman[99708]: 2024-04-25 16:15:22.993407978 +0000 UTC m=+0.431981016 container init e5b82495860a3df5f034aea10293b4ef5a6f71aef94adca0837c202f386d5cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-25T16:15:23.304 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:22 smithi031 podman[99708]: 2024-04-25 16:15:22.997335531 +0000 UTC m=+0.435908583 container start e5b82495860a3df5f034aea10293b4ef5a6f71aef94adca0837c202f386d5cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-25T16:15:23.304 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:23 smithi031 podman[99708]: 2024-04-25 16:15:23.093373044 +0000 UTC m=+0.531946089 container attach e5b82495860a3df5f034aea10293b4ef5a6f71aef94adca0837c202f386d5cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-04-25T16:15:23.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:22 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:22.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:23.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:22 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:22.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:23.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:22 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:22.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:23.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:22 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:22.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:23.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:22 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:22.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:23.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:22 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:22.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:23.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:22 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:22.989+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:23.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:23 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:23.112+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:23.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:23 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:23.112+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:23.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:23 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:23.112+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:23.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:23 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:23.112+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:23.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:23 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:23.112+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:23.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:23 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:23.112+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:23.598 INFO:journalctl@ceph.osd.9.smithi031.stdout:Apr 25 16:15:23 smithi031 podman[99708]: 2024-04-25 16:15:23.303153868 +0000 UTC m=+0.741726909 container died e5b82495860a3df5f034aea10293b4ef5a6f71aef94adca0837c202f386d5cbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T16:15:23.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.9.service' 2024-04-25T16:15:23.971 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:23.971 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-25T16:15:23.971 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-25T16:15:23.971 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.10 2024-04-25T16:15:24.309 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.006+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:24.309 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.006+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:24.310 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.006+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:24.310 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.006+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:24.310 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.006+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:24.310 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.006+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:24.310 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.006+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:24.310 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 systemd[1]: Stopping Ceph osd.10 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:24.310 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.307+0000 7f91680f9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:24.311 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.307+0000 7f91680f9700 -1 osd.10 80 *** Got signal Terminated *** 2024-04-25T16:15:24.311 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:24 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:24.307+0000 7f91680f9700 -1 osd.10 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:15:24.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:24 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:24.143+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:24.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:24 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:24.143+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:24.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:24 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:24.143+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:24.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:24 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:24.143+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:24.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:24 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:24.143+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:24.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:24 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:24.143+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:25.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:25 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:25.054+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:25.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:25 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:25.054+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:25.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:25 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:25.054+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:25.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:25 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:25.054+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:25.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:25 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:25.054+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:25.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:25 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:25.054+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:25.371 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:25 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:25.054+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:25.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:25 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:25.164+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:25.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:25 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:25.164+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:25.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:25 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:25.164+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:25.649 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:25 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:25.164+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:25.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:25 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:25.164+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:25.650 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:25 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:25.164+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:26.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:26 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:26.094+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:26.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:26 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:26.094+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:26.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:26 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:26.094+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:26.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:26 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:26.094+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:26.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:26 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:26.094+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:26.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:26 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:26.094+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:26.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:26 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:26.094+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:26.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:26 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:26.120+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:26.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:26 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:26.120+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:26.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:26 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:26.120+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:26.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:26 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:26.120+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:26.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:26 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:26.120+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:26.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:26 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:26.120+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:27.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:27 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:27.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:27.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:27 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:27.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:27.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:27 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:27.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:27.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:27 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:27.046+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:27.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:27 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:27.046+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:27.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:27 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:27.046+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:27.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:27 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:27.046+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:27.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:27 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:27.087+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:27.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:27 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:27.087+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:27.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:27 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:27.087+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:27.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:27 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:27.087+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:27.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:27 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:27.087+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:27.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:27 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:27.087+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:27.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:27 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:27.087+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:28.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:28 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:28.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:28.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:28 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:28.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:28.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:28 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:28.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:28.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:28 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:28.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:28.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:28 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:28.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:28.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:28 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:28.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:28.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:28 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:28.045+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:28.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:28 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:28.073+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:28.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:28 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:28.073+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:28.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:28 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:28.073+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:28.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:28 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:28.073+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:28.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:28 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:28.073+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:28.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:28 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:28.073+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:28.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:28 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:28.073+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:29.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:29 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:29.065+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:03.580130+0000 front 2024-04-25T16:14:03.579904+0000 (oldest deadline 2024-04-25T16:14:28.279958+0000) 2024-04-25T16:15:29.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:29 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:29.065+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:12.980959+0000 front 2024-04-25T16:14:12.981045+0000 (oldest deadline 2024-04-25T16:14:37.081132+0000) 2024-04-25T16:15:29.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:29 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:29.065+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:24.583544+0000 front 2024-04-25T16:14:24.583578+0000 (oldest deadline 2024-04-25T16:14:49.283252+0000) 2024-04-25T16:15:29.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:29 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:29.065+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6814 osd.3 since back 2024-04-25T16:14:30.384555+0000 front 2024-04-25T16:14:30.384460+0000 (oldest deadline 2024-04-25T16:14:54.484447+0000) 2024-04-25T16:15:29.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:29 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:29.065+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:39.785544+0000 front 2024-04-25T16:14:39.785493+0000 (oldest deadline 2024-04-25T16:15:02.085591+0000) 2024-04-25T16:15:29.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:29 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:29.065+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:45.586849+0000 front 2024-04-25T16:14:45.586832+0000 (oldest deadline 2024-04-25T16:15:09.686554+0000) 2024-04-25T16:15:29.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:29 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10[57744]: 2024-04-25T16:15:29.065+0000 7f91606a7700 -1 osd.10 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:54.987864+0000 front 2024-04-25T16:14:54.987795+0000 (oldest deadline 2024-04-25T16:15:17.887686+0000) 2024-04-25T16:15:29.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:29.089+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:29.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:29.089+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:29.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:29.089+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:29.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:29.089+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:29.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:29.089+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:29.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:29.089+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:29.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:29 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:29.089+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:29.948 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:29 smithi063 podman[68381]: 2024-04-25 16:15:29.691848583 +0000 UTC m=+5.517266601 container died c71d238c17e7de005545e2e20d73f4f8d1b9d5441ac2b5957e8d778424a0d042 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-25T16:15:30.290 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:30 smithi063 podman[68381]: 2024-04-25 16:15:30.04659275 +0000 UTC m=+5.872010767 container cleanup c71d238c17e7de005545e2e20d73f4f8d1b9d5441ac2b5957e8d778424a0d042 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-25T16:15:30.290 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:30 smithi063 bash[68381]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10 2024-04-25T16:15:30.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:30.041+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:30.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:30.041+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:30.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:30.041+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:30.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:30.041+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:30.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:30.041+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:30.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:30.041+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:30.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:30 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:30.041+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:30.620 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:30 smithi063 podman[68400]: 2024-04-25 16:15:30.289341255 +0000 UTC m=+0.591047243 container remove c71d238c17e7de005545e2e20d73f4f8d1b9d5441ac2b5957e8d778424a0d042 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-25T16:15:30.926 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:30 smithi063 podman[68489]: 2024-04-25 16:15:30.526028363 +0000 UTC m=+0.018235310 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:15:30.927 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:30 smithi063 podman[68489]: 2024-04-25 16:15:30.640065075 +0000 UTC m=+0.132272053 container create e5a218e382661b243f8ccdc53b3a33818cd05718351ef2d896b14ce8d556cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-25T16:15:30.927 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:30 smithi063 podman[68489]: 2024-04-25 16:15:30.842898028 +0000 UTC m=+0.335104963 container init e5a218e382661b243f8ccdc53b3a33818cd05718351ef2d896b14ce8d556cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, GIT_BRANCH=HEAD) 2024-04-25T16:15:30.927 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:30 smithi063 podman[68489]: 2024-04-25 16:15:30.845936123 +0000 UTC m=+0.338143063 container start e5a218e382661b243f8ccdc53b3a33818cd05718351ef2d896b14ce8d556cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-25T16:15:30.927 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:30 smithi063 podman[68489]: 2024-04-25 16:15:30.925989521 +0000 UTC m=+0.418196466 container attach e5a218e382661b243f8ccdc53b3a33818cd05718351ef2d896b14ce8d556cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T16:15:31.370 INFO:journalctl@ceph.osd.10.smithi063.stdout:Apr 25 16:15:31 smithi063 podman[68489]: 2024-04-25 16:15:31.147165915 +0000 UTC m=+0.639372839 container died e5a218e382661b243f8ccdc53b3a33818cd05718351ef2d896b14ce8d556cd69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T16:15:31.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:31.077+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:31.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:31.077+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:31.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:31.077+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:31.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:31.077+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:31.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:31.077+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:31.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:31.077+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:31.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:31 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:31.077+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:31.758 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.10.service' 2024-04-25T16:15:31.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:31.798 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-25T16:15:31.798 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-25T16:15:31.798 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.11 2024-04-25T16:15:32.086 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:31 smithi186 systemd[1]: Stopping Ceph osd.11 for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:32.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.084+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:32.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.084+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:32.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.084+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:32.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.084+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:32.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.084+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:32.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.084+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:32.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.084+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:32.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.111+0000 7f84dbaeb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:32.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.112+0000 7f84dbaeb700 -1 osd.11 80 *** Got signal Terminated *** 2024-04-25T16:15:32.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:32 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:32.112+0000 7f84dbaeb700 -1 osd.11 80 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-25T16:15:33.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:33.114+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:33.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:33.114+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:33.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:33.114+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:33.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:33.114+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:33.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:33.114+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:33.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:33.114+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:33.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:33 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:33.114+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:34.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:34.113+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:34.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:34.113+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:34.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:34.113+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:34.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:34.113+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:34.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:34.113+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:34.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:34.113+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:34.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:34 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:34.113+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:35.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:35.072+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:35.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:35.072+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:35.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:35.072+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:35.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:35.072+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:35.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:35.072+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:35.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:35.072+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:35.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:35 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:35.072+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:36.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:36.054+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:36.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:36.054+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:36.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:36.054+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:36.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:36.054+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:36.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:36.054+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:36.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:36.054+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:36.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:36 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:36.054+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:37.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:37.027+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2024-04-25T16:14:06.969840+0000 front 2024-04-25T16:14:06.969626+0000 (oldest deadline 2024-04-25T16:14:28.069630+0000) 2024-04-25T16:15:37.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:37.027+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6804 osd.1 since back 2024-04-25T16:14:13.271421+0000 front 2024-04-25T16:14:13.271540+0000 (oldest deadline 2024-04-25T16:14:36.771337+0000) 2024-04-25T16:15:37.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:37.027+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6804 osd.2 since back 2024-04-25T16:14:22.572735+0000 front 2024-04-25T16:14:22.572789+0000 (oldest deadline 2024-04-25T16:14:48.472877+0000) 2024-04-25T16:15:37.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:37.027+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6812 osd.4 since back 2024-04-25T16:14:38.474920+0000 front 2024-04-25T16:14:38.474935+0000 (oldest deadline 2024-04-25T16:15:02.574713+0000) 2024-04-25T16:15:37.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:37.027+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6812 osd.5 since back 2024-04-25T16:14:46.075837+0000 front 2024-04-25T16:14:46.075867+0000 (oldest deadline 2024-04-25T16:15:11.375875+0000) 2024-04-25T16:15:37.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:37.027+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.31:6822 osd.6 since back 2024-04-25T16:14:55.377531+0000 front 2024-04-25T16:14:55.377462+0000 (oldest deadline 2024-04-25T16:15:20.577988+0000) 2024-04-25T16:15:37.399 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:37.027+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.63:6820 osd.7 since back 2024-04-25T16:15:02.179394+0000 front 2024-04-25T16:15:02.179380+0000 (oldest deadline 2024-04-25T16:15:26.879366+0000) 2024-04-25T16:15:37.400 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11[58812]: 2024-04-25T16:15:37.027+0000 7f84d80a1700 -1 osd.11 80 heartbeat_check: no reply from 172.21.15.186:6820 osd.8 since back 2024-04-25T16:15:11.381162+0000 front 2024-04-25T16:15:11.381213+0000 (oldest deadline 2024-04-25T16:15:36.681269+0000) 2024-04-25T16:15:37.818 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 podman[70431]: 2024-04-25 16:15:37.567527499 +0000 UTC m=+5.575196164 container died 6b73f091e8d93c88e001b46a8d327102e5fadb2d5ea2be030b3bb63f3e261e27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-25T16:15:38.149 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 podman[70431]: 2024-04-25 16:15:37.916884168 +0000 UTC m=+5.924552829 container cleanup 6b73f091e8d93c88e001b46a8d327102e5fadb2d5ea2be030b3bb63f3e261e27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-04-25T16:15:38.149 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:37 smithi186 bash[70431]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11 2024-04-25T16:15:38.538 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:38 smithi186 podman[70447]: 2024-04-25 16:15:38.209845763 +0000 UTC m=+0.638748332 container remove 6b73f091e8d93c88e001b46a8d327102e5fadb2d5ea2be030b3bb63f3e261e27 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-25T16:15:38.848 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:38 smithi186 podman[70539]: 2024-04-25 16:15:38.439888736 +0000 UTC m=+0.018143005 image pull d53c96779a29b1ca3ec780b8b69a1863c25b0e59d8e476272e056726cdf012a4 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385 2024-04-25T16:15:38.848 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:38 smithi186 podman[70539]: 2024-04-25 16:15:38.569110725 +0000 UTC m=+0.147365056 container create d2631ac3cac423e8c5a60192a7b879a11514609d6cca723999eab0d50d916f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-25T16:15:38.848 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:38 smithi186 podman[70539]: 2024-04-25 16:15:38.771684877 +0000 UTC m=+0.349939147 container init d2631ac3cac423e8c5a60192a7b879a11514609d6cca723999eab0d50d916f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:15:38.848 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:38 smithi186 podman[70539]: 2024-04-25 16:15:38.774731318 +0000 UTC m=+0.352985590 container start d2631ac3cac423e8c5a60192a7b879a11514609d6cca723999eab0d50d916f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-25T16:15:38.848 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:38 smithi186 podman[70539]: 2024-04-25 16:15:38.846729424 +0000 UTC m=+0.424983696 container attach d2631ac3cac423e8c5a60192a7b879a11514609d6cca723999eab0d50d916f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-25T16:15:39.149 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:39 smithi186 podman[70539]: 2024-04-25 16:15:39.071969421 +0000 UTC m=+0.650223691 container died d2631ac3cac423e8c5a60192a7b879a11514609d6cca723999eab0d50d916f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-25T16:15:39.609 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:39 smithi186 podman[70565]: 2024-04-25 16:15:39.591742093 +0000 UTC m=+0.513659481 container remove d2631ac3cac423e8c5a60192a7b879a11514609d6cca723999eab0d50d916f1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-25T16:15:39.609 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:39 smithi186 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.11.service: Deactivated successfully. 2024-04-25T16:15:39.609 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:39 smithi186 systemd[1]: Stopped Ceph osd.11 for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T16:15:39.609 INFO:journalctl@ceph.osd.11.smithi186.stdout:Apr 25 16:15:39 smithi186 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.11.service: Consumed 8min 24.953s CPU time. 2024-04-25T16:15:39.610 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@osd.11.service' 2024-04-25T16:15:39.649 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:39.650 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-25T16:15:39.650 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-25T16:15:39.650 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.a 2024-04-25T16:15:39.950 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:15:39 smithi031 systemd[1]: Stopping Ceph mds.a for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:40.450 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:15:40 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a[71143]: 2024-04-25T16:15:40.065+0000 7f96c9611700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:40.450 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:15:40 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a[71143]: 2024-04-25T16:15:40.065+0000 7f96c9611700 -1 mds.a *** got signal Terminated *** 2024-04-25T16:15:42.517 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:15:42 smithi031 podman[99830]: 2024-04-25 16:15:42.216504208 +0000 UTC m=+2.363839463 container died 98c0e7e3c52e760fafe76452976403b380cd01a5c6c367425227d925ad804b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-25T16:15:42.950 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:15:42 smithi031 podman[99830]: 2024-04-25 16:15:42.598688413 +0000 UTC m=+2.746023666 container cleanup 98c0e7e3c52e760fafe76452976403b380cd01a5c6c367425227d925ad804b78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:15:42.950 INFO:journalctl@ceph.mds.a.smithi031.stdout:Apr 25 16:15:42 smithi031 bash[99830]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-a 2024-04-25T16:15:43.187 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.a.service' 2024-04-25T16:15:43.223 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:43.223 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-25T16:15:43.223 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-25T16:15:43.223 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.d 2024-04-25T16:15:43.561 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 16:15:43 smithi031 systemd[1]: Stopping Ceph mds.d for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:43.950 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 16:15:43 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-d[70146]: 2024-04-25T16:15:43.560+0000 7fa76bee9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:43.950 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 16:15:43 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-d[70146]: 2024-04-25T16:15:43.560+0000 7fa76bee9700 -1 mds.d *** got signal Terminated *** 2024-04-25T16:15:46.937 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 16:15:46 smithi031 podman[99949]: 2024-04-25 16:15:46.645862618 +0000 UTC m=+3.195108679 container died 625af22b8d0cedd6dbeaee5d46a6a77056058a46f7f0a3a68ce637a9196e76bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD) 2024-04-25T16:15:47.200 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 16:15:47 smithi031 podman[99949]: 2024-04-25 16:15:47.035389958 +0000 UTC m=+3.584636022 container cleanup 625af22b8d0cedd6dbeaee5d46a6a77056058a46f7f0a3a68ce637a9196e76bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T16:15:47.200 INFO:journalctl@ceph.mds.d.smithi031.stdout:Apr 25 16:15:47 smithi031 bash[99949]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-d 2024-04-25T16:15:47.482 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.d.service' 2024-04-25T16:15:47.515 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:47.515 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-25T16:15:47.515 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-25T16:15:47.515 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.g 2024-04-25T16:15:47.865 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:15:47 smithi031 systemd[1]: Stopping Ceph mds.g for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:48.200 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:15:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g[70756]: 2024-04-25T16:15:47.863+0000 7fb0263eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:48.200 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:15:47 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g[70756]: 2024-04-25T16:15:47.863+0000 7fb0263eb700 -1 mds.g *** got signal Terminated *** 2024-04-25T16:15:50.499 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:15:50 smithi031 podman[100071]: 2024-04-25 16:15:50.225727634 +0000 UTC m=+2.485845248 container died ad720772e5c1ecbe794667350dbd088298d8d89ccac9d7f108750223e0792bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=) 2024-04-25T16:15:50.841 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:15:50 smithi031 podman[100071]: 2024-04-25 16:15:50.614314797 +0000 UTC m=+2.874432423 container cleanup ad720772e5c1ecbe794667350dbd088298d8d89ccac9d7f108750223e0792bb5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:15:50.841 INFO:journalctl@ceph.mds.g.smithi031.stdout:Apr 25 16:15:50 smithi031 bash[100071]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-g 2024-04-25T16:15:50.969 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.g.service' 2024-04-25T16:15:51.001 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:51.001 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-25T16:15:51.002 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-25T16:15:51.002 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.j 2024-04-25T16:15:51.200 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:51 smithi031 systemd[1]: Stopping Ceph mds.j for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:51.700 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j[69797]: 2024-04-25T16:15:51.316+0000 7fd82d10c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:51.700 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:51 smithi031 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j[69797]: 2024-04-25T16:15:51.316+0000 7fd82d10c700 -1 mds.j *** got signal Terminated *** 2024-04-25T16:15:55.602 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:55 smithi031 podman[100192]: 2024-04-25 16:15:55.324045364 +0000 UTC m=+4.104008960 container died 43107973a76c1b3991bc7af971f9d9fd2c5120391a98a0f7d04ca07c493eb629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:15:55.886 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:55 smithi031 podman[100192]: 2024-04-25 16:15:55.70079126 +0000 UTC m=+4.480754854 container cleanup 43107973a76c1b3991bc7af971f9d9fd2c5120391a98a0f7d04ca07c493eb629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:15:55.886 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:55 smithi031 bash[100192]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j 2024-04-25T16:15:56.116 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.j.service' 2024-04-25T16:15:56.137 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:55 smithi031 podman[100207]: 2024-04-25 16:15:55.993530383 +0000 UTC m=+0.654668079 container remove 43107973a76c1b3991bc7af971f9d9fd2c5120391a98a0f7d04ca07c493eb629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-j, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:15:56.137 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:56 smithi031 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.j.service: Deactivated successfully. 2024-04-25T16:15:56.137 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:56 smithi031 systemd[1]: Stopped Ceph mds.j for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T16:15:56.137 INFO:journalctl@ceph.mds.j.smithi031.stdout:Apr 25 16:15:56 smithi031 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.j.service: Consumed 1.177s CPU time. 2024-04-25T16:15:56.147 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:56.147 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-25T16:15:56.147 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-25T16:15:56.148 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.b 2024-04-25T16:15:56.446 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:15:56 smithi063 systemd[1]: Stopping Ceph mds.b for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:56.869 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:15:56 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: 2024-04-25T16:15:56.447+0000 7fd2966ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:56.870 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:15:56 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b[62959]: 2024-04-25T16:15:56.447+0000 7fd2966ea700 -1 mds.b *** got signal Terminated *** 2024-04-25T16:15:58.523 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:15:58 smithi063 podman[68594]: 2024-04-25 16:15:58.203479636 +0000 UTC m=+1.859913815 container died 497d4c2b60ca14b9ddc7b51803b7ee3e90fb717b6aeeb3d6d1ea7f7bbe8c9894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-25T16:15:58.870 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:15:58 smithi063 podman[68594]: 2024-04-25 16:15:58.522680424 +0000 UTC m=+2.179114620 container cleanup 497d4c2b60ca14b9ddc7b51803b7ee3e90fb717b6aeeb3d6d1ea7f7bbe8c9894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-25T16:15:58.870 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:15:58 smithi063 bash[68594]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b 2024-04-25T16:15:58.870 INFO:journalctl@ceph.mds.b.smithi063.stdout:Apr 25 16:15:58 smithi063 podman[68619]: 2024-04-25 16:15:58.740613154 +0000 UTC m=+0.541165247 container remove 497d4c2b60ca14b9ddc7b51803b7ee3e90fb717b6aeeb3d6d1ea7f7bbe8c9894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-04-25T16:15:58.958 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.b.service' 2024-04-25T16:15:58.995 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:15:58.995 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-25T16:15:58.995 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-25T16:15:58.996 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.e 2024-04-25T16:15:59.319 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:15:59 smithi063 systemd[1]: Stopping Ceph mds.e for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:15:59.620 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:15:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e[62343]: 2024-04-25T16:15:59.318+0000 7f487b0c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:15:59.620 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:15:59 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e[62343]: 2024-04-25T16:15:59.318+0000 7f487b0c1700 -1 mds.e *** got signal Terminated *** 2024-04-25T16:16:02.469 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:16:02 smithi063 podman[68726]: 2024-04-25 16:16:02.200638442 +0000 UTC m=+2.988478619 container died 5d6e749ec588691d3c8cc5e75885914b8fea85c3b3bbc7cdb27d65db30a96479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-25T16:16:02.761 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:16:02 smithi063 podman[68726]: 2024-04-25 16:16:02.543042539 +0000 UTC m=+3.330882716 container cleanup 5d6e749ec588691d3c8cc5e75885914b8fea85c3b3bbc7cdb27d65db30a96479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-25T16:16:02.762 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:16:02 smithi063 bash[68726]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e 2024-04-25T16:16:02.995 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.e.service' 2024-04-25T16:16:03.016 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:16:02 smithi063 podman[68740]: 2024-04-25 16:16:02.760650828 +0000 UTC m=+0.556105739 container remove 5d6e749ec588691d3c8cc5e75885914b8fea85c3b3bbc7cdb27d65db30a96479 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:16:03.016 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:16:02 smithi063 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.e.service: Deactivated successfully. 2024-04-25T16:16:03.016 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:16:02 smithi063 systemd[1]: Stopped Ceph mds.e for 74fd3742-031b-11ef-bc93-c7b262605968. 2024-04-25T16:16:03.016 INFO:journalctl@ceph.mds.e.smithi063.stdout:Apr 25 16:16:02 smithi063 systemd[1]: ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.e.service: Consumed 2.391s CPU time. 2024-04-25T16:16:03.028 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:16:03.028 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-25T16:16:03.028 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-25T16:16:03.029 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.h 2024-04-25T16:16:03.356 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:16:03 smithi063 systemd[1]: Stopping Ceph mds.h for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:16:03.619 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:16:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h[62651]: 2024-04-25T16:16:03.354+0000 7f965ddde700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:16:03.620 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:16:03 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h[62651]: 2024-04-25T16:16:03.354+0000 7f965ddde700 -1 mds.h *** got signal Terminated *** 2024-04-25T16:16:06.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:16:06 smithi063 podman[68848]: 2024-04-25 16:16:06.20030476 +0000 UTC m=+2.956033433 container died 09bf47fc2733c8196aec808b0035b4001e13e8b2a721b9582c5251ea46dc5ec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-25T16:16:06.532 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:16:06 smithi063 podman[68848]: 2024-04-25 16:16:06.531735939 +0000 UTC m=+3.287464621 container cleanup 09bf47fc2733c8196aec808b0035b4001e13e8b2a721b9582c5251ea46dc5ec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-25T16:16:06.869 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:16:06 smithi063 bash[68848]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h 2024-04-25T16:16:06.870 INFO:journalctl@ceph.mds.h.smithi063.stdout:Apr 25 16:16:06 smithi063 podman[68862]: 2024-04-25 16:16:06.741183688 +0000 UTC m=+0.539820368 container remove 09bf47fc2733c8196aec808b0035b4001e13e8b2a721b9582c5251ea46dc5ec7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-h, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, ceph=True) 2024-04-25T16:16:06.967 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.h.service' 2024-04-25T16:16:07.000 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:16:07.000 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-25T16:16:07.000 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-25T16:16:07.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.k 2024-04-25T16:16:07.328 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:16:07 smithi063 systemd[1]: Stopping Ceph mds.k for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:16:07.619 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:16:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k[62024]: 2024-04-25T16:16:07.326+0000 7f89671b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:16:07.620 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:16:07 smithi063 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k[62024]: 2024-04-25T16:16:07.326+0000 7f89671b9700 -1 mds.k *** got signal Terminated *** 2024-04-25T16:16:10.461 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:16:10 smithi063 podman[68969]: 2024-04-25 16:16:10.199574315 +0000 UTC m=+2.980855079 container died 184ad245f379892aa90c82db4d8f6ca103857c91743809d920d3f33f685790b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-25T16:16:10.828 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:16:10 smithi063 podman[68969]: 2024-04-25 16:16:10.560085817 +0000 UTC m=+3.341366584 container cleanup 184ad245f379892aa90c82db4d8f6ca103857c91743809d920d3f33f685790b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-25T16:16:10.828 INFO:journalctl@ceph.mds.k.smithi063.stdout:Apr 25 16:16:10 smithi063 bash[68969]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-k 2024-04-25T16:16:11.000 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.k.service' 2024-04-25T16:16:11.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:16:11.033 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-25T16:16:11.033 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-25T16:16:11.033 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.c 2024-04-25T16:16:11.399 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:16:11 smithi186 systemd[1]: Stopping Ceph mds.c for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:16:11.666 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:16:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c[64212]: 2024-04-25T16:16:11.399+0000 7f27bb776700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:16:11.666 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:16:11 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c[64212]: 2024-04-25T16:16:11.399+0000 7f27bb776700 -1 mds.c *** got signal Terminated *** 2024-04-25T16:16:14.418 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:16:14 smithi186 podman[70645]: 2024-04-25 16:16:14.158190654 +0000 UTC m=+2.930313699 container died 0d441945cd3360d707634244da39c99453b4833afbce40849d48abfe41ef7844 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-25T16:16:14.809 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:16:14 smithi186 podman[70645]: 2024-04-25 16:16:14.516434783 +0000 UTC m=+3.288557828 container cleanup 0d441945cd3360d707634244da39c99453b4833afbce40849d48abfe41ef7844 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-25T16:16:14.810 INFO:journalctl@ceph.mds.c.smithi186.stdout:Apr 25 16:16:14 smithi186 bash[70645]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-c 2024-04-25T16:16:14.947 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.c.service' 2024-04-25T16:16:14.982 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:16:14.982 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-25T16:16:14.982 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-25T16:16:14.983 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.f 2024-04-25T16:16:15.399 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:16:15 smithi186 systemd[1]: Stopping Ceph mds.f for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:16:15.399 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:16:15 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f[63901]: 2024-04-25T16:16:15.295+0000 7f82d8321700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:16:15.399 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:16:15 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f[63901]: 2024-04-25T16:16:15.295+0000 7f82d8321700 -1 mds.f *** got signal Terminated *** 2024-04-25T16:16:18.422 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:16:18 smithi186 podman[70779]: 2024-04-25 16:16:18.16482964 +0000 UTC m=+2.966673457 container died 2d7978c8c71f54cf6c87c51bbf100fcdd151df00c0a8f57cc70ad9fbeb8a774a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-25T16:16:18.714 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:16:18 smithi186 podman[70779]: 2024-04-25 16:16:18.512134235 +0000 UTC m=+3.313978058 container cleanup 2d7978c8c71f54cf6c87c51bbf100fcdd151df00c0a8f57cc70ad9fbeb8a774a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-25T16:16:18.714 INFO:journalctl@ceph.mds.f.smithi186.stdout:Apr 25 16:16:18 smithi186 bash[70779]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-f 2024-04-25T16:16:18.928 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.f.service' 2024-04-25T16:16:18.959 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:16:18.960 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-25T16:16:18.960 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-25T16:16:18.960 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.i 2024-04-25T16:16:19.275 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:16:19 smithi186 systemd[1]: Stopping Ceph mds.i for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:16:19.649 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:16:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i[64519]: 2024-04-25T16:16:19.273+0000 7fa1be086700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:16:19.649 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:16:19 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i[64519]: 2024-04-25T16:16:19.273+0000 7fa1be086700 -1 mds.i *** got signal Terminated *** 2024-04-25T16:16:22.399 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:16:22 smithi186 podman[70899]: 2024-04-25 16:16:22.163031735 +0000 UTC m=+2.991718144 container died 55b78effe522dd14a8bdce313f11226064acb33cd85d53dc9a85798c61aeebea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2) 2024-04-25T16:16:22.776 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:16:22 smithi186 podman[70899]: 2024-04-25 16:16:22.499359028 +0000 UTC m=+3.328045443 container cleanup 55b78effe522dd14a8bdce313f11226064acb33cd85d53dc9a85798c61aeebea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:16:22.776 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:16:22 smithi186 bash[70899]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i 2024-04-25T16:16:23.059 INFO:journalctl@ceph.mds.i.smithi186.stdout:Apr 25 16:16:22 smithi186 podman[70913]: 2024-04-25 16:16:22.775769739 +0000 UTC m=+0.609872956 container remove 55b78effe522dd14a8bdce313f11226064acb33cd85d53dc9a85798c61aeebea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-25T16:16:23.081 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.i.service' 2024-04-25T16:16:23.112 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:16:23.113 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-25T16:16:23.113 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-25T16:16:23.113 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.l 2024-04-25T16:16:23.399 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:16:23 smithi186 systemd[1]: Stopping Ceph mds.l for 74fd3742-031b-11ef-bc93-c7b262605968... 2024-04-25T16:16:23.899 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:16:23 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-l[63593]: 2024-04-25T16:16:23.427+0000 7fa52fa36700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-25T16:16:23.899 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:16:23 smithi186 ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-l[63593]: 2024-04-25T16:16:23.427+0000 7fa52fa36700 -1 mds.l *** got signal Terminated *** 2024-04-25T16:16:26.450 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:16:26 smithi186 podman[71022]: 2024-04-25 16:16:26.16603385 +0000 UTC m=+2.841546836 container died 6116e03527e8a20c350c723ae3e12f4841711e3af70e7c2f65c89fa4b1a8f31f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-04-25T16:16:26.717 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:16:26 smithi186 podman[71022]: 2024-04-25 16:16:26.553372434 +0000 UTC m=+3.228885428 container cleanup 6116e03527e8a20c350c723ae3e12f4841711e3af70e7c2f65c89fa4b1a8f31f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:c1dfc10be9702848c35bd11791796a771cb431046d56ece349f8a36761441385, name=ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=wip-pdonnell-testing-20240425.015853-debug-994e2f2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-25T16:16:26.717 INFO:journalctl@ceph.mds.l.smithi186.stdout:Apr 25 16:16:26 smithi186 bash[71022]: ceph-74fd3742-031b-11ef-bc93-c7b262605968-mds-l 2024-04-25T16:16:26.919 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-74fd3742-031b-11ef-bc93-c7b262605968@mds.l.service' 2024-04-25T16:16:26.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-25T16:16:26.951 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-25T16:16:26.951 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74fd3742-031b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T16:16:27.153 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T16:16:29.699 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74fd3742-031b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T16:16:29.898 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T16:16:32.553 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74fd3742-031b-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-25T16:16:32.743 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T16:16:35.175 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:16:35.208 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:16:35.241 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-25T16:16:35.267 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-25T16:16:35.269 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/remote/smithi031/crash 2024-04-25T16:16:35.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/crash -- . 2024-04-25T16:16:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T16:16:35.304 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2024-04-25T16:16:35.306 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/remote/smithi063/crash 2024-04-25T16:16:35.307 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/crash -- . 2024-04-25T16:16:35.349 INFO:teuthology.orchestra.run.smithi063.stderr:tar: /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T16:16:35.349 INFO:teuthology.orchestra.run.smithi063.stderr:tar: Error is not recoverable: exiting now 2024-04-25T16:16:35.351 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/remote/smithi186/crash 2024-04-25T16:16:35.352 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/crash -- . 2024-04-25T16:16:35.385 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/74fd3742-031b-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-25T16:16:35.385 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-04-25T16:16:35.386 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-25T16:16:35.386 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-25T16:16:35.474 INFO:tasks.cephadm:Compressing logs... 2024-04-25T16:16:35.474 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T16:16:35.482 DEBUG:teuthology.orchestra.run.smithi063:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T16:16:35.487 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-25T16:16:35.514 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T16:16:35.516 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log 2024-04-25T16:16:35.517 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T16:16:35.517 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-25T16:16:35.517 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-25T16:16:35.517 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-25T16:16:35.520 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-25T16:16:35.520 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-25T16:16:35.520 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-25T16:16:35.521 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-25T16:16:35.521 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T16:16:35.521 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-25T16:16:35.521 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T16:16:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-25T16:16:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-25T16:16:35.522 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T16:16:35.523 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T16:16:35.525 INFO:teuthology.orchestra.run.smithi063.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-25T16:16:35.525 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log 2024-04-25T16:16:35.525 INFO:teuthology.orchestra.run.smithi063.stderr:gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T16:16:35.525 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-25T16:16:35.526 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-25T16:16:35.529 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log 2024-04-25T16:16:35.529 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-25T16:16:35.529 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-25T16:16:35.530 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-25T16:16:35.530 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-25T16:16:35.537 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.audit.log: --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-25T16:16:35.537 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-25T16:16:35.537 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-25T16:16:35.537 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-25T16:16:35.537 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T16:16:35.537 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-25T16:16:35.537 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-25T16:16:35.537 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T16:16:35.537 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-25T16:16:35.538 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T16:16:35.540 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-25T16:16:35.541 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-25T16:16:35.563 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/cephadm.log: 92.7% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-25T16:16:35.601 INFO:teuthology.orchestra.run.smithi063.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 82.3% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T16:16:35.647 INFO:teuthology.orchestra.run.smithi031.stderr: 87.0% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T16:16:35.648 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 93.0% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-25T16:16:35.652 INFO:teuthology.orchestra.run.smithi063.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T16:16:35.671 INFO:teuthology.orchestra.run.smithi063.stderr: 92.9% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-25T16:16:35.726 INFO:teuthology.orchestra.run.smithi186.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T16:16:35.769 INFO:teuthology.orchestra.run.smithi031.stderr: 91.1% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T16:16:35.771 INFO:teuthology.orchestra.run.smithi031.stderr: 82.7% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T16:16:35.776 INFO:teuthology.orchestra.run.smithi031.stderr: 94.5% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-25T16:16:35.818 INFO:teuthology.orchestra.run.smithi031.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-25T16:16:35.823 INFO:teuthology.orchestra.run.smithi186.stderr: 94.4% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-25T16:16:35.841 INFO:teuthology.orchestra.run.smithi186.stderr: 87.0% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T16:16:35.841 INFO:teuthology.orchestra.run.smithi186.stderr: 82.2% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-25T16:16:35.893 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-25T16:16:35.914 INFO:teuthology.orchestra.run.smithi186.stderr: 91.0% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T16:16:35.918 INFO:teuthology.orchestra.run.smithi063.stderr: 92.6% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-25T16:16:35.945 INFO:teuthology.orchestra.run.smithi063.stderr: 91.0% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-25T16:16:35.963 INFO:teuthology.orchestra.run.smithi063.stderr: 87.0% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-25T16:16:35.965 INFO:teuthology.orchestra.run.smithi186.stderr: 90.4% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T16:16:35.984 INFO:teuthology.orchestra.run.smithi063.stderr: 90.4% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T16:16:36.086 INFO:teuthology.orchestra.run.smithi186.stderr: 92.7% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-25T16:16:36.132 INFO:teuthology.orchestra.run.smithi031.stderr: 90.3% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-25T16:16:36.433 INFO:teuthology.orchestra.run.smithi063.stderr: 94.5% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-25T16:16:36.433 INFO:teuthology.orchestra.run.smithi063.stderr: 94.4% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-25T16:16:36.472 INFO:teuthology.orchestra.run.smithi031.stderr: 94.4% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-25T16:16:36.554 INFO:teuthology.orchestra.run.smithi186.stderr: 92.7% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-25T16:16:37.595 INFO:teuthology.orchestra.run.smithi031.stderr: 90.3% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-25T16:16:37.983 INFO:teuthology.orchestra.run.smithi186.stderr: 91.7% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-25T16:16:41.246 INFO:teuthology.orchestra.run.smithi063.stderr: 91.7% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-25T16:16:52.897 INFO:teuthology.orchestra.run.smithi031.stderr: 90.2% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-25T16:17:29.451 INFO:teuthology.orchestra.run.smithi186.stderr: 93.3% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-25T16:17:54.809 INFO:teuthology.orchestra.run.smithi186.stderr: 97.4% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-25T16:18:28.263 INFO:teuthology.orchestra.run.smithi186.stderr: 93.4% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-25T16:19:31.008 INFO:teuthology.orchestra.run.smithi031.stderr: 93.2% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-25T16:19:37.097 INFO:teuthology.orchestra.run.smithi186.stderr: 93.3% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-25T16:19:43.156 INFO:teuthology.orchestra.run.smithi186.stderr: 93.3% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-25T16:19:43.159 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-04-25T16:19:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:real 3m7.655s 2024-04-25T16:19:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:user 4m22.935s 2024-04-25T16:19:43.159 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m11.020s 2024-04-25T16:20:28.614 INFO:teuthology.orchestra.run.smithi031.stderr: 93.3% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-25T16:21:11.916 INFO:teuthology.orchestra.run.smithi031.stderr: 93.2% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-25T16:21:30.595 INFO:teuthology.orchestra.run.smithi031.stderr: 93.2% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-25T16:21:30.598 INFO:teuthology.orchestra.run.smithi031.stderr: 2024-04-25T16:21:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:real 4m55.103s 2024-04-25T16:21:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:user 4m22.688s 2024-04-25T16:21:30.598 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m10.471s 2024-04-25T16:28:53.050 INFO:teuthology.orchestra.run.smithi063.stderr: 93.2% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-25T16:29:38.733 INFO:teuthology.orchestra.run.smithi063.stderr: 93.4% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-25T16:33:35.921 INFO:teuthology.orchestra.run.smithi063.stderr: 93.3% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-25T16:34:54.302 INFO:teuthology.orchestra.run.smithi063.stderr: 93.3% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-25T16:37:13.329 INFO:teuthology.orchestra.run.smithi063.stderr: 94.2% -- replaced with /var/log/ceph/74fd3742-031b-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-25T16:37:13.330 INFO:teuthology.orchestra.run.smithi063.stderr: 2024-04-25T16:37:13.330 INFO:teuthology.orchestra.run.smithi063.stderr:real 20m37.831s 2024-04-25T16:37:13.330 INFO:teuthology.orchestra.run.smithi063.stderr:user 8m0.320s 2024-04-25T16:37:13.330 INFO:teuthology.orchestra.run.smithi063.stderr:sys 0m23.922s 2024-04-25T16:37:13.331 INFO:tasks.cephadm:Archiving logs... 2024-04-25T16:37:13.332 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/remote/smithi031/log 2024-04-25T16:37:13.333 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T16:37:53.635 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/remote/smithi063/log 2024-04-25T16:37:53.636 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T16:39:00.565 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/remote/smithi186/log 2024-04-25T16:39:00.567 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-25T16:39:42.264 INFO:tasks.cephadm:Removing cluster... 2024-04-25T16:39:42.265 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74fd3742-031b-11ef-bc93-c7b262605968 --force 2024-04-25T16:39:42.466 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T16:39:43.024 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74fd3742-031b-11ef-bc93-c7b262605968 --force 2024-04-25T16:39:43.228 INFO:teuthology.orchestra.run.smithi063.stdout:Deleting cluster with fsid: 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T16:39:43.987 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 74fd3742-031b-11ef-bc93-c7b262605968 --force 2024-04-25T16:39:44.167 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 74fd3742-031b-11ef-bc93-c7b262605968 2024-04-25T16:39:44.739 INFO:tasks.cephadm:Removing cephadm ... 2024-04-25T16:39:44.739 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T16:39:44.758 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T16:39:44.776 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-25T16:39:44.792 INFO:tasks.cephadm:Teardown complete 2024-04-25T16:39:44.792 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T16:39:44.808 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T16:39:44.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T16:39:44.811 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T16:39:44.818 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T16:39:44.931 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T16:39:44.941 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T16:39:44.941 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T16:39:44.944 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T16:39:44.946 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T16:39:44.957 INFO:teuthology.orchestra.run.smithi031.stderr:bash: line 1: ntpq: command not found 2024-04-25T16:39:44.959 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-04-25T16:39:44.960 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-04-25T16:39:44.962 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T16:39:44.962 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2024-04-25T16:39:44.962 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv01.front.sepia.ceph.com 3 6 377 50 +904us[ +902us] +/- 41ms 2024-04-25T16:39:44.962 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 45 -1107us[-1107us] +/- 64ms 2024-04-25T16:39:44.962 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 -387us[ -387us] +/- 88ms 2024-04-25T16:39:44.962 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:39:44.971 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T16:39:44.972 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-04-25T16:39:44.972 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 45 +917us[ +916us] +/- 42ms 2024-04-25T16:39:44.972 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 48 -1092us[-1088us] +/- 64ms 2024-04-25T16:39:44.972 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 -383us[ -383us] +/- 88ms 2024-04-25T16:39:44.972 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:39:44.985 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-25T16:39:44.986 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-04-25T16:39:44.986 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv01.front.sepia.ceph.com 3 6 377 46 +921us[ +921us] +/- 41ms 2024-04-25T16:39:44.986 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -1089us[-1090us] +/- 64ms 2024-04-25T16:39:44.986 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -390us[ -390us] +/- 88ms 2024-04-25T16:39:44.986 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-25T16:39:44.987 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T16:39:44.997 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T16:39:44.998 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T16:39:45.008 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T16:39:45.144 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T16:39:45.248 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-25T16:39:45.329 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T16:39:45.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:39:45.360 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T16:39:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:39:45.390 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-25T16:39:45.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:39:45.417 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T16:39:45.428 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T16:39:45.438 INFO:teuthology.task.internal:Duration was 3326.436894 seconds 2024-04-25T16:39:45.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T16:39:45.448 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T16:39:45.448 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T16:39:45.451 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T16:39:45.453 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T16:39:45.512 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T16:39:45.513 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T16:39:45.516 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-25T16:39:45.912 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T16:39:45.913 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2024-04-25T16:39:45.913 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T16:39:45.936 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-04-25T16:39:45.936 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T16:39:45.966 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-04-25T16:39:45.966 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T16:39:45.995 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T16:39:45.996 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T16:39:45.999 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T16:39:46.009 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T16:39:46.061 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T16:39:46.061 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T16:39:46.069 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T16:39:46.078 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T16:39:46.478 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T16:39:46.490 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T16:39:46.490 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T16:39:46.531 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T16:39:46.565 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T16:39:46.607 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T16:39:46.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T16:39:46.621 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T16:39:46.623 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T16:39:46.645 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2024-04-25T16:39:46.649 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-04-25T16:39:46.681 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-04-25T16:39:46.696 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T16:39:46.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:39:46.714 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T16:39:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:39:46.729 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T16:39:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T16:39:46.750 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T16:39:46.762 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T16:39:46.763 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/remote/smithi031 2024-04-25T16:39:46.763 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T16:39:46.805 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/remote/smithi063 2024-04-25T16:39:46.805 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T16:39:46.841 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-25_10:00:23-fs-wip-pdonnell-testing-20240425.015853-debug-distro-default-smithi/7673235/remote/smithi186 2024-04-25T16:39:46.841 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T16:39:46.876 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T16:39:46.876 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T16:39:46.879 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T16:39:46.881 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T16:39:46.925 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T16:39:46.938 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T16:39:46.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T16:39:46.949 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T16:39:46.949 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T16:39:46.951 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T16:39:46.954 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T16:39:46.964 INFO:teuthology.orchestra.run.smithi031.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 16:39 /home/ubuntu/cephtest 2024-04-25T16:39:46.966 INFO:teuthology.orchestra.run.smithi063.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 16:39 /home/ubuntu/cephtest 2024-04-25T16:39:46.979 INFO:teuthology.orchestra.run.smithi186.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 16:39 /home/ubuntu/cephtest 2024-04-25T16:39:46.981 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T16:39:46.992 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T16:39:47.017 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/iogen}} duration: 3326.4368937015533 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-04-25T16:39:47.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T16:39:47.098 INFO:teuthology.run:pass